DEV Community

Chris White
Chris White

Posted on

Private Python Packages With devpi

#python #tutorial #programming

There are cases where you want the flexibility of installing a python package via pip without having it available to the open public. This article will focus on using devpi to provide a self-hosted pip compatible python package server. Ubuntu will be used for the OS as it's a fairly common Linux distribution and easily available on Windows Linux Subsystem.

Install

Before beginning we want to make sure that devpi will run as an actual user and not the root process for security purposes:

$ sudo useradd -m -s /bin/bash devpi
Enter fullscreen mode Exit fullscreen mode

Now I'll switch to that user

$ sudo su - devpi
Enter fullscreen mode Exit fullscreen mode

Next is to create a virtual environment so devpi and its packages don't clutter the global namespace (you may need to apt-get install python3-virtualenv for this to work):

$ virtualenv --python=python3.8 venv
created virtual environment CPython3.8.10.final.0-64 in 123ms
  creator CPython3Posix(dest=/home/devpi/venv, clear=False, global=False)
  seeder FromAppData(download=False, pip=latest, setuptools=latest, wheel=latest, pkg_resources=latest, via=copy, app_data_dir=/home/devpi/.local/share/virtualenv/seed-app-data/v1.0.1.debian.1)
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator
$ src venv/bin/activate
Enter fullscreen mode Exit fullscreen mode

Finally installation of the devpi server:

$ pip install devpi-server devpi-web
Enter fullscreen mode Exit fullscreen mode

While devpi-web isn't a hard requirement it does enable better visualization of python packages and allows for pip search queries to work. Now we'll do install validation and initialize the server:

# devpi-server --version
6.9.0
# devpi-init
INFO  NOCTX Loading node info from /root/.devpi/server/.nodeinfo
INFO  NOCTX generated uuid: b7b5b93272124964b10bdb37cce942ad
INFO  NOCTX wrote nodeinfo to: /root/.devpi/server/.nodeinfo
INFO  NOCTX DB: Creating schema
INFO  [Wtx-1] setting password for user 'root'
INFO  [Wtx-1] created user 'root'
INFO  [Wtx-1] created root user
INFO  [Wtx-1] created root/pypi index
INFO  [Wtx-1] fswriter0: committed at 0
Enter fullscreen mode Exit fullscreen mode

This is setting up the basic backend for the devpi server to work. It's also creating a root user for later administrative tasks. Next up we'll need to create files which provide several options to run the devpi server:

# devpi-gen-config
It is highly recommended to use a configuration file for devpi-server, see --configfile option.
wrote gen-config/crontab
wrote gen-config/net.devpi.plist
wrote gen-config/launchd-macos.txt
wrote gen-config/nginx-devpi.conf
wrote gen-config/nginx-devpi-caching.conf
wrote gen-config/supervisor-devpi.conf
wrote gen-config/supervisord.conf
wrote gen-config/devpi.service
wrote gen-config/windows-service.txt
Enter fullscreen mode Exit fullscreen mode

A port can also be provided during execution if the default of 3141 doesn't suit you:

# devpi-gen-config --port 4040
Enter fullscreen mode Exit fullscreen mode

As you can see there are several methods of running devpi server including cron, launchd (OSX service), nginx, Windows service, and supervisord. It also has a systemd service file which we can use to manage the service easily as Ubuntu uses it for primary service management. First off though we're going to need a proxy script to ensure that devpi is running in the virtual environment:

/home/devpi/start-devpi.sh

#!/bin/bash
cd $HOME
source venv/bin/activate
devpi-server --restrict-modify=root
Enter fullscreen mode Exit fullscreen mode

--restrict-modify=root makes it so that the root user is needed to make administrative changes such as index and user creation. Then modify it to be executable by systemd:

$ chmod u+x start-devpi.sh
Enter fullscreen mode Exit fullscreen mode

Finally we'll adjust the systemd service file:

gen-config/devpi.service

[Unit]
Description=Devpi Server
Requires=network-online.target
After=network-online.target

[Service]
Restart=on-success
# ExecStart:
# - shall point to existing devpi-server executable
# - shall not use the deprecated `--start`. We want the devpi-server to start in foreground
ExecStart=/home/devpi/start-devpi.sh
# set User according to user which is able to run the devpi-server
User=devpi

[Install]
WantedBy=multi-user.target
Enter fullscreen mode Exit fullscreen mode

Now exit out of the devpi user to return back to the normal user you work on Ubuntu with. Then go ahead and:

$ sudo cp /home/devpi/gen-config/devpi.service /etc/systemd/system/
$ sudo systemctl enable devpi.service
Created symlink /etc/systemd/system/multi-user.target.wants/devpi.service → /etc/systemd/system/devpi.service
$ sudo systemctl start devpi.service
Enter fullscreen mode Exit fullscreen mode

Now the service is available on http://localhost:3141 by default unless the port was changed:

view of the devpi web interface to validate installation.

User Management

Before beginning we'll need to install the client which manages everything:

$ pip install devpi-client
Enter fullscreen mode Exit fullscreen mode

Due to this being a local pip install it will end up in $HOME/.local/bin which is generally not in path. I'll modify ~/.profile to add it:

# set PATH so it includes user's private bin if it exists
if [ -d "$HOME/bin" ] ; then
    PATH="$HOME/bin:$HOME/.local/bin:$PATH"
fi
Enter fullscreen mode Exit fullscreen mode

then refresh the session:

$ source ~/.profile
Enter fullscreen mode Exit fullscreen mode

Now devpi needs to know how to communicate with the server. This is done via:

$ devpi use http://localhost:3141
using server: http://localhost:3141/ (not logged in)
no current index: type 'devpi use -l' to discover indices
/home/cwprogram/.config/pip/pip.conf: no config file exists
/home/cwprogram/.pydistutils.cfg: no config file exists
/home/cwprogram/.buildout/default.cfg: no config file exists
Enter fullscreen mode Exit fullscreen mode

Where 3141 is the port you selected during setup if it's different. It also tries to update related pip config files if they exist, which is something we'll look at later. Now to setup the root password and add a new user:

$ devpi login root --password ''
$ devpi user -m root password=[something cool here]
$ devpi user -c cwprogram password=[redacted] email=[redacted]
user created: cwprogram
Enter fullscreen mode Exit fullscreen mode

Finally we need to create an index that the user can utilize for uploading and downloading:

$ devpi index -c cwprogram/stable bases=root/pypi volatile=True
http://localhost:3141/cwprogram/stable?no_projects=:
  type=stage
  bases=root/pypi
  volatile=True
  acl_upload=cwprogram
  acl_toxresult_upload=:ANONYMOUS:
  mirror_whitelist=
  mirror_whitelist_inheritance=intersection
Enter fullscreen mode Exit fullscreen mode

So what's happening here is that cwprogram/stable indicates this is a stable named index attributed to the cwprogram user. bases=root/pypi lets the system know to attempt PyPi if a package is not found. volatile=True will allow uploads to occur. Now that admin work is done we logoff as root:

$ devpi logoff
login information deleted
Enter fullscreen mode Exit fullscreen mode

Project Integration

The next part is going to be from the perspective of an end user. With this in mind I'll be switching to a Windows system for the next part. To facilitate logins that do not require having the password in plain sight I'll be utilizing the python keyring project along with client extensions for devpi. Which allow it to integrate with keyring. As keyring is more of a frontend to several keychain managers, a backend has to be available. Documentation lists this as:

  • macOS Keychain
  • Freedesktop Secret Service supports many DE including GNOME (requires secretstorage)
  • KDE4 & KDE5 KWallet (requires dbus)
  • Windows Credential Locker

If the plan is to use this on a command line linux interface then keyrings.cryptfile via an encrypted text file or sagecipher via ssh-agent can be used. I'll go ahead and install the devpi client along with the extensions and keyring to get started:

> pip install devpi-client devpi-client-extensions[keyring] keyring
> keyring set http://localhost:3141/ cwprogram
Password for 'cwprogram' in 'http://localhost:3141':
Enter fullscreen mode Exit fullscreen mode

Note: The last / in the URL is necessary or keychain won't be picked up

Now if I go to login as the cwprogram user:

> devpi login cwprogram
Using cwprogram credentials from keyring
logged in 'cwprogram', credentials valid for 10.00 hours
Enter fullscreen mode Exit fullscreen mode

Now I can also use pip to authenticate to this server. First I'll have a global setting to use the keyring provider:

> pip config set --global global.keyring-provider subprocess
Enter fullscreen mode Exit fullscreen mode

Note that using pip config set --site with the same options can also be used if you would rather handle this on a per virtual environment project basis. We'll also need to setup keyring again due to pip expecting a different host format:

> keyring set localhost cwprogram
Enter fullscreen mode Exit fullscreen mode

Now to show this works even with a virtual environment:

> virtualenv.exe --python=python3.9 venv_devpi
> .\venv_devpi\Scripts\activate
> devpi use cwprogram/stable
> pip install requests --index-url=http://localhost:3141/cwprogram/stable
Looking in indexes: http://localhost:3141/cwprogram/stable
Collecting requests
  Downloading http://localhost:3141/root/pypi/%2Bf/58c/d2187c01e70e6/requests-2.31.0-py3-none-any.whl (62 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 62.6/62.6 kB 3.3 MB/s eta 0:00:00
Collecting charset-normalizer<4,>=2 (from requests)
  Downloading http://localhost:3141/root/pypi/%2Bf/830/d2948a5ec37c3/charset_normalizer-3.1.0-cp39-cp39-win_amd64.whl (97 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 97.1/97.1 kB ? eta 0:00:00
Collecting idna<4,>=2.5 (from requests)
  Downloading http://localhost:3141/root/pypi/%2Bf/90b/77e79eaa3eba6/idna-3.4-py3-none-any.whl (61 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 61.5/61.5 kB ? eta 0:00:00
Collecting urllib3<3,>=1.21.1 (from requests)
  Downloading http://localhost:3141/root/pypi/%2Bf/48e/7fafa40319d35/urllib3-2.0.3-py3-none-any.whl (123 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 123.6/123.6 kB ? eta 0:00:00
Collecting certifi>=2017.4.17 (from requests)
  Downloading http://localhost:3141/root/pypi/%2Bf/c6c/2e98f5c7869ef/certifi-2023.5.7-py3-none-any.whl (156 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 157.0/157.0 kB ? eta 0:00:00
Installing collected packages: urllib3, idna, charset-normalizer, certifi, requests
Successfully installed certifi-2023.5.7 charset-normalizer-3.1.0 idna-3.4 requests-2.31.0 urllib3-2.0.3
Enter fullscreen mode Exit fullscreen mode

As shown here the package install is successful. Another interesting feature is the pypi downloads are cached:

~/.devpi/server/+files/root/pypi/+f/48e/7fafa40319d35$ ls
urllib3-2.0.3-py3-none-any.whl
Enter fullscreen mode Exit fullscreen mode

While the file structure might be different, we haven't done any uploads and yet urllib is on the devpi servers filesystem. As constantly using the --index-url option to point to the server is not ideal, devpi use can be ran with a --set-cfg option to write out local pip configuration options for the custom index to be used by default:

> devpi use --set-cfg cwprogram/stable
> pip install moto
Looking in indexes: http://localhost:3141/cwprogram/stable/+simple/
Collecting moto
  Downloading http://localhost:3141/root/pypi/%2Bf/6f4/0141ff2f3a309/moto-4.1.12-py2.py3-none-any.whl (3.0 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.0/3.0 MB 48.2 MB/s eta 0:00:00
Collecting boto3>=1.9.201 (from moto)
...
Enter fullscreen mode Exit fullscreen mode

Again it shows up on devpi's local filesystem:

$ find . -iname '*moto*'
./+files/root/pypi/+f/6f4/0141ff2f3a309/moto-4.1.12-py2.py3-none-any.whl
Enter fullscreen mode Exit fullscreen mode

Uploading

Now it's time to do the actual uploading part, as that's why most would want a private PyPi index right? To start out I'll clone tomchen's example PyPi repo template:

> git clone https://github.com/tomchen/example_pypi_package.git
Enter fullscreen mode Exit fullscreen mode

Then I'll install python build (the version designation was there because otherwise I got a python version error back):

> pip3.9 install build
> python -m build --sdist --wheel .
> dir dist\
Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
-a---          2023/06/30    22:45           8317 example_pypi_package-0.1.0-py3-none-any.whl
-a---          2023/06/30    22:45           9189 example_pypi_package-0.1.0.tar.gz
Enter fullscreen mode Exit fullscreen mode

So there's an sdist (tar.gz) and wheel (.whl) file output from the above command. The devpi client provides a simple way to upload:

> devpi upload --sdist --wheel --from-dir dist\
file_upload of example_pypi_package-0.1.0-py3-none-any.whl to http://localhost:3141/cwprogram/stable/
file_upload of example_pypi_package-0.1.0.tar.gz to http://localhost:3141/cwprogram/stable/
> pip install example_pypi_package
Looking in indexes: http://localhost:3141/cwprogram/stable/+simple/
Collecting example_pypi_package
  Downloading http://localhost:3141/cwprogram/stable/%2Bf/318/c1017f3670e4c/example_pypi_package-0.1.0-py3-none-any.whl (8.3 kB)
Installing collected packages: example_pypi_package
Successfully installed example_pypi_package-0.1.0
Enter fullscreen mode Exit fullscreen mode

The pip install from the private index works! Now to test to make sure the package itself works. I'll run this in a non-source code directory as well:

> python
Python 3.9.5 (tags/v3.9.5:0a7dcbd, May  3 2021, 17:27:52) [MSC v.1928 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from examplepy import Number
>>> n1 = Number(2)
>>> n2 = Number(3)
>>> n1.add(n2)
>>> n1.val()
5
Enter fullscreen mode Exit fullscreen mode

Twine is also an option if you'd rather stick with the traditional PyPi upload system. It does require an additional duplication of the keyring entry and a configuration file addition as well. First I'll create a $HOME\.pypirc file (make sure this is Powershell so $HOME expands properly) with the following contents:

[distutils]
index-servers =
    devpi-stable

[devpi-stable]
repository = http://localhost:3141/cwprogram/stable/
username = cwprogram
Enter fullscreen mode Exit fullscreen mode

keyring will need to be set for the value of repository:

> keyring set http://localhost:3141/cwprogram/stable/ cwprogram
Enter fullscreen mode Exit fullscreen mode

From there simply do a twine upload as usual, with a designation to the devpi-stable repository:

> twine upload -r devpi-stable dist\*
Uploading distributions to http://localhost:3141/cwprogram/stable/
Uploading example_pypi_package-0.1.0-py3-none-any.whl
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 28.4/28.4 kB • 00:02 • ?
Uploading example_pypi_package-0.1.0.tar.gz
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 29.1/29.1 kB • 00:00 • ?
Enter fullscreen mode Exit fullscreen mode

The packages also have a nice overview via the web interface:

example view of the uploaded package in devpi web interace

Conclusion

This concludes a look at using devpi for a basic standalone python package server. It's nice for a local development box or small teams where the manual user entry is not so much of an issue. That said, it is possible to have LDAP as an authentication backend via devpi-ldap though it will require a fair amount of technical skill to setup (along with running an LDAP server). In the next installment of this series I'll be looking at more user friendly options.

Top comments (0)

Read next

janahockenberger profile image

🧽 Cleaning up Security Hub with AWS Resource Explorer 🫧

Jana Hockenberger -

ipigantenk profile image

How to install SSH Server in Termux

Ipii -

cloudnative_eng profile image

A Power-Filled IDE for Neovim with Sane Defaults

Cloud Native Engineer -

mikeyoung44 profile image

New AI System Makes Math Models Think Like Creative Mathematicians

Mike Young -