There are cases where you want the flexibility of installing a python package via pip without having it available to the open public. This article will focus on using devpi to provide a self-hosted pip compatible python package server. Ubuntu will be used for the OS as it's a fairly common Linux distribution and easily available on Windows Linux Subsystem.
Install
Before beginning we want to make sure that devpi will run as an actual user and not the root process for security purposes:
$ sudo useradd -m -s /bin/bash devpi
Now I'll switch to that user
$ sudo su - devpi
Next is to create a virtual environment so devpi and its packages don't clutter the global namespace (you may need to apt-get install python3-virtualenv
for this to work):
$ virtualenv --python=python3.8 venv
created virtual environment CPython3.8.10.final.0-64 in 123ms
creator CPython3Posix(dest=/home/devpi/venv, clear=False, global=False)
seeder FromAppData(download=False, pip=latest, setuptools=latest, wheel=latest, pkg_resources=latest, via=copy, app_data_dir=/home/devpi/.local/share/virtualenv/seed-app-data/v1.0.1.debian.1)
activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator
$ src venv/bin/activate
Finally installation of the devpi server:
$ pip install devpi-server devpi-web
While devpi-web
isn't a hard requirement it does enable better visualization of python packages and allows for pip search
queries to work. Now we'll do install validation and initialize the server:
# devpi-server --version
6.9.0
# devpi-init
INFO NOCTX Loading node info from /root/.devpi/server/.nodeinfo
INFO NOCTX generated uuid: b7b5b93272124964b10bdb37cce942ad
INFO NOCTX wrote nodeinfo to: /root/.devpi/server/.nodeinfo
INFO NOCTX DB: Creating schema
INFO [Wtx-1] setting password for user 'root'
INFO [Wtx-1] created user 'root'
INFO [Wtx-1] created root user
INFO [Wtx-1] created root/pypi index
INFO [Wtx-1] fswriter0: committed at 0
This is setting up the basic backend for the devpi server to work. It's also creating a root user for later administrative tasks. Next up we'll need to create files which provide several options to run the devpi server:
# devpi-gen-config
It is highly recommended to use a configuration file for devpi-server, see --configfile option.
wrote gen-config/crontab
wrote gen-config/net.devpi.plist
wrote gen-config/launchd-macos.txt
wrote gen-config/nginx-devpi.conf
wrote gen-config/nginx-devpi-caching.conf
wrote gen-config/supervisor-devpi.conf
wrote gen-config/supervisord.conf
wrote gen-config/devpi.service
wrote gen-config/windows-service.txt
A port can also be provided during execution if the default of 3141 doesn't suit you:
# devpi-gen-config --port 4040
As you can see there are several methods of running devpi server including cron, launchd (OSX service), nginx, Windows service, and supervisord. It also has a systemd
service file which we can use to manage the service easily as Ubuntu uses it for primary service management. First off though we're going to need a proxy script to ensure that devpi is running in the virtual environment:
/home/devpi/start-devpi.sh
#!/bin/bash
cd $HOME
source venv/bin/activate
devpi-server --restrict-modify=root
--restrict-modify=root
makes it so that the root user is needed to make administrative changes such as index and user creation. Then modify it to be executable by systemd:
$ chmod u+x start-devpi.sh
Finally we'll adjust the systemd service file:
gen-config/devpi.service
[Unit]
Description=Devpi Server
Requires=network-online.target
After=network-online.target
[Service]
Restart=on-success
# ExecStart:
# - shall point to existing devpi-server executable
# - shall not use the deprecated `--start`. We want the devpi-server to start in foreground
ExecStart=/home/devpi/start-devpi.sh
# set User according to user which is able to run the devpi-server
User=devpi
[Install]
WantedBy=multi-user.target
Now exit out of the devpi user to return back to the normal user you work on Ubuntu with. Then go ahead and:
$ sudo cp /home/devpi/gen-config/devpi.service /etc/systemd/system/
$ sudo systemctl enable devpi.service
Created symlink /etc/systemd/system/multi-user.target.wants/devpi.service → /etc/systemd/system/devpi.service
$ sudo systemctl start devpi.service
Now the service is available on http://localhost:3141 by default unless the port was changed:
User Management
Before beginning we'll need to install the client which manages everything:
$ pip install devpi-client
Due to this being a local pip install it will end up in $HOME/.local/bin
which is generally not in path. I'll modify ~/.profile
to add it:
# set PATH so it includes user's private bin if it exists
if [ -d "$HOME/bin" ] ; then
PATH="$HOME/bin:$HOME/.local/bin:$PATH"
fi
then refresh the session:
$ source ~/.profile
Now devpi needs to know how to communicate with the server. This is done via:
$ devpi use http://localhost:3141
using server: http://localhost:3141/ (not logged in)
no current index: type 'devpi use -l' to discover indices
/home/cwprogram/.config/pip/pip.conf: no config file exists
/home/cwprogram/.pydistutils.cfg: no config file exists
/home/cwprogram/.buildout/default.cfg: no config file exists
Where 3141 is the port you selected during setup if it's different. It also tries to update related pip config files if they exist, which is something we'll look at later. Now to setup the root password and add a new user:
$ devpi login root --password ''
$ devpi user -m root password=[something cool here]
$ devpi user -c cwprogram password=[redacted] email=[redacted]
user created: cwprogram
Finally we need to create an index that the user can utilize for uploading and downloading:
$ devpi index -c cwprogram/stable bases=root/pypi volatile=True
http://localhost:3141/cwprogram/stable?no_projects=:
type=stage
bases=root/pypi
volatile=True
acl_upload=cwprogram
acl_toxresult_upload=:ANONYMOUS:
mirror_whitelist=
mirror_whitelist_inheritance=intersection
So what's happening here is that cwprogram/stable
indicates this is a stable
named index attributed to the cwprogram
user. bases=root/pypi
lets the system know to attempt PyPi if a package is not found. volatile=True
will allow uploads to occur. Now that admin work is done we logoff as root:
$ devpi logoff
login information deleted
Project Integration
The next part is going to be from the perspective of an end user. With this in mind I'll be switching to a Windows system for the next part. To facilitate logins that do not require having the password in plain sight I'll be utilizing the python keyring project along with client extensions for devpi. Which allow it to integrate with keyring. As keyring is more of a frontend to several keychain managers, a backend has to be available. Documentation lists this as:
- macOS Keychain
- Freedesktop Secret Service supports many DE including GNOME (requires secretstorage)
- KDE4 & KDE5 KWallet (requires dbus)
- Windows Credential Locker
If the plan is to use this on a command line linux interface then keyrings.cryptfile via an encrypted text file or sagecipher via ssh-agent can be used. I'll go ahead and install the devpi client along with the extensions and keyring to get started:
> pip install devpi-client devpi-client-extensions[keyring] keyring
> keyring set http://localhost:3141/ cwprogram
Password for 'cwprogram' in 'http://localhost:3141':
Note: The last /
in the URL is necessary or keychain won't be picked up
Now if I go to login as the cwprogram user:
> devpi login cwprogram
Using cwprogram credentials from keyring
logged in 'cwprogram', credentials valid for 10.00 hours
Now I can also use pip
to authenticate to this server. First I'll have a global setting to use the keyring provider:
> pip config set --global global.keyring-provider subprocess
Note that using pip config set --site
with the same options can also be used if you would rather handle this on a per virtual environment project basis. We'll also need to setup keyring again due to pip
expecting a different host format:
> keyring set localhost cwprogram
Now to show this works even with a virtual environment:
> virtualenv.exe --python=python3.9 venv_devpi
> .\venv_devpi\Scripts\activate
> devpi use cwprogram/stable
> pip install requests --index-url=http://localhost:3141/cwprogram/stable
Looking in indexes: http://localhost:3141/cwprogram/stable
Collecting requests
Downloading http://localhost:3141/root/pypi/%2Bf/58c/d2187c01e70e6/requests-2.31.0-py3-none-any.whl (62 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 62.6/62.6 kB 3.3 MB/s eta 0:00:00
Collecting charset-normalizer<4,>=2 (from requests)
Downloading http://localhost:3141/root/pypi/%2Bf/830/d2948a5ec37c3/charset_normalizer-3.1.0-cp39-cp39-win_amd64.whl (97 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 97.1/97.1 kB ? eta 0:00:00
Collecting idna<4,>=2.5 (from requests)
Downloading http://localhost:3141/root/pypi/%2Bf/90b/77e79eaa3eba6/idna-3.4-py3-none-any.whl (61 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 61.5/61.5 kB ? eta 0:00:00
Collecting urllib3<3,>=1.21.1 (from requests)
Downloading http://localhost:3141/root/pypi/%2Bf/48e/7fafa40319d35/urllib3-2.0.3-py3-none-any.whl (123 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 123.6/123.6 kB ? eta 0:00:00
Collecting certifi>=2017.4.17 (from requests)
Downloading http://localhost:3141/root/pypi/%2Bf/c6c/2e98f5c7869ef/certifi-2023.5.7-py3-none-any.whl (156 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 157.0/157.0 kB ? eta 0:00:00
Installing collected packages: urllib3, idna, charset-normalizer, certifi, requests
Successfully installed certifi-2023.5.7 charset-normalizer-3.1.0 idna-3.4 requests-2.31.0 urllib3-2.0.3
As shown here the package install is successful. Another interesting feature is the pypi downloads are cached:
~/.devpi/server/+files/root/pypi/+f/48e/7fafa40319d35$ ls
urllib3-2.0.3-py3-none-any.whl
While the file structure might be different, we haven't done any uploads and yet urllib is on the devpi servers filesystem. As constantly using the --index-url
option to point to the server is not ideal, devpi use
can be ran with a --set-cfg
option to write out local pip
configuration options for the custom index to be used by default:
> devpi use --set-cfg cwprogram/stable
> pip install moto
Looking in indexes: http://localhost:3141/cwprogram/stable/+simple/
Collecting moto
Downloading http://localhost:3141/root/pypi/%2Bf/6f4/0141ff2f3a309/moto-4.1.12-py2.py3-none-any.whl (3.0 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.0/3.0 MB 48.2 MB/s eta 0:00:00
Collecting boto3>=1.9.201 (from moto)
...
Again it shows up on devpi's local filesystem:
$ find . -iname '*moto*'
./+files/root/pypi/+f/6f4/0141ff2f3a309/moto-4.1.12-py2.py3-none-any.whl
Uploading
Now it's time to do the actual uploading part, as that's why most would want a private PyPi index right? To start out I'll clone tomchen's example PyPi repo template:
> git clone https://github.com/tomchen/example_pypi_package.git
Then I'll install python build (the version designation was there because otherwise I got a python version error back):
> pip3.9 install build
> python -m build --sdist --wheel .
> dir dist\
Mode LastWriteTime Length Name
---- ------------- ------ ----
-a--- 2023/06/30 22:45 8317 example_pypi_package-0.1.0-py3-none-any.whl
-a--- 2023/06/30 22:45 9189 example_pypi_package-0.1.0.tar.gz
So there's an sdist (tar.gz) and wheel (.whl) file output from the above command. The devpi client provides a simple way to upload:
> devpi upload --sdist --wheel --from-dir dist\
file_upload of example_pypi_package-0.1.0-py3-none-any.whl to http://localhost:3141/cwprogram/stable/
file_upload of example_pypi_package-0.1.0.tar.gz to http://localhost:3141/cwprogram/stable/
> pip install example_pypi_package
Looking in indexes: http://localhost:3141/cwprogram/stable/+simple/
Collecting example_pypi_package
Downloading http://localhost:3141/cwprogram/stable/%2Bf/318/c1017f3670e4c/example_pypi_package-0.1.0-py3-none-any.whl (8.3 kB)
Installing collected packages: example_pypi_package
Successfully installed example_pypi_package-0.1.0
The pip
install from the private index works! Now to test to make sure the package itself works. I'll run this in a non-source code directory as well:
> python
Python 3.9.5 (tags/v3.9.5:0a7dcbd, May 3 2021, 17:27:52) [MSC v.1928 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from examplepy import Number
>>> n1 = Number(2)
>>> n2 = Number(3)
>>> n1.add(n2)
>>> n1.val()
5
Twine is also an option if you'd rather stick with the traditional PyPi upload system. It does require an additional duplication of the keyring entry and a configuration file addition as well. First I'll create a $HOME\.pypirc
file (make sure this is Powershell so $HOME
expands properly) with the following contents:
[distutils]
index-servers =
devpi-stable
[devpi-stable]
repository = http://localhost:3141/cwprogram/stable/
username = cwprogram
keyring will need to be set for the value of repository
:
> keyring set http://localhost:3141/cwprogram/stable/ cwprogram
From there simply do a twine upload as usual, with a designation to the devpi-stable
repository:
> twine upload -r devpi-stable dist\*
Uploading distributions to http://localhost:3141/cwprogram/stable/
Uploading example_pypi_package-0.1.0-py3-none-any.whl
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 28.4/28.4 kB • 00:02 • ?
Uploading example_pypi_package-0.1.0.tar.gz
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 29.1/29.1 kB • 00:00 • ?
The packages also have a nice overview via the web interface:
Conclusion
This concludes a look at using devpi
for a basic standalone python package server. It's nice for a local development box or small teams where the manual user entry is not so much of an issue. That said, it is possible to have LDAP as an authentication backend via devpi-ldap though it will require a fair amount of technical skill to setup (along with running an LDAP server). In the next installment of this series I'll be looking at more user friendly options.
Top comments (0)