Python local-packages, à la npm node_modules

Updates

Follow-up post where I made this into simple program called pipm.
**************** ## ***********

Firstly, this thing doesn't exists. Unlike npm or php composer, packaging tools in python by default will install to the global packages location first.

In npm or composer, if you don't specify npm install -g, then by default the package will be installed to folder node_modules in the project root. This quite intuitive, local first, then global.

In python's pip, if you don't want to install to the global packages directory, you can restrict it --user flag. For example:-

python3 -mpip install --user requests

On linux, above command usually will install requests package to user's $HOME/.local/lib/python3.x/site-packages directory. You can see that this only "local" to the user, not project unlike npm node_modules above.

In python, the solution is to use virtualenv. To use virtualenv however, there are at least 2 schools of taught. The first, and also the majority is to put the virtualenv external to your project.

So you might create a directory in your $HOME/.virtualenvs to store all your virtual environment. Creating new virtualenv may look like this:-

python3 -mvenv $HOME/.virtualenvs/myproject

Then you will 'activate' the environment in order to make it active:-

source $HOME/.virtualenvs/myproject/bin/activate

From now on, when you run python it will actually invoke $HOME/.virtualenvs/myproject/bin/python. And any packages you install with pip, for example:-

python -mpip install requests

will install the requests library to $HOME/.virtualenvs/myproject/lib/python3.x/site-packages/requests.

But doing all this manually found to be tedious, so there's tools like virtualenvwrapper, or modern tools like pipenv or poetry that automatically create virtualenv for you.

However I don't really like this approach. Forgetting to 'activate' the virtualenv or worst, activating wrong virtualenv for your project is one of the most painful debugging experience.

So I am in the second camp, who create the virtualenv inside our project dir. For example, if I have a project in $HOME/myproject, I'll do:-

cd $HOME/myproject
python -mvenv venv

So above command will create the virtualenv in the project directory - $HOME/myproject/venv. From now on, whenever I need to invoke python, I'll run ./venv/bin/python from my project root. To install packages, I'll run:-

./venv/bin/python -mpip install requests

That will install requests to ./venv/lib/python3.x/site-packages. Explicit is better than implicit.

If using poetry, you can achieve similar effect by having in your project's poetry.toml:-

[virtualenvs]
in-project = true

Ok, back to the original topic. What if we don't want to use virtualenv? Fortunately, pip has a flag called --target where if specified it will install the package to the target's location. For example:-

python -mpip install -t local-packages requests

(Using name local-packages as a play to site-packages)

Above command will instead install requests to the directory local-packages, which you can create in your project root, similar to node_modules.

Recent version of pip will also created the package's executable script (if they have one) in local-packages/bin directory. For example if we install package httpie, which provide executable script named http in local-packages/bin/http.

Unfortunately, if we try to run the executable script it will fail:-

./local-packages/bin/http 
Traceback (most recent call last):
  File "./local-packages/bin/http", line 6, in <module>
    from httpie.__main__ import main
ModuleNotFoundError: No module named 'httpie'

This is not surprising as ./local-packages dir (which has our httpie packages) is not on python sys.path which is a list of paths where python look for modules to import. We can check it through the interactive console:-

python
>>> import sys
>>> sys.path
['', '/usr/lib/python37.zip', '/usr/lib/python3.7', '/usr/lib/python3.7/lib-dynload', '/home/kamal/.local/lib/python3.7/site-packages', '/usr/lib/python3.7/site-packages']

To add additional path, we can specify PYTHONPATH environment variable:-

PYTHONPATH=$PWD/local-packages ./local-packages/bin/http https://httpbin.org/get
HTTP/1.1 200 OK
Access-Control-Allow-Credentials: true
Access-Control-Allow-Origin: *
Connection: keep-alive
Content-Encoding: gzip
Content-Length: 177
Content-Type: application/json
Date: Mon, 25 Nov 2019 22:36:19 GMT
Referrer-Policy: no-referrer-when-downgrade
Server: nginx
X-Content-Type-Options: nosniff
X-Frame-Options: DENY
X-XSS-Protection: 1; mode=block

{
    "args": {},
    "headers": {
        "Accept": "*/*",
        "Accept-Encoding": "gzip, deflate",
        "Host": "httpbin.org",
        "User-Agent": "HTTPie/1.0.3"
    },
    "origin": "14.192.213.59, 14.192.213.59",
    "url": "https://httpbin.org/get"
}

So it works now. But there's one gotcha. If somehow we have httpie already installed in global or user site-packages, that one probably will get used instead of the one we have in our local-packages. This is bad and can cause another headache when debugging. And this is the reason we use virtualenv. But we don't want to use virtualenv, right?

Fortunately, python has another flag. The -S flag will disable processing site-packages. From the docs:-

Disable the import of the module site and the site-dependent manipulations of sys.path that it entails. Also disable these manipulations if site is explicitly imported later (call site.main() if you want them to be triggered).

But we run into another problem if we invoke python with the -S flag:-

PYTHONPATH=$PWD/local-packages python -S -mhttpie https://httpbin.org/get
Traceback (most recent call last):
  File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/kamal/python/testpip/local-packages/httpie/__main__.py", line 18, in <module>
    main()
  File "/home/kamal/python/testpip/local-packages/httpie/__main__.py", line 10, in main
    from .core import main
  File "/home/kamal/python/testpip/local-packages/httpie/core.py", line 23, in <module>
    from httpie.client import get_response
  File "/home/kamal/python/testpip/local-packages/httpie/client.py", line 8, in <module>
    from httpie import sessions
  File "/home/kamal/python/testpip/local-packages/httpie/sessions.py", line 11, in <module>
    from httpie.plugins import plugin_manager
  File "/home/kamal/python/testpip/local-packages/httpie/plugins/__init__.py", line 10, in <module>
    from httpie.plugins.manager import PluginManager
  File "/home/kamal/python/testpip/local-packages/httpie/plugins/manager.py", line 2, in <module>
    from pkg_resources import iter_entry_points
ModuleNotFoundError: No module named 'pkg_resources'

Let's try copying the pkg_resources module into our local-packages dir:-

cp -a /usr/lib/python3.7/site-packages/pkg_resources local-packages/

No luck. We will find out that we need these modules to be available:-

• pkg_resources
• six
• appdirs
• packaging
• pyparsing

And it turned out that all these modules actually part of setuptools, another beast in python packaging. So in short, in order to make our local-packages works, we need to install setuptools as well:-

python -mpip install -t local-packages setuptools
python -mpip install -t local-packages httpie

And now invoking python with -S flag should work:-

PYTHONPATH=$PWD/local-packages python -S -mhttpie https://httpbin.org/get
HTTP/1.1 200 OK
Access-Control-Allow-Credentials: true
Access-Control-Allow-Origin: *
Connection: keep-alive
Content-Encoding: gzip
Content-Length: 177
Content-Type: application/json
Date: Tue, 26 Nov 2019 00:15:54 GMT
Referrer-Policy: no-referrer-when-downgrade
Server: nginx
X-Content-Type-Options: nosniff
X-Frame-Options: DENY
X-XSS-Protection: 1; mode=block

{
    "args": {},
    "headers": {
        "Accept": "*/*",
        "Accept-Encoding": "gzip, deflate",
        "Host": "httpbin.org",
        "User-Agent": "HTTPie/1.0.3"
    },
    "origin": "14.192.213.59, 14.192.213.59",
    "url": "https://httpbin.org/get"
}

And that's it. We finally manage to invoke python, free from global site-packages and using our local-packages dir to place all dependencies for our project.

End notes: In practice, we're using buildout to achieve the same objectives.