Both docker and virtualenvs try to solve the same problem: separating the system side of things from the user side of things. In case of virtual envs you are getting basically a bit "shell magic" where paths are tweaked in such a way that things work as you were interfering with the system's python but you aren't. That makes it impossible to have interfering installations of python packages and versions.
Docker does the same on a more fundamental level:
Docker divides the kernel-land from the user-land. So it behaves more or less like a virtual environment; but instead of relying on "shell magic" it relies on "kernel magic".
From that point of view there is no real difference or advantage of using one over the other. It totally depends on your deployment model. If your deployment involves docker, it would make more sense to use docker from the first line of code written - and it would be a minimal smoketest for what you are doing anyways. If your deployment model doesn't involve docker, I see no advantage of using docker over a virtual environment.
Docker has a lot of uses, but merely as a replacement for virtual environments? It's overkill. If a virtualenv is a sandbox, Docker is a clean room. You can use it, naturally, but if the only thing you were needing was a controlled set of Python packages, it's a lot of wasted time and disk space.
Now, if you actually need to control your environment beyond what virtualenv allows you to control, Docker is fine. (It makes sense for web development, for example.)
I enjoy using Docker, mostly because I always have a database or something similar standing next to the actual application, plus deploying Docker is super easy, but perhaps I should try using venv more. Thank you :)
I do not share the point of docker being "overkill" because if one is fluent with it, there would be no "time wasted". And disk space is cheap - as long as you aren't on a Mac (SCNR - But admittedly one of my box is a Mac and I am too guilty of suffering from the lack of a bunch of GBs).
I put it more mildly: It has no advantages over virtual envs if you do not use docker for deployment.
It has no advantages over virtual envs if you do not use docker for deployment.
Well, and see, that's why I said that if it was merely a replacement for virtualenvs, it was overkill. A virtual environment only replicates the bare minimum necessary, in terms of the Python interpreter and libraries you're using.
If you're actually deploying with Docker, that's another story altogether.
First, not everyone has Docker configured on their machine. Second, resources like disk space are not always "cheap" on any system. (Macs aren't the only machines to suffer from low disk space.) Internet may be limited, slow, or inexpensive, making a Docker build prohibitive. Time may be of the essence, RAM might be limited, or CPU might be in demand. There are so many scenarios in which Docker is just not going to be work.
Virtual environments are the one canonical "works everywhere" solution for sandboxing Python. Anyone and everyone who can run Python 3 (and most versions of 2) can create a virtual environment. They require very little overhead, generally minimal network time, and nothing in the way of extra processing power. Ironically, they're one of the fastest and most reliable ways to deploy a Python project in a Docker!
To this aim, your project should be configured properly so that it can be run in a virtual environment. (It doesn't take that much work; certainly less work than a Dockerfile does.)
The third problem is, if the Python project in question has anything to do with GUI or the like, you're going to be beyond the "just start a Docker container" situation. You'll have to configure and work with VNC, which for testing a local application is generally impractical. The same could be said of working with local filesystems (you have to plan what to access and mount it), system integration features, and the list goes on.
Docker certainly has a purpose in some Python development situations, but it should be considered a separate tool altogether, and never as a "replacement" for virtual environments.
Hey Jason, thanks for the great write-up! It certainly solves a lot of questions for me. But one is still there – what to do when I have databases and I don't want to have pq, mysql and mongo installed on my system in various versions?
Assuming SQLite is not an option, this is exactly the sort of scenario where Docker is a good solution: you aren't just needing the sandbox the Python environment itself (virtual environment), but the actual system environment.
Virtual environments really become extra helpful here, too, because it simplifies your Dockerfiles. Assuming your Python project has a requirements.txt file, you'd only need the following:
RUN python3 -m venv venv &&\
venv/bin/pip install-r requirements.txt
You'll note, I had no need to "activate" the virtual environment. You can just use the venv's binaries directly; they know to use the virtual environment.
For further actions, you may consider blocking this person and/or reporting abuse
We're a place where coders share, stay up-to-date and grow their careers.
Hi, nice article, but honestly I've abandoned virtualenv in favor of Docker. What are your thoughts about this tool?
Both docker and virtualenvs try to solve the same problem: separating the system side of things from the user side of things. In case of virtual envs you are getting basically a bit "shell magic" where paths are tweaked in such a way that things work as you were interfering with the system's python but you aren't. That makes it impossible to have interfering installations of python packages and versions.
Docker does the same on a more fundamental level:
Docker divides the kernel-land from the user-land. So it behaves more or less like a virtual environment; but instead of relying on "shell magic" it relies on "kernel magic".
From that point of view there is no real difference or advantage of using one over the other. It totally depends on your deployment model. If your deployment involves docker, it would make more sense to use docker from the first line of code written - and it would be a minimal smoketest for what you are doing anyways. If your deployment model doesn't involve docker, I see no advantage of using docker over a virtual environment.
Docker has a lot of uses, but merely as a replacement for virtual environments? It's overkill. If a virtualenv is a sandbox, Docker is a clean room. You can use it, naturally, but if the only thing you were needing was a controlled set of Python packages, it's a lot of wasted time and disk space.
Now, if you actually need to control your environment beyond what virtualenv allows you to control, Docker is fine. (It makes sense for web development, for example.)
Just not as a straight virtualenv replacement.
Thanks for your reply, Jason.
I enjoy using Docker, mostly because I always have a database or something similar standing next to the actual application, plus deploying Docker is super easy, but perhaps I should try using venv more. Thank you :)
I do not share the point of docker being "overkill" because if one is fluent with it, there would be no "time wasted". And disk space is cheap - as long as you aren't on a Mac (SCNR - But admittedly one of my box is a Mac and I am too guilty of suffering from the lack of a bunch of GBs).
I put it more mildly: It has no advantages over virtual envs if you do not use docker for deployment.
Well, and see, that's why I said that if it was merely a replacement for virtualenvs, it was overkill. A virtual environment only replicates the bare minimum necessary, in terms of the Python interpreter and libraries you're using.
If you're actually deploying with Docker, that's another story altogether.
First, not everyone has Docker configured on their machine. Second, resources like disk space are not always "cheap" on any system. (Macs aren't the only machines to suffer from low disk space.) Internet may be limited, slow, or inexpensive, making a Docker build prohibitive. Time may be of the essence, RAM might be limited, or CPU might be in demand. There are so many scenarios in which Docker is just not going to be work.
Virtual environments are the one canonical "works everywhere" solution for sandboxing Python. Anyone and everyone who can run Python 3 (and most versions of 2) can create a virtual environment. They require very little overhead, generally minimal network time, and nothing in the way of extra processing power. Ironically, they're one of the fastest and most reliable ways to deploy a Python project in a Docker!
To this aim, your project should be configured properly so that it can be run in a virtual environment. (It doesn't take that much work; certainly less work than a Dockerfile does.)
The third problem is, if the Python project in question has anything to do with GUI or the like, you're going to be beyond the "just start a Docker container" situation. You'll have to configure and work with VNC, which for testing a local application is generally impractical. The same could be said of working with local filesystems (you have to plan what to access and mount it), system integration features, and the list goes on.
Docker certainly has a purpose in some Python development situations, but it should be considered a separate tool altogether, and never as a "replacement" for virtual environments.
Hey Jason, thanks for the great write-up! It certainly solves a lot of questions for me. But one is still there – what to do when I have databases and I don't want to have pq, mysql and mongo installed on my system in various versions?
Assuming SQLite is not an option, this is exactly the sort of scenario where Docker is a good solution: you aren't just needing the sandbox the Python environment itself (virtual environment), but the actual system environment.
Virtual environments really become extra helpful here, too, because it simplifies your Dockerfiles. Assuming your Python project has a
requirements.txtfile, you'd only need the following:You'll note, I had no need to "activate" the virtual environment. You can just use the venv's binaries directly; they know to use the virtual environment.