DEV Community

loading...

Discussion on: The best Docker base image for your Python application

Collapse
pmutua profile image
Philip Mutua Author • Edited

Hey David, Thank for the feedback. Could you please elaborate about why the line doesn't read right? When you’re choosing a base image for your Docker image, Alpine Linux is often recommended. Using Alpine, you’re told, will make your images smaller and speed up your builds. And if you’re using Go that’s reasonable advice.
But if you’re using Python, Alpine Linux will quite often:

  • Make your builds much slower.
  • Make your images bigger.
  • Waste your time.

On occasion, introduce obscure runtime bugs.

Let’s see why Alpine is recommended, and why you probably shouldn’t use it for your Python application.

Why people recommend Alpine

Let’s say we need to install gcc as part of our image build, and we want to see how Alpine Linux compares to Ubuntu 18.04 in terms of build time and image size.

First, I’ll pull both images, and check their size:

    $ docker pull --quiet ubuntu:18.04
    docker.io/library/ubuntu:18.04
    $ docker pull --quiet alpine
    docker.io/library/alpine:latest
    $ docker image ls ubuntu:18.04
    REPOSITORY          TAG        IMAGE ID         SIZE
    ubuntu              18.04      ccc6e87d482b     64.2MB
    $ docker image ls alpine
    REPOSITORY          TAG        IMAGE ID         SIZE
    alpine              latest     e7d92cdc71fe     5.59MB
Enter fullscreen mode Exit fullscreen mode

As you can see, the base image for Alpine is much smaller.

Next, we’ll try installing gcc in both of them. First, with Ubuntu:

FROM ubuntu:18.04
RUN apt-get update && \
    apt-get install --no-install-recommends -y gcc && \
    apt-get clean && rm -rf /var/lib/apt/lists/*
Enter fullscreen mode Exit fullscreen mode

We can then build and time that:


$ time docker build -t ubuntu-gcc -f Dockerfile.ubuntu --quiet .
sha256:b6a3ee33acb83148cd273b0098f4c7eed01a82f47eeb8f5bec775c26d4fe4aae

real    0m29.251s
user    0m0.032s
sys     0m0.026s
$ docker image ls ubuntu-gcc
REPOSITORY   TAG      IMAGE ID      CREATED         SIZE
ubuntu-gcc   latest   b6a3ee33acb8  9 seconds ago   150MB
Enter fullscreen mode Exit fullscreen mode

Now let’s make the equivalent Alpine Dockerfile:

FROM alpine
RUN apk add --update gcc
Enter fullscreen mode Exit fullscreen mode

And again, build the image and check its size:

$ time docker build -t alpine-gcc -f Dockerfile.alpine --quiet .
sha256:efd626923c1478ccde67db28911ef90799710e5b8125cf4ebb2b2ca200ae1ac3

real    0m15.461s
user    0m0.026s
sys     0m0.024s
$ docker image ls alpine-gcc
REPOSITORY   TAG      IMAGE ID       CREATED         SIZE
alpine-gcc   latest   efd626923c14   7 seconds ago   105MB
Enter fullscreen mode Exit fullscreen mode

As promised, Alpine images build faster and are smaller: 15 seconds instead of 30 seconds, and the image is 105MB instead of 150MB. That’s pretty good!

But when we switch to packaging a Python application, things start going wrong.

Let’s build a Python image

We want to package a Python application that uses pandas and matplotlib. So one option is to use the Debian-based official Python image (which I pulled in advance), with the following Dockerfile:

FROM python:3.8-slim
RUN pip install --no-cache-dir matplotlib pandas
Enter fullscreen mode Exit fullscreen mode

And when we build it:

$ docker build -f Dockerfile.slim -t python-matpan.
Sending build context to Docker daemon  3.072kB
Step 1/2 : FROM python:3.8-slim
 ---> 036ea1506a85
Step 2/2 : RUN pip install --no-cache-dir matplotlib pandas
 ---> Running in 13739b2a0917
Collecting matplotlib
  Downloading matplotlib-3.1.2-cp38-cp38-manylinux1_x86_64.whl (13.1 MB)
Collecting pandas
  Downloading pandas-0.25.3-cp38-cp38-manylinux1_x86_64.whl (10.4 MB)
...
Successfully built b98b5dc06690
Successfully tagged python-matpan:latest

real    0m30.297s
user    0m0.043s
sys     0m0.020s
Enter fullscreen mode Exit fullscreen mode

The resulting image is 363MB.

Can we do better with Alpine? Let’s try:

FROM python:3.8-alpine
RUN pip install --no-cache-dir matplotlib pandas
Enter fullscreen mode Exit fullscreen mode

And now we build it:

$ docker build -t python-matpan-alpine -f Dockerfile.alpine .                                 
Sending build context to Docker daemon  3.072kB                                               
Step 1/2 : FROM python:3.8-alpine                                                             
 ---> a0ee0c90a0db                                                                            
Step 2/2 : RUN pip install --no-cache-dir matplotlib pandas                                                  
 ---> Running in 6740adad3729                                                                 
Collecting matplotlib                                                                         
  Downloading matplotlib-3.1.2.tar.gz (40.9 MB)                                               
    ERROR: Command errored out with exit status 1:                                            
     command: /usr/local/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/
tmp/pip-install-a3olrixa/matplotlib/setup.py'"'"'; __file__='"'"'/tmp/pip-install-a3olrixa/matplotlib/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-install-a3olrixa/matplotlib/pip-egg-info                              

...
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
The command '/bin/sh -c pip install matplotlib pandas' returned a non-zero code: 1
Enter fullscreen mode Exit fullscreen mode

What’s going on?

Standard PyPI wheels don’t work on Alpine

If you look at the Debian-based build above, you’ll see it’s downloading matplotlib-3.1.2-cp38-cp38-manylinux1_x86_64.whl. This is a pre-compiled binary wheel. Alpine, in contrast, downloads the source code (matplotlib-3.1.2.tar.gz), because standard Linux wheels don’t work on Alpine Linux.

Why? Most Linux distributions use the GNU version (glibc) of the standard C library that is required by pretty much every C program, including Python. But Alpine Linux uses musl, those binary wheels are compiled against glibc, and therefore Alpine disabled Linux wheel support.

Most Python packages these days include binary wheels on PyPI, significantly speeding install time. But if you’re using Alpine Linux you need to compile all the C code in every Python package that you use.

Which also means you need to figure out every single system library dependency yourself. In this case, to figure out the dependencies I did some research, and ended up with the following updated Dockerfile:

FROM python:3.8-alpine
RUN apk --update add gcc build-base freetype-dev libpng-dev openblas-dev
RUN pip install --no-cache-dir matplotlib pandas
Enter fullscreen mode Exit fullscreen mode

And then we build it, and it takes…

… 25 minutes, 57 seconds! And the resulting image is 851MB.

Here’s a comparison between the two base images:

Base image Time to build Image size Research required
python:3.8-slim 30 seconds 363MB No
python:3.8-alpine 1557 seconds 851MB Yes

Alpine builds are vastly slower, the image is bigger, and I had to do a bunch of research.

Can’t you work around these issues?

Build time

For faster build times, Alpine Edge, which will eventually become the next stable release, does have matplotlib and pandas. And installing system packages is quite fast. As of January 2020, however, the current stable release does not include these popular packages.

Even when they are available, however, system packages almost always lag what’s on PyPI, and it’s unlikely that Alpine will ever package everything that’s on PyPI. In practice most Python teams I know don’t use system packages for Python dependencies, they rely on PyPI or Conda Forge.

Image size

Some readers pointed out that you can remove the originally installed packages, or add an option not to cache package downloads, or use a multi-stage build. One reader attempt resulted in a 470MB image.

So yes, you can get an image that’s in the ballpark of the slim-based image, but the whole motivation for Alpine Linux is smaller images and faster builds. With enough work you may be able to get a smaller image, but you’re still suffering from a 1500-second build time when they you get a 30-second build time using the python:3.8-slim image.

But wait, there’s more!

Alpine Linux can cause unexpected runtime bugs

While in theory the musl C library used by Alpine is mostly compatible with the glibc used by other Linux distributions, in practice the differences can cause problems. And when problems do occur, they are going to be strange and unexpected.

Some examples:

  1. Alpine has a smaller default stack size for threads, which can lead to Python crashes.
  2. One Alpine user discovered that their Python application was much slower because of the way musl allocates memory vs. glibc.
  3. Another user discovered issues with time formatting and parsing.

Most or perhaps all of these problems have already been fixed, but no doubt there are more problems to discover. Random breakage of this sort is just one more thing to worry about.

Don’t use Alpine Linux for Python images

Unless you want massively slower build times, larger images, more work, and the potential for obscure bugs, you’ll want to avoid Alpine Linux as a base image. For some recommendations on what you should use, see on an article on choosing a good base image.

Collapse
cairocafe profile image
David

ahh. yes, in the same sentence you say for "smaller" images alpine is recommended but then you say later on that one of the problems with alpine is it gives smaller images. when i looked at the article you referenced they say it sometimes gives "larger" images.

Thread Thread
pmutua profile image
Philip Mutua Author

@cairocafe just curious, which python docker images do you normally use and have you ever had builds taking too long and how did you resolve this?

Collapse
lucian profile image
Lucian BLETAN

Your answer is biased. You forget to mention that you can build an image as base and use the bins after for your needed docker image.

Thread Thread
pmutua profile image
Philip Mutua Author

@lucian Apologies if the you feel the answer seemed biased in your opinion. I would like to hear out your suggestions that might support your opinion .

Thread Thread
lucian profile image
Lucian BLETAN

Well, debian doesn't install code from source.
In your example I can see WHL

Collecting matplotlib
  Downloading matplotlib-3.1.2-cp38-cp38-manylinux1_x86_64.whl (13.1 MB)
Collecting pandas
  Downloading pandas-0.25.3-cp38-cp38-manylinux1_x86_64.whl (10.4 MB)
Enter fullscreen mode Exit fullscreen mode

WHL format was developed as a quicker and more reliable method of installing Python software than re-building from source code every time. WHL files only have to be moved to the correct location on the target system to be installed.

If I take a look on your alpine docker build I clearly see .tgz that means the pip manager install all from source.
I don't have much time for test and give you a clear feedback but you can test it yourself by building from source on debian and see how much time it's take.

Wheel packages pandas and numpy for example are not supported in images based on Alpine platform.

You can also wget a .whl in your alpine and install with cmd:
pip install pandas-**.whl