Create an Azure Self-Hosted Agent with Python without going Insane

I regularly use Azure for my day-job or other software work, and depending on the task at hand, it's necessary to setup self-hosted agents to get your work done. For those of you uninitiated, Azure Pipelines are a way of automating CI/CD and, once you've gotten over the initial learning curve, are genuinely great. At the time of writing this, open-source software projects could have up to 10 simultaneous "agents", hosted for free on Azure's cloud. However, for private projects you're a bit more limited. You get a single agent for a few thousand hours and this is generally insufficient for all but most basic projects. This is where self-hosted agents come in. You setup your own environment and plug it into Azure's job scheduler to get sorta the best of both worlds.

Being Saturday and COVID, I decided to work on making a private pipeline more efficient by adding a self-hosted agent to the existing pipeline. Since I'm cheap, I'm only willing on shelling out the $60 it takes to have two simultaneous, unlimited MS agents and waiting for the CI tasks to complete was really getting on my nerves. Since I have a home server/NAS that's mostly sitting there, I might as well make use of its processor. Should be easy right?

Setting Up the Agent

Let me prerequisite this by first saying that I'm no fan of Microsoft Windows. The cumulative time developers have spent in the past decade dealing with the OS idiosyncrasies, inefficiencies, and bad business practices enabled by a past corporate culture is a crime against humanity. That being said, the MS of today has genuinely pivoted to the cloud and open-source and rather than killing GitHub after it's acquisition, I would argue that it's improved under their leadership. Not something I'd expect from Microsoft. While I genuinely think that Windows, or at least the kernel, will be replaced with Linux one day as Windows slides into obsolescence, but think that Microsoft has a bright future ahead in the areas of cloud and software development. And honestly, there's far more money there than just in the OS anyway.

Windows has great documentation for setting up the agent and installing it locally at Self-hosted Linux agents, and there's very little I can add to the setup process. Note that for the agent to be started automatically to enable the service with ./svc.sh install.

Pipeline Configuration

With the agent installed, the next step is to modify your azure-pipelines.yml. For example, I added my agent, ds9, to the pool ds9_pool and added that into the yml with:

- job: Linux
  pool: ds9_pool

From the old configuration of:

  pool:
    vmImage: 'ubuntu-latest'

Next, create a branch with these changes and submit push it. Provide you already have the pipeline setup, this "Linux" job will automatically run on your new agent. I say "run" but I really mean, "fail almost instantly" because if you're migrating from a Microsoft-hosted agent, you'll need to do some extra work getting it compatible.

Pipeline Migration to Self-Hosted Agent

First, a little background on my pipeline. My pipeline compiles a python module that has several C/C++ extensions that are built using Cython. Overall, it's a great setup, I have the choice to write in Cython or to write pure C/C++ and then interface with it from Cython. For my project this saves a lot of time writing C extension interface code in pure C, which is often brittle and difficult to code.

However, using C extensions means that I can't just compile the package as a pure-python and ship it. I've got to build it on the environment and platform matching where it will be deployed. In this case, that's Windows and Linux for Python 3.6 - Python 3.8. In the past I would have to manually build it for each release, and all the build scripts and unit testing would take a minimum of a week per release (regardless of patch or major). Creating a pipeline is absolutely critical to getting the real work done of making better software.

The beginning of my "Linux" job looks like:

jobs:
- job: Linux
  strategy:
    matrix:
      Python36:
        python.version: '3.6'
        python.architecture: 'x64'
      Python37:
        python.version: '3.7'
        python.architecture: 'x64'
      Python38:
        python.version: '3.8'
        python.architecture: 'x64'
  pool: ds9_pool
  steps:
    - template: .ci/checkout.yml
    - task: UsePythonVersion@0
      inputs:
        versionSpec: $(python.version)
      displayName: 'Use Python $(python.version)'

This is where the first issue migrating will crop up. The UsePythonVersion@0 task requires Python to be installed on the self-hosted machine. For example, assuming you're using the default _work agent directory, your directory structure should look like:

$AGENT_TOOLSDIRECTORY/
    Python/
        {version number}/
            {platform}/
                {tool files}
            {platform}.complete

Microsoft provides documentation for setting this up at Use Python Version task, but it's not complete. For example, what's in {tool files}? Can I use 3.8 in place of 3.8.5 in {version number}. Is there an automated way of installing Python?

Turns out it takes a bit of work to figure out, but it's actually quite straightforward once you know a few tricks:

Installing and Setting up Python

The directory structure of Python 3.8 should look like:

$AGENT_TOOLSDIRECTORY/
    Python/
        3.8/
            x64/
                bin/
                include/
                lib/
                share/
            x64.complete

You could manually download, install, and copy over Python to this directory, but it's far better to simply use a Python virtual environment as this is repeatable and far more reliable. First, ensure you have python3.8-venv installed with:

sudo apt install python3.8-venv

Next, cd into your _tool directory and create a Python directory with mkdir Python. I need three versions of Python, 3.6, 3.7, and 3.8. Since the machine already has 3.8 installed from Ubuntu 20.04, it's easy to create a virtual environment since the base version is already installed. However, first you'll need to create a 3.8 directory as this is how Azure expects the directories to be setup. However, since you might wish to reference the exact version of python, it's best to create a directory of the full version of Python and then symlink it to the directory without the patch version. For example, the version installed after running python3 -V is Python 3.8.5, so you would create a symlink with ln -s 3.8.5 3.8. Your directory structure should now look like:

$AGENT_TOOLSDIRECTORY/
    Python/
        3.8/ -> 3.8.5/
        3.8.5/

cd into 3.8.5 and create your virtual enviornment with:

python3.8 -m venv x64

It's x64 because that's how Azure expects it to be setup (i.e. {version}/{architecture}). Finally, touch x64.complete to add the x64.complete file within the 3.8.5 directory. I'm not sure why this is necessary given that all the information is there in the directory structure, but alas, some poor programmer probably had a reason to do this.

The directory structure should now look like:

$AGENT_TOOLSDIRECTORY/
    Python/
        3.8 --> 3.8.5
        3.8.5/
            x64/
                bin/
                include/
                lib/
                share/
            x64.complete

This has the advantage of using the system Python as a symlink, not as a copy, yet retaining separate directories for all Azure installed packages. For all intensive purposes, it's a virtual environment setup just for the Azure agent.

The Other Python Versions

Ubuntu 20.04 comes with Python3.8, but for the older versions of Python, you'll need to add the deadsnakes repository and install them from there. Do this with:

sudo add-apt-repository ppa:deadsnakes/ppa

Since my pipeline requires the Python headers, I need to install the -dev version, and in order to create the virtual environment, you'll need to install -venv, so for the other two versions of Python, the full installation command is:

sudo apt install python3.7-dev python3.7-venv python3.6-dev python3.6-venv

Next, starting in the Python directory, create the virtual environments for 3.6 and 3.7 with:

PY36_VER=$(python3.6 -c "import sys; print('.'.join([f'{val}' for val in sys.version_info[:3]]))")
mkdir $PY36_VER
ln -s $PY36_VER 3.6
cd $PY36_VER
python3.6 -m venv x64
touch x64.complete
cd ..

PY37_VER=$(python3.7 -c "import sys; print('.'.join([f'{val}' for val in sys.version_info[:3]]))")
mkdir $PY37_VER
ln -s $PY37_VER 3.7
cd $PY37_VER
python3.7 -m venv x64
touch x64.complete
cd ..

The directory structure should look like:

$AGENT_TOOLSDIRECTORY/
    Python/
        3.6 --> 3.6.13
        3.6.13/
            x64/
                bin/
                ...
            x64.complete
        3.7 --> 3.7.10
        3.7.10/
            x64/
                bin/
                ...
            x64.complete
        3.8 --> 3.8.5
        3.8.5/
            x64/
                bin/
                ...
            x64.complete

Note that if your pipeline does not include a step to update pip, you may have to add it in. The version included with Python 3.6 was incompatible with my workflow and had to be manually upgraded.

Installing other Packages and Requirements

Azure hosted images contain several popular software packages, and if you're migrating to a fresh environment, it's necessary to install quite a few of them manually. In my case, I needed the following:

sudo apt install docker zip libglu1-mesa
sudo snap install docker

Zip was for the ArchiveFiles@2 step, libglu1-mesa was for vtk, and docker was for the building the Python wheels on quay.io/pypa/manylinux2010_x86_64.

Performance Improvements and Next Steps

Using Microsoft hosted agents is a great way to run CI/CD on the cloud without owning hardware; it scales well and is well suited to building software. However, in my case, I had extra hardware and spare time, so it was worth the time investment to add an additional agent to the pipeline. My builds run about 3x faster using the additional agent, and I could probably find another (perhaps Windows) computer to run the build.

However, I wasn't just going after performance improvements. There's some data that I'd rather not to be on the "cloud", and it's important to run these functional tests pre-release. I've been neglecting them because they're not automated, and there's usually an edge case that crops up because I failed to run them. The next step is to add in these functional tests, save the benchmarks (using pytest-benchmark), and add it in as a pre-release step.