Windows Jenkins Server Running WSL Docker Agents

This guide is meant to show a Windows setup for Jenkins which will run docker agents using WSL. Code will be stored in a private git server to make for easier local integration. Such an environment was created for the purpose of learning Jenkins to improve my job skills and as such is not a replacement for a production environment.

Hyper-V

You'll first want to make sure that Hyper-V is setup. Microsoft has documentation on how to do so along with the requirements for it. This is especially useful if you plan to do containers on the Windows side.

WSL Windows Subsystem for Linux

WSL is a system for Windows to allow you run a Linux distro from within various versions of Windows. Windows 10 and 11 Pro will work, but home editions will not. Version 2 will be used as it allows for systemd services to run, which there will be plenty of those to run. Installation should be as simple as;

wsl --install

Once done you'll want to verify WSL2 is being used by default:

wsl --set-default-version 2

From there various Linux distributions can be installed from the Windows Store. I personally picked the latest Ubuntu distrobution. For setup you will need to provide a username and password. There's not much besides that.

Windows Docker Install

To have docker tools running on Windows, I used the choco package manager for this:

choco install docker-engine docker-cli

This will let you run docker containers on both Windows and Linux depending on the Jenkins agent you require.

WSL Docker Install

The normal and easy way to do this would be Docker Desktop for Windows but...

Every install attempt leads to this with no detailed information. After about 10 different workarounds and closed as stale issues over several years it was apparent that the installer isn't a very solid piece of software. So instead we'll do it the old fashioned Ubuntu way using WSL!

First install make sure you're logged into the WSL Ubuntu instance. If you installed from the Windows Store you can simply search for Ubuntu in the start menu and it comes up as an app. You'll need to make sure systemd support is also setup so the docker service can run properly.

Then follow the instructions on the docker site. Now you'll need to add your regular user to the docker group so you don't have to be root every time you want to do container work:

$ sudo usermod -a -G docker [your_wsl_username_here]

Note that you'll need to re-load your WSL session after this so the group change takes effect.

Linking WSL Docker With Windows

WARNING: This method is insecure and primarily done for testing purposes. Realistically you should be using SSH/TLS to secure the connection. Do not do this in a production environment

To make things even more convenient, especially on the Jenkins side, we're going to do a bit of magic to expose WSL docker as a TCP service to Windows and then add it to the Windows side. I thank Markus Lippert for providing this particular method:

$ sudo cp /lib/systemd/system/docker.service /etc/systemd/system/
$ sudo sed -i 's/\ -H\ fd:\/\//\ -H\ fd:\/\/\ -H\ tcp:\/\/127.0.0.1:2375/g' /etc/systemd/system/docker.service
$ sudo systemctl daemon-reload
$ sudo systemctl restart docker.service

Then on the Windows side:

docker context create lin --docker host=tcp://127.0.0.1:2375

Now I tend to prefer using linux containers so I set that to the default context for working with containers:

docker context use lin

Jenkins Setup

Jenkins will need a Windows user and proper permissions for the Jenkins service to run under. Go to the search Windows task bar and:

1. Type in "Computer Management"
2. In the Window, expand "Local users and Groups"
3. Click on "Users"
4. Then from the menu select "Actions" -> "New User..."
5. Make sure to uncheck any of the checkboxes at the bottom
6. Click "Create"

This has created the Jenkins user. To make things easy docker wise:

1. Click "Groups" on the left side
2. Double click "docker-users"
3. Click "Add"
4. Where it says "Enter the object names to select" enter the username decided in the previous section
5. Click "Check Names" to get a fully qualified name
6. Click "OK"

Now Jenkins needs to run as a service, but it can't as is. To do so:

1. Type in "Local Security Policy" in the windows search bar
2. Expand "Local Policies"
3. Click on "User Rights Assignment"
4. Double click on "Log on as a service"
5. Click on "Add User or Group..."
6. Repeat steps 4-6 of the previous section

Now the Jenkins service user is setup to run Jenkins. One last requirement is a working Java 11 or 17. Doing this through the Oracle route requires an unwanted signup step, so instead we'll use the Microsoft OpenJDK builds. The one I used at the time of writing was microsoft-jdk-17.0.7-windows-x64.msi.

Jenkins Install

The Jenkins install on Windows is pretty well documented. You might want to consider the target install location though if your C drive is strapped for space. I found that WSL installs seem to do weird things if installed on anything but C drive so you'll need space for that anyways. From there:

1. Head to http://localhost:8080 to start the setup (the port number will depend on what was set during install)
2. Enter the initial admin password which the screen will make very clear the location of
3. I recommend just going with "Install suggested plugins" since it gave me most every plugin I needed for basic setup
4. Plugins should do there thing, and it usually takes a pretty decent amount of time
5. Enter in the admin details and things should be good from there

Docker Plugin

The Docker plugin (which is different from docker pipeline plugin) is used to setup on demand Docker agents later.

1. Click on "Manage Jenkins" on the left
2. Click on "Manage Plugins" in the top bar
3. Click on "Available Plugins" on the left
4. Type in "Docker" which should say "This plugin integrates Jenkins with Docker"
5. Select "Install without restart"

I do this without restart because I've noticed the Jenkins internal way of doing it seems flaky versus just restarting the service. Once the plugin is installed:

1. Enter "Services" in the Windows search bar and select it
2. Right click on "Jenkins"
3. Select "Restart"

Going to the Jenkins web interface again might show it's still working on getting started, and when it's done you'll be prompted to login. This completes the basic Jenkins setup.

Git Server Setup

Honestly GitHub is going to be the easiest functionality wise. However, I've always wanted to setup my own git server so that's what this will be about. Going with GitHub would also require exposing my local PC if I ever decided to do GitHub push notifications. At first I tried using the GitLab docker container but it was well beyond what my home PC could handle resource wise. Instead I went with gitolite.

Git connectivity will be done via SSH for both Jenkins and the docker agents. Which means that an SSH server will be required on WSL:

$ sudo apt-get install -y openssh-server

Next up a keypair will be need to be created. The GitHub setup page has instructions on how to do this. If the .ssh directory doesn't exist you'll want to make it now in your home directory (%userprofile% or ~ depending on what you're using).

Now, one of the issues will be having it to so both the host system and the docker agent can connect to the Git server, which is all hosted in the sample place. To get around this we'll use host mapping. Open up C:\Windows\System32\drivers\etc\hosts in editor of choice. Note that this will require admin privileges to actually make changes. At the end of the file add something like:

127.0.0.1 gitserver

Now when git repositories are accessed it will be in the form git@gitserver:repository-name-here.

Git Server Install

Gitolite has a fairly detailed install document. One of the things to note is that step 3 in the doc mentions copying over the pubkey from your workstation. If it's made on Windows you can simply do:

$ cp /mnt/c/Users/YOUR_USER_HERE/.ssh/*.pub /tmp

This will work if it's the first time creating an SSH keypair. If it's not then *.pub will need to be the exact filename of the public key pair you plan to use. Also YOUR_USER_HERE will need to be replaced with the appropriate Windows user name. Now simply replace ron.pub in in the instructions with whatever the resulting .pub file is in /tmp.

Git Server Testing

Now it's time to setup a repository and attempt to connect to it. The first thing to do is run:

ssh git@gitserver 

and it should ask you to verify an SSH fingerprint. If you go into WSL where the SSH server is installed and run ssh-keyscan -t ed25519 localhost you can validate that you're talking with the right server. To make connecting and authenticating with the git server via SSH less of a hassle, create a file called .ssh/config in the Windows user home directory with the contents:

Host gitserver
     user git
     hostname gitserver
     port 22
     identityfile ~/.ssh/[keypair name without the .pub]

This will make it so that when we connect to the git server it authenticates us properly making the experience seamless. Now it's time to setup a repo. On the Windows side, go to the user home directory and run:

git clone git@gitserver:gitolite-admin

Then enter it when it's cloned down. The settings are essentially a git repo itself. Edit conf/gitolite.conf so the file looks something like this:

repo gitolite-admin
    RW+     =   [ssh key name without the .pub]

repo jenkins-test
    RW+     =   [ssh key name without the .pub]

[ssh key name without the .pub] is whatever replaced ron.pub in the install instructions without the .pub extension. Now run git add conf/gitolite.conf && git commit -m "add Jenkins test repo" && git push origin master. This will make a new repo available called jenkins-test for use. Now back in the Windows home directory run:

git clone git@gitserver:jenkins-test 

Which should show the git repo setup works.

Jenkinsfile

Now in this new jenkins-test repo we're going to create a Jenkinsfile. This is similar in function to .circleci/config.yml on CircleCI and yaml files in .github/workflows for GitHub actions. In this example, I'm going to do a simple python version call:

pipeline {
    agent { 
        label 'jenkins-agent'
    }
    stages {
        stage('build') {
            steps {
                sh 'python3 --version'
            }
        }
    }
}

This is a somewhat minor modification of the Jenksinfile from the getting started tutorial. Assuming this is still the jenkins-test repository run:

git add Jenkinsfile && git commit -m "Initial Jenkinsfile" && git push origin master

Which will now make it easily available to Jenkins.

Customize The Jenkins Agent

Now by itself the jenkins agent docker file is missing some things for python support. To rectify this we're going to go ahead and make a custom docker image. You'll want to create this on the WSL side since it's going to be a Linux container. The Dockerfile looks like this:

FROM jenkins/agent:bullseye-jdk17 as agent

USER root

RUN set -ex && \
    apt-get update -y && \
    apt-get upgrade -y && \
    apt-get install git -y && \
    apt-get install -y \
        python3 \
        python3-pip \
        python3-venv \
        iputils-ping \
        unixodbc-dev

USER jenkins

Note that iputils-ping is there for discovering network issues but if you don't have any it can be removed and rebuilt. From there it just needs to be built with something like:

$ docker build -t jenkins-python-agent .

If you see any permissions errors it might be that you're not part of the docker group. You can verify this via groups. If you did add it with the usermod step then you may just need to exit out and start the WSL ubuntu instance to get the groups refreshed.

Jenkins Tool Configuration

Before setting up the agent there's a few bits of configuration prep work that go into play. In the Jenkins web interface:

1. Click on "Manage Jenkins" on the left
2. Next click on "Credentials" in the Security category

This should show something like:

1. Now click on "(global)"
2. Then click on "+ Add Credentials" in the upper right

Now you'll fill out the credentials like so:

The username should be git and ID can be whatever you want to identify it with. Where it says "Enter Directly" with blank space, that's where you want to put your private key from the generated keypair. Note that it's the key that doesn't have .pub as the extension. Note that it will be something like:

-----BEGIN OPENSSH PRIVATE KEY-----
[lots of text here]
-----END OPENSSH PRIVATE KEY-----

Copy that all into the text field (you may need to press "Add" to bring it up for copy/paste). There's also a passphrase entry box at the bottom in case you secured yours with one. Click on "Create" when finished. The nice thing about this is that Jenkins will do a temporary pass off of the key to the docker agent for git checkout to work. That means you don't have to have the private key included in the docker image itself. When done go back to "Manage Jenkins" and:

1. Click on "Global Tool Configuration" in the System Configuration category
2. Go to where it says Git and change git.exe to git

This is necessary because otherwise Jenkins will try to run git.exe within the docker agent which doesn't work since it's a linux container. Then click "Save" at the bottom. Due to wanting to actually do SSH key verification, we'll need to let Jenkins know about the SSH fingerprints. First go to "Manage Jenkins" on the left and:

1. Click on "Configure Global Security" under the Security category
2. Change Host Key Verification Strategy to "Manually provided keys"
3. Fill in the "Approved Host Keys" with the contents of c:\Users\YOUR_REGULAR_USER\.ssh\known_hosts
4. Click "Save" at the bottom

This file should be created when you first SSH'ed into WSL from Windows. If not I recommend doing so to populate it. This step is needed due to Jenkins being a different user from the regular one used to connect to SSH services. While something like symlinks could probably work I'd rather have a nice manual list to work with.

Docker Agent Configuration

Now it's finally time to setup the docker cloud. Since I only want to do docker agents It's time to first disable the built in one (which is also useful to see if the cloud setup is valid or not). Go to "Manage Jenkins" and:

1. Click on "Manage Nodes and Clouds"
2. Click on "Built-In Node"
3. Click on "Configure" on the left
4. Set "Number of executors" to 0

Now it's time to setup the docker cloud:

1. Go back to "Manage Jenkins" (the breadcrumb on the top)
2. Click on "Manage Nodes and Clouds"
3. Click on "Configure Clouds" on the left
4. Click on "Add a new cloud"
5. Select "Docker"
6. Leave the name as is

Now for the Docker Cloud Details:

Docker host is the WSL exposed docker service on port 2375 tcp. Click on "Test Connection" to see if everything is solid as well. Then click "Enabled" so it actually gets used. Click "Expose DOCKER_HOST" and set the container cap. I have this set to 5 in case something breaks causing a hectic container cleanup. Now it's time to setup a Docker Template. it's basically a very extensive way to configure how the agent docker image will be run:

The label here is meant to match the one in the Jenkinsfile for the repo:

    agent { 
        label 'jenkins-agent'
    }

Which explicitly states we want to use our docker agents for builds. "Enabled" is checked so it gets used properly. Docker Image is set to the name of the custom docker agent image we created. Remote file system root is set to the proper path for the Jenkins agent home /home/jenkins/agent. "Use this node as much as possible" doesn't really do much since we're directly asking for this by label, but it's the best option. Also under "Container settings" we want to scroll down towards the very bottom where it says "Extra Hosts" and set it to:

gitserver:host-gateway

This is special wizardry which routes the docker traffic meant for the git server (which is the host Jenkins will pass to the agent when doing git checkout) to the docker host where the SSH server is. This allows for a seamless git checkout.

The connect method I found was the most reliable way for Jenkins to connect to the docker agent. jenkins is set so it's working with the proper user. "Pull strategy" is set to "Never pull" since we're working with a local image. If set to anything else it will try and obtain the image from docker hub and fail spectacularly or give us an image we don't want. Click "Save" at the bottom when finished setting everything up.

Jenkins Git Integration

Now to setup the Git integration:

1. Click on "+New Item" on the left side
2. Give it a name such as "JenkinsGitTest" then select "Multibranch Pipeline"
3. Select "Okay" to confirm

Multibranch Pipeline essentially lets you set different builds depending on the branch. It's common for cases where you have something like a prod, dev, and test environment to work with. Each of these maps to a specific git branch. After setting up a name and description click "Add source" and select "Git":

Project repository will be using the example one we created or git@gitserver:jenkins-test. The format is similar to what you see in GitHub style pulls. Credentials is set to the one created for git during the Jenkins Tool Configuration phase. Under build configuration "Mode" should be set to "by Jenkinsfile" so it uses the Jenkinsfile from the repository (and thus our docker cloud agents). Everything else can be left as-is and press "Save" at the bottom. This will kickoff a process to register all the branches and do an initial kickoff build for them. The "Scan Multibranch Pipeline Log" should look like this:

If you click "Build History" on the left, then "Multibranch Setup > master" you'll get a view of the master build using our Jenkinsfile:

Hovering over "#1" under "Build History" until a down arrow appears and selecting "Console Output" will show you how the build ran. Checking towards the bottom:

The build is a success and the python version is printed out properly!

Conclusion

That's it for this very long setup process, but I hope it gives insight into setting up Jenkins in Windows with docker integration which also has a private git server. As mentioned this fully meant for testing there are some things like user separation in gitolite and push notification for Jenkins to automatically run the multibranch pipeline. I'll look into potentially doing a post on that and any other interesting Jenkins info I come across on a later post (maybe even minikube agent cluster if I'm feeling crazy enough).