Spawn a Jenkins from code

Through the CI looking glass

In a corporate CI infrastructure, configuration is key.

When it comes to Jenkins, we can easily look at it as if it's only a glorified collection of crontab jobs. Only later we realize that it became much more, and it manages sensitive data and configuration more than we first realize.

Credentials, host information, job descriptions, test, and production workflow definitions, and all the special settings and plugins to keep the show on.
Using Job DSL is one of the first big milestones that a CI team must achieve when working with Jenkins. This at least provides us some security. With Job DSL our job definitions that serve as blueprints for our product pipelines can be maintained in version control. This indeed takes off some weight from the shoulders of the DevOps guys.

It gives us peace of mind if you like.

Still, there is a malevolent secret in the hearth of our Jenkins, and that secret is the configuration that we wrought by tedious manual work on the configurations page. If our servers happen to bite the dust we can struggle to figure out how to make our jobs work again in a vanilla Jenkins, knowing nothing about these implicit dependencies, which are the result of some special port configuration, or a plugin install that was done years ago by some guy who's not even at the firm at the time. It can be quite a nuisance to resurrect a CI infrastructure after such an inconvenient accident.

I'm going to delve into a more advanced topic here, so if you are new to Jenkins it's better to start at the beginning. Which is not a documentation.
Come on. Don't lie to yourself. You only learn by doing stuff. Simply install a Jenkins and start breaking it in different ways. Seriously.

Not according to the manual

Let's declare our goal this way:

Set up a Jenkins straight from code, using configuration that is kept in files so it can be version controlled.

We want a scriptable process, that can be triggered from the command line with a single command and a few parameters.

One thing I must add:
There are a lot of implicit dependencies here that one should take care of when doing a zero-to-hero automated Jenkins setup. There are certain pitfalls (like handling necessary dependencies, managing the compatibility of our plugin requirements with the chosen Jenkins version, or figuring out how to control java instances of Jenkins or a certain plugin when configuring through cli). I will not give you a full solution here. Consider it a collection of ideas, or guidance if you like. It's the same process, that my team used when building our own configuration-as code solution.

Let's see the process

Before we jump into writing code, you might also want to look at the new configuration-as-code plugin that could help a lot to maintain Jenkins configuration. Anyway, according to the rumors, it could soon become the core component of Jenkins.

First step: Get your Jenkins war file.

No Jenkins runs without the necessary binary. You can get it from here: https://updates.jenkins-ci.org/download/war/

Create your custom Jenkins home

We will start from scratch. Literally, we will create a new empty directory .jenkins to serve as the home for our future Jenkins, and we will populate it in a way that it can be interpreted by Jenkins as a basic Jenkins config. For it, in order to be able to serve as the home directory for our new Jenkins instance, it should look something like this:

|-- config.xml
|-- jenkins.install.InstallUtil.lastExecVersion
|-- org.jenkinsci.main.modules.sshd.SSHD.xml
`-- users
|   |-- pencillr_xxxxxxxxxxxxxxxxxxx
|   |   `-- config.xml
    `-- users.xml

As we go through the process all the files listed above will be explained in detail.

Let's start with creating the home directory:

jenkins_home="/home/$USER/.jenkins"
mkdir "${jenkins_home}"

If you've ever started a Jenkins you know that first there is a setup process through which you create your admin user. All that we want to essentially pre-create.

First, we must avoid Jenkis starting this setup process. We achieve this by adding our chosen Jenkins version to our newly created jenkins.install.InstallUtil.lastExecVersion file.

jenkins_version=2.175.1 # your Jenkins version, see: https://jenkins.io/changelog/
echo "$jenkins_version" > "${jenkins_home}"/jenkins.install.InstallUtil.lastExecVersion

Add user record

We are adding the admin user from code, by creating the config files representing an admin user account. We create the users dir in our Jenkins home containing users.xml file. Jenkins from version 2.150.1 onwards uses users.xml file that holds username -> profile directory mapping. We will follow this convention here.

username=pencillr # the admin username you like
mkdir "${jenkins_home}/users"
touch "${jenkins_home}/users/users.xml"
mkdir "${jenkins_home}/${username}_1234567890123456789" # notice that this is an arbitrary hash
touch "${jenkins_home}/${username}_1234567890123456789/config.xml"

Our config.xml under "${jenkins_home}"/users will be like this, using our username at @username@:

<?xml version='1.1' encoding='UTF-8'?>
<hudson.model.UserIdMapper>
  <version>1</version>
  <idToDirectoryNameMap class="concurrent-hash-map">
    <entry>
      <string>@username@</string>
      <string>@username@_1234567890123456789</string>
    </entry>
  </idToDirectoryNameMap>
</hudson.model.UserIdMapper>

Now we jump to the user config definition under our "${jenkins_home}/users/${username}_1234567890123456789" directory. We need a config.xml file here to represent our admin user account. This is not a usual user registration procedure, so naturally you must do your password hashing yourself. We can do this, and also add password salt to make our configuration safer.

pass_salt=$(< /dev/urandom tr -dc 'a-zA-Z0-9' | fold -w 16 | head -n 1)
pass_hash=$(echo "${jenkins_password}{${jenkins_password_salt}}" | \
    sha256sum -b | cut -f 1 -d " ")

We want an ssh accessible jenkins, so we also add our public key at the config file, which will look like this in the end:

<?xml version='1.0' encoding='UTF-8'?>
<user>
  <properties>
    <hudson.security.HudsonPrivateSecurityRealm_-Details>
      <passwordHash>$pass_salt:$pass_hash</passwordHash>
    </hudson.security.HudsonPrivateSecurityRealm_-Details>
    <org.jenkinsci.main.modules.cli.auth.ssh.UserPropertyImpl>
      <authorizedKeys>$ssh_public_key</authorizedKeys>
    </org.jenkinsci.main.modules.cli.auth.ssh.UserPropertyImpl>
  </properties>
</user>

SSH accessibility

Our plan is to inject configuration through the CLI after our Jenkins instance is running. For that, we have to allow ssh access to the API explicitly. We do that by simply adding the desired ssh port to "$jenkins_home"/org.jenkinsci.main.modules.sshd.SSHD.xml file.

<?xml version='1.0' encoding='UTF-8'?>
<org.jenkinsci.main.modules.sshd.SSHD>
  <port>$jenkins_sshd_port</port>
</org.jenkinsci.main.modules.sshd.SSHD>

The last step is to enable our preferred security settings in Jenkins. We do this in our main config file at "$jenkins_home"/config.xml.

<?xml version='1.0' encoding='UTF-8'?>
<hudson>
  <authorizationStrategy class="hudson.security.FullControlOnceLoggedInAuthorizationStrategy">
    <denyAnonymousReadAccess>true</denyAnonymousReadAccess>
  </authorizationStrategy>
  <securityRealm class="hudson.security.HudsonPrivateSecurityRealm">
    <disableSignup>true</disableSignup>
    <enableCaptcha>false</enableCaptcha>
  </securityRealm>
</hudson>

Now we have our basic Jenkins setup. Start Jenkins on your newmade home dir as:

java -DJENKINS_HOME=$JENKINS_HOME jar jenkins.war --httpPort=$PORT"

Setting up plugins.

As we have an ssh accessible Jenkins we can do all configuration on our Jenkins through ssh. To see what CLI commands you can use on your own Jenkins check them out by opening a browser and going to localhost:$PORT/cli. You must also login with the newly created credentials. After this we are ready to run our commands.

Just to mention, there is indeed a CLI client for this purpose, but here for simplicity, we will not use that jar file.

You can test if the CLI is accessible through ssh with checking the session id of your Jenkins:

# ssh port is the one you defined in the SSHD file
ssh -p $SSH_PORT $USER@localhost session-id

Installing plugins are as simple as the CLI help suggests:

ssh -p $SSH_PORT $USER@localhost install-plugin SOURCE ... [-deploy] [-name VAL] [-restart]

See the CLI help, it's pretty straightforward:

 SOURCE    : If this points to a local file (?-remoting? mode only), that file
             will be installed. If this is an URL, Jenkins downloads the URL
             and installs that as a plugin. If it is the string ?=?, the file
             will be read from standard input of the command, and ?-name? must
             be specified. Otherwise the name is assumed to be the short name
             of the plugin in the existing update center (like ?findbugs?), and
             the plugin will be installed from the update center. If the short
             name includes a minimum version number (like ?findbugs:1.4?), and
             there are multiple update centers publishing different versions,
             the update centers will be searched in order for the first one
             publishing a version that is at least the specified version.
 -deploy   : Deploy plugins right away without postponing them until the reboot.
 -name VAL : If specified, the plugin will be installed as this short name
             (whereas normally the name is inferred from the source name
             automatically).
 -restart  : Restart Jenkins upon successful installation.

Apply configuration through CLI

You can apply config to Jenkins simply through groovy scripts. Jenkins is pretty awesome when it comes to accessibility. You can actually access the Java instances of the plugins and also Jenkins itself through the script console. You can inject any groovy script with the following method, through ssh:

ssh -p $SSH_PORT $USER@localhost groovy = < "${your_groovy_script_file}

Let's see an example where we set the scm retry count of Jenkins by injecting a simple script, so we have a more redundant checkout logic in our Jenkins.

#!/usr/bin/env groovy

import jenkins.model.Jenkins

def jenkins = Jenkins.getInstanceOrNull()
jenkins.setScmCheckoutRetryCount(5)
jenkins.save()
}

Another example: Here we make sure that the timestamper plugin adds timestamps globally to all pipeline console logs.

#!/usr/bin/env groovy

import hudson.plugins.timestamper.TimestamperConfig

//  Enable timestamper plugin globally
def timestamper = TimestamperConfig.get()
if (!timestamper.isAllPipelines()) {
    timestamper.setAllPipelines(true)
}

You can discover the interfaces of the Jenkins instance and it's plugins by scrolling through the Java docs. It's not a walk in the park I know.

Create jobs

All we need now is to create our jobs using our JobDSL definitions.

Unfortunately, we don't have any jobs now, but we can have a seed job with a simple trick. All you need is to create a seed job by hand, then save it's xml representation to a file. Then in your scripts, you can inject this xml file to be created as a seed job. You can trigger it also through the CLI.

ssh -p $SSH_PORT $USER@localhost create-job "$SEED_JOB_NAME" "${seed_job_as_xml}"

Profit

Indeed this is all a bit unconventional but sometimes one must go great distances to assure corporate level redundancy. Creating your groovy config scripts might be tedious work, but it may worth the effort. I also add here that there can be many pitfalls before you can pull-up and tear-down your Jenkinses flawlessly with one click.