Jenkins Upgrade from 2.1x to 2.4x

This Article speaks about the Jenkins upgrade from 2.1X to 2.4x Due to vulnerabilities observed in the Jenkins on 24th Jan 2024.

References
For more information on CVE-2024-23897, please refer to the following sources:]

1. https://www.jenkins.io/security/advisory/2024-01-24/
2. https://thehackernews.com/2024/01/critical-jenkins-vulnerability-exposes.html
3. https://www.bleepingcomputer.com/news/security/exploits-released-for-critical-jenkins-rce-flaw-patch-now/
4. https://github.com/jenkinsci-cert/SECURITY-3314-3315

Vulnerable versions

Jenkins 2.441 and earlier, LTS 2.426.2 and earlier.

Temporary mitigation

Access to the CLI needs to be disabled. Both of the following steps must be taken:

• Remove the CLI HTTP endpoint.
• Disable the SSH Port

Both steps can be performed by executing the below script in script console of Jenkins UI(jenkins-->mange jenkins-->script-->console)

def removal = { lst ->
  lst.each { x -> if (x.getClass().getName()?.contains("CLIAction")) lst.remove(x) }
}
def j = jenkins.model.Jenkins.get();
removal(j.getExtensionList(hudson.cli.CLIAction.class))
removal(j.getExtensionList(hudson.ExtensionPoint.class))
removal(j.getExtensionList(hudson.model.Action.class))
removal(j.getExtensionList(hudson.model.ModelObject.class))
removal(j.getExtensionList(hudson.model.RootAction.class))
removal(j.getExtensionList(hudson.model.UnprotectedRootAction.class))
removal(j.getExtensionList(java.lang.Object.class))
removal(j.getExtensionList(org.kohsuke.stapler.StaplerProxy.class))
removal(j.actions)

println "Done!"

if (j.getPlugin('sshd')) {
  hudson.ExtensionList.lookupSingleton(org.jenkinsci.main.modules.sshd.SSHD.class).setPort(-1)
}

Permanent solution/mitigation:

• Permanent mitigation can be done by upgrading the Jenkins to latest version.
• As per our current Jenkins setup automatic upgrade/migration is not possible and we have to replace the source file(i.e. Jenkins.war)
• What is Jenkins.War: The Jenkins Web application Archive (WAR) file bundles Winstone, a Jetty servlet container wrapper, and can be started on any operating system or platform with a version of Java supported by Jenkins

please install openjdk-17(install jdk not jre) with yum repositories before starting the upgrade which is required for 2.444 version(Latest version with fixed vulnerability at the time of writing this article

The command to install openjdk-17 is

yum install java-17-devel

and please don’t create any symbolic links if you have other version of jdk in your system instead please set the config to take the java 17 by below command.

alternative --config java

this command will prompt for the versions which are available in the system

Upgrade implementation steps on Linux:

Step 1: Stop the Jenkins service

sudo su -
service jenkins stop

Step 2: If the Jenkins is running in background, please kill the PID of the Jenkins by checking respective Jenkins port 8080

ps -ef | grep 8080
kill -9 PID

Step 3: Take backup of Jenkins home directory by zipping the file and move to temporary path. in my case the paths are given below,but it might different from system to system

cd /var/lib
tar -cvzf jenkins_date.tar.gz jenkins/
mv jenkins_date.tar.gz to /tmp path

Step 4: Take the backup of Jenkins current version binary(jenkins.war) using following commands.

cd /usr/lib/jenkins/
mv jenkins.war jenkins_old.war

Step 5: The webroot folder which is /var/cache/Jenkins has to be empty, when we are starting with new jenkins.war file.
so take the backup of war folder(mv war war_old)and empty the folder (so that it will extract new configuration in war cache folder)

mv war war_old
rm -rf war/*
chown Jenkins:Jenkins war
chmod 755 war/

Step 6: Download The New Jenkins Version and We can check the downloaded war file of SHA by

wget https://updates.jenkins-ci.org/latest/jenkins.war
sha256sum jenkins.war

Note: if the server is behind the proxy please execute the http and https proxy commands before downloading it.

step 7: Start The Jenkins Service

service jenkins start

There are several difficulties has been faced during this since it is a major version upgrade

1. Unable to start the Jenkins service.

• it is due to incompatible init.d/jenkins file where the --daemon is not supported anymore
• and comment the handlermaxCount, handlerMaxIdle (https://www.jenkins.io/doc/upgrade-guide/2.387/#de-duplicate-logging-implementations)

1. The Jenkins will start in the background but still the service shows as failed.

• To resolve this completely please take the Jenkins command which is used to start and create Jenkins.service file like below.

service Jenkins status

Note: the starting command can be shown while checking the status, so please take the command and convert to below service file.

• Create jenkis.service file in /etc/systemd/system with the following content(from the above copied command)
• ExecStart command should match with the above copied command

[Unit]
Description=Jenkins Service
After=network.target

[Service]
Type=simple
User=jenkins
Group=jenkins
ExecStart=/etc/alternatives/java -Djava.awt.headless=true -DJENKINS_HOME=/var/lib/jenkins -jar /usr/lib/jenkins/jenkins.war --logfile=/var/log/jenkins/jenkins.log --webroot=/var/cache/jenkins/war --httpPort=8080
Restart=always

[Install]
WantedBy=multi-user.target

• Please enable the service after creating the jenkins.service file

systemctl enable Jenkins.service

• And start with the below command

systemctl start Jenkins.service

Note: please delete the init.d/Jenkins file if it is present in server , that might cause multiple instance starts
this could be different from server to server please check your server config)

• var/lib/Jenkins—JENKINS_HOME
• /usr/lib/Jenkins—Jenkins.war
• /var/cache/Jenkins/war—webroot war extraction location
• Inti.d/jenkins—manually written service file location
• /etc/system/system/--- systemctl controlled services configuration locations