DEV Community

Jesse Houwing for Xebia Microsoft Services

Posted on • Originally published at jessehouwing.net on

Azure DevOps 2020 and 2019 (and 2018) patch for log4j vulnerability

Azure DevOps 2020 and 2019 (and 2018) patch for log4j vulnerability

Update: Official statement from Microsoft.

Azure DevOps Server ships with a pretty old version of Elastic Search and by default installs a non-auto updating JVM. I just checked the version of Elastic Search that ships with my version of Azure DevOps Server and it's 6.2.4. That version ships with log4j 2.9.1, which is unfortunately too old for the simple fix described by Elastic to do the trick:

Elasticsearch 6 and 7 are not susceptible to remote code execution with this vulnerability due to our use of the Java Security Manager. Elasticsearch running on JDK8 or below is susceptible to an information leak via DNS which is fixable by the JVM option identified below. This option is effective for Elasticsearch versions 5.6.11+, 6.4+, and 7.0+.

Elasticsearch 5 is susceptible to both remote code execution and an information leak via DNS. For versions 5.6.11 - 5.6.16, this can be mitigated by setting the JVM option. Users on an earlier version of 5.x, are recommended to upgrade to 5.6.16. We are exploring other options for cases where upgrading is not possible. Please note that while we provide these remediations, Elasticsearch 5 is not a supported version, and we always recommend updating to the latest release.

https://discuss.elastic.co/t/apache-log4j2-remote-code-execution-rce-vulnerability-cve-2021-44228-esa-2021-31/291476

  • For TFS 2017 RTM, Elastic Search 1.7.1, which ships with log4j 1.2.17, which is not vulnerable.
  • For TFS 2017 Update 1 - TFS 2018 Update 1, Elastic Search 2.4.1, which ships with log4j 1.2.17, which is not vulnerable.
  • For TFS 2018 Update 2+, Elastic Search 5.4.1, which ships with log4j 2.8.2, which is vulnerable and can not be resolved by setting the JVM option.
  • For Azure DevOps Server 2019, Elastic Search 6.2.4, which ships with log4j 2.9.1, which is vulnerable and can not be resolved by setting the JVM option.
  • For Azure DevOps Server 2020, Elastic Search 6.2.4, which ships with log4j 2.9.1, which is vulnerable and can not be resolved by setting the JVM option.

Luckily the default installation mode only binds Elastic Search to the local loopback network interface limiting access from the outside world, but if you've setup Search on a separate VM and didn't follow the Securing Search guidance to limit access you may want to look into that right now.

You currently have 3 options to deal with this issue:

  1. Remove Search from your installation
  2. Patch the log4j-core-*.jar files on disk
  3. Patch the log4j vulnerability at runtime

I also highly recommend you update the JVM that was installed when you configured Search. The default option does not auto-update and chances are the one you're running is already quite old.

Remove Search

The simplest solution, and most secure solution at the moment, is to uninstall Azure DevOps Search from your Application Tiers and disabling/uninstalling the Elastic Search service:

Azure DevOps 2020 and 2019 (and 2018) patch for log4j vulnerability
Disable Elastic Search

Or remove the service altogether:

Azure DevOps 2020 and 2019 (and 2018) patch for log4j vulnerability
Remove the Elastic Search Service

And remove the Search feature from the Azure DevOps Administration Console:

Azure DevOps 2020 and 2019 (and 2018) patch for log4j vulnerability
Remove Azure DevOps Search Service from the Azure DevOps Server Admin Console

Patch Elastic Search files Instead

If you don't want to turn Search off, you'll need to patch log4j-core to strip the vulnerable classes from the jar files. You can remove the vulnerable classes from the jar file with a single command. All of the docs I found give you this command which will work on Linux:

zip -q -d log4j-core-*.jar org/apache/logging/log4j/core/lookup/JndiLookup.class
Enter fullscreen mode Exit fullscreen mode

On Windows that command won't work, since the zip command isn't available. If you've got 7-zip installed, use the following the command for Azure DevOps Server 2020:

cd "C:\Program Files\Azure DevOps Server 2020\Search\ES\elasticsearchv6.2\lib"
..\bin\elasticsearch-service stop
"c:\Program Files\7-Zip\7z.exe" d log4j-core-2.9.1.jar org/apache/logging/log4j/core/lookup/JndiLookup.class
..\bin\elasticsearch-service start
Enter fullscreen mode Exit fullscreen mode

To resolve the issue for Azure DevOps Server 2019+:

cd "C:\Program Files\Azure DevOps Server 2019\Search\ES\elasticsearchv6.2\lib"
..\bin\elasticsearch-service stop
"c:\Program Files\7-Zip\7z.exe" d log4j-core-2.9.1.jar org/apache/logging/log4j/core/lookup/JndiLookup.class
..\bin\elasticsearch-service start
Enter fullscreen mode Exit fullscreen mode

To resolve the issue for Team Foundation Server 2018 Update 2+:

# cd "C:\Program Files\Microsoft Team Foundation Server 15.0\Search\ES\elasticsearch-5.4.1\lib"
..\bin\elasticsearch-service stop
"c:\Program Files\7-Zip\7z.exe" d log4j-core-2.8.2.jar org/apache/logging/log4j/core/lookup/JndiLookup.class
..\bin\elasticsearch-service start
Enter fullscreen mode Exit fullscreen mode

Alternatively, patch log4j at runtime

You can also install log4j-jndi-be-gone and instruct Elastic Search to load it to remove the vulnerability at runtime. This will also work for older versions of Team Foundation Server and Azure DevOps Server.

Download the log4j-jndi-be-gone-1.0.0-standalone.jar from their GitHub repository and save it to your server.

Azure DevOps 2020 and 2019 (and 2018) patch for log4j vulnerability
Save the jar file where Elastic Search can find it

Run elasticsearce-service.bat manager to open the configuration panel for Elastic Search and add the jar file to the configuration:

-javaagent:C:\AzureDevOpsData\log4j-jndi-be-gone\log4j-jndi-be-gone-1.0.0-standalone.jar
Enter fullscreen mode Exit fullscreen mode

Azure DevOps 2020 and 2019 (and 2018) patch for log4j vulnerability
Add log4j-jndi-be-gone agent to the Elastic Search configuration

Then restart the Elastic Search service:

Azure DevOps 2020 and 2019 (and 2018) patch for log4j vulnerability
Restart Elastic Search after adding log4j-jndi-be-gone.

Update the Java Runtime Environment for Search

While you're at it, update the JVM to the most recent version. You can find the latest Java 8 Runtime here.

Note

If you choose to use Azul Zulu OpenJDK, ensure that you download version 8.

Extract the zip file into the AzureDevOpsData\Java\ folder on your Application Tier:

Azure DevOps 2020 and 2019 (and 2018) patch for log4j vulnerability
Extract the downloaded JDK to your AzureDevOpsData\Java folder

Top comments (0)