My colleague Rob Bos has been working on analyzing the GitHub Actions Marketplace for security and it got me thinking about the level of security for Azure Pipelines.

I've been maintaining a set of tasks for quite a while and I maintain the tasks used to publish extensions to the marketplace itself from personal experience I can tell it takes up quite a bit of time to keep my extensions in tip-top state.

This project started as a Xpirit Innovation Day project. Bi-monthly our whole company gets together and works on anything we fancy. This report is the result of 1 day of hacking together a script and many more days digging through the data.

Gathering the data

My first problem was to get all the data required to analyze the extensions. Unlike the GitHub Actions Marketplace, where every action is a public GitHub repository, extensions for Azure Pipelines can be published without a reference to the source material. Instead, each extension is published as a vsix file which in turn contains a folder for each task. The contents of these folders is what is copied over to your agent and executed when you use the task.

To find all extensions in the Azure Pipelines category, I used the Marketplace Search API and to find the download location of each extension I ran the tfx extension show command.

After downloading each vsix file I extracted them to disk (a vsix file is essentially a zip file with a number of manifest files in them).

The result is:

• 1480 extensions
• 2610 tasks
• 2833 task versions
• 30GB of data

Tools used

• To download, extract, parse, recombine the data I used PowerShell.
• To analyze the dependencies I relied on Snyk open source security management.
• To analyze the code for security issues I relied on Snyk codescan.

I completely blew my scan-limits, but somehow Snyk didn't block me until the next day.

Thanks Snyk!

Raw results

If you're interested in digging through this data yourself, the required manifests and scan results are all available on my Gitub profile. This repo contains:

• The script to download all extensions yourself
• The script to analyze each extension
• The script to compile a single report file that sorts all the data by task
• All the key manifest files

It does not contain:

• The vsix files for each extension.
• The extracted sources for each extension.

You can run the scripts to re-download all that data if you wish.

Issues while scanning

During the analysis I ran into a number of issues, some I managed to work around, some I did not:

PowerShell

Tasks can specify a their handler type. Either PowerShell or PowerShell3. The PowerShell3 handler introduced the VstsTaskSdk that tasks are required to package and ship.

But PowerShell tasks are not required to declare their dependencies and when they do ship with dependencies, it's usually in the form of .NET assemblies, not NuGet Packages.

A note on .NET dependency management

PowerShell can call into .NET assemblies natively. The general way to get a .NET assembly is through a NuGet package. Both Snyk and Dependabot track security vulnerabilities at the NuGet package level. Unfortunately, people generally package the assembly files (.dll) and not the whole NuGet, making it very hard to query whether security vulnerabilities exist in the .dll.

Snyk Code also doesn't support PowerShell, nor did I find another scanner that looks purely at security issues to include in this report.

As a result of the above, PowerShell tasks have had a much lighter security screening in this report. It doesn't necessarily mean PowerShell tasks are better or more secure, it simply means their state is mostly unknown.

Webpack

Some extensions have used Webpack or another tool to reduce and optimize the code compressing the whole task into a single .js file. This strips the whole node_modules folder from the task and reduces the overall size of the task considerably.

Unfortunately it also removes the ability for Snyk to scan the dependencies.

A number of these tasks did package their package.json with the task. In those cases I've run npm install against the task folder. However, this won't result in a 1-to-1 comparison to the files used to originally build the extension to the dependency tree I'm now using to scan for vulnerable dependencies.

Development Dependencies

In this analysis I've only looked at the runtime dependencies in the tasks. Any security vulnerabilities in the tools used to develop the tasks is ignored. As a result this report looks at the vulnerabilities you may encounter when you run these tasks in your environment , but the author of the tasks may still be leaking their npm credentials from their dev or CI environment.

Samples and otherwise malformed

The 1480 extensions contain a number of extensions that seem to have been experiments, learning, examples or other non-production code. A few extensions seem to be duplicates of each other published under a different name. From the parsing issues I've had I highly doubt whether some of these tasks can even run if you were to install these in your environment.

My guess would be that these extensions should have never been published as a public extension. It's an indication of the level of screening Microsoft does when extensions are submitted.

Unwanted extension contents

It looks like quite a few extensions pack way more than needed to perform their tasks on your environment. I've found: tests, build-reports, sample files, dev-dependencies and other things that probably never should have ended up in the extension.

In many cases these files won't cause any harm, but some of the dev-dependencies might cause additional vulnerable code to be deployed to your build server.

In some cases the included tests caused false-positives in the code-scan report for embedded passwords (reported to the task owner) or other problems along those lines. I have not tried to filter detected problems out in these cases. These files don't belong in the packaged extension to begin with.

For most people these extra files result in 1 real problem, they cause overhead in storage and processor time every time a pipeline references one of these files.

Classifying problems

Vulnerabilities and vulnerability scores generally assume the worst possible problem a vulnerability can cause in an internet accessible deployment scenario. In that case a RegEx Denial of Service can cripple a whole website. But it generally still requires some payload to reach the vulnerable code and will be less of an issue on a CI/CD server.

Most extensions receive their settings through the YAML file that defines their input and in case an attacker wanted to slow down your build, it would be much easier to sneak a snippet into the build pipeline that prompts for user input or causes the process to pause until the agent kills the job.

On the other hand an arbitrary file write or a zip parsing bug and privilege escalation are more likely to provide sneaky attack paths in a CI scenario. Though, in most cases these are either not applicable, because the agent is already running as local admin, or won't cause too many problems because your agent may always run ephemeral.

I have not done any attempts to classify each vulnerability in the context of a CI/CD agent. In my eyes it's better to remove any detected vulnerability and to stay up to date than requiring your users to validate each vulnerability found.

Most important insights

Deprecated execution handlers

When the new Agent infrastructure first came out in 2015, the agent shipped with an in-process PowerShell handler and a now no longer supported Javascript handler.

The currently deprecated handlers are:

• Node (Node 6)
• PowerShell (in-process PowerShell)

The marketplace currently still serves 551 extensions (out of 1480) that contain tasks that only support these deprecated execution handlers. An extension may package more than one task

Out of 2610 tasks, 948 tasks only support a deprecated task handler in their latest version:

• 670 Node 6
• 278 PowerShell

When including older versions than just the latest of each task, the number goes up to 1005 out of 2833 (Node: 704, PowerShell: 301).

Even if these tasks don't contain direct vulnerabilities, they run on an unsupported runtime (that may itself contain vulnerabilities) and should have been replaced by an implementation that uses Node 10, Node 16 or PowerShell 3 handlers.

Most vulnerabilities are found in dependencies

Pipeline tasks generally contain a very small amount of custom code. For example, the task to publish an Azure DevOps Extension contains 41KB of custom code against 8.4MB of dependencies. These 8.4MB of dependencies are frozen at the time the extension is published, so the responsibility to provide any security updates falls to the extension author. Every version of the azure-pipelines-task-lib, except the latest v3 and v4 have vulnerable dependencies, so it's important for authors to regularly update their extension.

For a PowerShell3 task this ratio is a bit smaller, for example my TFVC Check-in task is 12KB and comes with 640KB of dependencies. The PowerShell dependencies are much smaller because PowerShell relies on the .NET Framework and the PowerShell version installed on the agent. In Windows these are automatically updated through Windows Update and these dependencies will thus be updated out of band and responsibility to update falls to the admin of the agent machine. The VstsTaskSdk for PowerShell has seen far fewer updates and most of these updates are not security related.

Due to their small size and specific usage scenario, it's relatively easy to keep the custom code secure. Due to the large amount of dependencies (that are all zipped up when the extension is published), chances are much bigger that a vulnerability is found there. This is probably why relatively few issues were detected by Snyk Code.

Of the Node based tasks, there is a total of 73824 vulnerabilities introduced through dependencies. 8853 are introduced by old versions of the azure-pipelines-task-lib.

There are 990 tasks (out of 2833) that have vulnerable dependencies. 958 tasks out of those 990 are vulnerable at least due to the fact that they haven't updated their task lib. Tasks that have any vulnerable dependencies almost always include vulnerabilities that are introduced by an older version of the task-lib.

This accounts for 591 extensions out of 1480.

The task-lib version is a good indication for other vulnerabilities. Below is a breakdown of all versions of the azure-pipelines-task-lib (and predecessors) in use by all tasks in the marketplace. As you can see only 192 tasks are using the most recent versions:

Count Name
----- ----
    2 4.0.2
    1 4.0.1-preview
   29 4.0.0-preview
    2 3.4.0        
  158 3.3.1        
   24 3.2.1        
   96 3.2.0        
   95 3.1.10       
   16 3.1.9        
    1 3.1.8        
    3 3.1.7        
   10 3.1.6        
    7 3.1.4        
    6 3.1.3        
   26 3.1.2        
   18 3.1.0
   12 3.0.6-preview.1
   12 3.0.6-preview.0
    3 3.0.1-preview  
   59 2.12.2         
    1 2.12.1         
   12 2.12.0         
    3 2.11.4         
   12 2.11.3         
    4 2.11.2         
    1 2.11.1         
    2 2.10.1         
   20 2.10.0         
   19 2.9.6          
   28 2.9.5          
  165 2.9.3          
  149 2.8.0          
   54 2.7.7          
    3 2.7.4          
    5 2.7.1          
   39 2.7.0          
   11 2.6.1          
   21 2.6.0          
    2 2.5.0          
   34 2.4.0          
    4 2.3.0          
    2 2.2.1          
   15 2.1.0          
    4 2.0.7          
    1 2.0.6          
   18 2.0.5          
    2 2.0.4-preview  
    6 2.0.3-preview
    2 2.0.2-preview
    4 1.5.86       
   30 1.1.0        
    2 1.0.0        
   16 0.9.20       
    1 0.9.7        
    3 0.9.6        
    9 0.8.5        
    2 0.7.4        
    1 0.7.1        
    1 0.7.0        
    1 0.6.2        
    4 0.5.13       
    1 0.5.11       
    4 0.5.10       
    4 0.5.5        
    1 0.5.4        

Table showing the occurrence of each (azure-pipelines|vsts|vso)-task-lib version used

It's interesting to see that certain versions (2.8.0 and 2.9.3) are used by relatively many tasks. 4.x and 3.4.0 had been released days before gathering this data, so there hadn't been much time for extension authors to take action.

A similar breakdown for the PowerShell3 VstsTaskSdk versions in use:

Count Name
----- ----
  486 0.11.0
  284 0.10.0
    1 0.9.0
    5 0.8.2
   34 0.8.1
   12 0.8.0
   26 0.7.1
    9 0.7.0
   28 0.6.4
    2 0.6.3
   26 0.6.2
   12 0.5.1
   48 0.1  

Table showing the occurrence of each VstsTaskSdk version used

As you can see, a lot fewer versions of this library were ever released and the large majority is on the latest 2 versions. As far as I can tell 0.11.0 only contains a bugfix and was not a security related update.

Where the last publication date of a Node based pipelines extension is a pretty good indicator of its health, this may not apply to PowerShell3 based pipelines extensions.

Note: These tables are incomplete because I was unable to detect the exact library version in a number of cases.

Node Versions and Azure DevOps Server.

The Azure Pipelines Agent packs one or more versions of Node to run the tasks. Historically when v2 of the agent shipped in 2017, it shipped with Node6. Over time support for Node10 was added and very recently Microsoft added Node16.

The Azure DevOps Service automatically updates its agents, so people using dev.azure.com will automatically gain support for running Node 16 tasks.

Unfortunately, Azure DevOps Server, and Team Foundation Server before it, only auto-updates the agent to the version that came with the latest update you install on the server. As such, the highest supported Node version is determined by the age of the server version.

Extension authors that need to support multiple versions of Azure DevOps Server must publish an extension that either targets the lowest common denominator, or package an extension that packs dependencies that support Node 6, Node 10 and Node 16. Of those versions, only Node 16 is currently still supported long-term by NodeJS. Node 10 is already unsupported by NodeJS at the time of writing.

Because the Node 6 tasks have always "just worked", the number of tasks that use that runtime and depend on node modules from that era are numerous.

• Extensions that only target Azure DevOps Services should target Node 16.
• Extensions that target Azure DevOps Services and Server should target Node 10 for Server Support and optionally Node 16 for Services.
• Extension that target Team Foundation Server, Azure DevOps Server and Azure DevOps Services should target Node 6, Node 10 and Node 16.

Azure DevOps automatically updates your pipelines to highest available version within the same major version that's configured in your YAML. But it won't auto-upgrade to a new major version. If you're using the old UI based Build and Releases, the UI will render a small 🚩 to signal a higher version is available.

Small flag signaling an update is available

Such an indicator is missing in the YAML editor. And its not very obvious in the UI based editors either.

Because of this, many users will still be using older versions, even if a newer version has been available for a long time. Extension authors can signal users to update by rendering a warning in the older versions, but very few do.

A warning is issues every time a task runs that uses the deprecated PowerShell task handler:

2022-12-23T08:51:14.4719150Z ##[section]Starting: Chuck Norris Quotes
2022-12-23T08:51:14.4840964Z ==============================================================================
2022-12-23T08:51:14.4841258Z Task : Chuck Norris Quotes
2022-12-23T08:51:14.4841520Z Description : Marker task to show Chuck Norris quotes in the build summary report.
2022-12-23T08:51:14.4841772Z Version : 0.2.9
2022-12-23T08:51:14.4841934Z Author : Mathias Olausson
2022-12-23T08:51:14.4842086Z Help : 
2022-12-23T08:51:14.4842294Z ==============================================================================
2022-12-23T08:51:14.5754200Z ##[warning]Task 'ChuckNorrisTask' (0.2.9) is using deprecated task execution handler. The task should use the supported task-lib: https://aka.ms/tasklib
2022-12-23T08:51:14.5757318Z Preparing task execution handler.
2022-12-23T08:51:23.0902590Z Executing the powershell script: D:\a\_tasks\ChuckNorrisTask_6785970c-2d58-4260-b047-0a54028ee9c1\0.2.9\ChuckNorrisTask.ps1
2022-12-23T08:51:23.2397147Z ##[section]Finishing: Chuck Norris Quotes

Execution log showing a warning when a task relied on the deprecated PowerShell execution handler.

But this warning is directed at the wrong audience, unless they submit a pull request or file an issue, there is nothing the person looking at this warning can do to resolve the issue, other than replacing the task with another task that doesn't have this issue.

Below is a breakdown of all of tasks and their supported Node version(s).

Count Name 
----- ---- 
  952 powershell3
  308 powershell (deprecated) 
    8 node16     
  742 node10     
  726 node (deprecated)      

As you can see there is a lot of work to do to upgrade all existing tasks to support the new Node 16 handler. And even though the PowerShell3 handler has been available since 2017, there are still 308 tasks that rely on the deprecated PowerShell handler (which is really only required for Team Foundation Server 2015 support, which doesn't even have Marketplace support).

Extensions can be created in many ways, these are the variables to play with:

• An extension can contain multiple tasks (must have a different ID and name).
• An extension can contain multiple versions of the same task (must have the same ID and name), but each task can have its own set of inputs.
• A task can provide multiple implementations in the same task version (e.g. Node, Node10, Node16 and/or PowerShell3), but each task must have the same set of inputs.

There are multiple strategies to publish extensions to the marketplace, it might be useful to understand these. It also explains why I'm not only looking at the latest versions in the extensions, but at all versions published in the extensions.

Not recommended

Because Azure Pipelines won't auto update tasks if their name and ID change, it's not recommended for extension authors to create separate extensions for each task. This also breaks upgrade scenarios for Azure DevOps Server.

- My Task for TFS 2018
  - Task.json 
    - execution: "Node" - index.js
- My Task for Azure DevOps Server 2020 and 2022
  - Task.json 
    - execution: "Node10" - index.js
- My Task for Azure DevOps Services
  - Task.json 
    - execution: "Node16" - index.js

The fact that no 2 public extensions can publish the same task (id & name) is a limitation imposed by the Marketplace. If Microsoft were to lift this constraint, it would make it easier to publish extensions for backwards compatibility or to publish a new version of a task abandoned by its previous owner.

Most used

To minimize surprises for both authors and users of tasks, most extension (almost all), consider a new Node version a breaking change and they publish a new major version. Backwards compatibility for older Azure DevOps server versions is provided by package the older version of the task as well:

- My Task
  - v1 targeting Node 6
    - Task.json 
      - execution: "Node" - index.js
  - v2 targeting Node 10
    - Task.json 
      - execution: "Node10" - index.js
  - v3 targeting Node 16
    - Task.json 
      - execution: "Node16" - index.js

This also results in the smallest package to deploy to the Pipelines Agent, as it can download only the content it needs to run the task.

Each task can have different inputs in this case.

This strategy is the most used by extension authors and it's how I personally updated my extensions as well. It provides good backwards compatibility, but also allows the author to focus most effort on the latest version.

Most compatible

An author can also put multiple implementations of the same feature in a single version of a task. This will allow the author to create a single version of a task that runs on any version of Azure DevOps Server:

- My Task
  - Task.json
    - v1
      - execution: "Node" - node6/index.js
      - execution: "Node10" - node10/index.js
      - execution: "Node16" - node16/index.js

The resulting package for the Pipelines Agent is much bigger and this way old (Node 6) dependencies may be downloaded to the pipelines agent, even if it doesn't need them. This can be unwanted in certain cases (e.g. organizations that are under certain governance regimes).

Issues of historical nature

Since the release in 2015, many new features were added to the agent. Tasks can enforce a minimum agent version, but most omit this in order to support older versions of Azure DevOps Server and Team Foundation Server.

You'll find the docs are littered with "support was added in 2.122.0" and "you need to use 2.125.0 or above version agent".

Some tasks query for this agent version in code and emit a warning if less secure behavior is used because the task is running on an old agent.

let agentVersion = tl.getVariable('Agent.Version');
if (agentVersion && agentVersion.length && semver.gte(agentVersion, '2.117.0'))
{}

Code snippet showing how to check for specific agent versions

Most tasks do not. The most common example of this behavior is surrounds disabling of certificate validation in Node. The Node10 handler introduced support for the NODE_EXTRA_CA_CERTS environment variable. So to use this feature, you require an agent that supports Node 10 and a task to register a Node 10 execution handler.

The right way to do it, would be to explicitly disable certificate checking only on the requests that need this:

if (usingSsl && this._ignoreSslError) {
    // we don't want to set NODE_TLS_REJECT_UNAUTHORIZED=0 since that will affect request for entire process
    // http.RequestOptions doesn't expose a way to modify RequestOptions.agent.options
    // we have to cast it to any and change it directly
    agent.options = Object.assign(agent.options || {}, { rejectUnauthorized: false });
}

Code snippet of a HttpClient that can be configured to ignore TLS security.

Most tasks turn off certificate checks for the whole task's execution using NODE_TLS_REJECT_UNAUTHORIZED=0. If the task itself or any of its dependencies try to contact an external resource, this resource can now be spoofed. At least an agent version check to tell the operator to upgrade their agent will help administrators to enable these new features (but fails to take into account the Azure DevOps Server version might not even ship with that agent version). This extension also uses a property to control this behavior and defaults to a secure configuration.

if (tl.getBoolInput('disableCertCheck', false)) {
    // Disable NodeJS certificate validation for the current process
    process.env.NODE_TLS_REJECT_UNAUTHORIZED = '0';
    // Log a warning if the agent supports the NODE_EXTRA_CA_CERTS variable
    let agentVersion = tl.getVariable('Agent.Version');
    if (agentVersion && agentVersion.length && semver.gte(agentVersion, '2.117.0')) {
        this.logger.log(tl.loc('DoNotDisableNodeJsCertCheck'));
        this.result.warning(tl.loc('DoNotDisableNodeJsCertCheck'), true);
    }
}

Example of an extension using a check against the agent version to warn operators to update the agent.

But quite a few tasks disable certificate checking altogether without any parameter or hidden variable:

"use strict";
process.env.NODE_TLS_REJECT_UNAUTHORIZED = "0";

Example of an extension that disables certificate security altogether without any configuration or warning.

Luckily Node will render a warning in this case, but given how many tasks do this, I doubt pipeline authors take this warning into account:

Starting: Task
=====================================================================
(node:4440) Warning: Setting the NODE_TLS_REJECT_UNAUTHORIZED environment variable to '0' makes TLS connections and HTTPS requests insecure by disabling certificate verification.
(Use `node --trace-warnings ...` to show where the warning was created)

Node logging a warning when TLS security is disabled when an outbound request is made.

There are 130 tasks that disable certificate validation of the total of 2833 tasks.

Unfortunately there is very little documentation available on which features were added in which version of the agent, which agent version shipped with what version of Team Foundation Server and Azure DevOps Server and which agent versions support which operating systems.

Recommendations

To extension/task authors

Many of the security issues are easily detected and mitigated. I recommend pipeline authors:

• Enable Snyk Open Source or GitHub Dependabot Security Updates - these tools are free for open source repositories and will automatically create pull-requests to update your extensions. This will resolve most of the dependency problems in the Azure Pipelines ecosystem.
• Enable GitHub Dependabot Updates - Instead of only updating when there is a security vulnerability, you can configure Dependabot Updates for any dependency that has an update available. By staying up to date, you reduce the chance of having to fix compatibility issues when a security problem is found.
• Update your dependencies now and regularly - When you first enable the tools above, you'll likely get a lot of dependency updates. But some need to be performed manually. Neither Snyk nor Dependabot will currently suggest replacing vsts-task-lib with azure-pipelines-task-lib for example.
• Enable Snyk Code or GitHub Advanced Security - Where the above tools solve security issues found in your dependencies, these tools will detect security issues in your code.
• Publish your extensions from CI/CD - Assuming you're using Azure DevOps, it's very easy to publish your extensions through Azure Pipelines. I maintain the Azure DevOps Extension Tasks to help you accomplish that. The docs provide a good starter recipe.
• Add support for Node 10 and Node 16 if you currently only support Node 6 - The Node 6 handler is now officially deprecated. Extensions should only ship with Node 6 for backwards compatibility with Team Foundation Server 2017 and 2018. To support Azure DevOps Server 2019, 2020 and 2022 you should add support for the Node 10 handler. For Azure DevOps Services you should add support for Node 16.
• Publish your Git repository in your extension manifest - Your extension manifest can publish your GitHub or Azure Repos repository information. This will allow administrators to verify the contents of your extension more easily.
• Deprecate tasks that are no longer maintained - A task can be marked "deprecated" in the task.json file. This will make the task harder to find in Azure Pipelines, reducing the chance of new pipelines using your old tasks. Optionally add a warning to your task to explain pipeline authors how to proceed. You could even go further and include a time-bomb that will fail the task in say 12 months from now.
• Unpublish extensions that are no longer maintained - An extension can be unpublished in the marketplace. This will make the extension invisible to new users, but existing users will still be able to use your extension. Optionally publish a final version that adds a warning and marks every task deprecated.
• Publish sample, test, demo tasks as private- when publishing an extension you can mark it as "private" and then share the extension only with the Azure DevOps accounts you want to grant access.
• Add contribution guidance to your GitHub repositories - I'm guilty on this one. It's usually hard for other people to contribute to extensions published on the marketplace. Even if you can find the GitHub repo for the extension, building it, running the tests (if even available) and deploying a test version to your environment safely is hard. Good documentation and a release pipeline that can provide a usable vsix file with instructions would go a long way to wards encouraging people to contribute.
• Reduce the amount of code shipped with your extension - many tasks contain way more than needed to run on the agent. This not only slows down every pipeline run by needlessly having to extract thousands of files, it also makes it harder to analyze the true security impact of the extensions we install into our Azure DevOps organizations. When publishing your extensions please strip out: tests, samples, dev-dependencies etc. Consider using Webpack or a similar tool to reduce your tasks to the smallest possible. But if you do, please do include the package.json and the package-lock.json to enable easy analysis.
• Do not disable security features lightly - I was shocked by the number of tasks that disable certificate security. These tasks potentially enable an attacker to send data to an untrusted location or to force a task to download files from an untrusted location. I understand why these authors chose to do this, it's hard to configure the agent correctly behind a corporate proxy or when talking to a host that's protected by a self-signed certificate. It's even harder to explain others how to use your tasks under these circumstances. But please try harder.

To Azure DevOps Administrators

As an Azure DevOps Administrator it's hard to tell whether your Organization has extensions installed that contain vulnerabilities and whether your pipeline authors are keeping their pipelines up to date.

• Verify your Azure DevOps Organization or Server extensions - Verify the security of an extension prior to installing it into your organization. Currently Microsoft provides no indicators for the state of the security of extension in the marketplace.
• Verify your Azure DevOps Organization or Server extensions regularly - It's not enough to verify the extensions only upon installation. Over time new vulnerabilities are found in existing code, so extensions should be updated regularly. You could leverage some of the scripts I used to generate this report, or another snippet of code I use to download the "taskzips" which are the packages downloaded by the agent prior to running a job.
• Clean up extension history - When an extension is installed, all tasks in that extension are made available. Every time an extension author updates their extension the new versions of any tasks in the extension are added to your Azure DevOps Organization or Server. These older versions aren't removed automatically. You can use tfx build tasks list to list all available versions on your server and tfx build tasks delete to delete specific (older) versions. Alternatively you can uninstall & reinstall an extension to purge the old versions of the tasks.
• Use the REST API - Check which versions of tasks are currently in use and whether a newer version might be available. Depending on whether you use templates in your organization, you may need to rely on the Timeline API to check recent job history for usages of specific task versions.
• Keep your Azure DevOps Server up to date - If you're still using Team Foundation Server or Azure DevOps Server 2019, it's really time to upgrade to Azure DevOps Server 2022 or consider migrating your instance to Azure DevOps Services. It's the best way to get the most recent version of Microsoft's built-in tasks as well as access to the latest security features of the Azure Pipelines Agent.
• Keep your agents up to date - Support for Node 10 and Node 16 may require an agent upgrade. Make sure your build and deployment pipelines are running the latest available agent version.
• Educate your users - Make sure your Pipeline authors and Developers are aware of the dangers and keep their pipelines up to date. Help them select healthy extensions and find alternatives to any extension you may need to part with.
• Sponsor your most used extensions - Many organizations depend on extensions to build and deploy their most precious bits to production. Many of those extensions were built by the community and are maintained in the spare time of the authors. Consider sponsoring or asking your employer to sponsor the publishers you depend, me for example.
• Install an agent that does not contain the deprecated task handlers - Microsoft now provides a separate agent download (see alternate agent downloads) that does not contain Node 6. This will remove the ability to execute Node 6 based tasks.
• Install the most recent agent, even if your Azure DevOps Server / Team Foundation Server doesn't support them - There is a magic environment variable you can set to allow you to configure an agent that is much newer than the version that shipped with your version of Azure DevOps Server.

To Azure Pipeline Authors

As pipeline author it falls to you to stay up to date with the latest versions of tasks and resolve any compatibility issues when updating. But there are almost no tools available for you to do this work.

• Inspect pipeline logs - Many security issues are logged to the pipeline log. Usage of deprecated task handlers, Disabling of TLS security in Node, etc. Pipeline authors should inspect their logs and see whether updates are available/required.
• Update your tasks - If a new major version of an extension becomes available, check your existing pipelines and check whether they need to be updated. Subscribe to the Azure DevOps Marketplace bot to get notifications in your twitter feed.
• Submit pull-requests to extensions - Many extension developers have built their extensions in their own spare time. Over the last 7 years building my extensions I've had many complaints, but very few pull requests. If you find an issue in an extension, consider helping the author by providing a pull-request.
• Sponsor your most used extensions - Many organizations depend on extensions to build and deploy their most precious bits to production. Many of those extensions were built by the community and are maintained in the spare time of the authors. Consider sponsoring or asking your employer to sponsor the publishers you depend, me for example.
• Not everything has to be a task - It's perfectly OK to use a snippet of powershell or bash instead of a task. You can even use templates to create reusable steps of those snippets of scripts.

To Microsoft

As a long-time extension author I know how hard it is to develop and maintain extensions, as long-time Azure DevOps Administrator I know how hard it is to keep my environment secure and as Azure Pipelines Author I know how hard it is to make sure I'm always on the latest version of everything. And neither Azure DevOps nor the Marketplace is really helping.

• Provide insights in usage of extensions - Currently an extension author has no visibility in the versions of Azure DevOps Server and the Azure Pipeline Agent used to run the tasks. This makes it very hard for authors to know whether they can safely deprecate old versions.
• Encourage extension authors to enable Dependabot on their repository - When an extension contains repository details, Microsoft and GitHub could verify that the repository has the free GitHub Advanced Security features enabled. Microsoft could even mandate that for popular or verified publishers.
• Encourage extension authors to enable 2FA their accounts - Presently it's not required to have 2FA enabled on your Microsoft account to publish an extension. At least verified publishers should have 2FA enabled, but better would be if all publishers had 2FA enabled. This also applies to the GitHub account used (if available).
• Make security state of extensions visible in the marketplace - I expect Microsoft to perform a regular scan of all extensions and for them to use tools like Dependabot and CodeQL to ensure te security of the tasks in the marketplace. The details should be available in the publisher portal and a global indicator for users browsing the marketplace.
• Make security state of extensions visible in Azure DevOps Services/Server - Currently administrators can't easily see the state of security of any of the extensions installed into the organization.
• Encourage pipeline authors to update their tasks - In the UI based pipelines a subtle 🚩 at least indicated an update was available. In YAML pipelines any indication is gone. This essentially means it's unlikely a pipeline author will ever update the task version after the initial development of the YAML file. An indicator should be added in the editor and potentially in the logs.
• Truncate/disable old task versions - Currently Azure DevOps will keep a copy of every version of every task that was ever installed/updated. This means that even when a Extension Author removed say v1 from their extension, it will remain available to any organization that had that older version installed previously. Only if the extension author publishes a final v1 version that logs a warning or an error will the old version be disabled (in most cases). An old version can also not be marked deprecated as that will mark all versions of that task deprecated. Azure DevOps should automatically trim the older versions of the tasks or at least disable them.
• Provide better documentation for extension developers - Current documentation describes how to support the latest version of Azure DevOps Server and Azure DevOps Services. There is very little documentation available about the agent versions that shipped with each version of Azure DevOps, which azure-pipelines-task-lib they support, what version of Node they support and how to best create an extension that supports all supported versions of Team Foundation Server and Azure DevOps Server our there in the wild.
• Better curate the marketplace - The number of low quality extensions that are published as public without proper ways for Administrators and Pipeline authors to verify the quality of the extension is bad. The ability to publish a public extension used to require manual validation, I can see how that has been time consuming and expensive, but the bar to publish a public extension should be higher. A review of existing extensions should be done.
• Vulnerability reporting - a number of extensions contain real vulnerabilities or secret information. At present there is no way, other than through an optionally registered GitHub link or through a review to submit a security finding to an extension author. Either extension authors should publish a report mechanism in their extension manifest (through GitHub Private Security Reporting for example), or the marketplace should add the ability to notify Extension publishers through the marketplace.
• Add Azure Pipeline YAML support for Dependabot - With Advanced Security features coming to Azure DevOps, those features should support Azure Pipelines as a first citizen. Extend Dependabot to automatically update the Azure Pipeline files. Use Dependabot and CodeQL to analyze the extensions installed in my organization.
• Provide better support for managing certificates on the agent - There's really no excuse for the difficulty to configure certificates and proxy configurations within the toolchain the Azure Pipelines Agent relies on. Ideally the agent would ensure each of these tools is configured correctly each time the agent starts instead of relying on administrators and task authors to know all of these details.
• Allow task authors to publish the same task (id & name) in different extensions - In the past (2017-2018), extension publishers could publicly release the same tasks in more than one extension. This was disabled through a Marketplace serverside validation at dome point. This prevents extension authors from publishing the same extension with explicit support for specific versions of Azure DevOps Server or Team Foundation Server. This would resolve much of the packaging and versioning problems extensions authors currently have trying to support older versions of the server.
• Make it easier for people running older versions of Azure DevOps or Team Foundation Server to run the latest agent - Please provide official documentation and guidance for administrators to run the latest available agent version on their self-hosted agents. Consider removing the need to set a magic environment variable to enable this scenario. Consider providing a patch for older server versions to provide a download link to the latest available agent version (instead of the latest version that came with the product).
• Lead by example - Microsoft has been very late in its steps to provide Node 10 compatible versions of even the built-in tasks, let alone the extensions published under publishers that are owned by Microsoft (1ES, ms-*, Premier-support, etc). This not only sets a precedent, it also means other extension authors have very little examples to rely on to use for inspiration.

Conclusion

At present the security state of the Azure Pipelines extensions of pretty bad. About 35% of all the tasks available in the marketplace have security issues. Across 40% of all extensions. And unfortunately this isn't limited to extensions that only have a few installations.

I've recently updated all my extensions again and in some cases I had gathered at least 100 new vulnerabilities since the last time I updated them. This has lead me to configure Dependabot and GitHub Advanced Security on all of my repositories and prompted the writing of this report. I'm adding Snyk Code next.

This problem isn't going away. Node 16 will be replaced by Node 18 sooner or later and we will keep finding security issues.

It is hard for Administrators and Pipeline authors to validate the security and quality of extensions prior to use and even post-installation.

I hope Extension Authors and Microsoft will see this report as a call to action and take steps to improve this situation. Extension Authors by updating their extensions. Microsoft by making it easier for everyone to do the right thing.

And again, thanks Snyk for graciously failing to enforce my scan limit as I burned through it in record tempo. I can highly recommend their tools.