DEV Community

Jesse Houwing
Jesse Houwing

Posted on • Originally published at jessehouwing.net on

Hey! You broke our pipeline!

Hey! You broke our pipeline!

One of the advantages of Azure Pipelines is that it automatically updates your tasks to the latest minor version. That way you don't have anything to stay up-to-date. But this also has disadvantages. Both Microsoft and Extension Authors can accidentally break your pipelines.

I've personally shipped a version of my Variable Toolbox extension that broke 10.000 developers at a single corporation that relied on my pipeline tasks.

And yesterday Microsoft had a world-wide outage due to the npmAuthenticate@0 task suffering from an unexpected breaking change.

Hey! You broke our pipeline!

Most people don't know what to do when something like this happens. But luckily there are a few things you can do to fix thing yourself when you find yourself in this situation.

Pin to an older version

If you're using YAML pipelines, then the simplest way to get unblocked, is to pin your workflow to a specific version of the task. This is one of the big advantages of YAML based pipelines. By default Azure Pipelines will reference a task by its major version @0 in the case of the npmAuthenticate task, but you can specify @0.208.1 or any other previous version instead:

# Broken
- step: npmAuthenticate@0
  inputs:
    workingDirectory: $(Build.SourcesDirectory)

# Fixed
- step: npmAuthenticate@0.208.1
  inputs:
    workingDirectory: $(Build.SourcesDirectory)
Enter fullscreen mode Exit fullscreen mode

The easiest way to find the version to enter here is to look at the last pipeline that succeeded, the Initialize Job step will list the exact versions used:

2022-12-23T09:05:03.7682146Z ##[section]Starting: Initialize job
2022-12-23T09:05:03.7684106Z Current agent version: '2.213.2'
2022-12-23T09:05:03.7719717Z Current image version: '20221215.2'
2022-12-23T09:05:03.8121128Z Download all required tasks.
...
2022-12-23T09:05:03.8228407Z Downloading task: npmAuthenticate (0.208.1)
2022-12-23T09:05:06.6642168Z ##[section]Finishing: Initialize job
Enter fullscreen mode Exit fullscreen mode

Overwrite the broken task

But if you're using classic build and release pipelines (UI based), then you can't pin to a minor version. In that case you'll need to upload a version of the task that works. You can use tfx-cli to do that:

TFS Cross Platform Command Line Interface v0.12.0
Copyright Microsoft Corporation

Syntax:
tfx build tasks upload --arg1 arg1val1 arg1val2[...] --arg2 arg2val1 arg2val2[...]

Command: upload
Upload a Build Task.

Arguments:
  --task-path Local path to a Build Task.
  --task-zip-path Local path to an already zipped task
  --overwrite Overwrite existing Build Task.
Enter fullscreen mode Exit fullscreen mode

But first you'll need to get a copy of the task that works. Your Azure DevOps instance already has a copy of that task, you just need to know where to find it:

$pat = "{{AZURE DEVOPS PAT TOKEN}}"
$org = "{{AZURE DEVOPS ORG NAME}}"
$url = "https://dev.azure.com/$org"
$header = @{authorization = "Basic $([Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes(".:$pat")))"}

$taskName = "npmAuthenticate"
$taskid = "ad884ca2-732e-4b85-b2d3-ed71bcbd2788"
$taskversion = "0.208.1"
$taskZip = "$taskName.$taskid-$taskversion.zip"

Invoke-WebRequest -Uri "$url/_apis/distributedtask/tasks/$taskid/$taskversion" -OutFile "$taskZip" -Headers $header
Enter fullscreen mode Exit fullscreen mode

Or you can grab a copy of an older version from my azure-pipelines-tasks-zips repository:

Hey! You broke our pipeline!

Now in order to "fix" the broken version we'll have to overwrite it. To do so we're going to patch the older working version with the broken version's version number.

First extract the zip file, then update the task.json file and set the version number to the one you want to overwrite (in our case 0.214.1):

Hey! You broke our pipeline!

Put the task.json back into the zip file.

We're now ready to upload the task. We can use tfx-cli to do that.

We first need to delete the existing task using tfx build task delete --task-id (warning this will delete all older versions):

Hey! You broke our pipeline!

And then we can upload the patched task tfx build tasks upload --overwrite --task-zips-path file-we-just-patched.zip:

Hey! You broke our pipeline!

Instead of deleting the existing task and re-uploading it, we can also patch the taskzip with a version number that's newer than the broken task, in our case 0.214.2 and then upload that.

As you can see here, that's what the Azure Pipelines team does too, but they roll out the task through their own provisioning process, they either hotfix the task, or simply roll back their changes:

Comparing master...releases/m215 ยท microsoft/azure-pipelines-tasks
Hey! You broke our pipeline!

Temporarily replace the broken task

Instead of overwriting, we can also install a temporary version of the task with a different id and name, reconfigure our pipelines to use the temporary replacement and switch back when the Azure Pipelines team has fixed theirs.

The process is very similar to replacing the existing task, but in this case we need to make a few more changes, we need to replace the id (choose any guid), name and update the friendlyName properties in both task.json and task.loc.json:

Hey! You broke our pipeline!

Or you can download a pre-patched (-sxs) version from the azure-pipelines-tasks-zips repository.

Then we can upload the patched file:

Hey! You broke our pipeline!

But now you'll need to go through all of your pipelines and replace the old task with the temporary replacement. A bit of scripting against the REST API would make that a quick change.

Top comments (0)