Vercel, a Platform as a Service (PaaS) for hosting Next.js web applications, offers some really convenient features like instant rollbacks. This post explores how to use this feature in a custom CI pipeline to allow colleagues without a Vercel login to carry out rollbacks.
Vercel is built on a serverless architecture, so it's cheap for them to keep copies of all your previous deployments. This allows for some cool features, like a vercel bisect
CLI command that works like git bisect
, except instead of checking out file changes locally to find a bug, it gives you URLs of each deployment to test before telling you which deployment introduced an issue.
Β Vercel instant rollbacks
Another cool feature is instant rollbacks. If you deploy a change that causes issues, rather than doing a git revert
or a hotfix and then re-running your build/test/deploy pipeline you can instead instantly revert to the last deployment, saving lots of time and stress. You can do a rollback via logging into Vercel's admin UI, via their CLI, or via an undocumented REST endpoint.
A downside to Vercel is that it's a premium service with a premium cost for the higher-end tiers. You're charged per-seat (per developer), so in many organisations not everyone will have a Vercel login. This is a bit of a problem for instant rollbacks β what if the developer who spots the production issue doesn't have a Vercel account so can't use their admin UI, CLI, or REST API?
To work around this issue at work I recently added a 'Rollback' step to the end of our CI pipeline. Every developer has access to the pipeline so will be able to initiate a rollback via GitLab CI instead of Vercel. The pipeline is authorised to connect to Vercel's API using an API token associated with an existing Vercel account.
How I did it
Firstly I generated a Vercel API token and added it to GitLab CI as an environment variable called VERCEL_TOKEN
. I also created a VERCEL_TEAM_ID
and VERCEL_PROJECT_ID
using the team and project IDs from the Vercel web interface.
Vercel's REST API documentation is missing docs for a 'rollback' endpoint, but Vercel's CLI is hosted on GitHub and you can see how they are using their own REST API (a good sign!) for the rollback
CLI command:
await client.fetch(`/v9/projects/${project.id}/rollback/${deployment.id}`, {
body: {}, // required
method: 'POST',
});
The rollback process, via API
Before we can send a rollback request to the API we need to find the previous production release's deployment ID. Here's how I did that in Bash, using jq:
# Fetch the rollback deployment ID
ROLLBACK_DEPLOYMENT_ID=$(curl -s "https://api.vercel.com/v6/deployments?teamId=$VERCEL_TEAM_ID&projectId=$VERCEL_PROJECT_ID&limit=2&rollbackCandidate=true&state=READY&target=production" -H "Authorization: Bearer $VERCEL_TOKEN" | jq -r '.deployments[1].uid')
echo "Rolling back to the previous deployment ID: $ROLLBACK_DEPLOYMENT_ID"
Note the query string parameters in that request:
-
limit=2
, as we want to get the previous production deployment, not the current one -
rollbackCandidate=true
, as we need a deployment we can rollback to -
state=READY
, to filter out deployments that might be in-progress -
target=production
, because we are not interested in preview deployments
We pipe (|
) the silent curl
output to jq and use it to get the second uid
from the results. We now have a variable called ROLLBACK_DEPLOYMENT_ID
that we can include in a POST
request to Vercel's undocumented rollback endpoint:
# Initiate a rollback to the previous deployment
curl -s -X POST "https://api.vercel.com/v9/projects/${VERCEL_PROJECT_ID}/rollback/${ROLLBACK_DEPLOYMENT_ID}?teamId=${VERCEL_TEAM_ID}" -H "Authorization: Bearer $VERCEL_TOKEN" -H "Content-Type: application/json"
If both those commands are successful then your site should rollback to the previous production release, hopefully before too many of your users have noticed the issue.
GitLab CI example
I added the above commands to a scripts/rollback-current-prod-deployment.sh
file, made it executable, and then referenced it in a GitLab CI job like this:
Rollback current prod deployment:
image: registry.gitlab.com/gitlab-ci-utils/curl-jq:latest
stage: deploy
needs:
- job: Upload to Vercel prod URL
optional: true
script:
- './scripts/rollback-current-prod-deployment.sh'
rules:
- if: $CI_PIPELINE_SOURCE == 'merge_request_event'
when: never
- if: $CI_COMMIT_BRANCH != "main"
when: never
- when: manual
This job:
- only runs manually
- only runs after the 'Upload to Vercel prod URL' job
- uses a Docker image that contains both
curl
andjq
. (Note that despite the name, this image is not an official GitLab product)
It'd be a very similar approach to use this with GitHub Actions or any other CI/CD pipeline provider.
How to take this idea further
I made a few other additions that are application-specific so I haven't covered here.
An 'undo rollback' job
I created a job to undo the rollback, in case it was triggered accidentally or was no longer needed. This means the original production deployment can be quickly restored, without re-running the whole pipeline
I did this by storing the original deployment ID and writing it to a file that a second pipeline step could read from.
Confirmation that the rollback worked by displaying the deployed version after the rollback succeeded
I created an /api/version
endpoint in my application that returned the version of the site in a JSON object. I query this after the rollback job so I can give the pipeline user confirmation that the rollback succeeded. I set the API route to have no caching with a max-age
header (public, max-age=0, must-revalidate
), but just to be sure I added some cache busting to the CI job too:
# Fetch version information twice to help bust cache and stale-while-revalidate
cache_buster=$(date +%s)
curl -s -o /dev/null "https://example.com/api/version?cache_buster=$cache_buster"
VERSION_INFO=$(curl -s "https://example.com/api/version?cache_buster=$cache_buster")
echo "Version after rollback: $(echo $VERSION_INFO | jq -r '.version')"
Top comments (0)