Testing for Breaking Changes in Fastify APIs

Recently I was approached by a team that needed help testing their Fastify API for breaking changes. Fastify was making it easy to quickly ship a lot of new functionality, but breaking changes were making it through Code Reviews. They were not finding out the changes were breaking until a consumer emailed them — not good. The developer who reached out saw my work on the Optic project and asked for help.

This post is about how I helped them test for breaking changes in CI. It’s written like a tutorial so you can follow along too.

API Lockfiles

APIs are a contract. When you publish a new endpoint, you are making a new promise to consumers. When you change how that API works, it might break that promise. The difficulty is knowing what promises you have already made, and figuring out if your code changes break them. With all the abstractions in a backend codebase, changing one line of code can easily change a dozen API endpoints without anyone realizing it.

What we need is a lockfile for our API’s behavior. That would make it easy to:

• Review API Diffs (the behavior changes) at the same time we do Code Review
• Make it possible to test for breaking API changes in CI
• Refactor our backend safety — lots of code can change, but the lockfile should stay the same

Fastify has great support for generating OpenAPI from code. Like many other popular backend frameworks, using JSON Schema for validation and exporting OpenAPI have first-class support. We can use this generated OpenAPI specification as a lockfile for the API.

Generating our API Lockfile (OpenAPI)

First let’s get our current OpenAPI specification out of Fastify and onto the file system. If you have not added the https://github.com/fastify/fastify-swagger plugin first go do that. Then use this simple script I called generate-spec.ts to write your OpenAPI specification to the filesystem.

import fs from "node:fs/promises";
import { setupApp } from "./app";
const FILE_PATH = "./openapi.json";

(async () => {
  const app = await setupApp();

  await app.ready();
  await fs.writeFile(FILE_PATH, JSON.stringify(app.swagger(), null, 2));
})();

I added it to the scripts section of my package JSON so it is easy to call.

"scripts": {
  "build": "yarn run tsc --build --verbose",
  "generate-spec": "yarn ts-node ./src/generate-spec"
},

Now the most recent OpenAPI is at openapi.json and is re-generated whenever I run:

yarn run generate-spec 

Tracking our API lockfile with Git

Lockfiles are one of the few generated artifacts it makes sense to track with Git. Adding generated OpenAPI specs to our git repos gives us a quick way to lookup how the API worked at various points in time using Git tags, commit SHAs, and branch names.

Suggestion: if you want to ensure that the OpenAPI on each branch is accurate, consider adding the following CI task:

yarn run generate-spec
git diff --cached --exit-code --quiet # will exit 1 if the checked-in spec is stale

Testing for breaking changes

Now that we have a way to lookup our API’s behavior with Git, we can start testing for breaking changes between versions of our API. We’ll be using Optic (an open source tool I created) to do just that. If you are looking for other options I recommend https://github.com/OpenAPITools/openapi-diff or https://github.com/Tufin/oasdiff.

First install the Optic CLI:

yarn global add @useoptic/optic

Then use the diff command to test for breaking changes between your feature branch and the main branch. This will flag any breaking changes during Code Review.

optic diff openapi.json --base main --check

This command will exit 1 if a breaking change is detected so it can be used like a test in CI.

Bringing it all together

• Generated OpenAPI specifications can be used as API Lockfiles. Many of the most popular API frameworks have been adding first-class support for generating OpenAPI.
• fastify + fastify-swagger make it easy to generate accurate OpenAPI specifications from your code
• optic makes it test for API changes between Git branches and tags

Now you can ship changes to your fastify APIs with confidence that you won’t break your consumers.