How to Run Snapshot Tests in CI/CD with Playwright

Visual tests are a powerful tool, but they're hell to work with in CI/CD. They're slow, flaky, a pain to debug, and difficult to maintain.

Getting them going is no easy task. After all, you need some way to store the snapshots between runs, and you need some way to let your pipeline know when it's OK to update the snapshot cache.

Thankfully, I took care of all this for you. :)
This article solves all of these problems, and lays the foundation for you to take your visual tests to the next level.

If you get value here, you'll love my Ultimate Guide to Visual Testing in Playwright. It covers advanced testing techniques, fine-tuning your snapshot configuration, and more.

Let's get started...

1. Configure Playwright for CI/CD

Your CI/CD environment is not like your local environment, and we should account for that in our Playwright configuration.

Update your playwright.config.ts with the following options. Feel free to merge these with any other options you've set so far:

import {defineConfig, devices} from '@playwright/test';

const isCI = !!process.env.CI;

export default defineConfig({
  timeout: 1000 * 60,
  workers: isCI ? 1 : '50%',
  retries: isCI ? 2 : 0,
  forbidOnly: isCI,

  outputDir: '.test/spec/output',
  snapshotPathTemplate: '.test/spec/snaps/{projectName}/{testFilePath}/{arg}{ext}',
  testMatch: '*.spec.{ts,tsx}',

  reporter: [
    ['html', {
      outputFolder: '.test/spec/results', 
      open: 'never',
    }],
    isCI ? ['github'] : ['line'],
  ],
});

Here's what's going on:

• The top few options account for the limited resources available in a CI environment. Tests will run slower, and they'll be a little more flaky, so we need to adapt accordingly.
• The next batch of options organize all of your tests' outputs into the .test directory. This will make it easy to cache. Add .test/ to your .gitignore file. None of this stuff should be committed to your repo.
• Lastly, we're going to generate html and github reports in CI/CD. The html report will be useful for debugging, and the github report will be useful for reviewing results with minimal output.

2. Run your tests in CI/CD

Next, let's create a basic pipeline. It doesn't do everything (yet), but it's a solid foundation to work from.

Create a file at .github/workflows/visual-tests.yml:

name: Visual Tests

on:
  push:
    branches:
      - "*"

  pull_request:
    branches:
      - "*"

jobs:
  run-tests:
    runs-on: ubuntu-latest
    timeout-minutes: 15
    steps:
      - name: Checkout
        uses: actions/checkout@v4

      - name: Set up Node
        uses: actions/setup-node@v3
        with:
          cache: pnpm
          node-version-file: .nvmrc
          check-latest: true

      - name: Install deps
        run: |
          npm install
          npx playwright install

      - name: Test
        run: npx playwright test --ignore-snapshots

Notice that we're running our tests with the --ignore-snapshots flag. Since we have no way to store snapshots between test runs, this is literally the only way to get our tests to pass.

If you push your repo to Github now, this workflow will run, and it will pass, but it won't guarantee anything about your app.

Let's remedy that...

3. Store snapshots between runs

For our visual tests to work, we'll need to store snapshots between runs. For this, we'll cache them using Github's "artifacts" feature.

Second, let's also write a step that generates new snapshots if they don't exist. This will solve the problem of our tests failing the first time they run on a branch.

Third, let's stop ignoring our snapshot assertions.

Here's the updates:

      - name: Install deps
        run: |
          npm install
          npx playwright install

      - name: Set up cache
        id: cache
        uses: actions/cache@v4
        with:
          key: cache/${{github.repository}}/${{github.ref}}
          restore-keys: cache/${{github.repository}}/refs/heads/master
          path: .test/**

      - name: Initialize snapshots
        if: ${{steps.cache.outputs.cache-hit != 'true'}}
        run: npx playwright test --update-snapshots

      - name: Test
        run: npx playwright test

Notice that our cache key includes the current branch as well as a fallback to the master branch. That way a new branch has some material to work from on the first push.

If you trigger the pipeline now, on the first run, you'll see snapshots generated, the tests pass, and the cache updated. On subsequent runs, you'll see the cache hit, no new snapshots generated, and the tests still pass.

Not bad, but what happens when the snapshots need to be updated?

4. Trigger a snapshot update

Now that we have visual tests that really work, let's change our workflow so that we can update snapshots on demand.

First, let's parameterize our workflow...

on:
  push:
    branches:
      - "*"

  pull_request:
    branches:
      - "*"

  # Allow updating snapshots during manual runs
  workflow_call:
    inputs:
      update-snapshots:
        description: "Update snapshots?"
        type: boolean

  # Allow updating snapshots during automatic runs
  workflow_dispatch: 
    inputs:
      update-snapshots:
        description: "Update snapshots?"
        type: boolean

workflow_call allows us to manually trigger the workflow with or without updating snapshots. workflow_dispatch allows other Github workflows the same set of options.

Here's what the manual widget looks like:

Let's also update "Initialize snapshots" to trigger an update when this parameter is enabled:

      - name: Initialize snapshots
        if: ${{steps.cache.outputs.cache-hit != 'true' || inputs.update-snapshots == 'true'}}
        run: npx playwright test --update-snapshots --reporter html

Lastly, let's upload our HTML report as an artifact, so that we can reference it before deciding if we should ask the workflow to update snapshots... or if we need to debug the problem:

      - name: Test
        continue-on-error: true
        run: npx playwright test

      - name: Upload test report
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: playwright-report
          path: .test/spec/results/
          retention-days: 30
          overwrite: true

Notice that our "Test" step now enables continue-on-error, so that we can upload our report even if the tests fail.

The new "Upload test report" step uploads the report as an artifact. In the workflow results, you'll see a direct link to a zip file containing our report.

5. Speed up your CI/CD pipeline... a lot!

Things are going great, but there's one more thing we can do to make visual tests even better: make them blazingly fast!

You've probably noticed two steps are slowing your pipeline down:

1. Installing Playwright browsers.
2. Running tests serially.

Thankfully, we can fix both of these issues, and most-likely for free!

BrowserCat hosts a fleet of Playwright browsers in the cloud that you can connect to with just a few lines of code.

If you do so, you won't have to install browsers in your CI/CD environment, and you'll be able to run your tests entirely in parallel.

And the best part? BrowserCat has an awesome forever-free plan. Unless you're running a huge team, you'll probably never have to pay a dime.

So let's get started!

First, sign up for a free account at BrowserCat.

Second, create an API key and store it as a secret named BROWSERCAT_API_KEY in your Github repository. You can do this by navigating to your repository, clicking "Settings," then "Secrets," then "Actions," then "New repository secret."

Third, let's update playwright.config.ts to use BrowserCat:

const isCI = !!process.env.CI;
const useBC = !!process.env.BROWSERCAT_API_KEY;

export default defineConfig({
  timeout: 1000 * 60,
  workers: useBC ? 10 : isCI ? 1 : '50%',
  retries: useBC || isCI ? 2 : 0,
  maxFailures: useBC && !isCI ? 0 : 3,
  forbidOnly: isCI,

  use: {
    connectOptions: useBC ? {
      wsEndpoint: 'wss://api.browsercat.com/connect',
      headers: {'Api-Key': process.env.BROWSERCAT_API_KEY},
    },
  },
});

In the updates above, you'll notice we've increased parallelization to 10 workers when using BrowserCat. This is a good starting point, but you can increase it significantly more without issue.

Notice that this config connects to BrowserCat whenever BROWSERCAT_API_KEY is defined. This is useful for running tests locally, as well as in CI/CD.

OK, now here's the last bit of magic... Let's stop installing Playwright's browsers every time we run our pipeline:

      - name: Install deps
        run: npm install

If you run your pipeline now, you'll witness a dramatic speed-up, even with just a single test in your suite. And with 1000 free credits per month, you can run your pipeline for hours on end with plenty of time left over.

Next Steps...

You've got your CI/CD pipeline working. Now it's time to write some tests!

Be sure to check out my Ultimate Guide to Visual Testing in Playwright for advanced snapshot testing strategies and ready-made solutions for running visual tests in CI/CD.

In the meantime, happy testing!