...A Practical Guide Using Playwright and GitHub Actions
๐๏ธ
Introduction
Visual testing is a crucial aspect of UI development that often goes underutilized. In this post, we'll explore how to set up an efficient visual testing workflow using Playwright and GitHub Actions. This approach is particularly beneficial for small to medium-sized teams looking to enhance their code review process and confidence without adding significant overhead or extra costs.
Why Visual Tests Matter
Uncover Hidden Issues:
Visual tests can reveal problems you weren't actively looking for (while also reducing the need for complex assertion code).
Ensure UI Integrity:
They're the most effective way to verify the visual aspects of your UI.
Improve Collaboration:
Visual tests serve as a great tool for communication between technical and non-technical stakeholders.
The Core Concept: Non-Assertive Visual Testing
Before we dive into the technical details, it's crucial to understand the core concept of our approach:
We're "piggybacking" Playwright tests to collect screenshots, but instead of automated visual assertions, we use the screenshots as an optional tool during the code review process.
Here's why this approach is powerful:
Stability: By not making automated assertions based on screenshots, we avoid introducing potential flakiness into our test suite. Our tests remain fast and reliable.
Flexibility: During code review, we can choose which screenshot comparisons are relevant to the changes being reviewed. We're not bound by rigid assertions.
Context-Aware Reviews: Screenshots provide visual context that might not be apparent from code changes alone, enhancing the effectiveness of code reviews.
This approach strikes a balance between the benefits of visual testing and the need for a smooth, efficient development process. It keeps our CI pipeline stable while still providing valuable visual information during code reviews.
The Solution: Playwright + GitHub Actions
We'll set up a workflow that integrates seamlessly with your existing development process:
- E2E tests run on every PR using Playwright.
- Tests capture screenshots only when the PR is ready for review. The screenshots are taken, not asserted on, and don't fail the tests.
- Screenshots that are now different are compared using ImageMagick to make sure there's an actual visual diff. Otherwise, they are reverted to avoid clutter.
- GitHub Actions commit these screenshots to the PR.
- Code reviewers can see image diffs directly in GitHub's file comparison view, using them as an optional aid in their review.
- Merged PRs set the new baseline for future comparisons.
Bonus Skills You'll Learn:
How to: optimize CI pipelines by creating custom Docker images with preinstalled programs, manipulate PRs through GitHub Actions, and write powerful scripts for CI environments. You'll also gain experience with ImageMagick's image comparison capabilities, enabling programmatic image analysis. These skills will enhance your ability to create efficient, automated workflows and improve your overall DevOps practices.
Let's dive into the implementation!
Step 1: Setting Up Playwright Tests
First, let's configure our Playwright tests to capture screenshots. We'll create a utility function that takes screenshots only for non-draft PRs.
Create a file named test-utils.ts
in your test directory:
const skipScreenshots = process.env.DRAFT === 'true';
export const screenshot = async (target: Page | Locator, name: string) => {
if (skipScreenshots) return;
await target.screenshot({
path: `e2e/screenshots/${name}.png`,
animations: 'disabled',
});
};
Now, you can use this screenshot
function in your tests to capture screenshots without making assertions:
import { test } from '@playwright/test';
import { screenshot } from './test-utils';
test('homepage visual test', async ({ page }) => {
await page.goto('https://your-app-url.com');
await screenshot(page, 'homepage');
});
Step 2: Creating a Custom Docker Image
To avoid reinstalling Playwright and ImageMagick on every CI run, we'll create a custom Docker image. This helps speed up the GitHub Actions workflow by ensuring all necessary tools are pre-installed.
Create a file named Dockerfile
in your project root:
# Use the official Playwright Docker image as a base
FROM mcr.microsoft.com/playwright:v1.43.0-jammy
# Install ImageMagick
RUN apt-get update && apt-get install -y imagemagick
# Clean up
RUN apt-get clean && rm -rf /var/lib/apt/lists/*
# Display installed versions
RUN playwright --version && identify -version
Now, build and push this Docker image to a registry. If you're using GitHub Container Registry, you can use these commands:
# Build the image
docker build -t ghcr.io/your-username/playwright-imagemagick:latest .
# Log in to GitHub Container Registry
echo $GITHUB_TOKEN | docker login ghcr.io -u your-username --password-stdin
# Push the image
docker push ghcr.io/your-username/playwright-imagemagick:latest
Make sure to replace your-username
with your actual GitHub username and set the GITHUB_TOKEN
environment variable to a personal access token with the necessary permissions.
Step 3: Configuring GitHub Actions
Next, we'll update our GitHub Actions workflow to use our custom Docker image. Create/update a file named .github/workflows/e2e-tests.yml
:
name: E2E Tests
on:
pull_request:
types: [opened, reopened, synchronize, ready_for_review]
jobs:
e2e:
name: 'E2E Tests'
timeout-minutes: 60
runs-on: ubuntu-latest
container:
image: ghcr.io/your-username/playwright-imagemagick:latest
credentials:
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 2
- uses: actions/setup-node@v4
with:
node-version: 20
- name: Install dependencies
run: npm ci
- name: Run Playwright tests
run: npm run test:e2e
env:
# Skip screenshots for draft PRs or non-main branch PRs
DRAFT: ${{ github.event.pull_request.draft || github.base_ref != 'main' }}
- uses: actions/upload-artifact@v4
if: ${{ !cancelled() }}
with:
name: screenshots
path: e2e/screenshots
retention-days: 3
commit-screenshots:
name: 'Commit Screenshots'
needs: e2e
runs-on: ubuntu-latest
if: github.event.pull_request.draft == false && github.base_ref == 'main'
permissions:
contents: write
container:
image: ghcr.io/your-username/playwright-imagemagick:latest
credentials:
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 2
- uses: actions/download-artifact@master
with:
name: screenshots
path: e2e/screenshots
- name: Compare Modified Screenshots
run: |
chmod +x $GITHUB_WORKSPACE/scripts/compare_screenshots.sh
$GITHUB_WORKSPACE/scripts/compare_screenshots.sh
- name: Commit changes
uses: stefanzweifel/git-auto-commit-action@v5
with:
commit_message: '[CI] Update Screenshots'
commit_user_name: 'GitHub Actions'
commit_user_email: 'actions@github.com'
- uses: geekyeggo/delete-artifact@v5
with:
name: screenshots
Step 4: Image Comparison Script
Create a file named scripts/compare_screenshots.sh
:
#!/bin/bash
set -e
compare_images() {
local img1="$1"
local img2="$2"
local diff_img="$3"
local diff_pixels
diff_pixels=$(compare -metric AE "$img1" "$img2" "$diff_img" 2>&1 >/dev/null)
echo "diff pixels for $img1, $img2: $diff_pixels"
if [ "$diff_pixels" -lt 11 ]; then
return 0
else
return 1
fi
}
SCREENSHOT_DIR="${GITHUB_WORKSPACE}/e2e/screenshots"
modified_files=$(git diff --name-only | grep '.png$' || true)
if [[ -z "$modified_files" ]]; then
echo "No modified PNG files found in $SCREENSHOT_DIR."
exit 0
fi
for screenshot in $modified_files; do
current_screenshot="${GITHUB_WORKSPACE}/${screenshot}"
git show HEAD~1:"$screenshot" > original_screenshot.png
baseline_screenshot="original_screenshot.png"
if [[ ! -s "$baseline_screenshot" ]]; then
echo "No baseline image for $screenshot. Assuming this is a new screenshot."
rm -f "$baseline_screenshot"
continue
fi
diff_file="${current_screenshot}.diff.png"
set +e
compare_images "$baseline_screenshot" "$current_screenshot" "$diff_file"
comparison_result=$?
set -e
if [ $comparison_result -eq 0 ]; then
echo "No visual difference for $screenshot. Reverting changes."
git restore "$screenshot"
else
echo "Visual difference detected for $screenshot. Keeping changes."
fi
rm -f "$diff_file" "$baseline_screenshot"
done
This script compares the new screenshots with the previous versions and reverts changes if the difference is minimal (less than 11 pixels).
Step 5: Setting Up Secrets
To allow the GitHub Action to commit changes, you need to set up a personal access token:
- Go to your GitHub account settings.
- Navigate to "Developer settings" > "Personal access tokens".
- Generate a new token with
repo
scope. - In your repository, go to "Settings" > "Secrets and variables" > "Actions".
- Add a new repository secret named
GITHUB_TOKEN
with the value of your personal access token.
Step 6: Bonus - Enhancing Image Comparison in GitHub
To make comparing screenshots even easier, Iโm working on an open source browser extension that improves GitHubโs built-in image diff tool. This extension will highlight visual differences more clearly, making it easier for reviewers to assess changes.
Stay tuned for the release!
Advantages of This Approach
- Integrated Workflow: Utilizes existing tools and processes.
- Enhanced Code Reviews: Visual diffs are part of the PR review process, providing additional context without enforcing rigid rules.
- No Additional Costs: Leverages GitHub Actions and your existing CI pipeline.
- Flexible: Can be easily adapted to different project sizes and needs.
Limitations and Considerations
- Repository Size: Frequent screenshots can increase repository size over time.
- Review Time: Large numbers of screenshots may slow down the review process.
- Limited Comparison Tools: GitHub's built-in image diff tool is basic.
- Manual Baseline Maintenance: It's up to you, the reviewer, to go through the diffs and check them with your own eyes - with no AI or smart algorithms to save you time on recurring diffs.
- Pulling Changes: After the CI commits screenshots, developers need to pull the changes to sync their local repository.
Conclusion
This setup provides a practical approach to incorporating visual testing into your development workflow, without the overhead of maintaining visual test assertions. By using screenshots as an optional review aid rather than a pass/fail criterion, we gain the benefits of visual testing while maintaining a smooth, efficient development process.
Remember, the key is to use these visual diffs as a tool to enhance your code reviews, not as a replacement for thorough code analysis. This approach allows you to catch visual regressions early, while still relying on human judgment to determine the significance of any visual changes.
As your team and project grow, you might consider more advanced solutions like Applitools, which offer powerful image comparison algorithms and cross-browser testing capabilities.
Happy testing, and please feel free to ask any question!
I encourage you to implement this in your workflow and share your experiences in the comments.
Top comments (0)