A/B Testing with the new Next.js 12 Middleware

Vercel recently released Next.js 12 which adds a number of exciting performance improvements as well as a new beta feature - Middleware. Middleware has many uses, but I'm going to focus in this post on A/B Testing.

You've always been able to run A/B tests on Next.js applications, but until this latest release there have been some major strings attached. For example, on static pages, there would often be a "flash" where users would see the original page for a split second before your variation popped in and replaced it. And on server-rendered pages, you would need to completely disable caching since two users on the same URL could get two different HTML responses.

Next.js middleware fixes these issues in an elegant way. You can create two different versions of a page and using a single URL, route traffic between them with a middleware function. The middleware is run at the edge, so it's globally distributed and super fast for your users.

Setting up the Next.js App

We'll start with a standard Typescript Next.js app:

npx create-next-app@latest --ts
cd my-app
npm run dev

Now you should be able to visit http://localhost:3000 and see a homepage.

Let's create a new version of the homepage at pages/new_homepage.tsx:

export default function NewHomePage() {
  return <h1>Welcome to my new site!</h1>
}

Now you should have two working URLs:

1. The original homepage - http://localhost:3000
2. The new (way better) homepage - http://localhost:3000/new_homepage

Our goal is instead of having these on two separate URLs, we want 50% of the visitors to the root URL (/) to see the original page and the other 50% to see the new one.

That sounds like an A/B test! To help do the traffic splitting, we're going to use GrowthBook, an open source feature flagging platform with really robust A/B testing support.

Setting up GrowthBook

You can self-host GrowthBook (https://github.com/growthbook/growthbook) or create a free Cloud account (https://app.growthbook.io/). Either way, once you login, there are a couple steps required before you can run an experiment.

First, click on Step 1: Install our SDK and run the npm install command:

npm i --save @growthbook/growthbook

Note: Next.js middleware runs outside of a React context, so we're using the vanilla Javascript SDK above instead of the React one.

Creating the Next.js Middleware

Now, we'll integrate the sample code in GrowthBook into our Next.js App. Create a file pages/_middleware.ts with the following contents (make sure to swap out the placeholder with the unique API endpoint you see in GrowthBook):

import { NextRequest, NextResponse } from 'next/server'
import { GrowthBook } from '@growthbook/growthbook'

const FEATURES_ENDPOINT = 'YOUR_GROWTHBOOK_ENDPOINT_HERE'

// Fetch features from GrowthBook API and cache in memory
let features = null;
let lastFetch = 0;
async function getFeatures() {
  if (Date.now() - lastFetch > 5000) {
    lastFetch = Date.now();
    const latest = fetch(FEATURES_ENDPOINT)
      .then(res => res.json())
      .then(json => features = json.features || features)
      .catch((e) => console.error("Error fetching features", e))
    // If this is the first time, wait for the initial fetch
    if(!features) await latest;
  }
  return features || {};
}

const COOKIE = 'visitor_id'

export async function middleware(req: NextRequest) {
  // We only want to run the A/B test on the homepage
  const pathname = req.nextUrl.pathname;
  if (pathname !== "/") {
    return NextResponse.next()
  }

  // Get existing visitor cookie or create a new one
  let visitor_id = req.cookies[COOKIE] || crypto.randomUUID()

  // Create a GrowthBook client instance
  const growthbook = new GrowthBook({
    attributes: { id: visitor_id },
    features: await getFeatures(),
    trackingCallback: (exp, res) => {
      console.log("In Experiment", exp.key, res.variationId);
    }
  });

  // Pick which page to render depending on a feature flag
  let res = NextResponse.next();
  if (growthbook.feature("new-homepage").on) {
    const url = req.nextUrl.clone();
    url.pathname = "/new_homepage";
    res = NextResponse.rewrite(url);
  }

  // Store the visitor cookie if not already there
  if (!req.cookies[COOKIE]) {
    res.cookie(COOKIE, visitor_id)
  }

  return res
}

There's a lot going on here, but it's not too hard to follow:

1. Function to fetch feature definitions from the GrowthBook API, cache them, and keep it up-to-date
2. Skip the middleware if the user is requesting any page other than /
3. Look for an existing visitor id stored in a cookie and generate one if it doesn't exist yet.
4. Create a GrowthBook client instance
5. Determine which page to render based on a GrowthBook feature flag
6. Set the visitor id cookie on the response if needed
7. Return the response

Creating the Feature Flag

At this point, if you visit http://localhost:3000 you'll always see the original homepage still.

This is because the code is looking for a feature flag named new-homepage, which doesn't exist yet. Flags that don't exist yet are always treated as if they are off so the middleware just returns the original page. Let's fix this.

Back in GrowthBook, close the SDK instructions modal and click on Step 2: Add your first feature. Enter the feature key new-homepage. Keep the feature type set to on/off, choose "A/B Experiment" as the behavior, and leave everything else set to the default (split users by id, even 50/50 split, "new-homepage" as the tracking key).

Click save, wait a few seconds, and refresh your NextJS app.

Depending on the random visitor_id cookie that the middleware generated, you may see either version of the homepage. You can delete that cookie and refresh a few times. You'll notice about half the time you get the new page and the other half you don't.

Also, if you look in the terminal where you're running the Next.js npm run dev command, you should see the log messages from trackingCallback.

Analyzing Results

Just running an A/B test by itself is fun, but not that useful. You also need to track and analyze the results.

In the trackingCallback in pages/_middleware.ts, instead of doing a console.log, we could use Mixpanel or Segment or another event tracking system.

Then, in the app, we could similarly track events when the users do something we care about, like sign up or buy something.

Once you do that, GrowthBook can connect to your event tracking system, query the raw data, run it through a stats engine, and show you the results. This process is a little more involved to set up, but I will walk through it in a followup post.

The Future

Hopefully in future releases, Next.js expands on their middleware feature to make A/B testing even more powerful. Imagine, for example, that middleware could inject props into your pages, similar to getServerSideProps. Then you wouldn't need to create new temporary pages every time you wanted to run an A/B test!