How I made AB-testing work for our Server rendered React app and GTM+Optimize

Originally published on Medium 2019-10-31

So you want to be data driven? So you want to use React, GTM, Google Optimize, Server rendering and a bunch of other buzzwords currently representing the current trend of web development? So did I. And it turned out to be harder than I thought.

Edit 2020–02–04: This solution limits you to one test at the same time. We are working on a solution.

Edit 2020–09–07: I previously stated that this method only worked for one test at a time. This was due to an error in our setExperimentCookie function. We checked if there was a cookie already, and didn't set a new one if there were. When that check was removed everything worked fine. So as long as you set your experiment cookie in componentDidMount in the Experiment, it should be fine.

Edit 2020–09–07: The summary section has been rewritten to reflect our results.

But first, a short disclaimer. As with most of the articles about development you find when googling your current conundrum, this represents my solution to my specific problem with my specific tech stack. I hope you will find something to take away from this that you can apply to your problem, but most likely this will fall on the pile of articles touching near your problem, but not quite solving it. If that's the case, I wish you luck on your hunt for solutions!

Now then, what problem did I have?

The problem

We are building and managing a web site for a non profit organisation, and have recently taken over the client from a smaller team. As we like to work data driven, making decisions based on more than our hunches, we wanted to get going with AB-testing.

Symptoms

So we booted up Google Optimize and made a simple test to see if it would work. It would not. Our control variant would flicker before showing the test variant, or our variant would flicker before showing our control. And only our control variant would be shown if we navigated within the site.

Cause

The problem lay in Server rendering and our Single Page App (SPA) built in React. Our server would serve the control variant, not having any clue about any Google Optimize scripts served by Google Tag Manager (GTM) running on the site. Then either Optimize would replace the test content with the variant, and then React would hydrate the server render (thus switching back to the control), or React would hydrate and then Optimize would show the variant. Either way, the user saw flickering of different variants.

Our solution

Our solution was to build a simple custom AB-testing framework that could select variant on the server based on your session, and then adding all our test variants manually in our code instead of using Optimize's interface, and lastly sending the experiment data to Google Analytics (GA), which then Optimize will pick up and show statistics for winning variant etc.

The process of setting up an experiment is now:

1. Create an experiment in Optimize and as many variants as you want to test, along with the goal you want to test for
2. Code the different variants in the code, using the ID from Optimize
3. Send experiment data via a cookie to GTM, which serves it to GA and Optimize
4. Analyse the data in Optimize and GA

With that brief explanation, let's get into the nitty gritty.

The nitty gritty (or, actual code and stuff)

Parts of our solution are inspired by https://github.com/pushtell/react-ab-test, but we found that to be a bit too complex for our liking. And I like to know what my code does.

Selecting the variant on the server

Or actually, generating a random value between 0 and 1 for later use in variant selection.

// Set a random number between 0 and 1 based on session id
const randomFromSession = randomFromSeed(req.sessionID);

// Set it when creating our Redux store
const defaultStoreState = { ab: { cohortId: randomFromSession } };
const store = createStore(
  reducers,
  defaultStoreState,
  applyMiddleware(thunk)
);

// Send back a cookie with the cohort ID to the client in the response
res.cookie('cohortId', `${randomFromSession}`, {
  maxAge: 86400000,
  httpOnly: false
});

We're using a function to generate the random numbers, involving some maths. I wish I could give credit here, but something like it was found in an answer somewhere on stack overflow.

We save the cohort ID in our redux-store as well, to be able to get it when server rendering the experiment. Lastly we send it in a cookie back to the client.

Note that this is not an article on server rendering, I leave it to you to adapt this into your server rendering code.

The experiment code

import { Experiment, Variant } from './components/ABTesting';
import { experiments } from './config/abExperiments';

class ComponentWithExperiment extends React.Component {
  render() {
    <Experiment experimentId={experiments.testExperiment}>
      <Variant>Variant 1</Variant>
      <Variant>Variant 2</Variant>
    </Experiment>
  }
}

Remember the first step from before? There we got an experiment ID from Optimize, which we’ve here put in a config-file (exporting a simple JS object).

We send the ID as a prop to our Experiment component, and nest our variants in Variant , just for clarity. The Variant component is just rendering its children, it is only there for code readability.

Rendering the experiment

Selecting which variant is getting used is, as mentioned before, set on the server and sent to the client as a cookie. On the first server render, we read the value from the redux store. On the client side we first check the store for the value, and if it's not there we check the cookie and put it into the store for easier retrieval later.

For brevity I'm not going to include the whole file, but you can read through it all if you want. I'm going to go through parts of it here.

The constructor

  constructor(props) {
    super(props);

    // Set cohort on server and client
    let cohortId = 0;
    let cohortSet = false;
    if (
      props.cohortId !== undefined &&
      props.cohortId !== null &&
      typeof props.cohortId === 'number' &&
      Number.isFinite(props.cohortId) &&
      props.cohortId >= 0 &&
      props.cohortId < 1
    ) {
      cohortId = props.cohortId;
      cohortSet = true;
    }

    this.state = {
      cohortId,
      cohortSet
    };
  }

The constructor will be called on both the server and client, so here we check if the cohort ID is set in the redux store (and is a number between 0 and 1), and add it to the state of the component.

The componentDidMount

  componentDidMount() {
    const variantChildren = React.Children.toArray(this.props.children);
    const numberOfVariants = variantChildren.length;
    const { experimentId } = this.props;

    // Set cohort on client
    if (!this.state.cohortSet) {
      const cohortIdFromCookie = cookie.get('cohortId');
      const cohortId = cohortIdFromCookie === null ? 0 : cohortIdFromCookie;

      this.setState({ cohortId: cohortId, cohortSet: true });
      this.props.abActions.updateCohortId({ cohortId });

      this.setExperimentCookie(
        experimentId,
        this.getVariantId(cohortId, numberOfVariants)
      );
    } else {
      this.setExperimentCookie(
        experimentId,
        this.getVariantId(this.props.cohortId, numberOfVariants)
      );
    }
  }

Now, here comes a tricky part. When server rendering, we don’t have a DOM, so the components never get mounted. This means that componentDidMount only will be called on the client.

So if we’re on the client, and the cohort has not been added to the redux store yet, we read it from the cookie we got from the server, and save it in the state as well as the store.

Here we also save another cookie, containing information about both the experiment and the variant chosen on a format that Google likes (experiment.variant). This cookie will be read by GTM and pushed to GA, more on that later.

The getVariantId is just a function returning a 0-indexed number based on the number of variants and the cohort-id. You can check it out in the full file linked above.

Based on the cohort id we select one of the React children and render that and only that.

Integrating with GTM, GA, and Optimize

This is not my area of expertise, we have a partner firm that does all the setup for this. But in the process of building this, I have picked up quite a few things, and will do my best to describe them.

Page views

Since our site is a SPA, only one regular page view is being registered normally; the initial request. All subsequent navigations have to be triggered manually, either by GTM or in the code.

We have a tag in GTM for sending a page view to Google, this fires on either a normal page view trigger (initial request), or a history change trigger (all subsequent navigations). In this tag, we send a gaProperty variable along.

Experiment data

Information about what experiment and variant is displayed to a user is read from the cookie mentioned earlier and saved to two custom JavaScript variables. The custom JavaScripts read the cookie, split the value on the period, and save the correct part to itself.

These two custom variables are then added to the gaProperty variable as expId and expVar under Fields to Set in the variable settings. Sending these along with the pageview makes Google Analytics aware of any experiments running for a user in a particular session. You can then track your variant’s progress in Optimize.

In conclusion

This was our solution, I hope it can help you as well. We’re now well on our way to add AB testing to our data driven workflow. Since this article has been written we have run a few AB tests with smashing results, so this method seems to work splendidly.

Cover image by: Caleb Jones
Thanks to: Robert Källgren and Regnskog