Knowing up front whether a new feature will improve or worsen user experience can be a challenging decision. If we don't adopt the proper mechanisms and processes to test new features we stand a high risk of introducing new bugs. By incorporating A/B testing into our feature release workflow, we can minimize these situations by testing the new feature with a small sample of users before deploying it.
With the help of a cloud-hosted feature flag management service like ConfigCat, you can effortlessly streamline your A/B testing process. Within the dashboard, you can create and manage feature flags that can be toggled on and off before and after testing. As a demonstration, let's look at an A/B testing experiment.
Exploring an A/B test scenario
To understand the concept behind A/B testing, we can break it down into two major components. A common analogy is to refer to each component as a bucket. The first bucket is called the control bucket - it represents the state of the app without any new improvements or updates and can be considered as a benchmark when making later comparisons. The second bucket is called the variation bucket and, unlike the control bucket, it includes the new features or improvements that need to be tested.
An A/B test aims to compare how each bucket impacts a fixed percentage of users. Usually, the bucket with a positive impact is selected and rolled out.
Before conducting the test, we must first identify these buckets. For this, I've created a sample app to help us along the way. Here's a snapshot of it:
Did you notice the green badge over the pro pricing plan? Assume that the website state shown above represents the variation bucket. The previous state of the website had the same badge, but it was colored yellow instead of green. We can mark this state as the control bucket:
Now that we know the two states of the app (variation and control), we can test them on a small subset of users to see if the variation state performs better (influencing more purchase clicks on the pro plan).
Let's consider the metrics recorded for the control state as follows:
The current number of purchase clicks (yellow badge):
Pro plan purchases - 200 per day
The number of purchase clicks (green badge):
Pro plan purchases - number of purchase clicks to be determined by the A/B test
The importance of feature flags in A/B testing
To easily toggle between the two buckets (variation and control), we can link the green badge feature to a feature flag. When the feature flag is on, we'll display the green badge, and when it's off, we'll display the yellow badge. Furthermore, by creating a feature flag, we can specify the percentage of users to display the new feature to, which is crucial for testing. Here is a short overview of how to do this:
Adding a feature flag
1. Sign up for a free ConfigCat account
2. In the dashboard, create a feature flag with the following details:
Name: canShowGreenPricingBadge
Key: canshowgreenpricingbadge
Description: Toggles the green pricing badge.
*Make sure to turn on the feature flag in your environment.
3. Click the ADD FEATURE FLAG button to create and save the feature flag
4. To create a user segment to run the A/B test on, click the TARGET % OF USERS button and configure a fixed percentage (20%) as shown below:
When we turn on the feature flag in the following step, this setting will ensure that the variation state/bucket (green badge) is only available to 20% of our users.
5. Toggle the feature flag on, then integrate the feature flag into your Angular application
To track and collect the click events for the variation bucket, I'll be using a product analytics tool called Amplitude. In a later step, we'll look at how we can compare the difference in clicks we collected to the control bucket.
Setting up Amplitude
1. Sign up for a free Amplitude account
2. Switch to the Data section by clicking the dropdown at the top left
3. Click the Sources link under Connections in the left sidebar, then click the + Add Source button on the top right
4. Select the Browser SDK from the SDK sources list and enter the following details to create a source:
5. You should be automatically redirected to the implementation page. We'll get back to this soon, but first, let's add an event:
Adding an event
By creating an event, we are instructing Amplitude to specifically listen for that event and collect it. Later, in our code, we will trigger this event whenever the buy now button is clicked.
1. Click the Events link in the left sidebar under Tracking Plan to access the events page
2. Click the + Add Event button at the top right to create an event and fill in the following details:
In the above, we created an event with the name pro_plan_click to represent the type of event we plan to trigger and send later on.
A category was then added. The event was then activated by selecting the is Active option. Last but not least, you'll need to add the correct source to this event.
3. Click the Save changes button in the left sidebar
Integrating with Amplitude
Click the implementation link in the left sidebar to see the integration instructions page.
1. Install the amplitude CLI with the following command:
npm install -g @amplitude/ampli
2. Install the amplitude JavaScript SDK dependencies:
npm install amplitude-js
3. Run the following command to pull the SDK into Angular:
ampli pull
In this command, Amplitude will create the file src/ampli/index.js and download all the required settings from the previous steps into it. We will import and use this file in the next section.
Sending an event from Angular
Let's look at how we can trigger and send the event we created from Angular.
1. Import and initialize amplitude in the component:
import { ampli } from '../../ampli'
export class SiteMainComponent implements OnInit {
constructor() {
// ...
ampli.load({environment: 'production'})
}
// ...
}
NOTE: Because we imported a .js file above, you'll need to add "allowJs": true, to your compiler options in tsconfig.js
2. Create a method that triggers and sends an event when clicking the buy now button on the pro pricing plan:
export class SiteMainComponent implements OnInit {
handleProPlanClick() {
ampli.proPlanClick();
}
constructor() {
// ...
}
}
To verify that Angular is connected to your Amplitude account, let's check for logged requests.
Checking for logged requests
1. Under Connections in the left sidebar, click on Sources
2. Clicking on the buy now button on the pro plan will log the event to amplitude as shown in the Successful Requests graph on the left:
5. Click Save on the top right to save the chart
Analyze the test results
To compare and analyze the results between the two states, you can click the Compare to past dropdown at the top of the chart as shown below. Start by selecting the appropriate time interval.
For example, if the total amount of purchase clicks on the pro plan was 1200 for the past week, you can compare that to the current week to determine if the feature (in this case, the green badge) is making a positive impact.
To keep a log of when you enabled and disabled your feature flags during testing you can add it to your charts on amplitude.
Recap
Let's take a moment to recap what we did so far:
1. We discussed the concepts of the control and variation states (buckets) and identified them in the sample app
2. Created a feature flag
3. Set up an Amplitude account
4. Created a source
5. Added an event
6. Pulled the amplitude configs and integrated them into Angular
7. Sent and tracked a few successful requests
8. Created an analysis chart to record the click events
9. Finally, we looked at how we can compare the variation state to the control state in Amplitude's analysis chart by using the Compare to past option
Final takeaways
A/B testing is a great method for measuring the impact of a new feature before deploying it to your entire user base. With the help of ConfigCat's feature flag services you can streamline this process better with the added ability to target specific user segments. Using their 10-minute trainable feature flag management interface, you can toggle experimental features on or off quickly without redeploying your app.
For more awesome posts like these and other announcements, follow ConfigCat on Twitter, Facebook, LinkedIn, and GitHub.
Top comments (0)