DEV Community

A/B testing: introduction

Elena on January 05, 2018

This article has been first published on my website smartpuffin.com. It is based on my (first ever!) conference talk about A/B testing - check out ...

Read full post

Rafal Pienkowski • Mar 25 '18

Thanks for a great article. It is a nice introduction to the A/B testing.

Can I ask you, based on your experience, aren't those tests annoying for end users? Let's the example with the color of the button from the article. If we will performing experiments over and over again, this could discourage potential clients in our shop. Are there any guidelines how many tests could we perform before we will start losing potential clients?

To be clear, I'm not against A/B testing, but I don't want to overdo at the beginning of my journey with A/B testing 😁

Elena • Mar 28 '18

Hi Rafal, that's a great question!
I probably need to write an article about "when not to run a/b tests". But in short, for your question: I wouldn't run a/b tests if I don't have enough customers (=enough statistical power). Also I would run a/b tests in a different manner depending on if my customers visit regularly or once in a while. Regular customers might not like constant back and forth changes; and if their work depends on your tool, then especially so.

Some more notes. One customer always sees one variant of the test when they come to the website, during the test runtime. I.e., I come first time, I am assigned the red button, and I will always see it until the test stops. It doesn't change every time for me.
If you change your website without experimenting, it's still a change, and the customers will see it, and possibly will be annoyed as well:).
If they don't come to the website very often, they won't be annoyed by constant changes (through a test or without).