DEV Community

Cover image for Show dev.to: an A/B test checker
Pierre
Pierre

Posted on

Show dev.to: an A/B test checker

Hi everyone,

Too many times I saw people on Facebook happy that their conversion increased from 1.1% to 2% with less than 500 visits, the truth is, there is no way to be sure that their conversion actually changed with only 500 visits and such a low conversion rate.

This is because of something called the "p-value". In short, the "p-value" is a Mathematical value that measures how statistically significant is a hypothesis.

In our case, the hypothesis could be: A is performing better than B in our A/B test or, this ad on Facebook is converting more.

This is, of course, an oversimplification, but in short, the p-value can tell you if you can draw a conclusion with your results or if you should wait to have a bigger sample.

I've made this small tool to help you draw a conclusion from your A/B tests that I hope you will like and find interesting.

Alt Text

This thing took me quite a bit of time to code, so if you think twitter would like it, do not hesitate to share it, it helps a lot πŸ™πŸ˜‰

(but do NOT create an account just for this though)

If you want to read more about it you can this excellent post on FCC.com.

Do not hesitate to follow me if you don't want to miss my next posts. I write about tech, my bootstrapping journey and I occasionally write more data analysis articles.

Top comments (8)

Collapse
 
mt3o profile image
mt3o

You should have explained what does it mean to have p=0.76 (and the meaning of % of confidence) because it's not obvious if you don't have degree related to statistics.
Often when people write such article wanting to explain something, they take some level of knowledge for granted and most often it is not true for most people. If you really want to explain something, you can use the "explain to me like I'm ten" approach. Or give the text to your mother and ask her what she understands (unless she has degree in math). You get the idea...

Collapse
 
daolf profile image
Pierre

Yes you are correct.

I’ve read it was called "curse of knowledge".

I thought it was clear on the website page but maybe it was not then :)

Collapse
 
mt3o profile image
mt3o

Never assume that other party has the knowledge :) regardless where you are, you will meet people who don't have your background, and even more often you will have to work with people of different background. Imagine that in corporate life you will be working under and reporting to a salesperson. If you can explain what you want to do in simple words, you win :)
I have the (un)luck of being a CS drop out, with degree in marketing, so I get what you are writing about, but often you will meet people who have no idea about the p value. In fact I saw a supposed-to-be-cool presentation destroyed by a lecture on p value. Ideas from the marketing team were cool and indeed they were a success but they couldn't measure it properly.

Collapse
 
mt3o profile image
mt3o

Besides, scientific books often share the same issues - lots of symbols which people don't understand and webdev don't know how to read - it makes the whole book useless. I prefer to read English books published by non-english authors, because they less often overintellectualize their writing. ;-)

Collapse
 
ben profile image
Ben Halpern

Neat

Collapse
 
daolf profile image
Pierre

Thanks Ben!

Collapse
 
abdisalan_js profile image
Abdisalan

Love the post, I want to try this with real data!

I don't think it works on firefox though, but it does on chrome.

Collapse
 
daolf profile image
Pierre

Oh thanks for pointing this out, I'll need to fix this once I'm home.