Hi everyone,
Too many times I saw people on Facebook happy that their conversion increased from 1.1% to 2% with less than 500 visits, the truth is, there is no way to be sure that their conversion actually changed with only 500 visits and such a low conversion rate.
This is because of something called the "p-value". In short, the "p-value" is a Mathematical value that measures how statistically significant is a hypothesis.
In our case, the hypothesis could be: A is performing better than B in our A/B test or, this ad on Facebook is converting more.
This is, of course, an oversimplification, but in short, the p-value can tell you if you can draw a conclusion with your results or if you should wait to have a bigger sample.
I've made this small tool to help you draw a conclusion from your A/B tests that I hope you will like and find interesting.
This thing took me quite a bit of time to code, so if you think twitter would like it, do not hesitate to share it, it helps a lot 🙏😉
Pierre de Wulf@pierredewulf
People tend to draw conclusion from their A/B tests when actually they should not📈.
You often can't be sure if B is better than A without computing the p-value 🤓
To make this process as easy as possible I've made this 👇
abtestchecker.com17:11 PM - 29 Feb 2020
(but do NOT create an account just for this though)
If you want to read more about it you can this excellent post on FCC.com.
Do not hesitate to follow me if you don't want to miss my next posts. I write about tech, my bootstrapping journey and I occasionally write more data analysis articles.
Top comments (8)
You should have explained what does it mean to have p=0.76 (and the meaning of % of confidence) because it's not obvious if you don't have degree related to statistics.
Often when people write such article wanting to explain something, they take some level of knowledge for granted and most often it is not true for most people. If you really want to explain something, you can use the "explain to me like I'm ten" approach. Or give the text to your mother and ask her what she understands (unless she has degree in math). You get the idea...
Yes you are correct.
I’ve read it was called "curse of knowledge".
I thought it was clear on the website page but maybe it was not then :)
Never assume that other party has the knowledge :) regardless where you are, you will meet people who don't have your background, and even more often you will have to work with people of different background. Imagine that in corporate life you will be working under and reporting to a salesperson. If you can explain what you want to do in simple words, you win :)
I have the (un)luck of being a CS drop out, with degree in marketing, so I get what you are writing about, but often you will meet people who have no idea about the p value. In fact I saw a supposed-to-be-cool presentation destroyed by a lecture on p value. Ideas from the marketing team were cool and indeed they were a success but they couldn't measure it properly.
Besides, scientific books often share the same issues - lots of symbols which people don't understand and webdev don't know how to read - it makes the whole book useless. I prefer to read English books published by non-english authors, because they less often overintellectualize their writing. ;-)
Neat
Thanks Ben!
Love the post, I want to try this with real data!
I don't think it works on firefox though, but it does on chrome.
Oh thanks for pointing this out, I'll need to fix this once I'm home.