Don't forget basic statistics here. If you're developing an A/B test for a case that will be used millions of times per day then a hundred tests is not going to be conclusive. You need to test a statistically significant number of times relative to your actual use case.
Yep. This is a conversation I've definitely had numerous times.
Yes ofc. Beside the statistically relevance there are other factors like user aquisition, cohorts have to be very similar and there is also an error margin of at least 5% which should be taken into consideration when comparing the stats.
That is part of the business part of doing tests, I tried to cover only the technical details.
The web is full of articles on how to do proper testing, but it was lacking on the implementation details, so I wrote this story.
We're a place where coders share, stay up-to-date and grow their careers.
We strive for transparency and don't collect excess data.