Hi Rafal, that's a great question!
I probably need to write an article about "when not to run a/b tests". But in short, for your question: I wouldn't run a/b tests if I don't have enough customers (=enough statistical power). Also I would run a/b tests in a different manner depending on if my customers visit regularly or once in a while. Regular customers might not like constant back and forth changes; and if their work depends on your tool, then especially so.
Some more notes. One customer always sees one variant of the test when they come to the website, during the test runtime. I.e., I come first time, I am assigned the red button, and I will always see it until the test stops. It doesn't change every time for me.
If you change your website without experimenting, it's still a change, and the customers will see it, and possibly will be annoyed as well:).
If they don't come to the website very often, they won't be annoyed by constant changes (through a test or without).
I'm focused on developing and expanding my knowledge and skills. Enjoying new challenges. I'm assuming that there are no stupid questions, there are only silly answers.
I'm focused on developing and expanding my knowledge and skills. Enjoying new challenges. I'm assuming that there are no stupid questions, there are only silly answers.
Hi Rafal, that's a great question!
I probably need to write an article about "when not to run a/b tests". But in short, for your question: I wouldn't run a/b tests if I don't have enough customers (=enough statistical power). Also I would run a/b tests in a different manner depending on if my customers visit regularly or once in a while. Regular customers might not like constant back and forth changes; and if their work depends on your tool, then especially so.
Some more notes. One customer always sees one variant of the test when they come to the website, during the test runtime. I.e., I come first time, I am assigned the red button, and I will always see it until the test stops. It doesn't change every time for me.
If you change your website without experimenting, it's still a change, and the customers will see it, and possibly will be annoyed as well:).
If they don't come to the website very often, they won't be annoyed by constant changes (through a test or without).
Thanks for your answer. You can be sure that if you write "when not to run a/b tests" post, I'll be one of the first readers of it :)
Cheers.
And now I proudly present to you... "When Not To Run A/B Tests" post! :)
dev.to/ice_lenor/when-not-to-ab-te...
Thanks. I’m going to read it as soon as possible :)