We use cookies to set the variant within the load balancer. The instances run within the same server so the only latency difference should be a result of changes to the code. Which is exactly what you want to test.
Also, how are cookies different than your approach?
I said users lifetime (across sessions). Cookies are session based/volatile, if the user uses other browser or cleans its cookies they will see different version, which result in the previous issues I mentioned.
I did not presented my approach because I don't know all the details, but all the tests we did in gaming were using a database for persistence, we had the luxury of having all our visitors authenticated users so we know who is in what test.
Depending on what is tested persistence is required or not. Usually our features require a few weeks of measuring its impact on the user behavior, without authentication this cannot be done properly.
Nevertheless, this would also be possible using load balancing, although it might require some customization.
Anyway, I use it for webshops, targeting new users. Most e-commerce websites do not have the luxery to use authentication before any actual conversion. Then this approach will work perfectly.
When testing an action like Conversion sounds great, you do not even care about user as a lifetime, but rather to which version reacted better/got converted.
Beside Round Robin, the LB/Proxy can also be used to make the cohorts, for example based on country, or limit an entire Test based on a property (example country, region, language, device, mobile vs web).
For further actions, you may consider blocking this person and/or reporting abuse
We're a place where coders share, stay up-to-date and grow their careers.
We use cookies to set the variant within the load balancer. The instances run within the same server so the only latency difference should be a result of changes to the code. Which is exactly what you want to test.
Also, how are cookies different than your approach?
I said users lifetime (across sessions). Cookies are session based/volatile, if the user uses other browser or cleans its cookies they will see different version, which result in the previous issues I mentioned.
I did not presented my approach because I don't know all the details, but all the tests we did in gaming were using a database for persistence, we had the luxury of having all our visitors authenticated users so we know who is in what test.
Depending on what is tested persistence is required or not. Usually our features require a few weeks of measuring its impact on the user behavior, without authentication this cannot be done properly.
Nevertheless, this would also be possible using load balancing, although it might require some customization.
Anyway, I use it for webshops, targeting new users. Most e-commerce websites do not have the luxery to use authentication before any actual conversion. Then this approach will work perfectly.
Good article though.
When testing an action like Conversion sounds great, you do not even care about user as a lifetime, but rather to which version reacted better/got converted.
Beside Round Robin, the LB/Proxy can also be used to make the cohorts, for example based on country, or limit an entire Test based on a property (example country, region, language, device, mobile vs web).