Common Pitfalls in Setting up A/B Tests

> I’ve heard “Darth Vader experiment” also called “Donald Trump experiment”, the idea is the same, you show a controversial picture which drives away some part of the population, so then you get less customers, but the experiment looks good.

\
Interesting, I didn’t know about the name!

\
> What are your thoughts on experiment overlap? 

That’s a good question, thanks. Different companies tackle it differently - I saw two ways 

* to automatically detect if any overlapping experiments running and   suggest experimenters to agree on priorities
* to split users to un-orthogonal sets so called ‘universes’, so several experiments can take place on the same part of the website. That keeps both experiments unbiased since the number of users exposed to the third change is equal between test and control of the experiments.

I hope it helps!

\


Kristina Fedorenko