A/B testing is one of the primary ways of verifying ideas in mobile apps on a fast scale. The arrangement is typically straightforward: you run a test for a few weeks, gauge its impact, and decide to roll out a feature or iterate further.
One of the essential but commonly overlooked aspects of A/B testing is setting the correct exposure points. In this article, we'll cover why correct exposure point setting is important, common pitfalls to avoid, and provide some tips and tricks I've learned along the way.
What are exposure points, anyway?
An exposure point is the real moment when the user initially encounters or interacts with the feature that you're testing. For example, timing when a user sees a new button, or when they see a redesigned landing page after having clicked on something.
Why is it important to choose a good exposure point?
If you've ever stared at A/B test results and they simply didn't make sense, bad exposure points might be the culprit. Bad exposure points can lead to:
- Confusing Data: It won't be clear whether your feature is a hit or miss.
- Hidden Bugs: Everything might appear okay on the surface, but serious issues like app crashes might be slipping under the radar.
- Missed Opportunities: Your data might falsely show no significant impact, even though the users indeed enjoyed the feature.
Example of a bad exposure point
Imagine you're testing an upsell for a new subscription in your app but trigger the exposure point on app launch rather than when users actually view the subscription page.
Problem: Maybe only 10% of users ever see the upsell page, making 90% of your test data worthless for this decision.
Loose vs. Tight Exposure Points
Loose Exposure Points:
These occur a bit too early. Users are exposed to the experiment prior to experiencing the tested feature. This premature exposure dilutes the data, and it is hard to find true impact.
Tight Exposure Points:
These take place at the exact moment that users experience the tested variant. Data collected with precise exposure points is more accurate and reliable, and easier to analyze.
Which is better?
It depends on your use case. Tight exposure points are preferable since they provide you with cleaner, more defined data, even at smaller sample sizes. Occasionally, however, tight exposure is not possible. In that situation, you can utilize loose exposure points with the knowledge you'll likely require a larger sample size to achieve significant results.
Avoid stacking changes
Never mix or pile multiple exposure points together in a single test. Split each change into its own A/B test. While it will take a bit more time, your data accuracy will be far greater, and you'll have more accurate conclusions on each individual feature.
Quick real-life example
Let's say you are adding a new row type in a table view:
- Good exposure point: Trigger exposure precisely when the new row is rendered in view.
- Bad exposure point: Fire exposure as soon as the table is loaded, even if the new row is not yet visible.
Final tips
- Plan ahead: Think through your exposure points at feature planning time.
- Iterate rapidly: If initial test results are not as expected, iterate rapidly.
I would love to hear your stories and experiences in determining exposure points for your A/B tests. Share them in the comments!
