QUIZ
A/B testing
14 questions on A/B testing, from 16 lessons. Pick an answer in your head, then open it.
On the roadmap for: ML Engineering, Data Science.
Question 1 of 14
Why do we randomly assign users to A and B?
- A. Randomisation isn't needed if groups are large
- B. To save money on running the experiment
- C. So the change is the only systematic difference
- D. To make each group smaller and faster
View answer
✓ C. So the change is the only systematic difference
Without randomisation, who gets the feature can be linked to the outcome, and the comparison stops being causal.
Lesson: Randomisation and Assignment
Question 2 of 14
What is a guardrail metric?
- A. A metric you only look at once the test has ended
- B. The number of users enrolled in the test
- C. The main metric you're trying to improve
- D. A metric that mustn't get worse, like latency
View answer
✓ D. A metric that mustn't get worse, like latency
Guardrails stop you shipping a win on one metric that quietly harms the business or users elsewhere.
Question 3 of 14
You want to detect an effect half as large as before, with the same power. How does the required sample size change?
- A. It halves
- B. It roughly quadruples
- C. It stays the same
- D. It doubles
View answer
✓ B. It roughly quadruples
Sample size scales with 1 / (effect)². Halving the effect you want to detect needs about 4× the users.
Lesson: Sample Size, Power and Runtime
Question 4 of 14
Your test is statistically significant. What else should you check before shipping?
- A. Whether the effect is big enough to matter, plus guardrails
- B. Nothing more: significance alone means you should ship it now
- C. Only that the p-value is also below 0.01
- D. Whether the test happened to start on a Monday
View answer
✓ A. Whether the effect is big enough to matter, plus guardrails
Significant isn't the same as worthwhile; decide using the interval and the costs.
Question 5 of 14
You randomise by user but measure click-through rate per page view. What goes wrong?
- A. CTR can't be computed when users differ
- B. The groups become unbalanced in size once some users view many more pages
- C. Views from one user correlate, so variance is understated
- D. Nothing: CTR per page view is the standard metric
View answer
✓ C. Views from one user correlate, so variance is understated
The analysis unit must respect the randomisation unit, or p-values come out too small.
Question 6 of 14
How does CUPED shorten an A/B test?
- A. By peeking early and stopping sooner
- B. By lowering the significance level to 10%
- C. By using pre-experiment behaviour to cut variance
- D. By dropping the least active users, who add noise
View answer
✓ C. By using pre-experiment behaviour to cut variance
Less variance means the same precision with fewer users, without biasing the estimate.
Lesson: Variance Reduction with CUPED
Question 7 of 14
Revenue per user is your main metric, and a few huge spenders swing every result. What's a common fix?
- A. Drop the test and pick a different main metric
- B. Use only the treatment group's median revenue
- C. Cap extreme values or transform, then bootstrap
- D. Only count users who spend small amounts
View answer
✓ C. Cap extreme values or transform, then bootstrap
Heavy tails inflate variance; capping trades a little bias for much more stable results.
Lesson: Revenue and Heavy-Tailed Metrics
Question 8 of 14
Your manager wants to check the test daily and stop as soon as it wins. How can that be valid?
- A. Use a lower p-value threshold just once
- B. It can't be made valid in any way
- C. Check only once a day, at the same fixed time
- D. Use a sequential test built for repeated looks
View answer
✓ D. Use a sequential test built for repeated looks
Ordinary p-values assume one look; sequential methods keep the false-positive rate at 5% across looks.
Lesson: Peeking and Sequential Testing
Question 9 of 14
You tested 20 metrics and one came back significant at p < 0.05. Is it a real win?
- A. Only if the significant metric is revenue or similar
- B. Yes, as long as the p-value is 0.049
- C. Yes, definitely: p < 0.05 means a real effect
- D. Not necessarily: about one false positive is expected
View answer
✓ D. Not necessarily: about one false positive is expected
Use Bonferroni or Benjamini–Hochberg, or treat the result as a hypothesis for a new test.
Lesson: Multiple Testing and Segments
Question 10 of 14
You planned a 50/50 split but got 50,500 vs 49,500 users out of 100,000. Does it matter?
- A. Yes: a sample ratio mismatch, likely a bug
- B. Only for small tests with under 10,000 users
- C. No: a 1% difference is well within normal noise
- D. Only if revenue also differs between the groups
View answer
✓ A. Yes: a sample ratio mismatch, likely a bug
χ² = 500²/50,000 × 2 = 10. Investigate before trusting any result.
Lesson: Trust Checks: SRM and A/A Tests
Question 11 of 14
A new feature lifted engagement by 5% in week one. Why might it not last?
- A. Engagement can't change that fast, so week one is wrong
- B. It always lasts once users have seen it
- C. Week one data is always wrong
- D. Novelty: interest at first can fade over time
View answer
✓ D. Novelty: interest at first can fade over time
Primacy effects run the other way: users resist change at first. Longer tests reveal the true long-term effect.
Question 12 of 14
A ride-app test showed treated riders booking 10% more, but after launch bookings barely moved. Why?
- A. Interference: treated riders took control riders' drivers
- B. The launch had a bug that the test didn't have
- C. Riders gave false answers during the test
- D. The test was too short to see the real, long-term effect of the change
View answer
✓ A. Interference: treated riders took control riders' drivers
In shared markets, randomise by region or time slot (switchbacks) so the groups don't compete.
Question 13 of 14
When should you prefer a classic A/B test over a multi-armed bandit?
- A. When you want the most clicks during the test
- B. When there are many variants to compare
- C. When you need a clean, unbiased effect estimate
- D. Never: bandits are always better
View answer
✓ C. When you need a clean, unbiased effect estimate
Bandits reduce the cost of showing worse options but give less reliable effect estimates.
Lesson: Bandits vs A/B Tests
Question 14 of 14
Your coupon test is flat overall. Could the coupon still be worth giving to some users?
- A. Only if you rerun exactly the same test again
- B. Yes: uplift modelling finds who it changes
- C. No: a flat test means it helps nobody at all
- D. Only for brand-new users who never bought before
View answer
✓ B. Yes: uplift modelling finds who it changes
Average effects can hide positive and negative groups; validate any targeting rule in a new test.