AIML Rounds

QUIZ

A/B testing

14 questions on A/B testing, from 16 lessons. Pick an answer in your head, then open it.

On the roadmap for: ML Engineering, Data Science.

  1. Question 1 of 14

    Why do we randomly assign users to A and B?

    1. A. Randomisation isn't needed if groups are large
    2. B. To save money on running the experiment
    3. C. So the change is the only systematic difference
    4. D. To make each group smaller and faster
    View answer

    ✓ C. So the change is the only systematic difference

    Without randomisation, who gets the feature can be linked to the outcome, and the comparison stops being causal.

    Lesson: Randomisation and Assignment

  2. Question 2 of 14

    What is a guardrail metric?

    1. A. A metric you only look at once the test has ended
    2. B. The number of users enrolled in the test
    3. C. The main metric you're trying to improve
    4. D. A metric that mustn't get worse, like latency
    View answer

    ✓ D. A metric that mustn't get worse, like latency

    Guardrails stop you shipping a win on one metric that quietly harms the business or users elsewhere.

    Lesson: Choosing Metrics: OEC, Drivers and Guardrails

  3. Question 3 of 14

    You want to detect an effect half as large as before, with the same power. How does the required sample size change?

    1. A. It halves
    2. B. It roughly quadruples
    3. C. It stays the same
    4. D. It doubles
    View answer

    ✓ B. It roughly quadruples

    Sample size scales with 1 / (effect)². Halving the effect you want to detect needs about 4× the users.

    Lesson: Sample Size, Power and Runtime

  4. Question 4 of 14

    Your test is statistically significant. What else should you check before shipping?

    1. A. Whether the effect is big enough to matter, plus guardrails
    2. B. Nothing more: significance alone means you should ship it now
    3. C. Only that the p-value is also below 0.01
    4. D. Whether the test happened to start on a Monday
    View answer

    ✓ A. Whether the effect is big enough to matter, plus guardrails

    Significant isn't the same as worthwhile; decide using the interval and the costs.

    Lesson: Reading the Result: Significance and Decisions

  5. Question 5 of 14

    You randomise by user but measure click-through rate per page view. What goes wrong?

    1. A. CTR can't be computed when users differ
    2. B. The groups become unbalanced in size once some users view many more pages
    3. C. Views from one user correlate, so variance is understated
    4. D. Nothing: CTR per page view is the standard metric
    View answer

    ✓ C. Views from one user correlate, so variance is understated

    The analysis unit must respect the randomisation unit, or p-values come out too small.

    Lesson: Ratio Metrics and the Delta Method

  6. Question 6 of 14

    How does CUPED shorten an A/B test?

    1. A. By peeking early and stopping sooner
    2. B. By lowering the significance level to 10%
    3. C. By using pre-experiment behaviour to cut variance
    4. D. By dropping the least active users, who add noise
    View answer

    ✓ C. By using pre-experiment behaviour to cut variance

    Less variance means the same precision with fewer users, without biasing the estimate.

    Lesson: Variance Reduction with CUPED

  7. Question 7 of 14

    Revenue per user is your main metric, and a few huge spenders swing every result. What's a common fix?

    1. A. Drop the test and pick a different main metric
    2. B. Use only the treatment group's median revenue
    3. C. Cap extreme values or transform, then bootstrap
    4. D. Only count users who spend small amounts
    View answer

    ✓ C. Cap extreme values or transform, then bootstrap

    Heavy tails inflate variance; capping trades a little bias for much more stable results.

    Lesson: Revenue and Heavy-Tailed Metrics

  8. Question 8 of 14

    Your manager wants to check the test daily and stop as soon as it wins. How can that be valid?

    1. A. Use a lower p-value threshold just once
    2. B. It can't be made valid in any way
    3. C. Check only once a day, at the same fixed time
    4. D. Use a sequential test built for repeated looks
    View answer

    ✓ D. Use a sequential test built for repeated looks

    Ordinary p-values assume one look; sequential methods keep the false-positive rate at 5% across looks.

    Lesson: Peeking and Sequential Testing

  9. Question 9 of 14

    You tested 20 metrics and one came back significant at p < 0.05. Is it a real win?

    1. A. Only if the significant metric is revenue or similar
    2. B. Yes, as long as the p-value is 0.049
    3. C. Yes, definitely: p < 0.05 means a real effect
    4. D. Not necessarily: about one false positive is expected
    View answer

    ✓ D. Not necessarily: about one false positive is expected

    Use Bonferroni or Benjamini–Hochberg, or treat the result as a hypothesis for a new test.

    Lesson: Multiple Testing and Segments

  10. Question 10 of 14

    You planned a 50/50 split but got 50,500 vs 49,500 users out of 100,000. Does it matter?

    1. A. Yes: a sample ratio mismatch, likely a bug
    2. B. Only for small tests with under 10,000 users
    3. C. No: a 1% difference is well within normal noise
    4. D. Only if revenue also differs between the groups
    View answer

    ✓ A. Yes: a sample ratio mismatch, likely a bug

    χ² = 500²/50,000 × 2 = 10. Investigate before trusting any result.

    Lesson: Trust Checks: SRM and A/A Tests

  11. Question 11 of 14

    A new feature lifted engagement by 5% in week one. Why might it not last?

    1. A. Engagement can't change that fast, so week one is wrong
    2. B. It always lasts once users have seen it
    3. C. Week one data is always wrong
    4. D. Novelty: interest at first can fade over time
    View answer

    ✓ D. Novelty: interest at first can fade over time

    Primacy effects run the other way: users resist change at first. Longer tests reveal the true long-term effect.

    Lesson: Novelty, Primacy and Long-Term Effects

  12. Question 12 of 14

    A ride-app test showed treated riders booking 10% more, but after launch bookings barely moved. Why?

    1. A. Interference: treated riders took control riders' drivers
    2. B. The launch had a bug that the test didn't have
    3. C. Riders gave false answers during the test
    4. D. The test was too short to see the real, long-term effect of the change
    View answer

    ✓ A. Interference: treated riders took control riders' drivers

    In shared markets, randomise by region or time slot (switchbacks) so the groups don't compete.

    Lesson: Interference: When Groups Affect Each Other

  13. Question 13 of 14

    When should you prefer a classic A/B test over a multi-armed bandit?

    1. A. When you want the most clicks during the test
    2. B. When there are many variants to compare
    3. C. When you need a clean, unbiased effect estimate
    4. D. Never: bandits are always better
    View answer

    ✓ C. When you need a clean, unbiased effect estimate

    Bandits reduce the cost of showing worse options but give less reliable effect estimates.

    Lesson: Bandits vs A/B Tests

  14. Question 14 of 14

    Your coupon test is flat overall. Could the coupon still be worth giving to some users?

    1. A. Only if you rerun exactly the same test again
    2. B. Yes: uplift modelling finds who it changes
    3. C. No: a flat test means it helps nobody at all
    4. D. Only for brand-new users who never bought before
    View answer

    ✓ B. Yes: uplift modelling finds who it changes

    Average effects can hide positive and negative groups; validate any targeting rule in a new test.

    Lesson: Heterogeneous Effects and Uplift Modelling