DATA SCIENCE · ROUND 2 OF 3
A/B testing and product
The interviewer wants to see you design a trustworthy experiment, read it honestly and turn it into a product decision.
Question 1 of 8
We want to test a new checkout page. Design the experiment.
Answer out loud first, then open the model answer.
View model answer
I'd start with the hypothesis and the decision it informs. I'd randomise by user, so each person sees one version consistently, and pick a primary metric close to the goal, such as purchase conversion per user, plus guardrails: revenue per user, latency, errors, refunds.
Then I'd choose the minimum effect worth detecting, compute the sample size and runtime for 80% power, run whole weeks to cover weekday patterns, write the decision rule in advance, and check sample ratio and data quality before reading results.
What the interviewer listens for
- Hypothesis and decision first
- Unit of randomisation
- Primary metric + guardrails
- Power, runtime, pre-registered rule, trust checks
Lessons: Randomisation and Assignment, Choosing Metrics: OEC, Drivers and Guardrails, Sample Size, Power and Runtime
Question 2 of 8
How do you work out how long a test must run?
Answer out loud first, then open the model answer.
View model answer
Sample size depends on the baseline rate and its variance, the smallest effect worth detecting, the significance level, usually 5%, and power, usually 80%. A useful rule: n per group ≈ 16 × variance / effect². Runtime is that sample divided by eligible traffic per day.
I'd round up to full weeks. If the runtime is too long, I'd pick a more sensitive metric, use variance reduction such as CUPED, or accept detecting only larger effects.
What the interviewer listens for
- Inputs: baseline variance, MDE, alpha, power
- n ≈ 16σ²/δ² rule of thumb
- Traffic → days; whole weeks
- Ways to shorten: CUPED, better metric
Lessons: Sample Size, Power and Runtime, Variance Reduction with CUPED
Question 3 of 8
The test is significant and the main metric is up 2%. Do you ship?
Answer out loud first, then open the model answer.
View model answer
Not automatically. First, trust checks: is the sample ratio as planned, did anything break in logging, and were there other changes at the same time? Then I'd look at the confidence interval, not just the p-value: is even the low end worth the cost of the change?
I'd check guardrails, then segments for anything alarming, without fishing for wins. If it all holds, I'd ship, ideally keeping a small holdout to confirm the long-term effect.
What the interviewer listens for
- Trust checks (SRM, logging)
- Effect size and interval vs practical value
- Guardrails
- Holdout for long-term effect
Lessons: Reading the Result: Significance and Decisions, Trust Checks: SRM and A/A Tests
Question 4 of 8
Your manager checks the dashboard daily and wants to stop when it turns green. What do you say?
Answer out loud first, then open the model answer.
View model answer
Each daily look is another chance to see p < 0.05 by luck, so stopping at the first green raises the false-positive rate well above 5%. That's how teams ship changes that do nothing.
If we need to stop early, I'd use a sequential test designed for repeated looks, such as always-valid p-values or alpha spending at set check-ins. Otherwise we fix the runtime up front and only read the result at the end.
What the interviewer listens for
- Peeking inflates false positives
- Explains why simply
- Sequential testing as the valid way to stop early
- Or a fixed horizon
Lessons: Peeking and Sequential Testing
Question 5 of 8
You planned 50/50, but the split came out 50.5/49.5 with 100,000 users. What do you do?
Answer out loud first, then open the model answer.
View model answer
I'd run a chi-squared test for sample ratio mismatch: it gives χ² ≈ 10, p ≈ 0.002, so this is very unlikely to be chance. Something is wrong with assignment or logging, such as bots, redirects or a crash in one variant that loses users.
I wouldn't trust any metric until I find the cause, because the missing users are probably not random. Regular A/A tests help catch these problems early.
What the interviewer listens for
- Runs an SRM check
- Recognises it as significant
- Lists plausible causes
- Doesn't trust results until fixed
Lessons: Trust Checks: SRM and A/A Tests
Question 6 of 8
A ride-hailing test shows treated riders book 10% more, but bookings barely move after launch. Why, and how would you test it properly?
Answer out loud first, then open the model answer.
View model answer
Interference: riders share the same drivers. Treated riders booked more partly by taking drivers that control riders would have used, so the test measured a shift between groups, not a gain for the whole market.
I'd randomise by something that keeps groups apart: by city or region, or by time slot in a switchback design, alternating the whole market between versions. Results are noisier, so the test needs to run longer.
What the interviewer listens for
- Names interference / SUTVA violation
- Explains cannibalisation of shared supply
- Cluster or switchback randomisation
- Notes the power trade-off
Lessons: Interference: When Groups Affect Each Other, Measure a Feed Change with Network Effects
Question 7 of 8
You tested 20 metrics and 15 segments. One segment shows a big win. Is it real?
Answer out loud first, then open the model answer.
View model answer
With that many comparisons, some will look significant by chance. At 5%, 20 metrics alone would give about one false positive on average. So I'd treat it as a hypothesis, not a result.
I'd correct for multiple testing, for example Benjamini–Hochberg, check whether the effect makes sense, and confirm it with a new test targeted at that segment.
What the interviewer listens for
- Multiple comparisons inflate false positives
- Correction methods
- Plausibility check
- Confirm with a follow-up test
Lessons: Multiple Testing and Segments, Heterogeneous Effects and Uplift Modelling
Question 8 of 8
Design a launch decision rule for a feature that wins on the main metric but is slightly worse on two guardrails.
Answer out loud first, then open the model answer.
View model answer
I'd agree the rule before the test: ship if the primary metric improves significantly and no guardrail gets worse by more than a set tolerance, using non-inferiority tests for the guardrails. Some guardrails, like crashes or legal issues, have zero tolerance.
Here I'd check whether the guardrail drops are beyond their tolerance. If not, ship and monitor; if they are, decide based on the agreed trade-off, or improve the feature and retest.
What the interviewer listens for
- Rule written before the test
- Non-inferiority margins for guardrails
- Hard-stop guardrails
- Clear decision path
Lessons: Set a Launch Decision Rule, Choosing Metrics: OEC, Drivers and Guardrails