QUIZ
Experiment design
5 questions on Experiment design, from 5 lessons. Pick an answer in your head, then open it.
On the roadmap for: Data Science.
Question 1 of 5
A new LLM tutor prompt looks better in demos. What's the right primary metric for a production test?
- A. The number of tokens used for each answer
- B. A real outcome like learning, with guardrails
- C. The average length of the tutor's responses
- D. How good the demo conversations looked to the team
View answer
✓ B. A real outcome like learning, with guardrails
Demos are cherry-picked; randomise real users and measure what the product is for.
Lesson: Test an LLM Prompt in Production
Question 2 of 5
AI-written email subject lines lost to the old rule-based ones. What's a sensible next step?
- A. Abandon AI-written subject lines for good
- B. Find out why using segments and examples, then retest
- C. Run the same test again until the AI version wins
- D. Ship them anyway, since AI models will improve
View answer
✓ B. Find out why using segments and examples, then retest
A loss is information: find where and why it lost before deciding.
Lesson: Fix a Failing AI Feature
Question 3 of 5
A feed change affects what a user's friends see. How do you measure its real effect?
- A. Randomise individual users as usual
- B. Ask users whether they noticed a change
- C. Randomise clusters of connected users together
- D. Use only offline data from before the change
View answer
✓ C. Randomise clusters of connected users together
Cluster (graph) randomisation keeps most interactions within one group.
Question 4 of 5
Why write the launch decision rule before the test starts?
- A. To make the test finish sooner, with analysis ready
- B. So results can't be reinterpreted after the fact
- C. It's a legal requirement for experiments
- D. It doesn't matter when it's written
View answer
✓ B. So results can't be reinterpreted after the fact
Deciding after seeing the data invites cherry-picking.
Lesson: Set a Launch Decision Rule
Question 5 of 5
What's a core part of a platform that runs hundreds of A/B tests at once?
- A. Running only one test at a time across the company
- B. Manual spreadsheets for tracking each test's results
- C. Hash assignment, SRM checks, shared metrics, overlaps
- D. A single dashboard showing total revenue every day
View answer
✓ C. Hash assignment, SRM checks, shared metrics, overlaps
Automation makes tests trustworthy and cheap enough to run many.