AIML Rounds

QUIZ

Experiment design

5 questions on Experiment design, from 5 lessons. Pick an answer in your head, then open it.

On the roadmap for: Data Science.

  1. Question 1 of 5

    A new LLM tutor prompt looks better in demos. What's the right primary metric for a production test?

    1. A. The number of tokens used for each answer
    2. B. A real outcome like learning, with guardrails
    3. C. The average length of the tutor's responses
    4. D. How good the demo conversations looked to the team
    View answer

    ✓ B. A real outcome like learning, with guardrails

    Demos are cherry-picked; randomise real users and measure what the product is for.

    Lesson: Test an LLM Prompt in Production

  2. Question 2 of 5

    AI-written email subject lines lost to the old rule-based ones. What's a sensible next step?

    1. A. Abandon AI-written subject lines for good
    2. B. Find out why using segments and examples, then retest
    3. C. Run the same test again until the AI version wins
    4. D. Ship them anyway, since AI models will improve
    View answer

    ✓ B. Find out why using segments and examples, then retest

    A loss is information: find where and why it lost before deciding.

    Lesson: Fix a Failing AI Feature

  3. Question 3 of 5

    A feed change affects what a user's friends see. How do you measure its real effect?

    1. A. Randomise individual users as usual
    2. B. Ask users whether they noticed a change
    3. C. Randomise clusters of connected users together
    4. D. Use only offline data from before the change
    View answer

    ✓ C. Randomise clusters of connected users together

    Cluster (graph) randomisation keeps most interactions within one group.

    Lesson: Measure a Feed Change with Network Effects

  4. Question 4 of 5

    Why write the launch decision rule before the test starts?

    1. A. To make the test finish sooner, with analysis ready
    2. B. So results can't be reinterpreted after the fact
    3. C. It's a legal requirement for experiments
    4. D. It doesn't matter when it's written
    View answer

    ✓ B. So results can't be reinterpreted after the fact

    Deciding after seeing the data invites cherry-picking.

    Lesson: Set a Launch Decision Rule

  5. Question 5 of 5

    What's a core part of a platform that runs hundreds of A/B tests at once?

    1. A. Running only one test at a time across the company
    2. B. Manual spreadsheets for tracking each test's results
    3. C. Hash assignment, SRM checks, shared metrics, overlaps
    4. D. A single dashboard showing total revenue every day
    View answer

    ✓ C. Hash assignment, SRM checks, shared metrics, overlaps

    Automation makes tests trustworthy and cheap enough to run many.

    Lesson: Design an Experimentation Platform