AIML Rounds RoadmapMocksJobs

Reading the Result: Significance and Decisions

Your A/B test came back significant. How do you decide whether to ship?

The situation

Your test ran to plan: 50,000 users per group, 10.0% against 10.4%, and p = 0.04. The product manager says it's significant, so ship it. Building and maintaining the feature isn't free. The interviewer asks:

Your A/B test came back significant. How do you decide whether to ship?

Why it matters

With enough users, tiny effects become significant. A real lift of 0.01% can have a p-value near zero and still not be worth the engineering cost. This lesson turns the numbers from hypothesis tests and confidence intervals into a decision.

The idea

Think of a bathroom scale that says you lost a tiny amount. You may be confident it's real, but that doesn't mean it matters. Two questions are separate: is it real, and is it big enough to care about? The interval answers both.

Above your barReal, and bigenoughShipCrosses both linesCould be nothing,could be goodNeed dataBelow zeroIt hurtsDon't ship
Check the interval against zero and against the smallest lift worth shipping.

Play

Try to break it. Set the lift to 0.1 points, then drag the users up to 500,000.

Read the interval

Significant against meaningful

Grey line: zero. Dashed line: your bar. Block: the 95% interval. The scale runs from −2 to +3 points.

95% interval, in points
–
p-value
–
decision
–

Illustrative inputs. The formula is the two-proportion z-test, using unpooled standard errors for the interval and the p-value.

Set the lift to 0.1 points, then drag the users up to 500,000. It becomes highly significant and still isn't worth shipping. That's the difference between statistical and practical significance.

Now say why: how can a result be highly significant and still not worth shipping?

Maths

In one breath

A low p-value only says the difference is unlikely to be pure chance. It doesn't say the effect is big enough to matter. So read the confidence interval against two lines: zero and the smallest lift worth shipping. Entirely above your bar: ship. Crossing both: you need more data. If a result looks too good, suspect a bug first.

Two lines, one interval

Where the interval sitsWhat it meansDecision
Entirely above your barReal, and large enough to matter.Ship, if guardrails hold.
Above zero, but crosses your barProbably real, but it might be too small to matter.Judgement call, or more data.
Above zero, entirely below your barReal, but too small to matter.Usually don't ship, unless it's cheap.
Crosses zero and your barCould be nothing, could be worthwhile.Not enough data.
Crosses zero, entirely below your barNo meaningful effect, either way.Don't ship, and you've learned something.
Entirely below zeroIt hurts.Don't ship.

What not to say

  • "p = 0.03, so there's a 97% chance B is better." The p-value assumes there's no effect. It isn't the chance your idea works.
  • "Not significant, so there's no effect." You may simply lack the data. Look at how wide the interval is.
  • "The lift is 8%." Give the interval too. It's an estimate, and significant results tend to overstate the truth.

Twyman's law

The more interesting or unusual a figure looks, the more likely it comes from a mistake. If a small change seems to lift revenue by 20%, check the logging, the assignment and the sample ratio before you celebrate.

The maths

    Worked example: 50,000 users per group, 10.0% against 10.4%. The difference is 0.40 points with a standard error of 0.19, so the interval is 0.02 to 0.78 points and p ≈ 0.04. Significant, but the interval also includes lifts too small to matter.

    In practice

    In practice

    • Check validity first: confirm the sample ratio and guardrails before you read the main metric.
    • Set your bar before the test: it should reflect the cost of building and maintaining the change.
    • Report the interval: "between 0.02 and 0.78 points" helps people decide far better than "significant".
    • Confirm surprises: rerun a big or unexpected win to see whether it holds.
    • Most ideas fail: at Microsoft, about one in three experiments was reported as a success, so a win should earn scrutiny.

    The answer that sounds right

    "p below 0.05, so ship it"

    It sounds right, since that's the agreed rule. But a low p-value only suggests the effect isn't zero. With enough users, tiny effects pass. Here, the interval runs from nearly nothing to 0.78 points, so the real question is whether the low end is worth the cost.

    Say it

    Your A/B test came back significant. How do you decide whether to ship?

    Say your answer out loud

    Aim for 30 seconds. It feels different from reading it, and that's the point.

    Tick what you covered

    The interviewer pushes further

    Follow-up 1 of 3

    What's the difference between statistical and practical significance?

    Answer it out loud, then check.

    Statistical significance says the effect is unlikely to be only chance. Practical significance says it's large enough to be worth it. With huge samples, a tiny lift can pass the first and fail the second.

    Follow-up 2 of 3

    The result isn't significant. Is the change useless?

    Answer it out loud, then check.

    Not necessarily. If the interval is narrow around zero, there's no meaningful effect. If it's wide and includes worthwhile lifts, the test was too small to tell. The width of the interval tells you which.

    Follow-up 3 of 3

    A small button colour change shows a 20% revenue lift. What do you do?

    Answer it out loud, then check.

    Suspect it first. I'd check the sample ratio, logging, outliers such as one huge purchase, and whether bots were included. If all of it holds, I'd rerun the test, because results this surprising are usually errors.

    First I'd check the test is valid: sample ratio and guardrails. Then I'd read the confidence interval against zero and against the smallest lift worth shipping. If it's entirely above that bar, ship. If it crosses zero and the bar, I need more data. If it's above zero but below the bar, the effect is real but too small to matter. Significance alone isn't enough, because with enough users tiny effects become significant. And if a result looks too good, I'd suspect a bug first.

    Key terms and sources

    Confidence intervals, statistical against practical significance, and a clear rule for ship, don't ship or get more data.

    Sources

    Lab numbers are illustrative unless a lab says otherwise. Explanations are ours, built from these sources and standard statistics.