AIML Rounds RoadmapMocksJobs

Novelty, Primacy and Long-Term Effects

Your new feature lifted engagement by 5% in week one. Will that lift last?

The situation

You redesigned the home page. Engagement jumps 5% in the first week, and the team wants to ship and celebrate. By week two, the lift looks smaller. The interviewer asks:

Your new feature lifted engagement by 5% in week one. Will that lift last?

Why it matters

A test measures what happens during the test, while you're shipping for what happens after it. Decisions based on a week of data can lock in a change that fades, or kill one that would have paid off.

The idea

Think of a new phone. You use it constantly for a week, and then it's just your phone. Or think of a new keyboard layout, where you're slower for a while and then faster. A one-week test can't tell which of those you're watching.

NoveltyStarts high,then fadesOverstated earlyPrimacyStarts low,then growsUnderstated early
Plot the daily effect, by days since first exposure.

Play

Try to break it. Set the bump to −4, and the long-term effect to +0.5. Then lengthen the test.

Run the test longer

Early result against lasting result
test reports (average lift)
–
long-term lift
–
error in the estimate
–

Illustrative: the bump fades with λ = 7 days. Assumes all users join on day one, and ignores noise.

Try an early bump of −4 with a long-term effect of +0.5. A one-week test says it's harmful, while it ends up helping. That's primacy: the short test gets the direction wrong.

Now say why: how can a short test get the direction wrong?

Maths

In one breath

People react to change itself. Novelty: they try something because it's new, then the lift fades. Primacy: they need time to get used to it, so it starts low and improves. So a short test can get the size or even the direction wrong. Plot the effect by days since first exposure, compare new users with existing ones, and keep a small holdback group to measure what lasts.

Recognise the shape

EffectWhat happensRisk
NoveltyPeople explore the new thing, then interest falls.You overestimate it, and ship a change that fades.
PrimacyPeople are used to the old way. Engagement grows as they learn.You underestimate it, and kill a good change.
Slow harmMore ads earn more now, while people gradually use the product less.Short-term gain hides long-term loss.

How to spot it

  • Plot the daily effect: a stable effect is flat. A trend suggests one of these.
  • Use days since first exposure: by calendar day, each date mixes people who've just joined with those who've been there for weeks.
  • Compare new and existing users: new users have no old habit, so a novelty effect shouldn't apply in the same way.
  • Check the cumulative effect too: it can look like it's shrinking simply because the interval is narrowing.

Measuring the long term

  • Run longer: simple, but it slows everything down and users drop out over time.
  • Holdback: after launch, keep a small share on the old version for months. It must be big enough to detect the effect.
  • User-learning designs: Google's Hohnhold et al. (2015) measured how exposure over time changes behaviour.

The maths

    An illustrative model, used for the lab. Real effects need not follow a smooth curve, so look at your data.

    In practice

    In practice

    • Check visible changes more: people notice redesigns and new features. Invisible improvements, like faster ranking, rarely show novelty.
    • Don't just wait: if a test runs a long time, users who remain are a selected group, and cookies get lost. Long tests have their own biases.
    • Size the holdback: a very small holdback can't detect a long-term change, so plan it like any other test.
    • Use cumulative learning: track what past short tests predicted against what holdbacks later found, and adjust your judgement.

    Surrogates: predicting the long term

    When you can't wait, use short-term metrics to predict the long-term one. Athey et al. combine many intermediate outcomes into a surrogate index, fitted on past data where the long-term result is known. It relies on a strong assumption: that the treatment affects the long-term outcome only through those short-term metrics. Check it against past tests and holdbacks wherever you can.

    The answer that sounds right

    "Just run every test for three months"

    It sounds right, since it measures the long term. But it slows the team to a crawl, and long tests have problems too: cookies get lost, and those who remain are a selected group. Check for a trend in the daily effect, and save holdbacks for major launches.

    Say it

    Your new feature lifted engagement by 5% in week one. Will that lift last?

    Say your answer out loud

    Aim for 30 seconds. It feels different from reading it, and that's the point.

    Tick what you covered

    The interviewer pushes further

    Follow-up 1 of 3

    Why plot by days since first exposure rather than calendar days?

    Answer it out loud, then check.

    Each calendar day mixes users who've just joined with users who've been there for weeks. Trends get blurred, and can even look like novelty when they're only changes in the mix. Aligning by first exposure compares users at the same stage.

    Follow-up 2 of 3

    How does comparing new and existing users help?

    Answer it out loud, then check.

    Existing users have habits to break, so novelty and primacy affect them. New users have no earlier version to compare with. If existing users show a fading lift while new users hold steady, it's probably a novelty effect.

    Follow-up 3 of 3

    What's the cost of a long-term holdback?

    Answer it out loud, then check.

    Those users keep the older experience, so you lose some of the benefit, and the team must maintain both versions for months. It also must be large enough to detect the effect. I'd save it for major launches, not every change.

    Not necessarily. People react to change itself. With novelty, they try it because it's new, and the lift fades. With primacy, they start worse and improve as they learn. So I'd plot the effect by days since first exposure, because a flat line suggests it's stable. I'd compare new and existing users, since new users have no old habit. For important launches, I'd keep a small holdback for months to measure the lasting effect, and size it so it can detect that.

    Key terms and sources

    Why short-term results can mislead, how to spot novelty and primacy effects, and how long-term holdbacks measure what lasts.

    Sources

    Lab numbers are illustrative unless a lab says otherwise. Explanations are ours, built from these sources and standard statistics.