AIML Rounds RoadmapMocksJobs

Heterogeneous Effects and Uplift Modelling

Your coupon test is flat overall. Could it still work for some users, and who should get it?

The situation

You have budget to send a coupon to 20% of users. A colleague trains a model on who's most likely to buy, and plans to send it to the top 20%. Last quarter's test of the same coupon was flat overall. The interviewer asks:

Your coupon test is flat overall. Could it still work for some users, and who should get it?

Why it matters

Many ML systems choose who gets something: a coupon, a notification, a retention call. Interviewers often move from A/B tests to this question. The usual mistake is to train a model on who's likely to buy, then send them the coupon. Many would have bought anyway, so the money is wasted.

The idea

Think of a doctor with a limited supply of a medicine. Giving it to patients who'd recover anyway is a waste, and so is giving it to those it can't help. The medicine belongs with those it changes the outcome for. Those aren't the same as the patients most likely to recover.

RandomiseTreatment andcontrol dataClean trainingModel upliftT, S, X learneror causal forestEffect, not responseEvaluateGaps by sliceof ranked usersQini curve
You're ranking by the change you cause, not by who buys.

Play

Try to break it. Treat 20%, then raise the model error. Then move the share to 100%.

Who gets the coupon?

Response against uplift
target likely buyers: extra purchases
–
target at random: extra purchases
–
target by uplift: extra purchases
–
best possible at this share
–

Our simulation of 20,000 users with a fixed seed. The numbers are invented: middling buyers gain the most, the keenest buyers can lose out, and about 18% of users are harmed. Green: by uplift. Grey dashed: at random. Red: by likely buyers. The uplift score here is the true effect plus error. Real data never shows you the true effect.

Treating 20% of users, targeting likely buyers gives about −168 extra purchases: you reach sure things and sleeping dogs, so the campaign does harm. Random targeting gives about 303. Even a noisy uplift model gives about 505. Set the model's error to 0 and it rises to 658. Now move the share to 100%. All three lines meet at about 1,515, because everyone gets treated, so the ranking stops mattering. The value of a model is in choosing who not to treat.

Now say why: why does targeting likely buyers do so badly?

Maths

In one breath

An A/B test gives one average, but a change may help some people and hurt others. Heterogeneous effects are those differences. If you can only treat some users, you should target those whose behaviour the treatment changes, not those most likely to buy anyway. Uplift models estimate each user's effect, using a randomised test as training data. You can never see one person's effect directly, so you evaluate on a held-out randomised group, by checking whether top-ranked users really show a larger gap between treatment and control.

Target the change

Four kinds of user

GroupWithout a couponWith a couponShould you treat them?
PersuadableDoesn't buyBuysYes: this is the only group that pays back
Sure thingBuysBuysNo: you give away margin
Lost causeDoesn't buyDoesn't buyNo: you pay for nothing
Sleeping dogBuysDoesn't buyNo: contact makes it worse, such as a reminder to cancel

This is the standard way to explain uplift modelling. In reality, you estimate each user's chance of being helped, because you can't sort people into these groups directly.

Three ways to build one

LearnerHow it worksWeak spot
S-learnerOne model, with treatment as a feature. Predict with it on and off, then subtractA regularised model can ignore treatment altogether
T-learnerTwo models, one per group. Subtract their predictionsErrors from two models stack up, especially with few treated users
X-learnerFit two models, compute each user's imputed effect from the other group's model, model those, then blend themMore steps, and it needs care when the data is thin

Künzel et al. found the X-learner performed well in simulations, but none of the learners was best everywhere. Causal forests instead extend random forests to estimate effects directly, and split the data so that the same users don't both choose the splits and estimate the effects.

Evaluating without seeing the truth

  1. Keep a held-out set from a randomised test.
  2. Rank its users by predicted uplift.
  3. In each slice, compare treated and control users. A good model shows larger gaps at the top.
  4. Plot the cumulative gain against the share treated: an uplift or Qini curve. Compare it with treating at random.

The maths

    Notation as in Künzel et al. These estimates need randomised data, or strong assumptions when treatment wasn’t random.

    In practice

    In practice

    • Train on random data: if past targeting was by rule, treated and control users differ. Keep a random holdout to supply clean training data.
    • Effects are small: individual effects are harder to predict than outcomes, so uplift models need a lot of data. The public Criteo benchmark has 13.9 million samples.
    • Plan your segments: mining many segments finds false winners. See multiple testing, and confirm a finding with a new test.
    • Include the cost: treat a user only when the expected gain beats the cost. A coupon that's used by a sure thing costs margin.
    • Keep testing the policy: compare the model's targeting with a simple rule in a live A/B test. Keep a share at random, or you'll lose your training data.

    The answer that sounds right

    "Send it to the people most likely to buy"

    It sounds right, since they're your best customers. But they were likely to buy anyway, so the coupon gives away margin for nothing, and some may even respond badly. What matters is the change the coupon causes, which only a randomised test can teach a model.

    Say it

    Your coupon test is flat overall. Could it still work for some users, and who should get it?

    Say your answer out loud

    Aim for 30 seconds. It feels different from reading it, and that's the point.

    Tick what you covered

    The interviewer pushes further

    Follow-up 1 of 3

    Why not target users with the highest predicted purchase probability?

    Answer it out loud, then check.

    They're the most likely to buy without help. The coupon then costs you margin for no extra sales. What you want is the change the coupon causes, which a response model doesn't estimate.

    Follow-up 2 of 3

    How do you evaluate an uplift model, if the true effect is never seen?

    Answer it out loud, then check.

    Use a randomised held-out set. Rank users by predicted uplift, split them into slices, and compare treated and control results in each slice. The cumulative gain against the share treated gives the uplift or Qini curve, which you compare with random targeting.

    Follow-up 3 of 3

    A test is flat overall, but a model finds a segment with a big lift. Do you ship to them?

    Answer it out loud, then check.

    Not yet, because searching for a segment finds false winners. Check that the segment holds on held-out data and makes sense, then run a new test on that segment alone.

    An average can hide who benefits. If I can only treat some users, I want those whose behaviour changes, not those likely to buy anyway, because sure things give away margin and sleeping dogs can be harmed. So I'd run a randomised test, then train an uplift model to estimate each user's effect. A T-learner fits one model per group and subtracts them, an S-learner uses one model with treatment as a feature, and an X-learner and causal forests are more advanced. You can't observe one person's effect, so I'd evaluate on a held-out randomised set: rank by predicted uplift, and check that top slices show larger gaps between treatment and control, using an uplift or Qini curve. Then I'd test the targeting policy live.

    Key terms and sources

    Why an average can hide who benefits, targeting uplift rather than response, T, S and X learners, causal forests, and how to evaluate uplift without seeing it directly.

    Sources

    Lab numbers are illustrative unless a lab says otherwise. Explanations are ours, built from these sources and standard statistics.