AIML Rounds RoadmapMocksJobs

Feedback Loops and Unbiased Training Data

Your recommender is retrained on its own clicks every week, and it keeps showing the same few items. Why, and how do you get data it can learn from?

The situation

Your recommender is retrained weekly on its own click logs. Each version looks better offline, but users see the same few items, and new items never take off. The interviewer asks:

Your recommender is retrained on its own clicks every week, and it keeps showing the same few items. Why, and how do you get data it can learn from?

Why it matters

Many models choose what happens next: what's recommended, which loans are approved, which transactions are reviewed. Their logs record outcomes only for the choices they made. Sculley et al. listed these hidden feedback loops as a form of technical debt in ML systems. Interviewers ask about them in recommender, ads, pricing and fraud designs.

The idea

Think of a restaurant critic who only revisits places they already like. They'll never find out the new place down the road is better, and their list gets shorter every year. Trying somewhere new now and then is the only way to learn.

ExploreA small shareof trafficEvery option gets dataLog chancesProbability ofeach shown itemCorrect laterHoldoutA group withoutthe modelReal effect
You only learn about what you chose.

Play

Try to break it. Run with no exploration. Then add 5%, then 20%.

A model that can't learn it was wrong

Exploration against lost clicks
item shown to most users on day 30
–
clicks per day on day 30
–
clicks lost over 30 days
–
traffic spent exploring
–

Illustrative, with expected values and no noise. 10,000 users a day. Real click rates: A 4%, B 6%, C 3%, D 2%, E 5%. The old model believes A 4.5%, B 2%, C 3%, D 2%, E 1%. Exploration spreads its share evenly over all five items, and the model updates its beliefs from each day's clicks.

With no exploration, the model keeps showing item A at 400 clicks a day. It never shows B, so it never learns B gets 6%, and it loses 6,000 clicks in 30 days. With 5% exploration, one day of data reveals B, the model switches, and it gets 590 clicks a day, losing 490 over the month, mostly on day one and the exploration itself. 20% exploration learns just as fast but keeps paying for it: 560 a day. So explore a little, all the time, and log the chances.

Now say why: why does a little exploration beat a lot?

How it works

In one breath

A recommender only sees clicks on what it showed. If it thinks item B is poor, it never shows B, so it never learns B is actually the best. Retraining on its own logs makes this worse each week, which is a feedback loop. The fixes: spend a small share of traffic on exploration, so every option gets some data; keep a holdout group served without the model; and log the probability that each shown item was chosen, so later training and evaluation can correct for what the old model preferred.

How loops form

You only see outcomes for what you chose

  • Recommenders: unshown items get no clicks, so they look unpopular.
  • Lending: you never learn whether rejected applicants would have repaid.
  • Fraud review: only flagged transactions get checked, so labels cover what the model already suspects.
  • Policing and inspections: Ensign et al. showed that sending patrols where incidents were recorded, and recording incidents where patrols go, can run away from the true rates.

Greedy models narrow fastest

Jiang et al. at DeepMind simulated recommenders acting on users whose interests shift with what they see. A model that always picked its current best items narrowed users onto a handful of items fastest. Random exploration and a growing pool of new items slowed this. Chaney et al. found that training on data shaped by an earlier recommender made recommendations more alike across users, and less useful.

The fixes

  • Exploration: show a small share of traffic other options, randomly or with a bandit, so every option keeps getting data. See bandits.
  • Log the chance of each action: record the probability that each shown item was chosen. Bottou et al. showed how such logs let you estimate how a different policy would have done, by weighting outcomes by one over that probability.
  • Holdouts: keep a small group served without the model, or with an older one, to measure its real effect.
  • Watch diversity: track how many distinct items are shown, and how concentrated exposure is.

The maths

    Inverse propensity scoring. It only works for actions the old system sometimes took, which is why exploration matters: with no exploration, many chances are zero.

    In practice

    In practice

    • Log propensities at serving time: the probability of each shown action, plus what else was eligible. You can't recover them later.
    • Keep a small exploration budget: a few percent, controlled and measured, rather than none.
    • Hold out a group: to measure the model's real effect against a simple baseline over time.
    • Evaluate offline with care: logs from the old model favour it, so weight by propensities, or confirm with an A/B test.
    • Track diversity and coverage: how many items get shown, and whether new items ever get a chance.

    The answer that sounds right

    "Retrain on the latest clicks, and it'll fix itself"

    It sounds right, since fresh data usually helps. But the clicks come only from items the model chose to show, so retraining reinforces its own picks. Explore a little, log the chances, and evaluate with weights or a holdout.

    Say it

    Your recommender is retrained on its own clicks every week, and it keeps showing the same few items. Why, and how do you get data it can learn from?

    Say your answer out loud

    Aim for 30 seconds. It feels different from reading it, and that's the point.

    Tick what you covered

    The interviewer pushes further

    Follow-up 1 of 3

    Your offline evaluation says the new recommender is worse, but it's never been shown to users. Can you trust that?

    Answer it out loud, then check.

    Not fully. The logs come from the old model, so they only have outcomes for items it liked. Weight by logged propensities if you have them, or run an A/B test.

    Follow-up 2 of 3

    Why not explore by showing random items to everyone sometimes?

    Answer it out loud, then check.

    Uniform random exploration is costly for users. Use a small share, a bandit that explores where it's uncertain, or explore within a shortlist of reasonable items.

    Follow-up 3 of 3

    A loan model approves fewer people each year, and its defaults look great. What's wrong?

    Answer it out loud, then check.

    You only see repayment for approved applicants, so the model never learns about the ones it rejects. Approve a small, controlled sample near the threshold to get unbiased labels.

    A model that chooses what users see only gets feedback on its own choices, so an item it wrongly rates low never gets shown, and retraining on its logs locks the mistake in. DeepMind's simulations showed greedy recommenders narrow fastest, while exploration and new items slow it down. I'd spend a few percent of traffic on exploration, log the probability of each shown action so I can weight outcomes later, keep a holdout group to measure real impact, and track how many distinct items get shown.

    Key terms and sources

    How models that act on the world shape their own training data, why a greedy system can never learn it was wrong, and the fixes: exploration, holdouts, logging the chance each action was taken, and correcting for it.

    Sources

    Explanations are ours, built from these sources. Lab numbers are illustrative unless a lab says otherwise. Many recent papers are preprints, and provider policies change.