AIML Rounds RoadmapMocksJobs

Re-ranking and Late Interaction

The right chunk is retrieved, but it's ranked 12th. What do you do, and what's late interaction?

The situation

Recall@50 is 94%, but recall@5 is only 58%. The right chunk is usually found, then buried. A colleague wants to send 50 chunks to the model. Another has read that ColBERT doubles results. The interviewer asks:

The right chunk is retrieved, but it's ranked 12th. What do you do, and what's late interaction?

Why it matters

Only the top few chunks reach the model, and the model uses early chunks most. So a good chunk ranked 12th might as well be missing. Re-ranking is often the cheapest big gain, and a common interview follow-up. See Search and Retrieval for BM25 and fusion.

The idea

Think of hiring. Sorting 10,000 CVs by keywords is quick but crude, so you use it to get a shortlist. Then you read those 50 carefully. Late interaction is like checking each requirement against the best line in a CV, rather than judging its overall impression.

Bi-encoderOne vectoreachFast, coarseLate interactionToken bybest matchMore memoryCross-encoderReads pairstogetherAccurate, slow
Retrieve broadly, then rank precisely.

Play

Try to break it. Add off-topic words to document A. Watch both scores.

Average against MaxSim

Why long chunks get diluted
single vector picks
–
MaxSim picks
–
vectors to store for A
–

Exact calculation on invented 4-number word vectors, with topics refund, delay, plan and food. Query: "gold refund late". Document A: "gold refund for late orders", plus off-topic words about food. Document B: "plan offers delivery refund", which is only partly relevant. Single vector: cosine of the averaged word vectors. MaxSim: the sum of each query word's best cosine. Scores are divided by the number of query words.

With no off-topic words, both methods pick document A. Add just 2 off-topic words, and the single vector flips to B: A falls from 0.99 to 0.70, against B's 0.89. At 10 words, A is down to 0.23. MaxSim stays at 1.00, because each query word still finds its exact match. The cost is storage: 14 vectors rather than 1 for this short document. MaxSim also has a blind spot: it ignores everything else in the document, so it can't tell that A is mostly about food. These are invented vectors, built to show the mechanism. See the conflicting results above for real data.

Now say why: why does MaxSim hold up, when averaging doesn't?

How it works

In one breath

A first-stage retriever must be fast, so it compares one vector per chunk. A cross-encoder reads the query and chunk together, which is far more accurate, but too slow for more than a short list. Late interaction sits between them: ColBERT stores a vector for every token, and scores each query token against its best match in the document. That keeps fine detail, but uses much more memory. Measured gains vary. In one study, late interaction doubled the hit rate, and in another it did worse than a single vector. So test on your own data.

Three ways to compare

Bi-encoderLate interactionCross-encoder
ComparesOne vector eachEvery query token with every document tokenReads both together
PrecomputedYesYes, one vector per tokenNo
Use forFirst-stage retrievalFirst stage or re-rankingRe-ranking a short list
Main costLeast memoryMuch more memoryLatency per candidate

What re-ranking delivers

  • Microsoft (vendor test): hybrid search scored 51.2 NDCG@3, and adding its new ranker raised that to 72.3. It re-ranked 50 documents in about 158 ms on average.
  • Anthropic (vendor test): re-ranking took retrieval failures from 2.9% to 1.9%, from 150 candidates down to 20.
  • Newer isn't always better: in the 450-setup study, a 2019 MiniLM cross-encoder beat all four recent rerankers it tested, on those corpora.
  • Gains overlap: that same study found re-ranking leaves less room for other upgrades, and for a strong embedding model. Test the full stack.

Does MMR help?

Maximal marginal relevance penalises results that look like ones already chosen. In the 450-setup study, it cut NDCG@10 by 3.6% on average, and ARAGOG found no notable gain. On focused corpora, similar passages are often exactly the evidence you need. Use it only if your problem is near-duplicates, and consider removing duplicates at ingest instead.

Late interaction: conflicting results

StudyResultCaveat
463,971 arXiv papersColBERT: Hit@3 of about 93 to 95%, against 46 to 47% for plain denseSynthetic questions, and a different encoder too
Financial and technical reportsText late interaction: nDCG@5 of 0.32, against 0.46 for dense. Visual late interaction was best, at 0.74Visually rich documents
LIMIT testMulti-vector: 54.8% recall@100, against under 20% for denseA deliberately hard test

So don't treat it as an automatic upgrade. The memory cost is real: in the second study, text late interaction used about 2.8 times the memory of dense, and page-image late interaction about 15 times.

The maths

    The MaxSim operator from ColBERT (Khattab and Zaharia, 2020). A single vector averages everything into one point before comparing.

    In practice

    In practice

    • Diagnose first: if recall@50 is high and recall@5 is low, a re-ranker should help. If recall@50 is low, it can't.
    • Tune the candidate count: more candidates raise the chance of including the answer, and add latency. Try 50 to 150.
    • Budget latency: measure the re-ranker at your p99, and cap the candidate length.
    • Use metadata where you can: a re-ranker can't tell a current policy from an old one that reads the same. Filter or add dates.
    • Compare to no re-ranker: keep measuring the gain, as a stronger embedding model can shrink it.

    The answer that sounds right

    "Just send more chunks to the model"

    It sounds right, since the answer is then in the prompt. But models use later text less, so a chunk at rank 30 may be ignored, while you pay for all of them. Put the right chunk near the top instead. That's a re-ranker's job.

    Say it

    The right chunk is retrieved, but it's ranked 12th. What do you do, and what's late interaction?

    Say your answer out loud

    Aim for 30 seconds. It feels different from reading it, and that's the point.

    Tick what you covered

    The interviewer pushes further

    Follow-up 1 of 3

    Why does averaging hurt long chunks?

    Answer it out loud, then check.

    A single vector squeezes everything into one point, so off-topic text pulls it away from the relevant part. That's why chunks are kept small, and why MaxSim, where each query token finds its best match, holds up better.

    Follow-up 2 of 3

    A re-ranker adds 200 ms. Is it worth it?

    Answer it out loud, then check.

    Compare it with the rest of the request. Next to a few seconds of LLM time, 200 ms is often fine if answer quality rises. Measure the gain on labelled questions, and try fewer candidates first.

    Follow-up 3 of 3

    Recall@50 is 60%. Will a re-ranker fix it?

    Answer it out loud, then check.

    No. A re-ranker can only reorder what it's given. Fix retrieval first with hybrid search, better parsing or chunking, query rewriting or a better embedding model.

    If the right chunk is found but ranked low, I'd add a second stage. A bi-encoder compares single vectors, so it's fast but coarse. A cross-encoder reads the query and chunk together, which is much more accurate, so I'd run it on the top 50 to 150. Late interaction, like ColBERT, stores a vector per token, and sums each query token's best match, which keeps detail without running a model per pair at query time, but costs much more memory. Results vary: it doubled the hit rate in one study, and lost to dense in another. I'd also check MMR before using it, as it hurt on focused corpora. And I'd measure the whole stack, because gains overlap.

    Key terms and sources

    Why a second ranking stage helps, cross-encoders, how ColBERT's MaxSim works, latency and memory costs, why MMR can hurt, and conflicting evidence on late interaction.

    Sources

    Explanations are ours, built from these sources. Lab numbers are illustrative unless a lab says otherwise. Most recent papers are preprints.