AIML Rounds RoadmapMocksJobs
Concept18 minBefore this:Embeddings and Vector Indexes

Keeping an Index Current

Documents change daily, and a better embedding model has come out. How do you keep your index current?

The situation

An outdated refund policy keeps appearing in answers, though it was deleted weeks ago. Meanwhile, the team wants to move to a newer embedding model, and someone suggests re-embedding documents gradually in the same index. The interviewer asks:

Documents change daily, and a better embedding model has come out. How do you keep your index current?

Why it matters

Tutorials end once the index is built, but real work starts there. A stale policy gives a confident wrong answer, and a deleted document that's still retrievable can be a legal problem. Migrations are expensive enough that you should plan for them. See RAG Wrong Answer for what stale data looks like from the outside.

The idea

Think of moving a library to a new shelving system. You can't mix the two systems on one shelf, as nobody could find anything. So you set up the new shelves beside the old ones, shelve new books in both, move the rest over time, check it works, then change the signs overnight.

UpdateHash, then embedwhat changedStay freshDeleteTombstonesneed cleanupCheck it's goneMigrateDual write,backfill, flipNo downtime
Time is usually a bigger limit than money.

Play

Try to break it. Raise the chunks, then the rate limit. Then raise the tombstones.

Plan a migration

Rate limits set the schedule
embedding cost
–
backfill time
–
chunks changing during it
–
live results, of 40 candidates
–

Exact arithmetic, at the $0.13 per million tokens used in the source's example. Batch endpoints usually run slower, which isn't modelled here. Live results assume tombstones are spread evenly among 40 search candidates.

100 million chunks of 400 tokens is 40 billion tokens: about $5,200, and 5.6 days at 5 million tokens a minute. The money is small next to the time. During those days, about 2.8 million chunks change, and that's why you dual write. Raise the rate limit to 50, and it takes about half a day for the same money. Now raise tombstones to 50%: only 20 of 40 search candidates are live, so recall and speed suffer until you clean up.

Now say why: why do you need dual writes during a backfill?

How it works

In one breath

An index goes stale from the moment you build it. Update it incrementally: hash each document, and re-embed only what changed. Deletes are harder than they look, as graph indexes usually mark items as deleted, and those tombstones slow searches and crowd out real results until a cleanup. Changing the embedding model means rebuilding everything, because old and new vectors can't be compared. The safe way is a second index, dual writes, a background backfill, a recall check, then an instant switch with an alias.

Updates, deletes and migrations

Incremental updates

  1. Detect changes: use events from the source, or compare a content hash per document or section.
  2. Re-chunk and embed only what changed: stable section IDs avoid re-embedding a whole document for one edit.
  3. Replace, don't add: write new chunks and remove old ones together, so the same document never appears twice in different versions.
  4. Track freshness: store the source version and indexed time, and alert when ingestion falls behind.

Why deletes are hard

A node in an HNSW graph can't simply be removed, because other nodes link through it. So databases usually mark it as deleted, a tombstone, and clean up later. Until then, search still walks through dead nodes. If 20% of candidates are tombstones, a search for 40 candidates finds about 32 live ones. Cleanup, such as a vacuum or segment rebuild, can be slow. So plan for it, and if you have a legal duty to erase data, check when it's actually gone, including backups, caches and summaries.

The tombstone mechanism and cleanup steps here come from engineering blogs. Check your own database's documentation for exact behaviour.

Migrating an embedding model

StepWhat to doWhy
1. Build beside itA second index, with the new model's dimensionsThe old one keeps serving
2. Dual writeEmbed every new or changed document with bothThe new index doesn't fall behind
3. BackfillEmbed history in the backgroundDon't overwrite a newer vector from a dual write
4. EvaluateCompare recall@k on a few hundred real queries, and optionally shadow live trafficA newer model isn't always better for you
5. SwitchFlip an alias, and keep the old index for rollbackInstant, and reversible

Can you avoid re-embedding?

Drift-Adapter trains a small mapping from old vectors into the new model's space, from a sample of paired embeddings. It reports recovering 95 to 99% of the recall of a full re-embedding, at over 100 times lower recompute cost, and under 10 microseconds added per query. Treat this as a bridge. One practitioner cites a lower range of 85 to 95%, with no source, so test it on your data before relying on it.

The maths

    Worked example from Tian Pan’s post: 100M chunks of 400 tokens is 40B tokens, about $5,200 at $0.13 per million, and over 5 days at 5M tokens per minute.

    In practice

    In practice

    • Version everything: store the embedding model and chunker version with each chunk, so you know what needs rebuilding.
    • Never mix models: query and document vectors must come from the same model and version.
    • Keep the raw text: you can't re-embed or re-chunk what you didn't keep.
    • Watch index health: track tombstones, recall on a fixed test set, and ingestion lag. Set a cleanup rule.
    • Test before you upgrade: run your golden queries against the new model, because a higher benchmark score doesn't guarantee a gain on your data.

    The answer that sounds right

    "We'll re-embed gradually, in the same index"

    It sounds right, as it avoids a big switch. But vectors from two models live in different spaces, so their similarity scores mean nothing together. A query can only match one of them. Use a second index, or a second named vector, then switch once it's complete and tested.

    Say it

    Documents change daily, and a better embedding model has come out. How do you keep your index current?

    Say your answer out loud

    Aim for 30 seconds. It feels different from reading it, and that's the point.

    Tick what you covered

    The interviewer pushes further

    Follow-up 1 of 3

    Why can't you upgrade embeddings gradually, chunk by chunk, in one index?

    Answer it out loud, then check.

    Vectors from different models live in different spaces, so comparing them is meaningless, and query vectors can only match one. You need a second index, a second named vector, or an adapter that maps between the two.

    Follow-up 2 of 3

    Users still get answers from a deleted document. Where do you look?

    Answer it out loud, then check.

    The chunks may not have been removed, or they may be marked deleted but still served. Check the search filter, and look at caches, summaries, semantic caches and any graph built from it. Then add a test that deletes a document and checks that it can't be retrieved.

    Follow-up 3 of 3

    How often should you re-index?

    Answer it out loud, then check.

    Update content as it changes, on events if you can. Do full rebuilds only when the chunker or model changes, or when tombstones and drift hurt recall. Let your freshness needs and a fixed recall test decide.

    I'd update incrementally: detect changes with events or content hashes, re-embed only changed sections, and replace old chunks with new ones together. Deletes need care, as graph indexes usually leave tombstones, which slow search and crowd out real results until cleanup, so I'd track them. For a new embedding model, vectors can't be mixed, so I'd build a second index, dual write new documents, backfill history, compare recall on real queries, then flip an alias and keep the old index for rollback. 100 million chunks of 400 tokens costs about $5,000, but takes over 5 days at 5 million tokens a minute, so rate limits often set the schedule. An adapter can bridge the gap, but test it.

    Key terms and sources

    Incremental updates and hashing, why deletes are hard in HNSW, tombstones and vacuuming, migrating embedding models with dual writes and an alias, costs, and adapters.

    Sources

    Explanations are ours, built from these sources. Lab numbers are illustrative unless a lab says otherwise. Most recent papers are preprints.