AIML Rounds RoadmapMocksJobs

Embeddings and Vector Indexes

You have 100 million chunks. Which embedding model and index do you use, and what does it cost?

The situation

You're indexing 100 million chunks. The team picked the top model on a leaderboard, and float32 vectors. The memory bill is huge, and since you added a department filter, some users say search has got worse, with no errors in the logs. The interviewer asks:

You have 100 million chunks. Which embedding model and index do you use, and what does it cost?

Why it matters

At a million chunks, almost anything works. At a hundred million, the choice of index decides your bill, and a filter can silently drop your recall. These are also common questions in a RAG design interview: how would you size this, and what happens when you filter?

The idea

Think of a giant library map. The embedding model decides where each book goes, and shorter coordinates take less paper but are less exact. The index is the road network. Close most roads for a filter, and you can't reach the books, even though they're still there.

ModelTest on yourown queriesNot a leaderboardCompressint8 or binarywith rescoring4× or 32× smallerFilterGraphs losetheir pathsMeasure recall
A filter can quietly destroy recall.

Play

Try to break it. Size your corpus, and switch encoding. Then move the filter down to 1%.

Size it, then filter it

Memory against recall
vectors in memory
–
HNSW links (M = 16)
–
quality kept, best case
–
recall@10, plain graph with filter
–
recall@10, graph built for the filter
–

Memory is exact arithmetic. Quality kept is Hugging Face's best case, and it varies by model, as binary needs rescoring. Recall is from Qdrant's own benchmark on 1 million deep-image vectors at hnsw_ef = 64, using a single keyword filter. It isn't linked to the other sliders, and your data will differ.

100 million 1,024-dimensional chunks in float32 need about 410 GB of memory for vectors alone. Binary cuts that to 12.8 GB, and the graph's links then cost about as much as the vectors. Now move the filter. At 10%, a plain graph's recall falls to 20.6%, and at 1% to just 0.1%, while the query still returns quickly. Nothing errors, so you'd only notice by measuring. A graph built with the filter in mind holds 99%. That's the main trap of filtered search.

Now say why: why is there no error when recall collapses?

How it works

In one breath

Your embedding model decides what can be found, and your index decides how fast and at what cost. Choose a model by testing it on your own queries, not on a leaderboard alone. Vectors can be made far smaller: Matryoshka models can be shortened, int8 is 4 times smaller, and binary is 32 times smaller if you re-score the top results. Filters can quietly break graph search, because the graph's links lead to points you've filtered out. And single vectors have a hard limit, so keep BM25 and a re-ranker in the stack.

Models, compression and indexes

Choosing a model

  • Shortlist from a benchmark such as MTEB, for your language and domain.
  • Test on your data: measure recall@k on a few hundred real labelled queries. A leaderboard rank needn't predict your result.
  • It matters a lot: in one 450-setup study, the embedding model was among the largest factors in retrieval quality. In Qu et al., it outweighed the chunking method.

Making vectors smaller

MethodSizeQuality keptWatch out for
MatryoshkaKeep the first N dimensionsOne study: 768 of 1,024 dimensions gave 25% less storage, and NDCG@10 rose 2.3%Only for models trained this way
int84× smallerAbout 99.3%Needs a calibration set
Binary + rescoring32× smallerUp to about 96%Strongly depends on the model: 96.45%, 87.7% and 74.77% across three tested models

Quantization figures are from Hugging Face's benchmark on retrieval tasks, measured against full float32 vectors. Rescoring means searching with binary vectors, then re-scoring the top results with higher precision vectors.

Choosing an index

IndexIdeaGood for
FlatCompare with every vectorSmall collections. Perfect recall
HNSWHop through a layered graphHigh recall at low latency, with memory for links
IVF, IVF-PQSearch the nearest clusters, with optional compressionVery large collections, at some recall cost

Why filters break graphs

HNSW walks from node to node through links. If a filter removes most nodes, the walk runs out of valid paths. Qdrant notes that with 96% of points filtered out, fewer than one link per node survives on average. The lab uses its numbers. Fixes include building extra links for filter fields, graph repair methods like ACORN, and simply scanning the matching rows when very few match. A separate 2026 study found partition indexes (IVFFlat) beat HNSW at very low selectivity, and that pgvector's planner often chose poor plans. So measure recall for each filter you use.

A single vector has limits

Weller et al. show that a fixed number of dimensions can't represent every possible set of top results. On their deliberately hard LIMIT test, with simple questions like "who likes apples?", the best single-vector models reached under 20% recall@100. BM25 reached 93.6%, and a multi-vector model 54.8%. This test is built to stress that weakness, so everyday scores won't look like this. But it's a good reason to keep BM25 and a re-ranker.

The maths

    The FAISS guideline for HNSW memory. Real systems also store IDs, metadata and the chunk text, and upper graph layers add a little.

    In practice

    In practice

    • Measure index recall: compare ANN results with exact search on your own vectors, at your real filters, and watch p99 latency too.
    • Keep exact search when you can: below a few hundred thousand vectors, or for very selective filters, a flat scan may be simpler and better.
    • Store full vectors on disk: keep float32 on disk for rescoring, and in-memory binary for the first pass.
    • Check the model card: don't truncate vectors unless the model was trained for it, or test first.
    • Plan for change: a new model means new vectors everywhere. See Keeping an Index Current.

    The answer that sounds right

    "The top model on the leaderboard is the best for us"

    It sounds right, since the benchmark is rigorous. But it measures other people's data and questions. Your names, codes and language may behave differently, and memory cost matters too. Shortlist from a benchmark, then decide by recall on a few hundred of your own labelled queries.

    Say it

    You have 100 million chunks. Which embedding model and index do you use, and what does it cost?

    Say your answer out loud

    Aim for 30 seconds. It feels different from reading it, and that's the point.

    Tick what you covered

    The interviewer pushes further

    Follow-up 1 of 3

    Estimate memory for 50 million chunks at 768 dimensions in float32, with HNSW at M = 16.

    Answer it out loud, then check.

    Vectors: 50M × 768 × 4 bytes = 153.6 GB. Links: 50M × 8 × 16 bytes = 6.4 GB. So about 160 GB, plus the chunk text and metadata. As int8, the vectors fall to 38.4 GB.

    Follow-up 2 of 3

    Why does a filter hurt recall in HNSW, but return quickly?

    Answer it out loud, then check.

    The walk follows links, and most neighbours have been filtered out, so it ends early with too few valid results. It's fast because it did little work. Nothing errors, so you only notice by measuring recall with that filter.

    Follow-up 3 of 3

    Should you truncate a 1,536-dimensional vector to 256 to save money?

    Answer it out loud, then check.

    Only if the model was trained for it, as in Matryoshka models. Then test recall on your data. Truncating other models usually hurts a lot. If you plan to use binary too, be careful, as very few dimensions leave very few bits.

    I'd shortlist embedding models from a benchmark like MTEB, then pick by recall on a few hundred of my own labelled queries. For memory, 100 million 1,024-dimensional float32 vectors need about 410 GB. int8 is 4 times smaller at about 99% quality, and binary is 32 times smaller at up to 96% with rescoring, though that varies a lot by model. I'd use HNSW for high recall at low latency, or IVF-PQ at very large scale. Filters are the main trap, as they cut the graph's paths: in one benchmark a plain graph fell to 20% recall on a 10% filter. So I'd use a filter-aware index and measure recall per filter. Single vectors also have limits, so I'd keep BM25 and a re-ranker.

    Key terms and sources

    Choosing an embedding model, shortening vectors with Matryoshka, scalar and binary quantization, HNSW and IVF memory, why filters break graph search, and the limits of a single vector.

    Sources

    Explanations are ours, built from these sources. Lab numbers are illustrative unless a lab says otherwise. Most recent papers are preprints.