AIML Rounds RoadmapMocksJobs
Concept20 minBefore this:RAG End to End

Parsing and Chunking

Your RAG misreads tables and splits answers in half. How do you parse and chunk documents?

The situation

Your policy assistant answers plain questions well, but gets refund amounts wrong. You look at the chunks, and the table's headers are in one chunk and its values are in another. A colleague suggests a smarter chunker. The interviewer asks:

Your RAG misreads tables and splits answers in half. How do you parse and chunk documents?

Why it matters

RAG End to End shows the chunk size trade-off. In real systems, though, the damage often starts earlier. Uber's team found PDF loaders "often fail to correctly capture structured text and formatting (such as bullet points and tables)", and tried several without finding a universal solution. If a table's headers are gone, no retriever can bring them back.

The idea

Think of photocopying a recipe book and cutting it into cards. If the copier scrambles the ingredients table, no filing system can save you. And if you cut through the middle of a recipe, each card is useless alone. So get a clean copy first, then cut along the natural breaks.

ParseKeep tablesand headingsComes firstChunkSplit onstructureTest sizesAdd contextContextual orlate chunkingIf needed
You can't recover what parsing lost.

Play

Try to break it. Start with flattened text at 20 words. Then raise the size. Then keep the table.

Break a table

Parsing against chunk size
questions answerable
–
chunks in the index
–
words sent per question
–

Exact calculation on a short invented refund policy with one table, and 6 questions. Retrieval takes the top 2 chunks by word overlap with the question. A question counts as answerable only if a single retrieved chunk holds the row, the column header and the value. The box shows how the chunk holding the first table row looks.

With flattened text at 10 words, only 2 of 6 questions can be answered, as the table's headers land in one chunk and its values in others. Keep the table row by row, and it's 4 of 6 at the same size, because every row carries its own headers. At 20 words both reach 6 of 6, but the kept table sends 23 words per question, against 40 for flattened. Raise flattened text to 120 words, and it still scores 6 of 6, but it sends 103 words, as the whole policy is one chunk. This lab also flatters flattened text: it only checks that the words share a chunk. A model still has to work out which number belongs to which column.

Now say why: why does a big chunk answer more, yet still cost more?

How it works

In one breath

Before you can retrieve anything, messy documents have to become clean text. Parsing is where tables lose their headers and scanned pages lose their layout. Chunking then decides what each retrieved piece contains. The best start is simple: split on the document's own structure, keep tables whole, and test a few sizes on your own questions. Semantic chunking is often not worth its cost. If a chunk doesn't make sense alone, add context with contextual or late chunking. For pages full of charts, you can embed the page image instead.

From documents to chunks

Parsing comes first

ContentWhat goes wrongWhat helps
TablesRows flatten, headers separate from valuesConvert to markdown, repeat headers per row, and never split one mid-table
Headings and listsLost, so a chunk doesn't know its sectionPrefer a source with structure, such as HTML, and keep the heading path as metadata
Scans and chartsOCR drops layout and labelsLayout-aware parsing, captions, or embedding the page image

Uber moved its policy documents from PDFs to a format with headings and tables it could extract reliably, used an LLM to turn tables into markdown, and gave each table chunk a short summary and keywords.

Chunking methods

MethodHowEvidence and cost
Fixed sizeEvery N tokens or sentencesCheap, and a strong baseline
Structure-awareSplit on headings, paragraphs, list items and tablesCheap once parsing works. Our suggested default
SemanticSplit where similarity between sentences dropsQu et al. found gains were inconsistent, and didn't justify the cost
ContextualAn LLM adds a sentence of context to each chunkAnthropic: failures fell from 5.7% to 3.7%, at $1.02 per million document tokens
Late chunkingEmbed the whole document, then pool tokens per chunkAbout 3% better than naive chunking on four test sets, with no LLM call
Small to bigSearch small pieces, then send their surrounding sectionPrecise matching with enough context. In ARAGOG, sentence window retrieval had the best precision

What the evidence says

  • The choice does matter: in one 450-setup study, 20 chunking setups spanned 0.126 in NDCG@10, a 36% relative gap.
  • Semantic chunking isn't a free win: Qu et al. saw it help mainly on artificial documents stitched from different topics. On real ones, fixed size often did as well or better.
  • Overlap isn't magic: Qu et al. saw little from a 1-sentence overlap, and a 16-token overlap showed no clear advantage in the late chunking paper. It can still rescue an answer cut at a boundary, so test it rather than assume it.
  • Chunking still beats no chunking: even long-context embedding models did better on chunks in most tests, with a 24% average gain at 512 tokens.

Pages as images

ColPali embeds page images directly and skips OCR. In one benchmark on financial and technical reports, page-image retrieval beat all the text pipelines. Single-vector page images were also cheap, while the multi-vector version used over 5 GB and took over 20 seconds per query. These documents are visually rich, so don't assume the same result for plain text.

The maths

    Simple arithmetic for fixed-size chunks with overlap. Structure-aware chunks vary in size, so use the average.

    In practice

    In practice

    • Look at your chunks: print a sample of real chunks, especially tables, before tuning anything. Most parsing bugs are obvious once you see them.
    • Keep metadata: store the source, section path, page, date and permissions with each chunk. You need them for filters and citations.
    • Test on your own questions: compare two or three chunk sizes on labelled questions, measuring recall and answer quality, because the best size varies by corpus.
    • Test in the full pipeline: a chunker that wins alone may not win alongside your re-ranker.
    • Re-indexing costs money: changing chunks means re-embedding. See Keeping an Index Current.

    The answer that sounds right

    "We just need a smarter chunker"

    It sounds right, since chunking is the obvious knob. But if parsing has already scrambled a table, no chunker can restore it. Studies also found that semantic chunking often doesn't beat fixed size. Look at your real chunks first, because the problem is usually visible.

    Say it

    Your RAG misreads tables and splits answers in half. How do you parse and chunk documents?

    Say your answer out loud

    Aim for 30 seconds. It feels different from reading it, and that's the point.

    Tick what you covered

    The interviewer pushes further

    Follow-up 1 of 3

    What's the difference between contextual chunking and late chunking?

    Answer it out loud, then check.

    Contextual chunking uses an LLM to write context, then adds it to the chunk's text, so BM25 benefits too, at a cost per chunk. Late chunking embeds the whole document, then pools token vectors per chunk, with no LLM call. It needs a long-context embedding model, and only helps vector search.

    Follow-up 2 of 3

    Should you always use overlap?

    Answer it out loud, then check.

    Not by default. Studies have found little average gain, though it can rescue an answer cut at a boundary. It also adds chunks and duplicates in results. Test it on your own questions, and prefer splitting on real boundaries.

    Follow-up 3 of 3

    A question's answer spans two sections. What do you do?

    Answer it out loud, then check.

    First check whether the chunks hold together. Try retrieving small pieces and sending their surrounding section, or merging neighbouring chunks at answer time. If the question really needs several facts, query decomposition or a second retrieval step may be the real fix.

    I'd start with parsing, because if a table loses its headers, no retriever can fix it. I'd keep tables whole, preserve headings as metadata, and look at real chunks by hand. For chunking, I'd split on document structure first, then test two or three sizes on labelled questions. Evidence for semantic chunking is weak: one study found its gains inconsistent and not worth the cost. If chunks don't make sense alone, I'd add context, either with an LLM-written sentence, as in contextual retrieval, or with late chunking, which embeds the whole document before pooling each chunk. For visual documents like financial reports, I'd consider embedding page images instead.

    Key terms and sources

    Why parsing often matters more than chunk size, keeping tables and structure intact, what the evidence says about semantic chunking and overlap, contextual and late chunking, and page images.

    Sources

    Explanations are ours, built from these sources. Lab numbers are illustrative unless a lab says otherwise. Most recent papers are preprints.