Compare fixed-size chunking, semantic chunking, and recursive chunking.

In this question, we will learn how fixed-size chunking, semantic chunking, and recursive chunking differ and why those differences matter in a RAG pipeline. We will keep the view practical so you can explain when each strategy is a good choice and how it affects embeddings and downstream generation.

We will cover the following:

• The direct answer
• The intuition
• How it actually works with a worked example
• When to use each and implementation notes
• Tradeoffs, failure modes, and interview follow-ups

Direct answer: Fixed-size chunking is simple and predictable but may split meaning across chunks; semantic chunking preserves natural language units at the cost of variable sizes and slightly more work; recursive chunking is a hybrid that guarantees fit and context coherence by splitting and refining with overlap, increasing index size and retrieval cost.

The intuition

Think of documents as loaves of bread and the model context window as a sandwich wrapper. Fixed-size chunking slices evenly, which is efficient if uniform slices are fine. Semantic chunking slices along natural breaks like sentence or chapter boundaries so you rarely get a slice that mixes two different topics. Recursive chunking first slices roughly and then trims and overlaps pieces so every sandwich gets full, coherent content without spilling out of the wrapper.

How it actually works

Fixed-size chunking

• Choose a chunk size in tokens or bytes, call it $c$.
• For a document of $T$ tokens you produce $m=\lceil T/c\rceil$ chunks.
• You usually add a small fixed overlap $o$ to preserve context between adjacent chunks.

Semantic chunking

• Use language structure: paragraphs, sentences, headings, or punctuation to define boundaries.
• Chunk sizes vary; some documents may produce many small chunks and others fewer large ones.

Recursive chunking

• First apply a coarse rule, for example split on headings, then for any chunk with size $>c$ split it recursively using a finer rule (sentences, clauses) and add controlled overlap.
• Guarantees every final piece is $\le c$ tokens while trying to keep natural boundaries.

Worked example

Suppose we have a 3,200 token document and a model context capacity per chunk of $c=512$ tokens with desired overlap $o=50$ tokens.

Search cost and embedding work scale with the number of chunks $m$. For vector search, a single query compares against $m$ vectors so cost is roughly $O(m)$. If embedding dimension is $d$, a single dot product is $O(d)$ so a query cost is $O(md)$. Smaller, more coherent chunks usually increase relevance per vector but may increase $m$.

When to use each and implementation notes

Fixed-size is good when:

• Documents are consistent and you want predictable index size.
• You need simple, fast preprocessing and have limited engineering time.
• The application tolerates occasional split sentences.

Semantic is best when:

• Document structure is meaningful, for example manuals, contracts, or Q and A where headings and paragraphs carry intent.
• You want fewer hallucinations from partial sentences because chunks align with semantic units.

Recursive fits when:

• You must guarantee every chunk fits the model context and also preserve semantic units as much as possible.
• Documents have irregular structure where some sections are extremely long.

Implementation notes

• Tokenization matters. Use the same tokenizer you use for model context sizing so $c$ reflects actual tokens.
• Overlap reduces boundary loss. Typical overlap $o$ is 20 to 200 tokens depending on document density.
• Index size grows with overlap and with recursive splits, so budget storage and retrieval accordingly.

Tradeoffs and failure modes

• Retrieval precision versus index size: smaller, semantic chunks often improve precision but increase $m$ and search time.
• Boundary fragmentation: fixed-size splits can separate entities and cause relevance loss or hallucination.
• Complexity and correctness: recursive chunking is more code and potential edge cases.

Questions the interviewer might ask

Some follow-up questions you might get:

How does overlap affect retrieval accuracy? Overlap increases the chance that a relevant phrase appears wholly inside at least one chunk, raising recall. It also increases index size and can lead to duplicate results unless de-duplicated at retrieval.

How do you pick the chunk size $c$? Pick $c$ based on the model's effective context window and average response length. Use the same tokenizer to measure tokens and aim to leave headroom for prompt and instruction tokens.

What metrics would you use to compare these strategies? Use retrieval precision at k, end-to-end task accuracy, and latency. Also track index size and average number of chunks per document.

When might fixed-size outperform semantic? When documents are short, uniformly structured, or when the cost of parsing semantics outweighs benefits. Fixed-size can also be faster for streaming preprocessing.

How do you handle very large documents like books? Use a hierarchical approach: chapter-level indexes for coarse retrieval, then chunk-level indexes for fine retrieval. Recursive chunking helps keep final pieces coherent for the model.

Some things to note:

• Always measure using your actual task and prompts. Numbers vary by domain.
• Tokenizer mismatch is a common source of errors; use the model's tokenizer for sizing.

What the interviewer is really testing

They want to see you reason about practical tradeoffs: retrieval accuracy, index and compute cost, and downstream hallucination risk. They also expect awareness of implementation details like tokenization, overlap, and how chunking interacts with vector search costs and model context windows. Show that you can choose a strategy based on constraints, not just preference.