What are chunking strategies, and how do you choose the right chunk size?

In this question, we will learn what chunking strategies are for RAG systems and how to choose the right chunk size for your retrieval and generation pipeline. We will keep the focus practical: tokens, overlaps, retrieval quality, and model context limits.

We will cover the following:

• The intuition
• How it actually works
• Choosing chunk size in practice
• Chunk overlap and indexing costs
• Tradeoffs and failure modes
• Questions the interviewer might ask

Direct answer: choose a chunk size that balances retrieval precision with the model context window and cost , start with token-sized chunks roughly one quarter to one half of your model context, add 10 to 30 percent overlap, and evaluate end-to-end on task performance. Keep chunks consistent and use tokenization aligned with your embedding model.

The intuition

Think of chunking as deciding how large each "searchable card" of your document should be. If the cards are tiny, retrieval finds very focused facts but you need many cards and the retriever might return many irrelevant cards. If cards are giant, each returned card likely contains the answer but you waste model context on irrelevant text. We want a sweet spot that returns concise relevant context without exceeding the model input budget.

How it actually works

We usually measure document length in tokens compatible with the model and embedding tokenizer. Let $L$ be the document token length, $c$ the chunk size in tokens, and $n$ the number of chunks:

If you add overlap $o$ as a fraction of $c$, each step moves by $c(1-o)$ tokens. For example, with $L=12000$ tokens, $c=1000$ tokens, and $o=0.2$ (20 percent):

• chunk count approximately

A worked example comparing three choices:

In practice, embeddings and retriever scoring interact with chunk size. Smaller chunks often produce higher cosine similarity for precise spans. Larger chunks can give stronger semantic matches when the retriever struggles with isolated phrases.

Choosing chunk size in practice

1. Align with the model context window. If your model has a $W$ token window and you expect the answer plus prompt to need $A$ tokens, keep the total retrieved context under $W-A$. A simple rule is $c \times k \le W-A$ where $k$ is the number of top retrieved chunks you will pass in.

2. Start with a heuristic: for a 4k token model, try $c$ between 512 and 1500 tokens. For an 8k model, 1k to 3k is reasonable. These ranges map to keeping $k=3$ or $k=5$ candidate chunks within budget.

3. Tokenize with the same tokenizer used for embeddings. Do not chunk by characters or words unless you control tokenization mapping.

4. Measure end-to-end: evaluate retrieval precision@k, downstream QA F1 or exact match, and generation quality. Trade off index size and latency.

Chunk overlap and indexing costs

Overlap reduces boundary misses when answers straddle chunk edges. Common overlap fractions are 10 to 30 percent. Too much overlap increases redundancy and index size. Compute index size roughly proportional to $n$; adding overlap increases $n$ by roughly $1/(1-o)$.

Example: with $c=1000$ and $o=0.2$, effective step is 800 so chunk count grows by factor 1.25 compared to no overlap.

Tradeoffs and failure modes

• Retrieval precision versus context: small chunks help precise matching, large chunks preserve narrative context.
• Cost and latency: more chunks means larger indexes, slower nearest neighbor search, and higher embedding compute when building the index.
• Boundary misses: no overlap risks splitting answers across chunks and losing them in retrieval.

Questions the interviewer might ask

Some follow-up questions you might get:

Why prefer token-based chunking over character or sentence counts? Tokenization matches the embedding and model input; tokens represent how the model consumes text, so chunk sizes in tokens are most consistent across languages and encodings.

How do you choose the number of retrieved chunks $k$ to pass to the model? Pick $k$ so that $k \times c$ plus prompt and expected answer fits in the model window $W$. Increase $k$ if retrieval precision is low, but watch cost and noise.

When is overlap unnecessary? If documents are short, structured with clear per-section headings, or you have robust sentence-level retrieval, you can reduce or remove overlap. For free-form text, some overlap helps.

How does chunking interact with vector index methods like HNSW or Faiss? Index performance is roughly proportional to number of vectors, so smaller chunks increase index size and search latency. HNSW handles large indexes well but cost and memory rise.

What metrics do you use to pick chunk size? Use retrieval precision@k, downstream QA F1 or exact match, generative quality checks, index size, and per-query latency. Prefer the metric closest to your product goal.

Some things to note:

• Use the embedding tokenizer and compute chunk sizes in tokens.
• Tune overlap and chunk size together, not independently.
• Validate on realistic queries and edge cases where answers cross chunk boundaries.

What the interviewer is really testing

They want to see you balance practical engineering constraints with retrieval quality. You should show understanding of tokenization, the model context window, index cost, and how overlap and chunk size affect downstream accuracy. A good answer explains heuristics, gives simple formulas, and stresses end-to-end validation.