How do you handle multi-document and multi-hop questions in RAG?

In this question, we will learn practical ways to handle multi-document and multi-hop questions in RAG and why each choice matters for correctness and cost.

We will cover the following:

• The direct answer
• The intuition
• How it actually works
• Practical patterns and a comparison
• Tradeoffs and failure modes
• Questions the interviewer might ask
• What the interviewer is really testing

You handle multi-document, multi-hop RAG by turning a hard question into a retrieval plan: decompose the hops, run targeted retrieval for each hop (or over-retrieve then rerank), integrate facts with focused generation and citation checks, and iterate until the answer is grounded. This balances recall and precision while controlling cost by limiting per-hop candidates and using rerankers.

The intuition (an analogy that makes it click)

Think of the process as navigating a library to solve a puzzle. You start with a lead from the question, pull a few relevant books, extract the page or paragraph with the clue, then use that clue to search for the next lead. We do not try to read every book. We move hop-by-hop and only keep the promising evidence. That keeps us efficient and focused.

How it actually works (the real mechanics, with one concrete worked example)

High level patterns:

• Hop-by-hop retrieval: convert the question into a sequence of focused subqueries. After each retrieval, extract the intermediate fact and form the next query.
• Over-retrieve and assemble: retrieve a larger set of candidate passages once, then rerank and chain them in the generator.

Worked example

Question: "Which researcher who worked on project X later collaborated with lab Y on method Z?"

Step 1: first-hop query to find project X authors. Step 2: extract author names and affiliated labs from top passages. Step 3: for each candidate author run a second-hop query: "author name lab collaborations method Z". Step 4: verify passes and generate a grounded answer with citations.

A simple cost model for $k$ hops with $m$ retrieved passages per hop: the total number of retrieved passages is

$D_{total} = k \cdot m$

If we apply a reranker at each hop to reduce candidates to $r < m$, later generation sees $k \cdot r$ passages. That reduces generator token cost but adds reranker compute.

Comparison of two strategies

Concrete numbers example: if $k=2$, $m=10$, $r=3$ then $D_{total}=20$, generator sees $6$ passages after rerank.

Practical patterns and implementation tips

1. Focused subquery design

We create subqueries that target the intermediate fact we expect. For example turn "Who collaborated" into "author name collaborated with lab Y method Z" using the extracted author as a slot. Prompt the extractor to return the minimal atomic fact, such as a name and a sentence citation.

2. Use entity and adjacency indices when available

If your vector store or index supports entity linking or fielded metadata, you can follow edges instead of raw text matching. That turns search from fuzzy text matching to a graph traversal. For large document bases this yields much higher precision for the next hop.

3. Reranking and verification

Use a lightweight cross-encoder or a classifier to rerank the top candidates before expensive generation. Then ask the generator to cite supporting passages. If citation confidence is low, trigger another retrieval or conservative failure.

4. Prompt engineering for composition

When composing multiple passages, feed the generator a short plan: list each passage id with its extracted fact, then ask the model to synthesize. That encourages explicit citation and stepwise reasoning.

Tradeoffs and failure modes

Multi-hop RAG trades off recall, latency, and hallucination risk. More aggressive over-retrieval raises recall but increases token cost and hallucination surface. Hop-by-hop reduces irrelevant context but can fail if an intermediate extraction is incorrect.

Common failure modes:

• Missing an intermediate entity because retrieval used a synonym or alias. Solve with alias tables or fuzzy matching.
• Reranker bias: a reranker tuned for surface relevance may demote the correct but less explicit passage.
• Token overload in the generator when too many candidates are fed at once, increasing hallucination.

Questions the interviewer might ask:

Some follow-up questions you might get:

How do you decide between hop-by-hop and over-retrieval? Choose hop-by-hop when the question naturally decomposes and each hop yields a crisp pivot. Choose over-retrieval when the hops are uncertain or when you can parallelize retrieval and reranking.

How do you measure intermediate extraction quality? Track precision and recall for extracted facts against a labeled set, and use confidence scores and human-in-the-loop sampling to calibrate thresholds.

What role do rerankers play and when are they necessary? Rerankers improve precision before generation. Use them when initial retrieval is noisy or when generator cost is high and you need a compact, high-quality context.

How do you prevent hallucinations when composing multiple passages? Require the generator to cite passage ids and only accept claims directly supported by cited text. If the generator invents links, mark the answer low confidence and fetch more evidence.

How does vector store quality affect multi-hop performance? Embedding quality and index granularity strongly affect recall. Better embeddings and smaller chunk sizes tend to help multi-hop retrieval because you can find the exact sentence that contains the pivot.

When should you stop adding hops? Stop when the answer is directly supported by retrieved evidence with sufficient confidence, or when additional hops no longer increase verification score. Use a budgeted hop limit to control latency.

Some things to note:

• Keep chunk size small enough to isolate facts but large enough to preserve local context.
• Monitor per-hop confidence and allow backtracking when confidence drops.

What the interviewer is really testing

They want to see that you can convert a reasoning challenge into a retrieval plan, understand tradeoffs between recall and cost, and design validation steps that limit hallucination. They also expect familiarity with practical tools: rerankers, entity indices, hop limits, and evidence-backed generation.