Explain the architecture of a basic RAG system.

In this question, we will learn the architecture of a basic RAG system and why each component matters. We will explain how queries flow through encoders and indices, how retrieved context is combined with a generator, and what design choices change behavior.

We will cover the following:

• System components and overall flow
• The intuition that makes it click
• How it actually works with a worked example
• Practical variations and when to use them
• Tradeoffs and failure modes

Direct answer: A basic RAG system has a query encoder and retriever that find top-$k$ relevant documents from a document store (usually via a vector index or a lexical search), and a reader or generator that conditions on the query plus those retrieved passages to produce the final answer. You must balance retrieval accuracy, context budgeting, and generation grounding to reduce hallucination while keeping latency acceptable.

The intuition (an analogy that makes it click)

Think of the RAG system as a student answering a question in an exam with access to a reference shelf. The retriever is the student scanning the shelf for the top few relevant books. The generator is the student who writes the answer while consulting the open pages. If the student grabs irrelevant books, the answer may be confident but wrong. If the student reads only tiny snippets they may miss crucial facts.

How it actually works

At a high level the pipeline has these components:

1. Document store: raw texts split into passages.
2. Indexing: embeddings for each passage if dense retrieval, or an inverted index for sparse retrieval.
3. Query encoder: converts the query into the same space as the passages.
4. Retriever: finds top-$k$ passages using similarity search.
5. Optional reranker: reorders or filters retrieved passages.
6. Reader / generator: conditions on the query and passages and produces the final output.

A small worked example. Suppose we have $N=10{,}000$ passages and an embedding dimension $d=768$. We encode the query and compute similarity with candidate passage embeddings, returning $k=3$.

We typically compute similarity using cosine score. Display form:

A simple retrieval step under a naive scan costs $O(Nd)$ per query. In practice we use approximate nearest neighbor search to reduce latency.

Example retrieval table for a single query:

After retrieval the generator receives the query and concatenated passages or attends over them. Two common fusion strategies:

• Retriever then generate by concatenation: the generator sees "[query] + passage1 + passage2 + ...".
• Fusion-in-Decoder: the decoder attends to encoder outputs for each passage, allowing cross-passage attention in the decoder.

Concatenation is simple but hits token limits sooner. Fusion-in-Decoder uses more compute but often gives better grounding because the model can weigh each passage per output token.

Variations and practical choices

Dense versus sparse retrieval

Passage size and overlap

Smaller passages reduce noise and increase precision for short answers, but increase index size and may break context. Overlapping windows mitigate context loss at the cost of more storage.

Reranking and filtering

A lightweight cross-encoder reranker that scores retrieved passages with an expensive model can boost precision. Typical pattern: retrieve $k=100$ cheaply, rerank to top $k'=5$, then pass them to the generator.

Tradeoffs and failure modes

Retrieval quality is central. If the retriever returns irrelevant documents the generator can confidently produce incorrect information. Increasing $k$ raises recall but increases latency and token budget usage. Reranking reduces hallucination but adds cost. Updating the index in streaming settings requires either incremental index support or periodic rebuilds.

Questions the interviewer might ask

Some follow-up questions you might get:

Why choose dense over sparse retrieval? Dense retrieval helps with semantic matches and paraphrase resilience. Choose sparse when you need cheap updates or exact lexical matches.

How do you prevent the generator from hallucinating? Improve retrieval precision, use rerankers, limit generator freedom with constrained decoding, and post-check answers against retrieved passages.

How would you scale to millions of documents? Use an ANN index like HNSW or FAISS with sharding and offline batching. Monitor index recall and rerank more candidates when needed.

How do you handle long documents? Chunk documents into passages with overlap. Store metadata so retrieved passages can be traced back to original documents.

What are latency vs quality knobs you would tune? Reduce $k$, use smaller models for reranking, lower embedding dimension, or tune ANN recall settings. Each reduces cost or latency at some quality expense.

How do you keep the index up to date? Use an index that supports incremental adds and deletes, or perform scheduled incremental rebuilds. For high update rates prefer systems designed for streaming inserts.

Some things to note:

• Reranking is a high-ROI step for precision at modest cost.
• Passage design and split strategy affect both recall and generator grounding.

What the interviewer is really testing

They want to see that you understand both components and their interaction: how retrieval shapes what the generator can reliably say, and how design choices trade off latency, cost, and factuality. They also want practical awareness: indexing and update strategies, reranking, and how to measure retrieval quality in service of safer generation.