How do you choose an embedding model for your RAG system?

In this question, we will learn how to choose an embedding model for your RAG system and what tradeoffs matter when you move from prototypes to production. We will keep the focus on practical checks you can run, simple math for resource planning, and testers you can present to an interviewer.

We will cover the following:

• The intuition
• How it actually works
• Evaluation and deployment considerations
• Tradeoffs and failure modes
• Questions the interviewer might ask

Direct answer: pick the smallest embedding model that meets your retrieval and downstream quality requirements within your latency and cost budget, and validate that choice with realistic retrieval and end-to-end generation tests. Fine tune or swap to a higher dimensional or domain-adapted model only when empirical metrics show material gains that justify extra cost.

The intuition (an analogy that makes it click)

Think of embeddings as a cataloging style for a library. A high dimensional embedding is like a detailed catalog entry with many fields. It can capture subtle distinctions, but it costs more to store and search. A smaller embedding is like a short catalog card. It is cheaper and faster but can miss fine-grained differences. You want the card that is detailed enough for your patrons to find what they need without bankrupting the library.

How it actually works (the real mechanics)

Key properties to inspect for any embedding model:

• dimension $d$ which controls expressivity and storage cost
• encoder architecture and training data, which affect domain fit
• distance metric compatibility, typically cosine or inner product
• latency and throughput for batch or online embedding

Concrete worked example. Suppose you are choosing between three candidates: a small general model ($d=384$), a mid model ($d=768$), and a large model ($d=1536$). You expect a corpus of 1 million passages and you store float32 vectors.

Display storage math:

$\text{bytes per vector} \approx 4d$

For the three models we get this table:

Next, compare retrieval quality and cost. You run a held-out retrieval evaluation (recall@k and MRR) and measure embed latency. Results might look like this:

From these numbers we see diminishing returns in moving up. If your downstream generator needs high precision, the mid or large models may be justified. If generator can re-rank or if you can use hybrid search with BM25, the small model may be enough.

Evaluation and deployment considerations

Evaluation plan you can present to an interviewer:

1. Offline retrieval test with realistic queries and ground truth passage labels. Measure recall@k and mean reciprocal rank $\text{MRR}$.
2. Downstream test: run full RAG generation for a sample of queries and evaluate answer correctness or human preference.
3. Latency and cost profiling: measure embedding throughput, index memory, and average end-to-end latency under expected QPS.
4. Robustness checks: test language and domain shifts, and adversarial or OOV content.

When reporting results, show both intrinsic metrics and end-to-end human-rated quality. A small improvement in retrieval metric does not always translate to better final answers.

When to fine-tune or train your own

Fine-tune or train when:

• Your domain uses specialized vocabulary that off-the-shelf models miss.
• You need multilingual coverage not provided by base models.
• You have labeled pairs or hard negatives to improve retrieval for your tasks.

Otherwise, prefer off-the-shelf models and focus effort on better negatives, re-ranking, and index design.

Tradeoffs and failure modes

Major tradeoffs:

• Accuracy versus cost: larger $d$ can improve quality but increases storage, memory, and compute.
• Latency versus throughput: larger encoders add per-request latency, affecting online systems.
• Index complexity: higher $d$ makes ANN indexes larger and sometimes less accurate unless tuned.

Questions the interviewer might ask

Some follow-up questions you might get:

Why not always use the largest embedding available? You can, but cost, latency, and diminishing returns matter. Large embeddings increase storage and slow down both indexing and query times. Empirical gains must justify those costs.

How do you pick the number of nearest neighbors $k$ for retrieval? Start with $k$ that balances recall and downstream re-ranking capacity. Typical ranges are $k=10$ to $k=100$. Larger $k$ raises recall at the expense of re-ranking cost and latency.

What distance metric should we use? Most embeddings work with cosine similarity or inner product. Choose the metric the model was trained with. If you mix models, standardize by normalizing vectors and using cosine.

How do you handle multilingual or domain-specific data? Prefer models pretrained on your target languages or fine-tune with in-domain examples. If you cannot fine-tune, consider separate indices per language or domain.

When would you retrain embeddings in production? Retrain when you see systematic drops in retrieval metrics, after major content changes, or when you add a new domain that was not covered in initial tests.

Some things to note:

• Measure end-to-end quality, not just embedding similarity scores.
• Factor index tuning and ANN parameters into your choice of $d$.

What the interviewer is really testing

They want to know you can balance multiple real constraints: retrieval quality, storage and compute budget, latency, and operational complexity. They also want to see a practical evaluation plan that uses both intrinsic retrieval metrics and end-to-end tests, plus an awareness of when fine-tuning is warranted versus using off-the-shelf models.