What are embedding models, and how do they convert text to vectors?

In this question, we will learn what embedding models are and how they convert text to vectors, so you can explain the pipeline and show a concrete similarity calculation. We will keep the explanation practical for retrieval-augmented generation scenarios.

We will cover the following:

• The intuition
• How it actually works
• Embedding choices and comparison
• Step-by-step example: compute cosine similarity
• Tradeoffs and failure modes
• Questions the interviewer might ask
• What the interviewer is really testing

Direct answer: Embedding models convert text to dense numeric vectors by tokenizing input, encoding tokens with a neural network, and pooling token representations into a fixed-size vector, trained so semantic neighbors are close in vector space. They enable fast similarity search and RAG by turning retrieval into nearest neighbor queries in $d$-dimensional space.

The intuition (an analogy that makes it click)

Think of the embedding space as a town map. Each phrase or document becomes a house on that map. Nearby houses share similar features. When you ask for related text, you look for houses within a short walking distance. The embedding model is the cartographer that draws the map from raw sentences so geography matches meaning.

How it actually works (the real mechanics, with one concrete worked example)

At a high level the pipeline is: tokenize the text, pass tokens through an encoder, then pool token outputs to produce an embedding vector of dimension $d$. Common encoders are transformers or convolutional networks.

Steps in more detail:

1. Tokenization: convert text into subword tokens. This maps words into integer ids.
2. Encoding: feed token ids into a neural encoder that produces token-level vectors. If the encoder has output states $h_1,\dots,h_n$, each $h_i$ is in $\mathbb{R}^d$.
3. Pooling: aggregate token vectors into one vector. Common options are the special CLS token, mean pooling, or attention pooling.
4. Normalization: often $L2$ normalize so embeddings lie on the unit sphere.
5. Indexing: store vectors in a nearest neighbor index for retrieval.

Training objectives. A popular choice is a contrastive (InfoNCE style) loss where a query vector $q$ should be close to a positive key $k^+$ and far from negatives $k_i$. One form is

$L = -\log \frac{\exp(q\cdot k^+ / \tau)}{\sum_i \exp(q\cdot k_i / \tau)}$

where $\tau$ is a temperature.

Worked example. Suppose we encode two short texts and get 3-dimensional vectors. The table shows components.

Compute dot product and norms. The dot product is

$v_a\cdot v_b = 0.10\times 0.00 + 0.30\times 0.40 + 0.90\times 0.80 = 0 + 0.12 + 0.72 = 0.84$

The norms are

$\|v_a\| = \sqrt{0.10^2 + 0.30^2 + 0.90^2} = \sqrt{0.01 + 0.09 + 0.81} = \sqrt{0.91} \approx 0.953$

$\|v_b\| = \sqrt{0.00^2 + 0.40^2 + 0.80^2} = \sqrt{0 + 0.16 + 0.64} = \sqrt{0.80} \approx 0.894$

Cosine similarity is

$\text{cosine}(v_a,v_b) = \frac{v_a\cdot v_b}{\|v_a\|\,\|v_b\|} = \frac{0.84}{0.953\times 0.894} \approx \frac{0.84}{0.852} \approx 0.99$

That high similarity means the model placed those texts very close in meaning on the map.

Embedding choices and comparison

There are several axes to choose from when building embeddings for RAG:

• Architecture: transformer encoders are common because they capture context well. Simpler models can be faster but may lose nuance.
• Pooling: CLS can be effective for models trained with it. Mean pooling is robust across architectures.
• Dimension $d$: larger $d$ can encode more subtle distinctions but increases index size and nearest neighbor cost.
• Training objective: contrastive training aligns paired examples; supervised classification can also produce useful vectors but may be less robust to open-domain similarity.

Comparison table for quick decisions:

Step-by-step use in RAG

1. Build or select an embedding model and decide $d$. 2. Encode all documents and normalize vectors. 3. Add vectors to a nearest neighbor index (HNSW, IVF, etc.). 4. For a query, encode and search top-k by cosine or inner product. 5. Use retrieved documents as context for the generator.

Tradeoffs and failure modes

Embedding models are powerful but not perfect. Common failure modes include:

• Anchoring on surface tokens: models sometimes focus on shared phrases rather than deeper meaning.
• Domain mismatch: embeddings trained on web data may misplace specialized jargon.
• Dimensionality limits: small $d$ forces tradeoffs between concepts.

Questions the interviewer might ask

Some follow-up questions you might get:

How does tokenization affect embeddings? Tokenization changes which subword units the encoder sees. Different tokenizers shift how semantic chunks are represented and can change embeddings, especially for rare or compound words.

Why normalize embeddings before indexing? Normalization makes cosine similarity equal to dot product which simplifies indexes and stabilizes similarity scores across lengths.

What is a good way to select negatives for contrastive training? Hard negatives that are similar but not correct help learning. Random negatives work for scale but may be less effective than mined hard negatives.

When should we use cross-encoder re-ranking? Use a cross-encoder when you need higher precision on a small candidate set. It is slower because it scores pairs instead of computing a single vector for each document.

How does embedding dimension influence index choice? Higher $d$ increases vector storage and search cost. Some indexes like product quantization handle large $d$ well, while others accelerate lower-dimensional searches.

How do you evaluate embedding quality? Use retrieval metrics like recall@k, MRR, or downstream task metrics in RAG. Also inspect nearest neighbors qualitatively.

Some things to note:

• Always match preprocessing and normalization between query and indexed vectors.
• Evaluate on domain-specific data to detect mismatch.

What the interviewer is really testing

They want to see you understand the full pipeline from raw text to searchable vectors: tokenization, encoding, pooling, training objectives, and indexing. They also want practical judgment about tradeoffs such as dimension, model cost, and whether to re-rank with a cross-encoder. Clear examples and a simple cosine similarity calculation show you can connect theory to implementation.