How do embedding models convert text to vectors?

In this question, we will learn how embedding models convert text to vectors and why each stage matters for retrieval and semantic similarity.

We will cover the following:

• The direct answer
• The intuition
• How it actually works with a worked example
• Choosing pooling and similarity metrics
• Tradeoffs and failure modes
• Questions the interviewer might ask
• What the interviewer is really testing

Direct answer: Embedding models map text into fixed-length numerical vectors by tokenizing input, encoding tokens with a neural encoder, pooling token-level outputs into a single vector, and often normalizing that vector so similarity corresponds to geometric proximity. This pipeline preserves semantic relationships when the encoder is trained to place related texts near each other in vector space, and we use metrics like cosine similarity or dot product to retrieve nearest neighbors.

The intuition (an analogy that makes it click)

Think of each sentence as a location on a conceptual map. Tokenization breaks the sentence into landmarks. The encoder measures how each landmark relates to many axes of meaning. Pooling collapses those per-landmark signals into one set of coordinates. Normalization then rescales the map so distance and angle correspond to semantic closeness. When two sentences are about the same idea, their coordinates land near each other.

How it actually works (the real mechanics, with one concrete worked example)

Step by step:

1. Tokenize: convert text to token ids using a vocabulary and tokenizer. Tokens preserve subword structure.
2. Encode: pass token ids through the model (for example a transformer) to get token embeddings of dimension $d$ for each token.
3. Pool: aggregate token embeddings into a single $d$-dimensional vector. Common choices are mean pooling or using a special CLS token.
4. Normalize: often apply L2 normalization so vectors lie on the unit hypersphere. That makes cosine similarity equivalent to dot product.
5. Index: store vectors in a vector database and search with an ANN algorithm.

Concrete toy example. Suppose an encoder outputs 3-dimensional token vectors and we use mean pooling. Two short texts produce token outputs that we pool into sentence vectors:

We then L2 normalize each vector. The cosine similarity becomes the dot product of the normalized vectors. Display the cosine formula:

$\cos(\theta) = \frac{u \cdot v}{\|u\|\,\|v\|}$

If the normalized vectors are very close, the cosine is near 1 and the texts are semantically similar. In our toy numbers, the dot product of the pooled vectors is high, showing semantic closeness.

Choosing pooling and similarity metrics

Pooling changes what the vector emphasizes. Mean pooling tends to capture the average content. CLS pooling can capture a representation the model was trained to place in that token, which may better serve tasks if the model was trained with that convention. Max pooling highlights the most salient features.

Similarity metrics behave differently:

If you normalize to unit length, cosine and dot product are equivalent.

When to normalize, and when not to

Normalize when you want similarity to depend on direction only. Keep norms if you want magnitude to encode confidence or importance, for example when a model's training assigns longer vectors to more informative inputs.

Batch encoding considerations: encode in batches for GPU efficiency, pad tokens to a max length, and be consistent about preprocessing. If you change tokenization or model, embeddings before and after are not comparable.

Tradeoffs and failure modes

• Larger embedding dimension $d$ can capture more nuance but costs more storage and slower nearest neighbor queries.
• A mismatch in preprocessing between index time and query time causes retrieval failures. Tokenizer, lowercasing, and punctuation handling must match.
• Models trained on a different domain can place similar-domain texts far apart.

Questions the interviewer might ask

Some follow-up questions you might get:

Why do we sometimes L2 normalize embeddings? L2 normalization ensures vectors lie on a unit sphere and makes cosine similarity equivalent to dot product. That simplifies nearest neighbor search and reduces the impact of vector magnitude.

When would you use dot product instead of cosine similarity? Use dot product if vector norms carry meaningful information, such as confidence or document length signals, or when the retrieval system expects raw model logits.

How does tokenization affect embeddings? Tokenization determines the atomic units the encoder sees. Different tokenizers change the token sequence and can change pooled representations, so consistency is critical between indexing and querying.

What is the impact of embedding dimension $d$? Higher $d$ can represent more features and improve accuracy up to a point, but increases storage and slows ANN search. There are diminishing returns beyond a practical range like 256 to 1536 depending on model and task.

How do you evaluate embedding quality for retrieval? Use downstream metrics like recall@k, mean reciprocal rank, or precision at k on labeled query-candidate pairs. Also inspect nearest neighbors qualitatively.

Some things to note:

• Keep preprocessing identical for index and query time.
• When swapping models, re-index your database.
• Small dimension reduction techniques like PCA or quantization can save storage with some accuracy loss.

What the interviewer is really testing

They want to see that you can explain the full pipeline from raw text to searchable vectors and reason about the effects of design choices like pooling, normalization, dimension, and similarity metric. They also expect practical awareness: batch encoding, consistent preprocessing, and the tradeoffs between accuracy, storage, and latency. Give a concrete example and mention common pitfalls to show you can apply concepts in production.