Explain cosine similarity, dot product, and Euclidean distance for vector search.

In this question, we will learn the differences between cosine similarity, dot product, and Euclidean distance for vector search and when to pick each one. We will keep math concrete and show a numeric example you can compute at the whiteboard.

We will cover the following:

• The intuition
• How it actually works
• When to use each
• Implementation notes and performance
• Tradeoffs and failure modes
• Questions the interviewer might ask

Direct answer: Dot product captures raw aligned magnitude, cosine similarity captures angular similarity and is invariant to vector length, and Euclidean distance measures absolute gap. Normalize vectors when you want angle-only comparisons, use dot product when magnitude matters and your index supports inner products, and use Euclidean distance when absolute difference is meaningful.

The intuition (an analogy that makes it click)

Think of each vector as an arrow from the origin. The dot product tells you how much one arrow projects onto another and scales with both length and alignment. Cosine similarity looks only at the angle between arrows, like comparing directions of two compasses. Euclidean distance measures the straight-line gap between arrow tips, like the literal walking distance between two points in the room.

How it actually works (the real mechanics, with one concrete worked example)

Start with two vectors $v$ and $w$ in $\mathbb{R}^n$.

Dot product definition: $v\cdot w = \sum_{i=1}^n v_i w_i$

Cosine similarity definition: $\text{cosine}(v,w)=\frac{v\cdot w}{\|v\|\,\|w\|}$

Euclidean distance definition: $\|v-w\|_2=\sqrt{\sum_{i=1}^n (v_i-w_i)^2}$

Worked example. Let $v=[1,2,3]$ and $w=[2,0,1]$.

• Dot product: $v\cdot w=1\cdot2+2\cdot0+3\cdot1=5$.
• Norms: $\|v\|=\sqrt{1+4+9}=\sqrt{14}$ and $\|w\|=\sqrt{4+0+1}=\sqrt{5}$.
• Cosine similarity: $\text{cosine}(v,w)=5/(\sqrt{14}\,\sqrt{5})=5/\sqrt{70}\approx0.597$.
• Euclidean distance: $\|v-w\|_2=\sqrt{(1-2)^2+(2-0)^2+(3-1)^2}=\sqrt{9}=3$.

Compare the three metrics side by side in value and property:

That table shows how the same vectors give different signals. A long version of $v$ would increase the dot product but not the cosine.

When to use each

• Use cosine similarity when you care about direction or relative pattern and want to ignore magnitude. This is common for normalized embeddings from language models.
• Use dot product when vector norms are meaningful and larger magnitudes should rank higher. Some learning pipelines produce embeddings where norm encodes confidence or frequency.
• Use Euclidean distance when absolute coordinate differences matter or when you want geometric proximity. This is common for coordinate-based sensor data or when raw distances are interpretable.

A common practical shortcut is to normalize vectors to unit length and then use inner product search. After normalization, dot product equals cosine similarity, and many indexing libraries optimize inner products efficiently.

Implementation notes and performance

We often build indexes with approximate nearest neighbor libraries. Two practical points:

• If you want cosine similarity, normalize all vectors to unit length and run an inner product index. Many ANN backends allow both inner product and L2 metrics, but normalized inner product costs the same as cosine.
• For Euclidean distance, indexes use squared L2 during search to avoid the square root. Squared distance preserves ranking, since square root is monotonic.

Runtime and space are largely driven by dimension $d$ and dataset size $N$. A rough cost per query with a linear scan is $O(Nd)$ for computing dot products or distances. ANN methods trade some recall for much lower average costs.

Tradeoffs and failure modes

Other tradeoffs:

• Interpretability: cosine gives a bounded similarity in [-1,1] which is easier to threshold; dot product is unbounded.
• Sensitivity: Euclidean responds to absolute differences and can be dominated by a few large coordinate differences in high dimension.
• Index compatibility: some ANN indices only support specific metric types or require tweaks.

Questions the interviewer might ask

Some follow-up questions you might get:

How does normalization change the math and results? If you scale a vector by a positive constant $\alpha$, dot product scales by $\alpha$ and Euclidean distance changes, but cosine stays the same after you normalize both vectors to unit length.

Can cosine and Euclidean give the same ranking? Yes, when all vectors are normalized to unit length, ranking by cosine similarity is equivalent to ranking by Euclidean distance because $\|v-w\|_2^2=\|v\|^2+\|w\|^2-2(v\cdot w)$ and the norms are constant.

What is hubness and why does it matter? Hubness is the tendency in high dimensions for some points to appear as nearest neighbors excessively often. It can distort retrieval and reduce diversity of results.

Why do some libraries prefer inner product search? Inner product is efficient on many hardware platforms and can be used to compute cosine after normalization. It also allows weighted ranking by norm when needed.

When would you use squared L2 instead of L2? Squared L2 is faster because it avoids a square root and preserves ordering, so for nearest neighbor ranking squared L2 is sufficient.

How do you debug metric choice on a dataset? Compare recall and downstream task metrics after switching metrics, inspect top-k results for obvious bias such as dominance by high-norm items, and test with and without normalization.

Some things to note:

• If norms carry semantic information, do not normalize away that signal without reason.
• Many prebuilt embedding pipelines normalize before indexing; confirm the pipeline behavior.

What the interviewer is really testing

They want to see you reason about scale versus angle, connect formulas to concrete behavior, and pick a metric that matches the data semantics. They also want awareness of practical issues like normalization, index support, and high-dimensional failure modes.