How does a Vector Database work?

In this question, we will learn how a vector database stores and searches embeddings so you can retrieve items by semantic similarity instead of exact keys. We will walk through embeddings, indices, distance metrics, a concrete worked example, and the practical tradeoffs you will discuss in an interview.

We will cover the following:

• The direct answer
• The intuition
• How it actually works
• Index types and comparison
• Tradeoffs and failure modes
• Questions the interviewer might ask

Answer: A vector database stores numerical embeddings and builds specialized indexes to perform nearest neighbor search quickly. It computes similarity using metrics like cosine or Euclidean distance, and uses approximate indexing algorithms such as HNSW, IVF, or LSH to reduce query cost from $O(n)$ to sublinear behavior while accepting some recall loss. Results are often re-ranked and filtered by metadata to deliver precise answers.

The intuition (an analogy that makes it click)

Think of a library where books are not organized by title or author but by the ideas inside each book. Each book gets a coordinate in an abstract idea-space. A vector database groups nearby books so you can run to the right shelf cluster instead of inspecting every book. The index is the set of labeled shelves and shortcuts that get you to promising clusters quickly.

How it actually works (the real mechanics, with one concrete worked example)

Step 1: Produce embeddings. Each document or item becomes a vector in $d$ dimensions, for example $d=768$ for many transformer embeddings.

Step 2: Choose a distance metric. Common choices are cosine similarity and Euclidean distance. Cosine similarity between two vectors $a$ and $b$ is

$\text{cos}(a,b)=\frac{a\cdot b}{\|a\|\,\|b\|}$

Step 3: Build an index. Instead of scanning $n$ vectors, the index organizes vectors into a graph or inverted structure. Popular approaches include HNSW (graph-based), IVF (inverted file), and LSH (hashing).

Worked example: two 3-d vectors.

Let $a=(1,2,3)$ and $b=(2,0,1)$. The dot product is

$a\cdot b=1\cdot 2 + 2\cdot 0 + 3\cdot 1 = 5$

Their norms are $\|a\|=\sqrt{1^2+2^2+3^2}=\sqrt{14}$ and $\|b\|=\sqrt{2^2+0^2+1^2}=\sqrt{5}$. Cosine similarity is

$\text{cos}(a,b)=\frac{5}{\sqrt{14}\,\sqrt{5}}\approx 0.597\,.$

This numeric similarity guides which items are returned for a semantic query.

Index performance comparison example. Suppose we have $n=1{,}000{,}000$ vectors and query throughput and recall differ by index type. Typical behavior can look like this:

Each index has tunable knobs. For HNSW you tune graph connectivity, for IVF you tune the number of centroids and product quantizer precision.

Index types and when to use them

HNSW is a hierarchical small-world graph. It usually gives the best latency and recall balance and is a strong default for many embedding sizes. IVF groups vectors by centroids and then searches a subset of clusters. It is efficient when you can quantize vectors and accept some recall loss. LSH hashes vectors to buckets and is simple and parallelizable, but typically gives lower recall for complex embeddings.

When choosing, consider: query latency requirements, dataset size $n$, embedding dimension $d$, memory budget, and recall target.

Re-ranking, filters, and metadata

A vector database often returns a candidate list via the index and then re-ranks those candidates using exact distances or an application-specific scorer. Metadata filters let you restrict the search to relevant subsets, for example by date, tenant id, or language. This two-phase approach keeps latency low while allowing precise final results.

Tradeoffs and failure modes

A key tradeoff is between recall and latency. More aggressive approximation reduces latency and memory, but increases the chance of missing relevant neighbors. High dimensionality can also hurt performance: as $d$ grows, distances can concentrate and indices may degrade.

Questions the interviewer might ask

Some follow-up questions you might get:

Why not just use a relational database for nearest neighbor? Relational databases can store vectors but they are not optimized for high dimensional nearest neighbor search. They would often require full scans and lack specialized indices like HNSW.

How does HNSW achieve sublinear search? HNSW builds a multi-layer graph where upper layers connect distant regions and lower layers provide fine-grained neighbors. Greedy search on upper layers routes quickly to a local region and then the lower layers refine results.

What is product quantization and when do you use it? Product quantization compresses vectors by splitting them into subspaces and quantizing each subspace. Use it when memory is limited and you can accept some loss in distance precision to store more vectors in RAM.

How do you tune recall versus latency in practice? Adjust search parameters such as the number of visited nodes in HNSW or the number of clusters searched in IVF. Measure recall on a held-out query set and choose a point on the latency-recall curve that meets your SLA.

How do you handle updates and deletes? Some indexes support incremental inserts quickly, but deletes are often lazy markers or require rebuilds. Plan capacity, use append-only patterns, or schedule periodic re-indexing for heavy churn.

Some things to note:

• Always evaluate with realistic queries and a labeled ground truth for recall metrics.
• Combine vector search with metadata filters to reduce candidate pools and improve precision.

What the interviewer is really testing

They want to see that you understand embeddings, similarity metrics, and the indexing algorithms that make nearest neighbor search practical at scale. They also want you to reason about tradeoffs: why one index might be chosen over another based on latency, memory, accuracy, and operational characteristics. Demonstrating a concrete example and discussing failure modes shows you can apply the concepts in real systems.