What is self-attention, and how does it work in Transformers?

In this question, we will learn what self-attention is and how it works inside Transformers, step by step. We will show the actual math, a concrete numeric example, and the practical tradeoffs so you can explain both intuition and mechanics in an interview.

We will cover the following:

• The direct answer
• The intuition
• How it actually works with a worked example
• Multi-head attention and positional signals
• Tradeoffs and failure modes
• Questions the interviewer might ask

Self-attention is a mechanism where each token computes attention weights to all tokens using Queries, Keys and Values, then uses those weights to form a context-aware representation. In Transformers this is implemented as scaled dot-product attention and extended with multiple parallel heads; the result is a set of outputs where each position contains information aggregated from the whole sequence.

The intuition (an analogy that makes it click)

Think of a classroom where each student has notes. When one student wants to answer a question, they look at cues from the other students to decide whose notes are most relevant. The Query is the question the student asks, the Keys are short summaries of what each student knows, and the Values are the detailed notes. The student computes which classmates to listen to, weights their input, and writes an answer that mixes those notes.

That is what self-attention does for tokens. Each token asks a query, checks keys of all tokens, and forms a context by mixing the values according to computed weights.

How it actually works (the real mechanics, with one concrete worked example)

At a high level for a single attention head we compute:

Here $Q$, $K$, and $V$ are the matrices of Queries, Keys and Values for all tokens, and $d_k$ is the dimensionality of each key vector. The scaling by $\sqrt{d_k}$ keeps dot products numerically stable.

Worked numeric example. Suppose sequence length $n=3$ and each $V$ is two dimensional. After projection we compute the scaled dot scores for one Query against all Keys and get raw scores $[2,1,0]$ after scaling. Apply softmax to get attention weights.

Now suppose the Values are:

The output for that Query is the weighted sum along each dimension:

• First component $=0.665\cdot 1 + 0.245\cdot 0 + 0.090\cdot 1 = 0.755$
• Second component $=0.665\cdot 0 + 0.245\cdot 1 + 0.090\cdot 1 = 0.335$

So the output vector is approximately $[0.755,0.335]$. This output blends token information according to those attention weights.

Multi-head attention and positional signals

Multi-head attention runs several independent attention computations in parallel. Each head uses different linear projections $Q_i,K_i,V_i$ and produces an output. The outputs are concatenated and linearly projected to the model dimension. This lets the model represent different types of relationships at the same token position.

Key points:

• Each head has dimension $d_k$; if model dim is $d_{model}$ and we use $h$ heads, typically $h\cdot d_k=d_{model}$.
• Heads can focus on local syntax, long-range coreference, or other patterns simultaneously.
• Transformers add positional encodings because attention alone is permutation invariant; positional signals tell the model token order or relative distance.

Complexity and practical comparison

Self-attention gives global receptive field in one layer, which is a major strength. The tradeoff is compute and memory growth with sequence length.

Here $n$ is sequence length and $d$ is model dimension. Self-attention is highly parallelizable on modern hardware but becomes costly for very long $n$.

Tradeoffs and failure modes

Self-attention strengths:

• Fast to train because of parallelism.
• Directly models long-range interactions in few layers.

Weaknesses and common failure modes:

• Quadratic memory and compute for long sequences can be prohibitive.
• Without clear positional signals or enough heads, the model can miss order-dependent patterns.
• Softmax can produce overly diffuse or overly peaky distributions depending on scale and training.

Questions the interviewer might ask

Some follow-up questions you might get:

Why scale by $\sqrt{d_k}$? Scaling prevents the dot product magnitudes from growing with $d_k$, which otherwise would push softmax into very small gradients. The $\sqrt{d_k}$ term keeps learning stable.

What does multi-head add beyond increasing dimension? Multi-head allows the model to attend to different subspaces and patterns simultaneously. Separate heads can specialize to syntax, long-range links, or entity alignment.

How do positional encodings work with attention? Positional encodings provide tokens with position-dependent vectors that are added or combined with embeddings so that attention can use order information when computing similarity.

How would you reduce memory for very long sequences? Options include sparse attention, windowed or block attention, low-rank approximations, or reversible layers. Each choice trades accuracy and ease of training.

What happens if the softmax is uniform? A uniform softmax means the Query could not distinguish Keys; the output becomes a simple average of Values. That may lose discriminative context.

How are Q, K, V computed from the token embeddings? They are linear projections: $Q=XW_Q$, $K=XW_K$, $V=XW_V$ where $W_Q,W_K,W_V$ are learned matrices. These projections let the model shape the similarity measure.

Some things to note:

• Attention matrices are dense unless explicitly sparsified.
• Position signals are necessary for order-sensitive tasks.

What the interviewer is really testing

They want to know you can connect intuition to the actual computations and tradeoffs. Specifically they test whether you can explain $Q,K,V$ and the scaled dot-product formula, show a concrete numeric flow, and discuss complexity and practical mitigations. Clear answers show both conceptual understanding and awareness of engineering limits.