What is Cross Attention in Transformers?

In this question, we will learn what cross attention is, how it differs from self attention, and how it computes context using queries, keys, and values from different sequences.

We will cover the following:

• The intuition
• How it actually works
• When to use cross attention
• Computational cost and optimization
• Tradeoffs and failure modes

Direct answer: Cross attention is the attention mechanism where queries come from one sequence and keys and values come from a different sequence, commonly used in encoder-decoder architectures to let the decoder read encoded context. The operation computes weights by comparing queries to keys with the softmax of scaled dot products and then aggregates values to produce context-aware outputs. It differs from self attention only by the origin of keys and values, and it is central to tasks that combine two streams of information.

The intuition (an analogy that makes it click)

Imagine you are answering a question in a classroom while holding a stack of reference cards. Your current question and partial answer are the queries. The reference cards are the keys and values. You scan the cards with the question in mind, rank which cards are most relevant, and then read the content from the top-ranked cards to form your final answer.

Queries ask, keys say how relevant each reference is, and values supply the content you borrow. Cross attention is simply this process when the questions and the reference stack come from two different places.

How it actually works (the real mechanics, with a concrete worked example)

Mechanically, if queries are $Q$, keys are $K$, and values are $V$, the basic attention output is

$ext{Attention}(Q,K,V)= ext{softmax}\left(\frac{QK^{T}}{\sqrt{d_k}}\right)V$

Here $d_k$ is the dimensionality of keys used for scaling. In an encoder-decoder model the decoder supplies $Q$ and the encoder supplies $K$ and $V$.

Worked numeric example. Suppose you have a single decoder query vector of dimension $2$ and three encoder key/value pairs. Let

$Q=[1,0]$,

$K_1=[1,0],\;K_2=[0,1],\;K_3=[1,1]$,

and corresponding values

$V_1=[2,0],\;V_2=[0,3],\;V_3=[1,1]$.

Compute raw scores by dot product $Q\cdot K_i$ and scale with $\sqrt{d_k}=\sqrt{2}$.

The raw dot products are:

Applying softmax to the scaled scores produces attention weights. Numerically softmax of $[1/\sqrt{2},0,1/\sqrt{2}]$ is roughly $[0.42,0.16,0.42]$.

Multiply weights by values and sum:

$0.42\times[2,0]+0.16\times[0,3]+0.42\times[1,1]=[0.42\times2+0.42\times1,\;0.16\times3+0.42\times1]$

which yields roughly $[1.26,0.9]$. That vector is the context the decoder reads from the encoder for that query.

Comparison to self attention. Here is a compact table showing the origin of Q K V and the main purpose.

When to use cross attention

Use cross attention whenever you must condition one sequence on another. Typical cases include:

• Machine translation with an encoder for the source and a decoder for the target. The decoder queries the encoder for relevant source tokens.
• Multimodal models where text queries image features or vice versa.
• Retrieval-augmented generation where the decoder queries a set of retrieved documents represented as keys and values.

Cross attention provides an explicit, learnable way to read from external information while producing sequence outputs.

Computational cost and optimization

Cross attention costs similar compute to self attention between sequences of size $n_q$ and $n_k$. The dominant cost is the matrix multiply that yields $QK^{T}$ which is $O(n_q n_k d)$ where $d$ is the head dimension. Memory for the full attention matrix is $O(n_q n_k)$. Practical optimizations include:

• Reducing $n_k$ by retrieving a small number of relevant context chunks instead of attending to a huge document.
• Using approximate attention or locality-sensitive hashing when $n_k$ is large.
• Caching projected $K$ and $V$ when the context is static across decoding steps.

Tradeoffs and failure modes

Cross attention is powerful, but it can fail or become expensive. Common failure modes include attention focusing on irrelevant keys, poor alignment between modalities, and out of memory errors when $n_k$ is large. Regularization, better tokenization of context, and retrieval filtering help mitigate these problems.

Questions the interviewer might ask

Some follow-up questions you might get:

How does cross attention differ from self attention? Self attention uses Q K and V from the same sequence so tokens attend over each other. Cross attention separates the query source from the context source so one sequence reads another.

Why divide by the square root of $d_k$? Scaling by $\sqrt{d_k}$ keeps the dot products from growing with dimension and keeps the softmax in a numerically stable, non-saturated regime.

Can cross attention be multiheaded? Yes. You split $Q$, $K$, and $V$ into heads, compute attention per head, then concatenate and project. Multihead attention allows the model to attend to different aspects of the context in parallel.

What happens when context length is huge? Compute and memory grow as $O(n_q n_k)$ which can be prohibitive. Use retrieval to shrink $n_k$, approximate attention, or sparse patterns to control cost.

How do you handle alignment between modalities? Use modality-specific encoders to produce compatible representations, and train cross-modal objectives or alignment losses so keys and queries are comparable.

When might you prefer concatenating sequences and using self attention instead? Concatenation can work when both sequences should interact symmetrically, but it forces $O((n_q+n_k)^2)$ compute and mixes positional encodings. Cross attention keeps a structured read mechanism that can be more efficient and clearer to control.

Some things to note:

• Caching $K$ and $V$ is cheap when context is fixed and saves repeated projection work.
• Cross attention typically sits in the decoder block after self attention over previously generated tokens.

What the interviewer is really testing

They want to see that you understand the data flow: where queries come from, where keys and values come from, and how the softmax of scaled dot products yields weights. They also want practical awareness of compute and memory implications and strategies to mitigate OOM or misalignment when using large or noisy contexts.