Explain Prefix Tuning and Prompt Tuning. How are they different from LoRA?

In this question, we will learn what Prefix Tuning and Prompt Tuning do, why they are considered parameter-efficient, and how they differ from LoRA. We will keep the explanation hands-on so you can describe mechanics and pick the right tool in interviews.

We will cover the following:

• The intuition
• How it actually works
• When to use each
• Tradeoffs and failure modes
• Questions the interviewer might ask
• What the interviewer is really testing

Direct answer: Prefix Tuning and Prompt Tuning both attach tunable soft tokens to the model input while keeping the base model frozen, whereas LoRA injects learnable low-rank updates directly into specific weight matrices. Prefix and prompt methods act on the input/attention side and are compact and simple to store, while LoRA modifies the linear layers inside the model and often achieves higher adaptation capacity for similar or slightly larger parameter budgets.

The intuition (an analogy that makes it click)

Think of the frozen model as a skilled musician who can play many tunes. Prompt Tuning and Prefix Tuning give the musician a small sheet of hints placed at the front of the music stand. The musician reads the hints and subtly changes performance without changing the instrument.

LoRA instead adds small adjustable gears inside the instrument so it produces different tones. Both approaches avoid replacing the whole instrument, but the gears change fundamental mechanics while the hint sheet nudges behavior at input time.

How it actually works (the real mechanics, with a concrete worked example)

Prompt Tuning stores a sequence of $p$ learnable embedding vectors and prepends them to the token embeddings each time you run the model. These vectors are optimized so the frozen model behaves as if it had seen a specialized prompt.

Prefix Tuning stores a sequence of $m$ virtual tokens that are mapped to attention key and value vectors at each attention layer. Instead of prepending embeddings that become normal tokens, the prefix provides extra key/value pairs that the attention mechanism can attend to. That lets the prefix influence internal attention patterns directly.

LoRA selects one or more weight matrices, for example a projection $W$ of shape $d_{in}\times d_{out}$, and models a low-rank additive update $\Delta W = BA$ with $B\in\mathbb{R}^{d_{in}\times r}$ and $A\in\mathbb{R}^{r\times d_{out}}$. During training you freeze $W$ and only learn $A$ and $B$.

The LoRA parameter count for one adapted matrix is roughly $\text{LoRA params} \approx r(d_{in} + d_{out}).$

Worked numeric example. Suppose a transformer with hidden size $d=768$, we use prefix length $m=50$, prompt length $p=50$, and LoRA rank $r=8$ applied to two matrices per layer across 12 layers. Extra parameter counts are:

This table shows why prefix/prompt tuning can be extremely small in parameter count. Prefix tuning often stores different virtual tokens per layer which increases the count compared to a single prompt vector set.

Practical worked step: fine-tuning a classifier with Prefix Tuning

1. Freeze the base model parameters.
2. Initialize $m$ virtual prefix vectors per layer or shared across layers.
3. Forward pass: compute attention using original keys/values plus prefix keys/values.
4. Backpropagate and update only prefix parameters and classifier head.
5. At inference, prepend same learned prefix and run the frozen model.

When to use each

• Use Prompt Tuning when you want the smallest storage overhead and your model responds well to input-space soft prompts. It is very light but can be limited on complex tasks.
• Use Prefix Tuning if you need more influence over internal attention and do not mind slightly more parameters because prefixes can be layerwise key/value vectors.
• Use LoRA when you want stronger representational changes inside the model with still modest parameter costs. LoRA often performs better on tasks that require deeper changes to internal transforms.

Tradeoffs and failure modes

Prefix/Prompt advantages: tiny checkpoints, simple to implement, easy to share. Limitations: they act through input-side steering and may not be able to change internal linear algebra as effectively as weight-space methods.

LoRA advantages: can approximate full weight updates with low-rank structure, often competitive with full fine-tuning for many tasks. Limitations: you must choose which matrices to adapt and rank parameter $r$, and storage grows with number of adapted matrices.

Questions the interviewer might ask

Some follow-up questions you might get:

How do prefixes differ from prompts under the hood? Prompts are learned embeddings prepended to input tokens and then processed normally. Prefixes produce additional attention key/value vectors, which directly influence attention without behaving as regular tokens.

Can you combine LoRA with prompt or prefix tuning? Yes. Combining methods is common. For instance you can apply LoRA to attention projections and also prepend a learned prompt. They can complement each other because they act in different parts of the model.

How do you choose prefix length or LoRA rank in practice? Tune them as hyperparameters. Start small, for example $m=20$ or $r=4$, and increase until validation performance saturates. Consider compute and storage constraints when scaling.

What about inference cost differences? Prompt and prefix tuning add minimal compute because they only add extra tokens or small key/values. LoRA adds small matrix multiplications but can be implemented efficiently. Full fine-tuning does not add inference cost but increases deployment complexity because you must store full model weights per task.

Are there security or privacy implications? Parameter-efficient checkpoints are small and easier to share but do not expose full model weights. They do not remove risks like memorization from the base model.

Some things to note:

• Prefix tuning may store different virtual tokens per layer, increasing parameter count.
• LoRA needs careful selection of which matrices to adapt and is often applied to attention and feedforward projections.

What the interviewer is really testing

They want to see you understand where additional parameters live and how that affects expressivity, storage, and inference. They also expect you to reason about when input-space steering is enough versus when weight-space updates are needed. Being able to compare parameter counts, describe training and inference changes, and recommend a method for a given constraint shows practical depth.