Explain the difference between full fine-tuning and parameter-efficient fine-tuning (PEFT).

In this question, we will learn how full fine-tuning and parameter-efficient fine-tuning (PEFT) differ, and why you might choose one approach over the other for a given task.

We will cover the following:

• The direct answer and short summary
• The intuition behind each approach
• How PEFT methods actually work with a concrete example and formulas
• Practical tradeoffs and failure modes
• Likely follow-up interviewer questions and concise answers

Full fine-tuning updates all model weights by backpropagating into every parameter, while PEFT freezes most of the pretrained parameters and trains small additional modules or low-rank updates to achieve task adaptation. Full fine-tuning often gets the best possible task performance but costs more in compute, memory, and model storage. PEFT usually reaches close-to-full performance with orders of magnitude fewer trainable parameters and smaller checkpoints.

The intuition (an analogy that makes it click)

Think of a pretrained model as a ship built for many oceans. Full fine-tuning is like rebuilding large parts of the ship for a new sea condition. You get a vessel tailored to the new water, but that takes time and materials. PEFT is like adding targeted attachments and lightweight sails that change how the ship handles without replacing the hull. Those attachments are cheaper to install, and you can switch them in and out quickly for different voyages.

How it actually works

Full fine-tuning: every weight matrix and bias scalar is a trainable variable. If the model has $P$ parameters, training updates all $P$.

PEFT: the pretrained weights stay fixed. We introduce additional small parameter groups or structured updates. A general low-rank scheme like LoRA adds low-rank matrices to existing weight projections. For a weight matrix $W$ of size $d\times k$, LoRA parameterizes the update as $\Delta W = BA$ where $B$ is $d\times r$ and $A$ is $r\times k$. The extra parameters are about

$\text{LoRA extra params}\approx r(d+k)$

and the relative fraction for that matrix is

$\text{fraction} = \frac{r(d+k)}{dk}.$

Worked example. Suppose you have a model with $1.5$ billion parameters and attention projection matrices that are roughly square with dimension $d=2048$. If you apply LoRA with rank $r=8$ across the major projections, the added parameters per matrix are about $2\times r\times d\approx 2\times 8\times 2048\approx 32{,}768$. Summing across many projection matrices gives an extra few million parameters, typically under $1\%$ of $1.5$ billion.

Compare typical outcomes:

Note that exact numbers depend on where you insert PEFT modules and the chosen ranks or hidden sizes.

Other PEFT flavors and how to pick

• Adapters: small bottleneck feedforward modules inserted between layers. They typically add a few million parameters and work well when you need a modular change per task.
• Prefix and prompt tuning: optimize continuous prompts or key/value prefixes that steer the transformer. These use very few parameters but can be less effective on tasks needing deeper representation change.
• LoRA: low-rank additive updates to specific projection matrices. Often a sweet spot for many NLP tasks because it targets attention and projection layers directly.

When to pick each: if you need near state-of-the-art and can afford storage and compute for multiple full checkpoints, choose full tuning. If you need to support many tasks, fast experiments, or limited GPU memory, choose PEFT.

Tradeoffs and failure modes

PEFT advantages: far lower training memory, quicker experiments, and tiny per-task checkpoints so you can ship many task adapters. Full tuning advantages: the maximum representational flexibility and often slightly better final accuracy on difficult distribution shifts.

Questions the interviewer might ask

Some follow-up questions you might get:

Why does LoRA use low-rank updates? Low-rank updates restrict the update space to a small subspace, which reduces parameters and focuses learning on dominant directions in weight changes. This often captures task-specific shifts without rewriting the whole matrix.

How do you store PEFT checkpoints in production? You typically store the base model once and save small delta files per task that contain adapter weights or LoRA matrices. At load time you merge or apply those deltas to the base model.

Does PEFT affect inference speed? Most PEFT methods add negligible latency. Adapters and LoRA add small matrix operations. Prompt tuning has essentially zero change to model FLOPs but adds token prep cost.

Can PEFT reach full-tuning performance? On many tasks PEFT approaches come very close. For some tasks that require wholesale feature reconfiguration, full fine-tuning can still win. Empirical validation matters.

How do you choose hyperparameters like LoRA rank or adapter bottleneck size? Start small and scale up until validation performance saturates. Use few-shot sweeps because these hyperparameters control capacity and overfitting risk.

Some things to note:

• PEFT makes experimentation with many tasks affordable because checkpointing cost is tiny.
• Always validate that the chosen PEFT method can represent the necessary task change.
• For safety and auditing, storing deltas separately keeps base model immutable.

What the interviewer is really testing

They want to see that you understand practical tradeoffs: compute, memory, storage, and final accuracy. They also test whether you can reason about how structural constraints like low-rank adapters change the model update space and how that affects generalization. Finally, they may probe deployment concerns, such as checkpoint management and inference implications.