How does fine-tuning work?

In this question, we will learn what fine-tuning is, why it works, and how to pick a practical strategy for different resource and data regimes. We will walk through a concrete example you can explain in an interview.

We will cover the following:

• The short direct answer
• The intuition
• How it actually works with a worked example
• Practical strategies and comparison of methods
• Tradeoffs and failure modes
• Questions the interviewer might ask

Fine-tuning is the process of adapting a pretrained model to a new task by updating some or all model parameters or by adding small task-specific modules; you choose how many parameters to change, set a learning rate appropriate to the pretrained weights, and validate to avoid overfitting. Fine-tuning usually gives better final performance than training from scratch, but costs more memory and can risk forgetting.

The intuition (an analogy that makes it click)

Think of a pretrained model as a chef who learned many cuisines. Fine-tuning is like teaching that chef one new signature dish. You can either let the chef rewrite their entire cookbook for the dish, adjust only the last steps of the recipe, or attach a short cheat-sheet the chef consults while cooking. Updating the whole cookbook can yield the best dish but takes time and risks changing other recipes. Attaching a small cheat-sheet is cheaper and keeps the chefs other recipes intact.

How it actually works (the real mechanics, with one concrete worked example)

Mechanically, fine-tuning changes parameters to minimize a task loss. If the pretrained parameters are $\theta_0$ and the task loss is $\mathcal{L}(\theta)$, standard gradient descent updates are

where $\eta$ is the learning rate. In parameter-efficient methods we freeze most parameters and only optimize a subset $\phi$. The loss then depends on $\phi$ and the update becomes

Concrete example: sentiment classification using a pretrained transformer with 100 million parameters. Options:

Suppose you have 2,000 labeled examples. A practical recipe is: freeze most layers, add a classifier head, use a batch size of 16 to 32, run 3 to 10 epochs, and set a head learning rate near $1\times10^{-3}$. If performance saturates, gradually unfreeze more layers with a smaller learning rate, like multiplying by 0.1 for lower layers.

Practical strategies and comparison of methods

Full fine-tuning

• Update every parameter. Best when you have a lot of labeled data and GPU memory. Use lower learning rates to avoid destroying pretrained features.

Head-only training

• Only train a final linear or MLP head. Fast and stable for small datasets. It can underperform if the task needs changes in earlier representations.

Adapter and small-module tuning

• Insert small trainable modules into transformer blocks and freeze base weights. This keeps memory low and enables multiple tasks without multiple full models.

Low-rank and prefix methods (LoRA, prefix-tuning)

• Represent updates in a low-dimensional subspace to reduce trainable parameters and memory. Works well for very large models.

Comparison table of common tradeoffs:

Hyperparameters and practical tips

• Learning rate: pretrained weights are already good. Start low for weights you fine-tune, often $1\times10^{-5}$ to $5\times10^{-5}$ for large models. For newly added parameters use higher rates.
• Weight decay: small values help, but be cautious if you tune bias or layernorm parameters.
• Warmup and schedulers: a short linear warmup helps stabilize updates on large models.
• Batch size and steps: small datasets need fewer steps to avoid overfitting; track validation metrics.

When you unfreeze layers gradually, use discriminative learning rates where deeper layers get smaller $\eta$.

Tradeoffs and failure modes

Fine-tuning can improve accuracy but introduces risks and costs. You must balance compute, memory, and the amount of labeled data.

Other failure modes:

• Tuning with too large a learning rate can destroy useful pretrained features and yield worse performance than random initialization in extreme cases.
• Label noise in a small dataset can make fine-tuning harm generalization. Consider regularization or training with robust loss functions.

Questions the interviewer might ask

Some follow-up questions you might get:

Why is the learning rate usually smaller for pretrained weights? Because pretrained weights already encode useful representations. Large updates can erase those representations. Small learning rates preserve them while nudging parameters toward the task objective.

When should you prefer adapters or LoRA over full fine-tuning? When memory is limited, you have many tasks, or you want to keep one shared base model and switch small modules per task.

What is catastrophic forgetting and how can you mitigate it? Catastrophic forgetting occurs when adapting to a new task makes the model lose performance on original tasks. Mitigations include regularization to stay near $\theta_0$, rehearsal with old data, or parameter-efficient tuning.

How do you choose how many layers to unfreeze? Start by training the head. If performance is insufficient, unfreeze top transformer blocks progressively. Monitor validation gains and stop when returns diminish or overfitting appears.

How does dataset size affect strategy? Small datasets: head-only or adapters. Medium datasets: unfreeze some layers. Large datasets: full fine-tuning is often best.

Some things to note:

• Save both the fine-tuned weights and the training recipe: learning rates, epochs, and seed matter.
• Parameter-efficient methods often match full fine-tuning for many tasks when designed well.

What the interviewer is really testing

They want to know you understand why pretrained representations help, how optimization choices affect those representations, and the practical tradeoffs between accuracy, compute, and memory. They also want to see you can select a sensible experimental plan and defend hyperparameter choices based on data size and constraints.