What is RLHF (Reinforcement Learning from Human Feedback), and how is it used to align LLMs?

In this question, we will learn what RLHF (Reinforcement Learning from Human Feedback) is and how it is used to align large language models during fine-tuning. We will keep the math light and show a concrete recipe you could sketch in an interview.

We will cover the following:

• The intuition
• How it actually works
• Practical recipe and worked example
• When to use RLHF versus supervised fine-tuning
• Tradeoffs and failure modes

Direct answer: RLHF trains a reward model from human judgments and then uses reinforcement learning to update the model policy so it prefers outputs humans like. It typically starts with supervised fine-tuning, collects human comparisons of model outputs to teach a reward model $r_{\phi}$, and then updates the policy network using a policy optimization algorithm such as PPO to maximize the learned reward while constraining drift from the base model.

The intuition (an analogy that makes it click)

Think of the base LLM as a trainee chef who can follow recipes but has broad tastes. Humans taste several dishes and say which they prefer. Instead of encoding every preference as a rule, we train a critic who predicts the human taste. Then we let the chef adapt recipes to please the critic. The critic lets us provide dense feedback from sparse human comparisons.

How it actually works (the real mechanics, with one concrete worked example)

There are three canonical stages:

1. Supervised fine-tuning (SFT). Start from a pretrained LLM and optionally fine-tune on human-written demonstrations to set a reasonable initial policy $\pi_{\theta_0}$.
2. Reward model training. Collect human comparisons where for a prompt humans say Response A is preferred to Response B. Fit a reward model $r_{\phi}$ that assigns higher scalar scores to preferred responses.
3. Policy optimization. Use a reinforcement learning algorithm such as Proximal Policy Optimization (PPO) to update the policy parameters $\theta$ to increase expected reward under $r_{\phi}$ while penalizing deviation from the SFT policy.

The optimization objective in RL form is to maximize expected reward:

$J(\theta)=\mathbb{E}_{\tau\sim\pi_{\theta}}\left[\sum_t r_t\right]$

In practice we use the learned scalar $r_{\phi}(\text{prompt},\text{response})$ and a KL penalty to the reference policy $\pi_{\theta_\text{ref}}$ so the practical objective per batch is something like reward minus $\beta$ times KL.

Worked example: suppose for a prompt you generate two responses A and B. Humans preferred A over B, so the dataset contains the comparison (A,B). Train the reward model so that $r_{\phi}(A) > r_{\phi}(B)$. After many comparisons we might have average scores $r_{\phi}(\text{good})=1.2$ and $r_{\phi}(\text{bad})=0.3$. During PPO we compute advantages using $r_{\phi}$ and update policy to increase probability of responses similar to A.

When comparing methods it helps to put them in a table:

Practical recipe and worked example

A compact recipe you can explain in an interview:

1. Collect SFT data and fine-tune the base model for safe, basic behavior.
2. For a set of prompts, sample multiple responses from the fine-tuned model. Have humans provide pairwise preferences forming tuples (prompt, A, B, choice).
3. Train a reward model $r_{\phi}$ to minimize a pairwise loss such as cross entropy on preference comparisons. A common loss is logistic: for pair (A,B) where A preferred, minimize $-\log\sigma(r_{\phi}(A)-r_{\phi}(B))$.
4. Run PPO to maximize expected $r_{\phi}$ with an additional KL penalty term $\beta \mathrm{KL}(\pi_{\theta}\|\pi_{\theta_\text{ref}})$ to prevent runaway optimization.

Concrete numbers: imagine 10k comparisons, a reward model that maps responses to scalars. During PPO you might collect 2k episodes per iteration and run 3 epochs of minibatch updates. Hyperparameters you will be asked about include the KL coefficient $\beta$, learning rate, and reward model capacity.

When to use RLHF versus supervised fine-tuning

RLHF is best when preferences are hard to express as explicit labels or rules, for example helpfulness, tone, or subtle safety judgments. SFT is simpler and more stable when you can specify correct outputs.

Tradeoffs and failure modes

RLHF improves alignment but introduces several failure modes.

Other common issues:

• Reward hacking: the policy finds loopholes in the learned reward that do not reflect human intent.
• Overoptimization: too strong updates degrade helpfulness or factuality.
• Data bias: human preferences reflect annotator bias and can amplify it.

Questions the interviewer might ask

Some follow-up questions you might get:

How do you train the reward model from pairwise data? You fit a scalar model $r_{\phi}$ using a loss such as logistic: for a preferred A over B minimize $-\log\sigma(r_{\phi}(A)-r_{\phi}(B))$. This encourages higher scores for preferred responses.

Why use PPO instead of simple policy gradient? PPO provides stable, clipped updates that control policy drift and reduce variance, which matters when optimizing a noisy learned reward.

What is the role of the KL penalty? The KL penalty constrains the tuned policy to remain close to a reference policy, preventing the model from drifting into degenerate behaviors while still improving measured reward.

How do you detect reward hacking? Look for high reward but poor human evaluations, unnatural phrasing, or outputs that exploit auxiliary signals. Regular human evaluation and adversarial prompts help catch it.

When would you retrain the reward model? Retrain when the policy explores new output modes not covered in the comparison data, or when annotator criteria shift. Periodic updates keep alignment up to date.

Some things to note:

• Human annotation quality controls are often the bottleneck.
• Calibration of $r_{\phi}$ matters but absolute scale is less important than rank order.

What the interviewer is really testing

They want to see that you understand the full RLHF pipeline end to end and can reason about how human data becomes a reward signal and how optimization affects model behavior. They also expect you to name practical safeguards such as KL penalties, validation with held-out human judgments, and ways to detect reward model failure.