Explain BLEU, ROUGE, and BERTScore. When would you use each?

In this question, we will learn about BLEU, ROUGE, and BERTScore and when to use each. These three metrics are common automatic ways to evaluate generated text, but they focus on different signals and therefore suit different tasks.

We will cover the following:

• The intuition (an analogy that makes it click)
• How it actually works
• When to use each
• Practical tips
• Tradeoffs and failure modes
• Questions the interviewer might ask
• What the interviewer is really testing

Direct answer: Use BLEU when you care about exact n-gram precision and phrasing, typically in machine translation. Use ROUGE when you care about recall and coverage, often in summarization. Use BERTScore when you want semantic matching and tolerance for paraphrase, at the cost of more compute.

The intuition (an analogy that makes it click)

Think of evaluation as comparing two paintings. BLEU counts how many exact brush strokes from the reference appear in the candidate. ROUGE checks whether the candidate includes the important areas of the reference, even if the strokes differ. BERTScore looks at the style and color of the strokes using a learned model and judges whether the paintings feel the same even if the brushwork differs.

This analogy highlights precision versus recall versus semantic similarity.

How it actually works (the real mechanics, with one concrete worked example)

BLEU

• BLEU computes modified n-gram precisions $p_n$ for $n=1..N$ and combines them with a geometric mean, then applies a brevity penalty $BP$.
• The formula is $BLEU = BP \cdot \exp\left(\sum_{n=1}^N w_n \log p_n\right)$ where typical weights $w_n$ are uniform and $BP$ penalizes candidates much shorter than references. For candidate length $c$ and reference length $r$: $BP = \begin{cases}1 & \text{if } c > r\\[4pt] e^{1-r/c} & \text{if } c \le r\end{cases}$

ROUGE

• ROUGE-N is recall of n-grams: $R = \frac{\text{overlap}}{\text{reference tokens}}$.
• ROUGE-L uses the longest common subsequence to reward sequence-level matches and order preservation.

BERTScore

• BERTScore encodes tokens with contextual embeddings, computes pairwise cosine similarities $\text{cos}(u,v)=\frac{u\cdot v}{\|u\|\|v\|}$, and then performs soft or hard matching to produce precision, recall, and F1 scores.

Worked example. Reference: "the cat sat on the mat". Candidate A: "the cat is sitting on the mat". Candidate B: "a feline sits upon the rug". The table shows rough, illustrative values to make differences clear.

Candidate A shares many exact n-grams and similar length, so BLEU and ROUGE favor it and BERTScore is high because embeddings match closely. Candidate B uses synonyms and rephrasing, so BLEU drops sharply, ROUGE drops because literal overlap is low, while BERTScore stays relatively high as embeddings capture semantic similarity.

When to use each

• BLEU: Use for machine translation and scenarios where exact phrasing and n-gram fidelity matter. BLEU performs better with multiple references and when surface-form fidelity is desired.
• ROUGE: Use for extractive and some abstractive summarization tasks where coverage and recall of important content matters more than exact wording. ROUGE-L helps when sequence order is important.
• BERTScore: Use when paraphrase, synonyms, or semantic equivalence are acceptable or the main goal, such as abstractive summarization evaluation or semantic similarity tasks.

Practical tips

• Report multiple metrics. A combination like BLEU or ROUGE plus BERTScore gives a fuller picture.
• For BLEU, include multiple references when possible because single references make BLEU conservative.
• For BERTScore, fix the embedding model and document tokenization choices. Be aware of domain mismatch between the chosen encoder and your data.
• Interpret small differences cautiously and use bootstrap or significance testing for system comparisons.

Tradeoffs and failure modes

Other failure modes include short references that make BLEU and ROUGE unstable, domain-specific vocabulary lowering embedding quality for BERTScore, and disagreement between metrics where each highlights different strengths.

Questions the interviewer might ask:

Some follow-up questions you might get: How does BLEU handle multiple references? BLEU uses the maximum counts of n-grams across references when computing modified precision, which increases the chance of matching paraphrases if they appear among any references.

Why is brevity penalty needed for BLEU? Without $BP$, a very short candidate that matches a few high-precision n-grams could score highly. $BP$ penalizes candidates shorter than the reference to discourage this behavior.

When would ROUGE-L be better than ROUGE-N? ROUGE-L captures the longest common subsequence and rewards in-order matches, so it is better when preserving ordering and sentence structure matters more than isolated n-gram overlap.

What hyperparameters affect BERTScore? Choice of embedding model, use of IDF weighting, normalization, and matching algorithm (greedy versus bipartite matching) all change BERTScore results and should be specified.

Can we combine these metrics into a single number? You can build composite scores or learn a regressor over metrics, but any combination must be validated against human judgments to avoid optimizing for metric quirks.

Some things to note:

• Provide multiple references and metrics when possible.
• Report confidence intervals or bootstrap results for system comparisons.
• Match the metric to the evaluation goal: precision, recall, or semantics.

What the interviewer is really testing

They want to see you distinguish surface-form overlap from semantic matching and to choose metrics that align with evaluation goals. They also expect awareness of practical issues like multiple references, length effects, and embedding-model dependence. Finally, they want you to recommend combining automatic metrics with human evaluation when results matter.