How do you conduct human evaluation for AI systems?

In this question, we will learn how to conduct human evaluation for AI systems so your results are reliable, explainable, and defensible. We focus on practical steps you can describe in an interview: choosing metrics, designing tasks, recruiting and monitoring raters, and analyzing results.

We will cover the following:

• The intuition
• How it actually works
• Designing tasks and instructions
• Quality control and analysis
• Tradeoffs and failure modes

Direct answer: Human evaluation for AI systems requires a clear measurement goal, a matching human task, careful rater instructions and qualification, and statistical analysis that accounts for agreement and sampling error. You should pilot the task, use quality control items, compute inter-rater agreement such as $\kappa$, and report confidence intervals for differences. Communicate limitations and ethical safeguards.

The intuition (an analogy that makes it click)

Think of human evaluation like running a small clinical trial for a software behavior. We choose a measured outcome, recruit raters or experts, randomize what they see, and use controls to ensure measurements reflect the model quality and not noise. We repeat and refine until the measurement is stable enough to support conclusions.

How it actually works (the real mechanics, with one concrete worked example)

Start by picking the right measurement method for your question. Common choices are:

• Pairwise preference: show two system outputs and ask which you prefer.
• Likert rating: ask raters to give a score on dimensions like coherence or helpfulness.
• Binary correctness: mark whether a claim is factually supported.

Worked example: evaluating two summarization models A and B for factuality and overall preference.

Design decisions:

• Metric 1: pairwise preference for overall quality.
• Metric 2: binary factuality check per sentence.
• Raters: crowdworkers with a short qualification test.
• Items: 200 source documents sampled stratified by length.
• Repeats: each pair shown to 5 different raters for redundancy.

Table of a small results snapshot:

To choose sample size for a proportion you can use the common formula

$n = z^2 \frac{p(1-p)}{e^2}$

where $z$ is the z-score for your confidence level, $p$ is the estimated proportion, and $e$ is the margin of error. For a conservative estimate set $p=0.5$ and $z=1.96$ for 95% confidence.

Measure inter-rater agreement to gauge noise. For categorical labels use Cohen or Fleiss $\kappa$; for ordinal ratings use Krippendorff or intraclass correlation. The kappa formula is

$\kappa = \frac{p_o - p_e}{1 - p_e}$

where $p_o$ is observed agreement and $p_e$ is chance agreement.

Finally run statistical tests for differences. For proportions use a two-proportion z-test or bootstrap for confidence intervals. For pairwise preferences, a simple binomial test can show whether a preference rate significantly differs from 50%.

Designing tasks and instructions

Clear examples are the most impactful single improvement you can make. Show positive and negative example annotations. Keep each question narrowly focused. When asking for subjective judgments like helpfulness or style, define anchors for each Likert point. For pairwise tasks instruct raters to ignore unimportant differences like punctuation when that is not part of your evaluation goal.

Pilot on 20 to 50 items. Use pilot data to refine instructions and to estimate $p$ for the sample size formula. Pay attention to order effects and randomize where relevant.

Quality control and analysis

Qualification tests: include a short quiz that replicates tricky cases. Interleave gold items: items with known ground truth. Insert attention checks but keep them subtle so you do not prime behavior.

Aggregation: choose an aggregation rule before analyzing. Options include majority vote, mean rating after outlier removal, or weighted votes by rater reliability. Report both raw per-rater variability and aggregated results.

Statistical tests: report effect sizes and confidence intervals. Bootstrap is useful when distributional assumptions are unclear. For paired data use paired tests; for independent samples use two-sample tests. For example a 95% bootstrap CI for difference in preference helps illustrate uncertainty.

Tradeoffs and failure modes

Human evaluation is limited by the construct validity of your task. If instructions are ambiguous, raters may optimize for a different target than you intend. Expert raters increase validity for complex tasks but cost more and scale poorly. Crowd raters scale but need stronger controls.

Questions the interviewer might ask

Some follow-up questions you might get:

Why choose pairwise preference over Likert scales? Pairwise comparisons reduce scale interpretation noise and often require fewer judgments to detect a preference, but they do not provide absolute quality measures.

How do you pick sample size? Use a power calculation or the proportion formula shown earlier. Pilot estimates of $p$ improve precision. For small expected differences increase $n$.

What do you do if inter-rater agreement is low? Investigate instruction clarity, ambiguous examples, and rater qualifications. You may need to revise the task or escalate to expert annotators.

How do you avoid bias in crowd evaluations? Randomize presentation order, anonymize system labels, and balance item sampling. Monitor per-worker bias patterns and remove malicious workers.

When should you use experts? Use experts for safety, factuality, or domain-specific judgments where lay raters lack knowledge. Experts give higher validity but at higher cost and lower throughput.

How do you measure and report uncertainty? Report confidence intervals or bootstrap distributions, p-values for pre-specified tests, and inter-rater agreement metrics.

Some things to note:

• Pre-register task definitions and analysis choices when possible.
• Always pilot and iterate instructions before full collection.
• Keep privacy and consent explicit for any data with personal information.

What the interviewer is really testing

We want to see that you understand experimental design, measurement validity, and practical constraints when measuring model behavior. They are looking for the ability to pick the right instrument, control for noise and bias, and communicate uncertainty and limitations. Give concrete operational steps you would take and mention common diagnostics like agreement metrics and sample size reasoning.