Articles

Building an LLM evaluation harness that catches regressions

Published 20 September 2026 · 10 minute read

The demo worked. You tweaked the prompt to fix a customer complaint, shipped it, and three weeks later found out the tweak had broken a different case nobody was watching. No test caught it, because the output is prose, and prose doesn’t have an equals sign.

An evaluation harness is the thing that replaces “it seemed better in the playground” with a number you can defend. It is the highest-leverage piece of infrastructure in an LLM project, and it is usually the last thing teams build. Here is how to build one that earns its keep.

What a harness actually is

Four parts, none of them exotic:

1. A dataset of cases: inputs plus whatever is needed to judge the output.
2. A runner that executes your real task function over those cases, repeatedly and in parallel.
3. Graders that turn each output into scores.
4. A report and a gate — results sliced by category, compared against a baseline, wired into CI.

The critical detail: the runner must call your application’s task function, not a bare model API. If retrieval, tool calls, parsing and retries are part of production, they’re part of the thing under test. Evaluating the prompt in isolation measures something you don’t ship.

The dataset is the hard part

Everything else is plumbing. The dataset is where the judgement goes, and it’s worth more than all the code around it.

• Source from real traffic. Cases you invent cluster around what you already thought of. Pull from logs, support tickets, and the inputs that produced complaints. If you have not launched, use whatever your users send to the human process the feature will replace.
• Start smaller than feels respectable. Fifty to a hundred well-chosen cases will find more real problems than a thousand scraped ones, and you can actually look at every failure. Growth comes later and for a reason.
• Stratify by failure mode, not by topic. Tag each case with what makes it hard: ambiguous request, missing information, multi-step reasoning, adversarial input, unusual format, long context, non-English, out-of-scope. Aggregate scores hide everything; per-category scores tell you where you actually stand.
• Every bug becomes a case. This is the habit that compounds. A production failure is not closed until it exists in the suite and the suite fails on it. That single rule turns the harness into a regression net over time.
• Hold some back. Keep a frozen slice you do not look at while iterating. Prompts get overfitted to eval sets just as models get overfitted to benchmarks — you will tune until the number goes up, and the number will stop meaning anything.
• Include cases that should be refused. Out-of-scope requests, prompt injection attempts, requests for data the user is not entitled to. Scored pass/fail alongside everything else, so that the security properties keep holding as prompts change.

Graders, cheapest first

Reach for the most deterministic grader that can answer the question. Most teams jump straight to an LLM judge and end up debugging the judge.

Using an LLM judge without fooling yourself

Judges are useful, and they’re also a measurement instrument you haven’t calibrated. Before trusting one:

• Label a sample by hand (50 or so outputs, graded by a person) and check the judge’s agreement with those labels. If agreement is poor, fix the rubric before running it over thousands of cases. An uncalibrated judge does not produce a weak signal; it produces a confident wrong one.
• Give it a rubric, not an adjective. “Rate helpfulness 1–5” produces mush. Enumerate the specific failure conditions and ask for a binary verdict against each.
• Prefer binary or pairwise. “Does this answer contain a claim not supported by the source? yes/no” is far more stable than a five-point scale.
• Control for position bias. In pairwise comparisons, judges favour one slot. Run both orders and average.
• Watch for self-preference. A judge from the same family as the generator tends to prefer its own style. Use a different model for judging where you can.
• Pin the judge version. If the judge model changes underneath you, your historical scores are no longer comparable, and you will spend a day hunting a regression that happened in the ruler.

Metrics worth putting on the dashboard

• Task success rate, overall and per category. The headline number.
• Cost per task — mean and p95. This is what makes the ROI model real rather than hypothetical.
• Latency — p50 and p95. Agent loops have terrible tails, and the tail is what users feel.
• Format failure rate — how often output failed schema validation and needed a retry. A leading indicator of cost blowouts.
• Refusal rate on in-scope requests, and compliance rate on out-of-scope ones. Prompt changes move these in opposite directions constantly.
• Adversarial pass rate on the injection cases.
• Escalation rate — the share of tasks handed to a human. The single number that determines whether the project pays for itself.

Noise: the mistake almost everyone makes

Two runs of the same suite will not give the same score, and temperature=0 does not fix it — batching, routing, and floating-point non-determinism all leak through. Treating a single run as ground truth means chasing ghosts.

What to do instead:

• Sample each case k times (3 is a reasonable default, 5 if the suite is small) and report the mean.
• Report an interval, not a point. On 100 cases, a score of 82% has a margin of roughly ±8 points at 95% confidence. A move from 82% to 85% is noise.
• Compare paired. Run variant A and variant B over the identical cases and compare per-case outcomes — a paired test on the cases that changed is far more sensitive than comparing two aggregate percentages.
• Read the diffs, not just the delta. The list of cases that flipped from pass to fail is more informative than any summary statistic, and it takes ten minutes.

A useful rule: if you can’t point at the specific cases that improved, the improvement isn’t real yet.

Wiring it into CI

A harness only changes behaviour once it can block a merge:

• Smoke suite on every pull request. Thirty to sixty cases, a few minutes, a few cents. Fast enough that nobody routes around it.
• Full suite nightly and before any release, with results stored per commit so you can bisect a regression to a change.
• Gate on regression, not on an absolute threshold. “No category may drop more than N points against the baseline” survives contact with reality; “must score above 90%” gets disabled the first time it is inconvenient.
• Cache aggressively. Key results on a hash of prompt, input, model version and tool schema. Unchanged combinations should not be re-billed.
• Re-run on provider changes, not only on your own. A model version rolling forward underneath you is exactly the event this catches, and it is the one you will not otherwise notice.
• Version prompts like code. In the repository, code-reviewed, with the eval diff attached to the pull request that changes them. Prompts edited in a console are changes nobody reviewed.

A minimal structure

Nothing here needs a framework. This is roughly the whole idea:

from dataclasses import dataclass
from typing import Any, Callable

@dataclass(frozen=True)
class Case:
    id: str
    category: str          # so results can be sliced
    inputs: dict
    expect: dict           # whatever the graders need

@dataclass
class Result:
    case_id: str
    category: str
    passed: bool
    scores: dict
    cost_usd: float
    latency_ms: int
    output: Any

Grader = Callable[[Case, Any], dict]

def run(cases, task, graders, samples: int = 3):
    """task() is the production entry point, not a bare model call."""
    for case in cases:
        for _ in range(samples):
            output, usage = task(**case.inputs)
            scores = {}
            for grade in graders:
                scores.update(grade(case, output))
            yield Result(
                case_id=case.id,
                category=case.category,
                passed=all(scores.values()),
                scores=scores,
                cost_usd=usage.cost_usd,
                latency_ms=usage.latency_ms,
                output=output,
            )

Graders are just functions returning a dict of named booleans:

def cites_a_source(case, output) -> dict:
    return {"cited": bool(output.citations)}

def no_unsupported_claims(case, output) -> dict:
    verdict = judge(
        rubric=FAITHFULNESS_RUBRIC,
        source=case.expect["source"],
        answer=output.text,
    )
    return {"faithful": verdict.is_supported}

Add a reporter that groups by category and diffs against the stored baseline, and you have the whole harness in a few hundred lines. Adopt a framework later if you want a UI or managed storage; do not let choosing one delay having a dataset, which is the part that matters.

What it costs

Budget roughly one to two weeks of engineering for the first working version, most of it spent on the dataset rather than the code. Running it is cheap: 100 cases × 3 samples at a fraction of a cent each is pocket change per run, and caching removes most of it. LLM judges add cost, which is another reason to use deterministic graders wherever the task allows.

Set against that: the first regression it catches before release usually pays for the whole thing, and it converts every “does this prompt change help?” argument from a matter of opinion into a matter of record.

Common mistakes

• Evaluating the model instead of the system — no retrieval, no tools, no parsing, so the score describes something you do not ship.
• One giant success metric with no category breakdown, so you can see that something broke but not what.
• An uncalibrated LLM judge, trusted because it produces numbers.
• A single sample per case, producing scores that move on their own.
• Cases written from imagination rather than from traffic.
• A suite that never grows, because fixed bugs are not being added back as cases.
• Results in someone’s notebook instead of in CI, where they would actually block a bad change.