AI Engineering Practice · 3 min read
How to Build an Evaluation Harness for LLM Agents
Why agent systems degrade without evaluation, how to build a trajectory-level harness, and what to measure beyond simple final output correctness.
An agent evaluation harness scores whole trajectories against a labelled set, runs automatically on every prompt or tool change, and measures intermediate steps rather than only final output. Without one, prompt changes are unverifiable and regressions ship invisibly while the system still sounds fluent.
Why agents degrade quietly
The dangerous property of language model systems is that they fail fluently. A traditional service that breaks returns an error, a stack trace, a five-hundred status code — something unambiguous that monitoring catches. An agent that breaks returns a confident, well-formed, entirely wrong answer, and nothing in your infrastructure notices anything unusual at all.
This is why agent systems degrade without anyone observing it happening. Someone adjusts a prompt to fix a reported case. The adjustment helps that case and subtly harms three others that nobody tests. Repeat weekly for a quarter and the system is measurably worse than it was, while every individual change was locally justified and appeared to work at the time.
An evaluation harness is the only reliable defence. It is also, consistently, the thing teams postpone longest, because it produces no visible feature and its value is entirely in the regressions it prevents — which are by definition invisible when it is working.
Building the harness
Collect real failures first
Do not invent test cases. Take actual production runs that went wrong, with the exact inputs that produced them, and use those as your seed. Fifty real failures produce a far more useful evaluation set than five hundred synthetic ones, because real inputs contain the ambiguity and mess that synthetic cases systematically omit.
Label at the trajectory level, not just the output
Record what the correct sequence of actions was, not only what the correct final answer was. An agent that reaches the right answer through three unnecessary tool calls is a cost problem you want to see. An agent that reaches a wrong answer through a correct process is a different bug from one that reasons badly.
Score multiple dimensions separately
Final correctness, tool selection accuracy, step count, token cost and latency should each be tracked independently. Collapsing them into a single number hides exactly the trade-offs you need to see when deciding whether a change was worth it.
Run on every change, automatically
The harness must run in CI on every prompt, tool and model change, with a threshold that blocks a merge. A harness that runs manually when someone remembers is a harness that stops running within about three weeks.
Grow the set from production continuously
Every new production failure becomes a new case. This is what keeps the evaluation set representative as usage patterns drift, and it turns each incident into permanent protection rather than a one-off fix.
What to measure beyond final correctness
- Tool selection accuracy: did it choose the right tool, even if the answer happened to be right?
- Step efficiency: how many actions were taken versus the minimum needed for that task
- Recovery behaviour: what happened after a tool returned an error or an empty result
- Groundedness: is every factual claim supported by something the agent actually retrieved?
- Token cost per resolved task, which is the number that actually appears on your bill
- Latency distribution rather than the mean, because the tail is what users complain about
- Escalation appropriateness: did it hand off to a human when it genuinely should have?
How large does the evaluation set need to be?
Smaller than most teams assume, and the obsession with size is usually a way of avoiding starting. Fifty well-chosen cases covering your genuinely distinct scenarios will catch the large majority of regressions. Five hundred cases that are variations on the same three situations will catch fewer, while costing far more to maintain and taking far longer to run.
Coverage matters far more than count. The useful question is not how many cases you have but whether every distinct failure mode you have observed in production is represented by at least one case. When a new kind of failure appears, add it. When a case has never once discriminated between a good and bad version of the system, consider removing it.
Practically, start with whatever failures you already have, even if that is a dozen. A harness with fifteen real cases running automatically on every change is enormously more valuable than a perfect two-hundred-case set that is still being designed six weeks from now.
Part of the AI Engineering Practice cluster · Read the pillar page
More in AI Engineering Practice
AI Engineering Practice
Reducing LLM Costs in Production: What Actually Works
The techniques that genuinely reduce language model spend in production, ranked by impact, and the ones that look promising but rarely move the number.
4 min read
AI Engineering Practice
RAG vs Fine-Tuning: How to Decide
A decision framework for choosing between retrieval and fine-tuning, the situations where each clearly wins, and why most teams should try neither first.
4 min read
AI Engineering Practice
Guardrails for Production AI Agents: A Design Guide
How to constrain what an agent may do, design escalation paths that preserve context, and decide which actions should never be automated at all.
4 min read
Frequently asked questions
What is trajectory evaluation?
Scoring the whole sequence of an agent's actions rather than only its final output. It catches agents that reach correct answers through wasteful or unsafe paths, which output-only scoring cannot distinguish from genuinely efficient behaviour.
How is agent evaluation different from traditional testing?
Traditional tests assert deterministic outputs. Agent behaviour is non-deterministic, so evaluation measures distributions and thresholds rather than exact equality, and it must score intermediate steps because the same output can arrive through very different processes.
Can I use a language model to grade agent outputs?
Yes, and it is common practice, but validate the judge against human labels periodically because judge models drift and have their own biases. Never make a judge model the only gate on a change with real consequences.
When should we build the evaluation harness?
Before optimising anything. It is the first thing a good agent engineer builds, because every subsequent decision is unverifiable without it. Teams that optimise first typically spend months on changes they cannot demonstrate were improvements.
How much does an evaluation harness cost to run?
Each full run costs real tokens, which is why teams sometimes resist automating it. Running a focused subset on every change and the full set nightly is the usual compromise, and it is far cheaper than shipping an unnoticed regression.