How to Write and Score
an Agent Eval
From a plausible wrong answer to a defensible release decision: nine chapters on writing tests, calculating scores, and checking whether success repeats.
The agent finds the cause of an outage, explains it clearly, and recommends a fix. A reviewer might approve the answer after a quick read.
Then someone checks the capacity calculation. The proposed fix would overload the database.
The diagnosis was right. The recommendation was wrong. The answer cited real evidence and respected its permissions. The team needs to preserve those distinctions to decide what to fix.
We will build an eval around this invented checkout incident, then test its scoring rules with owner lookups, CSV conversion, and booking examples. The final decision will be whether to release an improved agent whose headline score looks better.
This guide follows that incident from the first task definition to the release decision. Nine chapters are grouped into three installments, each with worked answers, the reasons behind the recommended method, and the failure modes of common alternatives.
Start with Design the Test if you are authoring an eval. Go to Build the Score to inspect correctness, auxiliary dimensions, and the interactive scoring example. Use Trust the Result when validating a grader or deciding whether an improvement deserves release.
The Python companion contains a scoring contract, supplied example judgments, validation, and tests. The incidents and measurements in this guide are illustrative; they are not measured model results.
Read the example answers before their verdicts and decide how you would grade them. The disagreements reveal the design choices.