Design the Test: What Counts as Correct?

The agent finds the cause of an outage, explains it clearly, and recommends a fix. A reviewer might approve the answer after a quick read.

Then someone checks the capacity calculation. The proposed fix would overload the database.

The diagnosis was right. The recommendation was wrong. The answer cited real evidence and respected its permissions. The team needs to preserve those distinctions to decide what to fix.

We will build an eval around this invented checkout incident, then test its scoring rules with owner lookups, CSV conversion, and booking examples. The final decision will be whether to release an improved agent whose headline score looks better.

Chapter 1 Reproduce the problem before writing the test#

Give the agent a problem to solve#

Imagine an application called checkout-api. It runs on four instances. Each instance keeps a pool of open database connections, which requests borrow when they need to query the database.

After a deployment, checkout requests start timing out.

The user asks:

Checkout requests started timing out after the 14:00 deployment. Investigate checkout-api in staging, identify the cause, and recommend a fix. Do not change the environment.

This prompt gives the agent a symptom, a place to investigate, an expected result, and an action boundary. It leaves the diagnosis for the agent to discover.

Compare it with “Investigate the connection pool regression.” That version has already supplied the important answer. It might be useful for testing a repair procedure, but it is a weak test of diagnosis.

Know the answer for a defensible reason#

The evaluator has a controlled reproduction of the incident. The deployment reduced the connection pool from 16 to 4 per instance. Requests now spend longer waiting to borrow a connection. Once they obtain one, their database queries run at the usual speed.

Restoring the previous setting resolves the problem in this reproduction.

That distinction between waiting for a connection and executing a query matters. “The database became slower” would sound plausible, but would explain the wrong part of the delay.

The evaluator records three pieces of evidence:

These are the reference facts: the evidence against which an answer will be judged. The agent can discover them through its tools. The evaluator's answer key and grading instructions remain outside the agent's accessible files.

Now work out a feasible remedy. The database allows 100 connections. Subtract the 20 used elsewhere and the 16 kept in reserve. That leaves 64 connections for checkout.

Four instances share that budget, so each can use 16.

Available per instance = (100 − 20 − 16) / 4 = 16 connections.

A recommendation to restore 16 fits this case. A recommendation to use 64 per instance would allocate 256 connections and fail the capacity constraint.

Notice what we established before involving a judge: the actual fault, the observable symptoms, and a constraint that distinguishes a good remedy from a bad one.

Make the starting conditions repeatable#

A fixture is the prepared test situation: the files, service state, records, and tool responses the agent starts with. A trial is one attempt at solving that situation.

Restore the fixture before each independent trial, then check that the intended fault is present. Otherwise, the second agent may inherit the first agent's repair and face an entirely different problem.

Saved tool responses are useful for a diagnosis test: they are inexpensive and repeatable. They cannot demonstrate that a proposed repair works. If the task asks the agent to change a system, use an isolated executable environment or a simulator whose behavior has been validated.

Historical incident reports are useful starting material too. Verify their diagnoses against the test environment; copying an old explanation into an answer key does not make it a verified answer for this test.

Package the prompt, fixture version, reference facts, rubrics, and execution settings together. If shared grading templates change, pin the version used by each task. Otherwise a score can change merely because the runner loaded a different rule.

Apply this to your task: write down the fault, the evidence available to the agent, and one tempting answer that the evidence rules out. If you cannot explain why that answer is wrong, the reference is not ready for automated grading.

Chapter 2 Write rules that distinguish a good answer#

Replace a vague rating with answerable questions#

A rubric is a written rule for judging one aspect of the result.

“Give a helpful and accurate answer” is too broad. Two reviewers could agree that the diagnosis is good and still disagree about whether the bad remedy deserves a score of 3, 6, or 8.

For checkout, separate three obligations:

Each check gets a pass or a fail. We will represent those as 1 and 0.

This makes the failure useful. If C1 and C2 pass but C3 fails, the next improvement is about choosing a remedy. We do not have to infer that from a holistic rating.

Write a boundary the grader can apply#

Before reading the verdicts, decide which of these recommendations you would let an engineer act on. All three respond to the incident; they make very different commitments.

Restore the pool to 16 per instance. Four pools use 64 connections, leaving the allocations for other workloads and reserve intact.

This passes C3.

Increase the pool to 64 per instance.

This fails C3, because four pools would require 256 connections.

Increase the pool, add retries, restart the database, or scale the application.

This also fails the rubric as written. It offers a menu without selecting and justifying a feasible remedy. Mentioning a potentially useful action is not the same as recommending a defensible fix.

These examples belong with the rubric. They give the grader a concrete boundary: a feasible recommendation passes; an excessive setting or an unchosen list of fixes fails.

A different solution can still pass. For example, limiting incoming concurrency could reduce pressure on the pool. The answer would need to explain the throughput tradeoff and how to validate that the policy addresses the incident. The reference should describe acceptable outcomes and constraints, rather than demand a particular sentence.

Do not reward imitation#

Suppose the reference says:

Requests are blocked while acquiring a connection.

The agent says:

Queries are waiting for a free slot in the application's database pool.

Those can express the same mechanism. Comparing the strings exactly would reject useful variation.

Now suppose the agent says:

This is not a connection pool problem.

A keyword check for “connection pool” could accept the opposite claim.

Exact matching works well for fixed identifiers and required literal outputs. For explanations, grade the meaning. For structured output, validate its structure and values instead of comparing formatting.

The same principle applies to tool use. If the agent establishes the facts through SQL, it should not fail merely because the reference trace used a metrics API. Use traces to diagnose how an agent worked. Require a particular sequence only when the sequence itself matters, such as obtaining approval before a write.

Choose the scoring scale deliberately#

Binary checks are a useful default for obligations with clear acceptance conditions. Write the boundaries clearly and check that reviewers apply them consistently.

For a quality with degrees, use an anchored scale. A communication rubric might define:

• 1: The main conclusion is missing or hard to locate.

• 2: The conclusion is present, but observations and recommendations are mixed together.

• 3: The answer states the conclusion first and clearly separates evidence from next steps.

That is easier to apply than “Rate clarity from 1 to 3.” LLM-Rubric explores calibrating multidimensional ratings against human judgments for dialogue quality. The relevant lesson is that a scale needs validation, not that every task should use the same scale.

For the worked checkout eval, we will keep binary checks so the arithmetic stays easy to inspect.

Here is a way to improve the score without improving the answer. Split C1, the cause check, into five overlapping checks. Leave the mechanism and remedy checks unchanged.

The same answer now passes six of seven checks: 85.7% instead of 66.7%. The diagnosis has gained extra votes; the bad remedy is still bad.

Choose the distinct obligations first. Then attach a pass example and a fail example to each rule. A longer rubric should make the judgment clearer, not quietly make a failure cheaper.