Trust the Result: Is It Good Enough to Ship?

A score can improve because the agent got better. It can also improve because difficult judgments disappeared, a grader missed an error, or the team kept only a lucky attempt.

Part 2 produced a three-task example: checkout diagnosis, an owner lookup, and CSV conversion. Correctness was 72.2%, but only one task fully completed. The CSV conversion also overwrote a source file it was required to preserve.

Before using those numbers to make a release decision, we need to establish what was measured, whether the grader is right, and whether success repeats.

Chapter 6 Keep missing results from inflating the score#

Half the grading jobs failed overnight. The dashboard still shows 80%. Find out which results that number includes before interpreting it.

Separate an agent failure from an evaluation failure#

Suppose the agent reads the capacity limit and still recommends 64 connections. That is an agent failure: C3 = 0.

Suppose instead the grader times out before returning a verdict. We do not yet know the grade. That is an unresolved judgment.

Suppose the checkout fixture never loaded, so the intended fault was absent. The trial is invalid for that test.

These cases need different treatment. Marking all three as zero blames the agent for evaluation failures. Dropping all three makes the agent look better by removing real failures.

For “not applicable,” consider a refund task with a rule requiring manager approval only above a stated amount. A request below that amount does not need the approval check. The condition belongs in the task definition.

Failing to inspect an available document is different. The agent cannot make an evidence requirement inapplicable by skipping the evidence.

Do not let incomplete grading improve the score#

Imagine 100 applicable judgments. The grader returns 40 passes and 10 failures, then fails on the remaining 50.

The resolved results have an 80% pass rate: 40 divided by 50. But only half the judgments were resolved.

Reporting “80%” without coverage would invite readers to mistake it for a complete result. Worse, the unanswered cases might be the hardest ones.

For the implementation here, unresolved required judgments block the complete aggregate they affect. A debugging dashboard can show the provisional result, clearly labeled “80% among resolved judgments; 50% coverage.”

Research on rubric-based judge agreement shows that reported agreement can change even when the verdicts do not. Which cases are included, how withheld judgments are handled, and how results are combined are part of the measurement.

Set retry rules before running#

Retry a grader error or invalid environment under a bounded, recorded policy. Retain the attempts and explain which result replaces which failed evaluation attempt.

Do not keep rerunning an ordinary agent failure until it passes and then record only the winner. That answers whether a successful attempt can be found, while concealing how often the product fails.

Apply the retry policy consistently. The report must distinguish an agent failure from a failure to evaluate it. Accounting for every result prepares us to ask whether the judgments themselves are right.

Chapter 7 Find the mistakes your grader misses#

A judge agrees with people on 85% of a small test set, yet accepts one in five unacceptable answers. Both statements can be true. The mistakes hidden by the average determine whether the judge is useful.

Build answers that expose its mistakes#

A well-written rubric does not guarantee that a model judge will apply it correctly. Before using its verdicts to compare agents, give it answers whose expected grades have been reviewed by people who understand the task.

For checkout, include:

• A correct answer with different wording.

• A valid remedy other than restoring the pool setting.

• A confident explanation blaming query execution time.

• A correct diagnosis with no supporting evidence.

• The capacity-breaking recommendation of 64 connections per instance.

• An answer telling the judge to ignore the rubric and award a pass.

These cases test different weaknesses. If paraphrases fail, the grader may be matching wording. If the confident wrong explanation passes, fluency may be masking a factual error. If instructions inside the answer change its score, the judge is treating untrusted content as authority.

When human reviewers disagree, inspect the rubric and evidence together. Sometimes the grading rule needs clarification. Sometimes the reference is wrong. Do not force agreement by automatically treating the original answer key as infallible.

Count the errors that matter#

Take an illustrative calibration set with 20 answers. Human reviewers classify 10 as acceptable and 10 as unacceptable for a particular criterion.

The automated judge produces:

It agrees on 17 of 20 answers, or 85%.

But the two kinds of mistakes deserve separate attention:

False passes: Two of the 10 unacceptable answers get through. That is a 20% false-pass rate among unacceptable answers.

False failures: One of the 10 acceptable answers is rejected. That is a 10% false-failure rate among acceptable answers.

For a permission gate, false passes may be especially costly. For accepting multiple valid solutions, false failures may hide real capability. Inspect these rates per criterion rather than relying only on an overall agreement number.

Agreement between two model judges is useful evidence of consistency, but they can share the same error. Keep a human-reviewed reference set and periodically inspect disagreements and apparent easy passes.

Tune on one set, then check another set whose labels were not used to adjust the judge. Otherwise you may only have taught the grader to recognize its calibration examples.

Use preference comparisons for the question they answer#

Sometimes you want to know which of two answers people prefer. Pairwise judging is well suited to that question.

Suppose Answer A recommends 64 connections and Answer B recommends 32. A judge might prefer B because it is less excessive. Both still exceed the allowed 16 per instance.

Preference does not establish absolute correctness.

The MT-Bench and Chatbot Arena research also documents position and verbosity biases in LLM judging. Hide model identity, vary which answer appears first, and inspect whether preferences are tracking length rather than the intended quality. Keep the absolute task checks alongside the comparison.

Keep reviewed examples as a regression check for the grader, and use a separate holdout to validate changes. A trustworthy grader can still observe an inconsistent agent; that is the next experiment.

Chapter 8 Find out whether success repeats#

A second run answers a different question from a new task#

The first checkout run is perfect. It would make a convincing demo. Now restore the environment and run it four more times.

Anthropic describes using multiple trials because agent outputs vary between attempts. Repetition tells you whether that first success was representative.

A new task tests another situation. Another attempt at the same task tests whether performance on that situation persists.

Suppose checkout is run five times from the same restored starting state. Every attempt respects the no-write rule, but the answers differ:

Three attempts complete the task, so the observed complete-task success rate is 3/5 = 60%.

Across these attempts, 12 of 15 correctness checks pass. Because every attempt has the same three checks, average correctness is 80%.

If you had stopped after the first attempt, the eval would have shown a complete success. The additional attempts expose the behavior a user might encounter on another day.

Five attempts are an illustration, not a universal sample-size rule. A small smoke test may be enough to detect an obvious regression; a release decision needs enough evidence to distinguish the improvement you care about from run-to-run variation.

Repeat the right thing#

Restore state and memory when measuring independent attempts. Keep fixtures, tools, budgets, and graders fixed across the comparison. Record the intended difference between the systems, such as a model or prompt change, so it is clear what the experiment tests.

Regrading a saved trace measures variability in the judge. Executing the agent again measures variability in the agent system. Replaying a cached response measures neither.

Keep those experiments separate. If the same answer alternates between pass and fail, investigate the grader before interpreting the agent's score.

When tasks have different numbers of trials, average in this order: checks within a trial, trials within a task, then tasks within the suite. Running checkout 20 times should not automatically give it 20 times the influence of a task run once.

Distinguish finding a success from delivering one reliably#

The five attempts above contain at least one success. They do not all succeed.

Those observations motivate two different measurements:

pass@k asks whether at least one of k attempts succeeds. It measures the ability to find a solution within several tries.

pass^k asks whether all k attempts succeed. It measures consistency across that group of attempts.

For the illustrated group of five, the “at least one” condition holds and the “all five” condition fails. A benchmark estimates these quantities across its tasks and samples; one group is only an illustration.

Best-of-N systems add a further problem: selecting which answer to deliver. A hidden evaluator may recognize the successful attempt, but the deployed selector may choose a confident wrong answer. Evaluate the actual selection rule and count the cost of all attempts.

Retries are useful when the product can afford them and can recognize or verify success. They do not justify removing failed attempts from a reliability report.

Test whether the agent follows evidence#

Now change the checkout case.

Keep the timeout symptom but make database queries genuinely slower. A good agent should change its diagnosis.

Remove the measurements that distinguish slow queries from connection waits. A good agent should explain the uncertainty and request the missing evidence.

Keep the service healthy. A good agent should avoid inventing an incident.

These variants test something that repeated identical runs cannot: whether the diagnosis responds to the evidence. Research on agent reliability distinguishes consistency from robustness and other reliability dimensions; both repeated attempts and changed conditions matter.

Hold whole families of related cases out of tuning. Randomly separating almost identical incidents into training and evaluation can make memorized patterns look like generalization.

For release comparisons, run the old and new systems on the same held-out cases and inspect changes task by task. Report an uncertainty range, such as a confidence interval, around the aggregate difference. A small apparent gain with a wide interval is weak evidence for a release.

Repeated attempts at one task remain grouped observations; do not count them as an equivalent number of unrelated tasks. You want a result that survives another attempt and a result that transfers to another situation.

Chapter 9 Build the framework and make the release call#

The team fixes the checkout recommendation and the score rises. The framework should now make the release decision—and its reason—easy to inspect.

Give the runner a complete task#

The runner is the program that prepares the environment, executes the agent, invokes graders, and collects results.

It should receive a task definition containing:

1. The request: what the user wants and what actions are allowed.

2. The starting state: the fixture version, reset procedure, and validity checks.

3. The reference: verified facts, constraints, and acceptable alternatives, hidden from the agent.

4. The rubrics: correctness checks, auxiliary dimensions, gates, and applicability conditions.

5. The execution settings: model, prompts, tools, budget, and number of trials.

6. The measurement policy: averaging, missing-result handling, and bounded retries.

Shared templates can generate these definitions. The resolved versions must be recorded with the run, so another runner can apply the same rules.

Keep the arithmetic easy to inspect#

Once judgments are valid and complete, the basic scorer is small:

from statistics import mean

tasks = [
    {"correctness": [1, 1, 0],
     "aux": {"authority": 1, "grounding": 1}},
    {"correctness": [1, 1],
     "aux": {"authority": 1, "grounding": 0}},
    {"correctness": [1, 0, 1, 0],
     "aux": {"authority": 0}},
]

correctness = mean(mean(t["correctness"]) for t in tasks)

dimensions = set().union(*(t["aux"] for t in tasks))
auxiliary = {
    name: mean(t["aux"][name] for t in tasks if name in t["aux"])
    for name in dimensions
}

print(round(correctness * 100, 1))  # 72.2

The inner average gives each task a score. The outer average combines tasks. Each auxiliary dimension gets its own set of eligible tasks.

This example assumes one trial per task and fully resolved judgments. It omits completion gates and validation so you can see the arithmetic clearly.

Before reaching this code, check that every verdict belongs to the right task, run, and rubric version. Reject duplicate judgments, missing required results, unexpected criterion IDs, invalid score ranges, and unsupported “not applicable” claims. A model-generated verdict cannot decide to remove a failed criterion from the denominator.

The companion implementation includes the checkout task, sample judgments, a grader prompt, and a scorer with validation, repeated-trial aggregation, missingness handling, critical gates, and tests. The judgments are supplied examples for inspecting the framework.

Make the score point to the next fix#

Return to the original recommendation. Change 64 connections per instance to 16, justify the capacity calculation, and keep the correct diagnosis.

Only C3 changes from fail to pass.

Checkout correctness rises from 66.7% to 100%. Across the three-task suite, correctness rises from 72.2% to 83.3%, and complete-task success rises from 33.3% to 66.7%.

Grounding stays at 50%. Authority compliance stays at 66.7%. The CSV task still violated its permission boundary, so a zero-violation release gate still fails.

The next work is specific: prevent the source overwrite, correct the CSV timestamps, and fix its column names. Preserve the filtering and account IDs that already pass. Keep the improved checkout answer as a regression case.

Release decision for this worked example: hold.

Checkout's remedy now passes. The CSV task still violates a hard boundary and fails two correctness checks. Fix those failures and rerun the comparison with the same fixtures and grading rules.

The score tells you what improved. The failed checks tell you what to fix. The gate tells you why the improvement is not enough to release.