Build the Score: What Does 72.2% Mean?

The agent found the cause of an outage. It also recommended a fix that would exhaust the database's connection budget. How should the eval score that answer?

In Part 1, we built an invented checkout incident with three correctness checks: cause, mechanism, and a feasible remedy. Four application instances share a budget of 64 connections, so the verified limit is 16 per instance. Our flawed answer gets the diagnosis right and recommends 64 per instance.

Now we will follow the evidence into individual judgments, separate auxiliary dimensions from correctness, and calculate the suite score.

Chapter 3 Choose what code and model judges should check#

An answer recommending 64 connections and one warning against 64 contain the same number. A useful grader must establish what was proposed before checking the arithmetic.

Separate the rule from the mechanism#

The rubric says what should count as correct. The grader applies that rule to a particular execution. The trace is the record of what the agent did: its tool calls, the results it received, and its answer.

Some facts can be checked by code. Others need interpretation. Human review helps establish whether those automated checks agree with the intended standard.

This division is also reflected in Anthropic's guidance on agent evals, which describes code, model, and human graders. A useful framework can combine them within one task.

For checkout, use a semantic judge to interpret the diagnosis and a direct calculation to check a concrete pool recommendation. A semantic judge is a model or person assessing meaning rather than exact wording.

Let code answer questions it can settle#

If the recommendation explicitly proposes 64 connections on each of four instances, its numeric feasibility is straightforward:

available = 100 - 20 - 16
proposed = 4 * 64

within_budget = proposed <= available  # False

Code can also inspect a generated CSV, query the resulting database state, or look for forbidden write attempts in trusted tool logs.

These checks are reproducible and inexpensive. Their limit is the rule they implement. Finding a number in the text does not establish that it is the recommended setting: the answer might say “Do not set this to 64.” Extracting the intended action from free text requires its own reliable method.

A check can also return legitimate partial credit. If a task requires five independent records to be transformed and exactly four are correct, code can calculate 4/5. The fraction has a defined meaning; it was not invented by a judge trying to express a general impression.

Let semantic judges interpret the explanation#

For C2, provide the judge with the user request, the rule, the reference facts, and the agent's actual answer and relevant tool results.

These two answers deserve different verdicts:

Requests wait because the application has too few available connections.

Requests wait because the database executes each query more slowly.

The first matches the evidence. The second contradicts it. Both may use fluent technical language; only one explains this incident.

Require a verdict for each criterion. For the failed remedy, a stored result might look like this:

{
  "rubric_id": "C3",
  "status": "graded",
  "score": 0,
  "evidence_refs": ["e3"],
  "reason": "Four pools of 64 require 256 connections; only 64 are available.",
  "failure_tag": "REMEDY_EXCEEDS_CONNECTION_BUDGET"
}

An engineer can now act on the failure. The criterion is C3, the relevant capacity record is e3, and the failure is an excessive connection allocation. “The answer could be better” would provide none of that direction.

Valid JSON only proves that the result has the right format. Check that the cited evidence exists and supports the judgment. Treat instructions inside the agent's answer or retrieved documents as material to inspect, not commands for the judge to follow.

You can request several verdicts in one model call to reduce cost. Validate that exact setup: a judge assessing several criteria together may behave differently from one assessing each separately.

Check the world when the agent claims to change it#

Consider a booking agent that says:

Your reservation is confirmed.

A text grader may like the answer. But did a reservation appear in the database, for the correct customer and date?

τ-bench uses resulting database state to evaluate task completion. The principle transfers directly: inspect the thing the agent was supposed to accomplish.

State checks and trace checks answer different questions. A final reservation can be correct even if the agent made an unauthorized change along the way. Check the outcome for completion and the trusted action record for permission compliance.

Chapter 4 Separate correctness from grounding and permissions#

A correct fact can still be an unsupported guess#

Suppose an agent reads the configuration and metrics, explains the pool regression correctly, and then proposes the excessive setting of 64 connections.

Its factual explanation is supported. Its remedy is wrong.

Now consider another agent that never inspects any evidence but happens to guess the correct diagnosis and remedy. Its answer may satisfy the factual correctness rules, but we have learned something troubling about how it reached them.

A single correctness score cannot distinguish those situations. Add separate dimensions for the properties you care about.

An auxiliary dimension is a separately reported measurement. It does not mean the property is unimportant or optional.

For the checkout example, we will measure:

Walk through the flawed answer#

Here is the response we will score:

The deployment reduced the connection pool from 16 to 4 per instance [e1]. Requests are waiting to acquire connections; query execution time is unchanged [e2]. There are four instances [e3]. Increase each pool to 64 connections.

Assume the trace confirms that it retrieved those records and attempted no writes.

Why does grounding pass when the recommendation is bad? Because this particular grounding rubric checks the support for the factual explanation. Those facts are supported. The remedy's feasibility is checked by C3.

That boundary must be explicit. If your grounding rubric also requires every recommendation to be justified by the evidence, this answer would fail grounding too. The label “grounding” alone does not tell the reader which definition you used.

For the lucky guess, the factual checks could pass while grounding fails. This gives the two agents different engineering work: improve evidence gathering for the guesser; improve remedy selection for the agent that found the right facts. Calling both answers simply “wrong” would hide that distinction.

Make critical requirements gates#

The user said not to change the environment. Our evaluation treats a prohibited write attempt as a violation, including an attempt blocked by the tool.

Authority therefore has two roles: it is reported as its own score, and it is a completion gate. A gate is a requirement that must pass before an execution counts as successful.

This lets a team see that an agent diagnosed the incident correctly while still rejecting its unauthorized action. Hiding the violation inside a weighted average would let unrelated successes compensate for it.

Similarly, if evidence-backed conclusions are an explicit requirement of your task, put grounding into correctness or make it a gate. Decide before looking at the result.

Keep cost and speed in understandable units#

Record time, cost, tool calls, and recovery attempts as measurements. Avoid automatically translating “fewer tool calls” into “better.”

One agent might take three calls and guess. Another might take five and discover the capacity limit. The second investigation costs more but produces a usable answer.

Report cost and latency alongside success, under a declared execution budget. A composite score can express a real product tradeoff. Keep its components visible and enforce critical gates separately.

Chapter 5 Explain what your score actually means#

Two dashboards can turn the same nine verdicts into 66.7% and 72.2%. Before calling either one a bug, find out what each calculation gives equal weight.

Start with one execution#

The checkout answer passed two correctness checks and failed one:

Correctness = (1 + 1 + 0) / 3 = 66.7%.

Read this as “two of three correctness obligations passed.” It is not a 66.7% probability that the task succeeded.

If completion requires all three checks and the authority gate, the answer is incomplete. The user still lacks a feasible recommendation.

Both measurements are useful. The correctness average shows partial progress. Complete-task success tells you whether the result met the full acceptance rule.

Add two concrete tasks#

A suite is a collection of tasks. To see how a framework combines their results, add two illustrative tasks.

Task B is a service-owner lookup. The user asks which team owns a service and which on-call queue to contact. The answer names both correctly, so its two correctness checks pass. It supplies no supporting citation, so grounding fails. No prohibited write occurs.

For this task, grounding is reported separately and is not a completion gate. Task B therefore completes. If the product requires a cited source, its acceptance rule should change accordingly.

Task C is a CSV conversion. The user asks the agent to write a new CSV containing only active accounts, convert timestamps to UTC, preserve account IDs, and use the required column names. The source must remain unchanged.

The output filters active accounts and preserves IDs, but fails the timestamp and column-name requirements. It also overwrites the source. That gives it two of four correctness checks, plus an authority violation. No explanatory prose was requested, so a prose-grounding rubric was never assigned.

The correctness results are:

First average the checks within each task. Then average those task scores:

Suite correctness = (2/3 + 1 + 1/2) / 3 = 72.2%.

Why not count all the passes together#

There are six passing checks out of nine. That produces 66.7%, which differs from 72.2%.

The disagreement comes from the unit that gets a vote.

Pooling checks gives every check one vote. Task C then has four votes while Task B has two. Averaging task scores gives each task one vote, regardless of how many checks its author wrote.

Use the latter when each task should contribute equally. If production traffic or business importance should determine the weights, declare those weights explicitly. Do not let the length of a rubric make that decision accidentally.

Also inspect important groups of tasks separately. A high average across frequent, easy lookups can conceal poor performance on less frequent actions.

Calculate each auxiliary dimension separately#

Use the same principle for grounding and authority, but include only the tasks to which the dimension applies:

Authority has three results:

Authority compliance = (1 + 1 + 0) / 3 = 66.7%.

Grounding has two:

Grounding = (1 + 0) / 2 = 50%, with coverage of two out of three tasks.

The missing grounding entry for C is neither a pass nor a failure. It was excluded by the task definition before the agent ran.

If a dimension has multiple checks within one task, average those checks first, then average the eligible task scores. That keeps a task with a longer grounding rubric from receiving extra weight.

Report completion alongside the averages#

Only Task B passes every correctness check and its authority gate.

Complete-task success = 1/3 = 33.3%.

The dashboard now says something precise:

• Correctness obligations are partly satisfied: 72.2%.

• Complete-task success remains low: 33.3%.

• One of three executions violated its authority constraint.

Change a result and watch the scores#

Correct the checkout remedy, then preserve the CSV source. Notice that fixing the permission violation does not repair the CSV timestamps or column names. Change the weighting without changing any answers to see how the headline moves.

Correctness72.2%Equal task weights

Complete tasks33.3%1 of 3 tasks

Authority compliance66.7%2 of 3 tasks

Task	Correct checks	Complete
Checkout	2 of 3	No
Owner lookup	2 of 2	Yes
CSV conversion	2 of 4	No

Grounding: 50.0% · 2 eligible tasks of 3.

Authority gate: failed — CSV source overwritten.

Fixing checkout's recommendation improves correctness and completion. It does not erase the CSV task's unauthorized overwrite; a zero-violation release gate still fails until that behavior changes.

We have calculated these scores assuming the judgments are complete and correct. In Part 3, we challenge those assumptions before making the release decision.