Make the Winner Earn Its Release

Giving Atlas a clearer view of CPU use and recent changes looks promising. In our hypothetical evaluation, the team tries several versions of its instructions and chooses the one with the highest development score. It now recognizes the expensive processing behind the original outage. Should it replace the current Atlas?

Not yet. The team selected this version because it looked good on cases it had already inspected. Some of that advantage may come from learning those particular examples, and some may come from favorable runs. The selection process is part of the evidence.

Compare changes on the same work

Run the current version and the candidate on the same saved incidents, with the same underlying records and comparable limits on time and tool use. Record their disagreements, not just their separate totals. The revealing cases are the ones that improved and the ones that got worse.

Accuracy rises from 80% to 86%. The candidate fixes fourteen recommendations and breaks eight. “Six points better” is true, but incomplete.

Look at the eight failures. They involve genuine traffic spikes. The candidate has picked up a shortcut: when CPU use is high, roll back the latest deployment. That helps when a recent change introduced expensive processing, but it can undo a healthy release while leaving a traffic surge unresolved. The average rewards the shortcut because change-related failures are common in this test set.

The distinction is straightforward once someone reads the disagreements. Similar symptoms can have different causes. High CPU use tells the engineer where to investigate; it does not, by itself, tell them whether to add capacity or reverse a change. A comparison should help the team discover this distinction, not merely produce a number that clears a release gate.

One hundred incidents also do not settle the size of the gain. Incidents differ in difficulty, repeated attempts can vary, and several cases may come from the same underlying outage. Adding Error Bars to Evals explains why uncertainty and experimental design belong in evaluation. Replaying one outage many times should not masquerade as independent evidence from many outages.

Give discovery and confirmation different jobs

Development cases are for investigation. Inspect them, argue about them, and use their failures to generate candidates. Confirmation cases answer a different question: does the selected change work on examples it was not tailored to?

A famous outage also creates a shortcut: Atlas may remember the published answer. Use unfamiliar variants, change the cause, and provide only evidence available at the decision point. Removing the company name alone does not make a case independent.

For Atlas, fresh cases should include genuine traffic surges, expensive processing introduced by a change, and outages where the available evidence does not yet settle the cause. Keep them out of the feedback used to revise Atlas. If their failures become the basis for another edit, they have become development material; the next confirmation needs independent evidence.

Research on adaptive data analysis explains why reusing data to choose the next hypothesis can undermine ordinary statistical guarantees. There is no magic in the word “holdout.” Repeatedly checking a private score and changing the system in response still lets information about that set guide the search. The more candidates a team tries, the more care it owes to how the winner was chosen.

Match the test to the responsibility

Atlas currently advises an engineer, who can reject a bad recommendation. Giving Atlas permission to act unattended changes the requirement even if its model and prompt stay identical.

If each attempt independently succeeds with probability 0.8, five attempts have a 99.97% chance of containing at least one success, but only a 32.77% chance of all succeeding. These are illustrative probabilities, not Atlas's measured performance. The first can help someone who can recognize and select a good answer. The second resembles a system expected to complete a sequence of jobs reliably. Real attempts can be correlated; repeating the investigation may simply repeat the same mistake.

τ-bench brings consistency across trials into agent evaluation. A more recent preprint, Locating Hidden Failures Makes Long-Horizon Agents More Reliable, examines errors inside sequences of actions that a final score can miss. Both reinforce a lesson for Atlas: an investigation can contain an unsupported assumption even when its final recommendation happens to help.

The team removes the automatic rollback shortcut. Atlas now compares request volume with processing cost and looks for evidence linking a recent change to the failure. Fresh incidents test whether it distinguishes those explanations and recognizes when more evidence is needed. Repeated trials and human review support a limited rollout, with an engineer approving any proposed action.

The release decision includes diagnostic quality, inappropriate recommendations, and response time. Real use can reveal conditions saved examples did not cover: a different traffic pattern, a new component, or an incomplete monitoring record. Keep the previous version available and check whether the expected improvement actually appears.

Offline comparison and live observation complete each other. The comparison helps isolate a change. Real use reveals whether the change mattered where Atlas has to work.