What the Score Conceals

On 2 July 2019, people visiting websites served through Cloudflare began seeing “502 Bad Gateway” errors. A newly deployed security rule was consuming excessive CPU while inspecting requests. Machines needed for serving web traffic were occupied evaluating the rule.

Cloudflare's engineers disabled the affected firewall component to restore service, then isolated the problematic rule and tested the correction. The outage lasted 27 minutes. The company's postmortem explains an important gap: the tests checked which requests the rules matched, but did not test for runaway CPU consumption.

This was one of several failed safeguards, including limits on CPU use and the scope of the rollout. The change passed the questions the tests asked. The failure lived in a question they did not ask.

A green test is an answer to a particular question

“Does this rule detect the intended pattern?” and “Can we afford to run it on production traffic?” are different requirements. Success on the first does not establish the second. A useful evaluation has to represent the job the system must perform, including constraints that can make an otherwise correct result unusable.

Now imagine evaluating Atlas, an AI assistant that helps an engineer investigate an unavailable website. It sees high CPU and recommends adding servers. The answer is concise, cites a scaling runbook, and contains a sensible-looking checklist.

But busy CPUs do not tell us why they are busy. There may be more requests, or each request may have become more expensive to process. The same symptom can require a different response.

If the evaluator rewards Atlas for writing a complete plan without checking that distinction, an optimizer can improve presentation while leaving the investigation weak. A runbook link is evidence only when its reasoning applies to the case.

In Scaling Laws for Reward Model Overoptimization, stronger optimization of a proxy reward eventually degrades the study's reference reward. The practical lesson is to inspect what the score allows the optimizer to neglect before making the search more powerful.

Write down the trade before choosing the weights

Suppose one version of Atlas diagnoses failures correctly more often, while another includes runbook links more often. A weighted average can rank either version first without changing a single answer.

Links, clear writing, and thorough explanations matter. They do not compensate for an unsupported diagnosis. Combining them into one average quietly makes them exchangeable.

A useful scorecard keeps separate questions visible: does the explanation fit the evidence, does the proposed action address the suspected cause, and does it respect the engineer's approval boundary? A plan to disable a security control also needs to account for the protection being removed. Restoring availability is part of the decision, not the whole of it.

The evaluator needs a difficult example too

Give the evaluator two equally polished answers: “add servers to handle the extra traffic” and “investigate the recent change that increased work per request.” Pair them first with a genuine traffic surge, then with evidence of an expensive new rule. A useful evaluator should change its judgment as the evidence changes.

Recent controlled probes of LLM judges find cases where meaningful changes to a response or rubric fail to produce the expected change in the verdict. That makes this a useful test to run, rather than an ability to assume.

Who Validates the Validators? studies how people's evaluation criteria evolve as they inspect outputs. The team may initially reward a plausible recovery plan and later recognize that it must also check the causal argument and the action's consequences. When the standard changes, rescore the previous version under the same rules.

The first step in improvement is a defensible answer to a simple question: what would have to change for us to call this behavior better?