Turn a Failure into an Experiment
A wrong answer points to a problem. A targeted change tests the explanation.
Part of The Quality Flywheel ↗Chapters in this collection 02 / 05
Atlas sees an unavailable website and high CPU. It recommends adding servers. “Reason more carefully” is an easy response to that weak diagnosis. It is not yet an engineering plan.
The useful question is what would let Atlas distinguish a traffic surge from an expensive new operation. Did it see the request volume? Could it inspect recent changes? Did it know where the CPU time was going?
Different explanations need different repairs
Suppose Atlas's monitoring summary includes error rate and total CPU use, but omits request volume and the breakdown of CPU work. Telling it to inspect that evidence will not help unless the system makes the evidence available.
| Possible problem | A useful test |
|---|---|
| Atlas cannot see the relevant evidence. | Add request volume, recent changes, and a CPU breakdown to the saved case. |
| The evidence is available, but Atlas does not inspect it. | Require the existing checks before choosing a remedy. |
| Atlas sees the evidence but interprets it poorly. | Keep the inputs fixed and change the decision procedure. |
These are proposed experiments for Atlas, not a reconstruction of Cloudflare's response. They separate a model problem from a missing-information problem. Both can produce the same confident recommendation.
Start by replaying the saved case with the missing evidence supplied. If Atlas now identifies an expensive operation, the information gap becomes a stronger explanation. If it still insists on a traffic surge, the next experiment should examine how it uses the evidence.
Make the explanation risk being wrong
A production symptom is not a cause. High CPU tells us the processor is busy. It does not tell us whether the work is necessary.
To test a suspected expensive rule, reproduce the workload in an isolated environment and compare processing with and without that rule. Keep the requests and resources comparable. If the extra cost disappears when the rule is removed, the result supports the diagnosis. If the cost remains, the investigation has more work to do.
That test concerns the website. A separate comparison concerns Atlas: does giving it the relevant evidence improve its diagnosis across more than one case? Repairing the service and improving the assistant are related tasks with different evidence requirements.
AgentRx applies a related principle to agent diagnosis: it checks execution steps and gathers evidence around failures. A record of the steps helps propose a cause. A targeted intervention tests whether that proposal is useful.
Find the nearest case where the lesson fails
“Roll back the latest change whenever CPU is high” is a tempting lesson. It also fails when a website has simply become popular and its latest deployment is unrelated.
| Website is slow, and… | What the evidence suggests |
|---|---|
| Request volume rises; work per request stays similar. | Investigate capacity and downstream limits before adding resources. |
| A new operation uses much more CPU per request. | Isolate that operation and assess a targeted rollback or correction. |
| Requests spend most of their time waiting for the database. | Investigate the database bottleneck; more web servers may increase pressure. |
These examples are useful because the visible complaint—“the website is slow”—does not determine the remedy. Atlas must use the evidence that separates the cases.
A useful failure review leaves behind an explanation someone can challenge, a change someone can test, and a counterexample that prevents the lesson from becoming too broad.