Search Beyond the Prompt

Atlas sees high CPU use and failed requests, then recommends adding servers. That advice could help during a genuine surge in traffic. It could also miss the problem completely: the website may be receiving an ordinary number of requests, with each request suddenly taking far more work to process.

The team could add “investigate carefully” to the prompt. But an instruction cannot reveal information the assistant never receives. If Atlas sees only total CPU use and the error rate, it cannot reliably distinguish more work arriving from each piece of work becoming more expensive.

Every hill has a boundary

Hill climbing starts with a candidate, tries nearby changes, and retains an improvement. For Atlas, a nearby change might be a sentence in the prompt. The score defines uphill; the allowed edits define which neighboring candidates exist.

A local peak means that none of the nearby changes being tried looks better. It does not mean the best system has been found. Rewording the instruction may stop helping because the limiting factor is missing evidence. Showing what is consuming CPU and what changed before the outage opens up a different kind of improvement.

There are practical reasons to search locally: small changes are cheap, easier to inspect, and easier to reverse. The mistake is treating the best result within one search space as evidence that the rest of the system no longer matters.

Use the failure to propose a change

A score can rank a candidate against the current version. It says little about what to try next. A record showing “Atlas assumed traffic increased without checking request volume” gives the team something specific to change.

That distinction runs through several strands of research. DSPy makes language-model programs modular enough to optimize their components. TextGrad propagates natural-language feedback through a computation. Its “gradients” are critiques, not exact derivatives; their value lies in proposing useful edits that can be tested.

GEPA uses execution feedback to evolve prompts and retain candidates with complementary strengths. This matters when a single average hides useful differences. One version of Atlas might handle traffic spikes well; another might investigate recent changes more effectively. Studying both can suggest a better revision than keeping only the overall winner.

For Atlas, the team puts three candidates on the table:

1. Prompt: require evidence for the proposed cause before recommending a fix.
2. Tools: show request volume, a breakdown of CPU use, and recent changes alongside the error rate.
3. Procedure: distinguish an increase in incoming requests from an increase in the work needed to process each request, then choose an intervention supported by that distinction.

Test them separately before combining them. If richer evidence fixes the cases while the prompt alone does not, the evidence points to missing visibility into the system. If Atlas still ignores the evidence until it receives a clear procedure, the combination deserves its own comparison. Changing everything at once makes it harder to know which part was worth keeping.

Keep a cost for being clever

It is possible to make the investigation unnecessarily elaborate. Atlas could inspect every service and produce pages of analysis while the website remains unavailable. A better procedure would first check the few observations that distinguish the leading explanations, then investigate further when those checks leave uncertainty.

The search therefore needs more than a quality score. It needs a reasonable response time, tool costs, and limits on what the assistant may change. AI Agents That Matter argues for evaluating agent performance together with its cost and for more careful benchmark design. Here, an improvement means giving the engineer better advice soon enough to use it.

Better feedback can help a system propose better changes. But a convincing explanation of why an edit should work is a reason to run the experiment, not a substitute for its result.