Building an Agent
That Improves Its Own Harness

Imagine a coding agent encountering the same failure for the third time. A build produces thousands of lines of output. The useful diagnostic disappears from the model’s working context. The agent edits the wrong file, retries the build, and runs out of time.

A developer adds another instruction: “Pay attention to compiler errors.” The next run still fails.

The instruction asks the model to compensate for an information problem. The harness could preserve the complete log, surface the relevant diagnostics, and make the rest easy to inspect. That would change what the agent can actually work with.

Now suppose the agent can propose that change, test it, and preserve it for future runs. Suppose it also remembers which experiments were useful and which apparently promising fixes caused regressions. We have the beginnings of a system that learns how to improve its own operation.

The useful question is what an execution has earned the right to change. A single success might justify another experiment. A shared default needs evidence from other tasks. And an improvement to the search process needs its own test: does it make later improvements easier to find?

Decide what should persist#

The harness is the software around a model: context construction, tool interfaces, state, recovery, and execution control. Harness learning can change those components while keeping the model’s weights fixed. The Self-Evolving Agent Harness paper makes persistence central to the definition: accepted revisions affect later runs, and an update mechanism remains available for further revisions. Its theoretical guarantees are conditional on assumptions such as independent validation; they do not make an arbitrary editing loop reliable.

For implementation, distinguish three kinds of state. They have different purposes and need different evidence.

A retry can repair the current task without changing future behavior. A procedure can help future tasks without teaching the optimizer anything. Improvement experience changes which patches and tests the optimizer considers next.

This extends the question in The Quality Flywheel: what useful knowledge survives the experiment? Here, some of that knowledge concerns how to run the next experiment.

Locate the change#

Several approaches can improve an agent while keeping model weights fixed. The distinction is what they retain and where that retained material acts.

These categories overlap: prompts belong to harnesses, and lessons can guide code edits. GEPA’s current implementation also supports code and configuration; the table refers to its original prompt study. Better task results support a better agent. Better subsequent search under a matched budget supports a better improver.

Harness² connects task-local adaptation with an accumulating improvement library. Each new task starts from a common base harness; earlier components and lessons guide its adaptation. Model weights, stage instructions, and the decision rule remain fixed. On WorkBuddy-Bench, Gemini 3.5 Flash under OpenCode improves from 72.4 to 77.5 with one improvement step; removing the library reduces that result to 74.4. This supports a contribution from accumulated experience, without establishing indefinite acceleration.

Harmony Harness tackles competing objectives through functionality and security evaluation, modular changes, and scope-aware composition. Its results motivate checking what an average conceals. Its small evaluation splits—30 tasks each for SecRepoBench and 28 for BaxBench—and narrow observed trade-off frontier also limit the conclusions.

Choose where improvement happens#

Harness² adapts around the task currently being solved, using execution contrasts when a reliable verifier is unavailable. The design below instead searches offline and uses trusted tests to admit shared versions. This is a deployment choice, not a reproduction of that paper.

For the compiler example, start offline: build output is reproducible, functional tests exist, and the failure recurs. A successful adaptation for one repository can enter the candidate pool, but it needs broader evidence before becoming a default.

Development results feed the improver. The audit stays outside that feedback loop. If a promotion failure is later opened for diagnosis, mark those cases as exposed and validate the revision on fresh cases.

Turn the failure into a patch hypothesis#

Start with the trace. Did the tool return the diagnostic? Did an output limit remove it? Did compaction discard it later? Did the model see it and misunderstand it? Each explanation points to a different component. A prompt cannot recover evidence that never reaches the model.

Write a hypothesis that can fail: “On long build outputs, preserving structured diagnostic spans will increase repair success because the agent can inspect the first relevant failure without losing access to the complete log.”

The candidate might modify a tool wrapper. It saves the original output in a task-scoped artifact, extracts diagnostics with locations, and returns a compact receipt. The complete log remains accessible under the same task permissions. An illustrative interface could return:

{
  "exit_code": 1,
  "full_output": "artifacts/build-17.log",
  "diagnostics": "artifacts/build-17.diagnostics.json",
  "preview": "src/parser.cc:148: incompatible return type",
  "preview_complete": false,
  "extraction_status": "partial"
}

These fields describe different facts. A complete raw log does not imply complete extraction. A short preview does not imply there is only one error. If extraction fails, the receipt must expose that failure and point to the raw output.

This builds on progressive disclosure for agent tools: keep large evidence available, and make the next useful read easy to discover.

Keep the first patch small enough to diagnose. For this experiment, change only the output adapter. Hold prompts, model, and retrieval policy fixed. Coordinated edits can come later, with comparisons that establish why the combination helps.

Give the improver a search procedure#

The improver can be the same generalist model, running in an isolated workspace. It needs a bounded editing task, relevant evidence, and an explicit selection rule.

Freeze the current production version, the incumbent, for one search cycle. Choose a recurring failure family by frequency and impact. Retrieve lessons that match its symptoms, component, and toolchain, including failed attempts. Require the proposer to return a hypothesis, applicability conditions, a patch, and a predicted regression to check.

Here is pseudocode for a deliberately small search. All candidates branch from the same incumbent; a rejected patch can inform a later proposal without entering production.

base, library = snapshot(incumbent, experience)
budget = search_budget_after_reserving_final_checks()
journal, candidates = [], []

while budget.can_start_attempt() and not stop_rule(journal):
    failure = choose_failure(development_traces, journal)
    lessons = retrieve(library, journal, failure)
    proposal = propose(base, failure, lessons, editable_files)

    candidate = apply_in_isolated_workspace(base, proposal.patch)
    checks = validate(candidate, allowed_diff, schemas, smoke_tests)
    if not checks.pass:
        journal.append(proposal, checks)
        continue

    report = compare_on_reset_tasks(
        base, candidate, development_tasks, protocol
    )
    journal.append(proposal, report)
    if protocol.development_eligible(report):
        candidates.append(candidate, report)

freeze(journal)
candidate = select_by_declared_rule(candidates)
return candidate, journal  # No candidate means keep the incumbent.

For example, allow at most eight proposals and stop after three attempts without improving the best eligible development score. A separate time and token cap covers every attempt, including invalid patches and retrieval; reserve budget for the final checks. These are starting settings, not research-derived constants.

Validate the actual diff against the allowed files, check that the receipt schema still parses, and run focused contract and smoke tests. The proposer’s description of what it changed is insufficient. Keep evaluator code, hidden tasks, permissions, and release controls outside the candidate workspace.

Declare selection before searching: require the minimum development gain and all regression and cost limits, then rank eligible candidates by task score. Break ties by lower execution cost, then smaller patch size. This simple policy may miss useful combinations. It makes the first implementation inspectable; an archive or broader search can be added when that limitation becomes measurable.

Comparisons need identical starting repositories, inputs, tool versions, and resource limits. Reset between attempts and randomize run order to reduce time-dependent bias. Recorded responses cannot substitute for a resettable environment when a new policy takes different actions. Use trusted outcome tests; candidate-generated tests can diagnose a defect but cannot independently validate the candidate.

Each journal entry records the parent version, patch, library snapshot, model, environment, task IDs, traces, outcomes, and total cost. Failed attempts remain available to the next proposal. The selected candidate goes to a separate promotion evaluator.

Let the first candidate fail#

Suppose generation one produces an error-only output filter. Evaluate it on 120 development tasks, with five runs per task. The following numbers are invented to illustrate the decision; no implementation was measured. Each family contains 40 tasks, giving 200 runs per family for each harness. A family is a task slice: a subset with a shared property worth checking.

The aggregate improves from 65% to 73%, but warning-dependent repairs get worse. If the declared limit permits at most a two-point decline in an important slice, this candidate is ineligible. A useful idea has produced an unacceptable default.

Inspect the failing traces before assigning a cause. The filter may have removed a useful warning, but run variability or another adapter bug could also explain failures. The measurement is evidence of a regression; warning loss is a hypothesis to test.

Calculate each task’s average score across repetitions, subtract its incumbent average, and average those differences:

task gain = mean(candidate runs) − mean(incumbent runs)
overall gain = mean(task gains)

Repetitions expose execution variability. They do not turn 120 distinct tasks into 600 independent task examples. Estimate uncertainty with a method that preserves task grouping, such as resampling tasks with all their runs together. If repositories or task templates introduce further dependence, preserve that grouping too.

Set the primary outcome, slices, acceptable regressions, and comparison budget in advance. Use declared workload weights if the task mix differs from deployment traffic. Development scores guide search; fresh promotion evidence must resolve uncertainty before release.

Make the second proposal remember the first#

The useful lesson is more specific than “shorter output is better.” Store the measured regression separately from its suspected cause.

# Hypothetical development record, not a framework schema.
trigger: long compiler output obscures actionable diagnostics
change: error-only diagnostic preview with raw-log access
observation: warning-dependent pass rate fell by 11 points
hypothesis: required warnings were omitted from the preview
next_check: trace whether omitted warnings explain failed repairs
scope: tested compiler versions and output formats
status: rejected_for_release
provenance: [parent_version, patch_id, task_ids, trace_ids, results]

Generation two starts from the unchanged incumbent. Its retrieval step finds this record before proposing another filter. The improver examines a failing trace and checks whether the relevant warning existed in the raw log, disappeared from the preview, and was never fetched. A controlled comparison that restores that warning can test the proposed mechanism.

It now proposes an adapter that preserves errors, warnings, and attached diagnostic notes with their locations. The preview has a size limit; a structured index makes omitted diagnostics discoverable. The receipt distinguishes a truncated preview from incomplete extraction, and raw-log access remains available when parsing fails.

The earlier failure also changes the tests chosen before the expensive task runs:

These checks test the adapter’s behavior. The next paired task comparison tests whether the agent actually uses that behavior to repair programs. Keeping a warning available does not ensure the model reads it.

This completes the second proposal, not a fictional success story: the new candidate still needs measurements. It can improve, tie, or regress again. The recursive step is already visible—the first experiment changed both the next patch and the next test plan.

Version the experience library separately from the execution harness. Freeze the historical library for each search cycle and retain that cycle’s development journal. Consolidate duplicate lessons, preserve counterevidence, and revalidate stale entries after model or tool changes. Keep task artifacts and retrieved lessons within their original access boundaries.

Promote the version that survives new evidence#

Freeze the selected patch and its applicability rule. Run it against the incumbent on fresh promotion tasks, using the outcome, uncertainty method, slice limits, and resource budgets declared before search. For example, require the lower confidence bound on overall gain to exceed a chosen minimum, while the uncertainty bounds for important slices rule out a loss larger than the allowed decline. Choose the error rate and comparison budget together; repeatedly testing candidates until one passes defeats the check.

Use historical failures as a regression suite alongside fresh tasks. Separate related repositories and near-duplicate tasks across partitions where possible. Route a specialized adapter using information available before execution, such as compiler identity and supported output format. A rule that depends on knowing whether a task will pass is unusable.

A passing candidate enters a canary, a limited rollout with predefined success and rollback criteria. Record the harness version on every run. If the rollout passes, update the production version pointer; otherwise restore the incumbent. Rollback affects future runs and cannot undo external writes, so action-taking workflows need isolated probes before live exposure.

An audit evaluates the frozen selection process on a further untouched set. If its outcomes change the design, that set has become development evidence and a later claim needs fresh audit data. Record rejected searches as well as releases; reporting only successful searches conceals the cost and failure rate.

For our example, the release record would connect the second adapter to its development report, fresh promotion report, applicability rule, canary decision, and rollback target. Until those reports exist, it remains a candidate.

Test whether the improver improved#

A better deployed harness establishes one result. A better improver requires another experiment.

Start each condition from the same base harness, proposer model, development tasks, and total search budget. Use only experience from earlier, separate tasks:

Keep retrieval limits and accounting consistent. Run multiple search trials, then evaluate frozen selections on unseen tasks with the same protocol. Compare C with B to estimate the added value of experiment lessons, rather than attributing inherited code to a better search process. Compare B with A to estimate component reuse. These are controlled comparisons, not guarantees of transfer.

Measure final task quality, regressions, total search cost, and the number of proposals needed to reach a predefined target. Include searches that never reach it; report their spent budget and failure to qualify. Additional attempts with the original harness and a static curated skill are useful simpler baselines under the same total budget.

Count proposal, probe, judging, retrieval, and memory-maintenance costs. An offline search costing 60 units and saving 0.2 units per later task breaks even after 300 tasks, assuming equal quality and no additional maintenance cost. That illustrative calculation fails if the change expires before 300 uses, or its quality loss outweighs the savings.

Repeat across fresh batches and after distribution changes. Task-order seeds do not necessarily control model sampling; Harness²’s experiment guide makes that distinction explicit. Record both. When gains flatten, revisit the search space, retrieval, and workload. Another recursive round may not help.

Build the smallest loop that can earn a release#

Build one loop around one recurring failure family. The first useful milestone is a complete, inspectable chain: a trace, a proposed patch, a failed comparison, a lesson that changes the next proposal, and a candidate that survives fresh evaluation.

The implementation can consist of a trace store, a versioned candidate workspace, a runner that resets task environments, an evaluator, an experience index, and a release pointer. It does not require permanent specialist agents. As argued in Before You Build Another Agent, keep expertise reusable and delegate bounded work when separate execution earns its cost.

Choose components you actually control. If an underlying SDK owns compaction and its internal execution loop, begin with exposed context files, tool wrappers, skills, or supported hooks. A proposal to edit inaccessible internals is not an executable improvement.

Expand the editable surface only when the existing loop finds useful gains and catches meaningful regressions. If the improver later proposes changing its own retrieval or proposal policy, evaluate that policy against the previous one from a common base with matched search budgets. Keep the external promotion standard fixed.

The next time the compiler failure appears, the agent should inherit a better way to inspect the evidence. The next time the optimizer proposes an output filter, it should inherit the warning regression and the test that caught it. That is a concrete form of progress: both the work and the method for improving the work have learned from the failure.