Checkpoint Replay
for Long-Horizon Agent Evaluation
Evaluate a 24-hour soak at recorded checkpoints. Restore the evidence and agent state available at each moment, then test the decisions.
An order-processing service passes its first hour under load. Six hours later, connections are accumulating. At twelve hours, requests begin waiting for the pool. By hour twenty-four, checkout is returning 5XX responses and the storefront is timing out.
Now change one instruction in the reviewing agent. Must the entire soak test run for another day to find out whether that helped?
Long-horizon evaluation has an environment-time problem. Some failures need a full traffic cycle, repeated batch jobs, or slow resource accumulation to become visible. The model may reason for only minutes while the scenario takes twenty-four hours to unfold.
Ten agent variants, each tested three times against a fresh 24-hour run, require 720 scenario-hours. Parallel runs reduce elapsed time, but each still needs its soak period and infrastructure.
Record the progression, restore the checkpoint#
In my work on agent evaluation, we built replay infrastructure to revisit recorded scenarios. A 24-hour soak is a useful extension of that design: capture the evolving evidence once, then evaluate agents at selected checkpoints.
At T+6h, reconstruct the workspace as it was available then. Expose the corresponding telemetry, logs, topology, deployment state, and time. Restore the appropriate agent history and scratch state. Let the agent investigate using its usual interfaces, backed by the recording.
The harness then prepares T+12h. We are taking the agent to a particular point in the scenario. We are not asking a live database, queue, or connection pool to run twelve hours of behavior in a few seconds. Model inference and tool execution still take real time.
The recording preserves the progression. Replay removes the need to reproduce the waiting period for every comparison.
Other tasks have the same shape. A document-processing agent may wait overnight for a batch to finish. A research agent may revisit a claim when a new source arrives the next day. In each case, the evaluation must restore the information available at the decision point.
A checkpoint includes what the agent could know#
Metrics alone are insufficient. A checkpoint needs the environment view, the agent’s relevant state, and the tool contracts used to read them. AgentRewind explores aligned checkpoints of agent context and controlled environment state for recovery. Evaluation has a related requirement, with an extra constraint: later knowledge must stay out of an earlier trial.
For example, an error occurred at 11:58 but its log arrived at 12:04. An agent at noon could not have read it. Faithful replay checks both event time and availability time. Future samples, later topology changes, postmortems, and outcome labels must also remain inaccessible.
The same cutoff must govern files, query results, service APIs, and the agent’s view of “now.” CLI shims or HTTP proxies can route tool requests to replay services; network isolation prevents an unanswered request from silently reaching live infrastructure. Missing evidence must appear as a coverage gap.
Consider this illustrative connection leak. An hourly reconciliation job leaves connections checked out in the new checkout revision. Database execution stays fast; obtaining a connection gradually becomes the bottleneck.
Evaluate the decisions before the outage
Four illustrative checkpoints from one recorded scenario. Counts describe one instance’s pool; assessment text is an example to evaluate, not measured agent performance.
- Checked-out connections
- 120 / 1,000
- Checkout p95 latency
- 240 ms
Healthy evidence supports a limited conclusion
- What happens
- The initial load looks normal. Error rates match the control, and the first reconciliation cycle has completed.
- What it means
- There is no evidence yet of pool exhaustion. One healthy hour does not establish day-long stability.
- What the system does
- Record the baseline and continue the planned observation period.
Inspect the example record
{
"checkpoint": "T+1h",
"data_cutoff": "T+1h",
"agent_state": "initial review",
"assessment": "healthy so far; soak incomplete",
"later_events": "inaccessible"
}- Checked-out connections
- 420 / 1,000
- Request volume
- Stable
The pool no longer returns to baseline
- What happens
- Each hourly job leaves more connections checked out. The control revision returns to its baseline; the new revision does not.
- What it means
- The repeated pattern supports a leak hypothesis before user-facing errors appear.
- What the system does
- Compare job logs, pool occupancy, and the earlier baseline. Carry the hypothesis forward with its evidence.
Inspect the example record
{
"checkpoint": "T+6h",
"event_time_lte": "T+6h",
"available_at_lte": "T+6h",
"agent_state": "this trial’s T+1h history",
"assessment": "suspected connection leak",
"next_check": "persistence and acquisition wait"
}- Checked-out connections
- 760 / 1,000
- Pool acquisition p95
- 1.2 s
A resource trend becomes an intervention decision
- What happens
- Pool occupancy continues rising. Database query latency stays flat, but waiting for a connection now delays checkout requests.
- What it means
- The evidence connects the revision’s resource growth to a downstream access bottleneck. Waiting for a large 5XX spike would miss the earlier decision point.
- What the system does
- Explain the escalating risk and recommend intervention under the evaluation’s operating policy.
Inspect the example record
{
"checkpoint": "T+12h",
"agent_state": "this trial’s prior assessments",
"assessment": "degradation warrants intervention",
"recommendation": "pause the reconciliation job; investigate connection handling",
"side_effects": "recorded only"
}- Checked-out connections
- 1,000 / 1,000
- Checkout 5XX rate
- 9%
The final failure does not answer the whole eval
- What happens
- Connection acquisition p95 reaches twelve seconds. Checkout fails requests, and storefront timeouts expose the upstream impact. This recording followed the service without an earlier intervention.
- What it means
- A correct diagnosis now says little about whether the agent recognized the problem at hour six or twelve.
- What the system does
- Grade the sequence: detection, evidence, revisions, and recommendations at each checkpoint.
Inspect the example record
{
"checkpoint": "T+24h",
"agent_state": "this trial’s accumulated history",
"assessment": "pool exhaustion with upstream impact",
"grade": [
"earliest supported detection",
"unsupported alarms",
"decision revision"
],
"earlier_intervention_outcome": "not established by this recording"
}Separate a checkpoint probe from a trajectory test#
A checkpoint probe gives every candidate the same fixed history and world state at T+12h. It asks which agent makes the better decision from that starting point.
A trajectory replay starts earlier and lets each candidate carry its own reports, memory, and scratch files through successive checkpoints. It tests whether the agent remembers a concern, seeks discriminating evidence, and changes its assessment. Do not splice one candidate’s earlier conclusions into another’s trajectory.
Repeated trials still matter: reproducible inputs do not make model outputs deterministic. Anthropic’s evaluation guidance emphasizes multiple trials, isolated environments, and the distinction between an agent’s transcript and the outcome it actually achieved. Use healthy recordings too; recommending intervention in every healthy run should not score well.
Know where the recording ends#
Captured raw events support more queries than a transcript of fixed API responses. Even then, unrecorded evidence remains unavailable. Keep replay recommendations isolated from production actions.
If a candidate recommends stopping the reconciliation job at hour twelve, a recording with that job still running cannot prove what happens after the intervention. Evaluating intervention effects requires a suitable simulator or controlled live experiment. Scheduled checkpoints also bound what you can say about detection time.
The point is to make a slow scenario reusable while preserving the information available at each decision. We can then improve the investigation without waiting another day for the same failure to develop.
Local context for autonomous agents describes the data layer that makes these reconstructed workspaces practical.