Verdicts require
Epistemics
An autonomous agent investigates 5XX errors—and learns to separate what it observes, what it believes, and what it can safely do next.
Part of Trustworthy Autonomy ↗Chapters in this collection 01 / 09
Epistemics is how we decide what to believe, what supports that belief, and when to change our mind.
Its roots lie in epistemology: the philosophical study of knowledge. A correct answer can be a lucky guess. Good reasons matter, and new evidence can reveal their limits.
Three questions make this practical:
- What did I observe?
- What am I assuming?
- What would change my mind?
“Server errors are rising” is an observation. “The latest deployment caused them” is a hypothesis. The distinction matters when an autonomous agent can act on that diagnosis.
An agent on call#
Imagine an API gateway's 5XX error rate rises from 0.2% to 12%, mostly 504 timeouts. Its p95 latency jumps from 250 milliseconds to three seconds. A Checkout deployment finished two minutes earlier.
5XX codes indicate server errors; 504 means the gateway waited too long for a response. The p95 is the response time that 95% of requests meet or beat.
The timing makes the deployment a suspect. The agent checks which requests fail and follows their dependencies.
An epistemic record#
The agent keeps a versioned record of its verdict, observations, inferences, confidence, alternatives, and unknowns. It states what would challenge the explanation and when to reassess it. Follow the six checkpoints below to see that record change.
Two details help: a connection pool is a reusable set of database connections. The agent's approved runbook allows restoring its previous size after checking database capacity, and limiting retries.
Follow the evidence
INCIDENT 042 / ILLUSTRATIVEThe newest deploy is a suspect
Low confidence · timing alone- Observed
- Gateway 5XX is 12% and p95 is three seconds. Checkout was deployed two minutes earlier.
- Current belief
- The deployment may be involved. There is no causal evidence yet.
- Next action
- Compare failing routes and trace their dependencies.
Inspect the structured record
{
"incident": "042",
"revision": 1,
"recorded_at": "10:00",
"supersedes": null,
"verdict": "degraded",
"scope": "Gateway symptoms; root cause unverified",
"observations": [
{
"id": "obs-1",
"statement": "Gateway 5XX is 12% and p95 is three seconds. Checkout was deployed two minutes earlier.",
"evidence_refs": [
"gateway metrics",
"deployment history"
]
}
],
"inferences": [
{
"statement": "The deployment may be involved. There is no causal evidence yet.",
"supported_by": [
"obs-1"
],
"alternatives": [
"A shared dependency is slow",
"The deployment changed request behavior"
]
}
],
"confidence": {
"level": "low",
"basis": "Low confidence · timing alone"
},
"unknowns": [
"Which routes fail; the cause of the timeouts"
],
"discriminating_checks": [
"A shared dependency could explain failures beyond Checkout."
],
"next_action": "Compare failing routes and trace their dependencies.",
"valid_through": "10:03",
"reassess_if": "New evidence contradicts the explanation, or traffic or configuration changes"
}Two APIs, one slow dependency
Moderate confidence · traces converge- Observed
- Checkout and Catalog both spend most of their request time waiting on Inventory. Payments stays fast.
- Current belief
- Inventory is the shared slow dependency. The trigger is still unknown: Checkout could also be overloading it.
- Next action
- Separate Inventory’s queueing time from database execution time.
Inspect the structured record
{
"incident": "042",
"revision": 2,
"recorded_at": "10:03",
"supersedes": 1,
"verdict": "degraded",
"scope": "Both API routes; sampled request traces",
"observations": [
{
"id": "obs-2",
"statement": "Checkout and Catalog both spend most of their request time waiting on Inventory. Payments stays fast.",
"evidence_refs": [
"Checkout traces",
"Catalog traces",
"Payments latency"
]
}
],
"inferences": [
{
"statement": "Inventory is the shared slow dependency. The trigger is still unknown: Checkout could also be overloading it.",
"supported_by": [
"obs-2"
],
"alternatives": [
"Checkout is overloading Inventory",
"Inventory changed independently"
]
}
],
"confidence": {
"level": "medium",
"basis": "Moderate confidence · traces converge"
},
"unknowns": [
"What triggered the shared slowdown"
],
"discriminating_checks": [
"Trace more requests; a shared delay does not identify what started it."
],
"next_action": "Separate Inventory’s queueing time from database execution time.",
"valid_through": "10:06",
"reassess_if": "New evidence contradicts the explanation, or traffic or configuration changes"
}The wait happens before the query
Leading hypothesis · mechanism and timing fit- Observed
- Inventory waits 2.4 s for a database connection; queries take about 20 ms. At 09:56, its pool limit changed from 40 connections to 4 per replica.
- Current belief
- The smaller connection pool is the leading explanation. Requests can queue before reaching an otherwise fast database.
- Next action
- Verify database headroom and restore the previous pool setting on one replica.
Inspect the structured record
{
"incident": "042",
"revision": 3,
"recorded_at": "10:06",
"supersedes": 2,
"verdict": "degraded",
"scope": "Inventory pool; database capacity must be checked",
"observations": [
{
"id": "obs-3",
"statement": "Inventory waits 2.4 s for a database connection; queries take about 20 ms. At 09:56, its pool limit changed from 40 connections to 4 per replica.",
"evidence_refs": [
"pool wait histogram",
"database spans",
"09:56 configuration diff"
]
}
],
"inferences": [
{
"statement": "The smaller connection pool is the leading explanation. Requests can queue before reaching an otherwise fast database.",
"supported_by": [
"obs-3"
],
"alternatives": [
"Another bottleneck coexists with pool starvation"
]
}
],
"confidence": {
"level": "medium",
"basis": "Leading hypothesis · mechanism and timing fit"
},
"unknowns": [
"Whether restoring the pool will relieve the bottleneck; database headroom"
],
"discriminating_checks": [
"If restoring the pool does not reduce waiting, reconsider the diagnosis."
],
"next_action": "Verify database headroom and restore the previous pool setting on one replica.",
"valid_through": "10:10",
"reassess_if": "New evidence contradicts the explanation, or traffic or configuration changes"
}An intervention tests the belief
Strong support · one replica, comparable load- Observed
- With database capacity checked, one replica gets its previous setting back. At comparable load, pool wait falls to 30 ms; unchanged replicas still queue.
- Current belief
- Strong evidence that the pool change caused the bottleneck. This canary—a trial on one replica—does not establish overall recovery.
- Next action
- Restore the setting across replicas and check end-to-end recovery.
Inspect the structured record
{
"incident": "042",
"revision": 4,
"recorded_at": "10:10",
"supersedes": 3,
"verdict": "degraded",
"scope": "Requests routed through the canary; other replicas still affected",
"observations": [
{
"id": "obs-4",
"statement": "With database capacity checked, one replica gets its previous setting back. At comparable load, pool wait falls to 30 ms; unchanged replicas still queue.",
"evidence_refs": [
"canary pool metrics",
"matched request traces",
"unchanged replicas"
]
}
],
"inferences": [
{
"statement": "Strong evidence that the pool change caused the bottleneck. This canary—a trial on one replica—does not establish overall recovery.",
"supported_by": [
"obs-4"
],
"alternatives": [
"Differences in workload may explain part of the canary improvement"
]
}
],
"confidence": {
"level": "high",
"basis": "Strong support · one replica, comparable load"
},
"unknowns": [
"Whether other replicas and request types will improve"
],
"discriminating_checks": [
"Check whether the improvement holds across replicas and request types."
],
"next_action": "Restore the setting across replicas and check end-to-end recovery.",
"valid_through": "10:14",
"reassess_if": "New evidence contradicts the explanation, or traffic or configuration changes"
}The trigger and the amplifier differ
Retry amplification likely · extra attempts measured- Observed
- After rollout, gateway p95 is 900 ms and 5XX is 2.8%. User traffic is unchanged, but traces show three Inventory attempts per original call on average.
- Current belief
- The pool change was a trigger. Upstream retries appear to be sustaining excess load after that trigger is removed.
- Next action
- Apply the approved retry budget, which caps extra attempts; watch queues and successful requests.
Inspect the structured record
{
"incident": "042",
"revision": 5,
"recorded_at": "10:14",
"supersedes": 4,
"verdict": "degraded",
"scope": "All replicas restored; upstream retries remain",
"observations": [
{
"id": "obs-5",
"statement": "After rollout, gateway p95 is 900 ms and 5XX is 2.8%. User traffic is unchanged, but traces show three Inventory attempts per original call on average.",
"evidence_refs": [
"gateway latency",
"attempts per request",
"Inventory queue depth"
]
}
],
"inferences": [
{
"statement": "The pool change was a trigger. Upstream retries appear to be sustaining excess load after that trigger is removed.",
"supported_by": [
"obs-5"
],
"alternatives": [
"Another source of load may be keeping queues busy"
]
}
],
"confidence": {
"level": "medium",
"basis": "Retry amplification likely · extra attempts measured"
},
"unknowns": [
"Whether reducing retries will drain the remaining queues"
],
"discriminating_checks": [
"If queues persist after retries fall, investigate another source of load."
],
"next_action": "Apply the approved retry budget, which caps extra attempts; watch queues and successful requests.",
"valid_through": "10:30",
"reassess_if": "New evidence contradicts the explanation, or traffic or configuration changes"
}Verify recovery at the user boundary
Recovery observed · current traffic level- Observed
- For ten minutes at comparable incoming load, queues stay drained, p95 is 270 ms, and 5XX is 0.3%.
- Current belief
- Recovery is supported for this traffic level. Peak-load behavior and the reason the configuration changed remain unverified.
- Next action
- Continue monitoring and investigate how the pool configuration changed.
Inspect the structured record
{
"incident": "042",
"revision": 6,
"recorded_at": "10:30",
"supersedes": 5,
"verdict": "recovered-under-observed-load",
"scope": "Observed load and time window; not a universal guarantee",
"observations": [
{
"id": "obs-6",
"statement": "For ten minutes at comparable incoming load, queues stay drained, p95 is 270 ms, and 5XX is 0.3%.",
"evidence_refs": [
"10-minute gateway window",
"queue depth",
"incoming traffic"
]
}
],
"inferences": [
{
"statement": "Recovery is supported for this traffic level. Peak-load behavior and the reason the configuration changed remain unverified.",
"supported_by": [
"obs-6"
],
"alternatives": [
"Recovery may not hold at higher load"
]
}
],
"confidence": {
"level": "high",
"basis": "Recovery observed · current traffic level"
},
"unknowns": [
"Peak-load behavior; why the configuration changed"
],
"discriminating_checks": [
"Reopen the incident if errors or queueing return, especially at higher load."
],
"next_action": "Continue monitoring and investigate how the pool configuration changed.",
"valid_through": "10:40",
"reassess_if": "Errors, queueing, traffic, topology, or configuration materially change"
}What changed the agent's mind?#
The error surfaced at the gateway. The bottleneck sat downstream in Inventory's connection pool. Slow responses affected upstream callers; their retries sent more load downstream. Several red dashboards described different parts of the same incident.
The record should link each observation to its evidence and each inference to the observations supporting it. Earlier versions stay available so another agent can see why the explanation changed.
Confidence needs a stated basis. Fluency cannot supply one. “Insufficient evidence” is a legitimate verdict when a required observation is missing. Competing explanations remain visible until discriminating checks rule them out.
A verdict also needs an expiry and an earlier reassessment condition. “Healthy at this load until the next checkpoint” is a claim someone can test. “Healthy,” left indefinitely in a report, invites much more trust than the evidence earned.
A successful intervention strengthens a diagnosis; it does not establish every claim about the system. The reason the configuration changed remains unresolved. Recovery at today's traffic level says little about tomorrow's peak.
The same discipline applies to any autonomous agent: keep observations separate from explanations, choose actions that test those explanations, and carry unresolved questions forward.
A trustworthy agent keeps its conclusions connected to evidence—and makes the limits of those conclusions visible.
The incident and figures are illustrative. Background: Google's SRE chapters on Effective Troubleshooting and Addressing Cascading Failures.