Trustworthy Autonomy
Trustworthy Autonomy / Chapter 01 of 09

Verdicts require
Epistemics

An autonomous agent investigates 5XX errors—and learns to separate what it observes, what it believes, and what it can safely do next.

Part of Trustworthy Autonomy ↗
Chapters in this collection 01 / 09

Epistemics is how we decide what to believe, what supports that belief, and when to change our mind.

Its roots lie in epistemology: the philosophical study of knowledge. A correct answer can be a lucky guess. Good reasons matter, and new evidence can reveal their limits.

Three questions make this practical:

  • What did I observe?
  • What am I assuming?
  • What would change my mind?

“Server errors are rising” is an observation. “The latest deployment caused them” is a hypothesis. The distinction matters when an autonomous agent can act on that diagnosis.

An agent on call#

Imagine an API gateway's 5XX error rate rises from 0.2% to 12%, mostly 504 timeouts. Its p95 latency jumps from 250 milliseconds to three seconds. A Checkout deployment finished two minutes earlier.

5XX codes indicate server errors; 504 means the gateway waited too long for a response. The p95 is the response time that 95% of requests meet or beat.

The timing makes the deployment a suspect. The agent checks which requests fail and follows their dependencies.

An epistemic record#

The agent keeps a versioned record of its verdict, observations, inferences, confidence, alternatives, and unknowns. It states what would challenge the explanation and when to reassess it. Follow the six checkpoints below to see that record change.

Two details help: a connection pool is a reusable set of database connections. The agent's approved runbook allows restoring its previous size after checking database capacity, and limiting retries.

Follow the evidence

INCIDENT 042 / ILLUSTRATIVE
At the API gateway
5XX error rate12%
p95 latency3.0 s
API gateway504 timeouts
Checkout APIRecent deploy
Catalog APINot yet checked
PaymentsNot yet checked
InventoryNot yet checked
PostgreSQLNot yet checked
Observed impactMeasured as normalUnchecked
Arrows follow requests downstream. The gateway calls Checkout and Catalog; both call Inventory. Checkout also calls Payments. Inventory calls PostgreSQL. Colors describe the displayed measurements, not every aspect of service health.
Record 01 / 06 · 10:00

The newest deploy is a suspect

Low confidence · timing alone
Observed
Gateway 5XX is 12% and p95 is three seconds. Checkout was deployed two minutes earlier.
Current belief
The deployment may be involved. There is no causal evidence yet.
Next action
Compare failing routes and trace their dependencies.
Basis: gateway metrics · deployment history
Inspect the structured record
{
  "incident": "042",
  "revision": 1,
  "recorded_at": "10:00",
  "supersedes": null,
  "verdict": "degraded",
  "scope": "Gateway symptoms; root cause unverified",
  "observations": [
    {
      "id": "obs-1",
      "statement": "Gateway 5XX is 12% and p95 is three seconds. Checkout was deployed two minutes earlier.",
      "evidence_refs": [
        "gateway metrics",
        "deployment history"
      ]
    }
  ],
  "inferences": [
    {
      "statement": "The deployment may be involved. There is no causal evidence yet.",
      "supported_by": [
        "obs-1"
      ],
      "alternatives": [
        "A shared dependency is slow",
        "The deployment changed request behavior"
      ]
    }
  ],
  "confidence": {
    "level": "low",
    "basis": "Low confidence · timing alone"
  },
  "unknowns": [
    "Which routes fail; the cause of the timeouts"
  ],
  "discriminating_checks": [
    "A shared dependency could explain failures beyond Checkout."
  ],
  "next_action": "Compare failing routes and trace their dependencies.",
  "valid_through": "10:03",
  "reassess_if": "New evidence contradicts the explanation, or traffic or configuration changes"
}
Record 02 / 06 · 10:03

Two APIs, one slow dependency

Moderate confidence · traces converge
Observed
Checkout and Catalog both spend most of their request time waiting on Inventory. Payments stays fast.
Current belief
Inventory is the shared slow dependency. The trigger is still unknown: Checkout could also be overloading it.
Next action
Separate Inventory’s queueing time from database execution time.
Basis: Checkout traces · Catalog traces · Payments latency
Inspect the structured record
{
  "incident": "042",
  "revision": 2,
  "recorded_at": "10:03",
  "supersedes": 1,
  "verdict": "degraded",
  "scope": "Both API routes; sampled request traces",
  "observations": [
    {
      "id": "obs-2",
      "statement": "Checkout and Catalog both spend most of their request time waiting on Inventory. Payments stays fast.",
      "evidence_refs": [
        "Checkout traces",
        "Catalog traces",
        "Payments latency"
      ]
    }
  ],
  "inferences": [
    {
      "statement": "Inventory is the shared slow dependency. The trigger is still unknown: Checkout could also be overloading it.",
      "supported_by": [
        "obs-2"
      ],
      "alternatives": [
        "Checkout is overloading Inventory",
        "Inventory changed independently"
      ]
    }
  ],
  "confidence": {
    "level": "medium",
    "basis": "Moderate confidence · traces converge"
  },
  "unknowns": [
    "What triggered the shared slowdown"
  ],
  "discriminating_checks": [
    "Trace more requests; a shared delay does not identify what started it."
  ],
  "next_action": "Separate Inventory’s queueing time from database execution time.",
  "valid_through": "10:06",
  "reassess_if": "New evidence contradicts the explanation, or traffic or configuration changes"
}
Record 03 / 06 · 10:06

The wait happens before the query

Leading hypothesis · mechanism and timing fit
Observed
Inventory waits 2.4 s for a database connection; queries take about 20 ms. At 09:56, its pool limit changed from 40 connections to 4 per replica.
Current belief
The smaller connection pool is the leading explanation. Requests can queue before reaching an otherwise fast database.
Next action
Verify database headroom and restore the previous pool setting on one replica.
Basis: pool wait histogram · database spans · 09:56 configuration diff
Inspect the structured record
{
  "incident": "042",
  "revision": 3,
  "recorded_at": "10:06",
  "supersedes": 2,
  "verdict": "degraded",
  "scope": "Inventory pool; database capacity must be checked",
  "observations": [
    {
      "id": "obs-3",
      "statement": "Inventory waits 2.4 s for a database connection; queries take about 20 ms. At 09:56, its pool limit changed from 40 connections to 4 per replica.",
      "evidence_refs": [
        "pool wait histogram",
        "database spans",
        "09:56 configuration diff"
      ]
    }
  ],
  "inferences": [
    {
      "statement": "The smaller connection pool is the leading explanation. Requests can queue before reaching an otherwise fast database.",
      "supported_by": [
        "obs-3"
      ],
      "alternatives": [
        "Another bottleneck coexists with pool starvation"
      ]
    }
  ],
  "confidence": {
    "level": "medium",
    "basis": "Leading hypothesis · mechanism and timing fit"
  },
  "unknowns": [
    "Whether restoring the pool will relieve the bottleneck; database headroom"
  ],
  "discriminating_checks": [
    "If restoring the pool does not reduce waiting, reconsider the diagnosis."
  ],
  "next_action": "Verify database headroom and restore the previous pool setting on one replica.",
  "valid_through": "10:10",
  "reassess_if": "New evidence contradicts the explanation, or traffic or configuration changes"
}
Record 04 / 06 · 10:10

An intervention tests the belief

Strong support · one replica, comparable load
Observed
With database capacity checked, one replica gets its previous setting back. At comparable load, pool wait falls to 30 ms; unchanged replicas still queue.
Current belief
Strong evidence that the pool change caused the bottleneck. This canary—a trial on one replica—does not establish overall recovery.
Next action
Restore the setting across replicas and check end-to-end recovery.
Basis: canary pool metrics · matched request traces · unchanged replicas
Inspect the structured record
{
  "incident": "042",
  "revision": 4,
  "recorded_at": "10:10",
  "supersedes": 3,
  "verdict": "degraded",
  "scope": "Requests routed through the canary; other replicas still affected",
  "observations": [
    {
      "id": "obs-4",
      "statement": "With database capacity checked, one replica gets its previous setting back. At comparable load, pool wait falls to 30 ms; unchanged replicas still queue.",
      "evidence_refs": [
        "canary pool metrics",
        "matched request traces",
        "unchanged replicas"
      ]
    }
  ],
  "inferences": [
    {
      "statement": "Strong evidence that the pool change caused the bottleneck. This canary—a trial on one replica—does not establish overall recovery.",
      "supported_by": [
        "obs-4"
      ],
      "alternatives": [
        "Differences in workload may explain part of the canary improvement"
      ]
    }
  ],
  "confidence": {
    "level": "high",
    "basis": "Strong support · one replica, comparable load"
  },
  "unknowns": [
    "Whether other replicas and request types will improve"
  ],
  "discriminating_checks": [
    "Check whether the improvement holds across replicas and request types."
  ],
  "next_action": "Restore the setting across replicas and check end-to-end recovery.",
  "valid_through": "10:14",
  "reassess_if": "New evidence contradicts the explanation, or traffic or configuration changes"
}
Record 05 / 06 · 10:14

The trigger and the amplifier differ

Retry amplification likely · extra attempts measured
Observed
After rollout, gateway p95 is 900 ms and 5XX is 2.8%. User traffic is unchanged, but traces show three Inventory attempts per original call on average.
Current belief
The pool change was a trigger. Upstream retries appear to be sustaining excess load after that trigger is removed.
Next action
Apply the approved retry budget, which caps extra attempts; watch queues and successful requests.
Basis: gateway latency · attempts per request · Inventory queue depth
Inspect the structured record
{
  "incident": "042",
  "revision": 5,
  "recorded_at": "10:14",
  "supersedes": 4,
  "verdict": "degraded",
  "scope": "All replicas restored; upstream retries remain",
  "observations": [
    {
      "id": "obs-5",
      "statement": "After rollout, gateway p95 is 900 ms and 5XX is 2.8%. User traffic is unchanged, but traces show three Inventory attempts per original call on average.",
      "evidence_refs": [
        "gateway latency",
        "attempts per request",
        "Inventory queue depth"
      ]
    }
  ],
  "inferences": [
    {
      "statement": "The pool change was a trigger. Upstream retries appear to be sustaining excess load after that trigger is removed.",
      "supported_by": [
        "obs-5"
      ],
      "alternatives": [
        "Another source of load may be keeping queues busy"
      ]
    }
  ],
  "confidence": {
    "level": "medium",
    "basis": "Retry amplification likely · extra attempts measured"
  },
  "unknowns": [
    "Whether reducing retries will drain the remaining queues"
  ],
  "discriminating_checks": [
    "If queues persist after retries fall, investigate another source of load."
  ],
  "next_action": "Apply the approved retry budget, which caps extra attempts; watch queues and successful requests.",
  "valid_through": "10:30",
  "reassess_if": "New evidence contradicts the explanation, or traffic or configuration changes"
}
Record 06 / 06 · 10:30

Verify recovery at the user boundary

Recovery observed · current traffic level
Observed
For ten minutes at comparable incoming load, queues stay drained, p95 is 270 ms, and 5XX is 0.3%.
Current belief
Recovery is supported for this traffic level. Peak-load behavior and the reason the configuration changed remain unverified.
Next action
Continue monitoring and investigate how the pool configuration changed.
Basis: 10-minute gateway window · queue depth · incoming traffic
Inspect the structured record
{
  "incident": "042",
  "revision": 6,
  "recorded_at": "10:30",
  "supersedes": 5,
  "verdict": "recovered-under-observed-load",
  "scope": "Observed load and time window; not a universal guarantee",
  "observations": [
    {
      "id": "obs-6",
      "statement": "For ten minutes at comparable incoming load, queues stay drained, p95 is 270 ms, and 5XX is 0.3%.",
      "evidence_refs": [
        "10-minute gateway window",
        "queue depth",
        "incoming traffic"
      ]
    }
  ],
  "inferences": [
    {
      "statement": "Recovery is supported for this traffic level. Peak-load behavior and the reason the configuration changed remain unverified.",
      "supported_by": [
        "obs-6"
      ],
      "alternatives": [
        "Recovery may not hold at higher load"
      ]
    }
  ],
  "confidence": {
    "level": "high",
    "basis": "Recovery observed · current traffic level"
  },
  "unknowns": [
    "Peak-load behavior; why the configuration changed"
  ],
  "discriminating_checks": [
    "Reopen the incident if errors or queueing return, especially at higher load."
  ],
  "next_action": "Continue monitoring and investigate how the pool configuration changed.",
  "valid_through": "10:40",
  "reassess_if": "Errors, queueing, traffic, topology, or configuration materially change"
}

What changed the agent's mind?#

The error surfaced at the gateway. The bottleneck sat downstream in Inventory's connection pool. Slow responses affected upstream callers; their retries sent more load downstream. Several red dashboards described different parts of the same incident.

The record should link each observation to its evidence and each inference to the observations supporting it. Earlier versions stay available so another agent can see why the explanation changed.

Confidence needs a stated basis. Fluency cannot supply one. “Insufficient evidence” is a legitimate verdict when a required observation is missing. Competing explanations remain visible until discriminating checks rule them out.

A verdict also needs an expiry and an earlier reassessment condition. “Healthy at this load until the next checkpoint” is a claim someone can test. “Healthy,” left indefinitely in a report, invites much more trust than the evidence earned.

A successful intervention strengthens a diagnosis; it does not establish every claim about the system. The reason the configuration changed remains unresolved. Recovery at today's traffic level says little about tomorrow's peak.

The same discipline applies to any autonomous agent: keep observations separate from explanations, choose actions that test those explanations, and carry unresolved questions forward.

A trustworthy agent keeps its conclusions connected to evidence—and makes the limits of those conclusions visible.

The incident and figures are illustrative. Background: Google's SRE chapters on Effective Troubleshooting and Addressing Cascading Failures.

Start typing to search the writing.