Learning requires
Outcomes
An invoice extractor passes its sample test. The next day, the accounting team finds incorrect totals. Who grades the original claim?
Part of Trustworthy Autonomy ↗Chapters in this collection 09 / 09
Learning requires connecting a decision to what actually happened, then using that comparison to improve future decisions.
An agent updates an invoice extractor, runs a sample batch, and writes an excellent summary. The summary becomes memory. Next time, the agent retrieves it as a successful precedent.
There is one missing step: nobody checked whether it was a success.
Reality arrives after the report#
The sample contains typed English invoices. Totals match, and the agent declares the parser ready. The next morning, reconciliation finds errors in German invoices: the parser treated decimal commas incorrectly.
Both document types were available during development. The problem lay in which examples the agent chose to test.
A system that closes the file when the report is written never connects the later discrepancy to that earlier omission. It can become more confident each time it remembers its own ungraded work.
Being right, being lucky, and being fair#
The philosophical concern is how experience justifies a belief. Repetition alone is weak evidence if every repetition is graded by the same unchecked assumption.
Outcomes must come from outside the agent’s verdict: reconciliation results, labeled documents, downstream validation, and human review where needed.
Fairness matters too. A failure on a newly introduced document format does not automatically invalidate an earlier claim limited to the formats then supported. Ask what the agent claimed, what it could have tested, and whether the later failure bears on that claim.
Grade the result after delivery
An illustrative document-processing workflow. Evaluation follows the artifact into use.
- Tested documents
- Typed English invoices
- Claim
- Ready for all suppliers
A narrow test passes
- What happens
- The agent validates totals on one document family, although examples from other suppliers were available.
- What it means
- The claim is broader than the evidence supporting it.
- What the system does
- Preserve the patch, test set, stated scope, and exact conclusion.
Inspect the example record
{
"outcome_horizon": "sample test",
"checked": "one document family",
"omitted": "decimal-comma formats",
"assessment_snapshot": "immutable"
}- Detected issue
- Decimal separator
- Evidence
- Labeled source documents
Reconciliation exposes the gap
- What happens
- Independent reconciliation finds incorrect totals in German invoices. Comparing the source documents with parsed fields establishes the error.
- What it means
- An available test case was missed. This is different from a failure on a format the system had never been asked to support.
- What the system does
- Link the outcome to the original artifact and record what was knowable at decision time.
Inspect the example record
{
"outcome_horizon": "next-day reconciliation",
"result": "incorrect numeric extraction",
"evidence_available_at_decision": true,
"grade": "avoidable coverage gap"
}- Candidate improvement
- Locale-aware checks
- Status
- Under evaluation
A lesson must improve held-out results
- What happens
- The team adds locale-aware parsing and tests it on held-out suppliers as well as the original failures.
- What it means
- Fixing one example is not enough if the change breaks previously correct documents.
- What the system does
- Measure extraction accuracy, rejected documents, and manual-review load before adoption.
Inspect the example record
{
"candidate": "locale-aware numeric parsing",
"evaluation": [
"known failure",
"held-out suppliers",
"previously correct documents"
],
"promotion": "requires agreed quality thresholds",
"autonomy_change": "not automatic"
}Close the loop without rewriting history#
Choose outcome horizons that cover meaningful delayed effects. Preserve the evidence the agent could actually access at decision time. Distinguish a justified decision, a lucky result, an avoidable miss, and an unnecessary rejection.
Track both incorrect extractions and unnecessary rejections. Record uncertainty in outcome labels when attribution remains contested. Independent collection does not make ground truth magically obvious.
Turn misses into candidate checks, better evidence gathering, or policy changes. Evaluate those changes before treating them as lessons. Keep the earlier version so their effect can be measured.
Only then should demonstrated quality inform a change in autonomy—and the permission should remain specific and revocable.
Experience becomes learning when the world can correct the story the agent tells about its work.
Further reading: Google SRE's Postmortem Culture explains how incidents can lead to concrete, reviewed improvements.