Classifiers
When labels become decisions
Ten chapters on designing categories, choosing evidence, measuring errors, and deciding when a label deserves to influence what happens next.
A classifier turns an untidy situation into a small number of names. That compression makes a support queue sortable, an incident searchable, or an agent's behavior measurable. It also gives downstream systems something they may trust more than they should.
Imagine a concert ticket hidden by an email filter. Urgent wording and an unfamiliar sender produced a suspicious label; a downstream policy moved the email out of sight. The machinery worked as designed. The person still missed something important.
The challenge is larger than selecting a model. Someone has to decide what suspicious means, which evidence supports it, when the decision must arrive, and how to discover that the classifier is wrong. Those choices determine whether a label becomes useful knowledge or a repeatable mistake.
The instrument behind the number#
These chapters treat a classifier as a measurement instrument embedded in a decision process. A rule, a trained model, a prompt, and an investigating agent can all produce categories. Each needs a defined question, a source of evidence, and a way to test its answer.
The collection moves from design to measurement and then to change over time. It connects label definitions with software contracts, ground truth with provenance, confidence with evidence, and dashboards with the history of the instruments that produced them.
The examples are illustrative. Their numbers are chosen to make the reasoning inspectable: a detector with 99% accuracy that finds no failures, an unchanged detector whose alerts become less reliable, and a multi-label answer that is partly correct while missing the decisive cause.
A path through the ideas#
Start with the first four chapters when building or changing a classifier. They establish the contract, category boundaries, decision rule, and placement. Chapters five through nine examine reference labels, metrics, prevalence, structured outputs, and uncertainty. The final chapter brings them together in a reproducible release and monitoring process.
For a shorter route, read A Label Is a Commitment, Accuracy Hides a Question, and Measurement Needs a Memory. They frame the central responsibility: a label should retain a defensible relationship to the evidence and decision it represents.
Connections and further reading#
Oddly's Evidence requires Provenance follows a number back to its denominator and source. The Next Useful Question examines when additional evidence is worth obtaining. Checkpoint Replay for Long-Horizon Agent Evaluation explores evaluation at the moment a decision was made.
Public technical references are linked in the relevant chapters: the scikit-learn evaluation guide, SetFit, On Calibration of Modern Neural Networks, Detecting and Correcting for Label Shift with Black Box Predictors, Item Response Theory for evaluation, tinyBenchmarks, and hierarchical evaluation measures.