Evaluation in practice01
How to Write and Score an Agent Eval
From a plausible wrong answer to a defensible release decision: nine chapters on writing tests, calculating scores, and checking whether success repeats.
By Chaitanya · Systems engineering
Engineering the systems that make agents work.
I’m Chaitanya. I build autonomous agents and the infrastructure around them: local context, reliable execution, and reproducible evaluation. Oddly connects the architecture, experiments, and engineering decisions behind that work.
Read the writing
Agent infrastructureLocal contextObservabilityReproducible evaluation
01
All writing
Featured writing
Agent interfaces02
Implementing Progressive Disclosure for Agent Tools
Build one discoverable CLI for MCP, SQL, and REST, with precise schemas, scoped execution, and honest result receipts.
Agent architecture03
Before You Build Another Agent
Build a strong generalist. Make expertise discoverable. Delegate only when the task earns it.
02
All writing
Recent writing
How to Write and Score an Agent EvalFrom a plausible wrong answer to a defensible release decision: nine chapters on writing tests, calculating scores, and checking whether success repeats.
1 min
Implementing Progressive Disclosure for Agent ToolsBuild one discoverable CLI for MCP, SQL, and REST, with precise schemas, scoped execution, and honest result receipts.
12 min
Before You Build Another AgentBuild a strong generalist. Make expertise discoverable. Delegate only when the task earns it.
7 min
The Quality FlywheelHow failures become experiments, better decisions, and knowledge that survives the next release.
1 min
Classifiers: When Labels Become DecisionsTen chapters on designing categories, choosing evidence, measuring errors, and deciding when a label deserves to influence what happens next.
3 min
03 / Connecting the work
Make evidence queryable, execution reliable, and decisions reproducible.Explore the focus areas
04
Open notebook
Currently thinking about
?
Which decisions should a classifier's labels be allowed to influence?
01 / 08