Evaluation in practice01
How to Write and Score an Agent Eval
From a plausible wrong answer to a defensible release decision: nine chapters on writing tests, calculating scores, and checking whether success repeats.
By Chaitanya · Systems engineering
Engineering the systems that make agents work.
I’m Chaitanya. I build autonomous agents and the infrastructure around them: local context, reliable execution, and reproducible evaluation. Oddly connects the architecture, experiments, and engineering decisions behind that work.
Find your starting point
Agent infrastructureLocal contextObservabilityReproducible evaluation
01
All writing
Start with these
A working philosophy02
Trustworthy Autonomy
Nine principles for systems that deserve the authority we give them. A field guide to evidence, boundaries, and learning from consequences.
Agent interfaces03
Implementing Progressive Disclosure for Agent Tools
Build one discoverable CLI for MCP, SQL, and REST, with precise schemas, scoped execution, and honest result receipts.
02
All writing
Recent writing
Building an Agent That Improves Its Own HarnessTurn repeated failures into tested changes, then use the evidence to improve the next experiment.
17 min
How to Write and Score an Agent EvalFrom a plausible wrong answer to a defensible release decision: nine chapters on writing tests, calculating scores, and checking whether success repeats.
34 min collection
Implementing Progressive Disclosure for Agent ToolsBuild one discoverable CLI for MCP, SQL, and REST, with precise schemas, scoped execution, and honest result receipts.
15 min
Before You Build Another AgentBuild a strong generalist. Make expertise discoverable. Delegate only when the task earns it.
8 min
The Quality FlywheelHow failures become experiments, better decisions, and knowledge that survives the next release.
24 min collection
03 / Connecting the work
Make evidence queryable, execution reliable, and decisions reproducible.Explore the focus areas
04
Open notebook
Currently thinking about
?
Which decisions should a classifier's labels be allowed to influence?
01 / 08