Which decisions should a classifier's labels be allowed to influence?
Proposed test · not yet measured
Hold predictions fixed and vary the permitted action. Inspect how false-positive cost changes the acceptance rule.
Open notebook8 open questions
Ideas in progress
Questions I have not resolved yet. Publishing them early makes the edges of the work visible.
How can we tell a change in the world from a change in the classifier measuring it?
Proposed test · not yet measured
Run old and new classifier versions over the same frozen overlap sample before attributing a dashboard change to the world.
When should a classifier gather more evidence or leave a case unresolved?
Proposed test · not yet measured
Compare coverage and error among accepted cases as the deferral threshold moves; record review workload too.
Which evidence should already be local when an agent begins an investigation?
Proposed test · not yet measured
Compare remote retrieval with a frozen local snapshot, including collection and indexing cost in the total.
Where should deterministic execution end and model-driven exploration begin?
Proposed test · not yet measured
Replace one repeated reasoning step with deterministic code and test edge cases that the code must reject.
How much of a 24-hour incident can a checkpoint preserve without leaking its outcome?
Proposed test · not yet measured
Insert a late-arriving record whose event time predates the checkpoint; verify it remains unavailable in historical replay.
When does evaluating an intervention require simulation instead of replay?
Proposed test · not yet measured
Choose an action that changes the environment and show exactly where a fixed recorded future stops being a valid comparison.
What should memory retain so the next agent can verify a lesson before reusing it?
Proposed test · not yet measured
Retrieve a lesson after changing its parser or policy dependency; require a review instead of unconditional reuse.