No Fluff AI
Agents that pass your evals still fail in production.I measure why.
Agentic evaluation, process-level rubrics, and where reinforcement learning meets eval design — with the code and the numbers attached.
The argument
Your eval set is your reward function.
Most teams evaluate agents the way they evaluate models: check the final answer, compute a score, ship when it's green.
Agents break that assumption. An agent can reach the correct answer through a broken process — a skipped verification, a lucky guess, a tool call whose output it ignored — and your eval will call it a pass. Then it fails in production and the dashboard offers no explanation, because the dashboard was never measuring the thing that broke.
Call that a false pass:
outcome correct, process invalid.
The fix is to score the trajectory instead of the outcome. And once you're scoring trajectories, you've built something more useful than a dashboard — you've built a dense, verifiable reward signal you can train against.
What I'm building
false-pass-bench
An open benchmark of tasks where the correct answer is reachable through the wrong process. Every trajectory published.
Of the runs an outcome eval calls green, what fraction were green by accident?
- Tasks
- 20
- Agent configs
- 3
- Runs each
- 3
- False-pass rate
- —
measured Aug 24, 2026
Writing
First essay · August 16, 2026
Your Eval Set Is Your Reward Function
Everything I publish carries a number I measured, a trajectory you can inspect, or an explicit note about what I don't know yet. If a post has none of those, it doesn't go up.
Get the findings
One email when the numbers land.
One email when the benchmark results land, and when I publish something with new data in it. No digests, no roundups, no "5 AI trends." Unsubscribe in one click.