Your organisation has always run on pass^k
For COO/CIO — stop evaluating agents on demos (pass@1) and price supervision like you already do for people: score the chain not the step, put checks where error cost × observed inconsistency is highest, relax supervision as pass^k accumulates, keep blind spot-checks after autonomy.
Everyone evaluates AI agents with a demo. A demo measures the wrong thing.
There are two ways to score k attempts at a task. pass@k asks: did at least one succeed? pass^k asks: did all of them? The first measures capability. The second measures whether you can run an operation on it. The gap is where agent projects quietly stall: an agent that succeeds 90% of the time per run clears eight runs in a row about 43% of the time. That is plain arithmetic. And a multi-step chain compounds the same way: every extra step multiplies in its own failure rate.
Human organisations solved pass^k long before anyone named it, and they found two different answers.
Routine organisations (aviation, fast food, back offices) engineer the metric directly. Checklists, standard procedures, four-eyes rules: strip the variance, raise per-step reliability, need no heroes.
High-performance organisations (deal teams, trauma units) accept that individual steps fail, and get chain-level reliability from catch-and-recover: drafts, reviews, second opinions. Steps fail; the chain usually holds.
Both use the same control lever: supervision priced by earned trust. A new hire is on probation, everything checked. A veteran gets spot-checks. Trust is accumulated pass^k.
With agents, that discipline broke. Princeton’s 2026 agent-reliability study (Rabanser et al., arXiv) puts numbers on it: across major providers, 24 months of capability gains produced only small reliability improvement, an industry-wide plateau. The divergence between pass@k and pass^k quantifies this consistency gap. So agents get hired on pass@1, the demo, and then either trusted like veterans or supervised like suspects forever. Neither matches how organisations manage people.
The playbook your organisation already knows:
- score the chain, not the step (unreliability compounds along a process);
- put checks where error cost and observed inconsistency are highest;
- relax supervision as pass^k accumulates, the way probation ends;
- keep blind spot-checks after supervision is reduced, the way audits work.
This is the frame behind agent graduation in Operstead, our agent execution platform: it is designed to let agents earn wider scope through recorded evals, not demos.
Where in your operation would an agent be on probation today, and who signs its graduation?
