Cost per task is not cost per outcome
For CFOs and CIOs — price the outcome, not the attempt.
The chart everyone shares answers the easy question
The comparison is easy to read and hard to misuse in only one direction. On the horizontal axis, weighted average cost per task at each model’s maximum reasoning setting; on the vertical, a composite intelligence score. As the July 2026 chart from Artificial Analysis is drawn, the spread across the field is more than fiftyfold, from a few cents a task at the cheap end to a few dollars at the expensive one. Near the top of the score range, models sit at prices that differ by roughly a factor of three while scoring almost the same. The middle of the range is crowded, which is the genuinely interesting part: a similar level of capability is available at very different prices.
If you are buying tokens, that chart is the right tool. If you are buying a business process, it prices one line item. The number describes a single attempt at a task by a single model. It does not describe what your organisation spends to end up with an answer it can act on, and those are different quantities with different owners.
What a usable result actually costs
Put the terms on the table. This is an accounting frame rather than a closed formula, and at minimum it has five entries; only the first appears on any vendor chart.
Cost per attempt is the number you already compare: model, tool and API spend for one run. It varies enormously, which is why it attracts attention, and it is the one term you change simply by choosing a supplier.
Attempts until a usable result follows from consistency rather than intelligence, and two metrics ask genuinely different questions here. pass@k asks whether at least one of k attempts succeeded, which is what a demonstration reports. pass^k asks whether all k succeeded, which is closer to what an operation needs. For independent attempts with per-attempt success probability p, the expected number of tries before the first success is roughly 1/p, so a step that works four times in five costs about a quarter more runs than the sticker price implies.
Chains behave worse, because success probabilities multiply: five steps at 0.9 each give about 0.59 for the whole chain, not 0.9. Whether that turns into full restarts depends on your harness. If a failure forces the process to begin again, you pay for the whole chain each time. If the run can resume from the failed step, you pay for the step. That single architectural difference can change this term by multiples, and it is decided long before anyone picks a model. The numbers here are arithmetic, not measurement, but it is arithmetic that lands on an invoice.
Verification is the term most organisations discover late, and it is separate from the two above: establishing that a result is usable is work in its own right. A judge model converts that work from hours into tokens, which is often the right trade, and it brings its own price and its own error rate, because a judge reading the same context as the worker shares its blind spots. Human review costs far more per unit, catches different failures, particularly the plausible ones, and carries an opportunity cost, since the reviewer is usually someone whose time has other claims on it.
The cost of an error that passed verification is the term that behaves unlike the others: probability of escape multiplied by what the escape costs. A repeat costs a few cents. A wrong markdown that reached the system of record costs margin, an incorrect replenishment costs working capital, and both cost the time of people who now have to unwind them. Small in probability and large in consequence is precisely the shape that spreadsheets handle badly.
The harness itself is the fifth entry and the one most often left out: building or buying the execution layer, integrating it, keeping data and connectors current. It is amortised across every run rather than charged per attempt, which is exactly why it disappears from per-task comparisons.
Together these give what we call process TCO: not the price of thinking once, but the price of arriving at something you can act on. Roughly, the amortised harness plus, per usable result, the run cost multiplied by the attempts it takes, plus verification, plus the expected cost of an escape. It is a frame for counting rather than a formula with fitted coefficients, and the point of writing it out is that four of the five entries never appear in the comparison that decisions actually get made on.
Working the numbers on a single queue
Frames are easy to nod at and hard to act on, so here is one worked through. The numbers below are illustrative arithmetic rather than a measurement of anything: yours will differ, and the point is the shape of the result, not the figures.
Take an exception queue that handles ten thousand decisions a month. The work runs as four steps, each of which succeeds ninety-five times in a hundred, and each step costs twenty cents to execute. That sounds robust until you multiply: four steps at 0.95 gives a chain that completes cleanly about 81% of the time.
What happens next depends entirely on how the harness handles a failure, and the spread is wider than one number suggests. If a failure forces the whole chain to run again and you are billed for all four steps each time, you need about 1.23 full runs per usable result and the monthly bill lands near $9,800. If the run aborts at the failing step and you pay only for the steps actually executed, the same restart logic costs about 4.55 step-executions per result, or near $9,100. If the run resumes at the failed step rather than starting over, you replay only that step: roughly 4.2 executions instead of 4, and a bill near $8,400.
Same model, same prices, same success rates. The difference between the top and bottom of that range is about $1,400 a month, and it is decided entirely by execution design, not by anything a model comparison would surface.
Now add verification, and the proportions change entirely. A judge model at three cents a result costs $300 a month. Human review at six minutes a result, charged at sixty dollars an hour, costs $60,000 for full coverage, or $6,000 if you check one result in ten. Verification is not a rounding error on the compute bill. In this example, checking one result in ten already costs most of what the compute does, and full human coverage costs several times more.
Then add the term that decides everything. Suppose that three results in every hundred are wrong in a way that verification fails to catch, and that unwinding one costs five hundred dollars in margin, corrections and people’s time. That is $150,000 a month, dwarfing every other entry combined. Halve the escape rate to 1.5% and it falls to $75,000; get it to half a percent and it falls to $25,000. Note what moved: not the model, but how much wrongness survives the check.
| Entry | Illustrative monthly cost |
|---|---|
| Compute, restart and re-bill the whole chain | $9,800 |
| Compute, restart but bill only executed steps | $9,100 |
| Compute, resume at failed step | $8,400 |
| Verification by judge model | $300 |
| Verification by human, one in ten | $6,000 |
| Verification by human, every result | $60,000 |
| Escapes at 3% of results, $500 each | $150,000 |
| Escapes at 0.5% of results, $500 each | $25,000 |
Read the table and the strategy writes itself. Compute is neither the largest column nor the one that moves much when you optimise it: the entire spread between the best and worst execution design here is smaller than the cost of checking one result in ten by hand, and both are dwarfed by escapes. The leverage sits in how much wrongness survives verification and what you pay to achieve that, and in this example neither is governed by which model runs inside the process.
Resume or restart, and why it is settled too early
The gap between $9,800 and $8,400 above looks modest, and it is the visible part of a much larger effect. Restarting a chain does not only cost compute. It costs elapsed time, which matters when the value of a decision decays by the hour. It costs idempotency risk, because a chain that half-completed may have already written something before it failed, and running it again can double an action unless the design prevents that. And it costs traceability, because two overlapping partial runs are harder to reconstruct afterwards than one run with a recorded resume point.
None of this is a modelling decision. It is settled when someone chooses how work is executed, months before a model is selected and usually without anyone framing it as a cost decision. That is the pattern this frame keeps exposing: the entries with the most leverage are decided earliest, by people who are not in the room when the model comparison happens.
Why a smarter model does not fix the second term
The instinctive response to unreliable output is to buy a better model, and the evidence makes that response weaker than it feels. Princeton’s 2026 study of agent reliability found that twenty-four months of capability gains produced only small improvements in reliability, and that this holds across all the major providers rather than being a vendor-specific weakness. Capability and consistency have been improving at different rates.
Read that against the frame and the consequence is uncomfortable for the usual purchasing motion. Moving up the intelligence axis buys you a better first attempt. On that evidence it does not reliably buy you proportionally fewer repeats. A better model does move the base error rate, and it can make a result easier to check, but verification cost and escape cost are also governed by things it does not touch: whether a failed step resumes or restarts, how an action behaves when it is retried, how much evidence travels with a result, and what stands between a wrong result and a system of record.
Review regimes, and how to stop paying for the wrong one
Because verification is a real cost centre, it deserves to be designed rather than defaulted. Four regimes cover most of the ground, and the choice among them should follow the cost of being wrong in that class of decisions.
Full review of every result is expensive and appropriate while a new agent is on probation, in exactly the way a new hire is checked before they are trusted. Sampled spot-checks cost a fraction and work once consistency has been demonstrated, which is to say once accumulated pass^k justifies the reduction. Blind control cases, where a known-flawed input is planted occasionally to see whether the reviewer catches it, are how you find out whether the reviewing function still works at all rather than assuming it does. A judge model with escalation to a human on low confidence scales differently from human review, trading price per check for a blind spot it shares with the worker, and it needs its own calibration, because a judge nobody audits is a control nobody has tested.
The mistake is uniform review: the same regime applied to every decision regardless of what an error would cost. It overspends on the reversible and underspends on the expensive, and it does both invisibly, because the review metric shows full coverage either way.
What the frame does not cover
A frame that claims to price everything prices nothing, so it is worth naming what sits outside this one.
It says nothing about latency, and there are processes where being slow is worse than being expensive. It says nothing about the quality ceiling: if a model simply cannot do the task, no amount of retrying and reviewing produces a usable result, and the honest answer is that the process is not ready for automation. It ignores risk appetite, which is a matter of governance rather than arithmetic, and two organisations with identical numbers can rationally choose different review regimes. It also leans on two independence assumptions that reality does not always honour. Retries are treated as independent, which fails when a failure is systematic: the same missing field will defeat every attempt, and repetition fixes nothing. And multiplying step probabilities assumes the steps are independent too, whereas in practice one shaky upstream step often degrades everything after it, which makes the chain worse than the arithmetic suggests rather than better.
Treat it as an accounting frame that makes the invisible entries visible, not as a model that decides for you.
Computing it for one process
The exercise is short enough to do in an afternoon, and it is worth doing on a single named process rather than in general.
Start by defining what counts as a usable result, in words a reviewer would agree with. Then measure how often you get one on the first attempt, over enough cases to mean something, and how many attempts a usable result takes on average. Establish what one attempt costs in compute, and separately how the harness behaves on failure, because that determines whether you pay for a step or for the chain.
Next, price verification honestly, including the opportunity cost of whoever does it: a reviewer whose day job is category management is not free just because their salary is already booked elsewhere. Finally, estimate the escape rate and what an escape costs downstream. Both are uncomfortable numbers, the second especially, and both can be approximated from incidents you have already had.
The output is not a precise figure. It is a ranking of the five entries by size, which is usually enough to tell you whether your next move belongs in model selection, in execution design, or in how results get checked.
What this changes in the buying conversation
Three questions change the shape of a procurement discussion, and none of them appear on a comparison chart.
The first is what one usable result costs today in a single named process. Many organisations have never computed it, and the exercise can reveal that repeats and review dominate the model bill. The second is how an abandoned run is billed: a process that fails halfway consumed tokens, produced nothing, and somebody paid for it, so the answer tells you whether your supplier’s incentives point at completion or at consumption. The third is what the supplier can show about escapes: how a wrong result is detected, who carries the consequence when one reaches a system of record, and what evidence arrives with each run to make that check possible at all.
Operstead, our execution harness for agent work, is designed around the later terms rather than the first: durable execution built on resumable steps and idempotent retries, receipts that travel with a result so it can be checked afterwards, and evaluation gates that widen an agent’s scope on recorded evidence rather than on a demonstration.
Where this leaves the chart
None of this makes the comparison useless. Cost per attempt is real money and a fiftyfold spread is worth attention. The argument is about boundaries: that chart is the correct answer to a question about tokens and the wrong answer to a question about processes, and the second question is the one with a budget attached.
The organisations that will spend well on this are not the ones that pick the cheapest column. They are the ones that can say what a usable outcome costs them, know which of the four terms dominates, and have decided deliberately who checks the work and how often.