Your model router has a setting called "cost-optimized." You turned it on. Your finance dashboard went green, and you felt smart. You are not smart. You optimized the one number that doesn't pay your bills.
The whole premise of cheap-model routing — send everything to Haiku, escalate only when forced — treats price-per-token as the cost of the work. It isn't. The cost of the work is cost per successfully completed task, and on hard or agentic jobs those two numbers diverge so violently that the "cheapest model" becomes the single most expensive line item you run. The damage is real; it just doesn't show up where you're looking, which is exactly why it survives quarter after quarter.
Here's the part nobody puts in the pricing blog.
Retries don't replay the failed step — they replay everything
This is the load-bearing fact most engineers get wrong, and it's why the math is so ugly.
When an agent step fails — bad tool call, malformed JSON, hallucinated function name — the framework does not re-run the marginal failed step. It resends the entire context: the system prompt, every prior tool result, every intermediate reasoning block, all of it, back through the model. Retry cost is not linear in step count. It's quadratic in context size.
This breaks the intuition everyone budgets with. The naive model says: 20% per-step failure, one retry, so roughly 25% overhead. Tolerable. Wrong. Once you account for full-context replay, the retry-budget analysis puts a 3-step agent at 20% per-step failure around 1.7–1.9x baseline tokens, and a 5-step agent at 2.2–2.5x. A failure rate you'd wave through on a dashboard quietly doubles the bill on any workflow long enough to matter.
And cheaper models fail steps more often. That's the entire trade you made. You picked the model with the higher per-step failure rate, then routed it into the workflow shape where per-step failure compounds quadratically. You didn't cut costs. You bought a multiplier.
Failed tool calls are pure waste, and the cheap model emits more of them
It gets worse, because not all of that replayed spend is even buying you a chance at progress.
A benchmark of ReAct-style agents across 200 tasks found that 90.8% of retries were spent on hallucinated or nonexistent tool names — the model confidently calling search_database() when the tool is query_db(), then burning a full retry to find out. In most billing models you pay full input tokens on a failed or timed-out call. Failure is not free. You are paying retail to learn the model can't read its own tool schema.
Weaker models hallucinate tool names and arguments at higher rates. So the cheap route doesn't just retry more — a larger share of those retries are structurally incapable of succeeding. You're not rolling the dice faster. You're rolling loaded dice.
This is why token deflation has been a trap. Per-token prices keep falling, and yet the bills keep climbing — because the cheapest-and-most-expensive model gap is roughly 10x per token, applied on every turn at an accumulating context size. Vantage's agentic-coding cost data is brutal on this point: a single 50-turn session runs ~1M input tokens against ~40K output, a 25:1 ratio where context accumulation, not headline price, drives the bill. The price went down and the bills went up, because cheap routing manufactured the volume.
The honest version: cheap routing genuinely works — on easy problems
Now I'll argue against myself, because the contrarian case is only worth making if it survives the strongest counter-evidence. And the counter-evidence is real and peer-reviewed.
RouteLLM (lm-sys) is the canonical result: ~85% cost reduction on MT-Bench while retaining 95% of GPT-4-class quality, sending only ~14% of requests to the strong model. That's a legitimate, replicable win. If your workload looks like MT-Bench, turn on the router and don't think twice.
But read the rest of the table, the part the enthusiasts skip. RouteLLM's savings were 85% on MT-Bench, 45% on MMLU, 35% on GSM8K. The pattern is not subtle: the harder and more reasoning-dependent the task, the less cheap routing pays. It wins big on conversational, classifiable, low-stakes queries and collapses toward break-even on math and knowledge-heavy reasoning. Routing isn't a free lunch you bolt onto everything. It's a tool with a domain, and the domain is easy work.
The price ladder makes the break-even concrete
Look at the actual Claude numbers (2026), per million tokens:
| Model | Input | Output | SWE-bench Verified | |---|---|---|---| | Haiku 4.5 | $1 | $5 | 73.3% | | Sonnet 4.6 | $3 | $15 | 79.6% | | Opus 4.6 | $5 | $25 | 80.8% |
The Opus input premium over Haiku is 5x. Feels like a lot. It isn't. Two cheap-model retries with full-context replay erase it outright — and the SWE-bench gap (73.3% vs 80.8%, a ~7.5-point first-pass deficit) is exactly the kind of gap that, on a multi-step task, cascades into those replay-retries. You don't lose 7.5 points once. You lose them at step 1, eat a full-context retry, lose them again at step 3, and the "5x cheaper" model has cost you more wall-clock, more tokens, and an engineer's afternoon.
That afternoon is the cost that never hits the dashboard. Cheap routing produces silent quality regression: the answer looks plausible, ships, and surfaces three days later as a customer ticket and an engineer doing rework. Tokens are the cheap part of a wrong answer.
The decision rule
Stop asking "what's the cheapest model that might handle this?" Ask "what's the cheapest model that succeeds on the first try?" Different question. Conflating the two is the entire mistake.
Route to the cheap model when the task is high-volume, classifiable, short-horizon (1–2 steps), shallow on tool calls, trivially verifiable, and failure is cheap to detect and retry: classification, extraction, file navigation, linting, simple edits, sub-agent fan-out. Here the cheap model's failure rate is low and its failures get caught instantly. Route away.
Send it to the strong model once when the task is multi-step agentic, tool-heavy, long-context, hard to verify, or where a wrong answer triggers human rework. The break-even heuristic:
if cheap_first_pass_success × (1 − retry_replay_overhead) < strong_first_pass_success:
the "cheap" route is already more expensive — send it to Opus once.
Plug in real numbers. Cheap model: 73% first-pass, 0.8x effective after replay overhead → 0.58. Strong model: 81% first-pass. 0.58 < 0.81. The cheap route lost before you sent a single token.
The guardrail that earns the savings safely
None of this is guessable from a vibe. Put a pre-merge eval gate of 50–500 representative cases in front of any router config before it touches production. Measure cost per successful task, not per token, on your actual workload. A router you haven't eval-gated isn't an optimization — it's an unmonitored bet with your reliability as the stake.
And while you're there, reach for the levers that actually beat naive downgrade-routing: prompt caching cuts cached input ~90%, batch processing cuts every price 50%. Those are real, compounding, quality-neutral savings. Downgrading the model to pocket 5x and paying it back in retries is not.
The cheapest model per token is routinely the most expensive model per result. Optimize the metric that empties the bank account, not the one that looks good in the demo.