Cascades are the oldest trick in the routing book: try a cheap model first, and only escalate to a stronger one if the cheap one wasn't confident. What changed in the last year is that we can now score confidence robustly across models using log-probs, self-consistency, and lightweight verifier models.
The pattern
- Call a cheap model with a confidence probe (self-verify prompt or logprob threshold).
- If confidence ≥ τ, return the cheap answer.
- Otherwise escalate to a stronger model — usually one class up, not straight to the frontier.
- Log the miss to adjust τ per topic over time.
Results
| Workload | Frontier baseline | Cascade (2-stage) | Cascade (3-stage) |
|---|---|---|---|
| Customer support | $1.00 · 82.1% CSAT | $0.34 · 81.7% | $0.28 · 81.9% |
| Code review | $1.00 · 78.4% | $0.48 · 77.9% | $0.41 · 78.5% |
| Legal Q&A | $1.00 · 71.0% | $0.55 · 70.4% | $0.52 · 70.8% |
The trap
Naive cascades double-charge you when the cheap model is wrong AND the escalation runs. Set τ using held-out data, and instrument the escalation rate — a healthy cascade escalates 20–40% of the time.