TTFT is the single metric users actually feel. Streaming disguises tail latency, but a slow first token still lands as dead air. Speculative routing races two models against each other and drops the loser mid-stream.
How it works
- Fire the cheap fast model (e.g. gpt-5.5-mini) and the quality model in parallel.
- Open both streams. The first stream to emit a token wins.
- Cancel the loser cleanly — most providers refund cancelled requests within the first 200 ms.
- Optional: keep the loser's stream if the winner is significantly cheaper and its quality Elo is within a threshold of the loser's.
Numbers from our fleet
Across a 2M-request sample, speculative routing cut median TTFT from 410 ms to 254 ms. Cost went up 1.42× because we occasionally pay for two prompts. If p95 latency matters more than absolute cost, this is the best knob we've shipped.
{
"model": "auto:quality",
"speculative": { "sidecar": "openai/gpt-5.5-mini", "cancel_after_ms": 800 }
}