Smart routing, explained: how auto:cost, auto:quality and auto:speed pick a model

Every request that hits api.llmcloud.ai carries a routing directive — either a specific model like anthropic/claude-opus-4.5 or a policy string like auto:quality. This post walks through what happens between the moment the request lands and the moment we open a stream to an upstream provider.

Stage 1 — Feature extraction

We extract 22 features from the request in under a millisecond: prompt token count, presence of images or audio, requested tools, JSON-mode flags, temperature, language hint, and a small semantic embedding used for topic classification (code, math, chat, agent, RAG, creative).

Stage 2 — Capability filter

We narrow the ~300 model pool down to candidates that can actually serve the request. A prompt with an image kicks out text-only models; a 400K-token context filters out anything under 512K headroom; a tool-call schema drops models with weak function-calling scores. This alone typically leaves 8–15 candidates.

Stage 3 — Policy scoring

Each policy weights the candidates differently:

Quality is the per-topic Elo we maintain from our internal eval harness plus public benchmarks. Latency is a 5-minute exponentially-weighted moving average of TTFT and tokens/sec across our probes and real user traffic. Price is the effective $/1M-token cost blended across input, output, and cached tokens.

Stage 4 — Contextual bandit

Scores feed a Thompson-sampling bandit that explores ~5% of traffic to keep quality estimates fresh. This is why identical requests occasionally take different routes — it's how we notice when a provider degrades or a new model quietly gets better.

POST /v1/chat/completions
Authorization: Bearer $LLMCLOUD_KEY

{
  "model": "auto:quality",
  "route": {
    "min_quality": 85,
    "max_ttft_ms": 500,
    "regions": ["us-east", "eu-west"],
    "exclude_providers": ["openai"]
  },
  "messages": [{"role":"user","content":"Draft an eng-lead memo."}]
}

Stage 5 — Fallback chains

If the winner returns 5xx, times out, or hits a rate limit, we fail over to the runner-up within 200 ms — the client never sees the retry. Every response includes an x-llmcloud-route header with the winning model and the chain of models we tried.