Features · Routing

One endpoint, six policies, 300+ models.

Point the OpenAI SDK at api.llmcloud.ai and swap the model string for an auto: policy. The router picks the upstream per request and streams back through the same connection.

auto:qualitydefault

Weights the request by task class (chat, code, extraction, agentic) and picks the highest-ranked model on the live leaderboard that fits your budget.

POST /v1/chat/completions
{ "model": "auto:quality", "messages": [...] }

auto:cost$

Picks the cheapest upstream that meets a minimum quality floor for the detected task. Great for background jobs, evals, and bulk extraction.

{ "model": "auto:cost", "messages": [...] }

auto:speedms

Optimizes p50 latency to first token. Filters to Groq, Cerebras, SambaNova, and any provider currently hitting the SLO.

{ "model": "auto:speed", "messages": [...] }

auto:reasoningthink

Routes to a thinking model (o-series, Claude thinking, DeepSeek-R, Qwen-thinking). Reasoning trace normalized into a single reasoning[] field regardless of vendor.

{ "model": "auto:reasoning", "messages": [...] }

auto:multimodalvision

Filters to models that accept the modalities present in the request (image, audio, PDF) and then ranks on quality within that pool.

{ "model": "auto:multimodal", "messages": [{ "role": "user", "content": [ {"type":"image_url", ...} ] }] }

customyours

Pin providers, exclude regions, cap price, require zero-retention. Policies are just JSON — commit them to your repo.

{ "model": "auto:cost", "route": { "allow": ["fireworks","together"], "max_price": 0.5, "zero_retention": true } }

How it decides

Contextual bandit over live signals

Every request is scored on task class, prompt shape, modality, and cost ceiling. We then sample from the top-k models on the live leaderboard, weighted by recent success rate, p50 latency, and price. Losers are exponentially decayed; winners get more traffic. The same signal powers automatic failover mid-stream.

Live leaderboard →Inside the router (blog) →Semantic caching →

The scoring function

Each candidate model/provider pair gets a score before the request is dispatched. The weights differ per policy, but the inputs are always the same five signals, all measured on real traffic in the trailing 60 minutes — never on vendor-published claims.

Signal	Measured as	Window	Weight in auto:quality
success_rate	Non-error, non-truncated completions over total attempts, per provider/model pair	60 min	0.35
quality_rank	Blended eval score for the detected task class, from the live leaderboard	24 h	0.30
p50_ttft	Time to first token, measured from our edge, not vendor-reported	15 min	0.15
blended_price	Input + output per-token list price weighted by observed output ratio	live	0.15
capacity_head	Remaining rate-limit headroom reported or inferred per upstream	5 min	0.05

Failover, mid-stream

A gateway that only retries on connect errors is not reliable — most real failures happen after the first token. The router treats a stream as failed on any of: a 5xx or 429 before first token, a stall longer than the provider's measured p99 inter-token gap, a truncated response with no finish reason, or a tool call that does not parse as valid JSON when tools were requested. Each triggers a retry on the next-best candidate with the same decoded prefix discarded, so the client never sees a spliced answer.

{
  "model": "auto:quality",
  "route": {
    "fallbacks": ["anthropic/claude-sonnet", "openai/gpt-5-mini"],
    "max_attempts": 3,
    "stall_timeout_ms": 8000,
    "on_exhausted": "error"        // or "best_effort"
  }
}

Every decision is auditable

Routing you cannot inspect is routing you cannot trust. Every response carries an x-llmcloud-route header and an optional route object in the body naming the chosen provider, the runners-up, the score each received, and the reason the winner won. Pipe it into your own analytics or read it in usage analytics.

"route": {
  "policy": "auto:quality",
  "chosen": { "provider": "fireworks", "model": "llama-4-maverick", "score": 0.91 },
  "considered": [
    { "provider": "together", "model": "llama-4-maverick", "score": 0.88, "why": "p50 +140ms" },
    { "provider": "llmcloud", "model": "llama-4-maverick", "score": 0.86, "why": "price +12%" }
  ],
  "attempts": 1,
  "cache": "miss"
}

Which policy for which workload

Workload	Policy	Why
Interactive chat UI	auto:speed	Perceived quality is dominated by time to first token; a 200 ms gap is more visible than a two-point eval delta.
Coding agent / IDE completion	auto:quality	Failed diffs cost more than tokens. Bias to the top of the coding leaderboard and keep a same-family fallback so tool schemas stay stable.
Bulk extraction, classification, evals	auto:cost	Volume dominates. A quality floor keeps accuracy acceptable while price does the picking.
Multi-step research and planning	auto:reasoning	Thinking budgets and normalized reasoning traces matter more than raw latency.
Document, screenshot, and chart ingestion	auto:multimodal	Filters to models that actually accept the modalities in the payload instead of failing at the provider.
Regulated or region-pinned traffic	custom	Pin allowed providers and regions explicitly; see data residency for the jurisdiction map.

Full per-workload breakdowns live in workload playbooks, and the model-by-model evidence behind them is in the live rankings.

Routing FAQ

Does llmcloud take a cut of what the router spends?

No. The gateway passes through provider list price with no platform margin, so the router has no financial reason to prefer one upstream over another. Revenue comes from Team seats and hosted inference, not from your token spend.

Can the router pick our own hosted fleet just because we own it?

No. llmcloud-hosted endpoints enter the same registry as third-party providers, carry the same published trust grade, and are scored with the same function. They win traffic only when they win on price, latency, or measured quality.

What happens if every candidate fails?

With on_exhausted set to error the request returns a 502 carrying the full attempt log, so you can retry deliberately rather than silently degrade. With best_effort the router returns the highest-scoring partial response and flags it in the route object.

Can we pin a single model and skip routing entirely?

Yes. Pass a concrete model ID instead of an auto: policy and the request goes straight to that model. You still get failover across providers serving the same weights unless you also pin a provider in the route object.

How often do routing decisions change?

Latency and capacity signals refresh every few minutes, quality ranks daily. In practice a stable workload sees the same winner for hours at a time; changes happen when an upstream degrades or a new provider undercuts the field.