llmcloud.ai
Get API key
Blog

Notes from the routing layer.

Deep dives on smart routing, ensembling, latency, cost, and reliability — plus independent scorecards for the open-weights models that actually moved the frontier.

Articles

8 posts
Routing· 9 min

Smart routing, explained: how auto:cost, auto:quality and auto:speed pick a model

A look under the hood of the llmcloud.ai router — feature extraction, live health scoring, and the bandit that decides which upstream wins each request.

Read →
Accuracy· 11 min

Model ensembles: 12–18% accuracy gains by blending multiple LLMs

We ran a 40K-prompt eval across arithmetic, code, and long-form QA. Ensembling three mid-tier models beat GPT-4-class single-model calls at 30% lower cost.

Read →
Latency· 6 min

Speculative routing: race two models, keep the faster stream

A trick borrowed from speculative decoding — launch a cheap model and a strong model in parallel, keep whichever hits first-token latency first.

Read →
Cost· 8 min

Model cascades: the quiet way to cut 60% of your LLM bill

Route to a tiny model first, escalate only when confidence is low. Works even better with a router that knows every model's per-topic confidence.

Read →
Cost· 7 min

Semantic caching for LLM gateways: 40% cost reduction without quality loss

Exact-match caches catch 3–8% of production traffic. Embedding-based caches with a smart threshold catch 30–45% — safely.

Read →
Reliability· 6 min

What actually happens when OpenAI goes down

A post-mortem-style walkthrough of the June 2026 OpenAI incident from a gateway's perspective. How auto-failover kept 99.98% of routed traffic healthy.

Read →
Routing· 8 min

Prompt-aware routing beats static rules by 14 quality points

Classify the prompt first, route second. A 120M-parameter router model, trained on 8M annotated prompts, decides which of 40 models each request goes to.

Read →
Evals· 7 min

Structured output across providers: a 30-day reliability study

We ran 5M structured-output requests across 12 providers. Malformed rates ranged from 0.02% to 6.4%. Here's the leaderboard and how to shrink your tail.

Read →

Model scorecards

Best OSS models of the past 12 months
Filter:AllFrontier OSSCodingMultimodalReasoningEfficiency king
Frontier OSSDeepSeek·

DeepSeek-V3.1

The OSS model that finally matched GPT-4-class quality at 1/20th the price.

V3.1 keeps the sparse MoE architecture from V3 but ships with a much cleaner post-training stack: hybrid RLHF, tool-use traces from R1, and a new reward model that pushes MATH and LiveCodeBench past every previous open release.

Params
671B MoE · 37B active
Context
128K
License
MIT-style (open weights)
$ / 1M tok
$0.27 in · $1.10 out
Strengths
  • Frontier-tier reasoning
  • MoE keeps cost near a 30B dense model
  • Excellent tool calling
Weaknesses
  • Long-context degrades past ~64K
  • Weaker in multilingual outside EN/ZH
Routing verdict

Default `auto:cost` fallback for anything not requiring vision. Route reasoning-heavy prompts here before touching GPT-4-class closed models.

Scorecard
Reasoning (MMLU-Pro)82
Coding (LiveCodeBench)79
Math (MATH-500)91
Speed (tok/s)74
68 tok/s on our fleet
Cost efficiency96
Try on llmcloud.ai:
curl https://api.llmcloud.ai/v1/chat/completions \
  -H "Authorization: Bearer $KEY" \
  -d '{"model":"deepseek-v3-1","messages":[...]}'
MultimodalMeta·

Llama 4 Maverick

Meta's answer to GPT-4o — natively multimodal and ridiculously long-context.

Maverick is the mid-tier of the Llama 4 family (below Behemoth, above Scout). Native early-fusion multimodality means images, audio and text share the same token stream, and the 1M context window is genuinely usable up to ~700K.

Params
400B MoE · 17B active · 128 experts
Context
1M
License
Llama 4 Community License
$ / 1M tok
$0.35 in · $1.40 out
Strengths
  • True 1M context that works
  • Native multimodal (vision + audio)
  • Great instruction following
Weaknesses
  • License blocks EU-based training use
  • Coding lags DeepSeek and Qwen3
Routing verdict

Route here for `mode:multimodal` and for RAG pipelines with >200K context. Not our first choice for pure coding.

Scorecard
Reasoning (MMLU-Pro)80
Vision (MMMU)73
Long-context (RULER 512K)84
Speed (tok/s)68
Cost efficiency88
Try on llmcloud.ai:
curl https://api.llmcloud.ai/v1/chat/completions \
  -H "Authorization: Bearer $KEY" \
  -d '{"model":"llama-4-maverick","messages":[...]}'
Frontier OSSAlibaba·

Qwen3-235B-A22B

The most permissively-licensed frontier OSS model, and it speaks 119 languages.

Qwen3 introduces a switchable 'thinking' mode: the same weights can answer instantly or emit long CoT when you set `enable_thinking=true`. Apache 2.0 makes it the default choice for enterprise self-hosting.

Params
235B MoE · 22B active
Context
256K
License
Apache 2.0
$ / 1M tok
$0.30 in · $1.20 out
Strengths
  • Apache 2.0 — no strings attached
  • Best-in-class multilingual
  • Toggleable reasoning depth
Weaknesses
  • Vision variant is separate SKU
  • Reasoning mode ~3× slower
Routing verdict

Our default recommendation for regulated / self-hosted deployments. Route multilingual traffic here even when GPT-4o would win on EN-only.

Scorecard
Reasoning (MMLU-Pro)81
Coding (LiveCodeBench)76
Multilingual (M-MMLU)88
Speed (tok/s)71
Cost efficiency92
Try on llmcloud.ai:
curl https://api.llmcloud.ai/v1/chat/completions \
  -H "Authorization: Bearer $KEY" \
  -d '{"model":"qwen3-235b","messages":[...]}'
CodingMoonshot AI·

Kimi K2

The first trillion-parameter open-weights model — and it's genuinely agentic.

K2 was trained explicitly for tool use and multi-step agents. On SWE-bench Verified it hits 65.8% single-attempt, edging out Claude Sonnet-class models at a fraction of the cost. The MuonClip optimizer keeps training stable at 15.5T tokens.

Params
1T MoE · 32B active · 384 experts
Context
128K
License
Modified MIT (open weights)
$ / 1M tok
$0.60 in · $2.50 out
Strengths
  • Best OSS agentic performance to date
  • Massive expert pool = strong niche knowledge
  • Long tool-call chains stay coherent
Weaknesses
  • Higher latency than DeepSeek-V3.1
  • 128K context (short by 2026 standards)
Routing verdict

Default target for `mode:agent` and Cursor/Cline-style coding agents. We route SWE-heavy traffic here before Claude.

Scorecard
Agentic (SWE-bench)87
65.8% pass@1
Coding (LiveCodeBench)82
Reasoning (MMLU-Pro)79
Speed (tok/s)62
Cost efficiency84
Try on llmcloud.ai:
curl https://api.llmcloud.ai/v1/chat/completions \
  -H "Authorization: Bearer $KEY" \
  -d '{"model":"kimi-k2","messages":[...]}'
Efficiency kingOpenAI·

gpt-oss-120b

OpenAI's first open-weights release since GPT-2 — and it fits on a single H100.

gpt-oss ships with configurable reasoning effort (low/medium/high) and full CoT visibility. Runs at 3000+ tok/s on a single 80GB H100 thanks to the tiny 5.1B active-parameter footprint. Not multimodal, English-first.

Params
117B MoE · 5.1B active
Context
128K
License
Apache 2.0
$ / 1M tok
$0.15 in · $0.60 out
Strengths
  • Ridiculous throughput
  • Apache 2.0 with commercial rights
  • Configurable reasoning budget
Weaknesses
  • English-only training focus
  • No vision / audio
Routing verdict

Best latency-sensitive OSS route. `auto:latency` sends everything here first unless the prompt requires vision or non-EN.

Scorecard
Reasoning (MMLU-Pro)78
Coding (LiveCodeBench)74
Math (AIME 2025)88
with high-effort mode
Speed (tok/s)96
3000+ on H100
Cost efficiency94
Try on llmcloud.ai:
curl https://api.llmcloud.ai/v1/chat/completions \
  -H "Authorization: Bearer $KEY" \
  -d '{"model":"gpt-oss-120b","messages":[...]}'
ReasoningMistral AI·

Mistral Large 3

Europe's flagship — dense architecture, function-calling that just works.

Large 3 doubles down on structured output: native JSON mode, parallel tool calls, and a reworked function-calling grammar that basically eliminates malformed calls. Dense (not MoE), so throughput scales predictably.

Params
141B dense
Context
256K
License
Mistral Research License (weights open, commercial via API)
$ / 1M tok
$1.50 in · $4.50 out
Strengths
  • Most reliable tool calling in OSS
  • EU-hosted, GDPR-clean
  • Predictable dense throughput
Weaknesses
  • Priciest OSS in this list
  • Non-commercial without paid license
Routing verdict

Route to Large 3 when the caller sets `strict_tools:true` or needs EU data residency. Otherwise DeepSeek/Qwen3 win on price.

Scorecard
Reasoning (MMLU-Pro)77
Function calling91
Coding (LiveCodeBench)72
Speed (tok/s)70
Cost efficiency78
Try on llmcloud.ai:
curl https://api.llmcloud.ai/v1/chat/completions \
  -H "Authorization: Bearer $KEY" \
  -d '{"model":"mistral-large-3","messages":[...]}'
Methodology

Scores combine public benchmark deltas (MMLU-Pro, LiveCodeBench, SWE-bench, MATH-500, RULER, MMMU), our internal eval harness (2.3M prompts across 14 task families), and 30-day live routing telemetry from the llmcloud.ai gateway. Prices reflect the lowest public per-1M-token rate available through a supported provider on the release date.