Notes from the routing layer.
Deep dives on smart routing, ensembling, latency, cost and hosting economics. For model grades and benchmarks, see the LLM Catalog rankings and scorecards.
Latest posts
- ·Cost· 10 min read
Kimi K3: what it actually costs to host a 1T-parameter sparse MoE
Moonshot's K3 lands with a huge sparse MoE and an agentic tool-use profile. We break down the serving footprint, the KV-cache tax, and the input/output token equation that decides whether you route or host it.
- ·Cost· 11 min read
Qwen's 2-trillion-parameter MoE: serving a model that doesn't fit anywhere
Alibaba's largest open-weight release pushes past the single-node era. What changes when weights outgrow a rack unit — sharding, cold starts, quantisation, and the real per-token math.
- ·Routing· 9 min read
Smart routing, explained: how auto:cost, auto:quality and auto:speed pick a model
A look under the hood of the llmcloud.ai router — feature extraction, live health scoring, and the bandit that decides which upstream wins each request.
- ·Cost· 9 min read
DeepSeek V4 and the hidden cost of reasoning tokens
Long-thinking models bill you for tokens you never read. A look at V4's serving profile, why reasoning breaks conventional capacity planning, and how to budget for invisible output.
- ·Accuracy· 11 min read
Model ensembles: 12–18% accuracy gains by blending multiple LLMs
We ran a 40K-prompt eval across arithmetic, code, and long-form QA. Ensembling three mid-tier models beat GPT-4-class single-model calls at 30% lower cost.
- ·Reliability· 6 min read
What actually happens when OpenAI goes down
A post-mortem-style walkthrough of the June 2026 OpenAI incident from a gateway's perspective. How auto-failover kept 99.98% of routed traffic healthy.
- ·Latency· 6 min read
Speculative routing: race two models, keep the faster stream
A trick borrowed from speculative decoding — launch a cheap model and a strong model in parallel, keep whichever hits first-token latency first.
- ·Cost· 8 min read
Model cascades: the quiet way to cut 60% of your LLM bill
Route to a tiny model first, escalate only when confidence is low. Works even better with a router that knows every model's per-topic confidence.
- ·Cost· 7 min read
Semantic caching for LLM gateways: 40% cost reduction without quality loss
Exact-match caches catch 3–8% of production traffic. Embedding-based caches with a smart threshold catch 30–45% — safely.
- ·Routing· 8 min read
Prompt-aware routing beats static rules by 14 quality points
Classify the prompt first, route second. A 120M-parameter router model, trained on 8M annotated prompts, decides which of 40 models each request goes to.
- ·Evals· 7 min read
Structured output across providers: a 30-day reliability study
We ran 5M structured-output requests across 12 providers. Malformed rates ranged from 0.02% to 6.4%. Here's the leaderboard and how to shrink your tail.