Model ensembles: 12–18% accuracy gains by blending multiple LLMs

Single-model calls are cheap, fast, and lossy. If you have any budget for extra tokens, ensembling two or three mid-tier models will almost always beat a single frontier call on hard tasks. We benchmarked three patterns on our public eval set.

Pattern 1 — Best-of-N with a judge

Fan out to N models in parallel, then ask a small judge model to pick the best answer. Works well when the answers are short and comparable (arithmetic, single-step reasoning, one-shot code). Cost scales linearly with N; latency stays at max(N).

Pattern 2 — Majority vote

Great for classification, multiple choice, and any task with a small discrete answer space. No judge needed — you just tally. Three cheap models voting will typically match a frontier model at 20–30% of the cost.

const votes = await Promise.all([
  llm("deepseek-v3.1", prompt),
  llm("qwen3-235b", prompt),
  llm("llama-4-maverick", prompt),
]);
const winner = mode(votes.map(parseChoice));

Pattern 3 — Debate / consensus

Two models produce answers, a third reviews the disagreement and issues the verdict. Slower and pricier, but the accuracy delta on GPQA-Diamond was the biggest we measured (+6.8 points over a single frontier call).

Doing this on llmcloud

Pass an ensemble spec on any request and we'll fan out, judge, and stream the winner back. Cost + trace headers still work.

{
  "ensemble": {
    "strategy": "best_of_n",
    "n": 3,
    "candidates": ["auto:cost", "auto:balanced", "auto:quality"],
    "judge": "openai/gpt-5.5-mini"
  },
  "messages": [ ... ]
}