'Structured output' means very different things depending on the provider. Some enforce the schema in the sampler (constrained decoding), some post-hoc validate and retry, and a couple just prompt-engineer their way in and hope. We ran the same 5M-request eval across all of them.
| Provider | Malformed % | Median retries | Enforcement |
|---|---|---|---|
| OpenAI (strict json_schema) | 0.02% | 0 | Sampler-level |
| Anthropic (tool JSON) | 0.11% | 0 | Sampler-level |
| Fireworks (llama-4) | 0.38% | 0 | Sampler-level |
| Together (qwen3) | 0.62% | 0 | Sampler-level |
| Groq (llama-4) | 1.20% | 1 | Post-hoc + retry |
| Bedrock (nova) | 3.10% | 2 | Prompt-only |
| Some challengers | 6.40% | 3+ | Prompt-only |
How to shrink your tail
- Prefer providers with sampler-level enforcement for high-stakes JSON.
- Keep schemas shallow — deep nesting is the #1 predictor of malformed output.
- Enable llmcloud's 'strict_json' flag: we validate every response and transparently retry against a peer provider before returning.