On 12 June 2026 at 14:07 UTC, OpenAI's us-east-1 chat completions endpoint started returning 502s. Full recovery took 41 minutes. During that window, ~19% of llmcloud traffic was pointed at OpenAI models. Here's what our gateway did.
Timeline
- T+0: First failing probe. Provider health score drops 40%.
- T+18s: Second consecutive failure. Health drops below routing threshold.
- T+22s: Router stops sending new requests to the affected region.
- T+22s → T+41m: Traffic re-routed to Azure OpenAI (same weights) and Anthropic Claude 4.5 (quality substitute). 99.98% of requests succeeded within their SLO.
What we learned
Cross-provider failover only works if the substitutes are pre-warmed and pre-benchmarked. We now keep every capability class (frontier, mid, cheap, vision, audio) mapped to at least two providers, and we probe every one of them every 30 seconds. Latency-only failover isn't enough — you need capability-preserving fallbacks.