Serve the answer, not the model.
Every request runs through cache-hit routing before it touches a provider. When a near-duplicate answer exists inside your workspace, we return it in under 100ms and skip the token bill entirely.
cache: "exact"SHA of the normalized request. Zero false positives — same messages, same tools, same seed = same reply.
cache: "semantic"Embedding cosine over the last user turn. Tunable threshold; defaults tuned per task class.
cache: "prefix"Prefix-hashed system + tool schema. Skips the KV rebuild on providers that expose prefix caching.
Enable per request
POST /v1/chat/completions
{
"model": "auto:cost",
"messages": [...],
"cache": { "mode": "semantic", "ttl": 3600, "threshold": 0.94 }
}Responses include x-llmcloud-cache: hit|miss|stale and an x-llmcloud-cost-saved header so you can attribute savings in the usage dashboard.
Never bleed cached answers across orgs or users.
Redaction runs before the cache key is computed.
Cache hits stream back as SSE, same shape as a live completion.