The dumb version: hash the prompt, cache the response. Works for FAQ bots and near-nothing else, because users paraphrase. The smarter version embeds every prompt, does an approximate nearest-neighbour lookup, and returns the cached response only when both the embedding similarity and a fast semantic-equivalence check pass.
What we cache
- System + user messages only (never partial conversations).
- Temperature ≤ 0.3 requests. Higher temperature = you asked for variance, so we don't cache.
- Non-tool calls only. Tool-call outputs depend on world state.
- Per-key namespace so tenant data never bleeds.
Safety rails
Semantic caches can hallucinate hits. We use a two-gate check: cosine similarity ≥ 0.94 AND a small NLI model returns 'entails' for both directions. That kept our observed hit-precision at 99.6% in a 6-week production study across 40M requests.
Set cache: { mode: 'semantic', ttl: 3600 } in the request body, or turn it on per-key in the dashboard. Cached responses come back with x-llmcloud-cache: hit and a synthetic latency ≤ 15 ms.