Semantic Cache

A semantic cache stores previous model responses and returns one when a new request means the same thing as an old one. Unlike an ordinary cache it does not compare strings; it embeds the incoming request and looks for a stored request close to it in vector space. The economics are why teams reach for it. Support and documentation assistants see the same twenty questions in a hundred phrasings, so a cache that understands paraphrase can cut both spend and p95 latency substantially — a cache hit is a lookup rather than a generation. The risk is the mirror image: two questions can be close in embedding space and different in the way that matters. "Can I cancel my plan?" and "Can I cancel my plan and get a refund?" sit near each other and have different answers. Tune the similarity threshold against real traffic, not intuition, and keep it conservative for anything that touches money, entitlements or legal wording. Cache invalidation is the other half. Answers grounded in documents must expire when those documents change, or the cache will confidently serve last quarter's pricing forever.