Cache

A cache is a storage layer, typically in-memory, that holds a temporary copy of data or a computed result so that a future request for the same thing can be served near-instantly instead of repeating an expensive database query, API call, or computation. Caching is one of the oldest and most universally applicable performance techniques in software engineering, and it takes on outsized importance in AI products specifically because LLM API calls are both slow (hundreds of milliseconds to several seconds) and metered by the token (every avoided call is direct cost savings, not just a latency win). Why it matters for AI/SaaS builders: a naive AI feature calls the model API on every single request, even when many requests are near-identical (the same FAQ question asked by different users, the same product description regenerated repeatedly, the same embedding computed twice for unchanged text). Caching those responses can cut both latency and API spend by an order of magnitude on features with any repeat-request pattern, and it's frequently the highest-ROI half-day of engineering work available on an AI product's cost sheet. How it works: the most common pattern is cache-aside (or "lazy loading") — on a request, check the cache first; on a hit, return immediately; on a miss, do the expensive work, store the result in the cache with a time-to-live (TTL), then return it. Cache invalidation — knowing when a cached value is stale and must be refreshed — is the genuinely hard part (famously one of the "two hard things in computer science"), typically handled either by a TTL (accept some staleness in exchange for simplicity) or explicit invalidation (bust the cache key when the underlying data changes, e.g., a webhook fires and deletes `cache:product:123` when that product is updated). Caches also need an eviction policy for when they fill up — LRU (Least Recently Used) is the most common, discarding whatever hasn't been accessed in the longest time to make room for new entries. Redis and Memcached are the dominant standalone cache stores; CDNs (Cloudflare, Fastly) cache at the HTTP layer for static and semi-static content. Worked example: an AI SEO tool generates meta descriptions for URLs on request. Since the same popular URL (e.g., a competitor's well-known blog post) gets analyzed by many different users, the backend caches the LLM's generated output keyed by a hash of the URL + prompt version: `SETEX cache:meta:{url_hash}:{prompt_v3} 86400 "<generated description>"`. The first user to analyze a given URL triggers a real LLM call (~1.5s, $0.002); the next thousand users requesting the same URL that day get a cached response in under 5ms at zero marginal cost — and bumping `prompt_v3` to `prompt_v4` in the cache key automatically invalidates old cached results after a prompt change, without needing a manual cache flush.