Rate Limiting

Rate limiting is a technique for controlling how many requests a client (a user, an API key, or an IP address) is allowed to make to a system within a given time window — for example, "100 requests per minute" — rejecting or delaying requests beyond that limit, typically with an HTTP `429 Too Many Requests` response. It serves two related but distinct purposes: protecting infrastructure from being overwhelmed (whether by a genuine traffic spike or a bug causing a client to hammer an endpoint in a retry loop) and enforcing business/pricing tiers (a free-tier API key might be limited to 60 requests/minute while a paid tier gets 6,000). Why it matters for AI/SaaS builders: rate limiting is essential wherever an API sits behind a metered cost — this is especially true for AI-powered endpoints, since a single LLM API call can cost meaningfully more in compute than a typical database read, so an unrate-limited AI endpoint is a direct, unbounded cost-exposure risk if a client (or a bug, or a malicious actor) calls it in a tight loop. It's also standard practice for third-party-facing APIs generally, both to prevent abuse and to create natural tiering for monetization (rate limits are one of the simplest, most common ways to differentiate a free plan from a paid one). How it works: common algorithms include the token bucket (each client has a bucket that refills at a fixed rate — say, one token per second up to a max of 60 — and each request consumes one token, so it allows bursts up to the bucket size while enforcing a steady average rate) and the sliding window (counting requests in a rolling time window, more precise than a simple fixed-window count that can allow a burst right at a window boundary). The rate limiter tracks usage per client (typically keyed by API key or authenticated user ID, stored in a fast in-memory store like Redis so the check adds minimal latency) and rejects requests exceeding the configured limit, often including `X-RateLimit-Remaining` and `Retry-After` headers so well-behaved clients can back off gracefully. Worked example: a SaaS company's AI-powered document-summarization API costs them real money per call to the underlying LLM. They set a rate limit at their API gateway: free-tier API keys are capped at 10 requests per minute, paid-tier keys at 500 per minute, tracked via a Redis-backed token bucket per API key. When a free-tier customer's integration has a bug causing it to retry a failed request in a tight loop, the rate limiter catches it after the 10th request within that minute, returning `429 Too Many Requests` with a `Retry-After: 45` header instead of letting the buggy loop rack up hundreds of expensive AI API calls — protecting the company's margins and giving the customer's client a clear, machine-readable signal about exactly when to retry.