Throughput

Throughput measures how much work an AI system can process per unit of time — typically expressed as tokens per second (for a single request's generation speed) or requests per second/minute (for how many concurrent users a given piece of infrastructure can serve). Throughput and latency are related but distinct, and the difference matters practically: latency is about how fast one single request completes, while throughput is about how much total volume a system can sustain — and optimizing for one can sometimes come at the cost of the other. A GPU serving LLM inference can often process multiple requests concurrently by "batching" them together (running several users' prompts through the model in the same forward pass), which increases total throughput significantly but can slightly increase the latency experienced by any individual request, since it may wait briefly for a batch to fill. This matters for SaaS builders scaling an AI feature from a prototype (handling a handful of test requests) to production (handling thousands of concurrent users): a feature that works fine with 10 test users can hit throughput ceilings at scale — either your own self-hosted infrastructure runs out of GPU capacity to serve requests fast enough, or you hit provider-side rate limits (API providers commonly enforce both requests-per-minute and tokens-per-minute caps per account/tier) that throttle your application under load. A concrete worked example: a SaaS company launches an AI-powered email-summarization feature and initially calls a frontier model API directly per-user-request; at 50 concurrent users this works fine, but at 5,000 concurrent users during a product-launch traffic spike, they start hitting the API provider's tokens-per-minute rate limit, causing requests to queue or fail. The fix involves several throughput-oriented architectural changes: requesting a higher rate-limit tier from the provider, implementing request queuing with graceful backpressure (so the UI shows "processing" rather than erroring), batching where possible, and potentially routing high-volume/low-complexity requests to a smaller, faster, higher-throughput model tier while reserving the frontier model for cases that need it. Understanding your expected throughput requirements — not just per-request latency — is essential before choosing model tier, provider rate limits, and self-hosted vs. API architecture. Throughput planning should also account for the fact that different task types have very different token profiles, and thus very different throughput costs at the same request volume: a simple yes/no classification task generates a handful of output tokens per request, while a long-form report-generation feature might generate thousands — meaning "requests per second" alone is an incomplete capacity metric, and teams sizing infrastructure or negotiating provider rate limits should model expected tokens-per-minute based on realistic output-length distributions for their specific feature mix, not just raw request counts.