Inference Latency

Inference latency is how long a model takes to respond, and it is two numbers rather than one. Time to first token is the wait before anything appears; throughput, in tokens per second, is how fast the rest arrives. Users experience them differently, and optimising the wrong one wastes effort. For a chat interface, time to first token dominates perception. A response that starts in 300ms and streams for four seconds feels faster than one that appears complete after two — which is why streaming is a product feature, not a technical detail. For a batch job nobody watches, only total time matters and streaming is irrelevant. The biggest contributors are usually not the model. Prompt length drives prefill time, so a system prompt that grew to 4,000 tokens taxes every request. Retrieval and tool calls add round trips. A chain of three model calls has three cold starts. And a router that picks a bigger model for hard requests trades latency for quality on exactly the requests users are least patient with. Measure p95, not the mean. The mean hides the tail, and the tail is what people remember.