Top-p (Nucleus Sampling)

Top-p sampling, also called nucleus sampling, is an alternative (or complement) to temperature for controlling how an LLM selects its next token during generation. Instead of rescaling the entire probability distribution like temperature does, top-p works by ranking all possible next tokens by probability, then keeping only the smallest set ("the nucleus") whose cumulative probability adds up to at least p, and sampling only from that restricted set — every token outside the nucleus is excluded entirely, regardless of how low temperature might otherwise let it slip through. A top-p of 1.0 considers the full vocabulary (no restriction); a top-p of 0.1 restricts the model to only the tiny handful of tokens that together account for 90% of the "obvious" probability mass, producing very safe, predictable output; a common default like 0.9 to 0.95 excludes only the long tail of highly improbable tokens while still allowing meaningful variety. This matters for builders because top-p adapts dynamically to the model's confidence in a way a fixed temperature doesn't: when the model is very confident (one token dominates the distribution, like completing "The capital of France is ___"), the nucleus is small even at high p, so output stays accurate; when the model is uncertain (many plausible next words, like continuing a creative story), the nucleus is larger, allowing more variety exactly where variety is appropriate. A concrete example: generating product description variants with top-p=0.9 for an e-commerce SaaS tool lets the model avoid the single most generic phrasing every time ("This product is great and useful") while still excluding truly bizarre or off-topic completions, striking a balance that a high flat temperature alone might not. Most API providers (OpenAI, Anthropic) recommend adjusting either temperature or top-p, not stacking aggressive changes to both, since their effects compound in ways that are hard to reason about — and for most production SaaS use cases, leaving top-p at its default and only tuning temperature is the simpler, more predictable approach. In practice, most builders leave top-p at its provider default (commonly 0.9-1.0) and reach for temperature as the primary tuning lever, since top-p's effect is more subtle and harder to reason about in isolation — reserving explicit top-p tuning for cases where temperature alone isn't producing the right balance, such as a creative-writing feature that needs more lexical variety without becoming incoherent, where a moderate top-p combined with a moderate temperature often outperforms pushing either parameter to an extreme on its own.