Beam Search

Beam search is a decoding strategy that keeps several candidate continuations alive at once instead of committing to a single token at each step. The decoder maintains a fixed number of partial sequences — the beam width — extends each of them, scores the results by cumulative likelihood, and keeps the best few. At the end it returns the highest-scoring complete sequence. Compared with greedy decoding, which always takes the locally most probable token and cannot recover from a bad early choice, beam search explores enough of the tree to find sequences that start unpromisingly and end well. Its natural home is tasks with one right answer: machine translation, speech transcription, structured extraction, code that must parse. It is much less suited to open-ended generation. Optimising for total likelihood systematically favours short, safe, repetitive text, which is why free-form assistants are usually served with sampling methods governed by temperature and top-p instead. Cost is the other consideration — a beam of width four is roughly four times the compute of greedy decoding for the same output length. Most hosted chat APIs do not expose beam search at all, so in practice it is a lever you reach for when running a model yourself, on a task where a single correct output exists and you would rather pay for it than sample repeatedly and pick.