Token Budget

A token budget is an explicit ceiling on how many tokens a request, a feature or a customer may consume. It is a product decision expressed in infrastructure terms, and without one, context grows until someone notices the bill. Context grows for good reasons, which is what makes it dangerous. Somebody adds three more retrieved documents to improve accuracy, somebody else appends the full conversation history, a third person includes the company style guide in the system prompt. Each change is defensible and the cumulative effect is a request that costs four times what it did last quarter for a few points of quality. A budget forces the trade into the open. Give each feature an allowance, measure the p95 against it, and make anything that exceeds it justify the spend — usually by showing a quality metric that moved. The practical enforcement points are truncation policy (what gets dropped first when the budget binds — usually the oldest turns, never the system rules), a retrieval cap, and a hard maximum on output length. Set them deliberately, because the defaults are set by whoever wrote the SDK example you copied.