Batch Inference

Batch inference runs model requests as a queued job instead of an interactive call. You submit many inputs, the provider processes them when capacity allows, and you collect results later — typically within hours, typically at a large discount to synchronous pricing. The test for whether work belongs in a batch is simple: does anybody wait for it? Classifying yesterday's support tickets, enriching a product catalogue, generating embeddings for a corpus, summarising documents overnight — nobody is watching the spinner, so latency costs nothing and the discount is free margin. The engineering difference is mostly failure handling. Interactive calls fail loudly and get retried in the moment; a batch fails partially, hours later, when nobody is at the keyboard. Write results incrementally, key them to inputs so a partial rerun is possible, and treat the job as resumable rather than atomic. The strategic point for a SaaS product is that batch changes what is affordable. Features that make no sense at interactive prices — re-scoring every record nightly, running an evaluation across the whole corpus — become routine when the same tokens cost half as much.