Concurrency Limit

A concurrency limit is a cap on how many requests an account may have in flight simultaneously, as distinct from a rate limit, which caps how many requests may be started within a window of time. The two are frequently confused and they constrain completely different things. A rate limit of sixty requests per minute is easy to satisfy with one request per second; a concurrency limit of five means that no matter how long you wait between batches, only five requests can be open at any moment. Concurrency is the limit that matters for AI workloads specifically, because inference requests are slow. A traditional API call returns in tens of milliseconds, so even a modest concurrency allowance supports high throughput. A model generating a long response may hold a connection open for many seconds, which means a low concurrency cap directly sets your maximum throughput regardless of how generous the per-minute quota looks. A job that needs to process ten thousand documents is bounded by concurrency and latency together, and a per-minute rate limit is often irrelevant to how long it takes. Three things to establish before you build on a tool. What the concurrency limit actually is on your plan — it is documented far less consistently than rate limits and sometimes only appears once you hit it. What happens when you exceed it: a queued request that eventually completes is very different from an immediate error that your code must catch and retry with backoff. And whether the limit is per key, per account or per model, since splitting work across keys is either a legitimate scaling path or a terms-of-service violation depending on the vendor. Design batch work to a measured concurrency ceiling rather than an assumed one, and treat an upgrade in that ceiling as a real commercial term to negotiate.