How limits apply
- Account concurrency covers requests across all models of the account.
- Model concurrency: some models have a separate limit of their own, independent of the account limit.
- Above the concurrency limit, generation requests are still accepted. They return a
request_idwith statusqueuedand start automatically as soon as one of your running requests finishes. You do not need to retry them. - Queue limit: the number of generations waiting to start is capped at a multiple of your concurrency limit, counting each output of a batch. When the queue is full, new generation requests return
429 Too Many Requestswith"code": "account_queue_full", the account’s queuelimitand aRetry-Afterheader; nothing is created or charged. See When the queue is full. - Some non-generation endpoints reject requests above their own limit with
429 Too Many Requestsand aRetry-Afterheader. See Rate limits.
Plan for scale
- Submit through a worker pool or semaphore sized to your concurrency, so the queue stays short, latency stays predictable and the queue limit is never reached.
- On
429, wait forRetry-Afterand resubmit with the sameIdempotency-Key. - Track every accepted
request_iduntil it reaches a terminal status, with webhooks and polling as a fallback. - Send an
Idempotency-Keywith every submission so retries after timeouts never create duplicates.