Concurrent request restriction

[sps]

Concurrency Rate Limiting: A $10,000 Issue

We are observing some irregularites in API response latency, despite being well within our Tier 5 rate limits. We’d love to better understand how the concurrency rate limits actually work, as they have very tangible, real impacts on our users’ experience with us. The irregularity we are observing is, when sending multiple API requests in parallel to gpt-4o-mini (138 requests in parallel; total token count ~15000 and roughly ~100 tokens/request), we have a return latency of 40 seconds. This is f…