We are tier 5 so we have pretty high limits on RPM and TPM, However our typical request take 10 to 40 seconds each to process. We found that we are able to process only about eight concurrent requests before getting an error message or being throttled back to a slow pace. Therefore it is absolutely…

Reported before. Metrics I would look at would be the time to first token, to see if requests are being held back by intermediary steps such as Cloudflare, workers to check your input limits, and other processes outside of AI generation that are impactful. The solution to that, and for an organiz…

Thanks. Please correct me if I’m wrong but I think your response is addressing performance bottlenecks whereas the main question has to do with concurrency restrictions. If I have some delays due to various bottlenecks on the API calls my tests show that I get best performance around a maximum of e…

I’m saying that the API behavior has been observed before, with no response from OpenAI, nor documentation that explains the futility of prepaying and aspiring to a service level advertised and implied. I’m also saying that this throttling appearance - limitation on the allocation of API resources …

[image] Concurrency Rate Limiting: A $10,000 Issue API We are observing some irregularites in API response latency, despite being well within our Tier 5 rate limits. We’d love to better understand how the concurrency rate limits actually work, as they have very tangible, r…

Thanks to everyone for the replies. OpenAI, if you are listening… This is a critical issue being reported by many. In the spirit of helping businesses become more successful using your services, it would be very helpful to provide full transparency on this in your documentation.

I have the same issue. I’m tier 2, and EIGHT is also about the number of concurrent API calls I can process normally before it slowing down significantly. sometimes it’s 7, sometimes it’s 9… My requests usually take less than a second to process. when I batch together 8 or 9, the last one or two get…

After implementing a 0.1 second space between calls this problem has decreased, and my requests are even faster now (half a second vs. 1 sec). I also implemented asyncio.wait_for with a timeout limit of 3 seconds which is still sometimes triggered, so I don’t think I have solved this sustainably…

I’ve observed similar issues as well, despite being Tier 5. I made a small library in an attempt to overcome this ( concurrent-openai ). It’s just a hypothesis, but I think everything depends on how OpenAI computes the usage metrics. For example, if OpenAI uses a rate-based approach, where they deriv…

I have seen a lot of people comment about this and it is truly a problem that can affect the success of a business. I find it really odd the OpenAI does not provide documentation on this so we can best navigate this constraint. OpenAI, please provide documentation of this!!!

Concurrent request restriction

API

sps December 22, 2024, 6:49am 5

Topic		Replies	Views
Gpt-3.5 concurrent requests limit API	3	6354	February 2, 2024
My request are getting throttled back API gpt-4 , api-rate-limits	0	180	November 21, 2024
Parallel API Requests - Very Long Response Times API	7	4251	August 18, 2024
Chat GPT's API is significantly slower than the website with GPT Plus API	35	37857	December 12, 2023
30K Tokens Per Minute limit vs 128K+ Context Models – Is Long-Context Usage Actually Possible via API? API	3	480	November 23, 2025

Concurrent request restriction

Related topics