So after writing here before, we decided to make an extensive test to confirm with ourselves that OpenAI is intentionally making the API slow, and it is not our imagination.
Our prompt produce code only.
We took 20 similar prompts, and for each one of them, we tried it via the API and the website in the same time.
Not only that, we tried the same 20, in different times of the day.
When trying online we counted also the time for the typing effect since it is not clear if its artificial yet. We setup the API to run from a server close to OpenAI both in the West Coast and US Central.
The result is not surprising. On average (average over the 20 prompts) the API respond time was x4.5 slower.
Here are some of the results:
38s API , 7.8s online
18s API, 6.3s online
9.8s API, 5.2s online
45s API, 8s online
So,
OpenAI - are you limiting developers? if so can you declare this and also the date you will stop doing this (if ever) ?
I may be in the minority here, but I donāt mind the looooooong response times, itās still a service in development, you can still develop and test your product. This is great when the waitlist is still being processed.
Letās say that company A gains access to GPT-4 and shorter response times, enabling to deploy their service. company B is still stuck on the waiting list and canāt launch their service. Said company might get angry and sue OpenAI claiming anti-competitive or unfair business practices.
Iād much prefer that OpenAI just open the waitlist floodgates and focuses on improving their servers, sure the response time will be worse, but only temporarily.
What are you talking about ? there are no waiting lists here, everyone has slow API because they just donāt really want you to be able to do anything big enough yet before they figure out their business.
Thatās fine if they can go out and say it, not telling us itās due to ānetworkā problems so we know we should leave.
I think they make a big mistake, because the next competitor is coming soon.
Are you a programmer? or at least did you read my post?
The same query takes 2 second on their website and 30s on the API. This has nothing to do with server load.
no it doesnāt, the amount of the API requests is maybe 0.1% of their total customers, this has 0 effect on them, they just donāt want you to go too fast as a developer, and they can just say it clearly.
In theory, the priority of the website should be higher because it can be used for training GPT, while the API becomes slower as usage increases.
However, this is one of the largest services with over billion users worldwide. It is estimated that they were not prepared with enough computing resources for this scale last year.
I also think it is highly likely that Openai intentionally slowed down our API speed, and my argument is as follows:
1ć I have been using the API every day since then, and I am particularly concerned about its response speed, so every time I ask a question, I pay attention to its speed. So, I am very sensitive and clear about his speed.
However, the APIās response underwent a mutation around April 6th Beijing time in China. His response speed suddenly changed from around 400ms to around a few seconds.
If the speed slows down due to increasing user numbers and server overload, why is it not a linear increase (although the probability of a sudden increase in a large number of users is also very low), but a sudden change? He basically increased his user base several times compared to before in less than 24 hours. This is worth everyoneās deep reflection.
2ć If itās because there are too many people, then it should also be divided into time periods. Why is it the same slow 24 hours a day?
For the above arguments, the conclusion points to the direction where OpenAI intentionally slows down its speed.
But I really hope my inference is wrong, but can anyone overturn my argument above?
āThat model is currently overloaded with other requests. You can retry your requestā
This message comes in the API now for an hour long, but online there is full, fast access.
We made our decision, we are out, you canāt base a company on this and they donāt want you to.
It is very likely that the stack servicing the API requests is completely different than the stack servicing the Web requests. It is also very likely that the API stack is sized differently than the Web stack. There are times when Iām using the APIs and they just timeout. There are other times when itās pretty fast. For a service as new and fresh as OpenAIās apis, I feel like they are doing a pretty decent job and letting us have it for very cheap. No complaints here.
I donāt see any reason why itās not as fast as the playground, if like someone said they have over a billion users and they can handle that but they canāt handle a small amount of developers API calls. Like the name suggests ChatGpt , itās not exactly chatting if I have to wait nearly over a minute for response, in fact developer API calls should be faster than even playground or bing. Itās clear they are deliberately doing this. Slow API calls, makes the whole thing almost useless, we would expect more transparency and a truthful response from OpenAI
So OpenAI can support 1 billion people but not say 200 developers? What the freakin hell is wrong with you guys OpenAI? will you have the odacity to respond and fix the stupid slow API calls? or are you too busy making money?
there are no waiting lists here, everyone has slow API because they just donāt really want you to be able to do anything big enough yet before they figure out their business.
