Interesting. But that’s still about hard token limits. Clearly, some users/organizations are being throttled, well within all those limits, and that’s not mentioned anywhere. The only mention of throttling in the docs is about throttling it yourself to stay with the provided limits.
I have 2 organizations, one of those is experiencing the slowness, the other is fast as we are used to with GPT 3.5 Turbo. However, the organization that did not have any traffic in the last couple of months is faster than the one i’m using in a production setting with not a lot of users, so very well within all the limits and quotas.
Demo of the issue:
Another topic about the slowenss, that mentions it is only happening for certain organizations: GPT-3.5 API is 30x slower than ChatGPT equivalent prompt
The slowness just doesn’t make any sense. It’s not documented anywhere. There’s no status update. And any support request to OpenAI is closed mentioning it is probably your own fault. EVEN when I send them the demo video you see above.
For now I have fixed it by using a different organization and by using Azure’s OpenAI API’s. But that’s not really a fix. Thinking about migrating every service I have to Azure OpenAI.