I have a system that sends about 7-10 large requests per task and receives large responses (4K - 8K tokens).
I am using GPT-4 via the API - 8K limit (I’d love to get 32K!)
For the whole time I’ve been developing it I have often received:
Error communicating with OpenAI: (‘Connection aborted.’, RemoteDisconnected(‘Remote end closed connection without response’))
I have a retry with an exponential backoff.
Depending on the day and time, my system may succeed in running, but I don’t get too many tries per day.
Worse, I am billed for all of these calls. I can see the times and token counts on the billing page.
Seemingly GPT-4 completes the work, but disconnects my connection after 5 minutes, so I never see the results.
This is crazy all around.
Don’t disconnect after 5 minutes!
Hey OpenAI, your system is slow because it’s heavily loaded and you know this, can’t you bump your timeout up? Why do you bill people for requests they never see the responses to? Why don’t you answer my inquiries via the Help Center? You have an amazing product, and we know you’re very busy, but this is pretty basic stuff!
If you don’t set a timeout the whole thing will cascade into something uncontrolable.
I think instead of timeout they’d should just stop accepting request and redirect to a “server busy” page on a round robin logic per user.
That together with a documentation entry that states: if you get “server busy you have to wait 2 seconds before you are allowed to send the next request”…