I tried the app engine for one of my use cases and unfortunately I needed streaming and SSE’s, Non of the App engine variants allow server side events so… was a no no for me.
That sounds about right. Google products, easy to set up, hair-pulling teeth gnashing difficult or impossible to customize.
My outdated urllib3 package was 100% the issue it turns out. Let me summarize for anyone else that stumbles across this.
I’m using the GPT-4 API with Python/Flask, deployed through Google App Engine.
I was having significantly more 500 server errors from one day to the next and thought it had something to do with an increase in OpenAI server traffic depending on the time of day. @Foxabilo confirmed that while it is somewhat slower during certain times, it’s faster overall and shouldn’t be the source of the problem.
- I confirmed that my app did not error while testing locally through Flask’s default server
- I introduced retry error handling. This worked to catch the rare 502 errors, but not the 500 server errors
- I confirmed that my completions were not crossing the 600 second default timeout time set by OpenAI
- I confirmed that it wasn’t GAE timing out, as the error would have been a
- I started digging into urllib3, which led me to the fix
The package urllib3, which auto-installs with openai, was version 1.26.16. The current version is 2.0.3. After updating to the current version, the 500 server errors have disappeared.
I’m still not sure why this issue popped up from one day to the next when I had been using the old urllib3 version for quite some time, but hey, a fix is a fix.
Thanks for your help @Foxabilo!