API calls fail to return results at high traffic times w/o error message

This is a report of behavior I’ve noticed with the API which might also explain some of the issues others have seen where there is no response from the API.

I’ve noticed that there are times when I get no response to queries via the API. A short ‘hello’ test might get a response but a longer message returns nothing.

By no response, I mean nothing: no error code and nothing is returned by the model, not even an empty result.

I’ve seen this with both 3.5-turbo and 4-turbo.

In most cases, not long after this I get a ‘server busy’ response. So my assumption is that there is a period immediately prior to the ‘server busy’ notification where the API isn’t able to respond to a query but the ‘server busy’ message hasn’t kicked in.

There might always be this lag between overload and the server returning the ‘busy’ message but it might explain the issues some people are experiencing.

1 Like

I’ve noticed this too a lot more this week for both 3.5 and 4 turbo. This is happening for API calls with long prompts and function calls. I’ve retries implemented with OpenAI python SDK. I can see the responses are so slow they’re causing http timeouts and retries.

Update: There were a lot of SSLRead timeout error in my logs as in this issue fix(client): retry if SSLWantReadError occurs in the async client by stainless-bot · Pull Request #804 · openai/openai-python · GitHub. Upgraded to SDK 1.3.7 and the retries are going through now.