Frequently getting API timeout error What am I doing wrong?

API
1

I frequently get API timeout errors, deeply frustrated

Screenshot 2024-02-02 at 2.45.56 PM
Screenshot 2024-02-02 at 2.45.56 PM1720×676 123 KB

I have tried various settings like:

client = OpenAI(api_key = os.getenv(“OPENAI_API_KEY”),timeout=20.0, max_retries=3)

or also:
granular_timeout = httpx.Timeout(25.0, connect=3.0, read=20.0, write=2.0)
client = OpenAI(api_key = os.getenv(“OPENAI_API_KEY”),timeout=granular_timeout, max_retries=2)

What am I doing wrong can anyone help?? Some requests are taking more then a minute as shown in the picture and I get timeout error that should not be happening.

2

The problem is likely that you are not giving time for the AI to respond.

A 20 second timeout terminates a non-streaming response before you receive any text, even if the AI is still generating. Then that setting retries uses the internal openai python library retry method to do the same thing (at your expense) and again hang up on the connection before the response is complete.

Besides setting the timeout higher to 120 seconds+:

You should try out, employ, and parse stream:true API parameter so you can see responses as they are generated and know what the AI is doing.

2 Likes
3

This is most likely the correct answer, I’m at 60 seconds for timeout and have no issues. Now if you’re in a production environment with customers, even 60 seconds would not be acceptable. You might want to try streaming responses.

4

I have streaming enabled already. Should I keep timeout to 60.0 ??

or more granular timeout like:

granular_timeout = httpx.Timeout(60.0, connect=3.0, read=20.0, write=2.0)

5

Until lately, I was using 30 seconds and five retries. My prompts are becoming increasingly complex, which made me get a lot of timeouts, too. This brought me to 60 seconds and one retry, which I am testing now—no streaming needed for my use cases.