I, like many of the rest of us, are running into the random 600s read_timeout error with the ChatCompletions api using gpt-3.5-turbo. I have looked through the python code and see that there is a timeout parameter that doesn’t seem to be documented in the api docs. [image] OpenAI API An…

I call the API from AWS Lambda in production, which has a configurable timeout value for each function you write. It has built in retry’s too. For high priority stuff, I would invoke the function and also write the invocation event to a database. Then check the database every minute to see if it f…

In addition to this: I’d also recommend having some sort of fallback strategy. Sometimes a model is completely off but the other ones are working seamlessly. Retrying the same model over and over will not help, but falling back to a different (usually worse) model will do. In my case, a generic ser…

[image] AgusPG: pinging them every minute, and updates this models’ health in a database An ICMP echo ping only check the network. You need to call the API to “ping” the model. :slight_smile:

You’re 100% right. Imprecise term here. I do not ping, but actually do a dumb API call. Thanks for the correction!

[image] AgusPG: I do not ping, but actually do a dumb API call. Haha welcome! I was confident, knowing you @AgusPG , that you were using a bare bones API call to “ping” the model! Thanks for the precision. :slight_smile:

@tdenton8772 , can you show an example? Thanks

Lord, please do not use undocumented parameters. They are undocumented for a reason. Use your own timeout feature with a library such as Retry. It’s so simple it is insane that it’s not already being done. retry · PyPI @retry(delay=1, backoff=2, max_delay=120) def func(): pass You can read more…

from wrapt_timeout_decorator import timeout import openai @timeout(35) def engine_abstraction(model, prompt, max_tokens=2046, n=1, stop=None, temperature=1, job=None): if model == "gpt-3.5-turbo": response = openai.ChatCompletion.create( model="gpt-3.5-turbo", …

[image] anon10827405: retry · PyPI Fyi, I can’t seem to get retry or tenacity to work with a timeout. It seems like openai.ChatCompletion.create is blocking it? Any other suggestions?

Configuring timeout for ChatCompletion Python

API

tdenton8772 March 20, 2023, 3:31pm 11

The actual answer here is to use the undocumented parameter request_timeout

1 Like

Topic		Replies	Views
Frequent API timeout errors recently API	39	50479	December 12, 2023
Timeout for OpenAI chat completion in Python API api , python	6	31474	December 16, 2023
Recommended way to limit the amount of time a Python ChatCompletion.create() runs API gpt-4	8	2780	September 15, 2023
Setting request_timeout in openai v1.2.2 API	3	20000	November 10, 2023
Timeout not honored in Python API? API	11	4675	June 8, 2023

Configuring timeout for ChatCompletion Python

Related topics