I, like many of the rest of us, are running into the random 600s read_timeout error with the ChatCompletions api using gpt-3.5-turbo. I have looked through the python code and see that there is a timeout parameter that doesn’t seem to be documented in the api docs. [image] OpenAI API An…

[image] tdenton8772: I have implemented wrapt_timeout_decorator but there really should be a configurable timeout in the api request with a retry and error handler wrapping the call. Hi @tdenton8772 Actually, normally managing the timeout for a client-server transitions, like the OpenAPI API…

I call the API from AWS Lambda in production, which has a configurable timeout value for each function you write. It has built in retry’s too. For high priority stuff, I would invoke the function and also write the invocation event to a database. Then check the database every minute to see if it f…

In addition to this: I’d also recommend having some sort of fallback strategy. Sometimes a model is completely off but the other ones are working seamlessly. Retrying the same model over and over will not help, but falling back to a different (usually worse) model will do. In my case, a generic ser…

[image] AgusPG: pinging them every minute, and updates this models’ health in a database An ICMP echo ping only check the network. You need to call the API to “ping” the model. :slight_smile:

You’re 100% right. Imprecise term here. I do not ping, but actually do a dumb API call. Thanks for the correction!

The actual answer here is to use the undocumented parameter request_timeout

@tdenton8772 , can you show an example? Thanks

Lord, please do not use undocumented parameters. They are undocumented for a reason. Use your own timeout feature with a library such as Retry. It’s so simple it is insane that it’s not already being done. retry · PyPI @retry(delay=1, backoff=2, max_delay=120) def func(): pass You can read more…

from wrapt_timeout_decorator import timeout import openai @timeout(35) def engine_abstraction(model, prompt, max_tokens=2046, n=1, stop=None, temperature=1, job=None): if model == "gpt-3.5-turbo": response = openai.ChatCompletion.create( model="gpt-3.5-turbo", …

Configuring timeout for ChatCompletion Python

API

ruby_coder March 20, 2023, 9:30am 10

Haha welcome!

I was confident, knowing you @AgusPG, that you were using a bare bones API call to “ping” the model!

Thanks for the precision.

1 Like

Topic		Replies	Views
Frequent API timeout errors recently API	39	50474	December 12, 2023
Timeout for OpenAI chat completion in Python API api , python	6	31463	December 16, 2023
Recommended way to limit the amount of time a Python ChatCompletion.create() runs API gpt-4	8	2780	September 15, 2023
Setting request_timeout in openai v1.2.2 API	3	19988	November 10, 2023
Timeout not honored in Python API? API	11	4674	June 8, 2023

Configuring timeout for ChatCompletion Python

Related topics