I, like many of the rest of us, are running into the random 600s read_timeout error with the ChatCompletions api using gpt-3.5-turbo. I have looked through the python code and see that there is a timeout parameter that doesn’t seem to be documented in the api docs. [image] OpenAI API An…

Timeouts are usually designed outside of the actual request. [image] backoff Function decoration for backoff and retry

That is a decent idea, thank you. I have implemented wrapt_timeout_decorator but there really should be a configurable timeout in the api request with a retry and error handler wrapping the call.

[image] tdenton8772: I have implemented wrapt_timeout_decorator but there really should be a configurable timeout in the api request with a retry and error handler wrapping the call. Hi @tdenton8772 Actually, normally managing the timeout for a client-server transitions, like the OpenAPI API…

I call the API from AWS Lambda in production, which has a configurable timeout value for each function you write. It has built in retry’s too. For high priority stuff, I would invoke the function and also write the invocation event to a database. Then check the database every minute to see if it f…

In addition to this: I’d also recommend having some sort of fallback strategy. Sometimes a model is completely off but the other ones are working seamlessly. Retrying the same model over and over will not help, but falling back to a different (usually worse) model will do. In my case, a generic ser…

[image] AgusPG: pinging them every minute, and updates this models’ health in a database An ICMP echo ping only check the network. You need to call the API to “ping” the model. :slight_smile:

You’re 100% right. Imprecise term here. I do not ping, but actually do a dumb API call. Thanks for the correction!

[image] AgusPG: I do not ping, but actually do a dumb API call. Haha welcome! I was confident, knowing you @AgusPG , that you were using a bare bones API call to “ping” the model! Thanks for the precision. :slight_smile:

The actual answer here is to use the undocumented parameter request_timeout

Configuring timeout for ChatCompletion Python

API

anon10827405 March 20, 2023, 1:53am 4

Could you imagine asking your chef to tell you if things are too busy?
He’d probably be too busy to tell you.

1 Like

Topic		Replies	Views
Frequent API timeout errors recently API	39	50493	December 12, 2023
Timeout for OpenAI chat completion in Python API api , python	6	31487	December 16, 2023
Recommended way to limit the amount of time a Python ChatCompletion.create() runs API gpt-4	8	2780	September 15, 2023
Setting request_timeout in openai v1.2.2 API	3	20007	November 10, 2023
Timeout not honored in Python API? API	11	4676	June 8, 2023

Configuring timeout for ChatCompletion Python

Related topics