I am using the OpenAI Python SDK for chat completions. I have specified a timeout of 2s but the the SDK is raising APITimeoutError after 10 seconds. If I set the timeout at 5s, it is erroring out after 18 seconds. Is this the expected behaviour? Is it possible to raise the exception after the specified timeout?
Hey there and welcome to the community!
Is the timeout in the docs? I’ve been looking through the API reference but I don’t see it anywhere yet. That right there might already be indicative of an issue, although it’s been a hot minute since I’ve touched the API.
Have you tried a 2s timeout with LLMs before? 2s might be a bit short for a model like 4o. 2s latency is usually for 4o mini/3.5 & Gemini Flash.