I’m using the OpenAI API in a Python 3.10 program using the python client library, the model is ‘gpt-3.5-turbo’. I’ve noticed that periodically the openai.chatcompletion call randomly takes a very long time to complete. I want to protect my users from having to wait for completion by timing out the API request.
After reviewing various sources (including these forums), I’ve tried a few different ways to implement the timeout:
- Setting the API parameter request_timeout=60 . This doesn’t seem to be honored by the API, calls infrequently last for 1000s of seconds.
- Using a variety of decorator functions around the API client call: timeout-decorator, timeout-function-decorator
- Directly using signal.alarm in python.
None of those work in actually timing out the function call; the program hangs until the call returns or until there is an eventual OpenAI timeout. I’ve tried using signal.alarm with a test program and it works perfectly. I’m running this on my Mac.
I’m wondering now if the OpenAI Python client creates its own threads? That would explain why signal doesn’t work. So I’m going to try calling the API directly using the requests library; if that doesn’t work, I’ve run out of ideas to try.
Has anyone else run into this problem? Any feedback much appreciated!