What are the best practices for retrying completion requests? When I occasionally encounter “That model is currently overloaded with other request” I would like to back off and try the same request again in a few seconds. Does OpenAI python library use a standard requests library that includes a retry function?
1 Like
At the moment we haven’t implemented this, but it’s a good idea, and I’ll add it to our backlog.
When using the API for processing batch jobs which shouldn’t fail I normally implement a retry logic which waits 1 second on first failure, and then a time doubling on each subsequent failure up to some maximum.
I have developed a simple Python package which provides retry logic around the OpenAI API. For some reason I am not allowed to provide a link, but the package is called openai-pygenerator and can be installed using pip install openai-pygenerator. Search for it on github to view the README.
Now-a-days and in some cases the API will return a string similar to “429 Too Many Requests - failed OpenAI request. Retry-After: 16”. In such cases the code can retry after 16 seconds and expect to have their request served. The 16 may be different from one call to the next and this particular message is received when your account limit has been reached, not because of general service overload.