Timeout not honored in Python API?

API
1

Python API supposed to take timeout parameter for API calls. But it appears the timeout parameter is not honored. Possible a bug?

Signature: openai.Completion.create(*args, timeout=None, **kwargs)
Docstring:
Create a new instance of this model.

Parameters:
timeout (float): the number of seconds to wait on the promise returned by the API, where 0 means wait forever.
File: /opt/anaconda3/lib/python3.7/site-packages/openai/api_resources/completion.py
Type: method

1 Like
2

Hi @kumarM

Can you share the code that you’re using to make the API call?

3

import openai

api_key = “your_openai_api_key”
openai.api_key = api_key

prompt = “Translate the following English text to French: ‘Hello, how are you?’”

try:
response = openai.Completion.create(
engine=“davinci-codex”,
prompt=prompt,
max_tokens=50,
n=1,
stop=None,
temperature=0.5,
timeout=10, # Set the timeout to 10 seconds
)
print(response.choices[0].text.strip())
except openai.error.ApiError as e:
print(f"Error occurred: {e}")

Timeout here is set to 10 seconds. Last 24 hrs there were too many timeouts with API calls but timeout error never occurred.

5

100% aligned with @RonaldGRuckus

1 Like
6

Ronald,

See attached screenshot of code for API.

Screenshot 2023-03-20 at 8.57.40 AM
Screenshot 2023-03-20 at 8.57.40 AM1536×810 72.5 KB

7

This is very neat. Nice catch.

Although I would still recommend using a client timeout and not relying on the server. It’s a very simple module to use and will be much more robust. Using hidden parameters is not recommended.

Looking a bit further, it could be possible to use the timeout currently - although again, it’s not really a good idea to use hidden parameters. There’s one other issue: You’re using Codex.

1 Like
8

@RonaldGRuckus - The OpenAI Python library referenced is code run by the client. So it’s perfectly sensible to want/expect the timeout argument to be used in some client-side timeout implemented by the library. I’d prefer that to having to pull in another library myself.

Also, your advice about not using Codex is neither what the OP asked, nor necessarily correct. See footer 2 of this paper.

10

@kumarM - I don’t think that timeout argument actually makes its way into the calling function (which is a few layers of wrapping around urllib3). The default client timeout in the openai library appears to be set here. It is absurdly long (10 minutes), but I don’t see any reason why you couldn’t shorten it. For example, when I run:

import openai

openai.api_key = OPENAI_KEY  # your api key
openai.api_requestor.TIMEOUT_SECS = 1  # set to 1 second just for demo, probably need it to be longer

completion = openai.ChatCompletion.create(
    model="gpt-3.5-turbo",
    max_tokens=256,
    messages=[
        {"role": "user", "content": "Why is the sky blue?"}
    ]
)

I get a Traceback ending with

Timeout: Request timed out: HTTPSConnectionPool(host='api.openai.com', port=443): Read timed out. (read timeout=1)

@RonaldGRuckus - my last comment may have been a little too abrasive. I just thought it was important to clarify for the OP and future readers that 1) it does make sense to have a working timeout argument in a client-side api library, and 2) it’s fine/reasonable to try using Codex for translation, there is evidence that it out-performs text-davinci-xxx on many NLP tasks.

11

Thanks for the clarification.

They use aiohttp so the timeout mechanism can be found here:

https://docs.aiohttp.org/en/stable/client_quickstart.html#timeouts

So the value that you are changing is the direct variable they use with this timeout parameter.
They also use the (I believe same) timeout parameter in other sections

        while True:
            try:
                return super().create(*args, **kwargs)
            except TryAgain as e:
                if timeout is not None and time.time() > start + timeout:
                    raise

Which is eventually added back into the request

response, _, api_key = requestor.request(
            "post",
            url,
            params=params,
            headers=headers,
            stream=stream,
            request_id=request_id,
            request_timeout=request_timeout,
        )
[...]
def wait(self, timeout=None):
        start = time.time()
        while self.status != "complete":
            self.timeout = (
                min(timeout + start - time.time(), MAX_TIMEOUT)
                if timeout is not None
                else MAX_TIMEOUT
            )
            if self.timeout < 0:
                del self.timeout
                break
            self.refresh()
        return self

Thank you. That, and the codex are two things I didn’t know. I would still rather my own client-side library to handle and retry the request though, but it’s nice to have. I didn’t intend to say that client-side timeouts are useless, I’ll try and be more clear in the future.

I believe what you are doing is a little “hacky” - you’re changing a constant instead of having the library manage it for you. If it works, it works though!

12

Agreed. It would be nice if this was exposed intentionally and documented.

1 Like
13

After this, “timeout” is popped again in the __prepare_create_request method in the superclass EngineAPIResource. This second pop would always set “timeout” to None. It does not make sense to me why it is popped twice, feels like a bug. But you can set request_timeout.

Screenshot 2023-05-23 at 18.00.40
Screenshot 2023-05-23 at 18.00.401274×862 80 KB

14

Using your own library to manage timeouts would mean paying for the requests that are of no use to us. It would work, but would also be quite expensive, at least for my use case.

In my opinion, the API needs to add support for a timeout.

1 Like