ChatCompletion API Call - HANGS without producing response

API
, ,
1

I’m using the openai.ChatCompletion.create endpoint and on an absolutely random basis when running the function in a loop it will halt execution without returning a response.
I can not even shut off the process via KeyboardInterrupt.

This is a new issue and has not happened before even though using the basically same function and API key. If it were due to a rate limit I believe I would get some response at least.

Please evaluate my functions’s logic. ChatGPT suggests adding a timeout but this doesn’t seem to be available to simply add to the call.

Any suggestions?

def call_gpt(api_key, gpt_model=3, prompt="How are you?", input_text=""):
    logger = get_logger()
    if api_key:
        # Set your OpenAI API key
        openai.api_key = api_key
    else:
        return "No OpenAI API key..."

    if gpt_model == 3:
        gpt_model = "3.5-turbo"
    if isinstance(gpt_model, int):
        gpt_model = str(gpt_model)

    # Concatenate the prompt and input input_text
    full_prompt = prompt + str(input_text)

    attempts = 0
    while attempts < 5:
        try:
            # Send the request to the OpenAI API
            logger.debug(f"About to make the GPT-{gpt_model} OpenAI API call - attempt {attempts}")
            response = openai.ChatCompletion.create(
                model=f"gpt-{gpt_model}",
                messages=[
                    {"role": "system", "content": "You are a helpful assistant."},
                    {"role": "user", "content": full_prompt},
                ],
            )

            # Extract the generated summary from the API response
            output_text = response.choices[0].message.content

            # print(f"\n{output_text}")

            # Remove non-ASCII characters from the output_text
            output_text = output_text.encode("ascii", "ignore").decode()

            return output_text

        except (
            openai.error.RateLimitError,
            requests.exceptions.ConnectionError,
            openai.error.APIError,
            openai.error.ServiceUnavailableError,
            openai.error.APIConnectionError,
            requests.exceptions.ReadTimeout,
        ) as e:
            logger.warning(f"{type(e).__name__} encountered. New API call attempt in {(2**attempts)} seconds...\n{e}")
            time.sleep((2**attempts))
            attempts += 1
    return f"No valid response from OpenAI API after {attempts} attempts!"
2

You should put a sleep(1) in the loop so it doesn’t immediately go again with blocking requests.

If you want the console or a user interface to remain responsive, you need to use asynchronous IO or threading.

For a known short expected output, chatCompletion can take a parameter request_timeout = 30 using an appropriate amount of time that forces closure.

1 Like
3

Thanks!

What exactly do you mean by the ‘short expected output’?
Where would I add the request_timeout = 30?
To my response = openai.ChatCompletion.create()?

And then I would catch the timeout error and retry I’m guessiing.

4

A chatbot may answer in just a few words. Or may write a story.

Instead, your loop task always generates near the same from its prompt, so you don’t have to allow for the time for long text to be written.

The timeout parameter goes alongside other parameters like model = “gpt-4”. It sets a connection time limit, it doesn’t monitor for getting no output or extend the time if it is receiving. Without streaming, and a typical answer time of 10 seconds, you don’t know somethings gone wrong at 15, and a slow AI could still be writing at 25.

2 Likes
5

Faaantastic! Great answer, thank you. Will see to set the request_timeout as my usecase requires very short answers. Also for now the simple time.sleep(1) has not have me encounter my issue so far.