Completion Endpoint Randomly Freezes

API
1
  1. Does anyone else run into the issue where “open.Completion.create” just hangs and doesn’t return anything sometimes?
2

No really. However, performance is model dependent.

The completion API (in general) performs much faster than the new chat completion API.

Maybe you can post your code so we can see it?

Thanks

:slight_smile:

3

Sure! I’m just using this. But sometimes (maybe 10% of the time) it just never returns a response and it times out.

 response = openai.ChatCompletion.create(
        model="gpt-3.5-turbo",
        messages=[
            {"role": "system",
                "content": "You are an intelligent, helpful, good teacher explaining to a student. Answer as concisely as possible."},
            {"role": "user", "content": f"{questionText}"},
        ],
        temperature=0,
        max_tokens=150,
        top_p=0.5,
        stream=True,
        frequency_penalty=0,
        presence_penalty=0
    )
4

Well, we learned a lot from you code, @derek8bai

You are talking about the chat completion endpoint and not the completion endpoint and you are using the gpt-3.5-turbo model,

It is well know these new chat completion models are very stressed with new users pounding on them and so they are very slow.

HTH

:slight_smile:

5

I’m having the exact same issue as @derek8bai. I’m also using the ChatCompletions endpoint with gpt-3.5-turbo with it hanging randomly. I’ve implemented a retry with exponential backoff, and it still doesn’t want to work. This is impacting the public image of our service. What other solutions have people found?

6

In my case, a time.sleep(2) solved the issue. I cannot say 100% but I was running a loop over 100 prompts to send, and it would always hang at some point. Very rarely reaching the 50th iteration. After adding the sleep command, it is now ending the loop