- Does anyone else run into the issue where “open.Completion.create” just hangs and doesn’t return anything sometimes?
No really. However, performance is model dependent.
The completion
API (in general) performs much faster than the new chat completion
API.
Maybe you can post your code so we can see it?
Thanks
Sure! I’m just using this. But sometimes (maybe 10% of the time) it just never returns a response and it times out.
response = openai.ChatCompletion.create(
model="gpt-3.5-turbo",
messages=[
{"role": "system",
"content": "You are an intelligent, helpful, good teacher explaining to a student. Answer as concisely as possible."},
{"role": "user", "content": f"{questionText}"},
],
temperature=0,
max_tokens=150,
top_p=0.5,
stream=True,
frequency_penalty=0,
presence_penalty=0
)
Well, we learned a lot from you code, @derek8bai
You are talking about the chat completion
endpoint and not the completion
endpoint and you are using the gpt-3.5-turbo
model,
It is well know these new chat completion
models are very stressed with new users pounding on them and so they are very slow.
HTH
I’m having the exact same issue as @derek8bai. I’m also using the ChatCompletions endpoint with gpt-3.5-turbo with it hanging randomly. I’ve implemented a retry with exponential backoff, and it still doesn’t want to work. This is impacting the public image of our service. What other solutions have people found?
In my case, a time.sleep(2)
solved the issue. I cannot say 100% but I was running a loop over 100 prompts to send, and it would always hang at some point. Very rarely reaching the 50th iteration. After adding the sleep command, it is now ending the loop