I was also having this problem and worse, I noticed that OpenAI’s API would hang but not kill the connection so it would wait for minutes on end doing nothing. My solution was to thread everything and externally kill the thread after a 30 second timeout and then retry. Here is what that looked like in Python, I had to use Async because neither futures nor Python threading have an effective way to externally terminate a thread.
async def async_openai_request(task_name, messages, chat_container, max_retries=5, attempt_timeout=15):
url = "https://api.openai.com/v1/chat/completions" # Corrected URL
for attempt in range(max_retries):
try:
async with httpx.AsyncClient(timeout=attempt_timeout) as client:
response = await client.post(
url,
json={
"model": "gpt-3.5-turbo",
"messages": messages,
"temperature": 0.1 # Adjust temperature as needed
},
headers={"Authorization": f"Bearer {openai.api_key}"}
)
response_data = response.json()
if 'choices' in response_data and len(response_data['choices']) > 0:
chat_container[task_name] = response_data['choices'][0].get('message', {}).get('content', '').strip()
else:
raise ValueError("Invalid response format from OpenAI API")
print(f"{task_name}: Success on attempt {attempt + 1}")
return
except Exception as error:
print(f"Attempt {attempt + 1} for {task_name} failed. Error was {error}. Retrying...")
if attempt < max_retries - 1:
await asyncio.sleep(2 ** attempt)
print(f"{task_name}: Failed to get a successful response after {max_retries} attempts.")
chat_container[task_name] = "Failed after retries"
and here’s what the call looks like. Replace create_translation_prompt with your function.
await asyncio.gather(
async_openai_request("chinese_translation",
create_translation_prompt(word, example, "Chinese", just_word=True),
chat_container, max_retries=5, attempt_timeout=15),
async_openai_request("chinese_example",
create_translation_prompt(word, example, "Chinese"),
chat_container, max_retries=5, attempt_timeout=15),
async_openai_request("spanish_translation",
create_translation_prompt(word, example, "Spanish"),
chat_container, max_retries=5, attempt_timeout=15)
)
I found it works quite quickly if you just terminate the thread and retry. Unless you’re asking it something pretty lengthy (in my case I’m not), 30 seconds should be more than enough for it to respond.
Output looks like this:
Processing: dekket
spanish_translation: Success on attempt 1
Attempt 1 for chinese_example failed. Error was . Retrying...
Attempt 1 for chinese_translation failed. Error was . Retrying...
chinese_translation: Success on attempt 2
Attempt 2 for chinese_example failed. Error was . Retrying...
Attempt 3 for chinese_example failed. Error was . Retrying...
Attempt 4 for chinese_example failed. Error was . Retrying...
chinese_example: Success on attempt 5
{'Chinese': '1. 餐桌已經準備好供晚餐使用。(Cānzhuō yǐjīng zhǔnbèi hǎo gōng wǎncān '
'shǐyòng.)\n'
'2. 地面被雪覆蓋。(Dìmiàn bèi xuě fùgài.)\n'
'3. 毯子 (Tǎnzi)',
'Chinese Example': '1. 餐桌已經準備好供晚餐使用。\n2. 地面被雪覆蓋。\n3. 他用一條暖和的毯子捲住自己。',
'Spanish': '1. Usage (Table Setting):\n'
' - Spanish: "puesta"\n'
'\n'
'2. Usage (Covered):\n'
' - Spanish: "cubierto"\n'
'\n'
'3. Usage (Blanket):\n'
' - Spanish: "cubrió"'}
(the context was an app that updated flashcards for me)
