Hi, I have made calls to the openai api chat completions many times, no problem, straight from Python.
Now I am trying to move my code to be run as tasks using the apache airflow as an orchestrator. Now my openai api calls hang without an error, silently and I put in logging/print statements, try/except constructs - no luck, I cannot extract any error. It fails silently.
I was looking around and I found the apache-airflow-providers-openai - but this one deals only with embeddings.
Did anyone else experience problems like this? calling openai api from inside an apache-airflow task?
Subject: Issue with OpenAI API Calls Hanging in Apache Airflow Tasks
Body:
Dear OpenAI Support Team,
I am experiencing a recurring issue where calls to the OpenAI API, specifically the chat completions, are hanging indefinitely without producing any errors when executed from within Apache Airflow tasks. These calls have been functioning flawlessly when made directly from Python scripts. However, the transition to executing these calls as tasks orchestrated by Apache Airflow has led to this problem.
Despite implementing extensive logging and try/except blocks in an attempt to diagnose the issue, the API calls fail silently, offering no indication of what might be going wrong. I am unable to extract any error messages or logs that could point me towards a solution.
In search of a potential solution, I came across apache-airflow-providers-openai, but it appears to focus exclusively on embeddings, which does not align with my use case centered around chat completions.
I am reaching out to inquire if this is a known issue and if there are any recommended practices, solutions, or workarounds for integrating OpenAI API calls, particularly chat completions, within Apache Airflow tasks. Any guidance or suggestions from the community or the OpenAI team would be greatly appreciated.