Hi @logankilpatrick - Streaming mode will ALWAYS stop after 5 minutes, which is not enough for all GPT4 answers. See Stream error after 5 minutes · Issue #399 · openai/openai-python · GitHub
And I can confirm that unfortunately this is still an issue today
It isn’t just GPT-4. gpt-3.5-turbo will do the same on long streaming (like if using high n trials). Same connection broken: InvalidChunkLength(got length b'', 0 bytes read of an empty delta.
Summary created by AI.
In this discussion, users are experiencing timeout errors and incurring charges on unsuccessful long requests sent using the GPT-4 API. julius.jacobsohn was the first to report his issue, stating that he is billed every time he sends a long request, regardless of whether it is fulfilled or not. He updated his experience, saying that even with the streaming variant, he receives an IOException indicating that the response ended prematurely. Similar experiences have been reported by shawn.veltman, THOMASD, gm.mikhailov, and Fusseldieb, all also reporting billing for these unsuccessful requests.
jwr calls this practice fraudulent, as he is billed for every unsuccessful request, hampering his product development. pproviamo offered some hope, mentioning that after tweaking his script and refocusing on GPT 3.5, he experienced a significant reduction in call times, implying the issue may be partially resolved.
Meanwhile, Quentin_Beillard suggested a modification to the API to improve it and possibly solve the problem. The suggested solution is a server response indicating a state of “break” that would allow for a POST on the conversation ID to resume generation. Nonetheless, as of the time of these posts, it seems the issue of gateway timeouts and being billed for unsuccessful API calls remains unresolved.
Summarized with AI on Nov 30 2023
AI used: gpt-4-32k