Issues with google cloud functions

phillip.diekmann · November 22, 2023, 3:55pm

Hi,
I currently working with the whisper, moderate and gpt-4-1106-preview API. I initally made the calls from some google cloud function inside a firebase project and had a extreme high failure rate (>90%) even though using the recommended retry strategies proposed in the cookbook.

When making the API calls from my local machine I almost never had issues.

Then I switched using Heroku as the platform to host my backend code and it works fine there.

However, I would much prefer using cloud functions and right now I am experimenting with azure cloud functions where I also have a high failure rate right now.

Any idea where those issues might come from?

anon10827405 · November 22, 2023, 3:57pm

What are you seeing in the logs?

redfive · November 22, 2023, 8:06pm

Possibly related?

https://groups.google.com/g/firebase-talk/c/vt8ILxFv-4Y

phillip.diekmann · November 23, 2023, 5:11am

So in the logs I can see that my function is triggered and does all the steps successfully previous to the API call. For the API calls I mostly see APITimeoutErrors and sometimes APIConnectionErrors. This keeps repeating until the cloud function itself times out what happens in 90% of the cases.

So regarding this I don’t think this is related to a CORS issue. However, eventually in the Browser logs I see a CORS error when the function itself times out though which is weird.

Could this be related to using the same API key in 3 different cloud functions shortly after another?

That is the only difference I can think of now that I have the same issue with azure cloud functions.

On Heroku all the API calls where made from a single worker, same running it on my local machine. With google and azure cloud functions I have 3 different functions which can each run on multiple VMs at the same time

antont · November 23, 2023, 9:39am

I haven’t had issues running from google cloud, using Cloud Run and the OpenAI Python lib. Firebase cloud functions use Google Cloud Run also, we’re also using those otherwise, but I ran the OpenAI requests in a different instance via task queues, to have proper async support, but that makes no difference for the env.

redfive · November 23, 2023, 1:18pm

What is your Open AI API rate limit? Possibly 3 concurrent threads (or in quick succession) are exceeding a limit?

https://platform.openai.com/docs/guides/rate-limits?context=tier-free

phillip.diekmann · November 23, 2023, 3:20pm

I switched from using the Python OpenAI async client to using the normal one and now everything works smoothly as expected. I still can’t really explain myself where those issues came from as I immediately awaited the results of the async calls…

Rate limits weren’t an issue

anon10827405 · November 23, 2023, 3:33pm

Are you sure you’re not spamming the endpoint? Can you please post some of your logs with timestamps and also the current python & openai version? Are you uploading the file using docker?

phillip.diekmann · November 23, 2023, 5:52pm

The logs from a google cloud function run where it didn’t work at all today is attached: cloud_run_async_client.log

I was also able to reproduce that behaviour earlier today running it from my local machine. After switching to the Python OpenAI standard client it worked fine from my local machine and now also runs fine using google cloud functions.

I also attached the logs from a run I just made from my local machine where I ran the moderation calls multiple times using the Python OpenAI standard client: local_standard_client.log.

Maybe I am spamming the endpoint. What do you suggest? Adding a delay between each call?

PS: I much appreciate the quick help here and it basically works now. But I’d still be interested where those issues came from as I’ve spent quite some time finding a way that works reliably.

cloud_run_async_client.log:

The request has been terminated because it has reached the maximum request timeout. To change this limit, see https://cloud.google.com/run/docs/configuring/request-timeout
2023-11-23 11:38:11,171 [INFO] Start moderation check.
2023-11-23 11:38:11,172 [INFO] Transcript split into 4 chunks.
2023-11-23 11:38:11,172 [INFO] Launch and append task 0
2023-11-23 11:38:13,173 [INFO] Launch and append task 1
2023-11-23 11:38:15,174 [INFO] Launch and append task 2
2023-11-23 11:38:17,175 [INFO] Launch and append task 3
2023-11-23 11:38:17,175 [INFO] Begin atttempt 1 for rate_limited_moderation
2023-11-23 11:38:17,175 [INFO] RetryCallState: <RetryCallState 68675814148816: attempt #1; slept for 0.0; last result: none yet>
2023-11-23 11:38:17,177 [INFO] Begin atttempt 1 for rate_limited_moderation
2023-11-23 11:38:17,177 [INFO] RetryCallState: <RetryCallState 68675814034512: attempt #1; slept for 0.0; last result: none yet>
2023-11-23 11:38:17,179 [INFO] Begin atttempt 1 for rate_limited_moderation
2023-11-23 11:38:17,179 [INFO] RetryCallState: <RetryCallState 68675814276176: attempt #1; slept for 0.0; last result: none yet>
2023-11-23 11:38:17,180 [INFO] Begin atttempt 1 for rate_limited_moderation
2023-11-23 11:38:17,180 [INFO] RetryCallState: <RetryCallState 68675814033744: attempt #1; slept for 0.0; last result: none yet>
2023-11-23 11:38:17,181 [ERROR] After attempt 1 'rate_limited_moderation'.
2023-11-23 11:38:17,182 [ERROR] RetryCallState: <RetryCallState 68675814148816: attempt #1; slept for 0.0; last result: failed (APIConnectionError Connection error.)>
2023-11-23 11:38:17,182 [ERROR] After attempt 1 'rate_limited_moderation'.
2023-11-23 11:38:17,182 [ERROR] RetryCallState: <RetryCallState 68675814034512: attempt #1; slept for 0.0; last result: failed (APIConnectionError Connection error.)>
2023-11-23 11:38:17,183 [ERROR] After attempt 1 'rate_limited_moderation'.
2023-11-23 11:38:17,183 [ERROR] RetryCallState: <RetryCallState 68675814276176: attempt #1; slept for 0.0; last result: failed (APIConnectionError Connection error.)>
2023-11-23 11:38:17,183 [ERROR] After attempt 1 'rate_limited_moderation'.
2023-11-23 11:38:17,183 [ERROR] RetryCallState: <RetryCallState 68675814033744: attempt #1; slept for 0.0; last result: failed (APIConnectionError Connection error.)>
2023-11-23 11:38:17,354 [INFO] Begin atttempt 2 for rate_limited_moderation
2023-11-23 11:38:17,354 [INFO] RetryCallState: <RetryCallState 68675814034512: attempt #2; slept for 0.17; last result: none yet>
2023-11-23 11:38:17,671 [INFO] Begin atttempt 2 for rate_limited_moderation
2023-11-23 11:38:17,671 [INFO] RetryCallState: <RetryCallState 68675814033744: attempt #2; slept for 0.49; last result: none yet>
2023-11-23 11:38:17,868 [INFO] Begin atttempt 2 for rate_limited_moderation
2023-11-23 11:38:17,869 [INFO] RetryCallState: <RetryCallState 68675814276176: attempt #2; slept for 0.68; last result: none yet>
2023-11-23 11:38:17,925 [INFO] Begin atttempt 2 for rate_limited_moderation
2023-11-23 11:38:17,926 [INFO] RetryCallState: <RetryCallState 68675814148816: attempt #2; slept for 0.74; last result: none yet>
2023-11-23 11:38:47,358 [ERROR] After attempt 2 'rate_limited_moderation'.
2023-11-23 11:38:47,359 [ERROR] RetryCallState: <RetryCallState 68675814034512: attempt #2; slept for 0.17; last result: failed (APITimeoutError Request timed out.)>
2023-11-23 11:38:47,674 [ERROR] After attempt 2 'rate_limited_moderation'.
2023-11-23 11:38:47,674 [ERROR] RetryCallState: <RetryCallState 68675814033744: attempt #2; slept for 0.49; last result: failed (APITimeoutError Request timed out.)>
2023-11-23 11:38:47,872 [ERROR] After attempt 2 'rate_limited_moderation'.
2023-11-23 11:38:47,873 [ERROR] RetryCallState: <RetryCallState 68675814276176: attempt #2; slept for 0.68; last result: failed (APITimeoutError Request timed out.)>
2023-11-23 11:38:47,928 [ERROR] After attempt 2 'rate_limited_moderation'.
2023-11-23 11:38:47,929 [ERROR] RetryCallState: <RetryCallState 68675814148816: attempt #2; slept for 0.74; last result: failed (APITimeoutError Request timed out.)>
2023-11-23 11:38:48,297 [INFO] Begin atttempt 3 for rate_limited_moderation
2023-11-23 11:38:48,297 [INFO] RetryCallState: <RetryCallState 68675814276176: attempt #3; slept for 1.11; last result: none yet>
2023-11-23 11:38:49,001 [INFO] Begin atttempt 3 for rate_limited_moderation
2023-11-23 11:38:49,001 [INFO] RetryCallState: <RetryCallState 68675814034512: attempt #3; slept for 1.81; last result: none yet>
2023-11-23 11:38:49,077 [INFO] Begin atttempt 3 for rate_limited_moderation
2023-11-23 11:38:49,077 [INFO] RetryCallState: <RetryCallState 68675814033744: attempt #3; slept for 1.89; last result: none yet>
2023-11-23 11:38:49,559 [INFO] Begin atttempt 3 for rate_limited_moderation
2023-11-23 11:38:49,559 [INFO] RetryCallState: <RetryCallState 68675814148816: attempt #3; slept for 2.37; last result: none yet>
2023-11-23 11:39:18,300 [ERROR] After attempt 3 'rate_limited_moderation'.
2023-11-23 11:39:18,300 [ERROR] RetryCallState: <RetryCallState 68675814276176: attempt #3; slept for 1.11; last result: failed (APITimeoutError Request timed out.)>
2023-11-23 11:39:19,006 [ERROR] After attempt 3 'rate_limited_moderation'.
2023-11-23 11:39:19,006 [ERROR] RetryCallState: <RetryCallState 68675814034512: attempt #3; slept for 1.81; last result: failed (APITimeoutError Request timed out.)>
2023-11-23 11:39:19,080 [ERROR] After attempt 3 'rate_limited_moderation'.
2023-11-23 11:39:19,080 [ERROR] RetryCallState: <RetryCallState 68675814033744: attempt #3; slept for 1.89; last result: failed (APITimeoutError Request timed out.)>
2023-11-23 11:39:19,561 [ERROR] After attempt 3 'rate_limited_moderation'.
2023-11-23 11:39:19,562 [ERROR] RetryCallState: <RetryCallState 68675814148816: attempt #3; slept for 2.37; last result: failed (APITimeoutError Request timed out.)>
2023-11-23 11:39:19,807 [INFO] Begin atttempt 4 for rate_limited_moderation
2023-11-23 11:39:19,807 [INFO] RetryCallState: <RetryCallState 68675814276176: attempt #4; slept for 2.61; last result: none yet>
2023-11-23 11:39:22,027 [INFO] Begin atttempt 4 for rate_limited_moderation
2023-11-23 11:39:22,027 [INFO] RetryCallState: <RetryCallState 68675814033744: attempt #4; slept for 4.83; last result: none yet>
2023-11-23 11:39:22,432 [INFO] Begin atttempt 4 for rate_limited_moderation
2023-11-23 11:39:22,433 [INFO] RetryCallState: <RetryCallState 68675814148816: attempt #4; slept for 5.24; last result: none yet>
2023-11-23 11:39:22,546 [INFO] Begin atttempt 4 for rate_limited_moderation
2023-11-23 11:39:22,546 [INFO] RetryCallState: <RetryCallState 68675814034512: attempt #4; slept for 5.35; last result: none yet>
2023-11-23 11:39:49,810 [ERROR] After attempt 4 'rate_limited_moderation'.
2023-11-23 11:39:49,810 [ERROR] RetryCallState: <RetryCallState 68675814276176: attempt #4; slept for 2.61; last result: failed (APITimeoutError Request timed out.)>
2023-11-23 11:39:52,030 [ERROR] After attempt 4 'rate_limited_moderation'.
2023-11-23 11:39:52,030 [ERROR] RetryCallState: <RetryCallState 68675814033744: attempt #4; slept for 4.83; last result: failed (APITimeoutError Request timed out.)>
2023-11-23 11:39:52,436 [ERROR] After attempt 4 'rate_limited_moderation'.
2023-11-23 11:39:52,436 [ERROR] RetryCallState: <RetryCallState 68675814148816: attempt #4; slept for 5.24; last result: failed (APITimeoutError Request timed out.)>
2023-11-23 11:39:52,549 [ERROR] After attempt 4 'rate_limited_moderation'.
2023-11-23 11:39:52,549 [ERROR] RetryCallState: <RetryCallState 68675814034512: attempt #4; slept for 5.35; last result: failed (APITimeoutError Request timed out.)>
2023-11-23 11:39:53,166 [INFO] Begin atttempt 5 for rate_limited_moderation
2023-11-23 11:39:53,166 [INFO] RetryCallState: <RetryCallState 68675814033744: attempt #5; slept for 5.97; last result: none yet>
2023-11-23 11:39:55,021 [INFO] Begin atttempt 5 for rate_limited_moderation
2023-11-23 11:39:55,021 [INFO] RetryCallState: <RetryCallState 68675814034512: attempt #5; slept for 7.82; last result: none yet>
2023-11-23 11:39:56,530 [INFO] Begin atttempt 5 for rate_limited_moderation
2023-11-23 11:39:56,530 [INFO] RetryCallState: <RetryCallState 68675814276176: attempt #5; slept for 9.33; last result: none yet>
2023-11-23 11:39:57,911 [INFO] Begin atttempt 5 for rate_limited_moderation
2023-11-23 11:39:57,912 [INFO] RetryCallState: <RetryCallState 68675814148816: attempt #5; slept for 10.71; last result: none yet>
2023-11-23 11:40:23,169 [ERROR] After attempt 5 'rate_limited_moderation'.
2023-11-23 11:40:23,169 [ERROR] RetryCallState: <RetryCallState 68675814033744: attempt #5; slept for 5.97; last result: failed (APITimeoutError Request timed out.)>
2023-11-23 11:40:24,513 [INFO] Begin atttempt 6 for rate_limited_moderation
2023-11-23 11:40:24,513 [INFO] RetryCallState: <RetryCallState 68675814033744: attempt #6; slept for 7.31; last result: none yet>
2023-11-23 11:40:25,024 [ERROR] After attempt 5 'rate_limited_moderation'.
2023-11-23 11:40:25,024 [ERROR] RetryCallState: <RetryCallState 68675814034512: attempt #5; slept for 7.82; last result: failed (APITimeoutError Request timed out.)>
2023-11-23 11:40:26,533 [ERROR] After attempt 5 'rate_limited_moderation'.
2023-11-23 11:40:26,533 [ERROR] RetryCallState: <RetryCallState 68675814276176: attempt #5; slept for 9.33; last result: failed (APITimeoutError Request timed out.)>
2023-11-23 11:40:26,786 [INFO] Begin atttempt 6 for rate_limited_moderation
2023-11-23 11:40:26,786 [INFO] RetryCallState: <RetryCallState 68675814276176: attempt #6; slept for 9.58; last result: none yet>
2023-11-23 11:40:27,915 [ERROR] After attempt 5 'rate_limited_moderation'.
2023-11-23 11:40:27,915 [ERROR] RetryCallState: <RetryCallState 68675814148816: attempt #5; slept for 10.71; last result: failed (APITimeoutError Request timed out.)>
2023-11-23 11:40:33,679 [INFO] Begin atttempt 6 for rate_limited_moderation
2023-11-23 11:40:33,680 [INFO] RetryCallState: <RetryCallState 68675814034512: attempt #6; slept for 16.47; last result: none yet>
2023-11-23 11:40:36,802 [INFO] Begin atttempt 6 for rate_limited_moderation
2023-11-23 11:40:36,802 [INFO] RetryCallState: <RetryCallState 68675814148816: attempt #6; slept for 19.6; last result: none yet>
2023-11-23 11:40:54,517 [ERROR] After attempt 6 'rate_limited_moderation'.
2023-11-23 11:40:54,517 [ERROR] RetryCallState: <RetryCallState 68675814033744: attempt #6; slept for 7.31; last result: failed (APITimeoutError Request timed out.)>
2023-11-23 11:40:56,789 [ERROR] After attempt 6 'rate_limited_moderation'.
2023-11-23 11:40:56,789 [ERROR] RetryCallState: <RetryCallState 68675814276176: attempt #6; slept for 9.58; last result: failed (APITimeoutError Request timed out.)>
2023-11-23 11:41:03,681 [ERROR] After attempt 6 'rate_limited_moderation'.
2023-11-23 11:41:03,682 [ERROR] RetryCallState: <RetryCallState 68675814034512: attempt #6; slept for 16.47; last result: failed (APITimeoutError Request timed out.)>
2023-11-23 11:41:05,024 [INFO] Begin atttempt 7 for rate_limited_moderation
2023-11-23 11:41:05,024 [INFO] RetryCallState: <RetryCallState 68675814034512: attempt #7; slept for 17.81; last result: none yet>
2023-11-23 11:41:06,805 [ERROR] After attempt 6 'rate_limited_moderation'.
2023-11-23 11:41:06,805 [ERROR] RetryCallState: <RetryCallState 68675814148816: attempt #6; slept for 19.6; last result: failed (APITimeoutError Request timed out.)>
2023-11-23 11:41:07,775 [INFO] Begin atttempt 7 for rate_limited_moderation
2023-11-23 11:41:07,775 [INFO] RetryCallState: <RetryCallState 68675814276176: attempt #7; slept for 20.57; last result: none yet>
2023-11-23 11:41:12,971 [INFO] Begin atttempt 7 for rate_limited_moderation
2023-11-23 11:41:12,971 [INFO] RetryCallState: <RetryCallState 68675814033744: attempt #7; slept for 25.76; last result: none yet>
2023-11-23 11:41:33,781 [INFO] Begin atttempt 7 for rate_limited_moderation
2023-11-23 11:41:33,781 [INFO] RetryCallState: <RetryCallState 68675814148816: attempt #7; slept for 46.58; last result: none yet>
2023-11-23 11:41:35,026 [ERROR] After attempt 7 'rate_limited_moderation'.
2023-11-23 11:41:35,026 [ERROR] RetryCallState: <RetryCallState 68675814034512: attempt #7; slept for 17.81; last result: failed (APITimeoutError Request timed out.)>
2023-11-23 11:41:37,777 [ERROR] After attempt 7 'rate_limited_moderation'.
2023-11-23 11:41:37,777 [ERROR] RetryCallState: <RetryCallState 68675814276176: attempt #7; slept for 20.57; last result: failed (APITimeoutError Request timed out.)>
2023-11-23 11:41:42,974 [ERROR] After attempt 7 'rate_limited_moderation'.
2023-11-23 11:41:42,974 [ERROR] RetryCallState: <RetryCallState 68675814033744: attempt #7; slept for 25.76; last result: failed (APITimeoutError Request timed out.)>
2023-11-23 11:41:56,711 [INFO] Begin atttempt 8 for rate_limited_moderation
2023-11-23 11:41:56,711 [INFO] RetryCallState: <RetryCallState 68675814033744: attempt #8; slept for 39.5; last result: none yet>
2023-11-23 11:41:57,334 [INFO] Begin atttempt 8 for rate_limited_moderation
2023-11-23 11:41:57,334 [INFO] RetryCallState: <RetryCallState 68675814276176: attempt #8; slept for 40.12; last result: none yet>
2023-11-23 11:42:00,992 [INFO] Begin atttempt 8 for rate_limited_moderation
2023-11-23 11:42:00,992 [INFO] RetryCallState: <RetryCallState 68675814034512: attempt #8; slept for 43.78; last result: none yet>
2023-11-23 11:42:03,784 [ERROR] After attempt 7 'rate_limited_moderation'.
2023-11-23 11:42:03,784 [ERROR] RetryCallState: <RetryCallState 68675814148816: attempt #7; slept for 46.58; last result: failed (APITimeoutError Request timed out.)>
2023-11-23 11:42:26,716 [ERROR] After attempt 8 'rate_limited_moderation'.
2023-11-23 11:42:26,716 [ERROR] RetryCallState: <RetryCallState 68675814033744: attempt #8; slept for 39.5; last result: failed (APITimeoutError Request timed out.)>
2023-11-23 11:42:27,337 [ERROR] After attempt 8 'rate_limited_moderation'.
2023-11-23 11:42:27,337 [ERROR] RetryCallState: <RetryCallState 68675814276176: attempt #8; slept for 40.12; last result: failed (APITimeoutError Request timed out.)>
2023-11-23 11:42:28,492 [INFO] Begin atttempt 8 for rate_limited_moderation
2023-11-23 11:42:28,492 [INFO] RetryCallState: <RetryCallState 68675814148816: attempt #8; slept for 71.28; last result: none yet>
2023-11-23 11:42:30,995 [ERROR] After attempt 8 'rate_limited_moderation'.
2023-11-23 11:42:30,995 [ERROR] RetryCallState: <RetryCallState 68675814034512: attempt #8; slept for 43.78; last result: failed (APITimeoutError Request timed out.)>
2023-11-23 11:42:44,354 [INFO] Begin atttempt 9 for rate_limited_moderation
2023-11-23 11:42:44,354 [INFO] RetryCallState: <RetryCallState 68675814033744: attempt #9; slept for 57.14; last result: none yet>
2023-11-23 11:42:46,714 [INFO] Begin atttempt 9 for rate_limited_moderation
2023-11-23 11:42:46,714 [INFO] RetryCallState: <RetryCallState 68675814276176: attempt #9; slept for 59.5; last result: none yet>
2023-11-23 11:42:58,495 [ERROR] After attempt 8 'rate_limited_moderation'.
2023-11-23 11:42:58,495 [ERROR] RetryCallState: <RetryCallState 68675814148816: attempt #8; slept for 71.28; last result: failed (APITimeoutError Request timed out.)>
2023-11-23 11:43:00,523 [INFO] Begin atttempt 9 for rate_limited_moderation
2023-11-23 11:43:00,524 [INFO] RetryCallState: <RetryCallState 68675814034512: attempt #9; slept for 73.3; last result: none yet>
2023-11-23 11:43:07,831 [INFO] Begin atttempt 9 for rate_limited_moderation
2023-11-23 11:43:07,832 [INFO] RetryCallState: <RetryCallState 68675814148816: attempt #9; slept for 80.62; last result: none yet>
2023-11-23 11:43:14,359 [ERROR] After attempt 9 'rate_limited_moderation'.
2023-11-23 11:43:14,359 [ERROR] RetryCallState: <RetryCallState 68675814033744: attempt #9; slept for 57.14; last result: failed (APITimeoutError Request timed out.)>
2023-11-23 11:43:16,717 [ERROR] After attempt 9 'rate_limited_moderation'.
2023-11-23 11:43:16,717 [ERROR] RetryCallState: <RetryCallState 68675814276176: attempt #9; slept for 59.5; last result: failed (APITimeoutError Request timed out.)>
2023-11-23 11:43:30,526 [ERROR] After attempt 9 'rate_limited_moderation'.
2023-11-23 11:43:30,527 [ERROR] RetryCallState: <RetryCallState 68675814034512: attempt #9; slept for 73.3; last result: failed (APITimeoutError Request timed out.)>
2023-11-23 11:43:32,787 [INFO] Begin atttempt 10 for rate_limited_moderation
2023-11-23 11:43:32,787 [INFO] RetryCallState: <RetryCallState 68675814033744: attempt #10; slept for 75.56; last result: none yet>
2023-11-23 11:43:37,650 [INFO] Begin atttempt 10 for rate_limited_moderation
2023-11-23 11:43:37,650 [INFO] RetryCallState: <RetryCallState 68675814034512: attempt #10; slept for 80.43; last result: none yet>
2023-11-23 11:43:37,835 [ERROR] After attempt 9 'rate_limited_moderation'.
2023-11-23 11:43:37,836 [ERROR] RetryCallState: <RetryCallState 68675814148816: attempt #9; slept for 80.62; last result: failed (APITimeoutError Request timed out.)>
2023-11-23 11:43:44,973 [INFO] Begin atttempt 10 for rate_limited_moderation
2023-11-23 11:43:44,973 [INFO] RetryCallState: <RetryCallState 68675814148816: attempt #10; slept for 87.75; last result: none yet>
2023-11-23 11:43:46,593 [INFO] Begin atttempt 10 for rate_limited_moderation
2023-11-23 11:43:46,593 [INFO] RetryCallState: <RetryCallState 68675814276176: attempt #10; slept for 89.37; last result: none yet>
2023-11-23 11:44:02,790 [ERROR] After attempt 10 'rate_limited_moderation'.
2023-11-23 11:44:02,790 [ERROR] RetryCallState: <RetryCallState 68675814033744: attempt #10; slept for 75.56; last result: failed (APITimeoutError Request timed out.)>
2023-11-23 11:44:07,655 [ERROR] After attempt 10 'rate_limited_moderation'.
2023-11-23 11:44:07,655 [ERROR] RetryCallState: <RetryCallState 68675814034512: attempt #10; slept for 80.43; last result: failed (APITimeoutError Request timed out.)>
2023-11-23 11:44:14,976 [ERROR] After attempt 10 'rate_limited_moderation'.
2023-11-23 11:44:14,976 [ERROR] RetryCallState: <RetryCallState 68675814148816: attempt #10; slept for 87.75; last result: failed (APITimeoutError Request timed out.)>
2023-11-23 11:44:16,596 [ERROR] After attempt 10 'rate_limited_moderation'.
2023-11-23 11:44:16,596 [ERROR] RetryCallState: <RetryCallState 68675814276176: attempt #10; slept for 89.37; last result: failed (APITimeoutError Request timed out.)>
2023-11-23 11:44:21,436 [INFO] Begin atttempt 11 for rate_limited_moderation
2023-11-23 11:44:21,436 [INFO] RetryCallState: <RetryCallState 68675814276176: attempt #11; slept for 94.21; last result: none yet>
2023-11-23 11:44:31,326 [INFO] Begin atttempt 11 for rate_limited_moderation
2023-11-23 11:44:31,326 [INFO] RetryCallState: <RetryCallState 68675814033744: attempt #11; slept for 104.1; last result: none yet>
2023-11-23 11:44:31,389 [INFO] Begin atttempt 11 for rate_limited_moderation
2023-11-23 11:44:31,389 [INFO] RetryCallState: <RetryCallState 68675814148816: attempt #11; slept for 104.16; last result: none yet>
2023-11-23 11:44:36,538 [INFO] Begin atttempt 11 for rate_limited_moderation
2023-11-23 11:44:36,538 [INFO] RetryCallState: <RetryCallState 68675814034512: attempt #11; slept for 109.31; last result: none yet>
2023-11-23 11:44:51,440 [ERROR] After attempt 11 'rate_limited_moderation'.
2023-11-23 11:44:51,440 [ERROR] RetryCallState: <RetryCallState 68675814276176: attempt #11; slept for 94.21; last result: failed (APITimeoutError Request timed out.)>
2023-11-23 11:45:01,329 [ERROR] After attempt 11 'rate_limited_moderation'.
2023-11-23 11:45:01,330 [ERROR] RetryCallState: <RetryCallState 68675814033744: attempt #11; slept for 104.1; last result: failed (APITimeoutError Request timed out.)>
2023-11-23 11:45:01,393 [ERROR] After attempt 11 'rate_limited_moderation'.
2023-11-23 11:45:01,393 [ERROR] RetryCallState: <RetryCallState 68675814148816: attempt #11; slept for 104.16; last result: failed (APITimeoutError Request timed out.)>
2023-11-23 11:45:06,541 [ERROR] After attempt 11 'rate_limited_moderation'.
2023-11-23 11:45:06,542 [ERROR] RetryCallState: <RetryCallState 68675814034512: attempt #11; slept for 109.31; last result: failed (APITimeoutError Request timed out.)>
2023-11-23 11:45:13,370 [INFO] Begin atttempt 12 for rate_limited_moderation
2023-11-23 11:45:13,370 [INFO] RetryCallState: <RetryCallState 68675814033744: attempt #12; slept for 116.13; last result: none yet>
2023-11-23 11:45:17,463 [INFO] Begin atttempt 12 for rate_limited_moderation
2023-11-23 11:45:17,463 [INFO] RetryCallState: <RetryCallState 68675814276176: attempt #12; slept for 120.23; last result: none yet>
2023-11-23 11:45:28,850 [INFO] Begin atttempt 12 for rate_limited_moderation
2023-11-23 11:45:28,850 [INFO] RetryCallState: <RetryCallState 68675814148816: attempt #12; slept for 131.62; last result: none yet>
2023-11-23 11:45:34,663 [INFO] Begin atttempt 12 for rate_limited_moderation
2023-11-23 11:45:34,663 [INFO] RetryCallState: <RetryCallState 68675814034512: attempt #12; slept for 137.43; last result: none yet>
2023-11-23 11:45:43,372 [ERROR] After attempt 12 'rate_limited_moderation'.
2023-11-23 11:45:43,372 [ERROR] RetryCallState: <RetryCallState 68675814033744: attempt #12; slept for 116.13; last result: failed (APITimeoutError Request timed out.)>
2023-11-23 11:45:47,467 [ERROR] After attempt 12 'rate_limited_moderation'.
2023-11-23 11:45:47,467 [ERROR] RetryCallState: <RetryCallState 68675814276176: attempt #12; slept for 120.23; last result: failed (APITimeoutError Request timed out.)>
2023-11-23 11:45:53,851 [INFO] Begin atttempt 13 for rate_limited_moderation
2023-11-23 11:45:53,851 [INFO] RetryCallState: <RetryCallState 68675814033744: attempt #13; slept for 126.61; last result: none yet>
...

local_standard_client.log:

2023-11-23 18:20:40,598 [INFO] Start moderation check
2023-11-23 18:20:58,270 [INFO] Transcript split into 4 chunks.
2023-11-23 18:20:58,270 [INFO] Launch and append task 0
2023-11-23 18:20:58,270 [INFO] Launch and append task 1
2023-11-23 18:20:58,270 [INFO] Launch and append task 2
2023-11-23 18:20:58,270 [INFO] Launch and append task 3
2023-11-23 18:20:58,271 [INFO] Begin atttempt 1 for rate_limited_moderation
2023-11-23 18:20:58,272 [INFO] RetryCallState: <RetryCallState 140286726304784: attempt #1; slept for 0.0; last result: none yet>
2023-11-23 18:21:06,294 [INFO] HTTP Request: POST https://api.openai.com/v1/moderations "HTTP/1.1 200 OK"
2023-11-23 18:21:06,298 [INFO] Begin atttempt 1 for rate_limited_moderation
2023-11-23 18:21:06,298 [INFO] RetryCallState: <RetryCallState 140287183401616: attempt #1; slept for 0.0; last result: none yet>
2023-11-23 18:21:06,734 [INFO] HTTP Request: POST https://api.openai.com/v1/moderations "HTTP/1.1 200 OK"
2023-11-23 18:21:06,736 [INFO] Begin atttempt 1 for rate_limited_moderation
2023-11-23 18:21:06,736 [INFO] RetryCallState: <RetryCallState 140287183401616: attempt #1; slept for 0.0; last result: none yet>
2023-11-23 18:21:07,211 [INFO] HTTP Request: POST https://api.openai.com/v1/moderations "HTTP/1.1 200 OK"
2023-11-23 18:21:07,213 [INFO] Begin atttempt 1 for rate_limited_moderation
2023-11-23 18:21:07,213 [INFO] RetryCallState: <RetryCallState 140286726370640: attempt #1; slept for 0.0; last result: none yet>
2023-11-23 18:21:07,624 [INFO] HTTP Request: POST https://api.openai.com/v1/moderations "HTTP/1.1 200 OK"
2023-11-23 18:21:07,626 [INFO] Moderation check successful. No abusive content detected.
2023-11-23 18:21:07,627 [INFO] Finished moderation check successfully
2023-11-23 18:21:09,668 [INFO] Start moderation check
2023-11-23 18:21:09,670 [INFO] Transcript split into 4 chunks.
2023-11-23 18:21:09,670 [INFO] Launch and append task 0
2023-11-23 18:21:09,670 [INFO] Launch and append task 1
2023-11-23 18:21:09,670 [INFO] Launch and append task 2
2023-11-23 18:21:09,670 [INFO] Launch and append task 3
2023-11-23 18:21:09,672 [INFO] Begin atttempt 1 for rate_limited_moderation
2023-11-23 18:21:09,672 [INFO] RetryCallState: <RetryCallState 140286725801488: attempt #1; slept for 0.0; last result: none yet>
2023-11-23 18:21:10,182 [INFO] HTTP Request: POST https://api.openai.com/v1/moderations "HTTP/1.1 200 OK"
2023-11-23 18:21:10,184 [INFO] Begin atttempt 1 for rate_limited_moderation
2023-11-23 18:21:10,184 [INFO] RetryCallState: <RetryCallState 140286726016080: attempt #1; slept for 0.0; last result: none yet>
2023-11-23 18:21:10,487 [INFO] HTTP Request: POST https://api.openai.com/v1/moderations "HTTP/1.1 200 OK"
2023-11-23 18:21:10,489 [INFO] Begin atttempt 1 for rate_limited_moderation
2023-11-23 18:21:10,490 [INFO] RetryCallState: <RetryCallState 140286726373136: attempt #1; slept for 0.0; last result: none yet>
2023-11-23 18:21:10,923 [INFO] HTTP Request: POST https://api.openai.com/v1/moderations "HTTP/1.1 200 OK"
2023-11-23 18:21:10,925 [INFO] Begin atttempt 1 for rate_limited_moderation
2023-11-23 18:21:10,925 [INFO] RetryCallState: <RetryCallState 140286725724176: attempt #1; slept for 0.0; last result: none yet>
2023-11-23 18:21:11,216 [INFO] HTTP Request: POST https://api.openai.com/v1/moderations "HTTP/1.1 200 OK"
2023-11-23 18:21:11,218 [INFO] Moderation check successful. No abusive content detected.
2023-11-23 18:21:11,218 [INFO] Finished moderation check successfully
2023-11-23 18:21:12,303 [INFO] Start moderation check
2023-11-23 18:21:12,304 [INFO] Transcript split into 4 chunks.
2023-11-23 18:21:12,305 [INFO] Launch and append task 0
2023-11-23 18:21:12,305 [INFO] Launch and append task 1
2023-11-23 18:21:12,305 [INFO] Launch and append task 2
2023-11-23 18:21:12,305 [INFO] Launch and append task 3
2023-11-23 18:21:12,305 [INFO] Begin atttempt 1 for rate_limited_moderation
2023-11-23 18:21:12,305 [INFO] RetryCallState: <RetryCallState 140286725718928: attempt #1; slept for 0.0; last result: none yet>
2023-11-23 18:21:12,746 [INFO] HTTP Request: POST https://api.openai.com/v1/moderations "HTTP/1.1 200 OK"
2023-11-23 18:21:12,748 [INFO] Begin atttempt 1 for rate_limited_moderation
2023-11-23 18:21:12,748 [INFO] RetryCallState: <RetryCallState 140286725718928: attempt #1; slept for 0.0; last result: none yet>
2023-11-23 18:21:13,150 [INFO] HTTP Request: POST https://api.openai.com/v1/moderations "HTTP/1.1 200 OK"
2023-11-23 18:21:13,152 [INFO] Begin atttempt 1 for rate_limited_moderation
2023-11-23 18:21:13,152 [INFO] RetryCallState: <RetryCallState 140286726007696: attempt #1; slept for 0.0; last result: none yet>
2023-11-23 18:21:13,498 [INFO] HTTP Request: POST https://api.openai.com/v1/moderations "HTTP/1.1 200 OK"
2023-11-23 18:21:13,500 [INFO] Begin atttempt 1 for rate_limited_moderation
2023-11-23 18:21:13,500 [INFO] RetryCallState: <RetryCallState 140286733408976: attempt #1; slept for 0.0; last result: none yet>
2023-11-23 18:21:13,867 [INFO] HTTP Request: POST https://api.openai.com/v1/moderations "HTTP/1.1 200 OK"
2023-11-23 18:21:13,870 [INFO] Moderation check successful. No abusive content detected.
2023-11-23 18:21:13,870 [INFO] Finished moderation check successfully

anon10827405 · November 23, 2023, 6:20pm

I just spent some time looking at the error code and am so far failing to understand wtf is going on. Your APIConnectionError is a catch-all safety net.

One thing though: the moderation endpoint accepts an array. You can send your chunks and get an array back of responses. You don’t need to process them in parallel.

What is the status code being thrown with the initial APIConnectionError?

class APIConnectionError(APIError):
    def __init__(self, *, message: str = "Connection error.", request: httpx.Request) -> None:
        super().__init__(message, request, body=None)

try:
            response = self._client.send(request, auth=self.custom_auth, stream=stream)
            log.debug(
                'HTTP Request: %s %s "%i %s"', request.method, request.url, response.status_code, response.reason_phrase
            )
            response.raise_for_status()
        except httpx.HTTPStatusError as err:  # thrown on 4xx and 5xx status code
            if retries > 0 and self._should_retry(err.response):
                return self._retry_request(
                    options,
                    cast_to,
                    retries,
                    err.response.headers,
                    stream=stream,
                    stream_cls=stream_cls,
                )

            # If the response is streamed then we need to explicitly read the response
            # to completion before attempting to access the response text.
            err.response.read()
            raise self._make_status_error_from_response(err.response) from None
        except httpx.TimeoutException as err:
            if retries > 0:
                return self._retry_request(
                    options,
                    cast_to,
                    retries,
                    stream=stream,
                    stream_cls=stream_cls,
                )
            raise APITimeoutError(request=request) from err
        except Exception as err:
            if retries > 0:
                return self._retry_request(
                    options,
                    cast_to,
                    retries,
                    stream=stream,
                    stream_cls=stream_cls,
                )
            raise APIConnectionError(request=request) from err

My guess is that it’s not liking the fact that you are sending multiple requests at the same time.

phillip.diekmann · November 24, 2023, 6:08am

They are a bit confusing. I’m using tenacity and added a before and after callback for additional logging which makes it hard to follow.

Thanks for the hint regarding the moderation endpoint.

I tried to reproduce the APIConnectionError but I don’t receive it anymore.

Yeah, that was it probably. Using the standard client avoids having that issue. Still weird that it follows up with so many APITimeoutErrors because after the inital API calls the retry delays for each task is different

Topic		Replies	Views
Infinite wait for OpenAI server response with GPT5 on /completions under specific conditions API gpt5	13	1216	January 12, 2026
RateLimitError: The server had an error with no reason given API	58	16423	December 2, 2023
Error: 429 Too Many Requests API	56	15269	December 2, 2023
Continuous gpt3 api 500 error: The server had an error while processing your request. Sorry about that! API	60	29277	December 2, 2023
ChatGPT Suddenly Stops Working on Our Web-based Platform API	9	3652	April 22, 2025

Issues with google cloud functions

Related topics