We are using OpenAI’s chat API (langchain Python module) to implement a document Q&A service. While testing the service locally on my developer machine (Mac Book Pro), the API response times are much smaller (varies from 2 - 5 seconds). However, once deployed as a GCP Cloud Run service, we observe s…

[image] GPT-3.5 Turbo API response is slow API Please can you answer the question from the list i posted above, thank you. @Foxalabs has some great suggestions

The delays we observed were due to h/w configuration issues with GCP. Once we upgraded our service to always have a dedicated CPU allocated, the API issues went away.

Thank you so much! This saved my day. Had similar issue because I had setup request based pricing and my service was running on websockets, guess cloud run only allocates resources based on HTTP requests

Significant degradation in API response time when running in GCP Cloud Run

API

varun_p October 31, 2023, 4:05pm 2

@Foxalabs has some great suggestions

Topic		Replies	Views
Seeking Advises on Optimizing openAI API Calls Feedback api	2	686	November 16, 2023
8-12 Seconds Response Delay with OpenAI API Using Node.js and WhatsApp API API api	2	757	January 15, 2025
How can I improve response times from the OpenAI API while generating responses based on our knowledge base? API chatgpt , api	3	25669	November 9, 2023
Discrepancy in Response Speed between GPT-3.5-turbo API and ChatGPT UI API gpt-35-turbo , chatgpt , api	3	3143	May 24, 2023
GPT-4 API to slow when you have to work with a 46 second time out API	11	3075	July 30, 2023

Significant degradation in API response time when running in GCP Cloud Run

Related topics