Slow response time with GPT-4

[ondrassssek]

Hi, in comparation with GPT 3.5 turbo, GPT-4 is much better, but response times are sometimes crazy from 30-100 seconds. With GPT 3.5 turbo I got 3-20 max ±.
Is there any way how to improve it from my side? I think its OpenAI related.
Thanks a lot

[linus]

Hi @ondrassssek,

I think with the switch in infrastructure to azure and the increase in computing power and the fact that this is a new model this is normal. I have the hope that this will improve further on

Problem with API request (long answer time) - General API discussion - OpenAI API Community Forum

Maybe you can build some more resilience to fall back to another model if response times surpass a for your case acceptable duration. I think what @Jeffers did is a great thing:

Problem with API request (long answer time)

I’m going to start monitoring the 3.5 Turbo response times across the server and when there are too many “curl_setopt($curl, CURLOPT_TIMEOUT, 30);” timeouts, I’ll revert to Davinci 003. or GPT4. It will be interesting to see how I can switch/convert the Turbo chat style resprequests into the completion style.

[ondrassssek]

yeah, hopefully in the future. that is great idea, thanks a lot for link!