ChatGPT API responses are very slow

ChatGPT API responses are very slow, even for short API calls with 200-400 tokens take 20-30 seconds. Is there any way to make the response faster.

12 Likes

Hi @nandha

Yes, things are slow based on the demand. I just checked for you by sending 300 words of lorem epsom text to the chat completion API for you and got these results:

gpt-3.5-turbo-0301

Total Tokens: 826, Completion API Time: 16.17 seconds
Total Tokens: 866, Completion API Time: 14.434 seconds
Total Tokens: 1313,  Completion API Time: 38.629 seconds

I donā€™t think there is much you can do at the moment as the issue is with the performance of the turbo model(s.) You could switch to another model, which have tested to be faster than turbo these days.

HTH

:slight_smile:

Appendix: Example Completion

3 Likes

hi ruby_coder,
im using api and model gpt 3.5 turbo too. but the response is very slow. im calling api by python.

Yeah, is it slow, for sure now, I tested again for you now, completion time was nearly 22 seconds:

My advice is to relax and do something less frustrating until the issue on the OpenAI infrastructure side improves, if you can.

HTH

:slight_smile:

2 Likes

Yes, may be ā€œturboā€ itĀ“s a little bit ā€œpretenciousā€ adjective for this model :slight_smile:
IĀ“m using curl with PHP on 500 tk max environment and the answers takes arround 30-50 secs to get ready.

Mine, tooā€¦ Iā€™m also having connection errors like this.

openai.error.APIConnectionError: Error communicating with OpenAI: (ā€˜Connection aborted.ā€™, ConnectionResetError(104, ā€˜Connection reset by peerā€™))

Same slowness here, plus occasional 502 Bad Gateway responses after a long wait.

Sadly, the API is throttled for normal paying users. And at the moment we are getting also a lot of errors. Not very usable in the current state and we hope OpenAI will find a solution soon.

Is there a way to avoid this error?
I got a loop that broke today after 5 minutes and I didnā€™t even notice when it did.

The best way to adjust, I think is trying to change your solution to avoiding invoke the API or classify your demands to reduce the times calling it to decrease the total amount of time

I came here looking to see if other people were encountering this. I guess it is reassuring that itā€™s not just me. But also unfortunate because Iā€™m hoping to launch my app in a few weeks and hope this improves.

Was gonna try using an another model but for this feature I need chat API to keep context. Guess Iā€™ll just have to wait it out like everyone else.

Iā€™m using the @backoff.on_exception(backoff.expo, openai.error.RateLimitError) from backoff library. Trying

for i in rlist: 
    try: 
        #mycode
    except TimeoutError:
        print("error")
        continue

but it still breaksā€¦

1 Like

How do you get those reports of your queries? Is there an OpenAI webpage for that? I dont see that fine grained results in OpenAI API

mark. Yesļ¼ŒNow heā€™s incredibly slow, and heā€™s getting slower and slower.I hope the official can improve it as soon as possible

2 Likes

Performance of the OpenAI API is horrible for the moment. Are there plans to improve this soon because this instability in performance is blocking the roll out of our project.

4 Likes

API responses have been consistently 20-50 seconds for about a week now- unusable when ChatGPT itself seems faster than it has ever been

2 Likes

Is there a way to get someone from OpenAI to comment on this? Why are paying customers being rate limited into unusable latencies? The model is supposed to be ā€œturboā€ 30-40 seconds is not very ā€œturboā€ for some 100s of tokens. The API is wayyyyyy slower than the free chat? Why? I doubt itā€™s a technical issue, is that a strategic decision to limit developers? If so, I think OpenAI should be more ā€œopenā€ with the community

3 Likes

I think there are too many people using OpenAI API services. Like, I am a bit shocked that people are now saying ā€˜gpt-3.5-turboā€™ is slow, because I remember ā€˜gpt-3.5-turboā€™ had a good speed, with +1000 tokens. Soā€¦ I feel like the server is packed now.

But my issue is more serious, because my company is using gpt-4, and gpt-4 is way slower though it is accurate. We are about to launch this internally, and I can imagine that our customer service team might say the chat bot is too slow and complain. :sob:

1 Like

totally suffered from the same problem, awesome response, but awful slow :smiling_face_with_tear:

I just signed up for an OpenAI subscription myself, and I expected a similar response time as what ChatGPT is using, but using the gpt-3.5-turbo model, the response times are 30-60 seconds, or timeout completely.

Iā€™m only using it for a demo application, but itā€™s almost unusable due to this performance, and itā€™s extra disappointing I had to pay for this to experience this.

1 Like