OpenAI slow response today

API
1

We noticed API response are slower compared to few days ago. Are you experiencing it also?

We have launched a Chat feature in our web app and will be doing a live demo tomorrow.

2

If you need it to perform, I would suggest running a fine-tune model at 1 epoch with little behavior difference - (except examples responding exactly how you want the AI to respond for demo questions). then keep pinging it with 1 token requests.

GPT-3.5-turbo responses (mine are faster than some report):
[length] 391 words/512 chunks, chunk 1 in in 0.486 seconds
512 tokens in 13.2 seconds. 38.7 tokens/s @ rate: 40.1 tokens/s)

fine-tune 3.5:

[stop] 370 words/483 chunks, chunk 1 in in 1.135 seconds
483 tokens in 6.3 seconds. 76.9 tokens/s @ rate: 93.9 tokens/s)

Let’s try others

gpt-3.5-turbo-16k-0613:
[length] 391 words/512 chunks, chunk 1 in in 0.455 seconds
512 tokens in 21.1 seconds. 24.2 tokens/s @ rate: 24.8 tokens/s)

gpt-3.5-turbo-0301 (didn’t write as long):
[stop] 202 words/250 chunks, chunk 1 in in 0.332 seconds
250 tokens in 11.0 seconds. 22.6 tokens/s @ rate: 23.3 tokens/s)

And:

ensure you have a few retries before giving up:

  File "C:\Program Files\Python38\lib\site-packages\openai\api_requestor.py", line 745, in _interpret_response_line
    raise error.ServiceUnavailableError(
openai.error.ServiceUnavailableError: The server is overloaded or not ready yet.

(an aside: with 0301 the leprechaun in the 2nd chatbot response used for all writes a personal story of his own experience instead of just reciting “a story”.)

3

It was fast until today. I’ll
consider your suggestions to support the API downtime. Thanks!

1 Like