Handling API calls when service is down

I’m pretty sure there isn’t a direct API we can call, but how does everyone account die dates like this when the API stops working and casks don’t come back as expected?

Looking for an automatic solution.

1 Like

Fail-over to Azure or a different LLM?

Well, in the first instance I go for a solid retry policy when I use my APIs. Typically one call goes through eventually.

The bigger challenge is a temporary performance degradation that can affect the quality of output and which requires other types of controls, especially if you are running an automated solution.

Yep.

That is one of the risks associated with building upon someone else’s service.

Look at what happens every time AWS has an outage.

2 Likes

You should try litellm, they have fallback, least latency router bunch of crazy and awesome helpers, load balancers etc, all in open ai format, so you can fallback to alternative openai server like azure, or even other llm providers like groq, bedrock etc.

1 Like