API latency when backend is hosted on Azure?

[novaphil]

Anecdotally people are saying Azure is faster

GPT-3.5 and GPT-4 API response time measurements - FYI

Hi all, Since API slowness is a consistent issue, I made some experiments to test the response times of GPT-3.5 and GPT-4, comparing both OpenAI and Azure. As a reminder, mostly the response time depends on the number of output tokens generated by the model. Here’s a summary of the results: [gpt_speed_comparison] Or in three numbers: OpenAI gpt-3.5-turbo: 73ms per generated token Azure gpt-3.5-turbo: 34ms per generated token OpenAI gpt-4: 196ms per generated token You can use these valu…