Hi there, I’m trying out a fine-tuned gpt-3.5-turbo model. The inference speed is good in terms of tokens/s (comparable to gpt-3.5-turbo), but sometimes the API calls (including via Playground) will have a 2-10 second delay. I assume this is because the fine-tuned part of the model isn’t in some h…

Fine-tuned gpt-3.5-turbo latency

SJ11 September 7, 2023, 12:36pm 11

I’m also experiencing delays in the fine-tuned model, it’s inconsistent, about a 1/3 of api calls are delayed by 5-12 seconds.

Topic		Replies	Views
High latency for fine-tuned gpt-4o-mini API	4	995	November 26, 2024
What is the expected inference latency of fine-tuned gpt-4 model? API gpt-4 , fine-tuning	3	1887	August 2, 2024
Fine-tuning flagged Moderation Policy customization API fine-tuning , moderation	17	1647	December 16, 2023
Gpt-4-0125-preview is slower than gpt-4-0613? Feedback gpt-4 , api	5	5599	January 30, 2024
What is considered as normal latency? API	3	3045	December 15, 2023