Hi,
I’m trying to make sense of another post on this forum: ”Whisper API costs 10x more than hosting an VM?” (I’m not allowed to link it)
From my tests, inference using both the OpenAI Whisper API and self hosting “insanely fast whisper” on a 4090 is taking roughly 2 minutes for an hour of speech.
Whisper API costs $0,36/h and you can rent a 4090 (spot instance) on runpod for $0,39/h.
I feel like I’m missing something, but to me OpenAI seems to have a very competitive pricing? 
One thing you’ve overlooked is that Whisper on OpenAI generates much faster than the length of the audio. That hour can be done in a minute.
You can also dispatch dozens of transcription tasks at the same time, lighting up a lot of datacenter to get you many hours back in a minute, something your $1000 GPU can’t do.
The price of not transcribing constantly on API is $0.00.
So there are many cost transition points to consider for your personal answer, even the time factor of making open source initially work. Math is fun.
Hi! I think you might have misunderstood what I meant, and I should have been more clear.
From my tests both OpenAI and a 4090 can transcribe 1 hour of audio in about 2 minutes, and they cost about the same ($0,39/h vs $0,36/h).
So my question is more related to the other post, where the author claims that they get a 20x cost benefit from running on a 4090, where to me it seems more like a 1 to 1 relationship 
It’s always a tradeoff
how long does it take you to set up a cloud instance? How long does it take you to set it up on you machine? how long does it take you to just call the API?
I think it’s geneally understood that most cloud services are significantly more expensive than self hosting.
edit: here’s the link: Whisper API costs 10x more than hosting an VM?
To transcribe 1 hour of audio with OpenAI, it would cost you:
$0.36, taking you 2 minutes.
Whereas…
To transcribe 1 hour of audio with self-hosted whisper, it would cost you 2 minute of run-time (1/30th of $0.39h, so basically 1.333 cents, or $0.013), taking you 2 minutes.
That’s a 30x cost difference.
You are incorrect when you state that they’re both 1 to 1.
They both take the same amount of time, but self-hosted is effectively 30x less expensive.
It’s only going to take the same amount of time on similarly speced hardware.
Are you taking into account the purchase cost of the hardware and depreciation?
I understand it’s tempting to say, I already own the GPU for another purpose so it’s effectively “free” and that’s totally fine if you are running your own home assistant setup, but it’s not an apples to apples comparison when it’s for commercial purposes where you have to rent/buy the land to host the hardware on and buy the hardware and pay for the air conditioning and electricity and water and gas and the staff to maintain it and the insurance costs and the costs of upgrades, account for depreciation etc, etc.
One could make the case: Whisper model on API had its pricing set three years ago, while hardware inference has been getting faster and cheaper.
However, the AI datacenter companies have recognized this fault, that many AI features are within reach of consumer high-end hardware. A problem with their profit model is general-purpose computing hardware being available to others.
The solution of course is to monopolize the supply chain, where in six months, we now see RAM at 4x the cost, solid state storage at 2x, and GPU SKUs with higher memory unobtanium unless you are an integrator, and then the same GPU dies going into servers with unbelievable pricing.
Isn’t the main question when it makes sense to rent a GPU in the cloud versus using the API?
For a one-time job, or something that only runs occasionally, a cloud GPU service can be cheaper. For a production app at scale that needs to run 24/7 all year, the OpenAI API is likely the more reliable option and, depending on the actual scale, potentially the cheaper one too.
OpenAI API: when you need 1000 of them now. 
