Hi, I am working on a web app. The frontend is in react and the backend is in express. My backend is receiving audio files from the frontend and then using whisper to transcribe them. For webm files (which come from chrome browsers), everything works perfectly. However, for mp4 files (which come fro…

If you use a library to transcode from mp4 to some other format/codec (mp3 or wav,) before you forward it to the Whisper model, does it work better?

Reminder: mp4 is a container, not an audio codec. It can contain multiple muxed streams, and obviously video also. It would be similar to saying something can’t understand the contents of your zip file. So one must look at the mp4 being generated: the muxer, interleaving, and ultimately, the audio…

It definitely seems like the Whisper API is unable to understand mp4 file types encoded with mp4a. I even tried transcoding the file into a webm/opus but it always just returns really short transcriptions that are seemingly unrelated.

“encoded with mp4a” is not a audio codec. That’s just Apple making different file names for the same .mp4, because their metadata environment of devices isn’t smart enough to figure out what’s in the file.

[image] jwatte: If you use a library to transcode from mp4 to some other format/codec (mp3 or wav,) before you forward it to the Whisper model, does it work better? [image] kwento: It definitely seems like the Whisper API is unable to understand mp4 file types encoded with mp4a. So, if…

I was using fluent-ffmpeg to transcode to wav, but it didn’t seem to work either. I might not be transcoding the data correctly though. I ended up recording the audio using mic-recorder-to-mp3 on the client side instead and it works perfectly.

https://chat.openai.com/share/0bbc444d-192b-42d1-b432-e6be59ffdf09 Maybe that works?

Are there intentions to make .mp4 work with whisper? Or maybe remove the .mp4 support from the API Reference. Using recorded mp4 audio from iPhone Safari (which is the only format supported as of now) does not give anything other than hallucinations. It works when converted into mp3 manually. Conve…

As has been mentioned before, MP4 is a compression standard, like a zip file. Your mp4 may not be adhering to the standards expected by the decompression side of things, that is not to say that all mp4’s do not work. Does apple do something proprietary or novel with the mp4 standard? some DRM thing …

Why is converting on production a real pain? Create a message, send it into a queue, let a microservice with ffmpeg create the mp3, create a message, send it to a queue and let whisper do it’s thing…

Whisper api completely wrong for mp4

API

jwatte July 29, 2023, 4:26pm 6

So, if you use a library to transcode from mp4 tomp3 or wav, before you forward it to the Whisper model, does it work better?

Topic		Replies	Views
Whisper doesn't work with mp4 API whisper	3	3269	May 29, 2024
Whisper API only transcribing first few seconds API whisper	6	3712	October 28, 2023
Whisper issues with mp4 saved by Safari API whisper	5	2577	December 16, 2023
Whisper API not transcribing audio files coming from an iphone API ios , whisper , javascript	10	3132	December 18, 2024
Whisper API rejecting MP4 from safari - but works with webm on chrome and edge Bugs whisper	2	1113	February 12, 2024

Whisper api completely wrong for mp4

Related topics