Whisper API cannot read files correctly

drewlesueur · 2023-03-24T11:13:26.403Z

I’m seeing same issue. Hoping it’s fixed on api side. Thank you.

WhyTho · 2023-03-24T12:37:54.023Z

I did some more controlled testing with ffmpeg versions and I just wanted to confirm that older versions cannot handle the m4a created by the web audio api.
Of course, it could be something different altogether on openai’s end but if you’re trying to capture audio from the browser, this problem will likely keep coming up.

DavidTompkins · 2023-03-24T15:00:41.361Z

Is anyone else experiencing the issue on firefox? I have recordings working on chrome, but not safari or firefox - unclear if that is one issue or two.

RePattern · 2023-03-24T15:04:11.413Z

Yes, experiencing issues on Firefox with webm as mime type - going to iterate through others…

WhyTho · 2023-03-24T18:03:48.932Z

I got another note on this from the server team. Apparently the Ubuntu LTS release (that 1/3 of the internet runs on) comes packaged with the older version of ffmpeg that doesn’t work with that codec. So, don’t be surprised when you struggle to record audio from iOS for the next year.

MIME types1658×598 55.8 KB

Here’s a good resource for working with the client end of the problem but you will still struggle sending the audio to openai:

stackoverflow.com

MediaRecorder iOS 14.6: mimeType not supported

ios, audio-streaming, web-mediarecorder

asked by Felix Dolderer on 03:44PM - 07 Jun 21 UTC

Phoenix · 2023-03-26T12:00:00.771Z

I did modify the suffix and openai api accepted the input

following guide helped changing the suffix:

stackoverflow.com

Renaming file extension using pathlib (python 3)

python, windows, pathlib

asked by user3398600 on 07:10PM - 11 Jan 19 UTC

def transcribe(audio):
print(audio)

myfile=Path(audio)
myfile=myfile.rename(myfile.with_suffix('.wav'))

audio_file= open(myfile, "rb")

transcript = openai.Audio.transcribe("whisper-1", audio_file)
print(transcript)

doda · 2023-03-27T00:37:29.175Z

After an absurd amount of trial and error I’ve found GitHub - kbumsik/opus-media-recorder: MediaRecorder polyfill for Opus recording using WebAssembly which can record webm audio entirely client side and send it to openAI.

itknowles · 2023-03-27T14:41:25.010Z

Having a similar issue with Safari on Mac 12.6.3. Audio from Chrome can be submitted without issue, as long as it is saved first. If I transmit the the blob directly via my Flask app, I get the Invalid file format regardless of whether I use Chrome or Safari. Taking my app to Windows to see if the issue persists.

vojto · 2023-04-04T08:00:51.464Z

Can we get an update here from OpenAI?

jordi.planas.ext · 2023-04-04T16:24:24.351Z

Same here,
I want to test Wishper from a databricks notebook, but doesn’t recognize the audio files.
Hopefully we get a fix because I spent quite some time testing different things none of which worked.

AIndieDeveloper · 2023-04-05T15:36:24.916Z

This API is the only thing that works for me. I have tried opus-media-recorder as well but couldn’t get it to work.

GitHub - ai/audio-recorder-polyfill: MediaRecorder polyfill to record audio in Edge and Safari just worked

skycope · 2023-04-06T11:17:13.021Z

this is working for me with chrome on mac. the audio is generated from a gradio app microphone:

def transcribe(audio):
    """Transcribe an audio file using OpenAI's Whisper API"""
    # Convert the audio file to MP3 format
    input_audio = audio
    with tempfile.NamedTemporaryFile(suffix=".mp3", delete=False) as temp_mp3:
        output_audio = temp_mp3.name

    # Use ffmpeg to convert the webm file to mp3
    command = f"ffmpeg -y -i {input_audio} -vn -ar 16000 -ac 1 {output_audio}"
    subprocess.call(command, shell=True)
    # Transcribe the audio
    audio_file = open(output_audio, "rb")
    transcription = openai.Audio.transcribe("whisper-1", audio_file).text
    
    # Close and remove the temporary MP3 file
    audio_file.close()
    os.remove(output_audio)
    os.remove(input_audio)

    return transcription

lz.groupchat · 2023-04-06T11:32:05.543Z

Tried to change the value of the ‘major brand’ bytes in beginning of ‘ftyp’ box, from [105,115,111,54] to [105,115,111,109] as [105,115,111,109] is also listed in the compatible brand. This is pass the Whisper type check.
As a result, now whisper ai sometimes can return good result, sometimes it return empty string.

aisurfer · 2023-04-08T10:44:48.142Z

Same for me, and all this time I thought it was the safari having the trouble. Using pydub to convert the mp4 recorded by safari to .wav works for me

FP · 2023-04-22T16:56:27.260Z

Same problem here, audio generated by Chrome via HTML5 is accepted by OpenAI Whisper, audio generated by Safari not.

Metadata of Safari generated file vs Chrome generated via ffmpeg:

Safari:

[fferr] Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '6444093fc6f2c.wav':
[fferr]   Metadata:
[fferr]     major_brand     : iso5
[fferr]     minor_version   : 1
[fferr]     compatible_brands: isomiso5hlsf
[fferr]     creation_time   : 2023-04-22T16:20:10.000000Z
[fferr]   Duration: 00:00:05.25, start: 0.000000, bitrate: 187 kb/s 0.9980952380952381
fferr]     Stream #0:0(und): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, mono, fltp, 184 kb/s (default)
fferr]     Metadata:
[fferr]       creation_time   : 2023-04-22T16:20:10.000000Z
[fferr]       handler_name    : Core Media Audio
[fferr] Stream mapping:
[fferr]   Stream #0:0 -> #0:0 (aac (native) -> aac (native))

Chrome:

[fferr] Input #0, matroska,webm, from '6443fbc4d28f7.wav':
[fferr]   Metadata:
[fferr]     encoder         : Chrome
[fferr]   Duration: N/A, start: 0.000000, bitrate: N/A
[fferr]     Stream #0:0(eng): Audio: opus, 48000 Hz, mono, fltp (default)
[fferr] Stream mapping:
[fferr]   Stream #0:0 -> #0:0 (opus (native) -> aac (native))

Any insights on solutoin?

FP · 2023-04-22T17:00:53.630Z

skycope:

this is working for me with chrome on mac.

Thank you, this solution appears sensible. Still working well for you?

WhyTho:

Apparently the Ubuntu LTS release (that 1/3 of the internet runs on) comes packaged with the older version of ffmpeg that doesn’t work with that codec.

Meaning, OpenAI’s servers on running Unbuntu LTS with the old ffmpeg versions that can’t properly process Safari-generated .wav files?

Experiencing similar issue on Firefox (though haven’t checked type of audio file generated.)

WhyTho · 2023-04-23T04:30:09.374Z

Safari doesn’t actually generate wav files. It uses an mp4 encoding. Either way, openAI’s servers don’t like it.

Your options are mostly to re-encode on the client-side with polyfill or on the server-side with something like pydub. There are also other transcription options out there that don’t have this specific problem.

omarsultan · 2023-04-25T10:54:37.729Z

I have a very strange issue, up to yesterday OpenAI Whisper API was accepting files recorded from the voice memo on iOS without any issues. Today they’re not working at all and I keep getting file format error and that my file format is not supported.

Could this be something they changed from their side, cuz I didn’t change anything in my code

michellep · 2023-04-25T16:35:13.104Z

Hi all! We did indeed make a change yesterday to try to support more files as described in this thread. Please let me know if that has helped with things!

@omarsultan - sorry to hear about this. I’ve tried out some voice memo files on my end and they all seem to be working - can you give any more details so I can repro? Maybe iOS version number, file extension, etc?

omarsultan · 2023-04-25T17:07:11.454Z

Hey Michelle,

It turns out that what was causing the crash was some preprocessing I was doing to the file because it initially didn’t work before. But not it just works out of the box with Audio Files recorded from the Voice Memos App on iOS 16

Thanks a lot