Whisper prompt leads to hallucinations

API
1

I’m using whisper through node. I’m calling the API directly, given that the openai-node package doesn’t have great support for the whisper API ([Whisper] cannot call `createTranscription` function from Node.js due to File API · Issue #77 · openai/openai-node · GitHub). I’m calling it like so:

import fetch, { FormData, File } from 'node-fetch';

const { data } = await supabase.storage
  .from('audio-transcripts')
  .download(`${interviewId}.mp3`);

const formData = new FormData();
const file = new File([data], `${interviewId}.mp3`);
formData.append('file', file);
formData.append('model', 'whisper-1');
formData.append('prompt', prompt);
formData.append('response_format', 'json');
formData.append('temperature', '0.01');
formData.append('language', 'en');

const response = await fetch(
  'https://api.openai.com/v1/audio/transcriptions',
  {
    method: 'POST',
    body: formData,
    headers: {
      Authorization: `Bearer ${process.env.OPENAI_API_KEY}`,
    },
  }
);

I’m running into a lot of issues using the prompt. It is usually a two sentence prompt that has (1) a high level overview of the transcript (it is a call about xyz), and (2) some proper nouns and competitor names that I want to make sure a spelt correctly. This follows the example provided in OpenAI’s docs.

An example prompt (made up for this post: This transcript is about Bayern Munich and various soccer teams. Common competitors in the league include Real Madrid, Barcelona, Manchester City, Liverpool, Paris Saint-Germain, Juventus, Chelsea, Borussia Dortmund, and AC Milan).

80% of the time I use the prompt, however, I get garbage, hallucinated, output. It ends up on a loop, repeating the same thing (eg. a competitor’s name, or a url made up from one of the competitor’s names). Sample output using the prompt above^:

In the past, the league has been a place of competition for the players. The league has been a place of competition for the players. The league has been a place of competition for the players. The league has been a place of competition for the players. The league has been a place of competition for the players. The league has been a place of competition for the players....

  1. Am I calling it incorrectly?
  2. What is happening, and how can I fix it? I’d like to ideally use the prompt to make sure spelling of competitor names is correct.

Note, this sometimes works better when calling the API directly from the terminal with the prompt.