Prompting issue inconsistencies

Prompting
1

Hi there

I use gpt3.5 turbo. Temperature=0
I am analyzing transcript of YouTube videos
My prompt basically says that OpenAI should analyze the transcript and see if a given topic (a user input) is mentioned or not. If it’s mentioned, it should return its sentiment (positive, negative or neutral) and why the sentiment was chosen in a few words. If it’s not mentioned it should return not mentioned

Yet after numerous attempts at prompting, I keep getting randomly some output such as:
Topic : Negative - the topic is not mentioned in the transcript.

Instead I would expect :
Topic : Not mentioned
And this is actually what I get in most cases.
But randomly I get the Negative sentiment associated with a not mentioned reasoning

(In the example above topic is in reality replaced by the user input)

I am starting to consider to make another call to OpenAI to simply refine the initial answer so that it cleans up the inconsistencies.

Any thoughts?

Thank you

2

This works for a structured output:

system

You are Extracto, a YouTube transcript analyzer. User provides their topic to search for, and a transcript from a video, and you choose and produce this binary json response, deciding if the topic is found or not, and if so, providing a sentiment analysis of transcript:

{“topic”: string,
“topic_found”: boolean,
“sentiment”: string,
“sentiment_reasoning”: string,
}

sentiment options:
positive, neutral, negative, n/a

sentiment is only measured if the topic is found, otherwise n/a

User:

Topic: Trump criminal trial

---

Transcript:

We have a voicemail number, you know, despite all of my criticisms of Joe Rogan, here’s someone who calls in and says, I’m not willing to criticize Joe Rogan.

I did like Rogan. I know, you know, you hesitate to talk badly about him.
…

(user topic + transcript labeled and separated by three hyphens)

Results:

{"topic": "Trump criminal trial",
 "topic_found": false,
 "sentiment": "n/a"
}

However, gpt-3.5-turbo is so crippled, broken, brain-damaged that it says false for the topic “healthcare” with a transcript including *“[0:15]*James Tyree: Hello and welcome to the TSET Better Health Podcast! This is James Tyree, health
communication consultant” I added more prompt (seen above), and for reliability, one has to go to gpt-3.5-turbo-0301 or gpt-4 for a bot that can still think:


{"topic": "Healthcare",
 "topic_found": True,
 "sentiment": "positive",
 "sentiment_reasoning": "The transcript discusses the latest research on the appeal and dangers of menthol, as well as the importance of community outreach and positive role modeling in relation to tobacco use. These are all positive aspects of healthcare and public health efforts."}