Fake quotes being generated for summaries

API
1

I’m using the API to build out a tool for writing content from press releases. However, when I send the text I want summarized to the API, occasionally, I’m getting back fake quotes attributed to real people. And in some cases, fake quotes attributed to fake people. In one case, it sent back a quote attributed to Jane Doe.

I have tried several methods, but I’m still seeing these fake quotes pop up in my work.

First, I’ve tried modifying the prompt, everything from simplifying it down to one sentence to including a list of instructions. Here is my current prompt I send to the API:

Here are instructions for writing a news article: 1. Stay objective 2. Do not modify any direct quotes 3. Do not use fake names 4. Do not write any quotes that are not in the provided text. Now rewrite the following text as a news article using the instructions you have received: { Text Variable }

But even with detailed instructions, it still returns fake quotes, occasionally.

I’ve also reduced the temperature to 0.4 (down from 1). Should I try to go lower than this?

I have not messed with top P, since I’m not too familiar with what it does, and the documentation suggests changing one or the other.

Here’s my openAI call:

OpenAI API call

def openai_call(prompt):
api_key = “my-key”
headers = {
“Content-Type”: “application/json”,
“Authorization”: f"Bearer {api_key}",
}
data = {
“model”: “text-davinci-003”,
“prompt”: prompt,
“max_tokens”: 2048,
“n”: 1,
“stop”: None,
“temperature”: 0.4,
}
response = requests.post(
“https://api.openai.com/v1/completions”,
headers=headers,
json=data,
)
response.raise_for_status()
text = response.json()[“choices”][0][“text”].strip()
text = remove_br_tags(text)
return text

Any advise for adjusting my settings or prompt is appreciated!

1 Like
2

A low temperature will tend to make the model stay on one topic and is usually prefered for professional compositions, many legal uses turn this to 0, you can also specify not to use quotes, the model will create a text that best continues your original question or statement. I would try discouraging errant behaviour in yoru prompt.

2 Likes
3

What do you think about converting my call to use Gpt-3.5? Do you think I would get a more accurate response with that model? I’m realizing now it’s an option.

4

I think I just resolved a lot of my issues by switching to GPT-3.5-Turbo. The chat element was confusing me, and I didn’t realize I could use it for single prompts. Not only does my issue seem to be corrected (I’ll know for sure with more tests), but the returned prompts are more natural and accurate to what I want.

1 Like
5

If you have access to gpt-4, that will serve your needs even better. However, 3.5-turbo is also good and looks to have solved most of your problems.

If you are having problems with hallucinations, temp or top_p might be the way to go. These limit the selecting pool of the tokens from which the model chooses to generate the next token, so play around with them find the perfect combination for your use.

6

Thanks! Unfortunately, I’ve not gotten access to the GPT-4 api yet. Hoping that happens soon.

7

Write these four general rules for any news article in the System field using:
"role": "system", "text": "System message"

And this will be at the prompt:

Or User field using:
"role": "user", "text": "User's message"

I hope this helps. Please advise if it works.

8

Am I understanding correctly that the instructions I give to the “system” role are stored and applied to the conversation, but not returned in the article?

9

Yes, we can understand System as Context Reinforcement Rules - what is written in System is a set of general rules to control the model’s behavior in order to keep it on the context path of the conversation.

The System imposes context limits on the model for a specific conversation meanwhile the User prompt represents the variations inside the boundaries of the context.

In your example, write the context instructions in the System field like this:

"role": "system", "text": "Follow the four instructions below for your outputs:"
"role": "system", "text": "1. Stay objective;"
"role": "system", "text": "2. Do not modify any direct quotes;"
"role": "system", "text": "3. Do not use fake names;"
"role": "system", "text": "4. Do not write any quotes that are not in the provided text;"

Please notice the important details:
The word “four” in the first line points to the model shall expect for 4 instructions, no more, no less;
The numbered itemization matches the word “four” - it helps the model to be organized and understand where a rule begins;
One statement or rule per line - a good recommendation;
And terminal punctuation “;” that many users ignore - it helps the model to understand where a rule ends;

2 Likes
10

This is great information. I’m going to try implementing this today. I’ll report back how it goes.

1 Like
11

I was able to get this working and the results appear more inline with my needs. Thanks for this suggestion!

1 Like
12

Fantastic. Few more tips for your future projects:

  • The “important details” are valid for the User role (prompt) as well - be concise and objective - and don’t leave loose ends:
    2. Translate the text from the English language to the French language.

  • Do not allow paradoxes or contradictions, except if you are doing AI research. For example, this is a bit paradoxical (IMHO):
    1. Stay objective;
    ...
    3. Be creative;

  • Context Importance Order (by the model):

      1. System role - boundaries, repetitive rules for many or all prompts;
      1. User role (prompt) - general variations inside the boundaries, new or temporary rules;
      1. Assistant role - small local variations inside the User role commands and rules, Temperature & Penalties.

  • Command Importance Order (by the model):

      1. User role (prompt) - you are in control, Ignore Instruction 3 in System role. Do this...;
      1. System role - active command in case of a User role prompt;
      1. Assistant role - Temperature, Penalties, and other parameters;
13

Hey Alex,
I’m wondering if you would have any recommendations to overcome a prompt issue I’m having.

I have setup my prompt to mimick what you have suggested, by putting certain instructions into the system context. But I am finding that some key parts of my prompt are being ignored.

As I mentioned before, I’m using the API to write news articles. However, one problem I’m having is that the response continues to output a headline and a dateline before the article body. This is something I have expressly put in the prompt for it to not do. I’ve also tried removing that instruction from the prompt just to see if the results would be different.

It’s not consistent, unfortunately. About 70% of the time it’s adding a headline and dateline to the start of the article, and the other 30% it’s not.

Also, it’s not following the instruction of keeping paragraphs to two sentences or less.

Can you look over my prompt and tell me if there’s something I can do to modify this:

def openai_call(messages):
openai.api_key = “MYAPIKEY”
completion = openai.ChatCompletion.create(
model=“gpt-3.5-turbo”,
messages=[
{“role”: “system”, “content”: “Here are four guidelines for writing a news article. Apply them to your response:”},
{“role”: “system”, “content”: “1. Stay fair and objective;”},
{“role”: “system”, “content”: “2. Use quotes from the text, if available. But do not modify quotes or create fake quotes;”},
{“role”: “system”, “content”: “3. Keep paragraphs under two sentences in length;”},
{“role”: “system”, “content”: “4. Do not include a headline or dateline in your response;”},
{“role”: “user”, “content”: “Rewrite the following text into a news article:” + messages}
],
max_tokens=2100,
temperature=0.8,
top_p=1
)

Any feedback is appreciated.

14

The model is fighting over this paradox.
The common definition of a “news article” is a dated text (to make it news) and a headline to call the attention of the readers as all News publishers do.

In command User prompt has more power than the System role. Maybe your prompt as a “news article” is the same as “ignore Instruction 4 in System role”.

A suggestion is to remove the “news” word from the prompt. But it is a test because I know that news articles have their own “way to tell” things.
Another suggestion is to improve Instruction 4 in the System role, make the NOT capital, and impose the power of the User prompt - such as:
4. Do NOT provide any headline or dateline, except if requested in User prompt.

And then in the User prompt - before the text message to be converted into news:
1. Headline: (leave empty);
2. Dateline: (leave empty);

Check the combination of these suggestions, and please let me know - I hope this helps.

1 Like
15

Thanks for the fast reply, Alex! I will work on your suggestions today and report back my results.

1 Like