How to ensure completions to be different when generating more (n) completions at the same time

Prompting
1

When I tell the model to generate 20 different quiz questions on a specific topic I get them.
But when I tell the model to generate a quiz question on a specific topic with a specific amount of completions (OpenAI API) the questions are repeating very often.
I want to go the second way because it is better to parse the result. Is it possible to tell the model the completions should be different?

2

What model and settings are you using? ie temperature, frequency_penalty, etc.

Do you have an example of your prompt?

3

The prompt is:Generate a quiz question with one correct and 3 incorrect answers each on the topic SOME_TOPIC. The scheme is ‘Question;Correct answer; Wrong answer; Wrong answer; Wrong answer’.

model: text-davinci-003
frequency_penalty: 0
temperature 0.7

4

I would use a one-shot or two-shot prompt… That is, I would include one or two examples of what you want in the prompt. It’s more expensive, but you usually get better results.

This would probably work with a one-shot example. You might even try to see what you get with Davinci model as it’s a lot cheaper. You would need to rephrase the question and supply answers in the example that show what you want GPT-3 to return…

example
example1908×983 104 KB

example2
example21898×973 94.5 KB

Hope this helps!

5

Hi Paul,

with “I would use a one-shot or two-shot prompt” you mean using one prompt instead of “n” completions? The original question was, if there is another way to do this with “n” completions for parsing reasons (n completions were more reliable and easier to parse).

6

I meant include one or two examples in the prompt itself so GPT-3 knows better what you’re after.

See the screenshot. The un-highlighted text is my prompt and the highlighted text is the output… which I thought is what you were after. Apologies if I misunderstood.

7

Ah, okay, sorry… now I understand what you mean. But adding example questions did not improve the results.

8

This is why you need to keep experimenting. There’s no “magic settings” for every use case.

I would play with the prompt and, like you said, different wording for “the scheme is…”

Sometimes, it just takes time to craft a good prompt. I’ve got over 1,000 hours in this now. (Started with GPT-2…)

Good luck!

1 Like
9

So the prompt is working:
Generate 20 questions on the topic xyz
is going great.
BUT
Generate a question on the topic xyz
n times (Rest-API)
generates the repetitions
:slightly_frowning_face:

10

Did you try to change the prompt to include a sample or two like I suggested?

11

Yes, I have included two samples.

12

Can you show the new prompt? And the output?

13

Here some prints from flutter:
flutter: {model: text-davinci-003, prompt: Generate a quiz question with one correct and 3 incorrect answers each on the topic German Wine. The scheme is ‘Question;Correct answer; Wrong answer; Wrong answer; Wrong answer’.What is the capitol of England?;London;Manchester;Glasgow;Newcastle###What is the biggest Fish?;Whale Shark;Blue Tuna; White Shark; Black Marlin###, temperature: 0.7, max_tokens: 2000, frequency_penalty: 2, n: 5}
flutter: [What is the most popular German wine?, Riesling, Merlot, Pinot Noir, Chardonnay]
flutter: [What type of wine is Riesling?, White Wine, Red Wine, Sparkling Wine, Rose]
flutter: [What is the most popular grape used in German wine?, Riesling, Merlot, Pinot Noir, Chardonnay]
flutter: [What is Riesling?, A white wine grape variety, An Italian red wine, A French white wine, An Austrian sparkling wine]
flutter: [What is the primary grape used in German Riesling?, Riesling, Chardonnay, Merlot, Pinot Noir]

and another time:

flutter: {model: text-davinci-003, prompt: Generate a quiz question with one correct and 3 incorrect answers each on the topic German Wine. The scheme is ‘Question;Correct answer; Wrong answer; Wrong answer; Wrong answer’.What is the capitol of England?;London;Manchester;Glasgow;Newcastle###What is the biggest Fish?;Whale Shark;Blue Tuna; White Shark; Black Marlin###, temperature: 0.7, max_tokens: 2000, frequency_penalty: 2, n: 5}
flutter: [What is the main grape used for Riesling?, MĂźller-Thurgau, Cabernet Sauvignon, Chardonnay, Pinot Grigio]
flutter: [What is the most popular type of German wine?, Riesling, Merlot, Pinot Noir, Sauvignon Blanc]
flutter: [What is the main grape variety used in German Wine?, Riesling, Cabernet Sauvignon, Merlot, Chardonnay]
flutter: [What is the main grape used to make Riesling wines?, Riesling, Chardonnay, Pinot Noir, Sauvignon Blanc]
flutter: [What type of wine is produced in Germany?, Riesling, Pinot Noir, Cabernet Sauvignon, Chardonnay]

all about Riesling…