How to design few shot prompt with API

API
,
1

Hello everyone,
I am currently struggling with designing few-shot prompt using the chat completion API. I’ve tired searching for information everywhere I could, including the OpenAI developer forum, the OpenAI cookbook, and the OpenAI help center. Unfortunately, it seems like clear instructions or tutorails are scarce.

The only resource somewhat worth referring to is the few-shot prompting example from this tutorial how_to_format_inputs_to_chatgpt_models
Following this example, I constructed my own prompt as below:

{
‘model’: ‘gpt-3.5-turbo-0125’,
‘messages’: [
{
‘role’: ‘system’,
‘name’: ‘example_user’,
‘content’: ‘Google Maps and other highway and street GPS services have replaced what? Answer Choices: A. united states, B. mexico, C. countryside, D. atlas. The answer is D. Because’
},
{’
role’: ‘system’,
‘name’: ‘example_assistant’,
‘content’: ‘the answer must be something that used to do what Google Maps and GPS services do, which is to give directions. Of the above choices, only atlases are used to give directions.’
},
{’
role’: ‘system’,
‘name’: ‘example_user’,
‘content’: ‘The man laid on the soft moss and looked up at the trees, where was the man? Answer Choices: A. niagra falls, B. forest, C. waterfall, D. ground, E. tree. The answer is B. Because’
},
{
‘role’: ‘system’,
‘name’: ‘example_assistant’,
‘content’: ‘the answer must be a place where the man is laying down and looking up at trees. Of the above choices, only forest is a place where people lay down and look at trees.’
},
{
‘role’: ‘user’,
‘content’: ‘If you’re speaking to a lawyer about getting a divorce, what relationship status are you trying to end? Answer Choices: A. being married, B. sisters, C. marriage, D. two people, E. lawyer. The answer is A. Because’
}
],
‘max_tokens’: 300,
‘n’: 4,
‘temperature’: 0.7
}

Brefliy speaking, what I am seeking is for the model to return just an explanatin similar to what I provided in the few-shot examples. The output should follow the template in the answer, specifically continuing with “Because” in the content and providing “some reasonable explanation”.

However, the answer will be like “The correct answer is A. "being married" is the relationship status that someone is trying to end when speaking to a lawyer about getting a divorce.”

Have I missed something? Any feedback would be greatly appreciated.

2 Likes
2

You can achieve few shot prompting by providing examples directly in the system message. There you add the examples you want the model to refer to when answering the user message.

So, you are still sending only one system message and one user message with the question. But the system message contains additional examples. There is no need to add additional system or user messages with replies.

System:

"You answer questions about Ice cream. Answer with ‘Yes’ or ‘No’ only.
When answering user questions follow these examples:
Question: Is ice cream cold?
Answer: Yes.

Question: Does Ice cream make children happy?
Answer: Yes

Question: Can I fix my car’s overheating problem with ice cream?
Answer: No.

And then you have the User message which will be answered according to your examples (few-shots).

Can you write a 500 word paragraph about the advantages of eating ice cream?

Answer:

Yes

3 Likes
3

Thank you so much! It solved my problem perfectly :smile:

3 Likes
4

Can I see what the final prompt looks like? I am also designing a few shot prompt and I need to include a lot of different and diverse examples, I am wondering how many is too many.

5

Sure, I’ve designed the final prompt as follows:

{
‘role’: ‘system’,
‘content’: ‘Reference examples delimited with “”" as a guide to outline the reasoning steps towards the correct answer in a step-by-step manner.
“”"
Q: What do people use to absorb extra ink from a fountain pen?
Answer Choices: A. shirt pocket, B. calligrapher’s hand, C. inkwell, D. desk drawer, E. blotter.
Answer with Reason: The answer is E. Because the answer must be an item that can absorb ink. Of the above choices, only blotters are used to absorb ink.
“”"
“”"
Q: Google Maps and other highway and street GPS services have replaced what?
Answer Choices: A. united states, B. mexico, C. countryside, D. atlas.
Answer with Reason: The answer is D. Because the answer must be something that used to do what Google Maps and GPS services do, which is to give directions. Of the above choices, only atlases are used to give directions.
“”"
“”"
Q: Before getting a divorce, what did the wife feel who was doing all the work?
Answer Choices: A. harder, B. anguish, C. bitterness, D. tears, E. sadness.
Answer with Reason: The answer is C. Because the answer should be the feeling of someone getting divorced who was doing all the work. Of the above choices, the closest feeling is bitterness.
“”"’
}
{
‘role’: ‘user’,
‘content’: ‘Q: The sanctions against the school were a punishing blow, and they seemed to what the efforts the school had made to change?
Answer Choices: A. ignore, B. enforce, C. authoritarian, D. yell at, E. avoid.
Answer with Reason: The answer is A. Because’
}

As you can see, I’ve used 3 examples for few-shot prompting. And it’s possible to use more examples, such as 6 to 8, for better(?) results.

And if anyone has a better suggestion, please feel free to share your ideas. :slight_smile:

2 Likes