Since yesterday I am struggling trying to fine tune the prompt to do the following. I would like GPT3 to ask 5 arbitrary questions to a text for which the answer can be either yes or no and if possible to put the correct answer next to the question.
The best engine to do this is I guess davinci or curie from the Instruct series, but I am not able to always get the expected result.
My prompt is the following:
Given a text, ask 5 questions for which the answer can be “yes” or “no”. Put the correct answer next to each question.
Text:
Any advise to optimize the prompt? Should I make multiple requests ? One per question/answer or 1 for the question(s) and one to get the answer(s)?
This problem is a bit too complex for zero-shot. Should work just fine with 1 or 2 examples. I’ve had plenty of success with far more questions and evaluations, like “Explain why the answer is ethical”.
Thanks for the quick reply. You suggest to only ask 1 question at a time. This is what i was thinking but the risk is to get similar questions with the subsequent requests and it will consume more tokens
But obviously I understand that as magic GPT3 seems to be, this is not magical but you see so many awesome prompts that you tend to be convinced that everything is possible.
This can probably be achieved by providing 2 or 3 full examples, as well as having high penalties for repetition. Otherwise it’s a bit too vague for the ai to understand what’s it’s supposed to do
You can ask multiple questions. I’ve had it keep track of up to 8, but sometimes it gets confused. Yes I understand that you want to optimize for token efficiency but you have to balance that against performance and accuracy.
Create 5 questions from the following that can be answered with yes/no, along with the correct answer.
Text:
The first railway was laid in 1854 between Goolwa and Port Elliot to allow for goods to be transferred between paddle steamers on the Murray River and seagoing vessels. The second railway was laid from the harbour at Port Adelaide, to the capital, Adelaide. This line was opened in 1856. Later on, branch lines in the state’s north in the mining towns of Kapunda and Burra were linked through to the Adelaide metropolitan system.
Questions:
Was the first railway between Goolwa and Adelaide? (No)
Was the first railway created to transfer passengers? (No)
Was the first railway laid in 1854? (Yes)
Was the second railway from Port Adelaide to Adelaide? (Yes)
Was Kapunda a mining town? (Yes)
Text:
Gordon James Ramsay (born 8 November 1966) is a British chef, restaurateur, television personality and writer. He was born in Johnstone, Scotland, and raised in Stratford-upon-Avon, England. Ramsay founded his global restaurant group, Gordon Ramsay Restaurants, in 1997. It has been awarded 16 Michelin stars in total and currently holds a total of seven. After rising to fame on the British television miniseries Boiling Point in 1999, Ramsay had become one of the best-known and most influential chefs in the UK by 2004.
Questions:
This is the answer I get from it:
Did Gordon Ramsay come from Johnstone, Scotland? (Yes)
Did Gordon Ramsay grow up in Stratford-upon-Avon, England? (Yes)
Did Gordon Ramsay begin his career as a chef in 1997? (Yes)
Has Gordon Ramsay been awarded 16 Michelin stars? (No)
Was a television miniseries called “Boiling Point” released in 1999? (Yes)
Has Gordon Ramsay been one of the most influential chefs in the UK by 2004? (No)
Did “Boiling Point” premiere in 1999? (Yes)
The answers are wrong, but the questions are good. Most of the time it does 5 questions. Here’s my settings:
thanks @smuzani ! I will definitively follow this path and see how gpt3 will behave. To be honest, I didn’t had the reflex to provide examples with the instructions engine because the examples I’ve seen were so impressive and without examples that I just expected to pass the text and basic instructions.
Sometimes it works without examples. Sometimes you need multiple. In this case, it seems to have some trouble understanding the prompt without a good example.
What makes the task difficult is the combination of a relatively creative task (needs higher TEMP) and a factual task (needs lower TEMP). So for that reason, I would split the steps and have one PROMPT for question generation and one PROMPT for answers.
==== PROMPT ====
“”"
I was asked to write five yes or no questions about this text:
“”"
Gordon James Ramsay (born 8 November 1966) is a British chef, restaurateur, television personality and writer. He was born in Johnstone, Scotland, and raised in Stratford-upon-Avon, England. Ramsay founded his global restaurant group, Gordon Ramsay Restaurants, in 1997. It has been awarded 16 Michelin stars in total and currently holds a total of seven. After rising to fame on the British television miniseries Boiling Point in 1999, Ramsay had become one of the best-known and most influential chefs in the UK by 2004.
“”"
Five yes or no questions about this the text are:
“”"
Let me give a little bit more context about what I am trying to achieve.
I want my google home being able to ask me 5 questions about the page that I am currently visiting in my laptop: “Hey google, please ask me 5 questions about the current page”.
What I did already is to create a small app running in background in my laptop that can capture the current tab of chrome and then summarise the text. I am using Login | MeaningCloud for that. I know that gpt3 can also do that but I don’t want to consume tokens for this purpose.
The text of the current page summarized is expose via an endpoint because the app running in my laptop is also a webserver.
I have also build an app for google home with Google Assistant | Google Developers
I can then capture the request and then send make a call to the server and let the assistant to read the output of the request.
The missing piece is actually to let gpt3 to ask X questions based on the summarized text. Obviously I want the google home to evaluate my answers and give me a score at the end.
The idea is to have something ready to ask you questions for any pages that you are visiting in your laptops with a minimum of efforts. I see an interest for the educations as an example.
