Answering a question based on multiple sources

Prompting
1

Hi Community, long time user and first time poster - please be kind :wink:

I’m attempting to write a prompt which will read multiple answers to a question from different sources then provide a definitive comprehensive answer. I’d like to use this approach so that I can provide GTP3 with human picked reliable answers allowing GTP3 to read knowledge which the model hasn’t been trained on then summarise and provide an answer.

Example of the type of question and knowledge I’m providing:

Question
What colour is the sky?

Source 1
The sky is blue.

Source 2
I see a blue sky during the day.

Source 3
We can see sky as blue because it has the smallest wavelength when scattering light.

Source 4
Depends, at night the sky is black.

Source 5
Usually blue, but at sunset the sky turns warm and glowing shades of gold, pink and orange.

Source 6
I don’t have any evidence but I’m sure I saw the sky turn green and yellow once.

Example of what I’m expecting GTP3 to return based on reading these 6 sources:

Question:
What colour is the sky?

Expected typical answer:
During the day the sky is blue due to being the smallest wavelength when scattering light. When the sun sets the sky turns gold, pink and orange, then at night the sky is black. Some sources believe the sky is green and yellow however this is unreliable.

So far I’ve used a TLDR prompt, the Q&A and 2nd grader templates - usually just getting back “The sky is blue” regardless of the engine used and any combination of temperature and penalties - no mention of other colour information mentioned in the statements. Is this coming from GTP3 training data or reading the sources?

I’ve also tried implementing the answer endpoint by uploading each answer then asking the question - however I’m mainly getting back verbatim statements when I was expecting some re-writing and summarisation to occur - tweaking the temperature starts getting too creative and other random non-related information starts creeping in which is usually completely wrong.

The main benefit of this summarise sources approach is that I can provide citations of where the information came from to combat misinformation, allowing greater trust with the reader. GTP3 is being used in this instance as a summarise system (“GTP3 - please answer this question by reading the following statements and ignore what you already know”), not relying on it’s own knowledge which might be out of date or incorrect.

Any suggestions or feedback on the correct prompt, approach or methodology would be appreciated!

Rob

2 Likes
2

Thanks for your question! Here’s the prompt using davinci-instruct-beta-v3

I just added a simple instruction at the top, then pasted your text.

The response I get seems decent:

The answer to this question depends on the time of day and weather conditions. In general, the sky is usually blue, but at sunset it can turn a range of warm colours.

6 Likes
3

Rob, I am working on something very similar. My text is super technical (legal rules). Check out the Q&A [example] (OpenAI API) in the playground which is extremely impressive in some respects, completely wrong in other respects, and quite funny at the end. For my use case, I’ll need to give strict instructions about citing sources and providing confidence levels.

2 Likes
4

Thank you for your assistance and advice Boris.

I’ve tried that prompt and tested with different colours, GTP3 at first does a great job however I’ve realised that it is bringing in blue, black and sunset colours into the response based on existing training data rather than solely rewriting the sources.

For example some of the sample responses I’m getting back based on the following prompt by providing only silver with pink dots which is unlikely to be contained within training data:

Answer the question based on the context from six different sources below:

Source 1
The sky is silver with pink dots.

Source 2
I see a silver with pink dots sky during the day.

Source 3
We can see sky as silver with pink dots because it has the smallest wavelength when scattering light.

Source 4
Depends, at night the sky is silver with pink dots.

Source 5
At sunset the sky turns silver with pink dots.

Source 6
I don’t have any evidence but I’m sure I saw the sky turn silver with pink dots.

Question:
What colour is the sky?

The answer to this question can depend on the time of day and weather conditions. In general, though, the sky is usually blue during the daytime, but it may turn pink or silver at sunset.

(Blue not mentioned by sources)

The answer to this question depends on the time of day. During the day, the sky is silver with pink dots. However, at night the sky is silver with black dots.

(Black not mentioned by sources)

According to Sources 2 and 3, the sky is silver with pink dots during the day. According to Source 4, the sky is silver with pink dots at night time. And according to Sources 5 and 6, the sky turns silver with pink dots at sunset.

(Technically correct but a very long way of saying the same information.)

The answer to this question can vary depending on the context, as is shown in the six different sources above. In general, however, the sky is usually described as being silver with pink dots.

(Correct answer, is there any way to design the prompt to request a self contained answer without talking about ‘sources above’).

Is there any way of asking GTP3 to ‘ignore what you know’ and simply rewriting only the knowledge which the sources have provided? I’ve tweaked the initial prompt to “Ignoring what you already know answer the question based on the context from six different sources below:” however the response still mentions blue:

The answer to this question can vary depending on the time of day and weather conditions. In general, however, the sky is usually described as being either silver or blue.

I hope this makes sense, I’ve been struggling to craft the prompt without wasting too many credits and I’ve been unable to see any documentation regarding this.

Thanks!

5

The best way to achieve more customization is with fine-tuning davinci on at least 50-100 examples of the behaviour you’re after.

2 Likes
6

Hey @boris, Could the instruct model be used as a base model for fine-tuning, or does the fine-tuning file have to teach the instract-like behavior?

7

At the moment you can only find tune the base models. Do you’d need to reach instruction following behaviour

8

I’ve been looking for a response to this question, but can’t find it. If I am fine-tuning, I understand that it is recommended to use “single-shot” prompting. But, if I have a completion which can provide the answer to multiple questions, what do you think about creating a prompt with multiple possible questions?