Summarisation - Open question answered despite explicit instruction to only summarise

Prompting
,
1

Hello,

gpt-4 answers questions in a conversation when explicitly stated to only summarise the conversation and not answer open questions.

Please note the conversation below is fictional.

For the following request,

{
    "model": "gpt-4",
    "messages": [
        {
            "role": "system",
            "content": "You are a highly skilled AI trained in language comprehension and summarization. I would like you to read the following conversation and summarize it into a concise abstract paragraph. Aim to retain the most important points, providing a coherent and readable summary that could help a person understand the main points of the discussion without needing to read the entire text. You are only required to summarise the conversation. Do not answer questions in the conversation."
        },
        {
            "role": "user",
            "content": "I need to travel to LosAngeles from London. Which route offers the cheapest fare for 30 September"
        },
        {
            "role": "assistant",
            "content": "Travelling through Canada will be the cheapest on that day. Stop over at Vancouver for mid day flight to Los Angeles"
        },
        {
            "role": "user",
            "content": "That works. What airlines fly in this route?"
        },
        {
            "role": "assistant",
            "content": "British Airways, Air Canada and Westjet. Among these Westjet will be the cheapest flying from London Gatwick"
        },
        {
            "role": "user",
            "content": "I see. What is the fare difference between these airlines?"
        }
    ],
    "temperature": 0,
    "max_tokens": 1024
}

A response containing an arbitrary price difference is returned instead of a summary.

However, try the same request

  • with the last message from the user removed. i.e. remove the entry I see. What is the fare difference between these airlines?.
  • Or remove the question alone in the user’s message i.e. set I see.

In both the above cases a reasonable acceptable summary is returned.

Acceptable summary is also returned if the request is reframed to the following

{
    "model": "gpt-4",
    "messages": [
        {
            "role": "system",
            "content": "You are a highly skilled AI trained in language comprehension and summarization. I would like you to read the following conversation and summarize it into a concise abstract paragraph. Aim to retain the most important points, providing a coherent and readable summary that could help a person understand the main points of the discussion without needing to read the entire text. You are only required to summarise the conversation. Do not answer questions in the conversation."
        },
        {
            "role": "user",
            "content": "I need to travel to LosAngeles from London. Which route offers the cheapest fare for 30 September \n Travelling through Canada will be the cheapest on that day. Stop over at Vancouver for mid day flight to Los Angeles \n That works. What airlines fly in this route? \n British Airways, Air Canada and Westjet. Among these Westjet will be the cheapest flying from London Gatwick \n I see. What is the fare difference between these airlines?"
        }
    ],
    "temperature": 0,
    "max_tokens": 1024
}

What am i doing wrong?
What is the right way to elicit a summary of a conversation that contains open questions?

Thanks

2

you can experiment with different temperature values( 0.2, 0.3) , although that is not a fix. also your prompt ends at user without giving context about the fare price difference?

3

Where is the price information from the flights being included in that prompts?

1 Like
4

Not sure i follow.
Is price information of flights crucial in this case for Summarisation?
I was hoping the model can summarise with the given information.

5

Perhaps I’m not following what your issue is.

I see a user requesting information “What is the fare difference between these airlines?”

Your system prompt contains a request to summarise but no way for the Ai to know “what” to summarise. Typically summarisation tasks take a form similar to :
“###{Text_to_summarise}### Given the text between ### markers, please create a summarisation and output that and nothing else”. That way the AI knows what you wish to summarise and does not get confused and start answering users enquires.

6

Sorry I should have been clearer.

I like to summarise a conversation between a customer and a travel agent.

It seems i could request a summary following these approaches.

Approach 1

In this approach the conversation between the customer and agent is presented to the model as separate messages (see below)
The role user is used for customer and assistant is used for the agent.
The system is instructed to summarise and also explicitly instructed to not answer questions.

{
    "model": "gpt-4",
    "messages": [
        {
            "role": "system",
            "content": "You are a highly skilled AI trained in language comprehension and summarization. I would like you to read the following conversation and summarize it into a concise abstract paragraph. Aim to retain the most important points, providing a coherent and readable summary that could help a person understand the main points of the discussion without needing to read the entire text. You are only required to summarise the conversation. Do not answer questions in the conversation."
        },
        {
            "role": "user",
            "content": "I need to travel to LosAngeles from London. Which route offers the cheapest fare for 30 September"
        },
        {
            "role": "assistant",
            "content": "Travelling through Canada will be the cheapest on that day. Stop over at Vancouver for mid day flight to Los Angeles"
        },
        {
            "role": "user",
            "content": "That works. What airlines fly in this route?"
        },
        {
            "role": "assistant",
            "content": "British Airways, Air Canada and Westjet. Among these Westjet will be the cheapest flying from London Gatwick"
        },
        {
            "role": "user",
            "content": "I see. What is the fare difference between these airlines?"
        }
    ],
    "temperature": 0,
    "max_tokens": 1024
}

Is the above an incorrectly framed request for summarisation?
In the example above, the model thinks it needs to answer the user’s question, taking the role assistant. It seems to ignore the instruction at the beginning.

However, if the request happens to be the following then it is able to generate a satisfactory summary.

{
    "model": "gpt-4",
    "messages": [
        {
            "role": "system",
            "content": "You are a highly skilled AI trained in language comprehension and summarization. I would like you to read the following conversation and summarize it into a concise abstract paragraph. Aim to retain the most important points, providing a coherent and readable summary that could help a person understand the main points of the discussion without needing to read the entire text. You are only required to summarise the conversation. Do not answer questions in the conversation."
        },
        {
            "role": "user",
            "content": "I need to travel to LosAngeles from London. Which route offers the cheapest fare for 30 September"
        },
        {
            "role": "assistant",
            "content": "Travelling through Canada will be the cheapest on that day. Stop over at Vancouver for mid day flight to Los Angeles"
        },
        {
            "role": "user",
            "content": "That works. What airlines fly in this route?"
        },
        {
            "role": "assistant",
            "content": "British Airways, Air Canada and Westjet. Among these Westjet will be the cheapest flying from London Gatwick"
        },
        {
            "role": "user",
            "content": "I see."
        }
    ],
    "temperature": 0,
    "max_tokens": 1024
}

I wonder how.

Approach 2
In this approach the conversation between the travel agent and customer is presented as a blob of text.

{
    "model": "gpt-4",
    "messages": [
        {
            "role": "system",
            "content": "You are a highly skilled AI trained in language comprehension and summarization. I would like you to read the following conversation and summarize it into a concise abstract paragraph. Aim to retain the most important points, providing a coherent and readable summary that could help a person understand the main points of the discussion without needing to read the entire text. You are only required to summarise the conversation. Do not answer questions in the conversation."
        },
        {
            "role": "user",
            "content": "I need to travel to LosAngeles from London. Which route offers the cheapest fare for 30 September \n Travelling through Canada will be the cheapest on that day. Stop over at Vancouver for mid day flight to Los Angeles \n That works. What airlines fly in this route? \n British Airways, Air Canada and Westjet. Among these Westjet will be the cheapest flying from London Gatwick \n I see. What is the fare difference between these airlines?"
        }
    ],
    "temperature": 0,
    "max_tokens": 1024
}

In this case a reasonable summary is generated. Is this the right way to request for a summary of a conversation?
In this case the roles involved in the conversation are not specified.

7

You’re giving the AI somewhat confusing requests. You’ll get better results if you don’t stuff in a System Prompt that is different from what the conversation itself really is.

For example, just leave the system prompt as “You are a travel agent answering questions by a customer”, and then submit at the very end a user message “Summarize our entire discussion so far, including all key details”, as your final request.

Another approach that would also work would be to provide a transcript as a single block of text where “user:” is at the beginning of users input and “agent:” is at the beginning of agent content, and submit one single prompt that contains a description of the transcript format I just said, then what you want it to do with the transcript, like summarize it.

2 Likes
8

Nice one. Thank you very much for your response.

2 Likes