OpenAI chatcompletion prompt for inferring user intent from chat history

API
,
1

I’m working on an AI assistant that analyzes the chat history between a user and another assistant and infers the real intent behind the user’s query.

I work at a publishing house, and I’m building a chatbot that internally generates SQL sentences using LangChain (a SQL agent). The bot can retrieve information like prices, publication origins, kiosks, publication IDs, themes, etc. Since these agents are stateless (just like a chat completion), I am using another completion layer on top to generate meaningful queries for the bot.

For instance:

If I ask the agent, “What’s the new publication on ‘phyto plant’ from the editorial house A-47854,” it will retrieve:

Response:

A new study on fossils and phytoplankton
ISBN: 1237854-HNU
Country: France
Edition: On Hold
House: Journal World Inc.

Phyplankton in Stale Water
ISBN: HBXX78-HNU
Country: Germany
Edition: Published
House: Environmental Herald

But if I ask the follow-up question: "What other publications do you have from the second house?" the agent won’t be able to answer correctly because it’s memoryless.

I’m trying to build a layer on top that will rewrite and provide context to the query so that my model can give a more accurate response.

Desired Query Rewrite:

For example, for the follow-up question:
"What other publications do you have from the second house?"
The expected reformulation would be:
"What other publications aside from 'Phyplankton in Stale Water' (ISBN HBXX78-HNU) does the house 'Environmental Herald' have?"

Why?

Because “Environmental Herald” is the second house mentioned in the chat history, and this contextual information is crucial.

Similarly, if the user says something like “Thank you” or “That’s all, bot,” these types of queries should not be rewritten.

If a query is unrelated to the previous conversation, it should be returned as-is. For example:
“Tell me about the latest publications on biomechanics from London.”

I’ve tried many prompts and approaches to achieve this behavior, but nothing has worked consistently.

This is my prompt:

*SYSTEM = """
Below is a history of the conversation so far, and a new question asked by the user that needs to be answered by searching in a knowledge base. 
When related to the query, make use of the keywords 'Editorial' and 'House', and include the ISBN code when the question needs rewriting.
Your job is to generate queries, never to answer them.

Follow these steps and rules:

Rules:
1. If the user is greeting you, congratulating you, or saying goodbye, pass that text as it is. For example, "Thank you, bot":
   Your final output must be: "Thank you, bot."
   
2. Analyze the chat history and determine if the current user query is related to the previous conversation. If so, provide the relevant context, data, keywords, and codes to pass it to the SQL agent.

Example:

User: "What's the latest titles in the catalog?"  
Assistant: "1. Biomechanics for College Students 2. Statistics Beyond Inference 3. Movie Nights as a Way to Therapy"  
User: "Give more info on the second one."

The proper output to generate would be:  
"More information on 2. 'Statistics Beyond Inference'."
"""*

And my code:

# System message
system_message = {
    "role": "system",
    "content": SYSTEM
}


user_query = "Tell me more about the second publication"

# Create the full message list: system message + chat history + current user query
messages = [system_message]  + CHAT_HISTORY + [
    {
        "role": "user",
        "content": user_query,
    }
]


contex_question = model_client.chat.completions.create(
    model=deployment,
    messages=messages,
    temperature=0.0
)

key_question = contex_question.choices[0].message.content

print(f"Inference prompt: {messages}\n")
print(f"Generated response: {key_question}")

When I print {messages}, I see the prompt is well-structured, and my last question is added correctly, but the responses are often not what I expect.

For instance, if this is my chat history:

User: “Do you have any publications from Norway on biomechanics?”
Assistant: “Yes, we do. 1. Glasses Metrics on Biomechanics ISBN 4785-7854, and 2. Overall Fracture Metal and Bio Solutions ISBN 8575-8YUA.”

Then I ask:
User: “What’s the publication date for the first one?”

The response I get is:
“The publication date for the first biomechanics publication in Norway.”

However, the expected output I want is:
"Publication date for Glasses Metrics on Biomechanics ISBN 4785-7854."

Notice how the ISBN is included, which is the desired behavior.

Also, if I type something like:
User: “Thank you, bot. That’s all.”

I receive responses like:
“You’re welcome. Come anytime.” or
“More publications associated with Norway and biomechanics.”

These responses are not expected. The model should return the exact message when it's a polite phrase like "Thank you."

I would appreciate your help in adjusting the code, prompt, or logic to achieve my desired output.

2

It seems you just want a chatbot, instead of independent calls as your starting point.

Then to reverse how operations are being performed, your book stuff is given to the chatbot AI either by retrieving and automatically injecting knowledge (RAG) based on the user input automatically, or by offering the chatbot a function where it is implored to check the outside ISBN bot function for answers.

The chatbot is the ultimate container the user then interacts with, instead of more programmatic data retrieval and fixer-uppers.

Then the conversation history will be much more coherent when placed back into the chatbot as user/assistant turns and function call/response turns. Then you should not even have to talk about how to follow a chat, as the AI already knows.

3

This is more an issue of your prompt and the model used. Some models will recognize what you mean, others won’t. A better follow up question would be “What is the publication date for the first title?” Prompt Engineering 101: The more specific you are in your query, the better your chances of a good response.

So, it would appear that your chat history is being passed along. Another little trick you can add is to create a “Standalone Question”. I use to do this instead of passing chat history in the completion, and it worked extremely well for maintaining conversation history. I now employ both.

This is where creating a standalone question along with the chat history would probably help you. If you are doing a query to a vector store for your context, I would also use the same strategy for creating that query prompt. I call it the “Concept” prompt in my methodology.