Classification Prompt and Overall Application Flow Suggestions

Prompting
1

Hi there, I am trying to create a simple system that answers user questions based upon documents.

I currently have it where I use an embedding to search through documents, based upon an asked question.

After I retrieve the most relevant document, I think have a prompt that attempts to answer the question based upon the content of the document. If it cannot answer the question, it tells the user to ask another question.

Here is my expected interaction flow:

Greeting intent

User sends a greeting: “Hi”
System classifies as a greeting.
Response: “Hi, please as a question.”

Question intent

User sends a question: “How do I find x?”
System classifies as a question.
System searches for relevant documents
System attempts to answer question based on to document context.
Response: [[answer]]
Response: Is this answer relevant, reply (Yes or No)?

Answer intent

User sends: “Yes”
System classifies as Yes or No response
Response: [[Whatever prompt]]

I am assuming with the Yes or No response it would be easiest to not even have it go through the openai, and determine this on the server.

For me the question is how do I determine if there is a question (or implied question) or a greeting? I have tried a few prompts and none of them seem to be consistent.

Is it best to just search off whatever the user enters and if the search score is less than a certain threshold then assume it isn’t a valid question and prompt them to ask another question? Or say hi?

Here is the first prompt I have tried:

Read the text and answer all of the questions with a single word response “Yes.” or “No.”.

Text: “Hi, where is the nearest dog park?”

Questions:

  1. Can words be found in the text?
  2. Does the text ask us a question?
  3. Does the text say hi to us?
  4. Can the word “Yes” be found in the text?
  5. Can the word “No” be found in the text?

Answers:

  1. Yes.

The prompt above wanders on lower models and sometimes provides more than just a “Yes” or “No”. And seems inaccurate on anything lower than Davinci.

I am not sure if it is best to have a series of boolean questions, or ask for the bot to determine the intent.

Or is it better do to something like this where you determine the purpose and then search off the generated response?

Something like this:

Read the passage and answer the question based on the passage. If you cannot answer the question, answer “No answer”.

Passage: Hi, I was wondering where is the nearest dog park?

Question: What is the purpose of this passage?

Answer:

I greatly appreciate any feedback!

2

Hello, I am dealing with a similar problem. What solution have you found ? I’m looking for how to determine whether an incoming question should be answered by embedding or by chatgpt.