Q&A Question Matching with Confidence Value

Prompting
1

Hi all,

I’m attempting to do something which I thought would be easy but I’m struggling!

I have a large set of question and answers for my bot. I want to use OpenAI to listen to the user and ascertain what they wanted to know - not to answer the question.

For example, I am able to give a prompt like

Decide which of the following questions most closely matches what the user asked.
1 What’s your role
2 What’s your job
3 What responsibilities do you have
4 What do you do
5 How can you help me
6 Can you answer my questions
7 What’s your purpose
8 Why are you here
9 Tell me about yourself
10 Is a financial assistant the same as a financial adviser
etc. etc…

And in the playground, using a Q&A model, I get excellent results. However, I only want to go with the answer if the AI is 90% certain that the question matches what was asked. So how can I get the correct question from the set along with a confidence value?

Also, I have almost 1, 000 questions. Does anyone have any ideas on how I can avoid sending them all every time?

Thanks so much for any pointers!

2

You can try embedding the questions and using similarity to the user’s question to calculate a confidence score.

2 Likes
3

@wfhbrian Is definitely right with the embeddings. Those are probably your best bet, and theres good documentation and support online for how they work and such.

If you want to be 100% positive that a given prompt is best paired with a given question, you will have to check with each individual question… It’s just a math/logic thing. But if you are okay with some uncertainty, you could take the first similarity that is above a certain threshold (like its above 90% certain), and that way you don’t run through the entire list of questions, and you can be (for all intents and purposes) 100% sure it is a good.

1 Like