Classify whether a question can be answered from the provided data

[erik9631]

I am kind of new to the OpenAI API
We have a custom company AI chat and we are using an embedding model for factual checks.

We don’t want to keep the context history within the memory because in some cases it might contain a lot of tokens.

• We want to decide whether it is actually worth it to look into the embedding model
• We want to avoid adding duplicate data into the context.
• There are cases where the AI might ask a question regarding the context
• Any question asked by the user returns a context from the embedding model to the chat.
  – A simple “hello, how are you” will add a context from the embedding model this is not necessary.
• We want to avoid doing this and only look up the embedding table when needed.
• Currently we solved this by only having ONE context in the memory.
  – This is a problem since if you ask another question within the context of the previous one, the chat will lose the data

In order to do that we need some way to classify whether the data in the context + the chat history is sufficient to give an answer to the user who is asking a question.

I tried experimenting with gpt-3 prompt to classify the data but I failed miserably. Is it even capable of doing this?

Could you suggest me how would I go about solving this issue?

[wfhbrian]

Instead of classification of the context, you could classify whether the incoming user input, or last few messages in the chat log, need the context from your corpus.

Also, I don’t find it necessary to store the past context from embeddings in the chat message history, since the assistant’s answer should provide a sufficient summary for use as context in future messages. This will reduce your token use substantially.

Lastly, if your inserting only one piece of context, and that context is over 250 tokens(~1000 characters), then you would probably benefit from smarter “chunking” of your embed inputs.

[erik9631]

Most of what you have said is already implemented that way. The issue we are facing is this:

Instead of classification of the context, you could classify whether the incoming user input, or last few messages in the chat log, need the context from your corpus.

That is the idea. The question is how to go about it. My idea was building a secondary embedding database from the chat history and seeing if there is a good match. Is there a more efficient method to do this?

[curt.kennedy]

You can certainly do it without embeddings using GPT-3 with a fine-tuned Ada or Babbage. More details below, but I do this regularly, and it is one of the hidden gems of GPT-3 as compared to other solutions out there. It’s quick, accurate, and relatively inexpensive too to run these fine-tuned models.

Do you fine tune? If so why?

Just curious… Do you fine tune your model and if so why? Are you trying to reduce the number of tokens you pass in? If so what’s the cost trade off? Fine tuned models are expensive… At Microsoft, lots of people are saying to avoid fine tuned models. They’re expensive and its not clear there’s a benefit over a well written prompt. Just curious what others in the community are finding…

How to "escape" user input?

You could also train a classifier with one token output (’ 1’, ’ 2’, …, ’ 10’). Then force the temperature to 0 and set it to max 1 tokens. That will fence it in so much, and there is no such thing as a prompt in the classifier, so no worries about a prompt injection. Plus trained classifiers perform well with lower (cheaper/less $$$) models such as Ada or Babbage. You can even use GPT-3 to create the training dataset for the classifier if you feel it’s accurate enough in its raw capabilities…