How do I determine whether a question will be answered by embedding or by chatgpt?

I have customized my model to respond the way I want, using my data on certain topics, taking advantage of embeddings. However, it should not use embeddings when answering all questions, in some cases chatgpt should also use self-generated answers. How can I determine whether to use embedding or chatgpt when creating an answer to a question.

1 Like

You could have a classfier before the GPT generation calls which classifies the question into whether it is for data for the embeddings or the chatgpt one. This would allow you to sort of use if else to make the correct gpt call to use

1 Like

Thank you very much, what you mean is to take advantage of an ml classification model. Is there a document, cookbook, or resource that will guide me to do this ?

You can find plenty of resources online that can help with question classification.

Another possible solution could be to prompt GPT as well by giving it the topics which you are using the embeddings for and asking whether the question will be answer by these topics or not, though this method will be very temperamental and you will need to structure the prompt quite well.

I really couldn’t find a proper document that would work for me, and I didn’t think about how I could make the classifier according to what. Can you send me a document as a suggestion?

same, I’m looking for something similar but no dice

i’m using the gpt-3.5-turbo model and unfortunately there are no fine-tune on this model. I’m obligatorily using embeddings to generate an answer with my data. My main problem is that when a question comes, how can I distinguish whether it is based on embeddings or not, the answer will be generated in the normal way ?

You should already know if the questions need embeddings or not as they are supplementing the knowledge of your chatbot. Most people use a semantic search using ada-embeddings to determine what pieces of information are relevant.

You would usually fine-tune a much smaller, cheaper model such as ada or babbage for any sort of classification task. In your situation though it’s not really necessary.

You should already know if the questions need embeddings or not as they are supplementing the knowledge of your chatbot.

How can I know exactly that. There may be a lot of type questions, there are millions of possibilities, I can distinguish whether there is a need to embed the incoming question as a person, but how do I make the model do this?

Maybe (context being build by semantic search):

Answer the question based on the context below, and if the question can't be answered based on the context, say "I don't know"\n\nContext: ${context}\n\n---\n\nQuestion: ${question}\nAnswer:

And if ChatGPT answers “I don’t know” you can use it without context

That’s what I do, but there are a variety of “I don’t know” responses from GPT-3.5. It usually starts with “The context does not provide” but I don’t think that’s completely reliable. It’d be nice if there were a flag included in ChatCompletionResult that indicated whether the completion could be supplied based on the given context.


I’m also looking into determining if the ChatGPT model was actually able to answer the question using the gpt-4-turbo model via the REST API.

There doesn’t seem to be anything in the REST response that would indicate it.

Trying to add system prompts such as “If you cannot answer the following question, reply ‘no’” don’t seem to work. I still get a generic “as an ai model I …”

Anyone have any success with this?

You would have to inject context, usually via RAG, and then stipulate to say “No” if the answer is not found in the context.

Simply asking the model if it knows something or not, without context, will likely be a hallucination, or a canned response.