How do I determine whether a question will be answered by embedding or by chatgpt?

klcogluberk · March 10, 2023, 7:27am

I have customized my model to respond the way I want, using my data on certain topics, taking advantage of embeddings. However, it should not use embeddings when answering all questions, in some cases chatgpt should also use self-generated answers. How can I determine whether to use embedding or chatgpt when creating an answer to a question.

udm17 · March 10, 2023, 7:54am

You could have a classfier before the GPT generation calls which classifies the question into whether it is for data for the embeddings or the chatgpt one. This would allow you to sort of use if else to make the correct gpt call to use

klcogluberk · March 10, 2023, 8:03am

Thank you very much, what you mean is to take advantage of an ml classification model. Is there a document, cookbook, or resource that will guide me to do this ?

udm17 · March 10, 2023, 11:39am

You can find plenty of resources online that can help with question classification.

Another possible solution could be to prompt GPT as well by giving it the topics which you are using the embeddings for and asking whether the question will be answer by these topics or not, though this method will be very temperamental and you will need to structure the prompt quite well.

klcogluberk · March 10, 2023, 11:53am

I really couldn’t find a proper document that would work for me, and I didn’t think about how I could make the classifier according to what. Can you send me a document as a suggestion?

vicmcorrea · March 10, 2023, 3:28pm

same, I’m looking for something similar but no dice

klcogluberk · March 11, 2023, 5:28pm

i’m using the gpt-3.5-turbo model and unfortunately there are no fine-tune on this model. I’m obligatorily using embeddings to generate an answer with my data. My main problem is that when a question comes, how can I distinguish whether it is based on embeddings or not, the answer will be generated in the normal way ?

RonaldGRuckus · March 11, 2023, 5:56pm

You should already know if the questions need embeddings or not as they are supplementing the knowledge of your chatbot. Most people use a semantic search using ada-embeddings to determine what pieces of information are relevant.

You would usually fine-tune a much smaller, cheaper model such as ada or babbage for any sort of classification task. In your situation though it’s not really necessary.

github.com

openai/openai-cookbook/blob/main/examples/Question_answering_using_embeddings.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "c4ca8276-e829-4cff-8905-47534e4b4d4e",
   "metadata": {},
   "source": [
    "# Question Answering using Embeddings\n",
    "\n",
    "Many use cases require GPT-3 to respond to user questions with insightful answers. For example, a customer support chatbot may need to provide answers to common questions. The GPT models have picked up a lot of general knowledge in training, but we often need to ingest and use a large library of more specific information.\n",
    "\n",
    "In this notebook we will demonstrate a method for enabling GPT-3 to answer questions using a library of text as a reference, by using document embeddings and retrieval. We'll be using a dataset of Wikipedia articles about the 2020 Summer Olympic Games. Please see [this notebook](fine-tuned_qa/olympics-1-collect-data.ipynb) to follow the data gathering process."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "9e3839a6-9146-4f60-b74b-19abbc24278d",
   "metadata": {},
   "outputs": [],

This file has been truncated. show original

klcogluberk · March 11, 2023, 6:02pm

You should already know if the questions need embeddings or not as they are supplementing the knowledge of your chatbot.

How can I know exactly that. There may be a lot of type questions, there are millions of possibilities, I can distinguish whether there is a need to embed the incoming question as a person, but how do I make the model do this?

louis030195 · March 12, 2023, 4:27pm

Maybe (context being build by semantic search):

Answer the question based on the context below, and if the question can't be answered based on the context, say "I don't know"\n\nContext: ${context}\n\n---\n\nQuestion: ${question}\nAnswer:

And if ChatGPT answers “I don’t know” you can use it without context

sammyjava · May 2, 2023, 2:13pm

That’s what I do, but there are a variety of “I don’t know” responses from GPT-3.5. It usually starts with “The context does not provide” but I don’t think that’s completely reliable. It’d be nice if there were a flag included in ChatCompletionResult that indicated whether the completion could be supplied based on the given context.

kevin.parker.372 · April 14, 2024, 5:53pm

Hello,

I’m also looking into determining if the ChatGPT model was actually able to answer the question using the gpt-4-turbo model via the REST API.

There doesn’t seem to be anything in the REST response that would indicate it.

Trying to add system prompts such as “If you cannot answer the following question, reply ‘no’” don’t seem to work. I still get a generic “as an ai model I …”

Anyone have any success with this?

curt.kennedy · April 16, 2024, 1:47pm

You would have to inject context, usually via RAG, and then stipulate to say “No” if the answer is not found in the context.

Simply asking the model if it knows something or not, without context, will likely be a hallucination, or a canned response.

rmp · November 19, 2024, 8:09am

How does a GPT model determine whether a given prompt is a question or an instruction? When working with a RAG technique how does it decide whether to perform a search or simply rephrase the input? If we use the model’s API, do we need to explicitly manage this behavior, or does the model handle it automatically?