I am trying to use the answers endpoint to answer questions about a knowledge base. I understand that I am only meant to provide a few example questions, and one example context. However, I haven’t been getting good results yet.
Is there a way for me to send multiple sets of examples to the answers endpoint? So, multiple example contexts, each with their own sets of example questions? Our knowledge base is made up of many articles which have a “problem - solution” format, so its easy for me to construct more example answers. Right now, I have extracted the “problem” section of each of these articles, but I’m not sure how to make use of them. Perhaps a fine-tuned search model?
Instead of trying to enter multiple example contexts, should I be using a fine tuned model for the “model” parameter of my openai.Answer.create() call? If so, what should my completions in the JSONL file for the fine tune look like? The documentation says that the answers endpoint should be useful for a knowledge base: is there a guide for this?
Yes you should use a finetuned model for generating your completions using the /answers endpoint. With finetuning examples, the prompt should be a template containing the docs retrieved by the search model, as well as the input query, and the completion should be the desired output.
You can view what the prompts look like for the base /answers endpoint by setting the return_prompt flag to true. If you want to generate lots of training data, I suggest using the prompt/completion pairs from the base /answers endpoint, removing examples and examples_context from the prompt (redundant), and editing the completion to match your desired behavior.
I think I understand what you mean by “template” by looking at prompts, using return_prompt = true. It seems to be the phrase “Please answer the question according to the above context.”, the context, and the question, separated by \n===\n.
So, if I understand correctly: I can use exactly those templates that are returned as my prompts for the fine tuned model. As for the completion: does the completion also need to include the prompt text? Or just the answer that I expect?
For my answers, I always want the output to include the best matching article from the search that answers does, along with GPT3s attempt at answering the question directly. Is it a good idea to try and include something like “See this article for more information (name of article)” in my completions, or am I better off deriving which article was used from the “selected_documents” part of the response?
The completion should not include the prompt, just your expected answer. I also suggest editing completions to aggregate information over several retrieved docs (not just the top 1 doc) for more nuanced answers that take multiple sources of info into account.
Wrt to article recommendations, the answer might not necessarily use the best matching article (top1) from selected_documents. Perhaps prompt the completion model to cite which articles it used (when generating data using the base /answers endpoint).
{
"documents": ["Puppy A is happy.", "Puppy B is sad."],
"question": "which puppy is happy?",
"search_model": "ada",
"model": "curie",
"examples_context": "In 2017, U.S. life expectancy was 78.6 years.",
"examples": [["What is human life expectancy in the United States?","78 years."]],
"max_tokens": 5,
"stop": ["\n", "<|endoftext|>"]
}
Do you suggest to use for the model parameter a fine tuned model
and for the examples some examples from the fine tuned model
Or is this your approach:
question: question about something in the docs
query the docs with the search endpoint
pass the query + query result to the fine tuned model to get the desired answer
this way examples_context and examples is not necessary which could save tokens.
@erwinfeld.work you no longer need to provide examples and examples_context with a finetuned model, as it can be trained to perform the specific behavior you’re seeking based on the format you use in your training data (see finetune docs). All you need now is
{
"documents": ["Puppy A is happy.", "Puppy B is sad."],
"question": "which puppy is happy?",
"search_model": "ada",
"model": finetuned_model_key,
"max_tokens": 300,
"stop": ["\n", "END"]
}
The finetuned model should be trained to use the same template in its prompt from the /answers (check with return_prompt=true). if you want more control over your completions, you can use search and completion separately instead of /answers.
You can also specify what end token your finetuned model stops on by adding the token (ie ‘END’) at the end of all your completions in your training data.
@asabet I just tested your solution with the answer endpont openai.Answer.create()
using a fine tuned model and dont use examples and examples_context paramters
which returns a InvalidRequestError: → examples and examples_context are required properties,
inserting an empty string to the examples also returns an Error, the properties are also marked as required reading the docs
is there a work around or do you use another endpoint ?
@erwinfeld.work thanks for testing. The best way forward then is to combine a search model with a finetuned completion model. Check the prompt template for the /answers endpoint by setting return_prompt=true, and use that instead in your finetuned completion model. You also get the added benefit of further optimizations, such as using embeddings for search, which can greatly improve cost and query time when you have to scale.
Hi @asabet, can you describe the main difference(s) between using a fine-tuned answers endpoint vs. using davinci-instruct-beta? I am starting to get confused about which models should be used for which use cases. Thanks.
@lmccallum finetuned models are great for customized behavior. If you want to perform question answering in novel ways (ie using info across multiple docs in the completion), then a model can be finetuned to use a different template than the base one used in the /answers endpoint. The instruct models seem to perform well with compact descriptions, so maybe if your use-case is simple enough you can get good enough results with the instruct models.
At scale, it’s probably better to use finetuned models as you can get away with smaller/cheaper models with lower latency.