OpenAI Model Struggle with Ambiguous Questions When Using Context

API
1

Good morning everyone,

I am attempting to use OpenAI’s Davinci model to answer technical questions using machine documentation. We are using an approach based on text embeddings and cosine similarity of the paragraphs in the documentation. We provide the most similar context to the model, using a specific prompt that includes the context and the question. The prompt is structured as follows:

Context: “xxx”
“You are the xxx technical assistant. Answer only with the provided context, and if the answer is not relevant, indicate that you cannot answer.”

The bot is able to answer relevant questions and correctly avoids answering completely irrelevant ones, but it has difficulty with ambiguous questions.

For instance, if we give the model the context related to changing the oil in a gearbox and ask the question, “how can I change the oil in the air fryer?”, the bot’s answer will be that, to change the oil in the air fryer you must follow the same instructions for changing the oil in the gearbox (and a description of the procedure for the gearbox).

We are looking for ways to make our model more robust and better able to handle these types of issues. Any assistance would be greatly appreciated.

Thank you, and cheers.

1 Like
2

Embedding vectors method are for search and retrieval, classification, clustering etc.

If you need a more “creative” answer which uses the corpus of data on the net, etc you will need to add a GPT test generation process component to your software architecture to complement your embeddings component.

HTH

3

Thank you for your response.

Perhaps I did not clarify earlier, but we are utilizing a specialized model for text embedding, as well as a standard algorithm to determine cosine similarity. Following this step, we combine the resulting context with the user’s query and input them into the Davinci model. Our issue is not related to the model’s creativity (on the contrary), but rather that it lacks the necessary robustness to decline answering irrelevant questions, as in the example i provided.

1 Like
4

Hi @f.ghelfi

What params are you you using for generating completion?