Assistant does not check ALL documents before answer

Hi all,
i’m developing a simple assistant that should help the user about a set of products.
I uploaded two very semple .pdf with this content:

PDF1:

ProductOne:
Engineered in 2021. Measures: 100x80, 100x50

PDF2:

ProductTwo:
Engineered in 2018. Measures: 100x100, 100x50

it seems that the assistant consider only the first pdf.

If I ask:
“Which product has been engineered after 1998?”
He replyes just “ProductOne”. But also ProductTwo is engineered after 1998!

If I ask:
“Which product is 100x50?”
He replyes just “ProductOne”. But also ProductTwo is 100x50!

Only if I reply “are you sure? Are there other products with that measures?” Then, he replyes: “yes sure, there also ProductTwo”.

Is there a way ti instruct the assistant to check all documents before reply and enumerate all the matching products?

I’m using gpt-3.5-turbo

Thanks a lot guys!

What other things did you tell it about the catalogue. What did you put in the Assistant instructions. The questions you list are only the ‘user part’ - in the Assistant instructions you could start with something like 'You are a warehouse manager that answers questions about our products. You have access to all the catalogues of the company. Each product has the following characteristics: year engineered and dimensions. Please note that a single product can be available in more than one dimension.
When answering questions make sure to always check all the catalogue documents and double check to list all products found.

Hi @jlvanhulst thanks a lot for your reply.
I tryed with this instructions:

You are a helpuf assistant that uses only the files as source of knowledge. Questions not inherent to the documents have to be declined.
You answer questions about our products. You have access to all the catalogues of the company. 
When answering questions make sure to always check all the catalogue documents and double check to list all products found.

But this is the chat:

User
which model is 180x34?

Bot-234
I’m sorry, but I couldn’t find the information about the specific model “180x34” in the document. If you have any other documents or relevant details, please feel free to provide them, and I will be happy to assist you further.

User
are you sure?

Bot-234
I will perform another check to ensure that I didn’t miss any relevant information about the model “180x34.”

Bot-234
The document does not provide any information about the model “180x34.” If there are additional documents or specific details related to the model, please provide them so that I can assist you further.

User
What about the dimensions of MODELLOUNO?

Bot-234
The dimensions available for MODELLOUNO are 100x50, 120x70, and 180x34【23†source】.

You are asking for a ‘model’ in singular form. My guess is that you need to speak in plural. Instead of this:

Which model is 180x34?

Try this:

Which models are 180x34?

2 Likes

Just to double check - you have both documents added to the Assistant. (Not the thread) So when you look in https://platform.openai.com/assistants and open the Assistant you see two files under Files?

Now also note that your question is Which MODEL is xxxx. You don’t use the word MODEL in your text file - or in your prompt. So you would have to do a better job explaining that. What is a model? A product? How could it know!

Also - I would probably consider formatting the document different (more like a CSV) , especially if you are expecting more rows. Also I’m curious if addeing quotes around the dimension will make a difference or not.

2 Likes

@grandell1234 thanks for you reply.
You are right, I used a singolar form, but using plurarl nothing changes unfortunately.

@jlvanhulst thank you too.
Yes, I’m sure the files are there. Infact, if I ask for further models it gets it as you can see my last thread:

User
Which models are 180?

Bot-234
The model available in the size 180 is the “MODELLOUNO”

User
And what about MODELLODUE?

Bot-234
The “MODELLODUE” is also available in the size 180x34

User
so why you did not report MODELLODUE in the first answer?

Bot-234
I apologize for the oversight in my initial response. Both “MODELLOUNO” and “MODELLODUE” are available in the size 180x34

.

He admits the error but it makes it in every new thread :smiley:

1 Like

Just make sure to expand on the prompt explaining explicitly what ‘MODELS’ are. If I have learned anything the last few months is that my prompts are usually not explicit and detailed enough before I get them right.

@factory.factory have you tried “gpt-4-1106-preview” just to see if this might be a problem with the model?

Hi @timpietrusky my last conversation was with gpt-4-1106:

User
Which models are 180?

Bot-234
The model available in the size 180 is the “MODELLOUNO”

User
And what about MODELLODUE?

Bot-234
The “MODELLODUE” is also available in the size 180x34

User
so why you did not report MODELLODUE in the first answer?

Bot-234
I apologize for the oversight in my initial response. Both “MODELLOUNO” and “MODELLODUE” are available in the size 180x34

@factory.factory my experience so far is that the retrieval is just not good enough yet. What I’m doing instead is adding the relevant information to the thread in a message and then asking my questions about the data. So a combination of OpenAI Assistants + RAG.

For this topic, I can also recommend the nice article https://gpts.luona.dev/research/knowledge-retrieval-experiment as it shows experiments of retrieval using files (they call them “knowledge files”). And your case is also tested there with horrible results (or at least I think the experiment is comparable).

Thank you @timpietrusky , I’m going to check the resources you posted!

Just an update: I noticed that if the assistant has already talked about both products, then it responds correctly.
It looks like Assistant loads the info in a lazy way but, once loaded, then it uses both documents.
This is my last thread:

User
tell me about MODELLOUNO and MODELLODUE

Bot-234
…it reports all the content of the pdfs…

User
Which models are 180?

Bot-234
The models that have a size of 180 are: - MODELUNO: available sizes 180x34【16†source】. - MODELDUE: available sizes 180x34【19†source】. Both models offer a variant with a width of 180 cm.



Here we go!

Does this happen in the same thread only or with every thread you start? What happens if you add a new document, that it didn’t talk about earlier?

@timpietrusky

If I open a new Thread without ask “tell me about MODELLOUNO and MODELLODUE” then it does not respond correctly. He forget about MODELLODUE every time.
But if I force to load both models with “tell me about MODELLOUNO and MODELLODUE” then it works good.

If I load a third pdf with a new model I get the same result. I have to ask explicitly about MODEL1, MODEL2, MODEL3.

This behaviur is not aceptable for a production environment. I hope it is possible to fix it in next versions of gpt4.
It would be awsome to inform the OpenAI dev team of this results.

1 Like

From what I observed the API does not check the knowledge base during load times or during any incidents (which occurred mostly due to model request load).

Does anyone think the same?

Are you specifying both file_ids in your Message? I don’t know for sure, but I think I got better results when I listed the files in the message and didn’t depend on the assistant using all the files it’s associated with.

I noticed the same wrong behaviour also merging the two PDFs in a single one.