I have an assistant that reads regulatory documents. The regulations update every year so I need to make sure that the assistant is using current documents and so I have attached the 2024 version of these docs in a vector store and then attached the store to the assistant. If I don’t explicitly name a file and tell the assistant to reference it the assistant says it does not know about the regulations even though they are in the attached vector store. I have started copy/pasting the names of all the files in the store into the instructions but I was wondering if:
- This is the correct way to make sure the assistant is using specific versions of information?
- Is this hugely inefficient?
- Is there a way to just tell the assistant to just reference everything in its store? I tried that and it once again said it did not have access to regulatory docs.
Thanks so much! This is such a fun and wonderful tool and I’m really excited about all the awesome things we can make with it!
Hi,
Welcome back.
AI doesn’t automatically know anything; nor do we, on our end, have the ability to train it on complex knowledge.
The AI makes decisions based on the information in it’s Context Window. The current model has a 128k token context window.
Both your responses and the AI’s responses are included in this window, which is how it converses. Every time you ask a question, it parses the full Thread, and appends it’s response at the bottom.
When you run a search via AI, it takes Chunks of information it finds in the Vector Store, which become a part of the Context Window.
So, yes, right now, you have to ask the AI to retrieve the information you wish to discuss every time. This is the essence of RAG, Retrieval Augmented Generation.
It definitely helps out if you know the file name, those files are named descriptively. There are some super new ways to work out how to find information more efficiently that I haven’t tested yet.
I always ask the model to look up information I want to discuss, then I quiz it a bit about that data to make sure it got everything. (It loves to summarize, beware. You have to ask it to explicitly report everything it finds.) You can include this process in your Instructions. You can use Function Calling to flag certain keywords for certain processes as well.
Then you have however long until the data falls out of the context window to have a rich and fulfilling conversation.
It’s hard to say if this is hugely inefficient. It’s AI, after all. This part is just “the way it is” for now.
2 Likes
Thanks! This was helpful. I guess I will have to play around with prompts to make sure I’m always getting the info I need. I noticed if I just ask it something like “Tell me what new regulations were added to xyz this year” it will just list a few, so I’m wondering if the docs I’m asking it to read are just too big (total of 66 MB)
2 Likes
When the first time it reads something it bounces around.
I’ve been working on getting it to read “sequentially,” because all I want right now is to pull subject headings from a very long document in multiple small stages.
You may consider converting these regulations to some form of a table—structured data lookups are far more efficient for that type of fact-checking.
2 Likes
Ah good idea. I could make the regulations a giant json file. Or I guess csv might be better. But yea, either is prob better than just lots of text. Thanks again! And that would also mean I could more easily point a user to the references used by the assistant. I guess I could make the table/json/csv a public link and the references returned by the assistant be used to make a link for the user.
1 Like