How to: Update Assistant file with constant product changes

I have created a “sales” Assistant to answer product questions from users. I have a file that was generated from a database dump of 3000+ products. The products are constantly changing, new products being added and others removed daily. What is the recommended method of updating the assistant file? Do I set a cron job to just upload a new file every few hours? Should I be using functions to accomplish the updates from the database? I can’t seem to find much information and appreciate your help.

Hi and welcome to the forum!

Personally, I’d use the API and one of the many vector database solutions out there to store product data.

Do bare in mind that you will need to experiment with chunking options with some solutions while others have automatic settings. if your product details fit under 8000 tokens each then you can chunk at the product level. See solutions like Pinecone, ChromaDB, Weaviate and Quadrant, perhaps combined with langchain or llamaindex as your framework.

Not sure the assistants system is setup for fast changing datasets currently

How does you assistant work? What is the starting point. Agree with some of the vector database solutions - which you could update from you the database perhaps based on triggers. Or indeed if you current solution WORKS but needs daily updating nothing wrong with a CRON job to make it happen? Function calling might also be a solution to do look ups or search. I use that for ‘accounts’ that are in Salesforce.

The OpenAI assistant has its own vector store where you can upload data. I encountered a similar use case when we needed to update our AI salesman’s database (build using OpenAI assistant). We utilized the Apify platform for data scraping to upload data using the Assistant API to the OpenAI vector store via an Apify - OpenAI integration. You can find detailed setup instructions here.

Disclaimer: I work at Apify.