I’m trying to develop an assistant using an Assistant API, and I have a PostgreSQL database containing all the data related to books in a library. I want the assistant to suggest books to users based on their interests provided in the message. However, I’ve encountered difficulties in implementing this suggestion mechanism, especially when considering that not all books are available to all users. Availability is determined based on factors like the user’s age and other parameters. For example, a user below 5 years old can only see books from the kids category, while a user under 15 can access books from both the kids and teen categories, and so on.
The challenge I’m facing is that more than 80 percent of the data is shared among multiple users, with only 20 percent being unique to each user. As a result, it doesn’t seem efficient or cost-effective to fetch all the data from the database, convert it to a JSON file, and upload it to the knowledge base each time a user requests book suggestions. Additionally, maintaining individualized data for each user in the knowledge base could lead to redundancy and increased costs.
I want the assistant to suggest books based on the user’s interests and availability, without recommending books that aren’t accessible to that specific user. Book availability status, new books, rental price, discounts, publicly available, etc values get updated frequently in the database so how can connect such changes with the assistant, do I need to create new embedding for complete data every time even if a single field gets changed . How can I achieve this effectively and efficiently?
Any suggestions would be greatly appreciated!