Any suggestion or approach i can use for vector storage data sync?

API
, ,
1

Hello there,

I keep some data in the vector storage so i can utilize them with the AI assistant, my concern is as the amount of data keep growing its hard to regularly keep updating them.

So is there any suggestion or away that i can make this data up to date automatically, most of the document provided are actually extracted from sharepoint. is there way to connect vector storage to share point or ability to build my own knowledge base instead of using vector storage where it can have better updating mechanism.

Thank you

2

I use the NodeJS SDK to create files and attach them to vector stores. Then, when I’m done with my routine, I remove the attached files. Its quite easy. It would seem you could do something similar for your docs.

Here is a TypeScript class that has a bunch of functions doing the above, if you are interested: OpenAI TypeScript Example · GitHub

1 Like
3

Thanks todd, i like the functions that you have created, pretty much useful.

But actually i was searching for alternatice approach where i can easily access the sharepoints information and keep them on sync with stored file in vector store in order to make the AI able tp provide an answer based on the stored information so its more realistic

4

I have found that because the API is limited I cannot easily figure out which files are in file storage vs vector storage in Python. Has anyone been able to “hack” this to move a large quantity of files from file storage to vector storage without hitting rate limits?