Daily Updation of Knowledgebase

API
, , , ,
1

Hi all,
I made a chatbot for which i had given 50 news article and and storing embedded in .npy file , I am not using any vector db.

Now the main problem is there are many news article coming on daily basis , What should i do for the the automate so that my knwoledgebase update automaticlly , without recomputing previous data . Knowledge source are different news website. I dont want to use any third party tools if not neccessary , except Vector DB.

What should be the best approach for this usecase.

2

Hi @sarthak.srivastava !

I am not sure I understand fully, but there is probably no need to re-compute anything, you just need to load your existing numpy array from the .npy file, get the embeddings on the new batch of articles, append those to the array, and save it back to your .npy file.

If it starts to get very large (millions of articles) you probably want to start looking at a proper vector db like Pinecone.