Help needed: how to make AI search and cite knowledge efficiently?

How to make a tiny version of a ““perplexity””-like tool.

Say I have 1000 press article previews: ID, Title, Abstract, URL. This is my knowledge… a total of about 50-80 thousand tokens on average

Then I have some user preferences: keywords, language, geo, age… (1000 tokens max)

Question for the pros: to the best of your knowledge, what is the best way to obtain the following result with AI:

select among the 1000 articles the top-20 relevant articles given the user preferences (and return me a JSON array with those 20 IDs.

Putting 1000 articles in the prompt can be an option, dumping them as JSON or CSV, many models accept more than that. It can be expensive but I don’t care much for now. I’m open to spending $1 for this task

What about using “RAG” instead? Embedding my knowledge in a vector db and asking the LLM to select for relevance? Will it work? Is a RAG able to return the exact ID of the source? this last part is something I’ve never done

Thanks

I recommend LlamaIndex for this.
If you want a fully operational “Chatbot” while using Pinecone or LlamaIndex with the OpenAI API or a local LLM, check out Flowise.
Cheers! :hugs:

thanks! would you mind walking me through my doubts before I dive into a tool?

What are the macroscopic steps invoved if I decided to go with llamaIndex? as I said , my use case is not a chatbot. is a mini-perplexity, no followup question.

I want to call “it” via an entry point, passing it 1000 articles+ user prefs, and get as output 20 ids of most relevant articles.