AI Search using big ammount Data without VECTOR

Prompting
,
1

I’m building an AI-powered search engine using OpenAI, where I need to process around 4,000 titles. The goal is to send a search query and have GPT suggest relevant titles from this dataset. However, there are challenges:

  1. Including all 4,000 titles in a single prompt would exceed token limits and significantly increase costs.
  2. If the search query (e.g., “I want to visit”) doesn’t closely match any title, embeddings might fail to return meaningful results. For instance, GPT might infer that “passport” is relevant because it’s essential for travel, but embeddings might not make this connection.

Please don’t suggest embeddings. Because in some cases embedding might not catch it. Is there another way to achieve this with:

  • Low cost
  • High accuracy

If you provide any suggestions, that will be very helpful. Thank you!

2

LlamaIndex, Flowise, Pinecone

There are many alternatives but from my experiences, these are the smoothest (especially Flowise!) :hugs:

There is going to be no way around embeddings.
What you are asking for is for someone to show you how to fit more water into an overflowing bathtub - it simply isn’t possible.

If you want an alternative, you could implement your own elastic search to limit the 4,000 titles that have to fit into the prompt to a way lower number - but then you’ll have a needle in a haystack situation with hallucinations unless you find some programmatic algorithm to bring the total amount down significantly (<100, if not even <10).

That being said - please look into vector databases and embeddings.
It really isn’t as difficult as it is the best solution for a reason.

Cheers! :hugs:

Edit: For your example case of “I want to visit” → An agentic approach is the perfect solution.
Vector Database + Indexing + Querying + AI Agents (i.e. Supervisor, Worker) until a satisfying result is returned.

1 Like
3

@j.wischnat
Thank you. I am actually a beginner that’s why. Can you elaborate the follow with more easy way. I will use that approve.

I did before generate-embedding using openai and saved vector in the database. Then search through vector. And as I mentioned, it didn’t give the response I expected.
So please for the results I want to ask you to please explain a little bit the flow?
Thank you so much

4

I recommend you to give Flowise a shot.
It’s pretty beginner friendly as it has a node system.
They offer a few presets that probably fit your needs in the marketplace section.
Just install the application and you’ll have a UI in the browser to use it.

2 Likes