working with gpt api, I need to send top 5 chunks based on similarity.
sometimes ,face this issue where some content is lost due to chunking size. Is there any possible way to achieve dynamic chunking so we can overcome this issue. Thus, based on content the chunking can vary and loos of content is reduced.
Think about it , If you got answer pin it up , could help a lot of folks.
also appreciate any research papers on this…
You can embed the data in two different vector databases with different chunk size and splitting techniques.
From the other context you have, you can use just the database chunk size that is preferable.
You can also run two embeddings simultaneously, and then do a quick search to see if the small one significantly overlaps the big one to discard it, even mix-and-match.
You also do not “need” to send top-5 chunks. You should use a similarity threshold. If your database has airplane part prices, and I ask for dog grooming tips, the amount of your database needed is zero.