From everything I have read so far (no personal experience as of yet), Assistant API is simply not designed to handle a dataset this large in neither an efficient nor cost-effective manner. As you quickly discovered:
So, 1/3rd of the total files you really want to use?
I’ve heard this a lot also. Imagine what the cost will be when you add the additional 2/3rds of files.
If you’re happy with the overall performance, then that’s all that matters.
But I really think you may want to consider the RAG approach using the Chat Completion API. It will be way more scalable, far less expensive (to operate on an ongoing basis), and give you at a minimum the same results you are getting now.
I mean, it’s certainly none of my business, but I’m just amazed at how many hoops people are willing to try and squeeze through to get the Assistants API to do what it was not designed to do. 20 file limit should have been the first clue. Just something to consider.
