I want to upload and search on a large number of files whats the best way I , am getting a max file limit
1 Like
Welcome to the forum!
Uploading large numbers of files and searching - what’s the best way?
Don’t do that.
If you have a large number of files odds are they are similar.
If they are similar then they probably have the same file type or magic number.
Before LLMs how was the same process done?
The point of all of this is that just because it may be easier to use an LLM to do this that does not make it the correct solution.
For example many programmers have been having great success with coding tools such as Codex that work with projects full of hundreds of the same files. The Codex via the AI does not load entire files and all of the files. What Codex does is use the tools such as grep, sed, PowerShell, Bash, Git, etc. to do the work. Saves on tokens, is faster, more accurate when done correctly, catches mistakes if syntax is correct, etc.
Since you did not give enough details, can only give a high level ananlogy.
HTH