I am using an assistent that gets a few files and then ask questions about these files.
I recognize that every question from my python app causes the use of a big amount of tokens (more then 15 thousands, even for short questions and answers).
When I remove all files that I uploaded the token amount is reduced by a lot.
For me it seems the files are automatically part of the input that is used to answer a question. Is there a way to pretrain it, so that the files are not part of the input anymore? Maybe the assistant is not the way to go for this case?
Especially for several textfiles with a lot of text this will make my AI agent very expensive.
You can train an AI that produces questions on demand.
The output still must be as a result of some kind of unique stimulus as context input. Otherwise a list of 100 random questions from files is likely better than asking an AI to write a question. An AI that would have to be trained on writing the file-knowledge type of question from its fine-tuning on 1000 examples of well-substantiated questions. An up-front cost in developing the training set that is exactly the type of large input you want to avoid, made mandatory (or rather every generation you receive and record now being possibly a fine-tuning example case in the future)
Instilling new knowledge answering capability via fine-tuning is quite elusive. It is much better for simply the style.
Then you are probably wanting more than a curious AI, maybe you want one that can administer a quiz…
I would try to solve this with simple vector store. I would put all the uploaded documents to the non-persistent vector store and then I would query the store. Long contexts can be handled with summarisation techniques as well.
Yeah, this architecture sounds inefficient, consider a RAG strategy with Completions instead.
Assistants are great and all, but I doubt many experts use them and will end up moving to Completions so they can have more control over what the LLM is doing call by call.