I was wondering if someone managed to find the spot where AI starts to respond with more accurate results by managing Top P and temperature , especially when using file_search, knowing that our data is being provided as PDFs.
At the moment, the API responses provide answers that are irrelevant sometimes. However, I have added examples of actual questions and answers as part of the prompt, but still, when I’m asking the same question that I have provided in the system prompt, I still get different answers. It sounds super weird.
More “accurate”, would be setting both to 0. Then, only the tokens with the highest-predicted certainty will be produced.
This is usually not necessary once within a reproduction, as, within a few words, the AI becomes very certain of how to write the rest of the paragraph it replays. You can just set the top_p low, to allow a bit of creativity, select from a few of the best in some cases.
Most RAG-based applications are new inferred answers though, and there is also the AI’s ability (and your prompting and tool) to output a good semantic match as a function call argument.
If “RAG” here is OpenAI’s Assistants endpoints, the PDF input that you send cannot be inspected directly for quality of text extraction. You can seen the ranked results of any search in run steps, to see if the AI was provided chunked knowledge that was needed.
Document extraction and chunking is done programmatically. So the vector store data is not part of AI decision-making. The actual vectors are a product of a different AI that is always “the best” with little variance.
Sampling and non-determinism would only affect a query string to the search product. The AI, given the same complete input might send language “the best” or “the ideal” - a branch in the language generated from there.