I’m building a chat with pdf type bot. I have tried using both 3.5 Turbo and 4o-mini, but am facing errors with both. The error is that they keep using information not mentioned in the pdf. I tried changing the system prompt to make it more strict, and sometimes it just starts saying ‘not mentioned in data’ to even questions which are relevant to my pdf, and I have to rephrase them to get an answer.
I have around 5k chunks in a csv which i have embeddings for (my RAG pipeline) and I’m sending the top 20 based on cosine similarity to the api. What can I do to try and fix this?
Either it uses all external info or gets wayy to restrictive.