How can I replicate ChatGPT’s behavior of attaching a document to the first message in a new chat and then asking different questions about it in successive messages? My first message usually says something like: “You are a helpful assistant. I’m going to ask you several questions about the attached document.”
When you use ChatGPT in this manner, you may notice that, each time you ask a new question, the interface says “Reading document” before answering. This suggests that the document is stored in some cloud storage and automatically accessed with each new message/question.
How would you replicate this using the OpenAI API? Alternatively, would simply pasting the original document as text with each question in successive API calls achieve the same outcome?
2 Likes
Hi and welcome to the developer forum.
What ChatGPT is doing is loading the contents of the document and inserting that into the prompt each time you ask a question along with any prior questions from you and answers from it.
The underlying model is stateless, so it needs to be sent everything every time. There is no way around this. So, you could simply include the document at the start and append extra text onto that, or you could try using retrieval augmented generation (RAG) if the document is very long, but that will require a more complex interaction to be built.
Posting the document each time you call the API has to be done anyway if you want to go down the simpler route, so yes, your original supposition was correct.
6 Likes
Thank you @Foxalabs. Very clear.
I didn’t know about RAG. I just researched it a bit and it looks like this could be very useful for us. I originally thought that ChatGPT was doing something similar to RAG when attaching a document – I mean that ChatGPT would identify and extract the relevant sections from the attached document for each new question/message, instead of loading and inserting the whole content each time…
2 Likes
ChatGPT “can” use RAG, it can make use of Microsoft AI Search which is a hybrid solution that can make intelligent decisions when to use RAG and when to just include all of a document given the current context size and conversation length.
At this stage it’s quite hard to tell which mode is being used RAG or whole document, as CHatGPT is not metered like the API.
2 Likes