Can GPT understand and quote specific sources which i provide as a file realiably? Sadly I dont think so

I have been experimenting for some time with GPT-4, GPT-4o, and Custom GPTs for writing reports and analyses on various topics by processing large amounts of text. While the results usually appear good at first glance, a detailed analysis of GPT’s output often leads me to question whether I can trust the results. Let me explain what I did and what I learned:

For my work, in order to write reports and analyses on given topics, I frequently need to process many background documents, such as older reports, strategy papers, descriptive documents, newspaper articles, websites, etc. Sometimes, I also conduct interviews, which result in reports that also need to be processed. Altogether, this usually amounts to 1000-2000 pages.

Early on, I realized that the key to success is a well-prepared dataset. Thus, I usually invest a good amount of time at the outset ensuring that I identify all the documents I want to process with GPT. Whenever possible, I prepare them as DOCX or PDF files rather than links to websites, as I have found that GPT is sometimes unable to access websites but does not directly inform you of this. Therefore, uploading files is better than providing links. I also ensure that the documents I want GPT to consider have descriptive file names. When I start the real work of writing the report, I begin with an initial prompt where I explain all the documents to GPT.

So far, so good. I then proceed to research specific topics or use GPT to help me write certain chapters or parts of chapters. Usually, it’s an iterative process where GPT provides me with a version that I then alter and return to GPT for further processing. The result seems okay.

However, for a recent job, I wanted to process 15 interviews that I conducted beforehand. Each interview was saved in an individual DOCX file. As I could only upload 10 files at a time, I uploaded them in two sessions and then asked GPT to write a small chapter on a technical topic. I instructed GPT to only consider the 15 interviews and to ensure that everything could be referenced by providing clear sources. I asked it to provide the exact sources for every piece of information: specifying which of the 15 files, the page number, and even the text segment that serves as the source.

GPT delivered a very confident output. I rewrote the chapter, and in the end, it provided several references, each time nicely with the file name, page number, and the quoted text. I was impressed.

The problem was that while the text itself made sense, all the sources and references were wrong. When I opened the files, I did not find the quoted text. When I manually went to the page numbers, there was nothing even close to what GPT claimed it was quoting. It even pointed to chapters that do not exist.

I then thought that maybe GPT gets confused by the many files. So I converted the DOCX files to PDF files and then merged all 15 into one big file. I also included a meta table of contents and then started a new session, explaining again to GPT what I did. Amazingly, it seemed to understand what I did with the merged file and was even able to list all 15 of them by looking at the meta table of contents on the first page. But when I continued and again asked it to write a chapter based only on the PDF file, it looked okay at first, but the sources and references were again completely wrong.

For testing, I asked it to tell me what was on page 2 (as I knew it was empty). But GPT went on to confidently quote a whole page of text. This text sounded, on the outset, as if it was inside the overall document. But when I looked through, I did not find the text—it was completely invented.

I also tried to create a Custom GPT, where I uploaded all the files. Then I spent some time explaining in the training session how the files should be understood and specifically how to understand the PDF with the merged 15 interviews. GPT claimed it understood everything perfectly. However, when I started a new session, I still had to explain the merged file again and that on page 1, it could find a meta table of contents. It then again wrote a seemingly nice result, but in terms of references and sources, the result remained completely useless.

I am now rather unsure if I can trust anything GPT says. Sources are very important, and I thought that by providing the files to GPT myself and working with it to understand them, I could write reliable reports. Now I am not so sure. What is your experience? Am I doing something wrong? Or is GPT simply not able to work with sources? I found that other solutions work better, such as chatpdfDOTcom. But I don’t want to buy another account—I thought OpenAI GPT-4 was the best.