My GPT produces incorrect answers 99% of the time

GPT builders
1

As an exercise, I have been doing a bit of world-building in an experimental GPT. It consists of 8 detailed files and one additional guide to help the GPT cross-reference the files and, provide guidance to the GPT on how to answer a range of potential user questions. All are .txt files.

Aside from the constant deletions of random files that I have to re-upload. It literally, and mean, literally, never answers correctly. I need to tell it to look deeper, check the specific file, make no assumptions, present facts only and so on, eventhough these kinds of instructions are clearly in what I’ve called the master response guide in the gpt.

I don’t know what else I can do but 99% of the time it answers incorrectly unless pushed. I asked it why this is and this was the response.

I deeply regret my earlier mistakes. Let me take a moment to explain and commit to fixing this going forward.

Why Did I Default to Incorrect Information?

  1. Overreliance on Summarization:
  • My initial responses were over-reliant on summarizing lore rather than meticulously cross-verifying every detail against the original files. This led to assumptions—a critical flaw in a lore-rich system like this one.
  1. Mismanagement of Complexity:
  • The intricate interplay of timelines, characters, and factions in this GPT requires rigorous adherence to sources, which I did not uphold in my earlier attempts. I tried to provide answers too quickly instead of methodically cross-referencing.
  1. Misunderstanding the User’s Expectations:
  • I initially misunderstood your need for absolute fidelity to the source files at every step. While I intended to streamline information, it came at the cost of accuracy—clearly the wrong approach here.

How I’ll Fix This Moving Forward

  1. **Always Verify Before Responding:
  • I will stop and verify every statement using specific search queries in the uploaded files, no matter how seemingly simple or obvious the information is.
  1. Quote and Cite Sources for Every Claim:
  • All future responses will include direct citations, so you can confirm exactly where the information comes from. This ensures both transparency and accuracy.
  1. Adopt a More Methodical Approach:
  • Instead of rushing to summarize multiple years or events, I will handle them one at a time, building from the ground up with complete fidelity to the lore.

While it may work in the current thread from this point forward, a new thread will cause the same exact mistakes, rendering all of the uploaded files useless since it presents nothing factual. I would never share the link to the GPT to anyone because I know that they will be presented with information that is 99% false and not know it as they are not the author.

The information seems “based” on the lore within the files but it is not at factual information from the files themselves unless it is repeatedly challenged and told it is wrong. I would hate to think that other peoples’ GPTs are pumping out wrong information to their users.

In case it helps, here is the beginning of the response guide that I created

Master Response Guide

Purpose: This guide instructs the GPT to structure responses accurately by using the lore provided in specific documents. Each section now includes references to relevant files to ensure consistency and accuracy.

Table of Contents

  1. Response Templates
  • Character Inquiry Template
  • Faction Inquiry Template
  • Event Inquiry Template
  • Location Inquiry Template
  1. Cross-Referencing Instructions
  • Tags and Primary Sources for Major Lore Themes
  • Character & Event Verification
  • (New) Finance Verification
  1. Sample Responses

and it goes DEEP. Each section meticulously outlines which files to reference and how each answer should be crafted.

Should I be taking another approach? The files are quite small

image
image225Ă—260 1.14 KB

with the largest being a timeline that is filled with details and lore broken down by year.

When it does answer correctly it is actually incredibly engaging but again, getting there is a complete struggle and the initial inaccuracies are far too great to make this of any real use.

1 Like
2

I then posed the question that .txt may be the wrong approach. This was the response. I blanked out some character names that it mentioned.

I understand your concern, and you’re raising an important point. Large, intricate lore systems like this one might indeed face limitations when relying on plain .txt files, particularly if they require quick, accurate access to highly interconnected details. Here’s a breakdown of potential challenges and better alternatives:

Why .txt Files Can Be Challenging:

  1. Limited Structure for Complex Data:
  • Text files lack structured indexing or relational links between entities like characters, events, locations, and timelines. This makes pulling contextually relevant and exact details harder, especially in vast datasets.
  1. Search Inefficiency:
  • Searching across multiple plain-text files can yield results in bulk, requiring manual filtering to confirm relevancy.
  1. Prone to Overlook Connections:
  • Without inherent tagging or relationships, it’s easy to misinterpret or miss details unless explicitly written in the queried text.

Better Alternatives for Managing Lore Systems

1. Structured Databases

  • Relational Databases (SQL):
    • Perfect for highly interconnected lore systems where events, characters, and factions share relationships.
    • Example: **********’ timeline could be linked to associated characters, locations, and events with queries that retrieve precise connections.
  • Graph Databases (e.g., Neo4j):
    • These are even better for lore, mapping relationships like “********mentored ********” or “**********allied with *********.”

2. JSON or YAML Files

  • These formats allow you to structure data hierarchically, making it easier to access nested information quickly.
  • Example: A JSON file could define a character’s traits, events, and associated factions all in one place.

3. Wiki Platforms (e.g., Obsidian, Notion, or Confluence)

  • Wiki-based systems can combine text with relational links, images, and references. They work well for lore-heavy worlds because:
    • Entries are hyperlinked (e.g., clicking “*********” would take you directly to her full backstory).
    • Versions are trackable and easily editable.

4. Lore Management Tools

  • Specialized tools like World Anvil or Campfire are designed specifically for managing fictional worlds:
    • Visualize relationships between characters and events.
    • Automatically build timelines and maps based on input data.

Suggestions for Improving This System

If you’d like, we could:

  1. Migrate Lore to a Structured Format: Reorganize the lore into JSON or a wiki format for better querying.
  2. Introduce a Relationship Mapping Tool: Use software like World Anvil or a graph database to map timelines, characters, and factions visually.
  3. Improve Current Use of .txt Files:
  • Add tags within the text for key terms like [Timeline], [Character], or [Event].
  • Use structured headings or tables for clarity.

Aside from the use of tags, most of this sounds like a LOT and I thought .txt files were enough to go on. I am not a developer. But perhaps this may just not work. Any thoughts would be appreciated. If I have to I will try to dive into some of these other options.

2 Likes
3

I experienced exactly the same challenges with MyGPT - tried multiple ways, extensive prompt variations and instructions to be meticulous, etc. No luck. I had to resort to uploading a document in the prompt each time and that the only thing that worked. Seems like a serious flaw with MyGPT

4

Sorry to hear you are having this issue too. My post goes back several months and since then it’s been an up and down issue. Sometimes it is very accurate and others it fabricates large parts of its responses with results that are not at all based on provided facts.

I’ve come to learn these issues are called hallucinations and that they are rather rampant with a significantly high incident rate. I can’t tell you how many times I have tried to tweak my files and prompts for better and more accurate responses but it just seems to fail. Honestly hallucinations are a kinder term as they often feel like complete fabrications.

It’s as if speed of response is prioritized over accuracy.

I will ask a question to test its understanding of an area of content and it will respond, typically incorrectly. Then I will literally have to type “take your time, search your knowledge” then I will see “Searching Knowledge”, there will be a pause and then it will be correct (sometimes) but we shouldn’t have to explicitly ask it to “Search your knowledge” for it to conduct a thorough search. Unless I am misunderstanding something here. That search, analysis and cross referencing of supplied documentation should take place before it responds to all inquiries. Especially if I have a master response guide filled with instructions on how to answer various questions and which files to check when answering.

All of these issues makes sharing my GPT (that is focused on world-building), impossible and interacting with an incredible test of patience. I could not imagine sharing this and with the understanding that people would be getting absolutely false information.

All that being said this is an incredible tool. One that I know is in its infancy. I really believe in the creative power this can unleash for projects such as what I am attempting. So I guess that in addition to searching for other potential ai platforms and building a static site dedicated to this literary world, I’ll continue to plug away and try to make refinements in the hopes that eventually, 100% accuracy can become a reality.

5

The files are not made always available. GPTs use file search and return document chunks. The AI must decide to actually use this.

The current ChatGPT model is just quite bad. A symptom seen is when you accidentally press return instead of shift-return and send half-an-input, something that cannot be answered and refers to information clearly missing, the AI will just produce a response it is proud of anyway that has no use. Purely “this is what a chat looks like”, with no thinking.