Feature Request: Create a Feedback Loop From Real-World ChatGPT Outcomes
Suggested tags: chatgpt, feature-request, feedback, training
I have a product suggestion that I believe could substantially improve ChatGPT over time.
ChatGPT has access to an enormous amount of information that people have published. But there is another potentially valuable source of knowledge that seems largely underutilized: the real-world experiences of people who actually use ChatGPT’s advice.
The missing feedback loop
Consider a typical conversation:
A user asks ChatGPT how to solve a practical problem. ChatGPT researches available information, reasons through the situation, and recommends a course of action.
The user then actually goes into the real world and does it.
Sometimes the advice is exactly right.
Sometimes it is technically correct but misses an important practical detail.
Sometimes the user’s experience reveals something that isn’t documented well anywhere.
And sometimes the advice is simply wrong.
Today, there doesn’t appear to be a clear mechanism for turning that subsequent real-world experience into structured knowledge that can improve future ChatGPT responses.
I think there should be.
A real example
I recently worked through an automotive repair problem with ChatGPT involving replacement wheel studs on a Kia Optima.
During the conversation, we reasoned through how the broken studs could be removed. When I actually performed the repair, I discovered a practical detail that wasn’t obvious from the generic procedure.
Because the stud heads were still intact, the heads contacted the steering knuckle before the studs could be driven completely through. I had to drive the studs approximately another 1/8 inch, cut the heads off with a die grinder, and then drive the remaining portions out.
The repair was successfully completed.
I subsequently performed the brake-bedding procedure without incident and rechecked the torque on all 20 wheel nuts. Everything checked normally.
The important point isn’t the Kia repair.
The important point is what happened:
AI recommendation → real-world execution → observed outcome → verified result → new practical information.
That’s a feedback loop.
Imagine if ChatGPT could capture that loop
Suppose ChatGPT offered an optional feature such as:
“Share this real-world outcome with OpenAI.”
After a user has actually followed advice, ChatGPT could help structure the feedback:
- What recommendation did you follow?
- What did you actually do?
- What happened?
- Did the result confirm or contradict the recommendation?
- Was there a practical detail missing from the original advice?
- How confident are you in your observations?
- Did you personally observe the result, or are you reporting something you heard from someone else?
The user could review exactly what would be submitted before consenting.
The resulting information could then be treated as first-hand experiential evidence, rather than simply another piece of text.
Don’t blindly train on it
I am NOT suggesting that OpenAI simply dump user conversations into a training set and assume everything users say is true.
In fact, I think that would be a bad approach.
One person saying, “This worked for me,” should not become universal truth.
Instead, the system could distinguish among:
- established technical information
- published documentation
- expert guidance
- first-hand user experience
- conflicting user experiences
- repeated experiences from independent users
- experiences that have subsequently been corroborated
The system could evaluate and aggregate these reports while retaining appropriate uncertainty and provenance.
For example:
“This procedure is documented by the manufacturer.”
is fundamentally different from:
“Three independent users report that this procedure worked on their specific vehicle.”
Both are useful. They simply represent different kinds of evidence.
Why this could be extremely valuable
The internet contains an enormous amount of published information.
ChatGPT conversations contain an enormous amount of problem-solving and experiential information.
Those are not the same thing.
A manual can tell you how something is supposed to work.
A person who actually performed the repair can tell you how it worked in practice.
This could be particularly valuable in areas such as:
- automotive repair
- home improvement
- appliance repair
- electronics
- software troubleshooting
- RVs and recreational equipment
- tools and machinery
- cooking
- gardening
- travel
- and countless other practical domains
Over time, ChatGPT could potentially become better at answering not only:
“What does the documentation say?”
but also:
“What happens when people actually try this?”
The larger opportunity
This could create a genuine closed-loop knowledge system:
Existing knowledge → ChatGPT recommendation → real-world experience → structured feedback → evaluation/corroboration → improved future knowledge
That is fundamentally different from simply collecting more web pages.
It would allow ChatGPT to learn from what people have actually experienced, while still treating those experiences as evidence that must be evaluated rather than automatically accepted as fact.
I believe this could become one of the most valuable sources of practical knowledge available to an AI system.
I’d be very interested in hearing whether OpenAI has considered something along these lines, and if not, whether this type of opt-in, privacy-protective, real-world feedback system could be incorporated into future versions of ChatGPT.