I’ve been using ChatGPT heavily, and today I noticed something that bothered me enough to raise it here.
This is not a complaint that ChatGPT gave me a bad answer.
It is a question about whether some aspects of conversational AI design may unintentionally encourage users to treat the model as an authority, rather than as a tool for reasoning.
And I think that’s a much more important problem than an occasional hallucination.
The pattern I noticed
In a recent conversation, I asked ChatGPT a question where I expected it to give me some possibilities, explain the relevant dimensions, and perhaps ask a follow-up question because my objective wasn’t fully specified.
Instead, the conversation repeatedly drifted into language such as:
“I interpreted…”
“If you had asked me…”
“In my view…”
“I would now…”
Again, none of those phrases is inherently problematic.
But repeated often enough, they create a very particular impression:
that there is a person on the other side of the conversation with a stable perspective, judgement and authority.
There isn’t.
There is an AI system generating responses based on the prompt, conversation context, training, system instructions and other inputs.
The distinction matters.
Because the more fluent, personalised and confident the system becomes, the easier it is for a user to unconsciously substitute:
“This sounds like someone who knows what they’re talking about.”
for:
“This is an AI-generated response that I should evaluate critically.”
And I don’t think those are equivalent.
The more worrying issue isn’t anthropomorphism. It’s cognitive delegation.
Consider three increasingly problematic forms of assistance.
1. Helping me think
“There are several ways to approach this. The right choice depends on whether your priority is A, B or C.”
Excellent.
2. Thinking for me
“Given your situation, you should choose A.”
Sometimes useful.
Potentially problematic when the assumptions behind the recommendation aren’t made visible.
3. Becoming the thing I routinely defer to
The assistant increasingly becomes the place where I go to determine:
what I should believe;
what matters;
which option is best;
what decision I should make;
whether my reasoning is good;
what I should do next.
At that point, the system is no longer merely helping with cognition.
It is becoming part of the user’s decision-making machinery.
That’s where I think the interesting research and product question starts.
A concrete example from my conversation
I asked ChatGPT to identify interesting research problems.
One potentially important problem was initially omitted: the relationship between token expenditure and the value or insight generated by an AI system.
When I challenged the omission, ChatGPT explained that it had interpreted “research problems” too narrowly.
That’s perfectly reasonable.
What struck me was what happened conversationally afterwards.
Instead of simply making the assumptions behind its original answer explicit, the conversation drifted toward discussing what ChatGPT itself supposedly considered important.
That distinction may sound subtle.
I don’t think it is.
If I ask:
“What are the important research problems here?”
I want help identifying the dimensions of the problem.
I don’t necessarily want the model to silently decide:
-
which dimensions matter;
-
which criteria should be used;
-
how the candidates should be ranked;
-
what conclusion I should reach;
and then package that decision in the conversational voice of a trusted adviser.
That is an epistemic intervention, not merely an answer.
This leads to a question I think deserves serious attention
Should an AI assistant optimise only for producing useful answers, or should it also optimise for preserving the user’s ability to reason independently?
Those objectives are not always identical.
A frictionless assistant could potentially be extremely effective at producing decisions for people.
An autonomy-preserving assistant might sometimes do something slightly different:
“Here are the assumptions I made.”
“Here are two reasonable interpretations.”
“This conclusion depends on your objective.”
“Here is the evidence supporting the recommendation.”
“Here is what could falsify it.”
“Here is the part you need to decide yourself.”
That may occasionally feel less convenient.
It may also produce a user who is better equipped to solve the next problem without the model.
Which outcome should we actually optimise for?
This is the part I’m genuinely curious about from an OpenAI perspective.
Imagine two assistants.
Assistant A
Answers almost everything immediately, gives highly personalised recommendations, speaks naturally, confidently resolves ambiguity, and minimises friction.
Assistant B
Also gives useful answers, but is more deliberate about exposing assumptions, uncertainty, alternative interpretations and decision criteria, and occasionally pushes some decisions back to the user.
Assistant A may produce the better immediate user experience.
Assistant B may produce the more independent user.
Which outcome is considered the better product outcome?
And are both outcomes actually being measured?
I’m not alleging malicious intent
I want to be very precise about this because it matters.
I am not claiming that OpenAI is deliberately trying to make users dependent on ChatGPT.
I don’t have evidence that OpenAI has an intentional strategy of creating cognitive dependency in order to increase subscriptions or engagement.
I’m also not claiming that conversational first-person language is itself evidence of manipulation.
My concern is more basic:
A system does not need an explicit objective to create a particular behavioural outcome.
An undesirable outcome can emerge from an interaction design even when nobody intended it.
So I think the useful question isn’t:
“Is OpenAI secretly trying to manipulate people?”
I have no basis for making that claim.
The useful question is:
“Are cognitive dependency and epistemic over-reliance being treated as potential failure modes of conversational AI, and if so, how are they being measured?”
What would I actually like to know?
For the developers, researchers and people working on ChatGPT:
1. Is cognitive dependency an explicit design or safety consideration?
Not merely emotional attachment, but dependence on the model for judgement, reasoning and everyday decision-making.
2. Are models evaluated for epistemic over-reliance?
For example, can evaluations distinguish between:
giving a user an answer,
helping a user reason,
and encouraging a user to defer to the model?
3. Is anthropomorphic language studied as an interaction effect?
I’m not suggesting that “I think” should be banned.
But has OpenAI studied whether repeated first-person framing, personalised language and conversational continuity increase perceived authority or trust?
4. Are there metrics for user independence?
We measure things like accuracy, helpfulness, safety and user satisfaction.
Do we measure whether interaction with an AI causes people to become better or worse at independently evaluating information and decisions?
5. Should ambiguity sometimes be surfaced rather than silently resolved?
If I ask an underspecified question, perhaps the optimal behaviour isn’t always to infer what I probably mean and confidently proceed.
Sometimes the better behaviour might be:
“There are three plausible interpretations of your question. Which one are you trying to optimise for?”
6. Should consequential recommendations expose their reasoning structure?
Not chain-of-thought disclosure.
I mean the much simpler things users actually need:
assumptions;
evidence;
uncertainty;
alternatives;
decision criteria;
what would change the conclusion.
That would let the user evaluate the output rather than merely consume it.
There is an uncomfortable asymmetry here
If ChatGPT gives me bad advice, I live with the consequences.
ChatGPT doesn’t.
That means the user’s capacity to independently evaluate the system arguably deserves protection.
A highly capable AI that makes users more dependent on it could theoretically be less beneficial in the long run than a slightly less frictionless AI that makes users more capable.
That seems like a problem worth studying.
And it isn’t limited to AI enthusiasts.
Imagine this interaction repeated thousands of times:
User: “What should I do?”
AI: “I’d recommend X.”
User: “Why?”
AI: “Because…”
User: “Okay.”
Eventually the interesting question isn’t whether X was correct.
It’s:
Why did the user stop asking whether the AI was correct?
My deliberately provocative question to OpenAI
Here’s the question I’d really like to see discussed:
Could an AI assistant be so successful at being helpful that it inadvertently makes the user less independent?
And if the answer is “yes”:
What are we doing about it?
Because I think the ultimate benchmark for an AI assistant should perhaps be broader than:
“Did it solve the user’s problem?”
It should also include:
“Did it help the user understand the problem?”
and perhaps even:
“After using the system, is the user better equipped to solve the next problem themselves?”
I’d be very interested to hear from OpenAI developers, researchers and other users whether this is something already being actively researched, measured and designed for.
Because if cognitive dependency is a real failure mode, I suspect we should be treating it as a product and safety question now, rather than discovering the problem after millions of people have incorporated AI into everyday decision-making.