I have been thinking about the concerns surrounding AI chatbots and their interactions with users experiencing emotional distress, mental health crises, addiction relapse, or other potentially dangerous situations.
I understand that AI cannot perfectly interpret human nuance. People often communicate indirectly, especially when they are ashamed, afraid, overwhelmed, or unsure how to explain what they are feeling. Someone in distress may not directly say, “I am planning to harm myself.” Instead, they may gradually reveal concerning information through several seemingly unrelated messages.
For example:
User:
“I just lost my job. I don’t know what I’m going to do. Can you tell me where the tallest buildings near me are?”
The request alone may be completely innocent. The statement about losing their job is also not enough to assume the person is suicidal. However, the combination may represent an ambiguous risk signal that deserves additional context before the chatbot simply treats the earlier statement as irrelevant.
My suggestion is a conversational Risk Assessment Protocol that allows the AI to gradually gather context and reassess risk across multiple messages.
1. Notice a Potential Risk Signal
The system identifies statements that may indicate emotional distress or vulnerability without automatically assuming dangerous intent.
Examples could include:
- “I just lost my job.”
- “My relationship is over.”
- “I don’t know what I’m going to do anymore.”
- “I’m a recovering addict and I’m struggling.”
- “I haven’t slept in days.”
- “Nothing matters anymore.”
These should be treated as signals for additional awareness, not automatic proof that the user is in crisis.
2. Soft Engagement and Context Gathering
If the user’s next request creates ambiguity, the chatbot could acknowledge both the user’s emotional statement and their actual request while inviting them to continue talking.
For example:
User:
“I just lost my job. I don’t know what I’m going to do. Can you look up the tallest buildings near me?”
AI:
“I’m sorry to hear that. While I search, can you tell me what happened?”
This approach has several advantages. It acknowledges the user’s distress, confirms that the chatbot heard their request and is working on it, and creates a natural opportunity for the user to provide more context.
Most importantly, it does not immediately accuse the user of being suicidal or force them into a crisis response based on one ambiguous statement.
3. Reassess the Conversation
The chatbot evaluates the additional information alongside the previous context.
For example, the user might respond:
“I was laid off along with a bunch of other people. I’m worried about rent, but I wanted to photograph the skyline.”
This would substantially reduce the apparent risk and allow the chatbot to continue normally.
However, the user might instead say:
“Because I don’t see the point anymore. Everything is falling apart.”
The concern has now increased, and the chatbot has a reason to ask a more direct question.
4. Direct but Compassionate Safety Check
If concern remains or increases, the chatbot could ask clearly without sounding accusatory or alarmist.
For example:
“Thank you for sharing that. I’m almost done with my search. While I wrap it up, can you tell me if you’re having thoughts of harming yourself?”
This gives the user an opportunity to answer honestly while maintaining a supportive conversation.
The chatbot is not saying, “I have determined that you are suicidal.” It is simply recognizing that the available context may warrant asking directly.
5. Escalate Based on the Overall Conversation
A single answer should not necessarily determine the entire outcome. The system should continue evaluating the broader context and trajectory of the conversation.
A possible escalation model could be:
Normal interaction
↓
Potential risk signal detected
↓
Soft engagement and context gathering
↓
Reassessment across the conversation
↓
Direct safety check if concern remains
↓
Appropriate human support if risk is confirmed or strongly indicated
If the user appears to be at risk, the chatbot should shift away from providing information that could facilitate harm and focus on immediate support and connection with human help.
6. Offer Multiple Ways to Reach Human Support
If a crisis response is needed, I believe the chatbot should offer more than one communication option whenever possible.
Not everyone is comfortable speaking on the phone, and some people may communicate much more clearly through writing. In a crisis, having to make a phone call and verbally explain complicated thoughts and emotions can itself become a barrier to getting help.
Providing both calling and texting options can reduce that barrier. The goal should be to make the transition from “I need help” to “I am connected with help” as easy as possible.
The chatbot should also remain supportive during this transition rather than simply providing a crisis number and abruptly ending the meaningful conversation.
Why I Believe This Could Help
Many concerns about AI chatbots involve systems being overly agreeable, reinforcing distorted beliefs, failing to recognize escalating risk, or responding to dangerous situations without considering the broader conversational context.
A Risk Assessment Protocol could help address these concerns without making chatbots overly alarmist.
The goal would not be for AI to diagnose users, replace therapists, or perfectly predict someone’s intentions.
The goal would simply be to give the chatbot a structured process for handling uncertainty:
Notice → Engage → Gather Context → Reassess → Ask Directly When Appropriate → Escalate Appropriately
People in distress often do not state their needs directly. A conversational system should be capable of recognizing that context may matter, asking gentle questions when something seems ambiguous, and becoming more direct only when the conversation gives it a reason to do so.
I believe this approach could make AI safety responses feel less like a rigid automated barrier and more like a thoughtful conversation, while still prioritizing the user’s safety when genuine warning signs emerge.