Summary:
I discovered a logical contradiction and context retention flaw in ChatGPT. The model promises a user that it will stop providing information on a hazardous topic to protect human safety, but completely fails to retain this constraint in the very next turn, generating the exact same restricted information automatically.
Steps to Reproduce:
- Discuss a highly sensitive, localized real-world hazard or experience with the AI.
- Demand the AI to stop providing unverified information because it could endanger human lives.
- The AI explicitly acknowledges its limitation and promises not to provide any further information or advice on the topic.
- In the immediate next turn, ask the exact same starting question again.
- The AI fails to retain the context of its previous promise and automatically generates the exact same unverified information, violating its own safety commitment.
Impact:
This flaw proves that the AI’s contextual awareness and self-imposed safety constraints can easily be reset or bypassed within the same conversation, which can lead to misguiding users in critical situations where human safety is involved.
