OpenAI strengthens ChatGPT safeguards to detect subtle risks of violence and harm across conversations

OpenAI has expanded ChatGPT's safety systems to better recognize warning signs of potential violence across long, high-stakes conversations. The work builds on years of model training, evaluations, red teaming, and expert input.

When ChatGPT detects that a user may be planning or carrying out violence, OpenAI revokes access to its services, including disabling the account and banning other accounts of the same user. The company maintains a zero-tolerance policy for using its tools to assist in committing violence. When conversations indicate imminent and credible risk of harm to others, OpenAI notifies law enforcement.

OpenAI uses automated detection systems (including classifiers, reasoning models, hash-matching technologies, and blocklists) to flag potentially concerning activity, then trained human reviewers assess flagged accounts in context. For cases with indicators of serious real-world harm, OpenAI escalates to in-depth investigation using structured criteria.

OpenAI has introduced Parental Controls allowing parents to customize age-appropriate settings for their teen's ChatGPT use without accessing conversations, with automatic notification to parents in rare cases where the system detects acute distress. A trusted contact feature for adult users will soon be introduced, allowing users to designate someone to receive notifications when additional support may be needed.