Safety

AI-generated text

OpenAI introduces child safety tools and support for investigators

OpenAI is making a new classifier for suspected child sexual abuse material available free of charge to vetted organizations, alongside new ChatGPT warnings and developer guidance.

OpenAI introduces child safety tools and support for investigators

OpenAI has announced new tools and partnerships intended to prevent, detect and respond to the online sexual exploitation and abuse of children. The package includes an image classifier, a warning in ChatGPT, developer guidance and support for child-safety investigators. The company says the measures build on its existing child safety program.

New image classifier

The classifier looks for potential child sexual abuse material (CSAM), including images generated by artificial intelligence. It examines visual signs of abuse, which can help teams identify images they have not previously encountered. Its scores help reviewers decide which images to examine first. OpenAI says it is the first broadly available CSAM classifier with a specific signal indicating that material may be AI-generated.

The company trained and evaluated the system and prepared it for practical use with assistance from the National Center for Missing and Exploited Children (NCMEC). The nonprofit operates CyberTipline, a centralized reporting hub. OpenAI is providing vetted organizations with free access to the tool through the OpenAI API.

The Tech Coalition and ROOST are helping distribute access. OpenAI has also joined the Tech Coalition's Lantern program, through which participants share signals concerning the online sexual exploitation and abuse of children to strengthen cooperation across platforms.

In tests conducted internally and with early partners, the company examined how much abusive material the classifier detected and how often it flagged other images. Based on an evaluation of user-uploaded images, OpenAI estimated that the new tool would flag 48% fewer images than its existing system while finding roughly the same number of abusive images. Images flagged by the new classifier were about twice as likely to contain abuse.

In tests by fal using human-reviewed production images, reviewers confirmed that 72% of the classifier's high-confidence flags were previously unseen CSAM. In invideo AI's testing, one in four flagged cases led to a ban. In SafetyKit's evaluations involving drawings and animations that sexualize children, the classifier detected 90% of cases. OpenAI plans to continue improving the system based on user feedback.

Warnings and developer guidance

OpenAI says it already prohibited CSAM, trained its models to refuse requests for it and took action against accounts that violated its policies. The new ChatGPT warning interrupts attempts to find abusive material, explains the consequences and directs people to help with stopping harmful behavior. It also states that sexualizing children is harmful even when the material is AI-generated.

The flagged conversation will be removed and ChatGPT access paused for 24 hours, while specialist support remains available. Repeated violations can lead to longer restrictions and ultimately account closure or reports to authorities.

The approach was developed with input from Protect Children and the Lucy Faithfull Foundation, as well as Dr. Caoilte Ó Ciardha, a researcher and honorary reader in forensic psychology at the University of Kent, and Joel Scanlan of the University of Tasmania. Scanlan independently evaluated a trial of deterrent messages.

The company has also published guidance for developers on preventing abuse and detecting, reviewing and reporting harmful content. It was developed with input from the Tech Coalition, Thorn, NCMEC and the Internet Watch Foundation. The guidance covers user reporting channels, safety identifiers that can connect misuse to individual users, and support for staff who review sensitive content. Alongside the new classifier, it can be used with tools including OpenAI's Moderation API, ROOST's self-hosted Coop review console and Thorn's Safer detection system.

OpenAI, Anthropic and Google also jointly commissioned separate guidance on how developers should respond when child sexual exploitation or abuse is disclosed in AI systems. OpenAI says that following the publication of those recommendations by the Tech Coalition, more developers have accessed child-safety tools and registered to report CSAM to NCMEC.

Support for investigators

OpenAI is also supporting specialist child-safety law enforcement units with new tools and training on using AI in investigations. The program includes dedicated plugins, investigative tools and complex data analysis, with access managed by program administrators.

The first tool available at launch will help investigators review lengthy chat logs and records returned through legal processes, identify potential connections and build timelines. Its findings will be linked to the source material.

The initial training will consist of an overview and demonstration, supported by a practical field guide that teams can consult afterward. Access will depend on agency authorization and program requirements. The company plans to propose future versions of the training at regional, national and international conferences.