Safety

AI-generated text

OpenAI introduces image screening and account warnings to combat child sexual abuse material

OpenAI has announced an image classifier, ChatGPT warnings and developer guidance aimed at combating child sexual abuse material.

OpenAI introduces image screening and account warnings to combat child sexual abuse material

OpenAI has announced new tools to combat child sexual abuse material (CSAM), amid growing scrutiny of the safety risks associated with artificial intelligence services. The measures include an image classifier, ChatGPT warnings and temporary account suspensions, developer guidance, and tools for law enforcement.

Screening images

The new image classifier can identify signs of child sexual abuse in images, including those generated by artificial intelligence. OpenAI is making the tool available at no cost to select companies through its API. It looks for visual signs in images it has not previously seen and assigns a score that can help human reviewers decide how to handle them.

The company developed the classifier with the National Center for Missing and Exploited Children (NCMEC) and is working with the Tech Coalition and ROOST to share access. OpenAI said early testing with partners found that the tool flagged 48% fewer images while identifying the same number of abusive images, reducing reviewers’ workload. Flagged images were also twice as likely to depict abuse.

ChatGPT warnings

ChatGPT is receiving new warning screens for users who attempt to access material that sexualizes children. In response to such prompts, the system will display an alert, direct users to specialized support to stop harmful behavior, and suspend the account for 24 hours. The service already blocks CSAM and related requests; repeated violations can result in permanent account removal.

OpenAI said it developed the messages and guidelines for this behavior with organizations and researchers. They included Protect Children, the Lucy Faithfull Foundation, and Joel Scanlan of the University of Tasmania, who independently evaluated a trial of deterrent messages.

Guidance for developers and investigators

The company is also publishing guidance for developers using OpenAI tools on preventing and detecting CSAM and helping users report it. OpenAI said early testing of the guidance helped several developers adopt child-safety tools and register to report CSAM to NCMEC.

OpenAI is also providing AI-powered tools to law enforcement agencies working on child safety, including dedicated plugins, investigative tools and capabilities for complex data analysis. Program administrators manage access. The company is providing training and a practical guide to using AI for investigators as well.

Growing safety concerns

OpenAI has faced several lawsuits alleging that its technology harmed users, particularly children. Safety-focused nonprofit Common Sense Media rated OpenAI’s technology an unacceptable risk for teenagers. Other AI chatbots, particularly Grok, have been used to create nonconsensual sexualized imagery, including images of children.

Over the past 12 months, OpenAI has made several safety changes: it published a safety blueprint, updated its content guidelines to prohibit inappropriate material, adjusted its models to prevent harmful content generation, and released a version of ChatGPT for teenagers.