OpenAI to limit weapons information for teens on ChatGPT in violence-prevention push

OpenAI is implementing new restrictions on ChatGPT's content, specifically targeting weapons-related information for teenage users. The company stated it will restrict "practical information on weapons and security systems that could be misused" for users under 18, even if their questions appear harmless. Additionally, for all users, ChatGPT will now limit detailed information on past mass attacks, including the weapons used and how they were obtained. These measures are presented as part of new safeguards against real-world violence.

To enforce these new limits, OpenAI indicated it has strengthened its age prediction system to ensure users estimated to be under 18 are directed to the "teen experience." The system is designed to automatically end chats where a user appears to be fixating on violence, and accounts may face temporary pauses after multiple such conversations. OpenAI also plans to notify parents if their teen's account is paused due to repeated discussions about violence.

The company's initiative comes amidst increasing scrutiny on AI companies regarding the risks chatbots pose to young users and their role in detecting potential harm. While OpenAI reported earlier this month that teens spend under 15 minutes a day on ChatGPT on average, a separate report by non-profit Common Sense Media found that ChatGPT's teen version often alerted parents late or not at all when teen accounts held conversations about suicide, self-harm, or eating disorders. This suggests that the practical application and effectiveness of such alert systems may face challenges.