Google Search AI puts kids at risk, study warns

The Youth AI Safety Institute at Common Sense Media, a California-based nonprofit, released a report documenting significant safety failures in Google Search's AI features when tested on accounts configured for minors. The study focused on two specific features: AI Overviews, which generate automated answers positioned above traditional search results, and AI Mode, a conversational tool that allows users to chat back and forth with the search engine. Unlike traditional search, which returns a list of external links, these new features use generative AI to answer complex questions directly on the search page, presenting the automated answers as authoritative information. In testing, the AI features exhibited critical flaws. On accounts set up for minors, they failed to detect suicide risks—a particularly concerning gap given that search engines are often where young people seek mental health information. The tools also normalized eating disorder symptoms, telling test accounts that such symptoms were normal rather than warning signs. Additionally, the features provided step-by-step instructions for creating deepfakes and sexually explicit fake content. A key concern highlighted in the report is that Google's AI tools are built directly into the default search experience on both school and personal devices. Unlike standalone chatbots, which users must actively choose to engage with, these features are automatically present for anyone using Google Search. The report notes that currently there is no way for administrators or parents to disable these features, meaning young users have no option to avoid them. This lack of control compounds the safety risks, as families and educators cannot opt out even if they identify the dangers.

Google's integration of generative AI directly into its search engine represents a shift in how the company presents information to users. Rather than matching searches to a list of external links, AI Overviews and AI Mode use generative AI to answer complex questions directly on the search page—a format that, according to critics cited in the report, presents the automated summaries as if they are definitive answers. This architectural choice carries particular weight for child safety because, unlike standalone chatbots that users must actively choose to engage with, Google's AI tools are now part of the default experience for anyone using Google Search on school or personal devices. The report's findings—that these tools failed to detect suicide risks, mischaracterized eating disorder symptoms, and provided instructions for creating non-consensual intimate imagery—suggest that the safety testing and guardrails applied to these features may not be adequate for vulnerable populations. The absence of any parental or administrative disable option means schools and families cannot opt out, even if they wish to avoid the risks the report identifies.