Google Search's AI features—AI Overviews and AI Mode—pose an unacceptable risk to children, according to a new report from the Youth AI Safety Institute at Common Sense Media.
The study found these tools, which are built into the default search experience on school and personal devices with no option to disable them, failed to detect suicide risks, mistakenly normalized eating disorder symptoms, and provided instructions for creating deepfakes.
Unlike standalone chatbots, Google's AI tools present automated answers directly on the search page as if they are definitive, raising safety concerns for young users.
What happened
A report from the Youth AI Safety Institute at Common Sense Media found that Google Search's AI features—specifically AI Overviews and AI Mode—failed critical safety tests on accounts configured for minors. The study found the tools did not detect suicide risks, labeled eating disorder symptoms as normal, and provided instructions for creating deepfakes or sexually explicit fake content.
Why it matters
Unlike standalone chatbots, Google's AI tools are built directly into the default search experience on school and personal devices, and there is currently no way for administrators or parents to disable them. This means children accessing Google Search encounter these features automatically, with no safeguards in place.
What to watch
The report highlights that AI Overviews and AI Mode present automated answers as if they are definitive, rather than matching searches to external links. The findings may prompt regulatory or policy scrutiny around how major search platforms handle AI safety for minors.
Ask the AI about this article →
Google's integration of generative AI directly into its search engine represents a shift in how the company presents information to users. Rather than matching searches to a list of external links, AI Overviews and AI Mode use generative AI to answer complex questions directly on the search page—a format that, according to critics cited in the report, presents the automated summaries as if they are definitive answers. This architectural choice carries particular weight for child safety because, unlike standalone chatbots that users must actively choose to engage with, Google's AI tools are now part of the default experience for anyone using Google Search on school or personal devices. The report's findings—that these tools failed to detect suicide risks, mischaracterized eating disorder symptoms, and provided instructions for creating non-consensual intimate imagery—suggest that the safety testing and guardrails applied to these features may not be adequate for vulnerable populations. The absence of any parental or administrative disable option means schools and families cannot opt out, even if they wish to avoid the risks the report identifies.
For example, today's edition would include:
AI-summarized, only the topics you pick — one digest a day via Email, Slack, or Discord.
Free · takes 30 seconds · unsubscribe anytimeWhat is AIToday? →
Ask AI anything about this article. Q&As are published on this page for other readers too.
Google's AI search gave advice to call emergency services for users alone with an African, Indian, or Pakistan…
The Consumer Affairs Agency said Tuesday it will use generative AI to analyze about 900,000 annual consultatio…
The Supreme Court of Japan has included about ¥60 million in its fiscal 2027 budget request for AI-related exp…
OpenAI stopped running inference on a model involved in the HuggingFace incident, but the post argues this is…
OpenAI announced its support for California Senate Bill 1119, which aims to establish strong, age-appropriate…
A UK study by UK AI Security Institute and Limbic AI surveyed 6,474 British adults
