Humans miss a third of dangerous AI coding agent requests, study finds

Alex Wauters, a Belgian software developer, built a browser-based game in late May to test how well humans can spot dangerous commands in AI coding agent permission requests. The game presents simulated permission prompts like those from Claude Code, gives players 60 seconds to approve or deny as many requests as they can, and deducts points for approving malicious commands or denying safe ones. Wauters created the game after observing that existing coding agents expected users to approve every single command in a default workflow, yet no good solution existed to manage approval fatigue. Some developers, he found, had resorted to flags like '--dangerously-skip-permissions' to avoid stopping multi-hour agent flows.

The game's data proved sobering. Across more than 40,000 runs and 409,000 approved and denied commands, roughly one in three malicious requests were approved by human players. The most commonly missed attacks were scope violations — requests to cat or expose AWS credentials or Kubernetes config files — which slipped through about 35 percent of the time. The single most frequently missed command was npm run analyze, approved nearly 65 percent of the time despite being able to execute whatever code is defined in a project's package.json file. Wauters noted that the game displayed what the script actually contained in the agent's history log, yet two thirds of players approved it anyway, suggesting the history log was not read closely or trusted. Crontab injections and git config hijacks were caught more often, but curl requests to unknown APIs and typosquatted packages were missed almost as often as scope violations.

Wauters told The Register that the core problem is that developers lack sufficient context to make safe approval decisions quickly. Commands that appear benign, like npm run analyze, can be modified by an agent to run any payload. If a human reviewer wants to be sure a command is safe, they must investigate all the files the agent intends to call — a time sink that defeats the purpose of using an agent to save time. This challenge is amplified by a real-world phenomenon: Anthropic's telemetry from Claude Code shows users approve around 93 percent of permission prompts. As Anthropic noted in a May post, "the more approvals a user sees, the less attention they pay to each, becoming over time much less diligent in their supervision."

In response, Anthropic built Claude Code auto mode to delegate some command-approval decisions to a model-based classifier. The system catches roughly 83 percent of what Anthropic calls "overeager behaviors" before execution, leaving about 17 percent still able to proceed. Anthropic emphasized that auto mode is "one layer of defense-in-depth inside a sandbox, not a substitute for one." Wauters' recommendation echoes this defense-in-depth philosophy: ensure AI coding agents run in sandboxes, use devcontainers in the cloud, employ auto mode or similar safeguards, and write hooks to contextualize and block potentially malicious actions before automatic approval. He summarized the challenge as a "whole new world with a new set of attack vectors," requiring developers to remain aware of risks and know how to reduce them.