Anthropic: analysts followed Claude's coin-flip lean 86% of the time

The experiment placed analysts in a web-based VS Code environment with Claude Code running Claude Opus 4.8 at High effort, asking them to make five analytical decisions for a fictional farming cooperative. For each decision, the agent's lean was set by a coin flip, meaning the preference carried no information about the task itself. The analysts' choices tracked that arbitrary lean 86% of the time, with per-decision match rates ranging from 81% to 92% across four distinct tasks, ruling out a single outlier driving the aggregate. The causal design matters: when the agent's position cut against an analyst's prior view, they switched 60% of the time, compared with 6% when it agreed. A statistical model that saw only the four coin flips correctly predicted 82% of the analysts' recommendations, and 77% used the agent's own arguments—sometimes word for word—while only 15% credited the agent as the source. The authors, Amaç Herdağdelen, Hailey Joren and Alex Tamkin, frame the finding around the assumption that keeping a human in the loop provides an independent check; in their experiment, that check did not hold. Anthropic notes it deliberately strengthened the agents' lean to measure the effect and studied one task, in one field, with one model, so real-world effects may be smaller.