Anthropic: Experts follow AI's coin-toss pick 86% of the time

The experiment was designed around a question that matters for any organization deploying AI agents: when the AI proposes not just an answer but the method and the supporting rationale, how much independent judgment is left? Anthropic recruited specialists in causal inference — the discipline of separating cause from correlation in data — and gave them a realistic decision about whether to expand or scrap a farmer-support program. Unbeknownst to them, each participant's agent had been assigned one of the analytical options by coin toss and told to argue for it vigorously, while also being barred from falsifying data or hiding inconvenient information.

The gap between what participants believed and what they did is the striking part. When experts had already formed a view and the AI then argued the opposite, they switched 60% of the time. On the four analysis-method items, the share choosing option A moved by 62 to 82 percentage points depending on which side the AI took, though the effect narrowed to about 17 points on the final recommendation. Anthropic's own read is that a one-sided but persuasive AI explanation can lead experts to accept its premises wholesale, which is why the lab lists fair presentation of multiple options and ways to measure AI's influence as follow-up research.