Seth, Block, Tononi: AI not conscious, computation claim "fatally flawed"

The Nature commentary enters a debate already shaped by the Butlin report, which drew on leading neuroscientific theories to build a 14-indicator checklist and concluded that none of the systems examined — including large language models like GPT-4 — is a strong candidate for consciousness. Butlin and colleagues explicitly adopted computational functionalism "for pragmatic reasons" and acknowledged their indicators would no longer be reliable if consciousness is tied to properties of living organisms, citing Anil Seth among proponents of that opposing view. The commentary picks up exactly there, arguing that in biological brains you can't separate what they do from what they're made of, and that a boundary between mind and living tissue comparable to the one between software and hardware doesn't exist.

The authors also draw on theories that don't rely on computation. Integrated Information Theory, developed by Tononi, holds that consciousness requires a physical system whose components are tightly interconnected in very specific ways — conditions certain brain regions meet and the cerebellum does not, and which computers running deep-learning systems can never meet because of how they're wired, regardless of how powerful the systems are. The Butlin report deliberately excluded IIT because it is incompatible with computational functionalism; what the report set aside, the commentary puts front and center. Approaches from biological naturalism add electrochemical signaling between nerve cells, electromagnetic fields binding activity across the cerebral cortex, and metabolism to the list of material properties that may matter.

The ethical stakes are where the two documents diverge most sharply. The Butlin report weighed both risks side by side — failing to recognize conscious AI systems and thereby causing them suffering, versus false attribution misdirecting resources and making people vulnerable to manipulation. The Nature authors come down firmly on one side: falsely attributing consciousness is far more probable than overlooking consciousness that's actually there, and granting rights to AI systems that merely appear conscious would make them harder to control exactly when oversight is most needed.