Hinton, Bengio, Pachocki warn of 'intelligence explosion' in new paper

The new paper by more than 20 AI researchers, including Geoffrey Hinton, Yoshua Bengio, and OpenAI research lead Jakub Pachocki, adds to a series of recent warnings about AI risks. Other notable warnings include 42 leading mathematicians calling for more attention to existential AI risks, and several AI lab employees voicing fears that AI could destroy humanity. Pachocki himself has said no lab has solved alignment well enough to continue responsibly scaling at maximum speed for much longer, and some Anthropic employees are reportedly already scouting safe havens in case AI goes sideways.

The authors argue that AI systems already write most of the code at the companies building them and could automate the entire AI R&D pipeline within years. If that happens, progress that normally takes years could occur in months, potentially outpacing society's ability to keep up. They warn that control over superhuman AI could slip away and that power balances between nations, companies, and governments could erode.

The outcome hinges on whether policymakers act on the researchers' call for greater visibility into AI research automation, and whether the concerns raised by this and other warnings lead to concrete oversight measures. For now, the paper underscores a growing unease among some of the field's most prominent figures about the pace and direction of AI development.