Pope's AI Adviser Warns of 'Cartel' Rule-Setting by Big Labs

The warning from Father Paolo Benanti lands in a moment when the loudest AI-risk voices are coming from the labs themselves. Anthropic CEO Dario Amodei, for example, has called for mandatory third-party evaluations before release and for a crackdown on distillation, the process of training rival models on another model's outputs. Others push to ban recursive self-improvement, where AI writes code for its own future versions.

Benanti's objection is less about the specific technical curbs than about who gets to write them. He says restricting development methods could quash innovation, so his focus is on guarding against models slipping from human control. His suspicion is aimed at the frontier labs' incentives: OpenAI and Anthropic have backed calls for a slowdown while preparing to go public, with many billions of dollars at stake, and US-China rivalry further muddies the picture. Anthropic declined to comment; OpenAI and Google DeepMind did not respond.

In May, Pope Leo XIV made artificial intelligence the subject of his first encyclical, warning of the technocratic logic that treats humans as projects to be optimized and urging rivals to set aside the desire for commercial or geopolitical dominance. Benanti reads the superintelligence debate as a potential distraction from that agenda. Whether his push for democratic regulation gains traction may hinge on how convincingly the public and policymakers separate genuine safety concerns from the labs' own market positions.