Safety researchers warn against accelerating AI capabilities for alignment work

The article challenges a growing idea within the AI safety community: that researchers should focus on selectively accelerating AI capabilities in domains that are seen as bottlenecking alignment work. The specific capabilities in focus are typically those involving abstract, conceptual, or philosophical reasoning—skills thought to be essential for making progress on alignment challenges.

The researcher raises two main objections to this strategy. The first is that the bottlenecks constraining safety research may overlap substantially with those limiting general AI R&D. AI systems today exhibit poor epistemics (reasoning about what they know and don't know), struggle with messy conceptual reasoning that lacks clean definitions or ground truth, and are unreliable on open-ended tasks. If the safety community accelerates progress in these areas—hoping to unlock AI-assisted alignment research—the same improvements will also unlock faster AI capability development across all domains. This creates a race-condition problem: speeding up safety-relevant capabilities also compresses the timeline for other time-sensitive safety initiatives (the article refers to these as "Plan A" agendas), potentially leaving less room to execute them before AI systems become harder to control.

The second concern goes deeper into the governance problem. Even if AI systems became highly capable at abstract reasoning, successfully handing off alignment research to AI requires not just capability but trust. The AI must be trustworthy at 1) actually conducting good alignment research and 2) deferring to human judgment where appropriate. The article suggests this prerequisite—reliable deference and integrity—is not automatically solved by making AI better at reasoning, yet the acceleration strategy does not appear to address it directly.

The article presents a critique of a strategy within AI safety circles that assumes targeted acceleration of reasoning capabilities can help solve alignment challenges without broader consequences. The premise is that certain AI abilities—particularly in abstract reasoning and philosophy—are holding back alignment research, so enhancing them would let AI systems assist human researchers more effectively.

However, the researcher identifies a fundamental problem with this logic: the bottlenecks in safety research may not be unique to safety. If AI systems are weak at epistemics, conceptual reasoning, and tasks without clear ground truth, these same weaknesses also limit general AI R&D. Removing them benefits everyone—both safety teams and teams working on capabilities with no safety focus. This creates a timing problem: accelerating shared capabilities compresses the window for time-sensitive safety work (referred to as "Plan A"), potentially making the overall situation worse rather than better.

A second concern, only briefly outlined in the body, touches on the deeper trust problem: even if AI becomes much better at reasoning, handing off alignment research to AI systems requires confidence that they will reason honestly and faithfully about their own behavior—a prerequisite that the acceleration strategy itself does not address.

No comments yet. Be the first to share your thoughts!