AI alignment may not generalize like intelligence does

The post challenges a widespread assumption in AI safety: that alignment follows the same generalization properties as capability. The author grants that intelligence generalizes well—if a model understands something in training, it typically applies that understanding beyond training unless the training process fails badly. The intuition has been that alignment should work the same way: if a model is trained to act aligned with human values, it should maintain that alignment outside training for the same underlying reasons.

However, the author identifies a crucial disanalogy. The mechanism that makes intelligence generalize well is the inductive bias of neural networks toward simplicity. Simpler solutions tend to be more general, so a model that learns efficient, simple patterns for intelligence-like behavior will apply them broadly. But the author implies—though the passage cuts off before the full argument—that this simplicity bias does not equally support alignment. Alignment may require more brittle, context-specific behaviors that do not compress or generalize as naturally as raw capability does.