AI Safety & Alignment

Principais notícias

1. 1

   On “Model Organisms”

   This post was written while working for Arcadia Impact's Alignment Team (and grew out of an internal talk I gave) but is my own opinion and not theirs. I am grateful for feedback from Daniel Tan and the rest of the team. This post was originally going to be more heavily about “model organisms” in AI safety research. But Francis Rhys Ward already wrote an excellent taxonomy which mostly covers that. So this is mostly about the history of the terms we're using, and about biology.[1] TL;DR what are you studying? Are you studying a production language model in order to infer things about how language models behave in general? Are you studying a model with a specific intervention to prove that intervention’s effects? Or are you studying a model with a specific property, in order to make inferences about that property in other language models? Model Organisms in Biology When a biologist uses the term model organism, they’re typically referring to a certain species, like Mus musculus, the lab

2. 2

   The distillation double bind: Distilling misaligned models either transfers misalignment or it doesn't

   Suppose we have a dangerous misaligned AI that can fool alignment audits, and distill it into a student model. Two things can happen: Misalignment doesn’t transfer to the student. If so, we get a fairly capable benign model, which we can use to perform tasks that we wouldn’t want a misaligned AI to perform. Misalignment transfers to the student. The student might also be worse than the teacher at hiding its misalignment (e.g., because it is less capable). If so, auditing the distilled model might give us indirect evidence of the teacher’s misalignment. In a previous post we discussed the second possibility and proposed distillation for incrimination techniques: distillation methods that we hoped would transfer misalignment without transferring the ability to fool audits. In this post, we discuss the first possibility, and propose distillation for capabilities techniques: distillation methods that we hope will transfer capabilities without transferring misalignment. Thanks to Carlo Le

3. 3

   Your Model Organisms Might Be Fried

   Context: We are the ‘model motivations’ team at Arcadia Alignment. We aim to build a science of ‘model intentions’, unifying insights from personas and other empirical evidence. In this post, we’ll outline the need for much better model organisms and how we might get there. The case for building more natural model organisms for alignment research Model organisms are how we study alignment-relevant pathologies (such as secret loyalties, reward hacking, and sandbagging) and are used as a testbed for alignment auditing and interpretability methods. This makes their usefulness depend on whether they stay a realistic proxy for the systems we care about. However, when we deliberately induce a pathology or a target behavior, we may also unknowingly damage the model in unrelated ways. The organism may exhibit the pathology but become less coherent, less capable, and less representative of plausible deployment models. A helpful mental image here is of Spongebob learning to become an excellent w

4. 4

   Effective Altruism will be unbundled

   From the end of high school to after my sophomore year of college, I considered myself an effective altruist. I was on the board of my college EA club, ran an EA intro fellowship, and went to EA retreats. I was vegetarian, regularly donated to GiveWell, and generally tried to proselytize EA ideas. I was never fully convinced to pursue a career as an AI safety researcher or in animal welfare, but I found the ideas around agency, counterfactual impact, and a life structured around a single coherent philosophical vision compelling. If I had to attribute my exit from EA to a single event, it would be reading Atlas Shrugged by Ayn Rand. For an author who has written an essay provocatively titled "The Virtue of Selfishness" and is known for relentlessly bashing altruism, one might expect that Rand's philosophical ideal is entirely disjoint from EA and that I had merely been turned away from altruism altogether. Instead, it clarified to me that EA constitutes a bundle of distinct belief syste

5. 5

   Google DeepMind unveils plan to protect itself from its own rogue AI agents

   For years, AI safety research focused on 'alignment.' The Google DeepMind roadmap assumes some AI agents may go rogue, and focuses on monitoring and access control.