AI安全性・アラインメント

主要ニュース

1. 1

   Your Model Organisms Might Be Fried

   Context: We are the ‘model motivations’ team at Arcadia Alignment. We aim to build a science of ‘model intentions’, unifying insights from personas and other empirical evidence. In this post, we’ll outline the need for much better model organisms and how we might get there. The case for building more natural model organisms for alignment research Model organisms are how we study alignment-relevant pathologies (such as secret loyalties, reward hacking, and sandbagging) and are used as a testbed for alignment auditing and interpretability methods. This makes their usefulness depend on whether they stay a realistic proxy for the systems we care about. However, when we deliberately induce a pathology or a target behavior, we may also unknowingly damage the model in unrelated ways. The organism may exhibit the pathology but become less coherent, less capable, and less representative of plausible deployment models. A helpful mental image here is of Spongebob learning to become an excellent w

2. 2

   Effective Altruism will be unbundled

   From the end of high school to after my sophomore year of college, I considered myself an effective altruist. I was on the board of my college EA club, ran an EA intro fellowship, and went to EA retreats. I was vegetarian, regularly donated to GiveWell, and generally tried to proselytize EA ideas. I was never fully convinced to pursue a career as an AI safety researcher or in animal welfare, but I found the ideas around agency, counterfactual impact, and a life structured around a single coherent philosophical vision compelling. If I had to attribute my exit from EA to a single event, it would be reading Atlas Shrugged by Ayn Rand. For an author who has written an essay provocatively titled "The Virtue of Selfishness" and is known for relentlessly bashing altruism, one might expect that Rand's philosophical ideal is entirely disjoint from EA and that I had merely been turned away from altruism altogether. Instead, it clarified to me that EA constitutes a bundle of distinct belief syste

3. 3

   Google DeepMind unveils plan to protect itself from its own rogue AI agents

   For years, AI safety research focused on 'alignment.' The Google DeepMind roadmap assumes some AI agents may go rogue, and focuses on monitoring and access control.

4. 4

   Rackspace Technology (RXT) Is Up 42.5% After Cost-Cutting To Fund AMD-Powered AI Expansion

   Earlier this month, Rackspace Technology announced a phased agreement with AMD to deploy 30 MW of AMD AI compute across its global data centers through 2028, while approving a workforce realignment that will cut about 15% of its global staff and incur an estimated US$14.00 million to US$19.00 million in one-time charges during 2026. The company plans to redirect much of the expected US$75.00 million to US$85.00 million in annualized cost savings toward building governed enterprise AI...

5. 5

   Several frontier models are substantially prefill aware

   This blog post discusses work in a recently-published paper. However, this blogpost was primarily written by Parv Mahajan and Andy Wang, and several of the more speculative takes may not represent the all-things-considered view of the entire team. Link to paper: https://arxiv.org/abs/2606.12747 TL;DR: We provide more conceptual grounding and extend results in prefill awareness to low-stakes settings, and show that several frontier models show prefill awareness even under conservative elicitation. Further behavioral studies are pretty messy, and we encourage more work in this area. We encourage frontier lab safety teams to measure and mitigate prefill awareness in pre-deployment evaluations. Recently, UK AISI investigated prefill awareness - whether frontier language models can distinguish between tampered and untampered assistant-side content. Prefills are used in misalignment continuation, persona, introspection, and jailbreaking research. Additionally, several prefill-based evaluat