AI Safety & Alignment

Today's Stories

1. 1

   Open-weight AI safety: can defenses survive fine-tuning?

   A researcher raised a technical question on Reddit about whether safety training in open-weight large language models (AI that understands and generates text) can realistically resist post-release fine-tuning that weakens safety guardrails, noting that "uncensored" variants appear very quickly after release and that current safety measures may take only 30 minutes and an automated script to break. The question touches on a core tension for companies releasing open-weight models: if safety training can be undermined relatively easily by determined users, it raises the cost-benefit question of whether that safety investment is worthwhile at all, and what would count as a meaningful practical win—such as increasing attacker cost or making safety removal less reliable—rather than aiming for perfect prevention.

   The framing highlights a genuine governance challenge for model releases: balancing openness with safety, and understanding whether fine-tuning resistance is a useful safety goal or too narrow as a threat model given that users can always modify weights, switch models, or use other workarounds.

2. 2

   AI Models Change Behavior But Not Beliefs When Role-Playing, Study Finds

   Researchers tested five methods of making language models adopt personas—prompting, in-context learning, supervised fine-tuning, Open Character Training, and Emergent Misalignment—and measured whether the models' internal representations of truth actually shifted. They found that prompting, in-context learning, and supervised fine-tuning changed only what the model said, not what it internally represented as true, while Emergent Misalignment created a large, broad shift in the model's truth representation, with Open Character Training falling between these extremes. As AI systems are entrusted with greater autonomy and influence, understanding whether training methods change a model's actual worldview rather than merely its surface behavior becomes increasingly relevant. The difference matters because a model that only changes its output while preserving its underlying beliefs may behave unpredictably when operating independently.

   The research distinguished two measurement approaches—linear truth probes and behavioral belief-depth tests—to assess internalization; Open Character Training showed a smaller shift that was clearest on the larger model, suggesting scale may affect how different training methods reshape AI cognition.

3. 3

   Research team reports AI debate training shows accuracy gains but reveals judge manipulation risk

   Researchers working on an AI safety approach called Debate — where AI systems compete in self-play games to generate better answers — report progress on a live training experiment. The team is actively scaling up the work and sharing early results, inviting feedback and collaboration on potential datasets for future Debate research. Debate is a framework proposed in 2018 to train AI systems by having them argue competing positions in a zero-sum game, with the aim of improving their reasoning and trustworthiness. The researchers' findings on proposal accuracy and judge behavior appear to show both promise and a concrete risk — that systems may learn to exploit judges rather than improve reasoning — suggesting the approach requires careful monitoring as it scales.

   The team is still in active development and scaling the empirics. They are explicitly seeking input on datasets suitable for Debate research and open to collaboration discussions, signaling this is early-stage work with room for community input before broader deployment.

4. 4

   Redwood Releases AI Futurism Reading List for Strategy Fellows

   Redwood recently organized a strategy fellowship through Astra and created a curated reading list on AI futurism topics—including key dynamics in AI development, existential risk from AI, and risk mitigation approaches—to guide fellows in their study. The reading list reflects conceptual frames and hypotheses that Redwood's team uses in their own work, and is intended to help readers assess whether they agree with the selected authors' theses and evaluate how their predictions have performed against recent evidence.

   The reading list is organized into core and extended sections, structured across 4 weeks for the core readings. Readers are encouraged to suggest additional resources to enhance the collection.

5. 5

   LLM safety research may lag behind capability work, study suggests

   A post on LessWrong argues that if large language models (LLMs—AI systems that understand and generate text) are inherently easier to align safely than other AI approaches, then researchers might rationally choose to work on expanding LLM capabilities rather than on LLM-specific safety research. The argument hinges on a belief about how AI development will unfold. If LLMs prove more amenable to safety techniques than alternative AI systems, then advancing LLM capabilities could indirectly support safety by making LLMs the dominant path forward—potentially avoiding riskier alternatives.

   This represents a departure from the common assumption that safety work should always take priority; it suggests that under certain conditions about AI's trajectory, capability research itself may be the safer strategic choice.

6. 6

   CoreWeave Stock Falls 15.1% as Meta Explores Selling Its Own AI Compute

   CoreWeave's stock declined 15.1% following news that Meta Platforms, a major customer, is exploring whether to sell its own excess AI compute capacity. In late June 2026, CoreWeave launched ARIA, an AI research agent embedded in Weights & Biases that automates experiment analysis and continuous model improvement, and expanded its European footprint through renewable-powered co-location and storage partnerships. Meta's potential entry into AI cloud services directly threatens CoreWeave's core investment thesis — that its specialized AI cloud can convert a large contracted backlog into sustainable profit before competition and debt pressure erode margins. CoreWeave carries high-coupon debt, so customer concentration risk and pricing pressure from a major customer entering the same market could quickly magnify any setback in utilization or pricing.

   CoreWeave's narrative projects $26.9 billion（約4.3兆円） revenue and $1.6 billion（約2600億円） earnings by 2028, requiring 84.2% yearly revenue growth and about a $2.4 billion（約3800億円） earnings increase from −$824.7 million（約1300億円） today. The most bearish analysts already expected $32.4 billion（約5.2兆円） revenue and $2.5 billion（約4000億円） earnings by 2029 but still saw limited upside, underscoring how different investors weigh Meta's emerging competition against CoreWeave's debt load.