AI Safety & Alignment

Today's Stories

1. 1

   Holden Karnofsky, a prominent AI safety researcher, publishes a list of ways AI safety efforts could backfire—acknowledging his own work might ultimately cause net harm.

   Karnofsky states he is 'not aware of a good list of downside risks for AI safety broadly' and has decided to make one. He identifies AI governance interventions as 'obviously high-variance,' noting that bad regulation can easily make things worse and many interventions could increase the risk of great power conflict. Karnofsky, who works in AI safety, says he thinks the field's impact is 'worse than 51/49'—meaning he genuinely lives with the possibility that his ultimate impact could be negative. This admission from an active researcher in the space suggests that even well-intentioned safety work carries real downside risk that warrants honest scrutiny.

   Karnofsky frames this as a personal list of risks he 'personally takes seriously,' not a comprehensive catalog. The acknowledgment that he may be 'prone to overestimate how robustly good' his actions are underscores the uncertainty inherent in safety-focused interventions.

2. 2

   Value investor Tobias Carlisle calls Adobe stock 'very compelling' despite AI uncertainty, pointing to a steep valuation discount and aggressive share buybacks.

   Adobe shares have fallen 44.24% year-to-date and are trading at a forward P/E of 8 and a PEG ratio of 0.53, prompting Carlisle to argue the stock is undervalued. In Q2 FY2026, Adobe posted record revenue of $6.62 billion（約1.1兆円） (up 13% year over year) and repurchased roughly 8.5 million shares for $2.111 billion（約3400億円） during the quarter. Adobe faces an open question about whether generative AI will ultimately disrupt its core editing tools or become a tailwind for the business. Carlisle frames the current discount as compensation for that uncertainty—if Adobe adapts or benefits from AI, investors are getting a favorable entry price. The company's 35.3% operating margin and 62.9% return on equity are under scrutiny as generative AI tools mature.

   Wall Street's consensus analyst price target is $282.27, compared with a current price near $195. Leadership changes are also unfolding, with CFO Dan Durn departing June 15, 2026, and CEO Shantanu Narayen announcing his transition after 18 years at the helm.

3. 3

   Anthropic faces questions about whether its founding commitment to AI safety will hold up if the company pursues a trillion-dollar IPO.

   Anthropic, an AI company built on a mission to develop safe artificial intelligence, is reportedly considering an initial public offering that could value it at around $1 trillion（約160兆円）. The company was founded by former OpenAI leaders and has positioned safety as central to its identity. Going public at such a large valuation would create pressure to prioritize shareholder returns and rapid growth. This sets up a potential tension between the company's original safety-focused mission and the financial incentives that come with being a publicly traded, trillion-dollar company.

   The outcome will signal whether Anthropic can maintain its founding principles once it has public shareholders demanding profitability and growth, or whether the IPO process forces a shift in priorities.

4. 4

   A governance researcher argues that the most impactful AI policy work happens invisibly inside government institutions, not in public statements and open letters.

   An article on LessWrong contends that a substantial portion of consequential AI governance work takes place within ministerial cabinets and international bodies, rather than in the visible public sphere of press coverage, statements, and open letters. The piece suggests the AI governance community may be overinvesting in public intellectual output while undervaluing the quieter work of people operating within national and international institutions. This invisible institutional work is described as some of the most impactful, yet it remains largely unseen by the broader community.

   The author raises hesitations about replicating a public advocacy model (ControlAI) in France, implying that the structure and visibility of governance efforts may need to match the actual locus of decision-making power in different institutional contexts.

5. 5

   AI researchers are proposing a debate-based approach to resolve the interpretive questions that arise when trying to verify whether AI models are safe—a new layer of difficulty in machine learning safety assurance.

   Researchers are framing AI safety assurance as a problem of answering interpretive questions about model behavior (e.g., whether a model is scheming or sandbagging). They propose using a debate protocol—relaxing strict adversarial robustness to allow defeasibility—as a way to investigate these questions empirically. Their "performative misalignment" work demonstrates one round of this debate conducted manually by human researchers. Safety cases and heuristic arguments for AI systems depend on understanding the model's internal motivations and mechanisms, but these interpretive questions are fundamentally difficult because models behave in non-human-like ways. This creates epistemic challenges: non-human motivations are hard to interpret, results are hard to assess, and it is unclear how findings generalize. The debate approach offers a structured way to investigate these questions without requiring absolute robustness—a practical necessity given how difficult the problem is.

   The research focuses on preventing a specific failure mode: non-converging empirical investigation, where attempts to answer interpretive questions about safety yield inconclusive or contradictory results. The debate protocol is intended to make this investigation more tractable and resolvable.

6. 6

   A LessWrong post examines how AI safety researchers use "model organisms"—a biology term—to study language models, exploring what researchers are actually testing and why the terminology matters.

   A researcher working with Arcadia Impact's Alignment Team published a post that traces the history and meaning of the term "model organisms" as it applies to AI safety research, building on earlier work by Francis Rhys Ward. The post asks three core questions about what researchers study: whether they are studying a production language model to infer general behavior, testing a specific intervention, or examining a model with a specific property to understand that property in other language models. The terminology and framing researchers use shapes how they design experiments and interpret results. By clarifying what "model organism" means in the AI context—borrowed from biology but applied differently—the post helps safety researchers be more precise about what they can and cannot conclude from their work.

   The post is primarily a conceptual contribution to how AI safety researchers think about their own methods, rather than reporting new experimental findings or timelines. Readers interested in AI safety methodology and the foundations of alignment research may find the taxonomy of study types useful for evaluating safety claims.