AI Safety & Alignment

Today's Stories

1. 1

   Redwood Research Publishes AI Futurism Reading List for Strategy Fellows

   Redwood Research compiled a curated reading list on AI futurism topics—including key dynamics in AI development, existential risk from AI, and risk mitigation approaches—that was used in a strategy fellowship run through Astra. The selection reflects the organization's opinionated views and focuses on topics the organization prioritizes. The reading list is designed to help people engage with conceptual frameworks and hypotheses regularly used by Redwood's team. Readers are encouraged to evaluate whether they agree with the theses presented and assess how well the predictions have held up against recent evidence.

   The reading list is divided into core and extended sections, with core readings organized into four weeks. The post invites suggestions for additions to the list.

2. 2

   AI safety research may take a backseat if LLMs prove easier to align

   An argument is made that if one believes LLMs are unusually well-suited to alignment compared to other AI development paths, it can be rational to prioritize capability research on LLMs over LLM safety research. The reasoning suggests that the choice between safety and capability work is not always straightforward—if the technical landscape favors LLMs as an alignment pathway, researchers working on capability advancement rather than safety could indirectly serve safety goals by steering AI development toward a more controllable regime.

   This framing challenges conventional wisdom that safety and capability research are always in tension, and may influence how AI researchers and funders allocate effort between the two domains.

3. 3

   CoreWeave Stock Falls 15.1% as Meta Explores Selling Its Own AI Compute

   CoreWeave's stock declined 15.1% following news that Meta Platforms, a major customer, is exploring whether to sell its own excess AI compute capacity. In late June 2026, CoreWeave launched ARIA, an AI research agent embedded in Weights & Biases that automates experiment analysis and continuous model improvement, and expanded its European footprint through renewable-powered co-location and storage partnerships. Meta's potential entry into AI cloud services directly threatens CoreWeave's core investment thesis — that its specialized AI cloud can convert a large contracted backlog into sustainable profit before competition and debt pressure erode margins. CoreWeave carries high-coupon debt, so customer concentration risk and pricing pressure from a major customer entering the same market could quickly magnify any setback in utilization or pricing.

   CoreWeave's narrative projects $26.9 billion（約4.3兆円） revenue and $1.6 billion（約2600億円） earnings by 2028, requiring 84.2% yearly revenue growth and about a $2.4 billion（約3800億円） earnings increase from −$824.7 million（約1300億円） today. The most bearish analysts already expected $32.4 billion（約5.2兆円） revenue and $2.5 billion（約4000億円） earnings by 2029 but still saw limited upside, underscoring how different investors weigh Meta's emerging competition against CoreWeave's debt load.

4. 4

   AI safety needs treaty enforcement, not a research lab, analyst argues

   A researcher argues that proposed international AI research collaborations modeled on CERN would likely prove ineffective for safety, and proposes instead an international treaty with enforcement mechanisms similar to the IAEA (the nuclear watchdog). The author contends that the main bottleneck in AI safety is not more research but political will and enforcement of existing best practices—a shift that reframes how governments should approach regulation. This suggests treaty-based governance with verification bodies may be more practical than building new research institutions.

   The author references successful precedents including the EU AI Act, the NPT/IAEA framework, and the Montreal Protocol, indicating a sequencing where red-line treaties come first, followed by an international verification body.

5. 5

   Model access for third-parties — it's a big deal!

   Model access for third-parties — it's a big deal!

6. 6

   Researcher proposes system-level defense against prompt injection attacks in AI agents

   A researcher introduced a middleware layer called Sentinel Gateway designed to prevent prompt injection attacks in tool-using AI systems. The approach enforces strict separation between trusted runtime commands (instruction channel) and untrusted external inputs like web data and APIs (data channel), requiring signed authorization tokens for all agent actions. Prompt injection has become a persistent failure mode in AI agents that interact with external data sources, and existing defenses focusing on input filtering or model alignment have struggled. This system-level strategy addresses the structural root of the problem by decoupling what the AI observes from what it is authorized to execute, potentially reducing risk for organizations deploying agentic workflows.

   The implementation uses FastAPI middleware, token-based authorization, a Streamlit interface for inspection, and audit logging of agent decisions—technical components that enable organizations to monitor and control AI agent behavior in production environments.