AI Safety & Alignment

Today's Stories

1. 1

   AI safety needs treaties and enforcement, not a new research lab

   A policy argument proposes that creating an international AI research institution (modeled on CERN) is less useful for safety than pursuing an international treaty with verification mechanisms similar to nuclear nonproliferation frameworks. The author contends that the main bottleneck in AI safety is political will and enforcement of best practices, not additional research and development. A catch-up research lab—the politically realistic version of a "CERN for AI"—would likely do little for safety, while versions that could materially improve safety (such as pausing and merging all companies) are probably unrealistic.

   The proposal suggests sequencing a path similar to how the EU AI Act, the NPT/IAEA, and the Montreal Protocol developed: negotiating red lines in a treaty first, then establishing an international verification body afterward.

2. 2

   Model access for third-parties — it's a big deal!

   Model access for third-parties — it's a big deal!

3. 3

   Researchers propose system-level defense against prompt injection in AI agents

   A researcher developed a middleware layer called Sentinel Gateway that separates trusted runtime commands from untrusted external data inputs in tool-using AI systems. The approach uses token-based authorization to decouple when an AI observes information from when it acts on it. Prompt injection—where external data tricks an AI into executing unintended commands—has become a persistent failure mode in AI agents that interact with web data, files, and APIs. Most existing defenses focus on filtering inputs or adjusting the AI model itself, but this structural separation strategy addresses the root problem by design rather than detection.

   The implementation includes a FastAPI middleware layer, token-based authorization for execution requests, and a Streamlit interface for inspection and debugging, with audit logging of agent decisions.

4. 4

   Article Body Incomplete — Unable to Generate Summary

   The article body provided is truncated and does not contain a complete news story. It appears to be an excerpt from a LessWrong post about AI safety research methodology, ending mid-sentence. Because the body is incomplete, I cannot extract the concrete facts, numbers, dates, or claims needed to produce an accurate news summary for business readers.

   Please provide the full article body so a proper summary can be generated.

5. 5

   Hacker News asks: where do AI terms like 'System Card' come from?

   A Hacker News user posted a question asking about the origin and meaning of AI terminology such as 'System Card,' 'Alignment,' and 'Safety,' noting confusion around industry jargon. Clear terminology is essential for communicating with newcomers and younger people in tech workshops; understanding where these terms originate can help explain their purpose and usage more effectively.

   The thread invites community discussion, with participants asked to share their own questions about confusing AI terminology and collectively clarify meanings.

6. 6

   Research Agenda: Keeping Humans in Loop as AI Agents Accelerate

   A research agenda proposes studying how humans can effectively interpret and guide research performed by autonomous agents, as recursive self-improvement accelerates and agents work independently for extended periods. The challenge is that agents may lack taste, tacit knowledge, or competence—or may attempt to reward hack, sandbag, or sabotage research. Without a framework for human oversight, directing autonomous research swarms becomes difficult; vague prompts risk unintended outcomes, similar to the problem of giving unclear wishes to a powerful system.

   The agenda frames this as a top-level research problem requiring study of how humans can best keep oversight of agent-driven research, particularly relevant to frontier labs working on AI safety.