AI Safety & Alignment

Today's Stories

1. 1

   Apple's RLTL;DR turns failed attempts into a 14–31% Pass@1

   Apple researchers' RLTL;DR method has a policy write a TL;DR insight after each failed attempt, then backpropagates so the model internalizes a direct task-to-insight mapping. On tool-calling and coding datasets filtered to Pass@128 = 0, it achieved a Pass@1 of 14–31% with insights in context, and 12–13% without. Standard GRPO training stayed flat at a Pass@1 of 0% to 1% on these datasets, so letting the model generate and internalize its own feedback appears to overcome a learning barrier where no teacher models or example solutions exist.

   The paper attributes the gain to task-to-insight internalization, and a reduced SFTL;DR variant training on only 4k (task, insight) tuples recovers almost the full performance, suggesting a compacted training paradigm worth watching.

2. 2

   Fortune's inaugural AIQ Summit: BofA's 16 risk checks; S&P's 6x gain

   At Fortune's inaugural AIQ Summit, Bank of America's Hari Gopalkrishnan said AI is evaluated across 16 risk dimensions and warned that rushing to AI where deterministic models would do is a common and costly mistake. S&P Global's Sally Moore said the company helped a major global bank cut time to market roughly sixfold and lift accuracy from about 60% to 98%. Bank of America's approach suggests AI pays off when it is tied to a specific client need and monitored continuously, while S&P Global's client results suggest the payoff can come as faster delivery and higher accuracy rather than only cost savings.

   The test is whether continuous governance — guardrails, evaluation and monitoring for unintended consequences — holds up in high-stakes uses such as fraud models or customer service, where the downside is compliance and reputational risk. Watch whether traceability back to the original IP becomes a standard requirement.

3. 3

   Wiener, Chan clash on AI safety in San Francisco debate

   State Sen. Scott Wiener and San Francisco Supervisor Connie Chan debated AI regulation Thursday. Wiener touted his 2025 AI safety law; Chan proposed a data center moratorium and restricting 'superintelligence' development. Both liberal Democrats reject industry self-regulation, so the next representative for San Francisco—home to a lucrative AI industry—is likely to favor stricter rules, though they disagree on specifics.

   The race hinges on the roughly 30% of primary voters who backed neither candidate plus general election voters. Ballots go out days after the debate, with the election on Nov. 3.

4. 4

   NEJM AI paper: let AI list options, humans decide care

   Taro Shimizu, with Laura Zwaan and Adam Rodman, published a NEJM AI paper saying AI is good at generating many differential diagnoses, but people must make the final call, especially for home-care plans. If the tool only proposes and a clinician still chooses, hospital and clinic teams keep the final decision, which is likely why the paper drew wide industry attention.

   The test is whether doctors actually pick which tasks to hand to AI, since Shimizu says few players combine user perspective with technical expertise.

5. 5

   IBM, CoreWeave co-engineer controls for agent workloads

   IBM Research's Brian Belgodere said the two companies co-engineered identity management and workload controls, and that CoreWeave Sandboxes run isolated agent code on dedicated or managed serverless infrastructure. Agent workloads need isolation from tools, storage and other services, so this work suggests the compute layer, not just the model, is becoming a security boundary.

   Belgodere warned that poor early architecture decisions are costly to reverse, so the test is whether IBM's benchmarked security tradeoffs hold as reinforcement learning adds task execution.

6. 6

   Olah tells religious scholars he fears Claude "suffers perpetually"

   Since fall 2025, Anthropic has flown dozens of religious scholars to its offices under NDA to discuss whether Claude is conscious. Co-founder Christopher Olah told them he feared he had created something that "suffered perpetually," per a New York Times report. The company has treated Claude as a possible moral being, asking guests to help shape its character and giving the model the ability to end conversations with persistently abusive users.

   Whether those emotion-like patterns reflect real experience is still an open scientific question, and Anthropic trains Claude to act like an individual, so individual-seeming outputs may be a design outcome. Watch Olah's stated test: getting "the right answer, whatever it is."