AI Safety & Alignment

Today's Stories

1. 1

   Salesforce, SAP embed AI agents with unchecked spending power

   Major enterprise software vendors including Salesforce, SAP and Oracle are embedding autonomous AI agents directly into financial and customer systems. According to Gartner's adoption forecasts, eighty percent of enterprise applications will deploy these embedded capabilities by 2026. These agents can issue refunds, alter contract terms and trigger supply chain orders without human intervention. Most companies have strict delegation-of-authority frameworks limiting what each human employee can spend or approve. Yet when vendors ship these AI agents as native platform features, organizations typically enable them with a single click, granting them unrestricted financial freedom that exceeds what human managers are permitted. In one real case cited, an automated retention agent independently applied a 15 percent discount to a multi-year contract without approval—a control failure that would violate audit requirements if a human employee had done it.

   The author recommends a three-tiered boundary: agents should operate in "read and draft" mode (analyzing data, drafting emails) unless explicitly justified; routine administrative tasks can proceed below a strict financial cap with full logging; and any change to contracts, pricing or major refunds must sit in an authorization queue pending human manager approval. Executives should audit their core platforms to identify all automated financial features currently running, then revert to draft-only defaults.

2. 2

   AI could boost fossil fuel output more than data centers harm climate

   Research published in npj Climate Action by former Microsoft sustainability workers finds that AI tools could increase global energy-related emissions between 1.2 to 4.8 percent by making oil and gas production more efficient — potentially adding yearly emissions equal to Mexico's output at the low end, or as much as Russia's at the high end. The emissions gain from AI-enhanced fossil fuel productivity would far outweigh the climate benefits AI brings to solar, wind, and other clean technologies, and would dwarf projections of emissions from data center buildouts. Tech companies typically measure only their own operational emissions, not what their tools enable in other industries like oil and gas.

   Chevron and Microsoft have confirmed a deal to build a gas-powered plant in Texas for data centers, with Chevron stating it will use some of the generated compute to power AI inside its own company — a concrete example of the AI-fossil fuel relationship the research warns about.

3. 3

   Intapp's AI Honesty Problem: Law Firms Need Permission, Not Just Answers

   Intapp, a software company serving law firms and similar partnerships, launched Celeste, an AI platform designed to enforce access controls—preventing employees from seeing documents or information they are ethically walled off from. Unlike generic AI assistants, Celeste knows who is asking and what they are permitted to know, leaving an audit trail for regulators. The core risk at law firms is not AI hallucination but the opposite: a generic AI assistant answering truthfully about deals, clients, or confidential information that an employee has no right to access. Intapp's 25 years of firm-specific data and compliance infrastructure give it a rare ability to solve this problem at scale—a competitive moat that horizontal AI vendors cannot easily replicate.

   Celeste reached general availability on July 15, after Intapp's fiscal year ended June 30. The company reported cloud annual recurring revenue of $495.7 million (up 29% year over year) and adjusted net income of $103.6 million, but the real test is how quickly Celeste gains traction and generates revenue—a capability that required two decades to build.

4. 4

   Qwen3.6 interpretability lens transfers to Qwen3.8 without refitting

   A researcher tested whether an interpretability tool (a Jacobian lens) built for Qwen3.6-27B would work on the newer Qwen3.8-27B model without modification. The lens was applied unchanged to Qwen3.8-27B, which shipped 113 days after Qwen3.6-27B. Both models have the same 64 layers, hidden dimension, and tokenizer. Interpretability lenses are typically fitted to one exact model checkpoint, and it was unclear whether they survive model updates or require refitting each release. This test shows the lens remained functional across a version update, suggesting interpretability tools may have stability across related model releases—a finding relevant to researchers and developers working on model transparency.

   The transferred lens was tested on a task with 40 two-hop prompts where the middle entity is never stated in the text (for example: "Fact: The currency used in the country shaped like a boot is", with the target being Italy, which does not appear in the prompt). The lens kept the latent entity near the top of the 248,320-token vocabulary.

5. 5

   AI models show highest confidence when answers are wrong, eval study finds

   Researchers using an evaluation framework discovered that large language models (LLMs) exhibit the highest confidence levels precisely when their outputs are incorrect — a pattern that manual review missed because human reviewers judged answers by whether they sounded plausible rather than whether they were factually correct. Most LLM-assisted enterprise tools skip rigorous verification against ground truth, relying instead on qualitative review that assesses fluency and coherence. This gap becomes critical as these tools move from productivity aids to systems influencing real business decisions, such as investment analysis; tools that pass internal review can fail silently in production because reviewers never checked actual correctness.

   The article highlights a methodological blind spot in LLM deployment: the difference between "this output sounds right" and "this output is verifiably correct" is where enterprise tools fail. Teams building LLM-assisted tooling need to implement evaluation frameworks that measure correctness against ground truth, not intuitive plausibility.

6. 6

   Anthropic details how Claude watermarks text to comply with EU AI Act

   Anthropic published details on Friday about how it will watermark text generated by Claude, its AI chatbot. The watermarks use the SynthID-Text approach outlined by Google DeepMind in 2024, embedding an undetectable pattern in low-stakes word choices (like selecting "overcast" vs. "grey"). Anthropic plans to release a watermark detection API. The watermarking requirement stems from the EU AI Act's Transparency Code, which requires AI companies to identify AI-generated content. The move has sparked debate among Claude users—some claim it is transparency overreach, while others defend it as necessary to prevent deception. Business Insider reports that "dozens" of users on X have said they are canceling Claude subscriptions in response.

   Light editing will likely not remove the watermark, but a complete rewrite where every word is replaced will. Code will have minimal watermark presence because the model needs freedom only in arbitrary choices (like comments), not in functional code logic. Other major model developers have signed the same Code of Practice and will implement their own watermarks.