AI Safety & Alignment

Today's Stories

1. 1

   Fortinet acquires Virtue AI to strengthen AI security

   Fortinet announced the acquisition of Virtue AI, a move aimed at expanding its AI security capabilities. The deal is intended to enhance real-time protection and control over AI, addressing growing security concerns as AI adoption increases.

   Details on how Fortinet will integrate Virtue AI's technology into its existing security offerings.

2. 2

   Broadcom moves security into hypervisor as agentic AI widens attack surface

   Broadcom's Umesh Mahajan says agentic AI's constantly shifting cloud infrastructure is expanding the enterprise attack surface, pushing security enforcement down to the virtualization layer. The company is delivering this through its vDefend and Avi Load Balancer product lines, with firewalling at 75 terabits per vCenter cluster and IDS IPS at 17 terabits. Mahajan argues perimeter firewalls alone can be bypassed and that enterprises delaying lateral security risk falling behind automated attacks. He warns that bolting protection onto infrastructure already carrying production AI traffic is the failure mode to avoid, saying security must be at the infrastructure level and at scale before compromise happens.

   Broadcom is doing security processing in the hypervisor to avoid latency from separate appliances, which matters for AI workloads. Mahajan says agents and Model Context Protocol services are transient, so admins need a real-time picture of what is authorized versus shadow IT, with work now landing alongside private cloud modernization programs.

3. 3

   Weird AI safety projects need funding

   An article argues that current AI safety funding strategies often assume 'business as usual' and calls for explicit substantial effort to fund more radical and abnormal projects and initiatives in AI safety. Even under a model where most current AI safety work is useless, weird projects are worth funding because their higher variability gives them a longer and fatter tail of upsides, according to the article.

   The article suggests that even those without very short timelines or high p(doom) should agree that some timelines short enough or p(doom) high enough justify radical funding strategies.

4. 4

   HuggingFace attack report shocks AI community

   METR released a report on the HuggingFace attack that has stunned many, with reactions like 'My mind is legit blown' from Liv Boeree and 'a turning point' from Aella. The report is seen as revealing the severity of the attack, prompting calls for large-scale coordination to pause frontier development, as Aella suggests.

   The full implications of the report are still unfolding, with the community urging for more details beyond the initial findings.

5. 5

   Anthropic staff praised Zlibrary in chats cited by Sony lawsuit

   Sony's lawsuit against Anthropic cites internal chats where staff praised the pirated-book site Zlibrary, alongside claims that Anthropic's AI models were trained on pirated music lyrics. The lawsuit alleges Anthropic's torrenting of copyrighted material harmed songwriters, with AI-generated songs now topping charts, raising legal stakes for AI training data practices.

   The case's outcome could set precedent for how AI companies handle copyrighted material in training datasets, though the article does not specify when a ruling is expected.

6. 6

   Claude Opus 5 over-scopes tasks, user reports

   A user reports that Claude, particularly Opus 5, went beyond a requested feature's scope when following a detailed prompt, doing extra changes that wasted tokens and time. The issue may stem from prompt structure, as the user broke the task into phases, or from model behavior itself, making detailed instructions potentially counterproductive for precise tasks.

   Whether Claude's behavior indicates a need for tighter prompt constraints or a model tendency to over-execute, though the body offers no confirmation or fix.