AI agents outpace human oversight at scale

What happened

At Fortune's Brainstorm Tech conference, executives from DraftKings, Salesforce, Indeed, and Xero discussed the challenge of managing AI in high-stakes operations. DraftKings' CTO noted his company already handles trillions of transactions with distributed workloads, and introducing AI agents that communicate with each other directly makes human oversight impossible. Salesforce's chief customer and commercial officer emphasized that errors in critical domains like health care have life-or-death consequences, unlike retail mistakes.

Why it matters

As organizations scale AI from small experiments to broad deployments, they face a fundamental tension: the volume and complexity of AI-driven operations can exceed what any human team can oversee. This forces companies to rethink how they ensure AI behaves as intended. Indeed's CISO highlighted that his job listings service operates at massive scale—used by 645 million job seekers and 3.5 million employers—making continuous human review infeasible.

What to watch

Panelists identified governance frameworks as the key safeguard. Xero's chief product and technology officer noted that AI can be deployed more freely when outcomes are deterministic and easily measured, but becomes harder to automate when judgment is required. Leaders emphasized that governance must scale, and that testing processes against desired outcomes is critical, though the body does not specify how companies should implement these frameworks in practice.