FDA, CDC AI use surges 148%, 87% — raises questions on healthcare oversight

Healthcare agencies' AI adoption has accelerated dramatically, driven by promises of efficiency and cost savings. Yet the scope of deployment—spanning eligibility determination, preauthorization, triage, and diagnostics—has outpaced the development of clear governance standards. The body of documented harms remains small relative to total AI use, but recent high-stakes failures illustrate that even rare errors can harm patients and violate rights.

The progression from rule-based algorithms (simple, auditable, legally challengeable) to machine-learning and generative AI systems (opaque, complex, error-prone) compounds the governance challenge. Early rule-based systems, like those used to allocate monoclonal antibodies during COVID-19, allowed transparent audit and legal accountability. But more advanced systems obscure their reasoning, making it harder to detect errors, trace their source, or hold decision-makers accountable. The UnitedHealth Group case—with its alleged 90 percent error rate in preauthorization denials—and the multi-state Medicaid eligibility failures demonstrate that automated decision-making at scale can inflict widespread harm before errors surface.

Existing federal frameworks require "high-impact" AI assessments that evaluate data quality, model fitness, privacy impacts, and civil rights implications. However, these frameworks assume AI deployment will proceed and focus on risk mitigation rather than asking the threshold question: Should AI be used for this decision at all? The article argues that agencies lack a principled method for determining when human judgment is non-negotiable, and what the nature of a governmental function—especially in healthcare—demands of its decision-making process.