U.S. healthcare agencies sharply increased AI deployment between 2024 and 2025—the FDA by 148%, the CDC by 87%—but the rapid adoption raises concerns about harm, especially where errors can affect patient access to care.
A federal court in March 2026 found that UnitedHealth Group's AI preauthorization system had alleged error rates of 90 percent, and defective eligibility software caused widespread improper Medicaid terminations.
Current federal oversight frameworks focus on mitigating risks rather than determining whether AI should be used in the first place, leaving agencies without clear guidance on when human judgment is required.
What happened
Between fiscal years 2024 and 2025, the FDA experienced a 148 percent surge in AI use, the CDC an 87 percent increase, the CMS a 78 percent rise, and the NIH a 51 percent jump, according to the HHS AI Use Case Inventory. Federal and state healthcare agencies are rapidly expanding AI adoption, citing efficiency, accuracy, and cost savings.
Why it matters
Healthcare AI decisions carry high stakes — errors can harm patients. Recent cases show real risks: UnitedHealth Group's AI preauthorization system had alleged error rates of 90 percent (nine of 10 appealed denials reversed, per a March 2026 federal court order), and defective eligibility software from Deloitte Consulting caused widespread improper Medicaid terminations across multiple states. Current federal frameworks require risk mitigation assessments but do not determine whether AI should be used at all, leaving agencies without guidance on when human judgment is mandatory.
What to watch
The article proposes a Five Factor Framework requiring agencies to examine who may be harmed, the nature of the decision itself, the technology's capabilities and limitations, the role of human oversight, and applicable legal constraints. Agencies using this framework for high-impact decisions should seek public input through notice and comment, town halls, and engagement with advocacy groups representing affected communities.
Ask the AI about this article →
Healthcare agencies' AI adoption has accelerated dramatically, driven by promises of efficiency and cost savings. Yet the scope of deployment—spanning eligibility determination, preauthorization, triage, and diagnostics—has outpaced the development of clear governance standards. The body of documented harms remains small relative to total AI use, but recent high-stakes failures illustrate that even rare errors can harm patients and violate rights.
The progression from rule-based algorithms (simple, auditable, legally challengeable) to machine-learning and generative AI systems (opaque, complex, error-prone) compounds the governance challenge. Early rule-based systems, like those used to allocate monoclonal antibodies during COVID-19, allowed transparent audit and legal accountability. But more advanced systems obscure their reasoning, making it harder to detect errors, trace their source, or hold decision-makers accountable. The UnitedHealth Group case—with its alleged 90 percent error rate in preauthorization denials—and the multi-state Medicaid eligibility failures demonstrate that automated decision-making at scale can inflict widespread harm before errors surface.
Existing federal frameworks require "high-impact" AI assessments that evaluate data quality, model fitness, privacy impacts, and civil rights implications. However, these frameworks assume AI deployment will proceed and focus on risk mitigation rather than asking the threshold question: Should AI be used for this decision at all? The article argues that agencies lack a principled method for determining when human judgment is non-negotiable, and what the nature of a governmental function—especially in healthcare—demands of its decision-making process.
For example, today's edition would include:
AI-summarized, only the topics you pick — one digest a day via Email, Slack, or Discord.
Free · takes 30 seconds · unsubscribe anytimeWhat is AIToday? →
Ask AI anything about this article. Q&As are published on this page for other readers too.
A UK study by UK AI Security Institute and Limbic AI surveyed 6,474 British adults
Broadcom's Clayton Donley says companies are doing mission-critical work with AI agents quickly, but without t…
Bank of England governor Andrew Bailey warned that advanced AI poses risks to financial infrastructure in a le…
As AI agents perform real business tasks, 'Agentic Identity' (giving each AI a unique employee-like ID) and 'D…
Andrew Bailey, head of the world's financial stability watchdog, warned in a letter to G20 finance ministers a…
The European Union is expanding regulation of ChatGPT and will mandate protections for minors
