California's AI transparency law found state agencies secretly used 6 high-risk systems

In 2023, California Governor Gavin Newsom signed Assembly Bill 302, a transparency law requiring the state's Department of Technology to inventory all high-risk automated decision systems used by any state agency and publish yearly reports. The law's definition of "high risk" covered systems that "assist or replace human discretionary decisions that have a legal or similarly significant effect," including those affecting housing, education, employment, credit, health care, and criminal justice—in other words, decisions that can shape whether vulnerable Californians receive essential benefits.

When the Department of Technology released its first report in 2025, it announced a surprising finding: California's state government uses no high-risk AI systems. To verify this claim, researchers at the UCLA Center on Resilience and Digital Justice and Georgetown Law's Center on Privacy and Technology filed a public records request and received a single spreadsheet with two columns—one labeled "Is ADS used" and one containing the word "no" for every agency listed. There was no evidence of any further investigation by the technology department into what systems agencies actually deployed.

Earlier this year, the Department of Technology finally interviewed some state agencies, and agencies that had previously claimed they used no high-risk systems suddenly reversed course and disclosed that they had been using automated systems all along. The discrepancy exposed a critical weakness in AB 302: the law depends entirely on agencies to self-report their use of AI, with no independent verification process and no penalties for false reporting. Moreover, the law fails to specify who decides whether a system qualifies as "high risk"—that determination appears to rest with the agencies themselves.

Known examples of automated systems that went unreported in the initial 2025 report illustrate the stakes. The Uniformity Assessment System, deployed by California's health and human services agencies, has led to reductions in In-Home Supportive Services for disabled Californians. The Risk Segmentation, Stratification, and Tier model is used by Medi-Cal to predict the "risk" and "service underutilization" of recipients—effectively automating decisions about who receives medical care. Neither system appeared in the Department of Technology's first report, suggesting that state agencies had defined them as falling outside the "high-risk" category. The authors argue this episode demonstrates that transparency laws without enforcement mechanisms, verification processes, or meaningful penalties fail to expose government use of automated systems and may instead legitimize and entrench their use by building official processes around them. Similar failures have occurred in other jurisdictions: New York's Public Oversight of Surveillance Technology Act, intended to constrain NYPD surveillance, was undermined when the police department exploited legal loopholes and used vague language to describe its tools, including its surveillance robot dogs. The authors conclude that if California intends to protect residents from harmful government AI, lawmakers must move beyond transparency requirements toward substantive restrictions on when and how state agencies can deploy automated decision systems.

Assembly Bill 302 was designed to bring transparency to California's use of automated decision systems in government—systems that can determine whether people receive cash assistance, housing, or medical care. The law's first test, however, revealed a fundamental flaw: it placed the burden of disclosure entirely on state agencies themselves, with no independent verification mechanism and no consequences for agencies that fail to report or mischaracterize their tools. When the Department of Technology's initial 2025 report showed zero high-risk systems, it appeared either the state had remarkably restrained itself or the law had been designed with no teeth. A public records request and journalist interviews proved the latter. Systems known to affect thousands of Californians—particularly the Uniformity Assessment System and the Medi-Cal risk model—had not been disclosed because agencies were able to define the threshold of what counts as "high risk" on their own terms.

This pattern is not unique to California. New York's Public Oversight of Surveillance Technology Act, meant to constrain the NYPD's use of surveillance tools, similarly failed when police exploited legal loopholes to avoid scrutiny and deployed vague language to describe their own capabilities. Community control bills in other jurisdictions have encountered the same problem: when the regulated entity is also responsible for deciding what to report, transparency becomes a tool for legitimating the technology rather than constraining it. The deeper problem, the authors argue, is that transparency-based regulation accepts the premise that government agencies should be using automated systems for life-altering decisions and then builds bureaucratic processes around that use—effectively normalizing and entrenching the practice rather than questioning it.