AI agent safety lacks data governance framework, experts warn

The rapid advancement in AI agent frameworks—particularly MCP (Model Context Protocol) and standardized function calling—has dramatically lowered the barrier to building agents that can interact directly with databases and APIs. Developers can now deploy such systems in roughly 20 minutes. However, this speed has outpaced the development of corresponding safety infrastructure.

The root issue, as described in the discussion, is that data governance for agents has not yet emerged as a distinct architectural layer. Instead, the industry relies on three approaches that each have critical gaps. System prompts are inherently unreliable because large language models (AI systems that generate text based on learned patterns) do not consistently follow complex instructions, especially when those instructions span multiple steps. Read-only database access prevents some disaster scenarios—such as accidental data deletion—but does nothing to stop agents from querying data they should not see or from generating fabricated database results that sound plausible. The third approach, "hope for the best," is explicitly acknowledged as not a strategy.

This asymmetry between capability and safety is the core concern. As agents become easier to build and deploy, the absence of a formal data governance framework means that risk—whether from unauthorized data access, expensive compute errors, or confidently delivered hallucinations—grows without corresponding countermeasures.