Microsoft Scales AI Agents to 80,000 Enterprises; Focus Shifts to Production Engineering

Microsoft shared production lessons from running AI agents at enterprise scale, revealing that more than 80,000 enterprises now build on Microsoft Foundry. The scale is significant: Microsoft 365 Copilot alone serves over 20 million users, with monthly active usage of first-party agents growing 6x year-to-date. Marco Casalaina, VP of Products for Microsoft Core AI, walked through what his team has learned from running these systems in production and the engineering challenges that distinguish prototype from production-ready agents.

The fundamental shift is from chatbots to agents. Chatbots answer questions; agents take real actions on behalf of the user—booking meetings, running analyses, sending emails, filing tickets. This shift changes the stakes. A chatbot returning a wrong answer is a bad experience. An agent taking a wrong action is a business incident. Enterprises are also adopting voice as a front-end via tools like Foundry's Voice Live, which lets teams convert an existing text agent to a voice agent without rebuilding it.

The gap between prototype and production is invisible in lab testing but fatal in deployment. A prototype can be coded in an afternoon with impressive demos and test prompts. Production reveals problems that never surfaced in the prototype: documents the agent depends on go stale, real users ask unanticipated questions, new edge cases emerge, model updates shift behavior subtly without notice, agents lack identity controls and audit trails, guardrails fail, and observability is missing. Casalaina's single biggest lesson: "the harness matters as much as the model." The harness is everything around the model—the runtime, tools, context retrieval, identity layer, guardrails, evaluators, and deployment pipeline. Models change every few weeks and cannot be treated like database versions; each model has different properties requiring the harness to re-tune and re-evaluate. When Anthropic released Claude Opus 4.8, GitHub Copilot CLI had to re-tune and re-evaluate before shipping.

The production harness has five layers. At the bottom is the inference layer, a single interface to reach models. Foundry supports more than 11,000 swappable models from OpenAI, Anthropic, xAI, DeepSeek, and Microsoft's MAI family. Above that is the agent runtime, which orchestrates the loop, handles tool calls, and manages conversation state. Well-built agents route only reasoning steps to the LLM; database lookups and extraction models run as ordinary code, faster and cheaper. The observability and governance layer provides fleet visibility, health scoring, token usage, latency metrics, drift detection, and cross-project rollups so platform teams can govern at scale. Foundry Control Plane provides this view and routes telemetry into Azure Monitor and Application Insights. The identity layer extends Entra, Microsoft's enterprise identity platform, to treat agents as a new class of principal with their own role assignments and audit trails, bounding misbehaving agents the same way misbehaving employees are bounded.

The context layer is the hardest and one Microsoft is "reasonably passionate about getting right." Enterprise context lives everywhere—unstructured documents in SharePoint and wikis, structured tables in data warehouses and OneLake, productivity apps like Outlook, Teams, and Word. Classic retrieval-augmented generation (RAG) never handled this. One-shot RAG takes a user's question, embeds it, searches a single index, returns top-k results, and passes them to the model. It works for simple questions against clean, small corpora. It breaks when the question is ambiguous, the corpus is heterogeneous, the answer requires combining sources, or the first retrieval comes back empty. The agent has no way to recover from a bad retrieval. The fix is to treat retrieval as an agentic loop: plan the query, try a source, evaluate the result, try a different source if the first came back empty, and combine what you find. Microsoft ships the context layer as four headless services collectively called Microsoft IQ: Foundry IQ handles unstructured data, Fabric IQ handles structured data, Web IQ handles real-time web retrieval, and Work IQ handles Microsoft 365 productivity surfaces (email, calendar, documents, Teams). Each IQ wraps retrieval in an agentic loop and can be called by agents through MCP. Work IQ adds a second engineering idea—giving an agent an identity and a place to act—so the agent can not only find information but take work on behalf of the user.

Microsoft's disclosure that 80,000 enterprises build on Foundry reflects a fundamental shift in how the industry thinks about AI deployment. The move from question-answering chatbots to action-taking agents reframes the engineering problem: a wrong answer is poor UX, but a wrong action is a compliance and business risk. This is why Marco Casalaina's core insight—that "the harness matters as much as the model"—carries weight. A prototype agent can be vibe-coded in an afternoon because it operates in a controlled lab setting with anticipated inputs and curated test data. The moment an agent enters production, it encounters real users asking ambiguous questions, documents that go stale, model updates that shift behavior, and the absence of audit trails when something goes wrong.

The production harness Microsoft describes is five layers deep: an inference layer that keeps models swappable (critical given the explosion of options from OpenAI, Anthropic, xAI, DeepSeek, and others), an agent runtime that orchestrates tool calls and state, observability and governance that provide fleet-wide visibility and cost control, an identity layer extending Entra to treat agents as enterprise principals, and a context layer that iterates across heterogeneous data sources. Each layer exists because its absence creates an invisible regression or uncontrolled risk. The context layer is particularly instructive: classic retrieval-augmented generation (RAG) was never designed to handle the reality that enterprise context "lives everywhere"—in unstructured SharePoint documents, structured data warehouses, email, Teams, and calendars. One-shot retrieval fails when the question is ambiguous, the corpus is heterogeneous, or the first search comes back empty. Treating retrieval as an agentic loop that plans, tries sources, evaluates results, and retries is the engineering idea that bridges this gap.

No comments yet. Be the first to share your thoughts!