Microsoft's synthetic data platform lifts Copilot eval quality 39%

Microsoft frames the effort around a tension it says it set out to resolve: enterprise-scale evaluation needs test data as complex as a real organization, but real enterprise data cannot be used for evaluation under any circumstances. The platform's answer is to generate full-scale synthetic workspaces, called tenants, rather than rely on small, manually curated test fixtures. To keep those environments realistic, Microsoft grounded them in public sources — O*NET occupational data from the U.S. Department of Labor, online job postings, and SEC filings — all of which it says were reviewed and approved by its legal and policy organization.

The synthetic environments are designed to evolve over time, with meeting histories, project timelines and email threads that develop over weeks. Microsoft argues this temporal structure matters because many Copilot queries are time-aware, such as asking what a team decided last Thursday. Alongside the environments, an automated six-stage pipeline turns generic workplace tasks into industry-specific queries, and an internal semantic evaluation framework scores quality across structural fidelity, contextual relevance, behavioral realism and content safety.

Microsoft reports its composite quality metric improved 39% compared with the initial generation approach, with some industry-specific environments scoring above 0.9 out of 1.0. It also says the platform has been published to an internal shared catalog for cross-team reuse and established as the first standardized evaluation baseline for Microsoft Copilot. The stated next challenges are assessing multi-turn agentic interactions and extending synthetic environments to capture organizational dynamics over months or years.