How Microsoft Tests Copilot Without Touching Customer Data: Synthetic Companies

Microsoft Research says it now tests Microsoft Copilot against fully synthetic companies instead of customer data, in a blog post published October 30, 2026. The system, called the Industry Evaluation Data Generation Platform, builds fake pharmaceutical, capital markets and software companies, fills them with emails, meetings and documents, and generates the test questions automatically. Microsoft reports a 39% quality gain over its first approach and roughly $100,000 saved per evaluation dataset.

It is a company-reported result with no public dataset, so treat the numbers as Microsoft's claims. The method is still useful to anyone shipping an enterprise AI assistant, because it solves a problem every such team has: you cannot test on real customer data, and toy test sets do not look like real workplaces.

TL;DR: the platform in one table

Why does Copilot need synthetic companies at all?

Copilot answers questions about your meetings, your email and your files. To know whether it is right, someone must check its answers against the underlying data. The Microsoft post gives a simple example: a user asks Copilot to summarize a meeting and extract action items. Is the summary accurate, complete and grounded in the transcript?

You cannot run that test at scale on customer tenants. Microsoft frames the "central tension" as needing data with the "full complexity of real organizations" (reporting structures, project histories, communication patterns, domain workflows) while being unable to touch the real thing. Small hand-built test fixtures miss that complexity, and generic benchmarks miss domain-specific failures.

This sits in a broader shift. Labs and tool vendors are discovering that standard public benchmarks do not predict how a model behaves inside a specific workplace; we have seen the same theme in FDE-Gym, a benchmark for forward-deployed engineers, where frontier models struggled on realistic customer-deployment tasks, and in the continual-learning exam for agents.

A machine producing green tiles beside a few gray ones, representing synthetic enterprise test data for Copilot evaluation

How are the synthetic enterprises built?

Microsoft describes three pillars: synthetic enterprise environments, automated query generation, and structured quality frameworks.

Grounding in public data

Each synthetic tenant is an isolated organization with its own users, data and configuration. To keep them realistic, Microsoft used three public sources:

• O*NET, the US Department of Labor's occupational database, to make roles and responsibilities realistic and to calibrate the mix and frequency of work activities for each role, including industry workflows such as clinical trial reviews or portfolio updates.
• Online job postings for realistic titles, reporting hierarchies and departmental makeup.
• SEC filings for organizational scale, business unit structure and financial context.

Microsoft says all of these are public and were reviewed and approved by its legal and policy organization.

Time-evolving organizations

The environments are not static snapshots. They simulate meeting histories, project timelines, email threads that develop over weeks, and organizational changes. That matters because many Copilot queries are time-aware, such as "What did the team decide last Thursday?" or "What's changed since the last quarterly review?" Those only test anything if the data has a genuine timeline.

Three industries

How do the test questions get made?

Fake data is the substrate; you still need realistic questions. Writing thousands of grounded, industry-specific queries by hand is slow and costly, so Microsoft built a six-stage pipeline. The post summarizes it as a flow that:

1. starts from a large library of common workplace tasks,
2. narrows them to what is relevant for a given industry,
3. adapts them with domain-specific language and context,
4. uses AI to generate queries grounded in the synthetic environment's data,
5. rewrites them to sound more natural and human, and
6. validates each query for accuracy and uniqueness.

The result, in Microsoft's words, is a set of "high-quality, lifelike queries" for scalable evaluation. Microsoft does not publish the stage-by-stage prompts or the size of the final query sets in this post.

How does Microsoft judge the synthetic data itself?

A synthetic benchmark is only as good as its realism, so Microsoft scores the environments with an internal semantic evaluation framework across four dimensions:

The headline result is a 39% improvement in the composite quality metric versus the initial generation approach, with some industry environments above 0.9 out of 1.0. Two cautions apply. First, this is Microsoft's own metric, so the score measures similarity to its own definition of realism. Second, a 39% gain is relative to the team's first attempt, not to real data or to any public baseline. It shows iteration worked; it does not show the synthetic tenants match real ones.

What did automation change?

Microsoft lists four problems with manual dataset creation: reproducibility, scale (too many industries and configurations to build by hand), iteration speed (days per dataset) and cross-team consistency. Automation turned a multi-day manual process into automated runs, which Microsoft estimates saves about $100,000 per evaluation dataset and $300,000 for the project. Those are the authors' estimates, and the post does not show the calculation.

The more practical claim: any team can regenerate a dataset from the same configuration and get identical results. For regression testing across Copilot releases, that reproducibility is arguably worth more than the savings, because you can compare a new build against the same baseline.

What about privacy?

Microsoft calls "no real customer data used at any stage" a non-negotiable principle. The safeguards it lists are privacy-preserving aggregation where real-world statistical distributions are referenced, legal review of every public source, and data provenance tracking. Because the data is synthetic and safety-screened, it says teams can reuse it "without privacy review overhead," which is exactly why it expects broad adoption inside the company.

Note the limit of this claim. "No customer data in the pipeline" is a statement about inputs. It does not address whether Copilot's real-world performance matches its synthetic-world performance, which is the question customers actually care about.

What is verified and what is not?

The last row is the notable gap. The post describes how Microsoft measures, not how Copilot scored. There are no accuracy numbers, so readers cannot yet tell whether the platform found failures, what kinds, or how fast they were fixed.

What does this mean for builders?

If you ship an AI assistant that touches private work data, the playbook transfers:

1. Build a fake organization, not a fake file. Include people, roles, history and change over time, because time-aware questions are where assistants fail.
2. Ground it in public statistics (job taxonomies, filings) so scale and role mix are plausible without exposing anyone.
3. Generate queries from templates, then rewrite and validate. The staged pipeline avoids both repetitive and unnatural test questions.
4. Score your synthetic data. Treat realism as something you measure, as Microsoft does with structural, contextual, behavioral and safety checks.
5. Make it reproducible. Same config, same dataset, so regressions are visible.
6. Move to agent evaluation next. Microsoft says it is tackling multi-turn agentic interactions and AI that takes actions, not just generates text. For action-taking agents, safety checks matter as much as accuracy; see APEX-Harm, an agent safety benchmark, and consider a runtime guard such as AgentBeam from the explainx.ai team.

For a related technique, see how another startup generates training data from nothing in Adaption's Invent API for zero-seed synthetic data.

Where this fits in Microsoft's Copilot push

Copilot is moving from question answering to long-running agents, which raises the evaluation bar. Microsoft recently put out Copilot Harness and Foundry Agent Service updates, and its coding models keep expanding, as in MAI-Code-1.1 Flash running locally on Windows. Reports about Copilot's economics and model sourcing also keep surfacing, such as the claimed-versus-verified look at Microsoft's reported Anthropic spend. A standard internal evaluation baseline is the unglamorous infrastructure that lets a company ship all of that with some confidence.

What to watch next

• Published results. Whether Microsoft shares Copilot scores or failure categories from these environments.
• Open release. Whether the synthetic tenants or the generation code ever become public, which would let outsiders replicate the work.
• Agentic extensions. Microsoft names multi-turn agent interactions and organizations evolving over months or years as next steps.
• Independent checks. Third-party comparisons of synthetic-environment scores against real-world user outcomes.

Related reading

• FDE-Gym: forward-deployed engineer benchmark
• Continual-learning exam for agents
• APEX-Harm agent safety benchmark
• Adaption Invent API for synthetic training data
• Copilot Harness and Foundry Agent Service
• MAI-Code-1.1 Flash on Windows

Figures and descriptions reflect Microsoft Research's October 30, 2026 post and may change as more detail is published.