The Test Behind Microsoft’s Claim That Its AI Beats Doctors

🔍 Read the full analysis: The Test Behind Microsoft’s Claim That Its AI Beats Doctors on ThorstenMeyerAI.com

Before you orderOffer from Amazon

Get the latest gadgets delivered free with Prime

• Fast, free delivery on millions of items
• Prime Video, Amazon Music and more included
• Member-only deals all year

Start your free Prime trial →Free trial for eligible customers · Cancel anytime

As an affiliate, we earn on qualifying purchases.Hide

Microsoft’s diagnostic AI system, MAI-DxO, scored 83.2% on a set of 304 difficult medical cases, compared with 36.1% for generalist physicians who could use textbooks and the internet, according to a peer-reviewed Science paper published October 22. The study tested historical case records in a text-based simulator, not the system’s ability to diagnose or treat people in a clinical setting.

The study also evaluated MAI-DxO on 325 emergency department cases, where it scored 80.3%, compared with 46.8% for physicians. Microsoft describes those figures as showing its system was 1.7 times as accurate as physicians on the emergency cases and more than twice as accurate on the complex cases when doctors had reference materials. These are results reported by the company’s research team, whose 15 authors are listed with Microsoft affiliations.

The benchmark, called the Sequential Diagnosis Benchmark, gives a doctor or AI a short case summary at the start. To obtain more information, the participant must request details or tests from a gatekeeper model holding the case file. The system is assessed on its final diagnosis and the tests it chooses, with estimated costs attached to those tests. MAI-DxO uses a language model to coordinate virtual physician agents that propose diagnoses, request information and challenge one another.

The paper’s reported accuracy is not solely a measure of the orchestration system. Microsoft’s chart shows GPT-5 alone scored 78.9% on the complex cases, close to MAI-DxO’s 83.2%. The company’s results indicate that the orchestration layer added about four percentage points and reduced estimated testing costs. In the study, costs were approximations based on billing codes and a 2023 U.S. hospital price list, not a record of spending in actual care.

What the Scores Mean for Care

The findings matter because they offer a test of AI’s ability to gather information and make diagnostic choices, rather than simply select an answer after receiving a complete case file. That process resembles one part of clinical reasoning: deciding what to ask or test next. The study also tries to account for the trade-off between additional information and its cost.

But the results do not show that MAI-DxO is more effective than doctors in routine practice. The comparison was against individual primary care physicians and hospital generalists working alone. It did not include specialists or medical teams, although the unusually difficult cases would often be referred to specialists or handled collaboratively. Nor did the benchmark measure patient outcomes, treatment decisions or how the system performs during an examination.

For hospitals and patients, the distinction is consequential. A high score on selected case records can support further research, but it cannot by itself establish safety, clinical benefit or readiness for use. The cost figures are estimates, and the benchmark does not include the full range of patients and explanations that clinicians encounter.

From Preprint to Science Paper

An earlier version of the research appeared on arXiv in June 2025 as a non-peer-reviewed preprint. That version reported an 80% score for an earlier system using OpenAI’s o3 model, against 20% for doctors. The later Science paper reports updated results and includes a physician comparison with reference materials. The journal paper was received in April and accepted in August 2026, according to the source account.

Microsoft’s blog post, titled “Building Medical Superintelligence,” highlighted comparisons including 1.7 times the physician score on emergency department cases and nearly four times the score of doctors without references on complex cases. Those ratios describe the study’s particular benchmark and comparison groups; they are not estimates of how much better the AI would perform across all medical diagnoses.

The benchmark’s methods also have limits. In details described in the preprint, a separate model graded diagnoses against the published final diagnosis, with physician re-grading showing good but imperfect agreement. If a requested test result was missing from a record, the gatekeeper generated a synthetic result. The authors characterized the cost estimates as first-order approximations. The final journal paper is the peer-reviewed publication, but some detailed methodology cited in the source account comes from the earlier preprint.

“I really believe that in 2027, we will be on the verge of a coding style breakthrough for medical records.”

— Microsoft AI chief executive Mustafa Suleyman, speaking to TechCrunch

Limits of the Doctor Comparison

The test does not establish how MAI-DxO would perform with real patients, physical examinations or imaging. The cases were historical and text-based, and the physician comparison involved solo generalists rather than specialists or teams. The difficult case set also did not include healthy patients or harmless explanations, according to the source account, so the benchmark could not measure how often the system would raise false alarms in a broader population.

It is also unclear how well the system’s accuracy and estimated savings would carry over to current hospitals, where patients, records, test availability and prices differ. The system’s 83.2% result and the comparisons are Microsoft’s reported findings; peer review does not mean researchers outside Microsoft have reproduced them. The source material does not provide results from an independent replication or from the planned patient study.

Testing MAI-DxO Alongside Patients

The next stated step is a prospective study at Beth Israel Deaconess Medical Center in Boston. In that study, consenting patients are expected to interact with the system alongside their usual care. Unlike the retrospective benchmark, this setting could provide evidence about how the tool operates in a clinical workflow.

The source account does not give a start date, enrollment target, study design details or an expected date for results. Until those findings are available, the Science paper supports a narrower conclusion: MAI-DxO performed strongly on the reported simulation, while its performance and effects in patient care remain to be tested.

Key Questions

Did Microsoft show that its AI is better than doctors in hospitals?

No. The paper reports higher scores than solo generalist physicians on selected simulated cases. It did not test MAI-DxO against specialists or medical teams treating patients.

What does the 83.2% figure measure?

It is MAI-DxO’s reported diagnostic accuracy on 304 difficult case records from the New England Journal of Medicine, within a text-based benchmark.

Was the study peer-reviewed?

Yes. The paper, “Sequential diagnosis with language models,” was published in Science on October 22, 2026. Peer review is not the same as independent replication of the results.

Did MAI-DxO outperform GPT-5 on its own?

Microsoft’s chart reports 78.9% accuracy for GPT-5 alone on the complex cases, compared with 83.2% for MAI-DxO. The system’s orchestration layer added about four percentage points in that comparison.

What is the next test of the system?

A prospective study is planned at Beth Israel Deaconess Medical Center, where consenting patients would interact with MAI-DxO alongside their usual care. Timing and further study details have not been provided in the source material.

Source: ThorstenMeyerAI.com