Microsoft’s AI diagnostic system correctly diagnosed 83.2% of complex New England Journal of Medicine cases, compared with 36.1% for generalist physicians using search and reference tools, according to a study published Oct. 22 in Science. Without those tools, physicians averaged 22.3%.
The system, called Diagnostic Orchestrator, or DxO, simulates a panel of physicians and ran on OpenAI’s GPT-5 in the study’s main results. It starts from a brief clinical presentation, then asks questions, orders tests and revises the differential diagnosis step by step. Microsoft researchers tested it on 304 NEJM clinicopathological conference cases and 325 emergency department admissions from the MIMIC-IV-ED dataset. The head-to-head comparison with physicians on the NEJM cases used a held-out set of 56 cases.
On the emergency cases, DxO reached 80.3% accuracy, compared with 46.8% for physicians.
The researchers also tracked estimated testing costs, using the chargemaster of Kaiser Permanente Washington’s Central Hospital in Seattle. With no budget limits, DxO averaged $5,431 per complex case. With budget constraints, it hit 80.9% accuracy at $2,241. Physicians using search and reference tools averaged $4,725.
DxO also took a different approach to workups. It asked an average of 25.5 questions before ordering its first test, compared with 5.6 for physicians. Across all models tested, the orchestration raised accuracy by an average of 7.9 percentage points on complex cases and 4.4 points on emergency cases.
The authors noted the results came from a controlled, text-only benchmark that compared DxO with individual generalists. They wrote that the comparisons “do not establish superiority over specialists, multidisciplinary teams, or real-world clinical practice.” All of the study’s authors are Microsoft employees who hold equity in the company, and Microsoft funded the research.
Microsoft plans to evaluate DxO in real-world settings through a partnership with Beth Israel Deaconess Medical Center in Boston, TechCrunch reported Oct. 22.
