AI Doctor Panel Beats Human Physicians in Complex Case Study

A new study suggests artificial intelligence is starting to pull ahead of human doctors in the hardest medical cases. The results, published this Thursday in the journal Science, show a significant gap in diagnostic accuracy between an AI system and generalist physicians. This finding marks a pivotal moment in the ongoing debate about the role of automation in healthcare.

Microsoft’s system, known as MAI-DxO, functions like a virtual panel of experts. It coordinates multiple AI agents that question patients, order tests, and debate diagnoses. In simulations using complex cases from the New England Journal of Medicine, the system was correct 83% of the time. By comparison, primary care doctors and general internists working individually with reference materials were correct only 36% of the time.

It is important to note that these specific cases are typically handled by specialists or teams in real-world settings. Those specialists were not part of the comparison. Despite this, Dominic King, vice president of Health at Microsoft AI, called the results substantial. He noted that the growing mismatch between the demand for medical expertise and its supply is the biggest challenge in healthcare delivery right now.

Microsoft joins Google and OpenAI in asserting that its AI can match or outperform doctors. However, the study’s data indicates that much of the improvement comes from using more powerful underlying models. Microsoft’s orchestration layer adds additional gains on top of that base performance. The best results were achieved using OpenAI’s GPT-5 and GPT-5.5 models, though Microsoft also tested systems from Anthropic, Meta, DeepSeek, and Google.

Cost efficiency was also a key metric. Running GPT-5 alone on complex cases cost about $8,400 per case with 78.9% accuracy. Adding the DxO system reduced the cost by roughly $3,000 but only improved accuracy by about four points. For emergency department cases, there was little difference in either cost or accuracy between using the model alone and using it with the DxO assist.

The benefits of the orchestration system become more apparent with less capable models. For example, Gemini 2.5 Flash scored 46.4% on its own but jumped to nearly 69% with DxO. However, the gap between weaker and stronger base models remained larger than the boost provided by the system. Researchers noted that accuracy gains naturally shrink as models approach their theoretical maximums.

Mustafa Suleyman, CEO of Microsoft AI, described the results as a step toward "medical superintelligence." He is leading Microsoft’s effort to build its own frontier AI. The company is currently working with the Mayo Clinic to train a foundation model for healthcare, with a release planned for next year. Suleyman believes this will enable enormous leaps in the field.

Suleyman predicts a coding-style breakthrough for medical records in 2027. He envisions a future where AI can stream highly accurate predictions about how a medical record might unfold. He views this foresight as invaluable for clinicians. He has spent his career applying AI to medical issues, previously co-founding DeepMind and later leading Microsoft’s AI unit since 2024.

Microsoft is moving beyond simulations to real-world evaluations through a partnership with Beth Israel Deaconess Medical Center in Boston. This study will assess DxO across different hospital settings, including emergency and urgent care. Patients will receive standard treatment while consenting patients will also engage with the AI. The team will compare the AI’s recommendations against those of human doctors. This follows a similar recent study by Google published in The Lancet.

Microsoft aims to integrate these tools into existing healthcare systems rather than just benchmarking individual tasks. The company already sells dictation and radiology tools to clinicians and sees this as an expansion of those efforts. Suleyman stated that Microsoft is committed to becoming the leader in medical clinical decision support products over the next few years.