4 minSociety
Microsoft AI outperforms generalist doctors in simulated diagnosis test
Microsoft's MAI-DxO system correctly diagnosed about 83% of complex medical cases in a simulated study, compared with 36% for generalist physicians using reference materials, though researchers caution about the study's limits.
Microsoft says a new artificial intelligence system outperformed generalist physicians in solving difficult medical cases in a simulated study, a result that could eventually help doctors tackle hard-to-diagnose illnesses. The findings were published on Oct. 22 in the journal Science, and the company's system correctly identified 83% of complex cases. Generalist doctors who could consult reference materials reached 36% accuracy under the same testing conditions, meaning the AI was more than twice as accurate.
The technology, which Microsoft calls the MAI Diagnostic Orchestrator, or MAI-DxO, works differently from a single program issuing an answer. It uses multiple AI agents that operate much like a panel of doctors discussing a complicated case. The system begins with limited information, asks follow-up questions, and considers which medical tests might identify the problem. As new information arrives, the agents challenge one another's conclusions and can revise their initial assessment.
That approach is designed for situations where symptoms overlap across several diseases, such as a patient with unexplained fatigue and several unusual lab results. Rather than settling on one explanation early, the system can weigh multiple possibilities as the evidence changes.
Microsoft evaluated the technology using 304 complex medical cases drawn from The New England Journal of Medicine, many involving unusual illnesses that challenge experienced physicians. According to the company's findings, MAI-DxO correctly diagnosed roughly 83% of those cases. Generalist physicians working alone reached 36% accuracy when they could use reference materials, and 22% without them. In a separate test using historical emergency department cases, the AI achieved about 80% accuracy, compared with 47% for the physician comparison group.
The study carries important caveats. The physicians in the comparison were generalists, including primary care doctors and general internists, who evaluated cases individually. Hospitals often bring in specialists for particularly difficult problems, and those specialists may collaborate and review additional evidence before reaching a diagnosis. Microsoft did not directly compare its system against teams of medical specialists.
The comparison groups were also not evaluated on identical workloads. The AI was tested across 304 cases, while physicians worked through a smaller set of 56 cases held out from the system's development. The researchers relied on existing written case reports rather than treating patients in real time. Actual care can be far less predictable, with patients who struggle to describe symptoms, incomplete medical records, and physical examination clues that never appear in a case summary.
The simulation also used an AI system to supply information from existing case reports when certain details were missing, generating plausible responses to create diagnostic conversations. Those interactions cannot fully reproduce the uncertainty doctors face with real patients. Moreover, identifying the correct diagnosis is only part of medical care. Doctors must also determine appropriate treatment and account for individual circumstances.
The underlying AI models play a major role in the system, and the researchers tested models developed by OpenAI, Google, Anthropic and other companies. Because the study's authors work for Microsoft, independent testing will be especially important before hospitals consider relying on the technology for patient care. For now, the results show how the system performed under controlled conditions, while whether it can deliver similar benefits in everyday practice remains an open question.
7
