Microsoft researchers say an AI tool that diagnoses patients step by step, as doctors do, was more than twice as accurate as general physicians at solving complex medical cases, according to a study published on Thursday in Science.
The model identifies and coordinates the questions that should be asked of a patient, as well as the exams and tests that should be ordered, using an “orchestrator” that organizes how AI will tackle a problem by employing a simulated panel of clinical experts collaborating to solve diagnostic cases, according to the report.
Existing medical AI models typically take an all-at-once approach that examines all of a patient’s symptoms, lab results and imaging studies to generate a diagnosis.
With the orchestrator coordinating the steps to reaching a diagnosis, the AI model first decides what to ask about the patient’s history and what is significant in the physical exam. Based on what it learns, it decides which tests to order, considers the results, and orders further testing as appropriate.
All the while, it is updating its working hypotheses and plans, the research leaders said in an interview.
Tasked with diagnosing more than 300 highly unusual and confusing medical dilemmas, the model employing the diagnostic orchestrator was more than twice as accurate at “working up” the patient and determining the diagnosis as general physicians using textbooks and the internet.
The tool also outperformed human physicians when presented with scenarios drawn from a major database of emergency room admissions.
Assessments comparing these types of older systems to human doctors usually use multiple-choice questions, the researchers said.
“AI has become very good at getting a 100% pass rate on those,” said Dr. Dominic King, the corporate vice president of the health unit at Microsoft AI, whose team is developing the new technology.
“But of course, that’s not real-world. No patient ever came to see me with a nice summary.”
The researchers also found that diagnostic processes cost less when the orchestrator designs the work-ups than when human doctors do.
“We’re not just measuring the accuracy of the systems. We’re also measuring how judiciously and carefully it goes through a diagnostic work-up,” said Harsha Nori, head of AI at Microsoft AI Health.
In a collaboration with Beth Israel Deaconess Medical Center in Boston, the researchers will next test the tool’s ability to triage patients in emergency department waiting rooms.
