The 'Second Opinion Machine': How an AI Panel of Virtual Doctors Could Crack Your Mystery Illness Before Your Next Test Is Even Ordered

Picture a patient who has seen three doctors in four months. Night sweats, a strange rash, aching joints, a fever that comes and goes. Each visit ends with a reasonable guess, a prescription and a "let's see how it goes." Each guess is wrong. By the time the right answer arrives, the patient has had a pile of tests, lost weeks of work and, in the worst cases, lost the window when treatment would have worked best.

That slow, frustrating loop is called the diagnostic odyssey, and it is one of the most common ways that modern medicine fails people. On October 22, 2026, a peer-reviewed paper in the journal Science put a provocative new tool into that story: an AI "panel of doctors" that works through a case one question and one test at a time, and that got the right answer far more often than the generalist physicians it was compared against.

The headline numbers are striking. The fine print matters just as much. Here is what the study actually showed, what it did not show, and what it could mean for you.

What Was Actually Published

The paper, titled "Sequential Diagnosis with Language Models," was written by a team of 15 researchers at Microsoft. It describes a system called the Microsoft AI Diagnostic Orchestrator, usually shortened to MAI-DxO or simply DxO. An earlier version of the work circulated as a non-peer-reviewed preprint in mid-2025; the Science version, received in April 2026 and accepted in August, is the first time it has passed formal peer review.

The key idea is not a smarter chatbot. It is a different way of testing and using one.

Most earlier headlines about "AI beats doctors" came from multiple-choice exams or from cases where the model was handed the entire medical file up front. Real medicine does not work that way. A clinician starts with a short complaint ("chest pain for two days") and has to decide what to ask next, what to examine, and which tests are worth ordering. Every test costs money and time, and some carry risk.

To mimic that, the researchers built the Sequential Diagnosis Benchmark, or SDBench. It turns real cases into step-by-step puzzles. The AI (or the human doctor) receives a brief description, then must request specific information or specific tests. A separate "gatekeeper" component answers only what is asked. When the player commits to a final diagnosis, it is scored against the confirmed answer, and the estimated cost of every test ordered is added up.

Two very different sets of cases

• 304 complex teaching cases drawn from the clinicopathological conference series in The New England Journal of Medicine. These are famously difficult cases chosen precisely because they stumped people.
• 325 emergency department encounters that led to hospital admission, taken from MIMIC-IV-ED, a de-identified research database from a Boston teaching hospital. These are closer to everyday acute care.

The Numbers That Made Headlines

On the hard teaching cases, the orchestrator running on OpenAI's GPT-5 model reached about 83% diagnostic accuracy. Generalist physicians given the same step-by-step format averaged about 36% when they were allowed to use textbooks and online references, and about 22% without them. The single best-performing physician reached roughly 52%.

On the emergency department cases, the orchestrator scored about 80%, against roughly 47% for the participating generalists. The best individual physician scored about 63%.

Two other findings are easy to miss but arguably more important for health systems:

• The "panel" adds value on top of the model. GPT-5 working alone scored about 79% on the teaching cases. Wrapping it in the orchestrator added roughly four percentage points. The design is model-agnostic, so the same structure can sit on top of different AI engines, and the researchers re-ran parts of the evaluation with a newer model before publication.
• Accuracy and cost can be traded off deliberately. With no spending limit, the orchestrator ran up an estimated US$5,431 in tests per teaching case, more than physicians using references (about US$4,725). With a budget constraint switched on, it still reached about 81% accuracy at roughly US$2,241 per case, and about 80% at roughly US$1,303. Those figures come from standardized price assumptions in the simulation, not real hospital bills.

How an AI "Panel of Doctors" Works

Instead of asking one model for one answer, the orchestrator splits the job into roles, a bit like a hospital case conference. One virtual participant keeps a running list of possible diagnoses and how likely each one is. Another proposes the most informative next question or test. Another plays devil's advocate, actively looking for evidence that would knock down the leading theory. Another watches the cost of every request. A final checker decides when the evidence is strong enough to commit.

That structure targets the classic thinking traps behind many human diagnostic errors: anchoring on the first idea, stopping the search once something "fits," and ordering tests that do not actually change the decision. Whether the software avoids those traps in a messy real clinic is exactly what has not yet been tested.

Why This Matters: The Hidden Scale of Diagnostic Error

Diagnostic error rarely makes the news, yet it is one of the largest sources of preventable harm in health care. The U.S. National Academies of Sciences, Engineering, and Medicine concluded in 2015 that most people will experience at least one diagnostic error in their lifetime. A 2023 analysis led by researchers at Johns Hopkins estimated that roughly 795,000 Americans die or are permanently disabled each year because of diagnostic errors, with stroke, sepsis and cancer among the most common culprits.

The problem is global. Family doctors in Canada, the United Kingdom and much of Europe juggle very short appointments, and wait times for specialists can stretch for months. In parts of Asia, the Middle East and Africa, a generalist may be the only clinician a patient sees for hundreds of kilometres. A tool that helps a generalist reason like a well-stocked team of specialists could, in theory, close some of that gap. That "in theory" is doing a lot of work, which is why the next section matters.

The Fine Print: What the Study Did Not Prove

To their credit, the authors list significant limitations themselves, and independent coverage has underlined them.

• It was a simulation, not patient care. Every case was historical and text-based. There was no physical exam, no reading of images or pathology slides, no patient who forgets a symptom or struggles to describe pain, and no follow-up over weeks.
• Some answers were synthetic. When the original record did not contain a requested detail, the gatekeeper could generate a plausible response. That keeps the game moving but is not how real charts behave.
• The comparison was generalists working alone. Physicians were not allowed to consult colleagues, and they were not specialists or multidisciplinary teams. The authors expect the gap to narrow against matched expert teams.
• Hard cases were over-represented. Healthy people and routine, benign complaints were largely absent, so we do not know how often the system would raise false alarms or push for unnecessary tests in a normal clinic.
• Diagnosis is not treatment. The scoring checked whether the disease was named correctly. It did not measure whether patients got the right treatment, were harmed, or recovered faster.
• Company-run research. All 15 authors are Microsoft employees who hold company equity, and several are named on a related patent application. Peer review strengthens the work, but it has not been independently replicated in a hospital.

The system also has no regulatory clearance anywhere and is not available as a product for patients or clinicians.

The Next Step: A Real Hospital

Alongside the paper, Microsoft announced plans for a prospective study with Beth Israel Deaconess Medical Center, a Harvard-affiliated teaching hospital in Boston. According to reporting on the announcement, consenting patients would interact with the system while still receiving standard care, and its conclusions would be compared with doctors' independent assessments. The company's health lead has said testing could begin within a couple of months, but no formal start date, sample size or timeline for any product has been published.

That study is the moment of truth. Plenty of medical AI tools have looked excellent on curated data and then stumbled when they met real patients, real workflows and real time pressure.

The Global Regulatory Picture

Canada

In Canada, software that diagnoses or informs diagnosis is generally regulated by Health Canada as a medical device, and Health Canada has published dedicated guidance for machine learning-enabled medical devices that emphasizes transparency and plans for managing model changes. Any patient-facing deployment would also have to respect federal privacy law under PIPEDA and provincial health-information statutes such as Ontario's PHIPA.

United States

In the U.S., a diagnostic tool like this would face review by the Food and Drug Administration, which opened a public consultation in August 2026 on how it should oversee generative AI-enabled medical devices. Patient data used in a hospital study falls under HIPAA, and for readers north of the border the comparable baseline is PIPEDA. Both laws were written long before conversational AI, which is why many experts argue they need modern guidance on consent, data reuse and model training.

Europe

In the European Union, AI used for medical diagnosis is treated as high-risk under the EU AI Act and must also satisfy the Medical Device Regulation. That means documented risk management, human oversight, high-quality data and post-market monitoring, with the heaviest obligations phasing in over the next couple of years. The GDPR adds strict rules on processing health data.

Asia and the Middle East

Regulators across Asia and the Middle East are building their own frameworks. Singapore's Health Sciences Authority, Japan's PMDA and Saudi Arabia's Food and Drug Authority have all issued guidance or pathways for AI-based medical software, and Gulf states such as the UAE and Saudi Arabia are investing heavily in hospital AI. For countries with severe specialist shortages, a validated diagnostic assistant could be transformative, but only with local validation on local populations. The World Health Organization's guidance on large multimodal models in health stresses exactly that, along with transparency and clear accountability.

Benefits and Risks at a Glance

The potential upside

• Faster answers for patients with confusing, multi-system symptoms.
• A built-in "second opinion" for generalists in under-served or rural areas.
• Fewer low-value tests if cost-aware settings hold up in practice.
• A transparent reasoning trail that clinicians can inspect and challenge.

The real risks

• Automation bias: busy clinicians may defer to a confident machine even when it is wrong.
• Over-testing: the unconstrained version spent more on tests than the doctors did.
• Uneven performance: accuracy on Boston and journal cases may not carry over to other countries, languages or populations.
• Accountability gaps: if the AI and the doctor disagree and the patient is harmed, who is responsible?
• Privacy: sequential diagnosis needs rich, detailed health data, which must be protected end to end.

What This Means for You

You cannot use this system today, and you should not expect to see it in your doctor's office soon. But the study offers several practical lessons right now.

• Bring your full story. The AI's strength came from collecting information step by step. Write down your symptoms, when each started, what makes them better or worse, and every medication and supplement you take. That timeline helps any diagnostician, human or machine.
• Ask the "what else could it be?" question. The orchestrator's devil's-advocate role exists because the first idea is often wrong. You can ask your clinician the same thing.
• Ask what a test will change. A reasonable question is: "If this comes back positive or negative, what will we do differently?"
• Be careful with consumer chatbots. A research system tested in a controlled benchmark is very different from a general chatbot on your phone. Use AI to prepare questions, not to replace medical advice or to change medication on your own.
• If you are invited into an AI study, ask questions. Who sees your data, how is it stored, can you withdraw, and will a human clinician still make the final call?

For Healthcare Professionals

• Read the benchmark, not the headline. Sequential, cost-aware evaluation is a meaningful step beyond multiple-choice exams. Expect it to become a reference point for vendors and regulators, and ask any supplier how their tool performs on step-by-step tasks.
• Design for oversight. Decide in advance how AI suggestions are shown, when clinicians must document disagreement, and how errors are reported and reviewed.
• Measure cost and harm, not just accuracy. Local pilots should track test volumes, time to diagnosis, false alarms and patient outcomes.
• Prepare governance now. Align procurement and privacy reviews with HIPAA and PIPEDA, the EU AI Act, and local regulators, and plan for model updates that may change behaviour over time.
• Watch the Boston study closely. Prospective results with real patients will tell us far more than any retrospective benchmark.

Wrapping Up

The Science paper does not prove that AI is a better doctor. It proves something narrower and still important: when an AI is forced to reason step by step, weigh the cost of every test and argue with itself, it can solve hard, text-based diagnostic puzzles far more often than generalists working alone.

The open question is whether that advantage survives contact with real patients, real time pressure and real accountability. If it does, the biggest winners may not be elite academic hospitals but the overstretched family doctor, the rural clinic and the patient stuck on a diagnostic odyssey. If it does not, we will have learned that cheaply, in a study, rather than expensively, at the bedside. Either way, the next chapter is being written in a hospital, not a benchmark, and that is exactly where it belongs.