Artificial intelligence is increasingly being used for medical diagnosis, with some systems demonstrating an ability to identify conditions that have stumped human doctors. One patient, after years of unresolved symptoms and visits to eight different clinicians, found a correct diagnosis within seconds using a publicly available AI chatbot.
Key Takeaways
- A patient with a complex digestive condition used ChatGPT to identify the cause of his symptoms after eight doctors could not provide a solution.
- In a formal academic challenge, a specialized AI model named CaBot correctly diagnosed a rare inflammatory condition, matching the conclusion of an expert physician but in a fraction of the time.
- Studies show that while AI can be highly accurate, it also carries risks, including providing incorrect medical advice and hallucinating data.
- Medical schools are beginning to integrate AI tools into their curriculum, raising concerns about potential over-reliance and the erosion of independent clinical reasoning skills among future doctors.
A Patient's Breakthrough with AI
In 2017, software engineer Matthew Williams underwent emergency surgery for cecal volvulus, a condition where the intestine twists on itself. The initial diagnosis was constipation, a mistake that could have had severe consequences. Following the surgery, which involved removing six feet of his intestines, Williams developed chronic and severe digestive problems.
For several years, he consulted a total of eight different medical professionals, including gastroenterologists and nutritionists. None were able to identify the specific cause of his distress. His diet became extremely restricted, significantly impacting his quality of life. "When your food is bland, your life becomes bland, too," he stated.
In 2023, Williams described his medical history and symptoms to ChatGPT. The AI model suggested his issues could be linked to foods high in oxalate, a compound he had never heard of from his doctors. The list of high-oxalate foods generated by the AI perfectly matched the items that caused his most severe reactions. Working with a nutritionist based on this new information, his symptoms improved dramatically. "I have my life back," Williams said.
The Challenge of Medical Misdiagnosis
Diagnostic errors are a significant problem in healthcare. In the United States alone, misdiagnosis is estimated to disable hundreds of thousands of people annually. Furthermore, autopsy studies have suggested that diagnostic errors may contribute to as many as one in every ten patient deaths. This highlights the potential for new technologies to improve patient outcomes.
Human vs. Machine in a Diagnostic Showdown
The potential of AI in medicine was recently tested in a high-profile academic setting at Harvard's Countway Library of Medicine. The event featured a competition between Dr. Daniel Restrepo, an expert diagnostician from Massachusetts General Hospital, and an AI model named CaBot.
CaBot was developed by Harvard researchers Arjun Manrai and Thomas Buckley. It is a custom version of an advanced OpenAI reasoning model enhanced with a process called retrieval-augmented generation (RAG), which allows it to pull information from external medical literature to inform its analysis.
The Case and The Competitors
The challenge was to diagnose a complex case involving a 41-year-old man with fever, swollen ankles, a rash, and fainting spells. A CT scan also revealed lung nodules and enlarged lymph nodes. Dr. Restrepo was given six weeks to prepare his analysis, while CaBot was given just six minutes.
Dr. Restrepo meticulously broke down the patient's symptoms, ruling out conditions like lymphoma and infection based on the timeline and clinical data. He concluded the patient had Löfgren syndrome, a rare form of the inflammatory condition sarcoidosis. His reasoning was met with applause.
The AI's Performance
Following the human expert, a presentation generated by CaBot was played. Using a synthesized, professional-sounding voice, the AI walked through the case, highlighting what it deemed the most relevant information. It focused on specific details from the CT scan, noting how small nodules tracked along the lung's lymphatic vessels.
After considering a range of possibilities, CaBot also concluded that the "single best fit is acute sarcoidosis, manifesting as Löfgren syndrome." The AI reached the same correct diagnosis as the human expert, demonstrating a sophisticated reasoning process that was created in a fraction of the time.
AI's Diagnostic Accuracy
When tested on several hundred historical medical cases from The New England Journal of Medicine, the CaBot AI correctly solved approximately 60% of them. According to the researchers, this success rate is significantly higher than that achieved by human doctors in a previous study analyzing similar cases.
The Risks and Limitations of Medical AI
Despite its successes, the use of AI for medical advice is not without significant risks. Publicly available chatbots are not designed as medical devices and can make serious errors. One study found that OpenAI's GPT-4 answered open-ended medical questions incorrectly about two-thirds of the time. Another showed GPT-3.5 misdiagnosed over 80% of complex pediatric cases.
"I trust A.I. more than doctors. I don’t think I’m the only one."
There have been documented cases of harm. In one instance, a 60-year-old man asked ChatGPT for salt substitutes and was advised to use bromide. After ordering and consuming it, he developed severe bromide poisoning, leading to hallucinations, paranoia, and an involuntary psychiatric hold. When doctors replicated the query, the AI again suggested bromide.
Privacy is another major concern. Medical information entered into a public chatbot is no longer private. This raises questions about data security and how personal health information is used by technology companies.
Integrating AI into Medical Education
As AI tools become more prevalent, medical schools are grappling with how to incorporate them into training. Dr. Adam Rodman, who leads efforts to integrate generative AI into Harvard's medical curriculum, highlights the different analytical strengths of AI. He noted that CaBot built its diagnosis around imaging findings that a human might overlook.
However, Dr. Rodman also warns of a risk known as "cognitive de-skilling," where over-reliance on technology could cause doctors to lose their own diagnostic abilities. A study involving gastroenterologists found that those who used AI to detect polyps during colonoscopies became significantly worse at finding polyps on their own.
The Student Perspective
Medical students are on the front lines of this transition. Benjamin Popokh, a student at University of Texas Southwestern, expressed his concern. "I’m worried these tools will erode my ability to make an independent diagnosis," he said. After finding himself using AI after almost every patient encounter, he realized he had not thought independently about a single case all day.
He has since made a conscious decision to formulate his own diagnosis before consulting an AI. "If all you do is plug symptoms into an A.I., are you still a doctor, or are you just slightly better at prompting A.I. than your patients?" Popokh questioned.
The Future Role of AI in Healthcare
Experts suggest the most effective use of AI may not be as a standalone diagnostician but as a supportive tool for clinicians. Dr. Gurpreet Dhaliwal, a diagnostician at the University of California, San Francisco, believes AI should be used for "wayfinding"—helping doctors identify trends, find missed details, or recall relevant research, rather than providing a final answer.
This approach is already being implemented. Penda Health, a network of clinics in Kenya, uses a tool called AI Consult that runs in the background, alerting clinicians to potential errors like overlooking anemia or prescribing unnecessary antibiotics. An evaluation found that clinicians using the tool made 16% fewer diagnostic errors and 13% fewer treatment errors.
For patients, AI can help them navigate the complex medical system. It can translate medical jargon, organize information from doctor visits, and help formulate questions for appointments. While current models can still make factual errors, their ability to process information can empower patients to become more active participants in their own care.