AI can’t distinguish between true and false medical claims once they’re packaged as realistic-looking hospital notes or social-media health discussions, researchers from New York City-based Mount Sinai Health System found.
The study authors analyzed over 1 million prompts across nine leading large language models after exposing them to actual hospital discharge summaries with a single fake recommendation added, widespread health myths from Reddit, and 300 brief clinical scenarios generated and validated by clinicians, according to the study published Feb. 9 in Lancet Digital Health. Several models treated the falsehoods as real medical guidance.
“AI has the potential to be a real help for clinicians and patients, offering faster insights and support,” said co-senior and co-corresponding author Girish Nadkarni, MD, chief AI officer of Mount Sinai Health System, in a Feb. 9 news release. “But it needs built-in safeguards that check medical claims before they are presented as fact. Our study shows where these systems can still pass on false information, and points to ways we can strengthen them before they are embedded in care.”
At the Becker's 11th Annual IT + Revenue Cycle Conference: The Future of AI & Digital Health, taking place September 14–17 in Chicago, healthcare executives and digital leaders from across the country will come together to explore how AI, interoperability, cybersecurity, and revenue cycle innovation are transforming care delivery, strengthening financial performance, and driving the next era of digital health. Apply for complimentary registration now.
