AI-drafted responses to patient portal messages can take physicians more time to edit than it would take to write the replies from scratch, Hanover, N.H.-based Dartmouth researchers found.
The team analyzed 146,000 messages exchanged between 10,105 patients and their primary care physicians on the online portal at Lebanon, N.H.-based Dartmouth Health. AI-generated drafts frequently introduced errors and extraneous details and often failed to ask relevant follow-up questions, according to the study presented July 7 at the 64th Annual Meeting of the Association for Computational Linguistics in San Diego.
Researchers tested drafts from Claude, Gemini and ChatGPT, along with three smaller commercial models — Llama, Aloe and Qwen — against a dataset of real clinician-written responses.
“We find that AI can sound like a doctor but not think like one,” said Sarah Preum, PhD, the study’s co-corresponding author and an assistant professor of computer science at Dartmouth College, in a July 6 Dartmouth news release.
Adapting the models to an individual physician’s communication style improved response accuracy by 33% and cut editing time by 26%, using a technique the researchers developed called TADPOLE. The gains have limits, however.
“If you have to edit 75% of the message, you may be spending more time and energy on making changes than if you were to just write it from scratch,” stated co-author Tim Burdick, MD, an associate professor of community and family medicine at Dartmouth’s Geisel School of Medicine and a family medicine physician at Dartmouth Health.
The findings add to a growing body of research questioning whether generative AI reduces or simply redistributes clinician workload.
At the Becker's 11th Annual IT + Revenue Cycle Conference: The Future of AI & Digital Health, taking place September 14–17 in Chicago, healthcare executives and digital leaders from across the country will come together to explore how AI, interoperability, cybersecurity, and revenue cycle innovation are transforming care delivery, strengthening financial performance, and driving the next era of digital health. Apply for complimentary registration now.
