AI-drafted responses to patient portal messages can take physicians more time to edit than it would take to write the replies from scratch, Hanover, N.H.-based Dartmouth researchers found.
The team analyzed 146,000 messages exchanged between 10,105 patients and their primary care physicians on the online portal at Lebanon, N.H.-based Dartmouth Health. AI-generated drafts frequently introduced errors and extraneous details and often failed to ask relevant follow-up questions, according to the study presented July 7 at the 64th Annual Meeting of the Association for Computational Linguistics in San Diego.
Researchers tested drafts from Claude, Gemini and ChatGPT, along with three smaller commercial models — Llama, Aloe and Qwen — against a dataset of real clinician-written responses.
“We find that AI can sound like a doctor but not think like one,” said Sarah Preum, PhD, the study’s co-corresponding author and an assistant professor of computer science at Dartmouth College, in a July 6 Dartmouth news release.
Adapting the models to an individual physician’s communication style improved response accuracy by 33% and cut editing time by 26%, using a technique the researchers developed called TADPOLE. The gains have limits, however.
“If you have to edit 75% of the message, you may be spending more time and energy on making changes than if you were to just write it from scratch,” stated co-author Tim Burdick, MD, an associate professor of community and family medicine at Dartmouth’s Geisel School of Medicine and a family medicine physician at Dartmouth Health.
The findings add to a growing body of research questioning whether generative AI reduces or simply redistributes clinician workload.
