AI-drafted responses to patient portal messages can take physicians more time to edit than it would take to write the replies from scratch, Hanover, N.H.-based Dartmouth researchers found.
The team analyzed 146,000 messages exchanged between 10,105 patients and their primary care physicians on the online portal at Lebanon, N.H.-based Dartmouth Health. AI-generated drafts frequently introduced errors and extraneous details and often failed to ask relevant follow-up questions, according to the study presented July 7 at the 64th Annual Meeting of the Association for Computational Linguistics in San Diego.
Researchers tested drafts from Claude, Gemini and ChatGPT, along with three smaller commercial models — Llama, Aloe and Qwen — against a dataset of real clinician-written responses.
“We find that AI can sound like a doctor but not think like one,” said Sarah Preum, PhD, the study’s co-corresponding author and an assistant professor of computer science at Dartmouth College, in a July 6 Dartmouth news release.
Adapting the models to an individual physician’s communication style improved response accuracy by 33% and cut editing time by 26%, using a technique the researchers developed called TADPOLE. The gains have limits, however.
“If you have to edit 75% of the message, you may be spending more time and energy on making changes than if you were to just write it from scratch,” stated co-author Tim Burdick, MD, an associate professor of community and family medicine at Dartmouth’s Geisel School of Medicine and a family medicine physician at Dartmouth Health.
The findings add to a growing body of research questioning whether generative AI reduces or simply redistributes clinician workload.
Becker’s 3rd Annual Fall Chief Pharmacy Officer Summit is the top tier event that the people leading the future of pharmacy convene. This 3-day event is built specifically for Chief Pharmacy Officers and senior pharmacy leaders. The sessions examine the workforce, financial and clinical priorities shaping pharmacy today. Becker’s is where candid discussions flourish and new ideas thrive. Becker's 3rd Annual Fall Chief Pharmacy Officer Summit
