As AI races into exam rooms and onto patients’ phones, the physician’s role is more important than ever — not because doctors must police the technology, but because chatbots cannot replace trust, judgment and the patient-physician relationship, according to Jan Carney, MD.
Dr. Carney, 2026-2027 president of the American College of Physicians, published a statement Nov. 3 about AI hype. Dr. Carney is associate dean for public health and health policy and chief of the division of public health at the University of Vermont’s medical college in Burlington, Vt.
Her message follows a September MAHA summit wherein HHS Secretary Robert F. Kennedy Jr. said AI can offer “a second opinion that is much better informed than any doctor in the country” — a claim six physician organizations, including ACP, publicly refuted. Those groups said physicians must “help lead the responsible development and use of AI in medicine.”
Dr. Carney acknowledged AI’s clinical promise — in documentation, imaging, cancer detection and diagnostic support — but anchored her case in patient safety data. She pointed to a Nature Medicine study in which ChatGPT misread the severity of emergencies such as diabetic ketoacidosis and imminent respiratory failure more than half the time, 52%. A separate systematic review in BMC Emergency Medicine found AI had a sensitivity of just 61% for the most severe emergencies.
She also tied consumer chatbot use to real harm, citing high-profile adolescent suicides linked to mental health chatbots and pending litigation. Usage is widespread: an estimated 25% of ChatGPT’s 800 million users seek health information, and a KFF poll found about a third of U.S. adults have used AI chatbots for health advice in the past year. In that same poll, 42% did not follow up with a clinician on physical health questions and 58% did not on mental health questions — despite most saying they did not trust the information AI provided.
ACP’s position paper, published this year in Annals of Internal Medicine, frames AI as “augmented intelligence” — a tool to assist, not replace, physicians. It lays out three ethical guideposts: relationality (AI should strengthen, not weaken, the patient-physician relationship), self-governance (physician integrity and independent clinical judgment) and competence (expert knowledge applied fairly and without bias).
Dr. Carney also urged physicians to build basic AI literacy, including “prompt literacy,” and warned of automation bias — the tendency to trust machine output as more reliable than it is — particularly as AI confabulates more often on complex medical questions.
Her bottom line: trust, accountability and clinical judgment applied to each patient are human elements “never reproducible through chatbots.” As she put it, “Patients trust us. Society counts on us. Patients’ health depends on us. We are physicians, not chatbots.”
Becker’s 3rd Annual Fall Chief Pharmacy Officer Summit is the top tier event that the people leading the future of pharmacy convene. This 3-day event is built specifically for Chief Pharmacy Officers and senior pharmacy leaders. The sessions examine the workforce, financial and clinical priorities shaping pharmacy today. Becker’s is where candid discussions flourish and new ideas thrive. Becker's 3rd Annual Fall Chief Pharmacy Officer Summit
