Healthcare has broadly settled on a reassuring principle for clinical AI: keep the human in the loop. For most health systems, clinician oversight has become the standard safeguard and the structural answer to concerns about AI errors, bias and patient safety.
“There will always be a clinician between our AI and the patient,” said Daniel Barchi, senior executive vice president and CIO of Chicago-based CommonSpirit Health, during a presentation at the Becker’s 26th Annual Meeting in April. “We never want to get to the point where there’s autonomous diagnosis and action, where a human — especially a well-trained doctor or nurse — is not in the loop making the decision.”
Paul Testa, MD, chief health informatics officer at New York City-based NYU Langone Health, believes that position, however reasonable today, may not hold. Evidence is accumulating, Dr. Testa said in a recent episode of the “Becker’s Healthcare Podcast,” that clinician involvement may not always improve performance and the industry has not fully reckoned with what that means.
“We will find circumstances where the addition of a clinician degrades an AI model’s performance,” he said. “And there is an ethical quandary of when we may need to remove the person in the loop to ensure higher quality. We’re getting really close to that moment.”
Dr. Testa said he is not arguing for autonomous AI in clinical settings today. Trust between clinicians, patients and AI models is still being built, and that process is appropriate and necessary.
But the question he believes the field will soon have to answer runs deeper than a governance discussion. Concerns about automation bias — the tendency for clinicians to over-trust polished AI output — are already well documented. Dr. The next question is what it will mean to practice medicine when stepping back from a decision is sometimes the responsible act.
“I’ve very curious as to how, frankly, disruptive in nature this is. Our definition of professionalism and our definition of who we are as clinicians is going to be shaken when there are moments we need to step back and say ‘the central thing for me to do is to let the process run and me not to interfere,'” said Dr. Testa. “But we still are encouraged to say, ‘I need to validate that. I need to be there.’ That is just another way of saying there needs to be a human in the loop and we’re going to learn that it isn’t always the protection that we think it is.”
He is not worried about replacement. If safer patient care sometimes requires ceding a decision to a model, that may become the responsible choice. What he is watching for is building the judgment to know when the model should lead.
“What I’m most curious about in the next one to three years is how we move into that space of acknowledging, at times, the performance of these models outperform me, outperform a team, and therefore, we have to understand what its role here would be moving forward,” he said.
Will his colleagues agree? Dr. Testa has reason to think eventually they will. Over time he’s seen technology win over even the most stubborn clinicians when the results clearly benefit the patients.
“As a profession, doctors and nurses have been assessing what tools we bring to bear for the betterment of our patients for a century,” he said. “We have the rubric, if not the line by line assessment tools. We have a model of questioning, proving, obtaining trust and deploying. That’s what I’m hearing from my colleagues. We’re still in the trust building phase, and I think that’s appropriate.”
Tomas Gregorio, senior vice president and chief digital information officer at Care New England Health System in Providence, R.I., sees the same horizon from a different vantage point. Where Dr. Testa is watching evidence accumulate about AI performance, Mr. Gregorio is watching the gap between what the models can do and what the governance frameworks built to oversee them can actually handle.
“The models are improving faster than our governance frameworks,” Mr. Gregorio said during a recent episode of the “Becker’s Healthcare Podcast.” “I think that within three to five years, we are going to face real hard questions about AI’s role in diagnosis and treatment planning that we’re not fully prepared for as an industry. We’re not ready for that.”
Dr. Testa thinks the mechanism for getting there is trust, built through the same rigor medicine has applied to every other new tool, from early clinical decision support systems to AI applications in radiology. He hears the same message from colleagues across the field.
“What physicians want is an explanation of how these things are doing what they do and when do they earn the level of trust that we can hand over some of the control,” said Dr. Testa. “I don’t think we’re there yet, but I don’t think it’s five to 10 to 15 years away. It’s much closer than that. The reckoning is coming and that reckoning will only be resolved or acknowledged or moved past once we build the trust together.”
At the Becker's 11th Annual IT + Revenue Cycle Conference: The Future of AI & Digital Health, taking place September 14–17 in Chicago, healthcare executives and digital leaders from across the country will come together to explore how AI, interoperability, cybersecurity, and revenue cycle innovation are transforming care delivery, strengthening financial performance, and driving the next era of digital health. Apply for complimentary registration now.