Ask health system technology leaders what AI can’t do, and the list keeps getting shorter. Ask what they won’t let it do, and the answers come back to two things: judgment calls and the human work of getting people to use technology.
Start with a cyberattack, the scenario Raza Fayyaz, CIO of Canton, Ohio-based Aultman Health System, laid out: AI can spot the threat, isolate affected systems and recommend how to recover. What it shouldn’t decide is when to pull a critical clinical system like the EHR offline, or when to bring it back.
“A decision that makes perfect sense from a cybersecurity perspective may have serious consequences for patient care,” he said.
Access management works the same way. AI can analyze access requests and flag suspicious behavior, Mr. Fayyaz said, but unusual exceptions call for judgment about privacy, urgency and legitimate clinical need. His staff will always handle those.
He sorts work into three buckets: search, synthesis and judgment. He expects AI to take over much of the first two across healthcare. Judgment on consequential decisions is where he draws the line, not because AI will never be capable of making those calls, he said, but because some decisions require human accountability no matter how intelligent the technology becomes.
“That is the part I do not intend to fully automate,” Mr. Fayyaz said.
Reid Stephan, senior vice president and CIO of Boise, Idaho-based St. Luke’s Health System, said he is a strong believer in using AI aggressively to reduce administrative work, synthesize information, identify patterns and help people make better decisions. But when a decision carries significant consequences for a patient, an employee or the organization, he said, a human should own it.
“My line isn’t whether AI participates in the work. Increasingly, it should. My line is whether we’ve delegated human accountability to it,” Mr. Stephan said.
Keith Rivera, vice president and CIO of Omaha-based Nebraska Medicine, said he would never let AI independently make decisions that materially affect patient safety, clinical care or significant organizational risk. AI can generate insights, cut administrative burden and speed up decision-making, he said, but humans must remain accountable for final calls on patient outcomes, ethics, resource allocation and risk management.
“In healthcare, trust and accountability cannot be delegated to an algorithm,” Mr. Rivera said.
The other half of the line is about people.
For Ashis Barad, MD, executive vice president and chief transformation and digital information officer of New York City-based Hospital for Special Surgery, informatics doesn’t end when technology lands in people’s hands. The goal is getting them to actually use it in ways that improve care.
“Healthcare happens at the bedside and in relationships built on trust. That trust has to extend to our tools, and we earn it by being there, listening, and helping people do their work,” Dr. Barad said.
Srinath Adusumalli, MD, vice president and chief health information officer of Philadelphia-based Penn Medicine, said the work his informatics team won’t automate is going deep with clinical and operational partners: mapping the problems they see, and surfacing the ones they haven’t thought to solve but now can. That means going to the gemba, the lean term for where the work happens, to observe and understand the problem firsthand.
“Those are conversations. Those are sitting shoulder to shoulder,” Dr. Adusumalli said.
AI can aid all of that, he said, but the interactions themselves shouldn’t be automated. The same goes for change management once a tool goes live. Rather than an automated or AI-created training, Penn Medicine is finding that what helps most is “white glove service”: going to people and helping them work through the change.
That doesn’t mean keeping AI out of IT support. A generative AI assistant that answers employees’ technology questions from a knowledge base, in place of a call to the service desk, is a good first step, Dr. Adusumalli said. But that is information retrieval. Turning the answer into something useful for a specific person is a next step AI may help with, he said, but ultimately much of it still comes down to sitting shoulder to shoulder with people.
Even so, he was careful with the word “never.” It’s hard to predict a year out, he said, let alone five or 10.
“It’s also hard to say never,” Dr. Adusumalli said.
Becker’s 3rd Annual Fall Chief Pharmacy Officer Summit is the top tier event that the people leading the future of pharmacy convene. This 3-day event is built specifically for Chief Pharmacy Officers and senior pharmacy leaders. The sessions examine the workforce, financial and clinical priorities shaping pharmacy today. Becker’s is where candid discussions flourish and new ideas thrive. Becker's 3rd Annual Fall Chief Pharmacy Officer Summit
