Health systems have found plenty of work for AI. At Salt Lake City-based Intermountain Health, an AI agent triages incoming IT incident tickets, and an AI-enabled human resources front door has resolved about 35,000 employee inquiries without anyone opening a case. Other systems have put agents to work on coding, scheduling and revenue cycle.
Ask what they won’t let AI do, and the answers are more consistent than the adoption curves. The same caution shapes which Epic AI tools health systems have kept switched off, and it extends well beyond any one vendor: Across interviews with Becker’s this year, the refusals kept returning to AI that acts on a patient rather than hands its work to a clinician.
Muhammad Siddiqui, CIO of Richmond, Ind.-based Reid Health, described the difference in June.
“We’ve already seen what ambient AI can do,” he said. “But there’s a real difference between AI that drafts something for a clinician to approve and AI that acts on its own. We’ve kept a person in the loop, and we’re not ready to remove them yet.”
The first category on hold is the autonomous clinical agent. Jane Moran, chief information and digital officer of Somerville, Mass.-based Mass General Brigham, explained her system’s position in February.
“Because these tools can influence patient care, strong governance, security and monitoring have to come first, so we haven’t broadly deployed autonomous clinical agents yet,” Ms. Moran said.
Chicago-based Northwestern Medicine has used AI agents for administrative workflows, documentation support and coding. Eric Shelley, vice president of information services, described where clinical use stood in February.
“When it comes to high-stakes clinical workflows where patient safety is paramount and error tolerance is essentially zero, we’re still in the first inning,” Mr. Shelley said.
The second is AI that diagnoses, orders or treats on its own. Aurora, Colo.-based UCHealth runs agents across administrative documentation, revenue cycle and scheduling with full audit logs, strict permissions, no access to ordering and an immediate kill switch.
“AI does not independently diagnose, order, or treat — it can strongly recommend and prebuild orders, but a clinician must click ‘accept,'” said Richard Zane, MD, chief medical and innovation officer of UCHealth. “The limit is driven by regulation, liability, and culture, not by what the models can already do.”
Yale New Haven (Conn.) Health writes separate rules for clinical and operational AI, said Lee Schwamm, MD, senior vice president and chief digital health officer.
“We require human-in-the-loop for all clinical decision and diagnostic support tools,” Dr. Schwamm said. “In operations and administration processes, we require extensive human-in-the-loop validation, but if the models perform as expected and exceed benchmark thresholds, then we will allow them to execute autonomously with periodic audits.”
The third is AI that deals with patients directly. Dr. Schwamm addressed that category, too.
“The greatest opportunity to improve health will be found in patient-facing agents that can actually deliver healthcare,” he said. “But the immature liability frameworks and risk of patient harm from unanticipated actions are major barriers to that next evolution.”
In some states, lawmakers have drawn part of that line for health systems. Illinois bars AI from therapeutic decision-making in mental healthcare under a law signed in August 2025, while still allowing it for administrative and supplementary tasks.
Some leaders haven’t turned on agentic AI at all. Newport Beach, Calif.-based Hoag had not deployed it as of April, according to Kathy Azeez-Narain, chief digital and customer innovation officer.
“I’d be skeptical of health system leaders who claim full agentic deployment today, because the oversight infrastructure required to do that responsibly doesn’t broadly exist yet,” she said.
Much of the hesitation turns on whether a mistake can be undone. Dustin Hufford, senior vice president and CIO of Camden, N.J.-based Cooper University Health Care, drew that comparison in June.
“The use of tools like ChatGPT or Gemini for summarization and augmentation carries a different risk profile, where errors, while not ideal, are generally recoverable,” Mr. Hufford said. “In contrast, an autonomous agent acting on a flawed inference within an operational or clinical workflow may not be. That distinction shapes our approach.”
Some leaders don’t want the boundary to move. Daniel Barchi, senior executive vice president and CIO of Chicago-based CommonSpirit Health, addressed it during a presentation at Becker’s Annual Meeting in April.
“There will always be a clinician between our AI and the patient,” Mr. Barchi said. “We never want to get to the point where there’s autonomous diagnosis and action, where a human — especially a well-trained doctor or nurse — is not in the loop making the decision.”
Paul Testa, MD, chief health informatics officer of New York City-based NYU Langone Health, expects the question to come to a head soon. He raised it on an episode of the “Becker’s Healthcare Podcast.”
“We will find circumstances where the addition of a clinician degrades an AI model’s performance,” Dr. Testa said. “And there is an ethical quandary of when we may need to remove the person in the loop to ensure higher quality. We’re getting really close to that moment.”
Until then, Reid Health’s CIO is keeping the person in the loop.
“I’m not skeptical of the technology. I’m disciplined about the timing,” Mr. Siddiqui said. “Technology works best when it fades into the background and shows up when lives depend on it. Until these tools can do that reliably, they stay on the radar.”
Becker’s 3rd Annual Fall Chief Pharmacy Officer Summit is the top tier event that the people leading the future of pharmacy convene. This 3-day event is built specifically for Chief Pharmacy Officers and senior pharmacy leaders. The sessions examine the workforce, financial and clinical priorities shaping pharmacy today. Becker’s is where candid discussions flourish and new ideas thrive. Becker's 3rd Annual Fall Chief Pharmacy Officer Summit
