When an AI agent goes off script, how do health systems react? And what is the script for such a new technology?
At Canton, Ohio-based Aultman Health System, the moment came from an internal employee handbook agent the team built to answer staff questions.
“Early on, we noticed the agent would occasionally ‘go off script’ when it couldn’t confidently find an answer,” Aultman CIO Raza Fayyaz told Becker’s. “Instead of admitting uncertainty, it would generate something plausible but incorrect, which is one of the core risks of deploying LLM-based agents without proper guardrails.”
The behavior pointed to something fundamental about how these systems work: An agent left to its own devices will reach for a fluent answer over an accurate one.
“That reinforced an important lesson: An AI agent without guardrails is often just a wrapper around an LLM,” Mr. Fayyaz said. “The underlying model is designed to complete, not necessarily to know when to stop.”
The three-hospital system’s fix was twofold. The team retrained the agent to say “I don’t know” or decline to answer when the source material didn’t support a response, and it implemented a retrieval-augmented generation (aka RAG) approach that gives the model a bounded set of context to draw from. Mr. Fayyaz considers that limitation essential for most enterprise query agents, both as a reliability measure and a cost one, since it reduces the volume of tokens the model processes.
“The broader takeaway is that AI agents can be powerful, but they should be treated like any other enterprise system: governed, tested and constrained before broad deployment,” he said.
At Philadelphia-based Jefferson Health, the unexpected behavior has been more understated, where a model’s reading of an instruction is noticeably not-human-like.
“At Jefferson, we’ve learned that even the most carefully designed AI systems can still surprise you — not through dramatic ‘rogue’ behavior, but through the quieter, more subtle ways they interpret the world,” said Luis Taveras, PhD, executive vice president and chief digital and information officer. “The real challenges tend to emerge in edge-case scenarios, where models reveal unexpected behaviors, misaligned priorities, or interpretations of instructions that differ from what a human would naturally intend.”
Dr. Taveras said Jefferson has not seen its agents act maliciously or unpredictably beyond those misinterpretations, and that when they do occur, users catch them quickly.
“That’s exactly why human-in-the-loop oversight isn’t just a safeguard for us — it’s a core design principle,” he said. “It ensures that our agentic systems remain aligned, reliable, and grounded in human judgment at every step.”
Ultimately, the goal is to keep AI agents on script. For example, Aurora, Colo.-based UCHealth moves quickly on experimentation but slows down before anything reaches patients, clinicians or core operations, leaning on tightly scoped use cases, preproduction validation and explicit guardrails.
“We also treat agents as governed identities in our ecosystem, with least-privilege access, continuous monitoring, and clear ownership, so if something did drift from the intended behavior, we could detect and intervene early,” said Richard Zane, MD, chief medical and innovation officer of UCHealth. “In fact, we joke internally that we may be holding our AI agents to a higher standard than our human ones as our people invariably go off script, so the software isn’t allowed to.”
At Houston Methodist, the worry isn’t launch day. It’s everything after.
“It’s important that we understand how agents behave not just at the moment they’re deployed, but over time, especially as the systems they interact with evolve and as they engage with their intended audiences,” said Roberta Schwartz, PhD, executive vice president and chief innovation officer of Houston Methodist. “Their performance and impact can shift in significant ways after upgrades or prolonged use. Monitoring and evaluating that progression is essential, even though it remains a complex challenge for both humans and bots alike.”
At the Becker's 11th Annual IT + Revenue Cycle Conference: The Future of AI & Digital Health, taking place September 14–17 in Chicago, healthcare executives and digital leaders from across the country will come together to explore how AI, interoperability, cybersecurity, and revenue cycle innovation are transforming care delivery, strengthening financial performance, and driving the next era of digital health. Apply for complimentary registration now.