Health systems have become increasingly disciplined about vetting AI before it goes live, building out governance committees, model reviews and risk assessments. What happens after go-live is often a different story.
Once an AI tool is deployed, responsibility for watching it tends to blur. Is it the data science team’s job? IT’s? The clinical department that requested it? Without a clear answer, performance can quietly slip for months before anyone notices, and the first sign of trouble is often a frontline complaint rather than a dashboard alert.
“The most common place teams get stuck is treating go-live as the finish line,” Geoff Patterson, senior vice president and chief digital and information officer at Detroit-based Henry Ford Health, told Becker’s. “Once a model is deployed, ownership quietly becomes ambiguous.”
UT Health San Antonio has built out dashboards for its AI use cases and reviews them with end users in recurring meetings, but assigning ultimate responsibility is still a work in progress.
“One of the challenges is: Who should be monitoring this?” said Edward Sankary, MD, vice president and chief health information and value officer. “Is it the application analyst, is it the AI analyst, or is it the operations team? Right now, most of it’s falling into our AI team to monitor these metrics.”
The health system holds AI office hours every other week, he said, where staff go through metrics with end users, asking questions like: “Are these metrics really reflecting how you’re feeling about the success and performance of the tools?”
Traditional IT applications did not create the same problem, noted Mitchell Schnall, MD, PhD, senior vice president for data and technology solutions at Philadelphia-based Penn Medicine.
“In the past, you had an application. It was rules-based. You tested a bunch of scripts against it, documented it, did all the things you needed it to do, and you implemented it. And the monitoring was just monitoring workflow through it, if you will,” he said.
“Here, you’ve got systems that aren’t rules-based. It’s very hard to anticipate inputs and outputs sometimes, and so the backend development of monitoring is way more important, a much larger fraction of the effort to implement and support an application than it traditionally is, and we have got to think about it that way.”
Several executives said the root of the problem is approaching monitoring as something to figure out later, rather than something built into the deployment itself.
“Where teams get stuck, in our experience, is treating instrumentation as an afterthought instead of a build-time requirement,” said Tim Skeen, executive vice president and CIO at Norfolk, Va.-based Sentara Health. “If usage, error rates and downstream workflow signal are not in the deployment on Day 1, retrofitting later is expensive and rarely fully catches up. Our approach is to make measurement part of the definition of ‘deployed,’ not a follow-on program.”
Some health systems are trying to close the ownership gap by tying monitoring directly to whether a tool is allowed to expand.
“This shouldn’t be different than any other piece of technology,” said Joel Klein, MD, executive vice president and chief digital and information officer at Edison, N.J.-based Hackensack Meridian Health. “When you deploy something, you need something — anything — to figure out if it’s adding value. It can be process-oriented, where you count how many times somebody’s using something, but that’s typically unhelpful. It needs to be value- or clinically oriented. We’ve started requiring these kinds of outcome measures for any non-trivial work we do, and we’ve set clear expectations that pilots will not get expanded unless they demonstrate those metrics moved positively.”
Ryan Smith, senior vice president and chief digital and information officer at Salt Lake City-based Intermountain Health, said the rigor required after launch is critical. “It’s building the discipline to monitor performance, learn from the data, and decide when to scale, adjust or to stop,” he said.
That discipline is also the first thing to erode as leadership attention moves elsewhere, said Jay Schultz, CIO of Sioux Falls, S.D.-based Sanford Health.
“It’s easy to celebrate a successful implementation,” Mr. Schultz said. “It’s much harder to continuously monitor performance, validate that models are still producing the intended outcomes, reassess value over time and establish clear ownership for ongoing optimization. As priorities shift, measurement can become inconsistent, making it difficult to determine whether an AI solution is still delivering the clinical, operational or financial impact it was intended to achieve.”
For Dr. Schnall, the fix starts before a tool is even deployed.
“I don’t think it’s talked about and thought about,” he said of the monitoring question. “It should lead — when thinking about an implementation.”
At the Becker's 11th Annual IT + Revenue Cycle Conference: The Future of AI & Digital Health, taking place September 14–17 in Chicago, healthcare executives and digital leaders from across the country will come together to explore how AI, interoperability, cybersecurity, and revenue cycle innovation are transforming care delivery, strengthening financial performance, and driving the next era of digital health. Apply for complimentary registration now.