While some health systems are pivoting away from AI projects more quickly than they had in the past, others are taking a cleaner approach: not starting them to begin with.
As AI solutions proliferate, health systems are building evaluation and adoption processes that bring consistency to a constantly changing technology — and don’t let most proposals get past the idea phase.
“At Yale New Haven [Conn.] Health System, we have a very structured process of predeployment evaluation so that some projects never move forward in the first place, others don’t make it past silent validation, and others are turned off after we do a limited test deployment with a small group of expert users and a formal evaluation method,” said Lee Schwamm, MD, senior vice president and chief digital health officer.
Dr. Schwamm said rapid evaluation and termination are more straightforward when analyzing a package of tools in a single platform like an EHR or AI imaging system, where algorithms can easily be turned on and off without disrupting those that work well.
“Once we launch a project into production, we need to allow it to run for the amount of time needed to test its performance in a reliable way, or to detect a signal of harm concerning enough to suspend it or turn it off,” he said.
New York City-based Mount Sinai Health System is making faster, more disciplined decisions about which AI projects deserve to scale, said Lisa Stump, executive vice president and chief digital information officer.
“We no longer treat launching pilots as success,” she said. “Instead, we define value up-front and set achieving value as the bar for continuation. At Mount Sinai, this approach is enabled by unified digital and AI governance — from intake to evaluation, validation, deployment, and retirement.”
When Ms. Stump arrived at Mount Sinai in late 2024, the health system had been piloting a single AI scribe vendor with no clear direction. So she decided to evaluate three solutions over three months, with explicit metrics for accuracy, cost, technical integration and user satisfaction, establishing a team to analyze, implement and train the solutions and engage with the vendors.
Mount Sinai narrowed it down to two choices, then entered into a competitive negotiation before reaching an agreement a few weeks later. About 1,500 providers have adopted the tool over the past year.
“The pace of rapid pilot and decision to stop or scale will accelerate for the next few years while organizations move to adopt and generate value from these solutions,” Ms. Stump said. “Once the early wins and low-hanging fruit are achieved, I think we will see a plateau in this pace. The health systems that succeed will be the ones that scale what works, stop what doesn’t, and build the discipline to tell the difference early and across the tool lifecycle.”
At Kansas City, Kan.-based University of Kansas Health System, leaders recently abandoned an AI tool that purported to reduce MRI turnaround time and boost capacity.
“From the test period, we discovered that a significant aspect of their projected time savings was derived from workflow and process changes,” said Keith Sale, MD, vice president of ambulatory services and chief medical information officer. “Our radiology team had already optimized our workflows and processes, so the realized gains by adding only the AI functionality did not meaningfully improve our throughput.”
But overall, the health system hasn’t had to kill a lot of AI projects because it performs due diligence beforehand to vet and validate the applications. “We look at AI as a feature of an existing software application, as a product that addresses a specific system problem, or as a platform on which we can codevelop solutions,” Dr. Sale said. “When we stick to this framework, it allows us to make informed decisions and prevent utilizing resources that might eventually lead to abandoned projects.”
KU Health System also focuses on the basics by not deviating from its core governance and decision-making mechanisms, continuously improves AI literacy at all levels, and staying hyperfocused on only solving challenges unique to its organization.
“There is a growing tsunami of AI features, products, and platforms,” said Senior Vice President and CIO Chris Harper. “Determining how to navigate that wave so we do not get crushed by it is our primary goal.”
The health system takes a platform-first approach to AI but is always looking for innovation partners that can help it improve care and outcomes, he said.
“We’re being strategic and selective upfront and seeing projects through to completion because we’re making better-informed decisions together from the start,” noted Kristin Myers, executive vice president and chief digital officer of New Hyde Park, N.Y.-based Northwell Health.
This includes having a structured framework that ensures proposals solve real problems, align with Northwell strategies and come with clear success metrics.
“Our teams can now pursue high-impact pilots with the support and infrastructure they need to succeed,” Ms. Myers said.
Fort Myers, Fla.-based Lee Health has become more “disciplined” rather than “reactive” on AI, according to Executive Vice President and CIO Chris Akeroyd. For a contender to be greenlit, it must check some important boxes: clinical value, patient safety, data integrity, operational impact.
“When an initiative doesn’t meet those standards, we may decide not to move forward, reflecting our strong governance, rather than a diminished commitment to AI innovation,” Mr. Akeroyd said.
At King of Prussia, Pa.-based Universal Health Services, ideas never advance to full pilots or deployments if they don’t show feasibility and return on investment, said Vice President and CIO Eric Goodwin.
“That front-end governance is intentional,” he said. “It allows us to focus resources on initiatives that are scalable, sustainable and aligned with strategic priorities instead of pursuing technology for technology’s sake.”
Health systems are becoming more experienced and sophisticated after a few years of AI trial and error. And they are defining successes in advance, analyzing important factors like workflow integration, scalability and security — and pressing vendors to “prove it” — way before trying out use cases.
“Ultimately, the organizations that succeed with AI will not necessarily be the ones running the most pilots,” Mr. Goodwin said. “They will be the ones most effectively identifying where AI can create sustainable value on an enterprise scale.”
At the Becker's 11th Annual IT + Revenue Cycle Conference: The Future of AI & Digital Health, taking place September 14–17 in Chicago, healthcare executives and digital leaders from across the country will come together to explore how AI, interoperability, cybersecurity, and revenue cycle innovation are transforming care delivery, strengthening financial performance, and driving the next era of digital health. Apply for complimentary registration now.