Most health systems have a story about an AI pilot that never went anywhere. The tool was promising. The vendor was convincing. The proof of concept checked out. And then — nothing. The project lingered, the costs accumulated, and the clinical or operational problem it was supposed to solve stayed unsolved.
Getting from pilot to measurable, sustained value requires more than good technology. It requires a framework. Four executives — Drew Smith, chief data and analytics officer at Wilmington, Del.-based ChristianaCare; Kalyani Gopalan, executive director of analytics at Presbyterian Healthcare Services in Albuquerque, N.M.; T.Y. Alvin Liu, MD, who directs an endowed AI center at Johns Hopkins School of Medicine in Baltimore and leads AI implementation and governance on the health system side; and Darrell Keeling, chief technology officer at Bronson Healthcare in Kalamazoo, Mich. — gathered for a panel at the Becker’s 11th Annual IT + Revenue Cycle Conference outlined five concepts they say determine whether AI investments pay off.
1. Define AI and match ROI to the right type. Not all AI is built the same, and the way an organization measures return depends entirely on which kind it’s deploying. Mr. Smith draws a sharp line between two categories: generative AI, built on natural language processing and convolutional neural networks and classical machine learning models. The distinction matters for how ROI gets calculated.
“Generative [AI] is a bit more difficult to decipher and therefore difficult to actually establish — like true hardcore finance nerd ROI,” Mr. Smith said. “Machine learning models are a bit more classical, but they are often easier to build the baseline for and therefore the lift, and then therefore some sort of ROI calculation, even if the return is not financial.”
Dr. Liu offers a three-part framework that applies regardless of which type of AI is in play: clinical ROI (better patient outcomes), operational ROI (improved efficiency) and financial ROI. His rule of thumb is that any implementation worth pursuing should demonstrate positive returns in at least two of the three categories.
“If you are talking about non-clinical AI applications, the vast majority of the low-hanging fruit will be improvement in operational efficiency. So that should be pretty easy to demonstrate. However, it’s also very important to translate those operational KPIs to believable financial KPIs,” Dr. Liu said.
Ms. Gopalan frames AI more broadly, from statistical analysis supporting treatment decisions all the way to generative tools that draft communications or surface next-best actions. The most immediate ROI opportunity lies in personalizing care and getting earlier interventions in front of patients with chronic conditions.
2. Track hidden costs and plan for sustainability. The line items that kill AI programs rarely appear in the original business case. Mr. Keeling points to two that health systems routinely underestimate: per-user platform licensing costs that compound across large workforces and token economics, which is the cost of every query sent to an AI model.
“I believe tokens are going to be one of the bigger costs. I think we’re going to see it similar to what we saw when we went to the cloud, when we pushed a lot of systems out to the cloud and we let a lot of costs run really fast,” Mr. Keeling said.
Ms. Gopalan flags a different category of hidden cost: the technical labor that accumulates on the back end after a vendor product goes live. As AI vendors update their tools, someone on the health system’s data science team has to review, optimize or rebuild the code. That maintenance burden rarely shows up in the procurement conversation.
“I can see that there is going to be hidden costs in the implementation from the data science perspective, where we start to use it,” Ms. Gopalan said.
Mr. Smith adds two more: the cost of wasted effort on pilots that fail to sustain and opportunity cost of the work that doesn’t happen because resources are committed elsewhere. He cites a prior employer’s experience in which only two of 10 AI tools deployed were still in use after a year.
“Every time you work on this, you cannot work on that. We are all in finite resource land, and so the rigor for me comes in to put the right process by which you’re making sure things can be vetted quickly, not bureaucratically, assessed against your entity strategy and then ruthlessly implemented quickly or killed,” he said.
3. Manage user adoption and the culture around AI. A technically sound AI model that clinicians don’t use is not an asset, it’s a cost center. Dr. Liu pointed to the gap between FDA clearance and real-world adoption as one of the most underdiscussed problems in healthcare AI: more than 1,500 AI-enabled medical devices have been cleared, but the science of how to validate and implement them in ways that generate trust remains unsettled.
“On the clinical side, the major barrier is people just don’t trust it. Clinicians will say, I’m not sure if this works or not, then I don’t want to use it because I don’t want to create harm,” Dr. Liu said.
Ms. Gopalan describes the trust problem from the analytics side. A sepsis predictive model her team developed can identify risk within a seven- to 10-day window — but its value is limited because it sits with the analytics team rather than in clinicians’ hands.
“The ROI really comes from when that model gets in the hands of a physician or a clinician and also when that model gets to be used by whoever goes to the doctor and sees the benefit of it,” she said.
Mr. Smith urged organizations to treat the pilot phase as a learning exercise about users, not just technology. The goal of a test deployment is to understand how decision-makers and care providers perceive the model so that when a full rollout happens, the design amplifies what users value and directly addresses what erodes their confidence.
The cultural piece compounds the adoption problem. Health systems rarely kill a model that isn’t delivering. The inability to declare something a failure and move on means underperforming tools accumulate and drain resources while producing no value.
“Rarely do people kill a model, and that to me is the biggest risk and cultural shift we may need to do,” Ms. Gopalan said.
4. Account for the cost of inaction. The risks of moving too fast on AI dominate most boardroom conversations. The risks of moving too slowly rarely get the same attention. Dr. Liu draws a parallel to financial services digitization, where a clear performance divergence between early movers and laggards emerged roughly a decade after industry-wide adoption began. He expects healthcare to follow the same curve.
“Integrated health systems that deploy AI at scale in the correct way will outperform organizations financially that don’t. That has a lot of downstream effects all the way down to patient care,” Dr. Liu said.
Ms. Gopalan argued that healthcare’s data infrastructure lag compounds the urgency. Despite generating more data per patient than virtually any other industry, health systems haven’t evolved the underlying data architecture fast enough to act on it, which means the gap between what AI could do and what it is actually doing continues to widen.
Many health systems don’t have the luxury of pausing to get everything organized before moving forward.
“There is an opportunity cost,” Ms. Gopalan said.
Mr. Keeling connected the inaction problem to technical debt. Aging infrastructure — from EHR systems to physical equipment — has to be addressed in parallel with AI deployment. Skipping that foundational work doesn’t avoid the cost; it defers it and makes AI harder to sustain when it arrives.
5. Position AI as an enabler, not a replacement. AI will automate tasks and some job displacement is likely over time. But the more immediate and strategically useful framing is AI as a tool for doing what the organization already does better. Mr. Smith’s plan has been to deploy AI against strategic advantages over the most hyped products.
“You need to think about your AI strategy from a focus point of view, and that focus is already done. Hopefully your organization probably has already built a strategic roadmap that is the starting point of your AI journey and if you go anywhere else, you’re just going for distractions and little bits of value that aren’t going to add up to anything,” Mr. Smith said.
Dr. Liu has started defining jobs by purpose instead of task. In revenue cycle, the purpose is maximizing the organization’s financial health. AI may automate many of the tasks that currently serve that purpose of submitting claims or following up on denials but the purpose itself is the same.
“AI will automate some of your tasks, but this is your chance to really upgrade your skill set to move you up the value chain,” Dr. Liu said.
Mr. Keeling emphasizes the opportunity AI creates for high performers who have been underutilized. Health systems that take time to understand the actual tasks and volume their people manage rather than thinking at the title level can use automation to free those employees for more complex and rewarding work.
Looking ahead, Dr. Liu sees the ROI conversation itself changing shape. Rather than evaluating AI tools in isolation, health systems will increasingly think in terms of human-AI ratios at the business unit level, toggling the percentage of AI involvement in a function and modeling the cost and output implications in real time.
“We will be in a place — even now, definitely in two years — where you can decide to be 100% human, 100% AI and somewhere in between,” Dr. Liu said. “Then you can just adjust the cost and do your decision.”
There are a variety of new roles and opportunities for job descriptions to evolve with AI.
“For those who say AI is going to replace people, look at how many people are here talking about AI,” Ms. Gopalan said. “I really do think it’s just going to change the way we work.”
