Health systems have poured money and attention into scaling AI over the last several years — but, of course, not every pilot makes it that far.
For Tinu Tadese, MD, chief medical informatics officer at Boston Medical Center, the clearest AI failure pattern has been vendors that expect a health system’s IT staff to do the heavy lifting of turning a solution into a usable product.
“Some didn’t turn out the way we thought they would, because sometimes the vendors or partners come to the table with a platform expecting you to put in the elbow grease to make this what it’s supposed to be,” Dr. Tadese said during a July 21 Becker’s virtual panel.
One example came when operational leaders expected an experience closer to what they get from major EHR vendors — a mostly finished, packaged product — rather than months of internal development work. “That was not the way we thought it would go, and that was a big flop,” she said.
During the panel, Crystal Broj, enterprise chief digital transformation officer of Charleston, S.C.-based MUSC Health, recalled an AI pilot that dragged on for “several months.”
“It was an automation workflow, and it did automate a certain percentage of whatever was in the work queue that we wanted it to do,” she said. “The vendor thought it was a great success, because it was 30% or 40% of the queue. The problem was the stakeholders thought it should be 85%, because it was AI, and so AI should take that much work out of the system.”
The 85% figure, she said, came from other organizations’ results that weren’t a fair comparison — different use cases, different levels of production maturity, different settings entirely.
“At the end of the day, 40%, 50% was about as good as it was going to get, and they were not going to use this if they could not get 85% clearance on it,” Ms. Broj said.
“I think it’s really important when we do these types of pilots or rollouts that we have an expectation of what it is you’re expecting it to do,” she said. “It really has to hit that, or it’s not worth it for them to put in the time and the effort.”
Not every stalled project traces back to the vendor, according to Anita Iyenger, senior vice president, chief strategy officer and interim CIO at Portland, Ore.-based Legacy Health. Sometimes the obstacle is inside the health system’s own infrastructure.
“You can have a vendor who has a great product, is willing to work with you really well,” Ms. Iyenger said in the session. “I think in one of our projects, what we’ve run into is some Epic design and choices that we made 15 years ago as an organization, and now we’re going to have to go revisit that.”
She said the fix isn’t limited to the AI tool itself. “It’s become something that you have to change manage across the organization,” she said, “because you’re changing something in your Epic model that hundreds of people work with every day.”
At Danbury, Conn.-based Nuvance Health, part of New Hyde Park, N.Y.-based Northwell Health, the informatics team has tried to head off that kind of mismatch before a pilot ever starts, screening any new AI project against five criteria.
“We have developed what’s called guiding principles,” Nuvance Chief Medical Information Officer Albert Villarin, MD, said at the panel. “We created five specific guiding principles to say it has to be integrated in the EMR. If we have a project coming in, and they haven’t already integrated into our EMR, we don’t even look at them, because we don’t have the person-power or the finances or the time. We want to stand something up quickly, not over 9, 10, 12 months. We have 9, 10, 12 weeks to get something going.”
Beyond EHR integration, the criteria include a measurable financial, operational or clinical benefit, and grounding in evidence-based practice, he said.
“We get rid of the things that will add work to us, and only focus on those projects, AI or otherwise, that will be easy lifts, quote-unquote, for the team to facilitate best in our environment,” Dr. Villarin said.
Dr. Tadese put it more bluntly, describing what it feels like to realize a pilot wasn’t worth the investment only after the fact: “You get to the end of the project, that pilot, and you wish you never did it. Hindsight is 20/20; you’re looking at it now. It’s like, OK, I have to be smarter going into this.”
At the Becker's 11th Annual IT + Revenue Cycle Conference: The Future of AI & Digital Health, taking place September 14–17 in Chicago, healthcare executives and digital leaders from across the country will come together to explore how AI, interoperability, cybersecurity, and revenue cycle innovation are transforming care delivery, strengthening financial performance, and driving the next era of digital health. Apply for complimentary registration now.