Health systems are spending heavily on AI, but proving what it delivers remains a challenge. Only 4% of systems have scaled AI with measurable outcomes, and 4 in 5 IT leaders struggle to measure AI’s return, according to an April report from Qventus.
That raises a question for CEOs about which AI metrics deserve their attention. Becker’s asked six health system CEOs what they should track about the technology itself. Several said AI does not necessarily need its own daily set of KPIs. They pointed instead to tracking when problems arise and whether AI is improving performance.
Note: Responses have been lightly edited for length.
Bob Garrett. CEO of Hackensack Meridian Health (Edison, N.J.): We are tracking KPIs around various AI initiatives, and we’re actually building it into our leadership incentive programs as well, to be sure that we’re developing AI in a responsible way, but one that we can measure and that can benefit patients, whether that be in terms of enhanced patient outcomes or patient experience.
This past year, we rolled out an AI agent that is calling patients 48 to 72 hours after discharge. As a result of those conversations that the AI agent is having with our discharged patients, it’s avoiding readmissions. As a matter of fact, our readmissions are down 11% as a result of these types of conversations. They’re also referring some of these patients who might be experiencing complexities or complications to caregivers and to clinicians on an expedited basis, and it’s also enhancing our patient experience.
So it’s just one example where there are several KPIs being monitored really on a daily basis, and those are the types of AI initiatives that are showing the most promise.
Josiah “Sy” Johnson. Vice President and CEO of Samaritan Health Services Valley Region Hospitals (Corvallis, Ore.): To maintain simple awareness and fluency, CEOs need to monitor the general news and general tech industry news as well as healthcare industry tech news. If possible, add to that a portfolio of three to five credible AI historians, prognosticators, tech leaders who don’t agree about the future and look for their published thoughts on at least a monthly cycle.
On that foundation, areas of AI impact most important to your circumstances can be identified. For the fortunate few who have friends in the tech industry involved near the forefront of the AI world, hang out with them when you can and listen a lot. The real forefront of what’s going on probably won’t be published for a while.
G. Mark O’Bryant. President and CEO of Tallahassee (Fla.) Memorial HealthCare: Leaning heavily on Epic’s continued investment in AI dashboard assist tools, our leaders are better equipped to leverage information to drive performance. After first passing the test of data integrity, AI can do the heavy lifting of running and correlating reports, turning disparate threads of info into a complete and powerful fabric. Looking at changes in trends or advancement towards targets, questions arise more quickly. OR utilization is down, patient experience is beginning to drag in a certain facility, why? While not always at a CEO level, things could roll up. As they should. Focus and attention shifts. And we all know that attention is the currency of leadership.
Ettore Palazzo, MD. CEO of EvergreenHealth (Kirkland, Wash.): Tracking AI usage and adoption alone doesn’t tell us whether we are improving patient care or the experience of the people providing that care. Adoption is important, but it is primarily an operational measure, not a measure of value or impact.
For EvergreenHealth, the more important question is: What changed, and did that change improve our performance? Whether the change involves AI, new technology, a redesigned workflow, or a different care model, we should understand when it was introduced and then monitor the core measures we already use to evaluate performance: quality and safety, patient experience, financial performance, and employee engagement.
Ambient documentation is a good example. Our teams tracked adoption and worked with providers to ensure the technology was functioning as designed. But the more meaningful question was what it did for our providers and patients. We saw improvements in both provider and patient satisfaction following deployment.
We need to connect AI and other innovations to the performance framework we already use. Yes we should understand where AI is being deployed and whether it is being adopted, but the real measure of success is whether it improves the outcomes that matter most to EvergreenHealth: safer, higher-quality care; better patient and employee experiences; and stronger financial and operational performance.
Timothy Pehrson. CEO of Integris Health (Oklahoma City): What needs daily attention reaches us through our Daily Tiered Escalation Huddles, where leaders see and learn about any performance issue that requires their attention. Our job is then to remove barriers or deploy resources when front-line staff and leaders can’t solve the problem themselves. AI is no different from any other KPI: our leadership team learns about AI operational issues in the DTEH each day. For example, from time to time, we hear that our ambient documentation tool is not working as it should, or that there was an AI-related safety event or near miss.
AI KPIs and strategy are something we review regularly, but with a longer-term view rather than a daily one. For example: Do we have a complete AI inventory, including AI embedded in existing products from vendors we never bought AI from? And what share of the AI in use has a named owner, a documented purpose, a completed risk review and a monitoring plan?
Because AI is still new, our AI governance council measures our progress against the NIST AI Risk Management Framework to keep our efforts aligned with national best practices.
Chris Van Gorder. President and CEO of Scripps Health (San Diego): My top three, which link to my top three “P’s” at Scripps:
Patients: AI-related safety events and measurable quality/access improvements
People: Hours returned to caregivers and reduction of administrative burden
Performance: Validated financial benefit versus AI operating costs
I’d also suggest creating an AI Risk & Trust Index, Green/yellow/red combining clinical safety events, hallucinations or material inaccuracies, privacy/security events, patient complaints and significant clinical overrides so I would know if AI did anything yesterday that should concern me?
