Health systems accumulating AI pilots faster than they can evaluate them are not building capability; they’re building technical debt and clinician fatigue. Nirav Shah, MD, associate CMIO for AI and innovation at Evanston, Ill.-based Endeavor Health, is striking that balance. The health system has more than 100 active AI tools currently deployed across the system.
Managing that portfolio without a structured filter means ceding the evaluation process to vendor optimism and internal momentum. Dr. Shah’s alternative is a framework built on a standard his team applies uniformly before any pilot advances to enterprise scale.
“To cut through the noise of the AI hype cycle, we can’t rely on gut feelings or vendor promises,” Dr. Shah said. “In our system, which has over 100 active AI tools, we evaluate the readiness of any pilot through a conceptual compass or framework built around three fundamental features.”
The first is model performance. The tool has to perform accurately without adding stress to the team in the real-world clinical and administrative environments. Just doing well in a controlled environment is not enough to scale.
“Does the model actually perform reliably in our specific clinical environment or administrative environment, delivering a high quality signal without drowning our teams in alert fatigue, false positives, or hallucinations?” Dr. Shah said.
The second lens is technical integration and workflow fit. No matter how accurate a model is, a tool that cannot embed naturally in how clinicians actually work creates friction rather than reducing it.
“Can this tool integrate into our existing technical stack, and does it slide naturally into the daily workflow of our end users without requiring disruptive bespoke workarounds?” he said.
The third lens is value, which Dr. Shah breaks into two parts: potential value and real value. The new tools have to include both.
“Is it solving a valuable problem that needs to be solved?” he said. “We must see a clear, measurable reduction in cognitive burden, a drop in administrative costs for a tangible clinical outcome.”
Dr. Shah anchors the value question to the quintuple aim, extending the traditional triple aim to include clinician well-being and health equity. A tool that does not measurably advance at least one of those dimensions does not clear the bar.
“If a pilot doesn’t meet these criteria, we optimize, pause it, or kill it,” Dr. Shah said.
The principle behind the entire framework is what Dr. Shah calls scaling at the speed of trust, a standard defined not by timeline but by demonstrated performance across all three lenses.
“We pilot to learn, but we only scale at the speed of trust, and that trust is earned in solving real problems, accurate models, seamless workflows, and value creation,” he said.
Becker’s 3rd Annual Fall Chief Pharmacy Officer Summit is the top tier event that the people leading the future of pharmacy convene. This 3-day event is built specifically for Chief Pharmacy Officers and senior pharmacy leaders. The sessions examine the workforce, financial and clinical priorities shaping pharmacy today. Becker’s is where candid discussions flourish and new ideas thrive. Becker's 3rd Annual Fall Chief Pharmacy Officer Summit
