Health systems have grown more disciplined about vetting AI before it goes live. What comes after has proven harder to standardize.
Post-deployment monitoring responsibilities frequently become ambiguous once an AI tool is deployed. Ownership might shift without a clear designated holder, with the first sign of trouble often arriving as a frontline complaint rather than a dashboard alert. The same gap shapes how vendors treat their health system relationships after the launch event is over.
Two CMIOs who have built formal expectations around that problem describe what they require from their partners and what they’re willing to do when those requirements aren’t met.
Nirav Shah, associate CMIO for AI and innovation at Evanston, Ill.-based Endeavor Health told Becker’s the mental model for an AI vendor relationship starts with biology.
“The days of a technology vendor dropping software on a health system, cutting a ribbon, and walking away are completely over,” Dr. Shah said in an interview for the “Becker’s Healthcare Podcast.” “AI is a living organism. Models drift continuously as clinical population shifts, and models and workflows constantly evolve.”
A model trained on a health system’s patient population at one point in time will encounter a different population as demographics shift, as clinical protocols change and as the system expands or contracts services. A vendor monitoring only initial performance metrics misses the degradation that follows. The clinical team is the first to feel it.
“Post go-live, a partner must remain accountable for continuous model validation and workflow adoption,” Dr. Shah said. “If a model’s accuracy degrades over time, or if our clinical or administrative teams start building manual workarounds to bypass the tool, the vendor must be in the trenches with us fixing it.”
Workarounds are the signal Dr. Shah watches most closely. When a clinical or administrative team quietly routes around a tool rather than through it, the tool has already failed its core purpose, even if deployment metrics still show it as active. Endeavor’s response is to surface the problem formally and give the vendor a defined window to respond.
“If we share concrete workflow issues or performance degradation and we see zero road map velocity or improvement within a fixed amount of time, typically a few months, which is the amount of time it takes to validate a model, we pivot,” he said.
The logic behind a months-long window rather than an open-ended one is deliberate. Validation timelines are knowable, and a vendor that cannot demonstrate measurable improvement within them signals the solution treats the health system’s clinical environment as a development environment.
“Health systems shouldn’t fund a vendor’s R&D on our clinical time, on our patients’ clinical time,” Dr. Shah said. “The relationship is only valuable if the tool continuously earns its place in our toolkit and in our workflow.”
Danny Lee, MD, chief medical informatics officer at Johns Hopkins Medicine Community Physicians, part of Baltimore-based Johns Hopkins Medicine, frames the vendor relationship as something that begins rather than ends at signature.
“Our relationship with our vendors doesn’t end with us signing a contract,” Dr. Lee said. “It is only the beginning, and we expect our vendors to be fully engaged, to interact with us, to get feedback, and continually improve the product. We don’t expect the product to be completed or finished when we sign the dotted line, but we expect to see improvement as we move along.”
The health system collects data and presents it to providers in dashboards in the most effective ways possible.
“With one of our current AI vendors, we now have the ability to do real time clinical decision support at the time that we’re seeing the patient to help synthesize all the data before the physician sees the patient and then in real time during conversations with the patient,” said Dr. Lee. “This gives the provider little nudges and possible suggestions, and then if during that conversation questions arise, the CDS tool can actually pull up that answer without ever leaving the EMR.”
Dr. Lee said his team is looking for clinical facing AI products that reduce provider burnout, making their lives easier and making clinicians more effective. Patient safety also factors into the decision-making process.
“If [the tech] addresses any of those things then it will become self evident and we will enroll on a pilot at that point,” he said.
Becker’s 3rd Annual Fall Chief Pharmacy Officer Summit is the top tier event that the people leading the future of pharmacy convene. This 3-day event is built specifically for Chief Pharmacy Officers and senior pharmacy leaders. The sessions examine the workforce, financial and clinical priorities shaping pharmacy today. Becker’s is where candid discussions flourish and new ideas thrive. Becker's 3rd Annual Fall Chief Pharmacy Officer Summit
