When an AI vendor walks into Mass General Brigham with a bold ROI claim, the conversation doesn’t start with a pilot proposal. It starts with a form.
Rebecca Mishuris, MD, chief health information officer at Somerville, Mass.-based Mass General Brigham, requires every AI vendor to complete a standard assessment questionnaire before any procurement conversation advances. The form ties directly to the system’s responsible use of AI principles and covers how a model works, how it was validated and whether the vendor looked for bias and equity issues during development.
“Even being able to fill out that form is a non-negotiable,” Dr. Mishuris said.
That kind of structured scrutiny is becoming a defining feature of AI procurement at health systems nationwide. As an increasingly crowded vendor market produces increasingly confident claims, IT and informatics leaders say they are building more formal evaluation processes — to hold vendors accountable for what they promise and to share the risk when no one has proven results yet. Across health systems, the pressure to translate AI spending into measurable gains is intensifying how procurement decisions get made.
When Mass General Brigham breaks new ground, it must sometimes build its own evidence base. The system ran what was at the time the largest ambient documentation pilot in the country, with 800 providers, and published the initial performance data where none had previously existed at that scale. In those circumstances, Dr. Mishuris looks for vendors willing to share the accountability and risk, believing the arrangement produces better outcomes for both parties and sets the precedent for performance accountability.
“It leads to a better product,” she said. “It leads to us really understanding how to incorporate it into our environment, and it leads to that kind of precedent that the vendor needs to go talk to other companies, and that we need internally to demonstrate that there’s value in scaling.”
Kevin Bidtah, chief information and inclusion officer at Evergreen Health, a federally qualified health center with seven sites in Buffalo, N.Y., starts the accountability conversation before a contract is signed. Scope of work, expected outcomes, timelines and the consequences of missed commitments are negotiated in writing. Then he asks a question vendors don’t always hear.
“When this fails, where are you headed? What are we going to do? How are we going to handle this failure?” Mr. Bidtah said. “Because not everything is going to just work right out of the box. Most things don’t.”
His vetting process extends beyond the negotiating table. Before entering any significant vendor relationship, Mr. Bidtah reaches out to peers who have worked with the same company — not to learn about features, but to understand what the working relationship actually looked like. That accountability standard holds for the lifecycle of the product’s use.
“Go-live isn’t the end, isn’t the finish line,” he said. “It’s beyond that.”
Stephanie Hines, CIO of Valleywise Health in Phoenix, draws a clear line between vendors who can earn a contract and those who can’t. For any product that isn’t being co-developed with the health system, she expects vendors to show their work by outlining their training methodology, model maturation and outcomes data from comparable organizations.
“I want to see the evidence,” she said. “I want to see what the results have been. I want to see how you’ve trained your model. I want to see behind the curtain essentially of how you have matured your product.”
A vendor who can’t provide that transparency is sending a clear message.
“Anyone who has a bold claim around an ROI but they can’t articulate that information, they can’t share what’s behind that curtain — then that’s obviously a red flag,” Ms. Hines added.
As the market matures, Ms. Hines expects vendor performance to become easier to benchmark and weaker products to fall away. In the meantime, she sees shared accountability for outcomes — wins and losses — as the clearest signal a vendor is worth a long-term relationship.
Joseph Izzo, MD, chief medical information officer at San Joaquin General Hospital in French Camp, Calif., said the strongest vendor presentations he has seen share one trait: the vendor doesn’t make the case to IT alone. They understand all stakeholders within the hospital’s ecosystem and can draw a clear value line for each.
“Being able to actually sit down and have a vendor meet with our stakeholders, make those convincing arguments alongside me — I think it goes a long way,” Dr. Izzo said.
At the Becker's 11th Annual IT + Revenue Cycle Conference: The Future of AI & Digital Health, taking place September 14–17 in Chicago, healthcare executives and digital leaders from across the country will come together to explore how AI, interoperability, cybersecurity, and revenue cycle innovation are transforming care delivery, strengthening financial performance, and driving the next era of digital health. Apply for complimentary registration now.