‘Not everything was a 10-month IT project’: URMC’s AI build-a-thon drew 550 and tested its governance model

Advertisement

Health systems building AI governance frameworks tend to work from the top down. Policies are drafted, risk matrices approved, vendor contracts reviewed and then the executive team signs off on the deal. At Rochester, N.Y.-based University of Rochester Medicine, the governance model got its most revealing stress test from an unexpected direction.

Lisa Nelson, PharmD, associate vice president of IT applications, chief applications officer and co-CIO at the eight-hospital academic health system, has spent much of the last two years building the infrastructure underneath URMC’s AI strategy. The system has data centers of excellence, a clinical AI governance model and a partnership with AI platform Qualified Health to manage the organization’s secure AI environment. When the system’s senior director of AI and automation proposed a build-a-thon, which would be six weeks of classes and drop-in sessions on how to build an AI agent, Dr. Nelson’s team planned for modest turnout.

“We thought we would get 100 folks,” Dr. Nelson said during an interview with the “Becker’s Healthcare Podcast.” “We had over 550 people join.”
The participants were not predominantly technology staff; physicians, nurses and educators from URMC’s schools of nursing and medicine joined as well. IT staff were present but oriented toward internal operations projects while the clinical and administrative problems driving most of the participation belonged to people outside the IT department.

“It was just the exercise of being able to put those tools in the hands of our clinicians, our nurses,” Dr. Nelson said. “It was really just a really cool place to bring a community of practice together to learn how to build in something with everyone being able to kind of do it at the same time.”

Participants had the option at the end of the six weeks to submit agents they believed were worth further evaluation, and 50 did. That 10% conversion rate from a group that began as an open invitation to learn became far exceeded expectations.

“10% of the folks involved in this felt like they had something that was worth building on and improving and maybe potentially sharing,” she said.

The agent submissions built across clinical, administrative and educational functions and spanning a wide range of risk levels arrived on a timeline that the standard IT project pipeline wasn’t designed to handle. The build-a-thon had inadvertently created a real-world, diverse portfolio of AI tools built by end users rather than vendors, all requiring a determination about how to handle them.

“It allowed us to kind of test our clinical AI governance model to understand how do we use governance when it needs to be, by using a risk matrix,” Dr. Nelson said. “But then when there’s something that’s really just looking at approved guidelines that we’ve already published on the internet and just bringing all the information together, how do we fast follow? Not everything was a 10 month IT project.”

The practical result was a governance distinction URMC is now building around: clinical AI tools that touch patient-facing workflows at meaningful risk require full risk-matrix review. Agents that surface already-published, internally approved guidelines, drawing existing information together without introducing new clinical judgment, can move faster. The build-a-thon forced that line to be drawn in practice rather than on paper.

It also surfaced organizational readiness that pre-existing assumptions had underestimated. The 550-person enrollment challenged a premise that health system IT leaders often carry into AI strategy discussions, showing that capability must be built through the IT organization before staff can meaningfully engage. At URMC, the demand was already there. The build-a-thon gave it somewhere to go.

Looking into the future, Dr. Nelson sees a lot of possibility for growing AI knowledge and applications.

“The clinicians, frontline administrators and other folks need to get their data, but also they have helped us realize that if we democratize in IT, we can take care of the platform and we understand all the regulations, but the folks who are closest to the data, our research groups where we have some folks that can pull their own data and know their data, they’re the center of excellence,” said Dr. Nelson. “They help with the blue ribboning. If people need data either for a proposed research project, quality assurance or something that’s actually been funded people can go there and access the data in the electronic health records.”

Advertisement

Next Up in Artificial Intelligence

Advertisement