During a fireside chat at Becker’s 10th Annual Health IT + Digital Health + RCM Conference, two leaders discussed a pressing challenge for health systems: translating innovative ideas into scalable digital health solutions.
The speakers — Dr. Simon Kos, global chief medical officer at Heidi, and Bhrandon Harris, MD, associate chief medical information officer at UI Health in Chicago — offered real-world insights into piloting new technologies, especially amid today’s generative AI boom and healthcare workforce constraints.
Both speakers brought perspectives from health IT and clinical operations to unpack what makes a pilot succeed or stall.
Five takeaways:
Note: Quotes have been edited for length and clarity.
1. Define success before launching
Dr. Kos emphasized that successful pilots start with clearly defined goals and baseline metrics. “If you don’t have the data to know where you’re starting from, it’s actually very difficult to prove how far you’ve come,” he said. He advocates for reframing “pilots” as either proof-of-concept or proof-of-value initiatives, stressing that organizations should only pilot technologies they are prepared to scale.
2. Rigor and readiness are essential
Both panelists underscored the importance of due diligence before initiating any pilot. Dr. Harris described a recent ambient listening pilot at UI Health that involved selecting two vendors, conducting a head-to-head crossover evaluation and maintaining momentum through monthly leadership updates.
“We probably spent more time defining and trying to create metrics than we did actually doing the pilot,” he said.
Dr. Kos praised the approach: “That’s an example of a great pilot.” He noted the importance of representative pilot groups across clinical settings to ensure scalability and relevance.
3. Clinician demand and organizational alignment drive adoption
Clinician interest can signal a technology’s readiness. Dr. Kos said that when clinicians clamor to join a pilot, it’s often a good sign of product-market fit. However, readiness also includes infrastructure, staffing and leadership support.
Dr. Harris acknowledged that engagement from senior leadership is important when designing a pilot but stressed the importance of also incorporating feedback from the front lines.
“It’s the people on the ground who are actually doing the work,” he said. “The wheels will fall off your very well-designed pilot if those people aren’t part of the conversation early on also.”
4. New technologies require new evaluation frameworks
Generative AI has accelerated the need for updated pilot models. Dr. Kos said the traditional approach — designed for fixed, deterministic tools — doesn’t apply to AI systems that evolve rapidly. He emphasized that governance structures for AI pilots must be informed by ethical and legal expertise.
5. Define ROI beyond dollars
While financial return remains critical, the speakers urged organizations to consider qualitative impacts, such as reduced cognitive load and lower burnout rates among clinicians.
The speakers made clear that pilots are far from obsolete, but they must be designed with intentionality, data rigor and a path toward real-world results.
