When examining artificial intelligence in healthcare, Eric Cheng, MD, CMIO of UCLA Health Sciences, takes a systemic, data-driven approach.
In a recent interview on the “Becker’s Healthcare Podcast,” Dr. Cheng described the health system’s strategy to evaluate AI technologies thoroughly before implementation. This includes a focus on collecting high-quality, comprehensive data to ensure accuracy and minimize bias. He referenced a well-known epidemiological study that found only 20% of early mortality is attributable to healthcare itself, while the remaining 80% is linked to factors not routinely captured in clinical systems—such as patient behavior, social determinants of health, the built environment, and genetics. Access to this broader dataset is critical for clinicians and researchers to make informed treatment decisions — and the same principle applies when using AI.
“I’ll tell a story from my research days. We were wrapping up a trial. We had some difficulty getting the final survey results from some participants, so I asked a statistician, can’t we run some fancy imputation methods to overcome this,” said Dr. Cheng. “He looked at me and said, ‘You know, imputation is certainly better than no imputation, but there is no substitute for primary data collection.’ Likewise right now, if we don’t collect the right predictors, then the outcomes will be inaccurate due to omitted variable bias and AI can’t overcome that.”
AI can generate powerful predictions from available data, but for the technology to be truly effective in clinical care, the data itself must be both accurate and diverse. Ensuring strong foundational data is essential to operationalizing AI within health systems.
“If the notes are inaccurate, then certainly the predictions are inaccurate. But in addition to the quality data, the types of data should also be expanded,” said Dr. Cheng.
UCLA Health has applied this research-based philosophy to its selection of ambient listening technology. To objectively evaluate potential partners, the system launched an observational study and randomized 250 physicians into two groups — each testing a different vendor — and a control group. Physicians rotated between groups at set intervals, and researchers gathered detailed feedback about each tool’s performance and how it compared to the control condition.
“Based on my health services research days, I know there are methodological flaws, especially like self-selection bias, that can make it difficult to interpret observational studies,” said Dr. Cheng. “When you really need confidence in an answer, there isn’t a substitute for randomization.”
The two-month study is still under evaluation. Researchers are analyzing physician time spent in documentation, burnout levels from surveys, and the number of clinic days per week. They’re also considering patient safety metrics and conducting an economic analysis to guide decision-making before selecting a long-term technology partner.
“The overhead to conduct a randomized trial is too high to run for every AI implementation, but when a project has a potential seven-digit price tag, it’s worth slowing down the implementation just a bit so they can study it properly,” said Dr. Cheng. “I definitely encourage others to do that as well.”