As AI becomes increasingly embedded in healthcare operations and strategy, organizations are working on how to govern its use responsibly amid evolving regulations and rising risk. Becker’s Healthcare spoke with Jennifer Richards, PhD, PharmD, JD, senior director of product management at URAC, a nonprofit that developed the first Artificial Intelligence in Health Care Accreditation Program, about the state of AI oversight, common governance gaps and how accreditation can help organizations move forward with confidence.
Note: Responses have been edited for length and clarity.
Q: AI is now foundational to healthcare strategy, what key trends, concerns or opportunities are you seeing around AI oversight?
Jennifer Richards: It starts with developers: they need to be compliant with applicable laws and regulations, and they need to be transparent with the organizations using their tools. That transparency is what enables health systems to build an effective governance plan, because oversight must be tailored to the specific system and how it was designed and deployed.
Health systems should be asking questions like: What data was used to train the model? How was it configured at the outset, and for what intended use? What are the known limitations, and what guardrails are in place?
From there, it’s about making sure teams have the right data and documentation to interpret outputs, validate that results make sense clinically, and monitor performance over time. And just as important is having a clear plan for what happens when something doesn’t add up. For example, how issues are flagged, investigated, documented, and corrected.
Q: Many organizations are still building out AI governance programs. Where are the most common gaps today?
JR: What we’re seeing most is that AI feels scary, and people don’t know where to start. Healthcare organizations manage data all the time and already have governance plans, even if they don’t always think of them that way. AI feels totally foreign but it isn’t entirely new, it’s just different. A good early strategy is to look at how you’ve governed data before, what data is being used, how it’s collected, where it’s stored. Then you start filling in what’s different with AI and that’s how you build a more complete data management and governance plan.
Q: With the regulatory landscape still uncertain, how can accreditation help organizations establish governance while maintaining momentum?
JR: Our goal is to set up frameworks and guardrails so organizations can be successful as things change. Regulations are limited in how broadly they can govern across different types of AI, especially as new forms like generative and agentic AI emerge. Accreditation helps organizations understand the foundational quality frameworks they need so they can introduce new tools into their processes responsibly. It’s also national, helping organizations navigate compliance across states. We don’t tell them whether they are compliant, but we give them the structure to understand what they need to do long-term.
Q: On the operational side, what are leading organizations prioritizing to build trust in AI and scale adoption responsibly?
JR: This isn’t new, physicians initially mistrusted EHRs, too. There’s a lot that organizations can learn from that experience. One priority is being clear about what outcomes users should expect and training people to recognize whether those outcomes make sense. Another is clearly outlining the “what if” pathways, what if the output doesn’t look right, what if someone wants to override the system, and how clinical responsibility and licensure factor in. When those pathways are clear, clinicians better understand that AI is a tool, not a replacement.
Q: Any final thoughts you’d like to share with readers?
JR: AI is here. It’s not a question of “if” it will happen, it’s already happening. Organizations should be proactive about setting up quality frameworks and processes around AI so they can move forward and use it effectively as a tool to streamline workflows, reduce clinician burnout, and, ultimately, improve patient care.
At the Becker's 11th Annual IT + Revenue Cycle Conference: The Future of AI & Digital Health, taking place September 14–17 in Chicago, healthcare executives and digital leaders from across the country will come together to explore how AI, interoperability, cybersecurity, and revenue cycle innovation are transforming care delivery, strengthening financial performance, and driving the next era of digital health. Apply for complimentary registration now.
