Sponsored

Building trust in AI: Guardrails, governance and oversight

Advertisement

More than 70% of healthcare organizations either have already adopted or plan to implement generative AI. Expanding the use of this technology however, requires organizations to pay special attention to patient safety.

To learn more about the early application of generative AI in healthcare and opportunities for the future, Becker’s Healthcare recently spoke with Rhett Alden, chief technology officer at Elsevier.

Editor’s note: Responses have been lightly edited for length and clarity.

Question: As generative AI gains traction in healthcare, what do you see as the biggest opportunities and risks for clinical decision-making?

Rhett Alden: Generative AI can ease the burden of administrative tasks, freeing time for clinicians to spend with patients. It can also identify insights that lead to better diagnoses and interventions. Patients today are older. They have many comorbidities that drive challenges, such as drug-drug interactions, complex treatment models and more. AI can be beneficial in solving these problems.

With AI, given the right patient context, we can address health equity in a more granular way. This is important to Elsevier and to me. Using social determinants of health and other factors, clinicians can democratize care by customizing and personalizing treatment models.

At the same time, we worry about over-reliance on AI. Guidelines and governance are needed to ensure that what clinicians see is well-articulated. We must reinforce critical oversight and limit misinformation and hallucinations. Another big risk is bias — either unintentional or intentional. Many AI solutions are trained on biased information. Our role as a vendor is to disambiguate that.

Q: When evaluating new AI tools for clinical use, what principles or guardrails do you believe are most important to ensure safety, reliability and impact?

RA: Systems must be trustworthy and provide the right level of oversight. Most large language models are trained on data gleaned from the Internet. Some of that data is high quality, some isn’t.

Elsevier relies on retrieval augmented generation (RAG) architecture. We carefully curate the set of trusted content used for summarizations. More importantly, we expose them to clinicians so they can see that references are coming from trusted sources. Transparency is critical because it exposes information sources that are respected by clinicians.

Q: How are these principles informing Elsevier’s approach?

RA: In healthcare, it’s critical to have a guiding framework for AI. Elsevier has established a set of responsible AI principles, but what’s more important is the manifestation of those.

For example, we transparently expose all journals, books and guidelines used in the AI model. In addition, all information is copyright-cleared.

Within the RAG system, we surface the references used to construct summarizations. Clinicians can interrogate the source material and see the exact sections that were used. That transparency gives users confidence.

Another critical element is our novel “clinician-in-the-loop” testing framework, which utilizes subject matter experts from different disciplines. We recruit clinicians to determine whether AI responses are correct, comprehensive, complete and helpful at the point of care. These clinicians also evaluate the potential for clinical harm. This testing framework is used for every major product release. The preliminary results of the framework are outlined in a press release and more details will be published in the Journal of the American Medical Informatics Association.

Q: What specific steps can healthcare leaders take to build greater clinician confidence and understanding around AI technologies?

RA: Although Elsevier’s products aren’t FDA-regulated medical devices, we use similar risk controls to deliver safe and effective solutions to clinicians. Information quality, bias and health equity are huge concerns. Our approach combines training, user interface design and collaboration with bioethics teams.

By establishing appropriate guardrails, we ensure that clinicians understand the boundaries of use. We want a UI model that preserves clinicians’ clinical judgment. While the system provides high-quality responses, it can’t make autonomous decisions on the clinicians’ behalf.

Q: Looking ahead, what developments in generative AI and clinical decision-making are you most excited about — and how can leaders best prepare to integrate these tools responsibly?

RA: I’m excited about AI integration with the electronic medical record.

With generative AI models, we rely on clinicians to develop a prompt, such as, “What is effective management for an elderly patient with hypertension?” However, unless manually added, that prompt may not include that the patient has early-onset dementia or neuropathy due to diabetes. That longitudinal information is available in the electronic medical record.

AI agents for specific, high-friction points in healthcare is another exciting area. Transitions of care are a high-friction point where AI agents offer great potential; that’s because information is often lost at those points in the patient journey. AI agents can prevent loss of information and can reduce the risk of complications. Other opportunity areas for AI agents include referrals and physician order entry.

Q: Is there anything else you’d like to touch on?

RA: AI is moving very fast. The challenge is determining what innovations will hit the mark. Strong partnerships between innovators like Elsevier and healthcare institutions are critical. You need clinicians in the room to address the complexity and ensure that patient needs are front and center.

To learn more about Elsevier’s AI evaluation framework, download the white paper.

Advertisement

Next Up in Health IT

Advertisement