The conversation around safe and effective use of artificial intelligence (AI) has largely focused on issues like bias in training data and bad outputs like hallucinations. But as pilots evolve into enterprise-wide deployments, our industry cannot overlook how human users interact with these systems.
“Automation complacency” is a psychological phenomenon that care organizations and clinicians must now begin to understand and prepare to combat.
Identifying the issue
Automation complacency is a natural human tendency to become overly reliant on or overly trusting of automated systems with increased use. This is not unlike the “alert fatigue” clinicians already experience, where constant notifications can dull responsiveness. And while this pattern occurs across industries, healthcare is naturally a high-stakes environment.
Let’s use ambient listening, a now widespread AI use case, to illustrate how seemingly minor instances of automation complacency have the potential to affect care decisions and outcomes. If automated documentation is not reviewed and thoroughly scrutinized each time it is used, nuanced inaccuracies could propagate through the patient’s record. While the note may look correct and bill properly, subtle errors could accumulate over time undetected and ultimately create a distorted clinical picture.
Medicine rarely deals in absolutes, and even the most advanced AI systems are not infallible. While clinical AI systems are more tightly trained and governed than consumer-grade models, edge cases, such as patients who don’t fit the textbook profile, will always test the limits of technology. Just as AI cannot replace the human connection clinicians offer, it cannot replace their deep clinical expertise and hands-on experience, either.
Curbing complacency
The strongest safeguards against complacency are human curiosity and healthy skepticism. AI-generated insights should prompt review and questioning, not automatic acceptance. This requires a multi-faceted approach involving both organizational strategy and individual diligence.
When organizations introduce complex AI systems, such as those for triage or clinical decision support, it’s crucial to run them alongside existing processes so you can compare the AI’s data and recommendations against established workflows. AI models are often trained on very large national or even international data sets that may not be completely representative of your organization’s community or your patients. Implementing phased and parallel rollouts can help reveal any potential pitfalls to address before they impact patient care.
Creative approaches to training can also help reinforce the need to actively evaluate AI-generated information. Because we often learn best through direct experience, organizations could create education with AI outputs that intentionally contain very subtle inaccuracies. Continuing with ambient listening as an example, a simulation could prompt clinicians to evaluate a series of AI-generated notes. If they miss those intentional slight errors in the documentation, the training system could flag that nuance and share best practices to help identify it in real clinical practice.
Additionally, organizations can encourage ongoing critical evaluation of AI through “forcing functions.” Processes should be built that require active provider engagement and approval instead of autonomously pushing data to the patient record. Simple reminders and system design that keep the clinician in the loop as the final line of review are essential. Existing controls like chart audits are also very helpful to ensure accuracy and quality while rolling out AI tools.
Scaling with care and confidence
With greater awareness of and education on automation complacency, hospitals and practices can help providers remain vigilant while also offering real relief through these assistive technologies. Because ultimately, AI adoption is not the goal, but rather a means to enable clinicians, empower patients and transform care organizations safely, sustainably and strategically.
Learn how Altera Digital Health is leveraging AI to elevate healthcare for everyone involved here.
Disclaimer: This article is intended for general informational and educational purposes only. The content reflects the author’s professional opinions and does not constitute legal, regulatory, or compliance advice. Readers should not rely on this article as a substitute for consultation with qualified legal counsel regarding their specific circumstances. Laws and regulations governing artificial intelligence in healthcare are evolving rapidly and vary by jurisdiction; organizations should seek independent legal guidance before implementing any AI-related policies or practices.
At the Becker's 11th Annual IT + Revenue Cycle Conference: The Future of AI & Digital Health, taking place September 14–17 in Chicago, healthcare executives and digital leaders from across the country will come together to explore how AI, interoperability, cybersecurity, and revenue cycle innovation are transforming care delivery, strengthening financial performance, and driving the next era of digital health. Apply for complimentary registration now.
