The American Hospital Association is urging the FDA to make AI developers primarily responsible for monitoring the safety and performance of their medical devices, arguing that hospitals often lack the technical visibility and resources needed to evaluate the tools on their own.
In an Oct. 19 letter to FDA Center for Devices and Radiological Health Director Michelle Tarver, MD, PhD, the AHA outlined recommendations for regulating generative AI-enabled medical devices, including how the agency should assess risks, evaluate products before clearance and monitor their performance after deployment.
The association called for a flexible, risk-based regulatory framework that protects patients without creating unnecessary barriers to innovation.
“While hospitals and health systems support evaluation and monitoring activities, vendors and developers must be responsible for the ongoing performance, maintenance and integrity of the tools they market,” the AHA said.
The association said hospitals have established governance structures to evaluate AI technologies, but many lack access to the underlying information needed to independently assess how those tools perform.
According to an AHA survey cited in the letter, 74% of hospitals have multiple teams responsible for evaluating predictive AI, including senior leaders, department leaders and IT staff.
However, the association said the “black box” nature of many AI models can make it difficult for hospitals to identify flaws that could affect the accuracy of their outputs and recommendations.
The AHA recommended that the FDA establish post-market evaluation standards for developers, including performance metrics, thresholds that trigger additional evaluations and requirements for communicating performance information to healthcare organizations.
The association also raised concerns about the resources required to oversee AI tools, particularly at rural, critical access and safety-net hospitals.
Some of these organizations lack the staffing and infrastructure necessary to support ongoing monitoring, the AHA said, potentially widening disparities in access to AI technologies.
Beyond developer accountability, the association recommended several changes to the FDA’s proposed approach to regulating generative AI-enabled devices.
The AHA urged the agency to expand its risk assessment framework to account for the likelihood of errors, whether potential harm can be reversed, safeguards and the traceability of AI-generated outputs. The FDA’s proposed framework focuses on a device’s level of autonomy and the potential consequences of its actions.
The association argued that AI systems providing information or recommendations can still pose significant risks, even when they do not independently take clinical actions.
It cited research showing that certain commercial AI tools incorrectly triaged patients, including cases involving potentially serious medical emergencies.
The AHA also called for clearer labeling requirements to help patients and clinicians distinguish between informational AI applications and tools that take actions. It recommended that the FDA work with the Federal Trade Commission on guidance explaining how products should disclose their intended uses, level of autonomy and clinician involvement.
The association further urged the FDA to clarify which AI applications fall outside medical device oversight, including certain clinical decision support tools that allow clinicians to independently review recommendations and administrative applications.
The AHA also recommended aligning premarket clearance requirements with post-deployment monitoring, potentially allowing developers to seek clearance for additional AI applications through more streamlined pathways supported by ongoing evaluation plans.
The association said such changes could reduce regulatory costs while allowing hospitals to adopt safe and effective uses of AI more quickly.
The recommendations were submitted in response to an FDA discussion paper seeking feedback on the regulation of generative AI-enabled medical devices.
Becker’s 3rd Annual Fall Chief Pharmacy Officer Summit is the top tier event that the people leading the future of pharmacy convene. This 3-day event is built specifically for Chief Pharmacy Officers and senior pharmacy leaders. The sessions examine the workforce, financial and clinical priorities shaping pharmacy today. Becker’s is where candid discussions flourish and new ideas thrive. Becker's 3rd Annual Fall Chief Pharmacy Officer Summit
