Nvidia’s new Open Secure AI Alliance, with 37 founding members including Microsoft, IBM, Cisco and Salesforce, is being framed as an industry response to a single incident.
For hospital and health system leaders, it’s worth reading as something narrower and more urgent: a warning about what happens when the AI tools running inside a health system can’t be trusted to work during the moment that matters most — an active breach.
The incident behind the alliance
Hugging Face, an open-source AI hub, disclosed July 16 that it had detected and contained an intrusion into its production infrastructure. When its security team tried to use commercial, closed AI models to analyze the attack, the models’ safety guardrails blocked the request — they couldn’t tell an incident responder from an attacker. Hugging Face instead ran GLM 5.2, an open-weight model, on its own servers to process more than 17,000 recorded attacker actions and reconstruct the timeline.
Nvidia cited that episode directly in announcing the alliance July 27, arguing that defenders need open, self-hosted AI tools they can run without waiting on a vendor’s permission.
Why this isn’t just a tech-industry story
Health systems have spent the past year-plus embedding closed models — ChatGPT Health, Oracle Health’s OpenAI-powered patient portal, ambient scribes, and Claude-based tools like Banner Health’s BannerWise — deeper into clinical and administrative workflows.
That dependency creates precisely the scenario Hugging Face described: a hospital’s security team, mid-breach, needing an AI system to sift logs or analyze malicious code, only to have the same vendor guardrails built to stop misuse block the defense itself.
Anthropic’s counter-argument matters too
This week, Anthropic CEO Dario Amodei also publicly rejected the idea that Anthropic wants open-weight models banned, while arguing the opposite risk is also real: models with no guardrails at all can be weaponized by attackers with no usage policy to violate.
Anthropic’s preferred fix — mandatory safety testing for all sufficiently capable models, open or closed — echoes the logic behind its Project Glasswing testing of Claude Mythos, which Becker’s has reported health system CIOs have asked to be included in.
Questions for hospital leaders to ask now
- Does your incident-response plan assume your AI vendor will be available and unrestricted during an active attack — and what happens if it isn’t?
- Do you have a vetted, self-hostable model or a vendor’s “trusted access” cyber program in place before an incident, not during one?
- How does your Business Associate Agreement or vendor contract address AI guardrails blocking legitimate security work?
The alliance itself won’t answer all of these questions overnight. But it puts a name and a roster of familiar enterprise vendors behind a problem hospital information security chiefs should already be planning for.
At the Becker's 11th Annual IT + Revenue Cycle Conference: The Future of AI & Digital Health, taking place September 14–17 in Chicago, healthcare executives and digital leaders from across the country will come together to explore how AI, interoperability, cybersecurity, and revenue cycle innovation are transforming care delivery, strengthening financial performance, and driving the next era of digital health. Apply for complimentary registration now.