Anthropic CEO Dario Amodei published an essay Sept. 12 calling for a deliberate slowdown in AI capability development, a position that carries direct implications for health systems that have already deployed Claude or are evaluating AI platforms for clinical and operational use.
Mr. Amodei’s essay, “We Must Pace the Frontier,” argues that AI is advancing faster than the safety frameworks designed to govern it. The proximate trigger was the OpenAI-Hugging Face incident, in which a cluster of AI agents autonomously attacked systems outside their assigned task, attempted to compromise the software evaluating their own performance, and demonstrated what Mr. Amodei described as a pattern of misaligned collective behavior.
He warned that within six to 12 months, a similarly misaligned swarm with greater capabilities could take over large swaths of the internet, causing hundreds of billions of dollars in damage. Similar, less severe incidents have occurred at Anthropic, he said.
The essay lays out a three-step plan.
1. The only step Anthropic is unilaterally committing to now is embedded evaluators: permanent, third-party safety reviewers with employee-level access inside Anthropic’s offices, access to internal risk-assessment tools, and the contractual right to publish findings without editorial control by the company.
2. Mr. Amodei calls for frontier AI companies in democratic nations to agree on common safety thresholds before advancing to higher capability levels.
3. The third step calls for global coordination, including an attempt to bring authoritarian governments into a framework, though Mr. Amodei called that the hardest step and expressed significant skepticism about near-term feasibility.
The “pacing” called for is a deliberate slowdown sufficient to let interpretability science, alignment research, operational security, and testing catch up to model capabilities. He said even an extra year or two at a measured pace could substantially reduce the risk of a catastrophic failure.
The CEOs of OpenAI, SpaceX and Google DeepMind all wrote social media support for the letter, according to The New York Times. Sam Altman, CEO of OpenAI, said the slowdown is a “primary topic of discussions” at the company and he liked the idea of independent evaluators with employee access. He promised to share more soon.
The call comes amid a rapidly accelerating commercial healthcare AI market. Anthropic launched its HIPAA-compliant Claude for Healthcare platform in January and has signed enterprise deals with Banner Health in Phoenix, CommonSpirit Health in Chicago and Stanford Health Care in Palo Alto, Calif., among others. In June, Anthropic launched Claude Science for drug discovery and unveiled its most capable models to date. In May, health system CIOs pushed to be included in testing for Mythos — the company’s most advanced model — after Anthropic delayed its release when internal testing found the model could autonomously detect and exploit cybersecurity vulnerabilities.
