UC San Diego Health is working with Harvard Medical School and Anthropic to develop an open-source oversight system designed to check the safety of medical recommendations made by patient-facing AI.
The collaboration is part of the Advanced Research Projects Agency for Health’s ADVOCATE program, which focuses on AI-supported care for patients with heart failure, Karandeep Singh, MD, UC San Diego Health’s chief health AI officer, told Becker’s.
The team’s approach centers on a “clinical constitution,” a set of rules it plans to develop with communities and stakeholders to guide the technology’s safety assessments. Those assessments would also draw on medical evidence and clinical guidelines, Dr. Singh said.
The work addresses a challenge as AI becomes more directly involved in patient care: how to evaluate its recommendations and maintain oversight after deployment.
“In the long run, simply evaluating an AI tool before it goes live is not going to be enough,” Dr. Singh said.
Oakland, Calif.-based Kaiser Permanente and Duke University were selected to lead real-world implementation under ADVOCATE. The program aims to develop an FDA-authorized agentic AI system for cardiovascular care.
Dr. Singh described three groups participating in the program: teams developing patient-facing AI, teams building technology to check those tools’ recommendations, and health system partners where studies will evaluate safety and effectiveness.
UC San Diego Health, Harvard and Anthropic are participating in the second group. Their team is separate from the Stanford-led collaboration with OpenAI, the University of Washington in Seattle and Tuple Health, which is also developing supervisory technology under ADVOCATE.
The UC San Diego Health team’s technology is intended to check recommendations such as increasing a medication dose, Dr. Singh said. Physicians would use its interface to review both the recommendations and the oversight system’s assessments.
The system’s intended role is to identify safety risks while recognizing that some decisions require consideration of patient preferences.
“We’re going to only comment on those things that are true safety risks to patients,” Dr. Singh said.
The team calls its approach constitutional oversight. The clinical constitution would have governance attached to it to make transparent how the tool is supposed to behave in different scenarios, he said.
The partners intend to make the oversight technology open source so others can implement it. Dr. Singh said the component would not be exclusive to Anthropic.
“Our component is actually intended to be open source,” he said.
The program has formally kicked off, contracts are signed and participating teams must meet milestones every three months to remain in the program.
Dr. Singh said the partners’ experience developing and adopting open standards informs their goal of making the oversight technology available for others to implement.
“I think we embrace open standards and we hope to kind of collectively benefit the community,” he said.
