As AI takes on more clinical tasks, health systems are introducing safeguards to preserve physicians’ independent judgment and prevent clinical skills from deteriorating, even as leaders acknowledge they have yet to determine how to measure AI’s long-term effects on competency.
At Children’s Hospital of Philadelphia, trainees must demonstrate proficiency in certain clinical tasks before gaining access to AI tools that can perform them, part of an effort to prevent overreliance on the technology.
“To defend against overreliance, we worked with our GME [graduate medical education] leaders to introduce ‘graduated privileges,’ the idea that trainees only gain access to certain AI tools (like AI scribes for note writing) after first demonstrating proficiency in that clinical task,” Bimal Desai, MD, vice president and chief health informatics officer at CHOP, told Becker’s.
The approach is one example of how health systems are confronting a growing question: As AI becomes capable of performing work clinicians have traditionally been trained to master, which skills must they continue to demonstrate independently?
The issue is also drawing national attention. On Oct. 27, the Association of American Medical Colleges published AI competencies for physicians across different stages of education and practice, including expectations for preserving independent clinical reasoning, recognizing AI errors and avoiding de-skilling.
The framework, intended to guide education and assessment rather than replace existing competency standards, identifies the potential for AI to weaken critical thinking and foundational skills. It also calls for physicians to recognize the risk of never developing essential clinical skills because of reliance on the technology.
Protecting skills before introducing AI
At CHOP, leaders believe clinicians must develop foundational abilities before using AI to support their work.
“We believe real clinical learning is the result of productive cognitive struggle with authentic problems,” Dr. Desai said.
The organization recommends “AI Free Zones” for learners and strategies that position AI as a supplement rather than a replacement for generating differential diagnoses, synthesizing clinical information and interpreting tests.
Leah Carr, MD, attending neonatologist and clinical informaticist at CHOP, said the organization is also developing recommendations to help graduate medical education leaders and faculty navigate AI use among trainees.
“GenAI is here, in the practice environment, and we need to ensure our clinicians understand the potential impacts of overreliance such as never-skilling, de-skilling, mis-skilling,” Dr. Carr told Becker’s.
Those concerns encompass clinicians failing to develop necessary skills, losing existing abilities or developing inappropriate practices through AI use.
CHOP is expanding its School of AI educational materials with content tailored to clinicians at different training levels, including a new module focused on critically evaluating AI-generated information and identifying inaccuracies.
The organization has also developed local guiding principles for AI use while working to determine which clinical skills should be preserved without technological assistance.
Baltimore-based Johns Hopkins Medicine is taking a similar approach to preserving foundational clinical reasoning.
Khyzer Aziz, MD, chief medical information officer at Johns Hopkins Hospital, said the organization prioritizes establishing clinical competence before teaching clinicians to critically evaluate AI output and introducing AI under supervision.
“Taking a history, examining the patient, reasoning from physiology to a differential diagnosis, communicating with patients and families, and recognizing a patient who is deteriorating all remain core skills that clinicians demonstrate independently,” Dr. Aziz told Becker’s.
Under its AI governance framework, some Johns Hopkins pilots are structured to require clinicians to make their own assessments before receiving AI-generated input.
The health system is also teaching clinicians when to trust, verify or override AI recommendations. Through an educational boot camp, clinicians work with a deidentified dataset covering approximately 8 million patients to examine data quality, bias, missing information and how language models interpret clinical notes.
Johns Hopkins’ collaboration with OpenEvidence, announced Oct. 13, similarly emphasizes preserving first-principles reasoning, evidence-based medicine and clinical judgment as AI becomes more integrated into medical education and patient care.
Clinical expectations remain unchanged at Iowa
Despite those changes in education and technology access, AI has not necessarily altered how health systems formally assess clinicians’ competency.
At Iowa City-based University of Iowa Health Care, credentialing assessments have not changed as AI use expands.
James Blum, MD, chief health information officer at the health system, told Becker’s that physicians and nurses must remain capable of practicing independently of AI.
The health system provides education and certain tool-specific competency assessments as it introduces new technologies, including training on potential risks such as generative AI hallucinations. Those requirements are separate from formal credentialing.
“Even something as rudimentary as note writing, people still need to be able to write a note in the event the system is down,” Dr. Blum said.
The same expectation applies to more complex tasks. Radiologists must be able to interpret images without AI assistance, and physicians must retain the ability to diagnose patients using resources that are not AI-enabled, he said.
While Dr. Blum expects AI to eventually change clinical practice expectations, he does not anticipate fundamental skills becoming unnecessary within the next five years.
The distinction underscores a challenge facing hospitals: Teaching clinicians how to use AI safely is not the same as changing how their underlying clinical abilities are evaluated.
Can AI make clinicians worse at their jobs?
Research into AI-related clinical de-skilling is beginning to examine that possibility, although findings are not uniform.
An observational study published in The Lancet Gastroenterology & Hepatology in 2025 found that experienced endoscopists’ ability to detect precancerous growths during colonoscopies performed without AI declined after exposure to AI-assisted detection tools.
Researchers analyzed 1,443 colonoscopies performed without AI at four centers in Poland. The adenoma detection rate decreased from 28.4% before AI implementation to 22.4% afterward, a 6-percentage-point decline, or roughly 21%.
The findings suggested that repeated exposure to AI could affect clinicians’ performance when the technology is unavailable, although the observational study did not establish that AI caused the decline.
A separate prospective study published in Endoscopy in 2026 produced different findings. Researchers examined 5,013 colonoscopies performed by 13 endoscopists before, during and after exposure to AI-assisted polyp detection.
AI improved detection of polyps measuring at least 5 millimeters among less-experienced endoscopists while it was in use. However, researchers did not find a statistically significant change in their unassisted performance after AI exposure, suggesting no clear evidence of either skill improvement or deterioration.
Because the Lancet study focused on experienced endoscopists while the Endoscopy study compared endoscopists with different levels of experience, the two results are difficult to compare directly. Together, they illustrate why the long-term implications of AI reliance remain unsettled.
For health systems, the challenge extends beyond whether an AI tool performs accurately. Leaders must also consider whether clinicians can maintain their abilities when that assistance is removed.
Measuring what AI changes
Dr. Blum said evaluating whether AI strengthens clinicians’ abilities or creates overreliance remains difficult.
At Iowa, clinicians can rate the quality of documentation generated by ambient AI tools, while the health system periodically solicits feedback on how its technologies are performing.
Predictive AI models can be evaluated through performance metrics, including monitoring for model drift. Generative AI presents a more complicated challenge because assessments often depend on clinicians’ perceptions of a tool’s usefulness, he said.
Those measures can provide insight into technology performance but do not necessarily establish whether AI is preserving clinical competency or improving patient outcomes.
Johns Hopkins is also evaluating AI tools before allowing them to influence clinical care. Dr. Aziz described a retrospective pilot that compared a generative AI tool’s performance with physician-adjudicated chart reviews at two hospitals without affecting patient care.
The health system’s collaboration with OpenEvidence also includes plans to evaluate AI’s effects on clinical decisions, care quality and patient outcomes.
Meanwhile, the new AAMC framework encourages physicians to use AI in ways that preserve independent reasoning, recognize errors and support continued professional development. It also emphasizes monitoring for skill deterioration as AI becomes more deeply integrated into care.
Together, the approaches at CHOP, Johns Hopkins and Iowa show health systems changing how clinicians learn about and gain access to AI. Iowa’s experience suggests formal competency assessments have not necessarily followed.
Hospitals are developing ways to ensure clinicians can use AI appropriately while retaining essential skills. Whether those safeguards are enough, and how organizations should measure their effectiveness, remains unresolved.
“I think that that’s a very difficult question that we, as a discipline, particularly in medicine, have not figured out yet,” Dr. Blum said.
Becker’s 3rd Annual Fall Chief Pharmacy Officer Summit is the top tier event that the people leading the future of pharmacy convene. This 3-day event is built specifically for Chief Pharmacy Officers and senior pharmacy leaders. The sessions examine the workforce, financial and clinical priorities shaping pharmacy today. Becker’s is where candid discussions flourish and new ideas thrive. Becker's 3rd Annual Fall Chief Pharmacy Officer Summit
