Researchers from the Icahn School of Medicine at Mount Sinai found that widely used AI chatbots designed for medical support are vulnerable to repeating and elaborating on false health information.
Published Aug. 2 in Communications Medicine, the study tested how large language models respond to fictional patient scenarios containing made-up medical terms. The chatbots often accepted the false information without question, generating detailed responses about nonexistent conditions, symptoms or treatments.
Researchers then introduced a simple safeguard: a one-sentence warning at the start of the prompt reminding the AI that the information provided may be inaccurate. With this added prompt, the rate of errors dropped significantly — suggesting that even minimal guardrails can help reduce misinformation generated by AI systems.
The findings highlight the risks of relying on AI chatbots for medical decision-making and underscore the importance of prompt engineering, system design and human oversight. According to the research team, the results support the need to test AI tools using fabricated inputs before they are deployed in clinical settings.
Researchers plan to apply the method to deidentified patient data and explore additional safety prompts and retrieval tools as AI technology continues to evolve.
