Researchers fed New England Journal of Medicine pediatric case challenges to ChatGPT version 4 — the newest model from OpenAI, which costs $20 a month. The chatbot misdiagnosed 72 of 100 cases and wrote a too-broad diagnosis for another 11.
In the study, ChatGPT also couldn’t identify relationships such as the one between autism and vitamin deficiencies.
The researchers said this is the first known accuracy test for learning language model chatbots and pediatric medical scenarios, “which require the consideration of the patient’s age alongside symptoms.” Clinical AI uses also require up-to-date information, which can be hard to come by for tech companies.
At the Becker's 11th Annual IT + Revenue Cycle Conference: The Future of AI & Digital Health, taking place September 14–17 in Chicago, healthcare executives and digital leaders from across the country will come together to explore how AI, interoperability, cybersecurity, and revenue cycle innovation are transforming care delivery, strengthening financial performance, and driving the next era of digital health. Apply for complimentary registration now.
