In a study published April 19 in NEJM AI, the researchers gathered more than 27,000 distinct diagnosis and procedure codes from a year of routine care at Mount Sinai Health System. They then utilized descriptions for each code to trigger models from OpenAI, Google and Meta to produce the most precise medical codes.
The codes generated were then compared with the original ones, and any patterns in errors were examined, according to an April 22 news release from Mount Sinai.
All of the large language models analyzed — GPT-4, GPT-3.5, Gemini-pro and Llama-2-70b — demonstrated accuracy below 50% in replicating the original medical codes, the researchers found.
GPT-4 showed the best performance among the models. It achieved the highest exact match rates for ICD-9-CM (45.9%), ICD-10-CM (33.9%) and CPT codes (49.8%).
At the Becker's 11th Annual IT + Revenue Cycle Conference: The Future of AI & Digital Health, taking place September 14–17 in Chicago, healthcare executives and digital leaders from across the country will come together to explore how AI, interoperability, cybersecurity, and revenue cycle innovation are transforming care delivery, strengthening financial performance, and driving the next era of digital health. Apply for complimentary registration now.