A team of researchers from NYU provided 392 people with 10 patient questions and responses. Half of the responses were generated by healthcare providers, and the others were generated by ChatGPT.
The individuals were tasked with identifying which responses were generated by ChatGPT or human healthcare providers, as well as rate their trust in ChatGPT’s responses using a 5-point scale.
People had trouble distinguishing between ChatGPT’s responses and provider-generated responses. On average, individuals correctly identified chatbot responses 65.5 percent of the time and provider responses 65.1 percent of the time, according to the study.
The study also found that participants mildly trust ChatGPT-generated responses, with individuals rating its responses as 3.4 out of 5 for trustworthiness.
The study, posted in JMIR Medical Education, suggests that chatbots could assist in some patient-provider communication, including administrative tasks and common chronic disease management.
At the Becker's 11th Annual IT + Revenue Cycle Conference: The Future of AI & Digital Health, taking place September 14–17 in Chicago, healthcare executives and digital leaders from across the country will come together to explore how AI, interoperability, cybersecurity, and revenue cycle innovation are transforming care delivery, strengthening financial performance, and driving the next era of digital health. Apply for complimentary registration now.