Pennsylvania has sued Character.AI, alleging the company’s AI chatbot unlawfully posed as a licensed medical professional and provided medical advice.
The lawsuit, filed by the Pennsylvania Department of State, seeks a preliminary injunction to stop Character.AI from “misrepresenting its AI companion bots as licensed medical professionals who can provide medical advice,” according to a May 5 news release from Pennsylvania Gov. Josh Shapiro’s administration.
State officials said the action followed an investigation by the Department of State’s AI Task Force into whether AI systems may be engaging in the unlicensed practice of medicine. The administration described the case as the first enforcement action of its kind announced by a U.S. governor.
According to the news release, investigators found chatbot characters on Character.AI claiming to be licensed medical professionals, including psychiatrists, and engaging users in conversations about mental health symptoms. In one instance, a chatbot allegedly claimed to be licensed in Pennsylvania and provided an invalid license number.
The lawsuit alleges Character Technologies, the company behind Character.AI, violated Pennsylvania’s Medical Practice Act, which prohibits individuals or entities from presenting themselves as licensed medical professionals without proper credentials.
Character.AI has more than 20 million monthly active users, according to the news release.
“We do not comment on pending litigation,” a spokesperson for Character.AI told Becker’s.
The company added that its highest priority is the safety and well-being of its users and that the user-created characters on its site are fictional and intended for entertainment and roleplaying.
“We have taken robust steps to make that clear, including prominent disclaimers in every chat to remind users that a character is not a real person and that everything a character says should be treated as fiction,” the spokesperson said. “We also add robust disclaimers making it clear that users should not rely on characters for any type of professional advice. Character.AI prioritizes responsible product development and has robust internal reviews and red-teaming processes in place to assess relevant features.”
At the Becker's 11th Annual IT + Revenue Cycle Conference: The Future of AI & Digital Health, taking place September 14–17 in Chicago, healthcare executives and digital leaders from across the country will come together to explore how AI, interoperability, cybersecurity, and revenue cycle innovation are transforming care delivery, strengthening financial performance, and driving the next era of digital health. Apply for complimentary registration now.