Philadelphia-based Penn Medicine has launched Observer, a multimodal medical dataset that captures anonymized, real-time interactions between patients and clinicians.
Developed by researchers at the university’s School of Engineering and Applied Science, Observer links video, audio and transcripts to clinical data and EHRs, according to a paper published Oct. 27 in the Journal of the American Medical Informatics Association. The dataset offers a new way to study how communication, technology use and environmental factors influence care delivery.
Leaders said the dataset allows HIPAA-compliant review of real-time visits, creating a foundation for improving clinical practice and building responsible AI tools, according to a Jan. 12 health system news release.
Pilot grants have been awarded to other research teams to begin using Observer, with the long-term goal of expanding it into a national resource.
By tying EHRs to visual and audio data, Penn Medicine researchers aim to study how clinicians use technology during visits, how patients respond to diagnoses and how subtle cues — such as laughter or eye contact — affect outcomes.
At the Becker's 11th Annual IT + Revenue Cycle Conference: The Future of AI & Digital Health, taking place September 14–17 in Chicago, healthcare executives and digital leaders from across the country will come together to explore how AI, interoperability, cybersecurity, and revenue cycle innovation are transforming care delivery, strengthening financial performance, and driving the next era of digital health. Apply for complimentary registration now.