Abstract
Context
This study explores the potential application of artificial intelligence (AI) in facilitating communication in electronic health record (EHR) systems to reduce the burden and risk of clinician burnout. We leveraged previously extracted real EHR patient messages from a study of physician burnout, generated responses using ChatGPT, and then qualitatively compared them to actual physician responses.
Objectives
Assess the potential use of AI in reducing clinician burnout caused by electronic messaging by generating responses to patient messages using ChatGPT. The study also evaluates the AI-generated responses based on their relational connection, informational content, recommendations for next steps, and the extent of editing required before they can be used.
Study Design and Analysis
Qualitative analysis of AI-generated responses to patient messages compared to actual physician responses.
Dataset
EHR messages
Population studied
EHR patient messages
Intervention
Previously extracted real EHR patient messages were used as prompts to generate responses using ChatGPT. Qualitative comparisons were made between the generated responses and actual physician responses for different categories of patient messages, evaluating their relational connection, informational content, follow up recommendations, and the amount of editing needed.
Outcome Measures
Outcome measures include the qualitative assessments of ChatGPT-generated responses to patient messages compared to actual physician responses.
Results
The study found that AI-generated responses lacked relational connection, appearing mechanical and impersonal, while physicians’ responses varied widely, ranging from personal and empathic to instrumental and prescriptive. The informational content of AI-generated responses was also general, compared to physicians’ responses, which were more specific. Additionally, AI-generated responses were on average three times longer than physicians’ responses and required substantial editing. Recommendations from AI were generally generic, while physicians provided tailored recommendations based on the patient’s specific needs.
Conclusions
While some users have started using generative AI language models in healthcare communication, this study demonstrates significant challenges to making them useful to clinicians, and more efforts are needed to harness the potential of AI to support human critical thinking, judgment, and creativity in healthcare.