ChatGPT's potential role in non-English-speaking outpatient clinic settings

Abstract

Researchers recently utilized ChatGPT as a tool for composing clinic letters, highlighting its ability to generate accurate and empathetic communications. Here we demonstrated the potential application of ChatGPT as a medical assistant in Mandarin Chinese-speaking outpatient clinics, aiming to improve patient satisfaction in high-patient volume settings. ChatGPT achieved an average score of 72.4% in the Chinese Medical Licensing Examination's Clinical Knowledge section, ranking within the top 20th percentile. It also demonstrated its potential for clinical communication in non-English speaking environments. Our study suggests that ChatGPT could serve as an interface between physicians and patients in Chinese-speaking outpatient settings, possibly extending to other languages. However, further optimization is required, including training on medical-specific datasets, rigorous testing, privacy compliance, integration with existing systems, user-friendly interfaces, and the development of guidelines for medical professionals. Controlled clinical trials and regulatory approval are necessary before widespread implementation. As chatbots’ integration into medical practice becomes more feasible, rigorous early investigations and pilot studies can help mitigate potential risks.

Ali et al. recently reported on the use of ChatGPT as a tool for composing clinic letters, a time-consuming process crucial for facilitating communication between physicians and patients. ¹ The authors concluded that ChatGPT could generate accurate and empathetic letters, closely resembling those written by humans. As the technology behind ChatGPT evolves rapidly, this study offers a timely and highly relevant insight.

In China, many physicians face the challenge of managing high patient volumes in outpatient clinic sessions. Physicians may see over 50, and sometimes even over a hundred, patients in a single session, compelling them to limit the time spent with each patient to just a few minutes. This constraint contributes to patient dissatisfaction and is a common source of complaints. ² In this study, we aim to explore the potential of implementing ChatGPT as a medical professional to interact with patients, collect patient history, and improve patient satisfaction in non-English-speaking (mandarin Chinese) outpatient clinic settings. Informed consent is waived since there is no test subject involved in this study.

To begin, we evaluate ChatGPT's medical knowledge using scores from the Chinese Medical Licensing Examination's Clinical Knowledge section, comprised of 600 multiple-choice questions, for the past 3 years (2021, 2020, and 2019). Subsequently, we assess ChatGPT's ability to gather relevant clinical information from patients. We utilize 10 patient-history-inquiry questions from various disciplines, including gastroenterology, hepatology, nephrology, endocrinology, cardiology, rheumatology, traumatology, and neurology, in the Chinese Medical Licensing Examination's Clinical Skills section and score them based on the successful retrieval of key information, communication skills, and humanness demonstrated.

ChatGPT (GPT-4) achieved an average score of 72.4% in the Chinese Medical Licensing Examination's Clinical Knowledge section, ranking within the top 20th percentile based on estimation. In the Chinese Medical Licensing Examination's Clinical Skills section, ChatGPT is capable of passing Clinical Skills examinations, demonstrating its potential for clinical communication in non-English speaking environments (Figure 1). We conclude that ChatGPT is able to pass the Chinese Medical Licensing Examination's Clinical Knowledge Section with a higher-than-average score, and it has the potential to engage in conversations to extract pertinent information for diagnostic use in Mandarin Chinese.

Figure 1.

Performance of ChatGPT (GPT-4.0) on the Chinese Medical Licensing Examination Clinical Knowledge (left) and Clinical Skills (right).

ChatGPT is multilingual by training. Our study shows that ChatGPT has the potential to serve as an interface between physicians and patients, enhancing patient experiences in Chinese-speaking, outpatient settings with staff shortages. It is probable that ChatGPT will achieve on-par performance in other languages. Other applications, including strengthening doctor–patient communication and more technical involvement in different subspecialties, are also currently under investigation.

However, further optimization is required before implementing ChatGPT in real-world scenarios to ensure safety, efficacy, and compliance with regulations. The AI model must be trained on a comprehensive dataset specifically designed for medical use to ensure it has the appropriate domain knowledge. Rigorous testing and validation should be conducted to ensure the AI model's accuracy and reliability in interpreting and extracting relevant clinical information. The AI system should adhere to strict privacy regulations, such as HIPAA, and incorporate safeguards to protect sensitive patient information. The AI model also needs to be integrated with existing Electronic Medical Records (EMR) systems, patient management platforms, and other clinical tools to ensure seamless data exchange and to maximize efficiency. A user-friendly interface for both medical professionals and patients, especially patients with less experience with electronic devices, is needed to easily interact with the AI model. Guidelines should be made for medical professionals on how to use the AI model, including its limitations, potential risks, and when to rely on human expertise. Eventually, controlled clinical trials demonstrating ChatGPT's safety and efficacy are necessary before regulatory approval.

The integration of chatbots into medical practice appears more feasible and imminent than ever before.^3,4 While this development presents challenges for the medical community, rigorous early investigations and pilot studies can help mitigate potential risks.