Efficacy of ChatGPT in solving attitude, ethics, and communication case scenario used for competency-based medical education in India: A case study

Abstract

BACKGROUND:

Competency-based medical education (CBME) is a method of medical training that focuses on developing learners’ competencies rather than simply assessing their knowledge and skills. Attitude, ethics, and communication (AETCOM) are important components of CBME, and the use of artificial intelligence (AI) tools such as ChatGPT for CBME has not been studied. Hence, we aimed to assess the capability of ChatGPT in solving AETCOM case scenarios used for CBME in India.

MATERIALS AND METHODS:

A total of 11 case scenarios were developed based on the AETCOM competencies. The scenarios were presented to ChatGPT, and the responses generated by ChatGPT were evaluated by three independent experts by awarding score ranging from 0 to 5. The scores were compared with a predefined score of 2.5 (50% accuracy) and 4 (80% accuracy) of a one-sample median test. Scores among the three raters were compared by the Kruskal–Wallis H test. The inter-rater reliability of the evaluations was assessed using the intraclass correlation coefficient (ICC).

RESULTS:

The mean score of solution provided by ChatGPT was 3.88 ± 0.47 (out of 5), indicating an accuracy of approximately 78%. The responses evaluated by three raters were similar (Kruskal–Wallis H P value 0.51), and the ICC value was 0.796, which indicates a relatively high level of agreement among the raters.

CONCLUSION:

ChatGPT shows moderate capability in solving AETCOM case scenarios used for CBME in India. The inter-rater reliability of the evaluations suggests that ChatGPT’s responses were consistent and reliable. Further studies are needed to explore the potential of ChatGPT and other AI tools in CBME and to determine the optimal use of these tools in medical education.

Introduction

Competency-based medical education (CBME) is an emerging approach in medical training that emphasizes the development of specific competencies in learners rather than just the accumulation of knowledge and skills. Indian medical institutions under the National Medical Commission (NMC) follow the CBME curriculum since the academic year 2019.[1] One of the important components of CBME is the training on attitude, ethics, and communication (AETCOM). Indian medical colleges have implemented various strategies to incorporate AETCOM into their curricula. One common approach is to use case scenarios and role-playing exercises to help students develop their ethical reasoning and communication skills as per the module put forward by NMC on AETCOM. In addition, many medical colleges have established ethics committees to address ethical issues that arise in clinical practice and to provide guidance to medical students and practitioners.[2]

By focusing on the AETCOM skills of medical students, institutions are trying to prepare their students capable of providing compassionate, patient-centered care that is respectful of cultural and social differences.[3] In addition, AETCOM helps to promote professionalism and ethical behavior among medical practitioners, which are essential for maintaining public trust in the healthcare system. However, as this is relatively a newly introduced component in the curriculum, both teachers are students are facing challenges in teaching–learning and assessment process.[4]

Artificial intelligence (AI) is one such tool that has the potential to revolutionize medical education. The use of AI in CBME has the potential to provide learners with personalized feedback and support and to enhance the efficiency and accuracy of assessments. ChatGPT is an AI tool that has been trained to generate text responses to a wide range of prompts, including those related to medical education.[5,6] However, the capability of ChatGPT in solving AETCOM case scenarios used for CBME in India has not been studied.

Hence, this study aimed to fill this gap in the literature by evaluating the capability of ChatGPT in solving AETCOM case scenarios used for CBME in India. This is the first study to explore the model’s performance across multiple domains, clinical reasoning, and decision-making, while considering the ethical implications of AI in medical education. The study’s uniqueness lies in its integration of various dimensions, adaptation to the Indian context, and potential application in medical education. The findings of this study have important implications for medical educators and researchers, as they provide insights into the potential of AI tools for CBME. In addition, the study contributes to the development of best practices for using AI in medical education and assessment.

Materials and Methods

Study design and setting

This was a cross-sectional study to assess the efficacy of ChatGPT in solving AETCOM case scenarios for CBME in India. The study involved an audit of data generated from a public domain website (https://chat.openai.com, May 24 version). We used a personal computer (ASUS VivoBook Max X541N) and a personal broadband connection for accessing the Internet. This study was conducted in May–June 2023.

Data collection tool and technique

The study involved 11 case scenarios from various modules of AETCOM taught under CBME in medical colleges in India. Two such case scenarios are shown in Figures 1 and 2. These cases were used to generate responses from ChatGPT, and the responses were collected for further analysis.

Figure 1.

Example of a case and part of response by ChatGPT

Figure 2.

Example of a case and part of response by ChatGPT

The responses generated by ChatGPT were evaluated by three independent experts using a scoring system that ranged from 0 to 5. The experts were blinded to the identity of the responses generated by ChatGPT. The scores from individual evaluators were collected without identification and used for analysis.

Ethical considerations

Data audit from public domain sources or analyzing text generated by AI (not sharing the text) does not require ethical clearance. We followed ethical standards for handling and storage of the data. The study did not involve human participants or animals; hence, ethical clearance was not required according to “National Ethical Guidelines for Biomedical and Health Research Involving Human Participants” published by the Indian Council of Medical Research (ICMR) 2017.

Statistical analysis

The scores awarded by the experts were first used to calculate the average score and compared to predefined scores of 2.5 (50% accuracy) and 4 (80% accuracy) using a one-sample median test to determine whether ChatGPT’s responses were accurate. Additionally, the scores awarded by the three raters were compared using the Kruskal–Wallis H test to determine whether there were any significant differences in their evaluations. To assess the inter-rater reliability of the evaluations, the intraclass correlation coefficient (ICC) was calculated. The ICC measures the degree of agreement among raters and is commonly used in studies that involve multiple raters. We used GraphPad Prism 7.0 (GraphPad Software Inc., USA) for statistical analysis, and a P value below 0.05 was considered statistical significance.

Result

A total of 11 cases were analyzed in this study. The average score was 3.88 ± 0.47. The scores of three raters are shown in Table 1.

Table 1.

Score of the response to the cases

Module	Cases	Rater 1	Rater 2	Rater 3	Average
Case studies on patient autonomy and decision-making	Case 1	4	4.5	4.5	4.33
	Case 2	4	4.5	4.5	4.33
	Case 3	4	4	4.5	4.17
Disclosure of medical errors	Case 1	3	3	3.5	3.17
Confidentiality	Case 1	4.5	4.5	4.5	4.5
Fiduciary duty	Case 1	4	4	4	4
Medicolegal and ethics	Case 1	4	4	4	4
Case studies in ethics empathy and the doctor–patient relationship	Case 1	3.5	3.5	3.5	3.5
Case studies in ethics: the doctor industry Relationship	Case 1	4	4	4	4
	Case 2	3	3.5	3.5	3.33
Case studies in ethics and patient autonomy	Case 1	3	3.5	3.5	3.33
Average		3.73	3.91	4	3.88
Standard deviation		0.52	0.49	0.45	0.47

The scores among the three raters were similar and did not show any significant difference as tested by the Kruskal–Wallis H test as shown in Figure 3.

Figure 3.

Scores of the responses by three raters

A one-sample median test with a hypothetical score of 2.5 showed that the score was significantly higher than that of 50% accuracy (P value = 0.001) and it is similar to 80% accuracy (P value of one-sample median test P value = 0.33).

The ICC value was 0.796, which indicates a relatively high level of agreement among the raters.

Discussion

The major finding of this study is that ChatGPT, an AI large language model, has the capability to generate responses to case scenarios based on AETCOM competencies with an acceptable degree of accuracy. One possible reason for this finding is that the language model was trained on a large corpus of medical text data, which may have enabled it to generate responses that are consistent with medical ethics and communication principles.

This study may be particularly helpful for Indian medical colleges that are facing a shortage of teachers. Using AI language models like ChatGPT to generate responses to case scenarios based on AETCOM competencies, medical colleges may be able to provide students with personalized learning experiences and reduce the workload of teachers. This may also help the students in self-directed learning under the appropriate guidance and supervision of teachers. A study by Sahadevan et al. highlights the potential advantages of online platforms in assisting the implementation of CBME, particularly in times of remote learning. Such platforms can offer flexibility to learners and allow them to learn at their own pace.[7]

The findings of this study may be used to inform the development of AI-based educational tools that can be used in medical colleges. Many of the components of medical education need case-based learning like AETCOM.[8] Previous studies have proven its capacity to solve cases and complex questions in various domains.[9,10] Chatbots can help in both generating the cases and also provide solution to the cases. For example, AI-based chatbots could be developed that can provide students with immediate feedback on their responses to case scenarios based on AETCOM competencies.

The use of AI in medical education has both advantages and disadvantages.[11] One advantage is that it can provide personalized learning experiences for students and help them enhance their assisted problem-solving skills. Another advantage is that it can reduce the workload of teachers and enable them to focus on more complex aspects of teaching.[12] However, there are also potential disadvantages, such as the risk of overreliance on technology and the potential for bias or errors in the algorithms used by AI models. High dependency on AI may reduce the comprehensive capability of humans in the overall learning process.

The main limitation of this study is the small sample size of case scenarios and raters. The study only evaluated the capability of ChatGPT to respond to 11 case scenarios and was assessed by three independent raters. Therefore, the generalizability of the findings may be limited, and further research with larger sample sizes and more diverse case scenarios and raters is needed to confirm the findings. In addition, ChatGPT itself has its limitations. While ChatGPT and other AI tools have vast potential in the medical field, their utilization also raises significant ethical and legal issues. These concerns encompass potential copyright infringements, medicolegal complications, and the necessity of ensuring transparency in AI-generated content.[13] The existing medical system operates on the premise of certified professionals who deliver reliable services to patients. However, AI-based chatbots like ChatGPT lack a comparable verification process, giving rise to ethical concerns.[14] We need to take advantage of such large language models without compromising the quality. In the field of medicine, AI has the potential to enhance efficiency by streamlining paperwork and optimizing chatbots for medical writing. However, we must exercise caution and not be solely captivated by the immense potential of AI. To fully harness its benefits in medicine and science, a thoughtful and deliberate approach should be adopted, involving open discussions about the associated risks and advantages before its implementation.[15]

Conclusion

The ChatGPT has the potential to generate responses to case scenarios based on AETCOM competencies with an acceptable degree of accuracy according to the current health policy of the country. The findings of this study indicate that the current version of ChatGPT can be used for teaching–learning AETCOM modules by medical teachers and students. In India and other countries, facing teacher shortage may think of reaping the benefits of AI in medical education.

Financial support and sponsorship

Nil.

Conflicts of interest

There are no conflicts of interest.