Abstract
Background
Artificial intelligence (AI) is an increasingly popular tool in medical investigations. However, AI’s potential of aiding medical teaching has not been explored. This study aimed to evaluate the effectiveness of AI-tutoring problem-based-learning (PBL) in ophthalmology clerkship and to assess the student evaluations of this module.
Methods
Thirty-eight Grade-two students in ophthalmology clerkship at Sun Yat-Sen University were randomly assigned to two groups. In Group A, students learned congenital cataracts through an AI-tutoring PBL module by exploring and operating an AI diagnosis platform. In Group B, students learned congenital cataracts through traditional lecture given with the same faculty. The improvement in student performance was evaluated by comparing the pre- and post-lecture scores of a specific designed test using paired-T tests. Student evaluations of AI-tutoring PBL were measured by a 17-item questionnaire.
Results
The post-lecture scores were significantly higher than the pre-lecture scores in both groups (Group A: P<0.0001, Group B: P<0.0001). The improvement of group A in the part of sign and diagnosis test (Part I) was more significant than that of group B (P=0.016). However, there was no difference in the improvement in the part of treatment plan test (Part II) between two groups (P=0.556). Overall, all respondents were satisfied and agreed that AI-tutoring PBL was helpful, effective, motive and beneficial to help develop critical and creative thinking.
Conclusions
The application of AI-tutoring PBL into ophthalmology clerkship improved students’ performance and satisfaction. AI-tutoring PBL teaching showed advantage in promoting students’ understanding of signs of diseases. The instructors play an indispensable role in AI-tutoring PBL curriculum.
Keywords: Artificial-intelligence, problem-based learning, ophthalmology clerkship
Introduction
The development of artificial intelligence (AI) provides new tools to solve problems in various medical fields (1). AI algorithms have obvious advantages in identifying and extracting image features (2). Therefore, AI models have been shown to performed equally with or better than human doctors in disease diagnosis based on signs or morphologies, such as skin cancer (3), diabetic retinopathy (4-6), congenital cataract (7), and pathological section identification (8-10). However, AI’s potential of aiding medical teaching has not been explored.
Problem-based learning (PBL) is a learner-centered pedagogy, in which students learn knowledge by solving several open-ended problems found in a trigger material, and take a center stage in case-based, self-directed learning (11,12). PBL enables students to learn more profoundly and improves students’ self-confidence and self-learning ability (13-15).
In recent years, the rapid development of AI algorithm has contributed to the innovation of pedagogy (16). Currently application of AI in teaching is mainly in the form of a robot teacher and/or a robotic examination system, and the AI system replaces teachers’ role in teaching and/or performance-evaluating. Taking the advantages of AI algorithm in image recognition and its widely use in medical field, we proposed an alternative approach of applying AI in medical teaching. We have developed an AI-tutoring PBL module, in which students learn ophthalmic knowledge through solving questions related to the CC-Cruiser, the AI pediatric cataract consultation platform developed by Zhongshan Ophthalmic Center. We validated the effectiveness of the AI-tutoring PBL module by comparing it to the traditional lecture-based teaching in ophthalmology clerkship.
Methods
Participant enrollment and assignment
Thirty-eight Grade-two students participated in ophthalmology clerkship at Sun Yat-Sen University were enrolled in August 2018. The participants were randomly assigned to two groups. In Group A, students learned congenital cataracts through an AI-tutoring problem-based-learning module by exploring and operating an AI diagnosis platform. In Group B, students learned congenital cataracts through traditional lecture given with the same faculty. All the procedures in this study were arranged strictly with the approval of the institutional review board of Zhongshan Ophthalmic Centre of Sun Yat-sen University (IRB-ZOC-SYSU). Written informed consents have been obtained from all students. Flowchart of the study was specific (Figure 1). Both groups had a pre-lecture test, a 45-minute lecture and an after-lecture test. The same examination paper was used for pre- and after-lecture test. The test consisted of two parts. Part I was about the signs and diagnosis of pediatric cataract, and Part II was about the treatment of pediatric cataract. Both groups were taught by an experienced instructor who is a physician in Home of Cataract Children in Zhongshan Ophthalmic Centre.
Figure 1.
Flowchart of the study.
CC-cruiser, the AI consultation platform for pediatric cataract
CC-Cruiser is a pediatric cataract consultation platform developed by our team. This AI platform uses slit lamp photos of pediatric cataracts to identify and classify the location, density and extent of cataract lesions, and provides treatment recommendations for either conservative treatment or surgical treatment. Trained with a large number of labeled ocular images, this AI platform performed as good as ophthalmic experts in diagnosis and evaluation of pediatric cataract (7).
AI-tutoring PBL module
One day before class, students in group A got the introduction materials on CC-cruiser as well as key points for diagnosis and treatment of pediatric cataract. Besides, the tutor provided five questions on diagnosis and treatment of pediatric cataract to students and encouraged them to discuss the questions before class. In the class, 10 minutes were given to students to finish the pre-lecture test. Then the students used 25 minutes to diagnose and evaluate patient with the AI consultation platform, CC-cruiser. Finally, a 10-minute wrapped-up discussion about the diagnosis and treatment of congenital cataract was carried out.
Design of the traditional lecture
One day before class, students in group B got key points for diagnosis and treatment of pediatric cataract. The tutor provided five questions, the same as for Group A, on diagnosis and treatment of pediatric cataract to students and encouraged them to discuss the questions before class. During the class, 10 minutes were given to students to finish the pre-lecture test. Then the tutor gave a 25-minute traditional lecture on congenital cataract. Finally, a 10-minute wrapped-up discussion about the diagnosis and treatment of congenital cataract was carried out.
Student evaluations
Student evaluations of AI-tutoring PBL were measured by a 17-item questionnaire, including 16 one-choice question and one open-ended question (Table 1).
Table 1. Seventeen-item questionnaire.
| No. | Question |
|---|---|
| One-choice questions (A, strongly agree; B, agree; C, disagree; D, strongly disagree) | |
| 1 | AI-tutoring PBL helped me to acquire a higher level of knowledge |
| 2 | AI-tutoring PBL is more effective and motivate compared with traditional didactic lecture |
| 3 | AI-tutoring PBL challenged me to do my best |
| 4 | AI-tutoring PBL promoted the learning of essential concepts or skills |
| 5 | AI-tutoring PBL promoted effective cooperative learning |
| 6 | AI-tutoring PBL promoted increased reading of the textbook by the students |
| 7 | Overall, I am very satisfied with this AI-tutoring PBL |
| 8 | AI-tutoring PBL should be offered more frequently in the curriculum |
| 9 | I will recommend AI-tutoring PBL to other students |
| 10 | This activity was preferable to the traditional lecture format |
| 11 | The AI-tutoring PBL module is well organized |
| 12 | I study with colleagues frequently |
| 13 | The instructor highly facilitated the learning process during the process of AI-tutoring PBL. |
| 14 | The instructor responded well to questions asked by students. |
| 15 | The instructor encouraged and provided opportunities for discussion. |
| 16 | AI-tutoring PBL is beneficial to help develop critical and creative thinking. |
| Open-ended question | |
| 17 | Compared with the traditional teaching method, what do you think are the advantages and disadvantages of AI-tutoring PBL? |
AI, artificial intelligence; PBL, problem-based learning.
The questionnaire in our study were designed according to the previous studies in other subjects of medical education (17). The questionnaire was conducted in Group A after class to collect and assess student satisfaction on AI-tutoring PBL module. Information of the questionnaire have several topics, which included: knowledge acquisition (2 items), motivational dimension (3 items), group cooperation (1 item), creative and critical thinking (1 item), instructor performance (3 items), organization (1 item), overall rating (2 item), recommendations (2 items). Statements were rated on a 4-point scale ranging from “strongly agree” (with the highest score) to “strongly disagree” (with the lowest score).
Data analysis
All data were collected non-anonymously. The effects of the AI-tutoring PBL module on students’ performance were measured by comparing pre- and after-lecture scores using a paired t-test. Independent t-test was used to compare the improvement of student performance between Group A and B. A P value less than 0.050 was considered statistically significant. All analyses were conducted by SPSS software version 22.0 (SPSS Inc., Chicago, IL, USA).
Results
Baseline characteristics
All participants took part in the class and completed examinations as required. There was no difference in baseline characteristics between two groups including gender, pre-lecture scores of Part I (P=0.469), Part II (P=0.097) and total scores (P=0.158) (Table 2).
Table 2. Baseline characteristics of subjects.
| Characteristics | Group A | Group B | P value |
|---|---|---|---|
| Male (n, %) | 7, 36.8% | 7, 36.8% | |
| Pre-lecture scores (mean ± SD) | |||
| Part I: diagnosis | 1.12±1.09 | 1.36±0.94 | 0.469 |
| Part II: treatment | 4.21±1.99 | 3.26±1.37 | 0.097 |
| Total scores | 5.33±1.90 | 4.62±0.98 | 0.158 |
SD, standard deviation.
Improvement of student performance in each group
The post-lecture scores were significantly higher than the pre-lecture scores in both groups (Group A: P<0.001, Group B: P<0.001). The improvement of both groups in the part of sign and diagnosis test (Part I) was significantly. (Group A: P<0.001, Group B: P<0.001). However, the improvement of group A in treatment plan test (Part II) was not obvious (P=0.083) (Table 3).
Table 3. Improvement of student performance in each group.
| Test content | Difference in post- to pre-lecture scores (mean ± SD) | P value |
|---|---|---|
| Group A | ||
| Part I: diagnosis | 7.11±1.94 | 0.000* |
| Part II: treatment | 0.95±2.25 | 0.083 |
| Total scores | 8.06±2.86 | 0.000* |
| Group B | ||
| Part I: diagnosis | 5.39±2.26 | 0.000* |
| Part II: treatment | 1.37±2.11 | 0.011* |
| Total scores | 6.76±3.33 | 0.000* |
*, statistically significant. SD, standard deviation.
Comparison of student performance between Group A and B
The total and the Part I score of group A were higher than those of group B (P=0.034 and 0.033 respectively). Compared with the pre-lecture test, the improvement of group A in Part I was more obvious than that of group B (P=0.016), which may be the main reason why the total post-lecture score of group A was significantly higher than that of group B (Table 4).
Table 4. Comparison of student performance between Group A and B.
| Test content | Group A (mean ± SD) | Group B (mean ± SD) | P value |
|---|---|---|---|
| Post-lecture scores | |||
| Part I: diagnosis | 8.24±1.84 | 6.75±2.26 | 0.033* |
| Part II: treatment | 5.16±2.14 | 4.63±1.89 | 0.427 |
| Total scores | 13.39±2.40 | 11.38±3.17 | 0.034* |
| Increase in post- to pre-lecture scores | |||
| Part I: diagnosis | 7.11±1.94 | 5.39±2.26 | 0.016* |
| Part II: treatment | 0.95±2.25 | 1.37±2.11 | 0.556 |
| Total scores | 8.06±2.86 | 6.76±3.33 | 0.203 |
Student satisfaction
All students in group A responded to the questionnaire. Overall, all respondents were satisfied and agreed that AI-tutoring PBL was helpful, effective, motive and beneficial to help develop critical and creative thinking. Meanwhile, they agreed that the instructor played an important role in AI-tutoring PBL curriculum (Figure 2).
Figure 2.
Students’ responses to Artificial-tutoring problem-based learning courses. Each survey question used a four-point scale.
We also collected students’ answer to the open-ended question: what are the advantages and disadvantages of AI aided PBL teaching methods? Many students confirmed that AI-tutoring PBL benefits them, in terms of preview, communications, collaborations, self-learning ability and critical thinking. However, some students complained on higher requirement of hardware and that they need to spend more time previewing.
Discussion
Clerkship is a transition from general education to learning in a specific field (18). Students in this phase have the advantages of active thinking, strong learning ability, and are eager to accept new concepts and technologies (19,20). Therefore, we combined an AI diagnosis platform with a model of active learning, PBL to set up an AI-tutoring PBL module to improve student performance in ophthalmic clerkship.
We found some interesting results. Group A improved more greatly in the sign and diagnosis test (Part I) compared to Group B, which indicates that the AI model based on image features is better than traditional teaching in deepening medical students’ understanding of signs and morphological features. The AI platform does not directly replace part of the teacher’s work. What CC cruiser presents to students is the disease-evaluating ability that AI has gained through learning from human experts. The AI platform extracts the characteristics of the patient’s ocular image using convolutional neural networks for diagnosis and treatment decision-making (7). Based on the input ocular images and the corresponding evaluation made by the AI platform, the students were encouraged to discuss and collaborate to learn knowledge on diagnosis and treatment of pediatric cataract. The post-lecture scores of Part II were similar in both groups, which demonstrated that the AI module was equally effective to the traditional lecture in conveying knowledge on treatment strategy.
Our anonymous questionnaire gave a subjective assessment of AI-tutoring PBL from students in terms of knowledge acquisition, motivational dimension, cooperation, creative and critical thinking, instructor performance, and preferences. Consistent with the results of the test scores, students agreed that AI-tutoring PBL helps improve student initiative, collaboration, creativity and critical thinking. What is more, the students all agreed that the role of instructor in AI-tutoring PBL is indispensable. The instructor highly facilitated the learning process, also encouraged and provided opportunities for discussion. Consistently, studies showed that the sophisticated role of PBL tutor strikes the right balance between process facilitation and information delivery (21).
The results of this study should be assessed within the context of its limitations. Our data showed that AI-tutoring PBL module could be more effective in conveying knowledge on diseases with specific signs, such as pediatric cataract. However, the effects of this module in teaching other types of diseases require further study. Secondly, the number of participants is relatively small and the results require to be validated with larger sample size.
In summary, application of AI-tutoring PBL into ophthalmology clerkship improved students’ performance and satisfaction. AI-tutoring PBL teaching showed advantage in promoting students' understanding of signs of diseases. The instructors play an indispensable role in AI-tutoring PBL curriculum.
Supplementary
The article’s supplementary files as
Acknowledgments
Funding: This study was funded by the National Key R&D Program of China (2018YFC0116500) and the National Natural Science Foundation of China (81873675).
Ethical Statement: The authors are accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. All the procedures in this study were arranged strictly with the approval of the institutional review board of Zhongshan Ophthalmic Centre of Sun Yat-sen University (IRB-ZOC-SYSU). Written informed consents have been obtained from all students.
Footnotes
Provenance and Peer Review: This article was commissioned by the Guest Editors (Haotian Lin and Limin Yu) for the series “Medical Artificial Intelligent Research” published in Annals of Translational Medicine. The article was sent for external peer review organized by the Guest Editors and the editorial office.
Conflicts of Interest: All authors have completed the ICMJE uniform disclosure form (available at http://dx.doi.org/10.21037/atm.2019.12.15). The series “Medical Artificial Intelligent Research” was commissioned by the editorial office without any funding or sponsorship. HL served as the unpaid Guest Editor of the series. The other authors have no other conflicts of interest to declare.
References
- 1.Jordan MI, Mitchell TM. Machine learning: Trends, perspectives, and prospects. Science 2015;349:255-60. 10.1126/science.aaa8415 [DOI] [PubMed] [Google Scholar]
- 2.Litjens G, Kooi T, Bejnordi BE, et al. A survey on deep learning in medical image analysis. Med Image Anal 2017;42:60-88. 10.1016/j.media.2017.07.005 [DOI] [PubMed] [Google Scholar]
- 3.Esteva A, Kuprel B, Novoa RA, et al. Dermatologist-level classification of skin cancer with deep neural networks. Nature 2017;542:115-8. 10.1038/nature21056 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Takahashi H, Tampo H, Arai Y, et al. Applying artificial intelligence to disease staging: Deep learning for improved staging of diabetic retinopathy. PLoS One 2017;12:e0179790. 10.1371/journal.pone.0179790 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Ting DSW, Cheung CY, Lim G, et al. Development and Validation of a Deep Learning System for Diabetic Retinopathy and Related Eye Diseases Using Retinal Images From Multiethnic Populations With Diabetes. JAMA 2017;318:2211. 10.1001/jama.2017.18152 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Gulshan V, Peng L, Coram M, et al. Development and Validation of a Deep Learning Algorithm for Detection of Diabetic Retinopathy in Retinal Fundus Photographs. JAMA 2016;316:2402. 10.1001/jama.2016.17216 [DOI] [PubMed] [Google Scholar]
- 7.Lin H, Li R, Liu Z, et al. Diagnostic Efficacy and Therapeutic Decision-making Capacity of an Artificial Intelligence Platform for Childhood Cataracts in Eye Clinics: A Multicentre Randomized Controlled Trial. EClinicalMedicine 2019;9:52-9. 10.1016/j.eclinm.2019.03.001 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Wilson ML, Sayed S, Horton S, et al. Artificial intelligence can augment global pathology initiatives – Authors' reply. Lancet 2018;392:2352. 10.1016/S0140-6736(18)32170-6 [DOI] [PubMed] [Google Scholar]
- 9.Ehteshami Bejnordi B, Veta M, Johannes Van Diest P, et al. Diagnostic Assessment of Deep Learning Algorithms for Detection of Lymph Node Metastases in Women With Breast Cancer. JAMA 2017;318:2199. 10.1001/jama.2017.14585 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Golden JA. Deep Learning Algorithms for Detection of Lymph Node Metastases From Breast Cancer. JAMA 2017;318:2184. 10.1001/jama.2017.14580 [DOI] [PubMed] [Google Scholar]
- 11.Hartling L, Spooner C, Tjosvold L, et al. Problem-based learning in pre-clinical medical education: 22 years of outcome research. Med Teach 2010;32:28-35. 10.3109/01421590903200789 [DOI] [PubMed] [Google Scholar]
- 12.Norman GR, Schmidt HG. The psychological basis of problem-based learning: a review of the evidence. Acad Med 1992;67:557-65. 10.1097/00001888-199209000-00002 [DOI] [PubMed] [Google Scholar]
- 13.Vernon DT, Blake RL. Does problem-based learning work? A meta-analysis of evaluative research. Acad Med 1993;68:550-63. 10.1097/00001888-199307000-00015 [DOI] [PubMed] [Google Scholar]
- 14.Faisal R, Bahadur S, Shinwari L. Problem-based learning in comparison with lecture-based learning among medical students. J Pak Med Assoc 2016;66:650-3. [PubMed] [Google Scholar]
- 15.Bate E, Hommes J, Duvivier R, et al. Problem-based learning (PBL): getting the most out of your students - their roles and responsibilities: AMEE Guide No. 84. Med Teach 2014;36:1-12. 10.3109/0142159X.2014.848269 [DOI] [PubMed] [Google Scholar]
- 16.Hassabis D, Kumaran D, Summerfield C, et al. Neuroscience-Inspired Artificial Intelligence. Neuron 2017;95:245-58. 10.1016/j.neuron.2017.06.011 [DOI] [PubMed] [Google Scholar]
- 17.Huang Z, Li M, Zhou Y, et al. Modified Team-Based Learning in an Ophthalmology Clerkship in China. PLoS One 2016;11:e0154250. 10.1371/journal.pone.0154250 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Adelsheimer A, Berman RS, Pachter HL, et al. Surgical clerkship or medical clerkship first: Does it make a difference? Am J Surg 2018;215:304-8. 10.1016/j.amjsurg.2017.08.054 [DOI] [PubMed] [Google Scholar]
- 19.Haak DC, HilleRisLambers J, Pitre E, et al. Increased structure and active learning reduce the achievement gap in introductory biology. Science 2011;332:1213-6. 10.1126/science.1204820 [DOI] [PubMed] [Google Scholar]
- 20.Freeman S, Eddy SL, McDonough M, et al. Active learning increases student performance in science, engineering, and mathematics. Proc Natl Acad Sci U S A 2014;111:8410-5. 10.1073/pnas.1319030111 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Wang Q, Li H, Pang W. From PBL tutoring to PBL coaching in undergraduate medical education: an interpretative phenomenological analysis study. Med Educ Online 2016;21:31973. 10.3402/meo.v21.31973 [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
The article’s supplementary files as