Implication Statement
Our approach addresses the urgent need for AI experience for the doctors of tomorrow. Through a medical education-focused approach to data labelling, we have fostered medical student competence in medical imaging and AI. We envision our framework being applied at other institutions and academic groups to develop robust labelling programs for research endeavours. Application of our approach to core visual modalities within medicine (e.g. interpretation of ECGs, diagnostic imaging, dermatologic findings) can lead to valuable student experience and competence in domains that feature prominently in clinical practice, while generating much needed data in fields that are ripe for AI integration.
Énoncé des implications de la recherche
Notre approche répond au besoin urgent de familiariser les médecins de demain avec l’IA. Nous avons cherché à développer leurs compétences en imagerie médicale et en IA par une approche à l’étiquetage de données axée sur la formation médicale. D’autres établissements et groupes universitaires souhaitant mettre sur pied des programmes d’étiquetage solides pour leurs projets de recherche pourraient adopter notre modèle. L’application de notre approche aux principales modalités visuelles en médecine (par exemple, l’interprétation des ECG, l’imagerie diagnostique, le diagnostic des lésions dermatologiques) peut permettre aux étudiants d’acquérir une expérience et des compétences précieuses dans des domaines importants de la pratique clinique, tout en procurant des données indispensables dans des secteurs qui sont mûrs pour une intégration de l’IA.
Introduction
Artificial intelligence (AI) is poised to significantly influence medical training. In their recent white paper, the Royal College of Physicians and Surgeons of Canada (RCPSC) has asserted AI as a “new fundamental competency” and stated that introduction into medical school curricula should be prioritized.1 Despite calls for greater digital health literacy among physicians and frameworks for implementation,1,2 little to no curricular attention to AI has been applied within medical schools.3,4 Thus, there is urgency for strategies to engage students and promote understanding of data science and AI.
Medical AI model development relies on large volumes of labelled data. While this work is costly to AI developers, there exists a source of motivated individuals for whom the scholarly collaboration, AI experience, and medical competence derived from data labelling is to be celebrated: medical students. Involving medical students provides access to understanding how medical data are used for AI model development and assures engagement in research efforts with associated faculty mentorship. Here, we describe an example of a rigorous labelling program using lung ultrasound (LUS) images that confers both AI fluency and domain expertise for interested medical students.
Innovation
Central to our LUS labelling project was a core leadership structure consisting of the project’s lead supervisor (physician/ultrasound expert), labelling team leader (medical student), and two physicians. The labelling workforce consisted of six first- and second-year medical students recruited through a social media interest group dedicated to point of care ultrasound. With an extended team, it is essential to have a robust labelling platform to facilitate teaching, scalability, and ultimately, productivity. Thus, we designed a workflow using Labelbox,5 a web-based interface, under an educational license. Using this platform, labellers annotated the findings of short LUS video clips obtained from our institutional database and uploaded to Labelbox after anonymization.
Initial training employed a multimodal strategy including a written ‘Labelling Guide’, videos, practice datasets, iterative feedback, and workshops. The Labelling Guide included a Labelbox introduction, the project’s workflow, descriptions of key LUS findings, the project’s standard operating procedures, and definitions. Screen recordings included LUS findings and a labelling demo. After initial practice conducted as a group, labelling tasks were assigned in 205-clip datasets; once a dataset was complete, a core team member provided individualised feedback and evaluated accuracy (the proportion of clips correctly labelled). This iterative feedback at frequent, fixed intervals was crucial for learning and success.
As the labelling team gained knowledge and experience, in-person workshops were held, consisting of interactive labelling demos, instruction on LUS acquisition at the bedside, and AI in medicine talks. The workshops provided opportunities for student engagement, networking, faculty mentorship, and fostered continued interest in the project. Further, these workshops provided insight into the data life cycle for a medical AI project.
Our project received research ethics board approval from Western University (REB 116838) on January 28th, 2021. Permission from participants was granted for data included in this manuscript.
Outcomes and next steps
We observed considerable improvement in the first independent dataset, with further improvement in subsequent datasets (Figure 1). Of note, the resulting data directly contributed to the creation of a classifier and a publication,6 demonstrating successful incorporation of student-labelled data into AI model development.
Figure 1.
Plot representing the accuracy (proportion of correctly interpreted and labelled clips) of the four medical students that completed initial training. Dataset number ‘0’ indicates the practice dataset completed as a group(a) denotes the point where students were given the Labelling Guide, videos, and practice datasets, (b) the points of iterative feedback, and (c) the first workshop.
Next steps include involving experienced students in the training of new recruits to increase efficiency and participation. Other research programs can start incorporating our program by assigning a labelling team leader and incorporating standardized teaching and feedback sessions into labelling workflows. The framework described here can then be used to create a high output labelling program for other visual fields of medicine, such as ECGs, pathology slides, diagnostic imaging, and dermatologic photos, which are prominent areas for AI and clinical practice.
Acknowledgements
The authors would like to acknowledge Derek Wu and Rushil Chaudhary for their contributions to the training process. We also acknowledge Labelbox, Inc. for providing an educational license for this work.
Funding Statement
Funding: This work received no external or internal funding.
Conflicts of Interest
The authors declare that they have no conflicts of interest with the contents of this article.
Individual Contributions
Conception or design: Jared Tschirhart, Amadene Woolsey, Jamila Skinner, Khadija Ahmed, Courtney Fleming, Justin Kim, Chintan Dave, Robert Arntfield; Drafting and revision: Jared Tschirhart, Amadene Woolsey, Jamila Skinner, Khadija Ahmed, Courtney Fleming, Justin Kim, Robert Arntfield; Final approval: Jared Tschirhart, Amadene Woolsey, Jamila Skinner, Khadija Ahmed, Courtney Fleming, Justin Kim, Chintan Dave, Robert Arntfield
References
- 1.Richard Reznick, Harris K, Horsley T, Sheikh Hassani M. Artificial intelligence (AI) and emerging digital technologies [Internet]. Royal College of Physicians and Surgeons of Canada; 2020. Feb Available from: https://www.royalcollege.ca/rcsite/health-policy/initiatives/ai-task-force-e [Accessed on Sep 9, 2021].
- 2.Law M, Veinot P, Campbell J, Craig M, Mylopoulos M. Computing for Medicine: Can We Prepare Medical Students for the Future? Acad Med J Assoc Am Med Coll. 2019. Mar;94(3):353–7. 10.1097/ACM.0000000000002521 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Rampton V, Mittelman M, Goldhahn J. Implications of artificial intelligence for medical education. Lancet Digit Health. 2020. Mar;2(3):e111–2. 10.1016/S2589-7500(20)30023-6 [DOI] [PubMed] [Google Scholar]
- 4.Paranjape K, Schinkel M, Nannan Panday R, Car J, Nanayakkara P. Introducing artificial intelligence training in medical education. JMIR Med Educ. 2019. Dec 3;5(2):e16048. 10.2196/16048 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Labelbox [Internet]. 2022. Available from: https://labelbox.com [Accessed on Apr 2, 2022].
- 6.Arntfield R, Wu D, Tschirhart J, et al. Automation of lung ultrasound interpretation via deep learning for the classification of normal versus abnormal Lung Parenchyma: a multicenter study. Diagn Basel Switz. 2021. Nov 4;11(11):2049. 10.3390/diagnostics11112049 [DOI] [PMC free article] [PubMed] [Google Scholar]