Preparing Radiologists to Lead in the Era of Artificial Intelligence: Designing and Implementing a Focused Data Science Pathway for Senior Radiology Residents

Abstract

Artificial intelligence and machine learning (AI-ML) have taken center stage in medical imaging. To develop as leaders in AI-ML, radiology residents may seek a formative data science experience. The authors piloted an elective Data Science Pathway (DSP) for 4th-year residents at the authors’ institution in collaboration with the MGH & BWH Center for Clinical Data Science (CCDS). The goal of the DSP was to provide an introduction to AI-ML through a flexible schedule of educational, experiential, and research activities. The study describes the initial experience with the DSP tailored to the AI-ML interests of three senior radiology residents. The authors also discuss logistics and curricular design with common core elements and shared mentorship. Residents were provided dedicated, full-time immersion into the CCDS work environment. In the initial DSP pilot, residents were successfully integrated into AI-ML projects at CCDS. Residents were exposed to all aspects of AI-ML application development, including data curation, model design, quality control, and clinical testing. Core concepts in AI-ML were taught through didactic sessions and daily collaboration with data scientists and other staff. Work during the pilot period led to 12 accepted abstracts for presentation at national meetings. The DSP is a feasible, well-rounded introductory experience in AI-ML for senior radiology residents. Residents contributed to model and tool development at multiple stages and were academically productive. Feedback from the pilot resulted in establishment of a formal AI-ML curriculum for future residents. The described logistical, planning, and curricular considerations provide a framework for DSP implementation at other institutions.

Supplemental material is available for this article.

© RSNA, 2020

Summary

Authors described the initial experience with a data science curriculum for senior radiology residents who wished to develop expertise in applied clinical artificial intelligence and machine learning; this flexible, integrated curriculum may serve as a model for other institutions.

Key Points

• ■ The next generation of radiologists needs clinical leaders in artificial intelligence and machine learning to help drive the field toward clinical utility and to improve patient outcomes.

• ■ The senior year of a radiology residency is an ideal time for immersive involvement in a curriculum covering essential skills in data science, artificial intelligence, and machine learning.

Introduction

Artificial intelligence and machine learning (AI-ML) applications in radiology are emerging at a staggering rate, although full clinical use has yet to be realized (1–3). The role of radiologists as clinical data scientists is decades old; from the pre-PACS era through modern applications for results notification, clinical decision support, and precision imaging, radiologists have served as stewards of medical imaging data for our referring colleagues. Radiologists are essentially information and data specialists, working at the intersection of patient care and informatics (4). While many have expressed anxiety about AI-ML tools as a threat to job security, such fears are largely overstated (5). Optimists in turn share the belief that radiologists are well-positioned, arguably more than any other medical specialty, to harness the power of data toward a goal of delivering better and more efficient health care.

Motivation for a Data Science Pathway for Radiology Residents

Data science has the potential to reinvigorate our specialty, but radiologists must be ready and able to adapt to changes in practice in the coming years. Radiologists-in-training should embrace data science in preparation for guiding machine learning model development and application translation into the clinical arena. Recent landmark studies applying AI-ML techniques in other specialties including ophthalmology (6) and dermatology (7), as well as applications in chest radiography (8) and neuroimaging (9), have sparked interest in AI-ML training opportunities among radiology residents and fellows (10). Nonetheless, organized AI-ML curricula are limited to a few institutions, and formal training opportunities are lacking (11,12). While a formal training pathway is neither necessary nor sufficient to become a leader in radiology data science—as many existing leaders in the field have not received formal data science or AI-ML training—such formal training may serve as a cornerstone for early career radiologists.

Context of the Data Science Pathway

The final year of radiology residency is an opportunity to tailor education and training to the interests of individual residents. The recent change in timing of the American Board of Radiology Core Exam has stimulated development of both clinical and nonclinical elective “pathways” for the 4th year. In addition to more traditional pathways for research and dual-board certification in diagnostic radiology and nuclear medicine, the establishment of formal pathways for other interests including clinician-educator (13), global health, quality improvement (14), leadership (15), and focused clinician-scientist research pathways (16) have been described. Our residency program has established 4th-year elective pathways in global health, informatics, leadership, and medical education and thus has a preexisting infrastructure for elective learning in the 4th year.

Imaging informatics training programs and fellowships have been established at institutions across the country with a mission to train new radiologist leaders in imaging informatics. Given the prominent role of imaging informatics and informaticists in access to data and analytical tools for AI-ML, there have been calls in the radiology and imaging informatics literature to integrate AI-ML training into existing informatics training programs (17,18). The Informatics Pathway and Fellowship at Brigham and Women’s Hospital Department of Radiology is administered through the Brigham and Women’s Hospital Center for Evidence-Based Imaging and is designed to give broad exposure to imaging informatics with a focus on clinical effectiveness research. While this experience is well-suited to many residents and fellows interested in receiving further training in informatics, the program was not sufficiently integrated with AI-ML research efforts across the institution to provide the immersive AI-ML research experience the residents involved in creating the Data Science Pathway (DSP) were seeking. However, it is reasonable to assume that informatics programs at other institutions might be well suited to integrate the concepts discussed herein into their curricula, as suggested in the articles previously referenced.

The MGH & BWH Center for Clinical Data Science (CCDS) is a collaborative effort by the flagship academic medical centers of the Mass General Brigham Healthcare System and Harvard Medical School. The CCDS employs a multidisciplinary team of data scientists, software engineers, and product specialists who partner with clinicians from the affiliated hospitals in the development of clinically relevant AI-ML tools. With industrial partnerships including Nvidia, General Electric, Nuance, and Fuji, the CCDS has built a large research and development data warehouse with extensive computer infrastructure needed to engage in large-scale data science efforts. Clinicians, including radiologists, are heavily involved in project design, strategy, and development. Trainee involvement in collaborative research and tool development is highly valued.

To address the growing interest among residents and the need for radiology-trained leaders in data science, we developed a DSP for senior radiology residents. The goal of the DSP was to provide an introduction to AI-ML through a flexible schedule of educational, experiential, and research activities. Here, we describe our experience with the first trainee cohort and the establishment of a formal AI-ML curriculum for future residents based on pilot feedback. A desire for flexibility in the DSP was expressed among the first cohort, given varying preferences for the relative time spent on clinical versus research activities during the 4th year. Within this limitation, our goal was to provide a broader framework that may be generalized to other institutions.

Intervention

Trainee Characteristics

Three senior residents with varying technical backgrounds independently proposed the DSP under the guidance of a common mentor. The trainees shared a common goal of using the 4th year to build relevant skills for a career in AI-ML applications in radiology. Two of three trainees had graduate degrees (PhD in computational neuroscience and PhD in cell biology). One of three trainees had extensive experience with multiple programming languages. One of the trainees had neither additional advanced degrees nor formal training in computer programming but had a strong interest in acquiring basic programming skills as part of the DSP. None had prior formal training in AI-ML theory or model development.

Mentorship

Adequate mentorship is critical for growth and productivity in clinical research and personal development (19). The structure of the DSP incorporates a layered/overlapping mentorship approach (20). The CCDS Director of Research (K.P.A.), who is also nonclinical faculty of radiology at Brigham and Women’s Hospital Department of Radiology, served as chief mentor for each trainee, overseeing integration into the CCDS workflow, project selection, research guidance, networking opportunities, and core curricular didactics. Two residents were comentored by a lead clinical project manager at the CCDS, and the third resident was mentored by a clinical radiologist (M.H.R.) engaged in AI-ML research at Brigham and Women’s Hospital. This form of collaborative, layered mentorship with experts from different fields is suitable for the multidisciplinary nature of AI-ML.

Curricular Design

Trainees proposed 3-, 6-, and 8-month DSP plans based on individual learning goals and desire to reserve time for advanced clinical electives (see Fig 1). A mixture of consecutive and sporadic DSP blocks was allocated for each resident, accounting for clinical rotations, vacation, and parental leave time. Residents provided detailed reports documenting their accomplishments and progress toward predefined learning objectives at the end of a 6-week trial period for review by the Brigham and Women’s Hospital/Harvard Medical School Radiology Residency Clinical Competency Committee to receive approval for continued participation in the program.

Figure 1:

Schematic of individualized Data Science Pathway (DSP) curricula. The three pilot trainees independently devised curricular proposals for the DSP during the 3rd year. Each proposal was reviewed and approved by the Brigham and Women’s Hospital/Harvard Medical School Radiology Residency Clinical Competency Committee prior to schedule construction. Individualized 4th-year schedules are shown as 52 1-week blocks with green representing DSP time. The trainees requested different lengths of DSP time based on individual learning goals and desired balance of clinical training. DSP blocks were interpolated over the course of the year, respecting required clinical rotations and call responsibilities. AY = academic year.

The individual components of the DSP experience were different for each resident, but with common thematic elements. The curriculum was designed to address all stages of clinical ML model development, outlined in Figure 2. A brief summary of strategies to address each curricular component is outlined below.

Figure 2:

Overview of learning tasks in the Data Science Pathway (DSP). Fundamentals: Trainees begin with individualized learning plans to build foundational knowledge, including formal and informal study of relevant mathematics, coding, and AI-ML theory. Data curation: Trainees are heavily involved in cohort selection, data cleaning, and imaging annotation throughout the DSP. Model development: Trainees collaborate with data scientists to design and test algorithms with definable clinical end points and play a critical role in performance analysis and quality assessment. Clinical integration: As clinical personnel, DSP trainees contribute to user-interface development and orchestrate scenario-testing and workflow simulation prior to model deployment. Finally, they assess the impact of AI once translated into clinical systems at the point-of-care.

Fundamentals

Radiologists must become proficient in the terminology of AI-ML and imaging informatics to meaningfully engage in clinical data science. Our experience is that dedicated study of math, coding, and theory can be both stimulating and rewarding even for trainees with limited prior exposure to AI-ML. However, an understanding of the life cycle of an imaging study and the systems involved is also of fundamental importance. Following initial didactic sessions with the Director of Research, general concepts in AI-ML were covered through weekly journal clubs, participation in the National Imaging Informatics Course and Curriculum cosponsored by the Radiological Society of North America and Society for Imaging Informatics in Medicine (SIIM) (21), informal instruction, and daily collaboration with data scientists and other center staff.

Mathematics.—At its core, AI-ML is a subdiscipline of statistical mathematics but shares a conceptual and operational kinship with neuroscience, thermodynamics, and information theory. Many concepts can be understood schematically without detailed knowledge of the underlying mathematics, but a degree of fluency in math may be helpful if one intends to delve deeper. Differential calculus is a prerequisite for most allopathic medical schools, but review of these concepts may be of benefit to many DSP participants. A review of probability theory, including Bayesian methods, is recommended. Concepts from linear algebra and matrix calculus are also fundamental to understanding the data structures and mathematics of AI-ML algorithms. Familiarity with mathematical notation and concepts is useful in interpreting the literature. Self-directed review of these concepts may be achieved through freely available online courses (Appendix E2 [supplement]).

Basic coding.—There are many roles for radiologists in the AI-ML pipeline that do not require programming expertise; however, basic proficiency in at least one coding language is beneficial for manipulating data, particularly in notebook environments where one can simply step through another person’s code or easily implement functions from open source libraries. Python (Python Software Foundation, https://www.python.org) is an increasingly popular object-oriented programming language and has emerged as the lingua franca of AI-ML. Therefore, if your goal is ML code development, Python is a logical starting point, and there are thousands of adaptable libraries, online code repositories, and many online self-paced modules available for free. Experimentation with simplified, self-contained coding environments (eg, Kaggle Notebooks, https://www.kaggle.com/notebooks/; Google Colaboratory, https://colab.research.google.com/) also serves as a fertile opportunity for self-exploration. Trainees noted that skills in data wrangling are essential for study cohort design and metadata and results analysis.

ML theory.—Deep convolutional neural networks (CNNs) are currently the most widely used ML method employed in clinical data science, particularly image analysis (22) because of their ability to capture local spatial patterns in pixel data. We therefore advocate that DSP candidates develop familiarity with the basic architecture of CNNs. However, basic knowledge of other ML methods is useful because certain clinical questions may be better targeted with other supervised learning strategies, including support vector machines (texture analysis), random forest models, naive Bayes, and regression models (23). Unsupervised learning techniques, such as k-means clustering, affinity propagation, and Gaussian mixture modeling, may also be useful for some exploratory projects of heterogeneous datasets. Self-paced online tutorials provide a means for individualized learning in the first several weeks of the DSP (see Appendix E2 for additional readings and Figure E1 for additional online resources available at https://wfwiggins.github.io/resources.html [supplement]).

Data curation and model development.—Radiology trainees can play an important role in the strategy and tactics of formulating a tractable clinical question and defining and curating the right data cohort. Trainees can also accelerate dataset annotation and spearhead model quality control and failure analysis. The challenges of data curation and preparation can further refine trainees’ critical thinking and clinical reasoning skills. In the pilot program, residents were involved with model development at multiple stages for several ongoing projects.

Clinical integration.—DSP trainees engaged in ongoing discussions for strategic deployment of clinical AI-ML tools including cross-institutional validation studies of models for spinal stenosis grading and brain hemorrhage detection. Trainees also contributed to user-interface design for delivery of ML inference to the clinical point of care. They consulted on the clinical utility of numerous proposed projects. Going forward, DSP trainees will engage in “live” clinical beta testing of several models. This will involve recruitment of clinical radiologists to participate in simulated reading room sessions using AI-ML tools. The results will further improve model design and user-interface considerations prior to prospective clinical deployment and validation.

Scheduling and Logistics

During the spring of the 3rd year, rising senior residents in our radiology residency program must propose a curriculum for their elective time in the 4th year. The 4th year at our institution comprises 13 4-week-long blocks, including several weeks of call. Outstanding requirements for American Board of Radiology–mandated rotations in breast imaging and nuclear medicine are also completed during this year. Each resident performs a self-assessment of milestones and clinical requirements in conjunction with program leadership prior to drafting their curricular proposal (see Appendix E1 for a template that may serve as a framework for similar program proposals at other institutions [supplement]). The proposal which requires Clinical Competency Committee approval must (a) integrate an overarching theme with a narrative description of postgraduate specialty training plans and career aspirations, (b) delineate specific skills that will be developed, (c) enumerate five tangible learning objectives, and (d) identify a faculty mentor. Given the range of potential opportunities in the DSP, each resident must reflect on his or her individual needs and goals (eg, academia, AI leadership in private practice, etc). The SWOT analysis (strengths, weaknesses, opportunities, threats) serves as a useful tool in strategic planning and may serve as a template for needs assessment in DSP design (Fig E2 [supplement]) (24).

Outcomes of the Initial DSP Experience

In the pilot period, each of the three residents successfully integrated into the CCDS environment. Two of the residents participated in an array of ongoing neuroimaging projects, including the development of an automated stenosis grading algorithm for lumbar spine MRI and a suite of stroke imaging algorithms for CT and MRI. The third resident was involved in an abdominal imaging project focused on the optimization and implementation of an algorithm for automated body composition segmentation of abdominal CT examinations for large-scale clinical research (Fig 3).

Figure 3:

Individual AI-ML projects from the Data Science Pathway (DSP). Each trainee contributed to design, data curation, and model development of individual projects including, A, hemorrhage detection on CT, B, abdominal body composition, and, C, lumbar spine segmentation and stenosis assessment.

Skill Acquisition

The residents also dedicated substantial time to learning staple tools and programming languages for AI-ML based on their individual levels of prior experience. The Director of Research was available for consultation as questions came up regarding core concepts at each step of the model development pipeline. One trainee had formal weekly tutoring in Python programming by a data scientist at CCDS. This same trainee also partnered with another data scientist to enter a ML datathon competition. Residents participated in weekly ML roundtables which included journal club as well as ML concept and tool discussions.

Clinical “Value-added” to AI-ML Projects

Residents were initially introduced to the CCDS workflow and model development pipeline (Fig 2), which prepared them to engage in all aspects of AI-ML model development, including data cohort identification, data curation and annotation, model design, quality control, and failure analysis. Participation in meetings with nonclinical stakeholders (hospital leadership, external industry partners, business consultants) introduced residents to the strategic and financial considerations of AI-ML implementation. By reviewing thousands of “raw” cases as part of annotation, failure analysis, or quality assurance pipelines, two of the trainees gained substantial interpretive experience during the data curation process. Additionally, residents took a primary role in orchestrating clinical testing of algorithms prior to staged widespread clinical deployment.

Academic Productivity

Although the focus of DSP is on skill-set acquisition and contribution to the clinical aspects of medical AI-ML, there exist many opportunities for scholarship and such work is encouraged. During the initial DSP pilot year, the three trainees produced 12 accepted abstracts and gave oral presentations at several national meetings including SIIM, the Society of Abdominal Radiology, and the International Stroke Conference. Six manuscripts based on the work performed are in progress.

Teaching Opportunities

A principle objective of the DSP is an enduring curriculum for future cohorts of motivated trainees. The recirculation of knowledge from trainee to trainee is a critical component of the long-term success of DSP. The lessons learned by pilot DSP participants will improve the training experience going forward, and ongoing work will be transitioned to rising 4th-year residents. DSP trainees will provide near-peer mentorship to junior residents and aid in proposal writing, goal setting, and networking. The pilot DSP cohort developed a self-contained hands-on “Introduction to AI” module which guides novice users through the fundamental code necessary to execute a simple classification CNN. This module has been tested on multiple cohorts of trainees locally and nationally. It was presented at the 2019 SIIM Annual Meeting and was an invited topic of the monthly AI Journal Club of the American College of Radiology’s Resident and Fellows Section (25) (Fig E3 [supplement]).

Feedback from the First Cohort

Overall, the feedback from the first cohort was very positive. The trainees felt they were able to achieve their goals for skill acquisition and research experience. Academic productivity exceeded the expectations of residency program leadership. The greatest strength identified by trainees was the experiential component. Among other strengths identified was the overall flexibility of the DSP, specifically the ability to tailor the program to individual backgrounds and interests. Constructive feedback identified a desire for a more formal didactic curriculum.

Future Directions

We described our initial experience with a novel DSP for senior radiology residents. The MGH & BWH CCDS DSP provided a flexible, yet well-rounded introductory experience in AI-ML model development, as evidenced by the academic productivity of the participating residents. Residents were introduced to key concepts in medical imaging informatics, the business and regulatory considerations for commercialization of medical devices, and to the myriad of ways that AI-ML can be used to conduct groundbreaking clinical radiology research.

Our experience showed that an integrated DSP was feasible for trainees with varying technical backgrounds. Lessons learned from the first iteration of the DSP will improve the experience for subsequent trainees. Based on feedback from the first cohort, a more formal structured curriculum was developed for the second cohort of trainees (Appendix E2 [supplement]). This core curriculum covers concepts throughout the ML processing pipeline from cohort development through to model development, performance assessment, and clinical integration. Tutorial, hands-on, live, and video-taped didactic sessions are included as well as background literature references and links to useful online resources and educational materials.

Implementing the DSP at Other Institutions

This curriculum and the description of our experience may serve as a model for comparable programs at other institutions. For a brief, self-directed introduction to key concepts in AI-ML, one could complete the module referenced in Figure E3 (supplement), diving deeper into unfamiliar concepts by following the hyperlinks contained therein. However, this pathway is intended for individuals who seek a more formalized, mentored approach. For such individuals, mentors, or programs, Appendix E2 (supplement) presents a 7-week curriculum with plans for didactics, independent hands-on learning, and readings. While the existence of a dedicated center for AI-ML research was beneficial for the establishment of the DSP at MGH & BWH, this was not a necessary condition of success. The key factors that contributed to the success of the program that could be extrapolated to any other training institution were motivated learners, dedicated mentors, and integration into a multidisciplinary AI-ML research environment. It is our belief that each of these facets of the DSP experience will prepare residents to serve as leaders in multidisciplinary clinical data science and in radiology as a whole.