Improving Radiology Oncologic Imaging Trainee Case Diversity through Automatic Examination Assignment: Retrospective Study from a Tertiary Cancer Center

Abstract

In a retrospective single-center study, the authors assessed the efficacy of an automated imaging examination assignment system for enhancing the diversity of subspecialty examinations reported by oncologic imaging fellows. The study aimed to mitigate traditional biases of manual case selection and ensure equitable exposure to various case types. Methods included evaluating the proportion of “uncommon” to “common” cases reported by fellows before and after system implementation and measuring the weekly Shannon Diversity Index to determine case distribution equity. The proportion of reported uncommon cases more than doubled from 8.6% to 17.7% in total, at the cost of a concurrent 9.0% decrease in common cases from 91.3% to 82.3%. The weekly Shannon Diversity Index per fellow increased significantly from 0.66 (95% CI: 0.65, 0.67) to 0.74 (95% CI: 0.72, 0.75; P < .001), confirming a more balanced case distribution among fellows after introduction of the automatic assignment.

© RSNA, 2023

Keywords: Computer Applications, Education, Fellows, Informatics, MRI, Oncologic Imaging

Summary

An automated imaging examination assignment system effectively increased the diversity of subspecialty examinations and the number of uncommon cases reported by oncologic imaging fellows.

Key Points

• ■ The implementation of an automated imaging examination assignment system doubled the number of uncommon cases reported by oncologic imaging fellows.

• ■ Use of the system also led to a statistically significant increase in the Shannon Diversity Index from 0.66 to 0.74 (P < .001), indicating a more equitable distribution of cases among fellows and a more comprehensive training experience.

• ■ These results suggest that technology can be leveraged to enhance the training experience in oncologic imaging and ensure that trainees are prepared for the challenges that will be encountered in real-world practice.

Introduction

Dedicated clinical training programs are vital cornerstones of radiology education. Specialty-trained radiologists exhibit, for example, increased cancer detection rates (1) and fewer interpretation discrepancies (2). The overarching objective of these programs is to ensure a thorough grasp of specialized clinical and radiologic knowledge through wide exposure to diverse clinical scenarios within each specialty. While some of this can be achieved through reviewing archived case libraries, teaching files, and published literature, it is widely considered that direct hands-on reporting under real-life conditions, supervised by dedicated subspecialty faculty, forms the centerpiece of radiology training (3).

However, limited availability of certain types of examinations, especially those performed less commonly or for rarer indications, can impede a trainee's ability to acquire the necessary skills and knowledge within the duration of the training program. This becomes even more pronounced in departments where traditional workflows, which allow trainees to review all performed examinations before consultation with faculty, are rendered infeasible owing to surging clinical workloads. Traditional methods for selecting examinations for first-line reporting by trainees, such as radiologic worklists with manual selection by instructors or self-selection by fellows, are prone to biases. One notable bias is preferential selection of cases with perceived low difficulty, colloquially known as cherry-picking. This not only results in a lack of exposure to certain types of examinations but also has a potential negative effect on turnaround times for uncommon and challenging cases and, ultimately, may lead to delays in patient care (4).

An automated imaging examination assignment system, which algorithmically assigns studies for reporting considering the trainee's progression level (be it resident or fellow) and case availability, has the potential to improve case balance and diversity. The purpose of this study was to evaluate the increase in case diversity after implementation of an automatic case assignment system for trainees within a dedicated oncologic body MRI training curriculum.

Materials and Methods

Examination Assignment Algorithm and Software

This retrospective study was Health Insurance Portability and Accountability Act compliant and institutional review board exempt as an institutional quality-improvement project. The implementation of the automated examination assignment software was described elsewhere in detail (5). In short, every MRI examination was assigned a category based on one or more of the following factors: (a) the anatomic coverage (ie, abdomen, pelvis); (b) image acquisition protocol (ie, MR cholangiogram, renal mass evaluation); (c) the known or suspected cancer site or type (eg, prostate, rectum, lymphoma, etc), as entered by the ordering provider in the electronic examination request; and (d) the referring physician's subspecialty. For example, an MRI examination of the prostate is automatically placed in the genitourinary category, whereas a CT examination of the chest, abdomen, and pelvis for staging may be placed in the genitourinary category if referred by a urologist, in the hepatobiliary category if referred by a liver surgeon, or in the general category if referred by a medical oncologist and the purpose is general staging of the disease. Then, the list of available examinations was stratified by category (ie, ordered so that the categories were spread evenly across the whole list). For instance, an examination list with three possible categories A, B, and C was ordered ABCABCABC instead of AAABBBCCC.

This fully automated process was executed by the software at several time points throughout the day. At each time point, trainees received their cases in batches, where each batch contained a diverse case mix with various disease categories. The only manual input to this process was that the number of cases per batch was gradually increased by the program director throughout the duration of the 1-year fellowship, to account for increasing confidence and reporting speed as the training advanced. For example, in the first quarter of the fellowship, the abovementioned example list would be split as follows among three fellows: AB + CA + BC, whereas in the second quarter this may increase to ABC + ABC + ABC, and later to ABCA + BCAB + CABC and so on.

Study Design

Given the high examination volumes, a large proportion of cases at our institution are reported by faculty alone. Moreover, because there is no stand-alone residency program at the institution, only a few external residents from an affiliated academic center rotate through the MRI section at irregular intervals. Given that resident seniority varies widely, case selection is done manually from picture archiving and communication system worklists according to the individual level of comfort, with guidance by attending physicians.

All MRI examinations initially interpreted by a trainee were extracted from our radiology information system. Two separate study periods were defined. In the period defined as “before” (July 2019–June 2020), trainees selected cases off a single worklist ad libitum. In the period “after” (July 2021–June 2022) examinations were assigned automatically. Data from the period in between were excluded given the extraordinary circumstances and workflow during the COVID-19 pandemic. Uncommon disease categories were defined as constituting less than 10% of the total examination volume in the “after” period.

Statistical Analysis

The Shannon Diversity Index (entropy) was calculated per week and per trainee: Assignments being evenly distributed over 10 different subspecialties would result in an index of 1, whereas assignments all falling within a single subspecialty would result in an index of 0. Relative changes for different subspecialties were measured in percentages.

Entropy was compared using a two-tailed Welch two-sample t test. Differences in assignment categories before and after implementation of the automatic assignment software were compared with a χ² test. Statistical significance was set at a P value of < .05 as per common consensus. All calculations were performed in R version 4.3.0 (https://www.r-project.org/).

Results

The mean number of examinations per trainee before and after automatic assignment were comparable (before, 756 ± 206 [SD]; after, 761 ± 286). There were 11 fellows in 2019 and 16 fellows in 2021. Given the growth of the fellowship program and increase in class size, both percentages and absolute numbers are reported in the Table. The distribution of cases was significantly different before and after automatic assignment (P < .001).

Breakdown of MRI Fellow Reports by Disease Category before and after Implementation of Automatic Case Assignment

Before automatic assignment, the five highest-volume categories, or common cases (ie, gynecologic, hepatobiliary, musculoskeletal, genitourinary, and general), constituted 91.3% of examinations reported by fellows. This value decreased to 82.3% after automatic assignment. Of the total examinations, those from the four lowest-volume categories (ie, iron quantification [liver], multiple myeloma, syndromes [usually whole-body screening examinations], and lymphoma) more than doubled from 0.8% before to 3.3% after automatic assignment. Overall, the total number of reported uncommon cases increased by 9.1% from 8.6% to 17.7%. The data are summarized in the Table and Figure.

Alluvial diagrams demonstrate (A) overall case distribution and (B) increase of uncommon (defined as < 10%) disease categories among fellows before and after implementation of automatic case assignment. For illustrative purposes, the “general” category (from 8.4% to 12.1%) is displayed in A.

The mean weekly entropy increased significantly (P < .001) between the two study periods from 0.66 (95% CI: 0.65, 0.67) to 0.74 (95% CI: 0.72, 0.75), while the mean number of examinations per week and fellow remained unchanged at 16.8 ± 6.9 before and 16.3 ± 3.4 after automatic assignment.

Discussion

In addressing challenges in radiology training, we assessed an automated imaging examination assignment system for oncologic imaging. Our primary motivation was to counteract biases in traditional assignment methods, such as cherry-picking. Key findings include roughly a doubling in exposure to rare conditions and a significant (P < .001) increase in case diversity (ie, breadth of exposure), highlighting the potential for tailored software to improve subspecialized training.

In the present study, we used a handcrafted algorithm to ensure case diversity. In recent years, numerous efforts have been undertaken to assign cases with the help of artificial intelligence. Usually, this is done to prioritize clinically relevant or urgent findings (6–8). However, because artificial intelligence has the power to take into account imaging findings before a radiologist sees them, future projects may focus on building systems that attempt to route imaging findings to trainees that the trainees have not previously encountered.

The main limitation of this study was its single-center retrospective design. Hence, applicability to other centers would require adaptations to the respective workflow. For example, the lack of a dedicated residency program made our focus on fellows possible without a detrimental effect on resident education. However, at institutions with a residency program, all levels of trainees ought to be considered when implementing automatic case assignment. Differences in educational objectives and clinical experience necessitate a nuanced approach when considering automatic case assignment for residents. Residents who are in their early formative years need exposure to a wide spectrum of cases to build a foundational understanding of diagnostic imaging. This breadth is considered crucial to develop a solid base before they delve into subspecialized domains. An automated assignment system for residents could prioritize giving them a broad mix of common and uncommon cases across multiple modalities. Moreover, having access to the resident's cases from the beginning of training could ensure that the resident receives exposure to previously unseen or only sparsely seen imaging protocols or body regions.

An additional positive side effect of automatic case assignment is the setting of a clear numerical target for the minimum number of image interpretations per day. First, it equips both residents and fellows with transparent milestones, offering them a clear vantage point to assess their evolution and ensuring they maintain the desired training pace. From the educator's lens, it instills a systematic standardization in the training process. By guaranteeing that each trainee reviews a set number of specific common abnormalities by a certain point in their training, for instance, there is increased confidence in the consistent attainment of foundational competencies. This ensures a uniform quality in training outcomes, elevating the general proficiency of all graduating trainees. Last, this model may ease the feedback process. Instructors can swiftly spot trainees who may need additional support and can promptly provide the necessary guidance or resources. Conversely, for trainees who consistently outperform, advanced learning avenues or elective rotations could be unlocked sooner, thus enriching their educational experience.

In conclusion, implementation of automated case assignment in our subspecialized oncologic imaging fellowship program led to a significant increase in exposure to rare conditions and infrequently performed imaging protocols. The increased exposure potentially allows fellows to develop a deeper understanding of uncommon and challenging cases, improving their competence in interpretation. This enhanced competence is crucial for fellows’ future practice because it prepares them for a more specialized and diverse caseload. These findings bear potential significance for shaping innovative case allocation strategies for residents and fellows, impacting radiology education and molding the expertise of the next generation of radiologists.