Abstract
Accurate disease diagnosis is the cornerstone of patient care, directly impacting the quality of healthcare delivery and patient outcomes. Developing diagnostic competence is a critical goal of pathology residency training. This study uses longitudinal data from trainees during pathology residency to examine the association between year of training and performance in four key interpretive phases: detecting critical regions, recognizing their relevance, describing histopathological features, and rendering an accurate diagnosis. Using a study set of 32 digital whole slide images of breast biopsies, 155 pathology residents from 10 U.S. academic medical centers each reviewed 14 cases across multiple years of residency training. Image viewing behavior, annotation accuracy, and diagnostic decisions were recorded and analyzed. Generalized estimating equations were used to assess the relationship between residency year and metrics capturing the four interpretive phases. Year of residency training was not significantly associated with detection of critical regions or recognizing their importance. However, each year of residency training was associated with 2% less time spent viewing critical regions (p = 0.025), and 1% more attentional coverage of the image space (p = 0.01), suggesting that residents adopt broader scanning strategies over time. Further, each year of residency training was associated with 19% higher odds of writing an accurate annotation (OR = 1.19, p < 0.001) and 18% higher odds of making an accurate diagnosis (OR = 1.18, p < 0.001). Exploratory analyses indicated that accurate feature annotation was the key predictor of diagnostic accuracy, highlighting its importance as a foundational skill in clinical decision-making. Both the ability to describe histopathological features and diagnostic accuracy increase with years of residency training, and the former appears to be a key component of the latter. Competency-based training in pathology should incorporate targeted interventions to improve histopathological feature recognition and description, which appears most likely to improve overall diagnostic performance.
Keywords: breast cancer, pathology, decision-making, medical image interpretation, eye-tracking, diagnosis, medical education, longitudinal design, accuracy
Introduction
In all areas of medicine, residency training is intended to equip physicians with the knowledge and skills necessary to achieve clinical competence in a specialty and prepare for independent practice. In anatomical/surgical pathology training programs, residents must learn to recognize and interpret the complex visual features of images present on glass or digitized slides and integrate that information with their developing clinical knowledge to render an accurate diagnosis. In examining an image, pathologists generally proceed through four interpretive phases: inspecting a case and detecting an area of significant pathology, deeming the area important for the diagnosis, recognizing perceived features, and mapping features to a diagnosis.1–3 Identifying how residency training affects each phase of the interpretive process is critical for improving the delivery and assessment of training practices and for reducing the diagnostic errors that carry serious implications for patient health.4,5
To better understand how surgical pathology training during a residency program affects the process of image interpretation, we recruited pathology residents from multiple training programs in the U.S. to view sets of breast biopsy images. While they viewed the images, we tracked their image viewing behavior to quantify experience-based changes in interpretive behavior. Only one previous study has longitudinally examined interpretive changes in pathology residents.6 In that study, 4 pathology trainees viewed the same set of 20 digitized breast biopsy whole slide images during their first through fourth years of residency training. Images were viewed at a fixed low power (i.e., no zooming or panning was permitted) while their eye movements were tracked; participants marked an area they would be interested in zooming into, then rendered a diagnosis. Results suggested that residency training reduces the amount of time taken to review a slide, and helps pathologists better prioritize attention towards relatively critical image regions. There was also high interparticipant variability in the data, demonstrating the importance of larger sample sizes for adequate statistical precision and generalizability of results.
The current study expands upon this prior work6 in breast pathology in five important ways. First, we allowed participating trainees to freely navigate the digital whole slide images while recording their zooming and panning data. This allowed us to develop a more comprehensive understanding of how trainees distribute visual attention towards various image regions. It also allowed us to collect viewing data when we could not collect eye-tracking data due to restricted site access during the SARS-CoV-2 pandemic. Second, we used a larger set of 32 standardized digital whole slide breast biopsy images each with a single consensus reference diagnoses and region(s) of interest (ROI) containing the most diagnostic features on the slide as determined by an expert panel of breast pathologists.7,8 This allowed us to understand the distribution of trainees’ visual attention across critical (ROI-related) and non-critical image regions, and the extent to which their diagnoses converged with those rendered by a consensus panel of experts. Third, in addition to tracking slide viewing behavior and diagnoses, we used a mixed methods approach to collect richer data on trainees’ subjective descriptions of critical image regions; these descriptions were scored by a second panel of expert breast pathologists. Fourth, given high inter-pathologist variability in interpretive behavior, we collected data from a relatively large group of pathology trainees from multiple residency programs across the U.S.
To isolate and quantify phases of the interpretive process, we leveraged our conceptual framework that quantifies sources of interpretive error across four phases (Figure 1). During the diagnostic interpretive process pathologists visually scan a digital whole slide image and detect critical features.1,9 Second, pathologists realize the relevance of detected features for solving the task, and focus their attention on the region.10,11 Third, pathologists recognize perceived features by drawing on their expertise to describe histopathological features in the region. Finally, pathologists map features to a diagnostic category and render a diagnosis by classifying disease progression along a diagnostic spectrum.12–14
Figure 1.
Our conceptual model of the diagnostic interpretive process, including four phases beginning with inspection of the medical image and ending with the rendering of a diagnosis.
Herein, we leverage our conceptual framework to quantify how residency training affects all four phases of the interpretive process. Our research design allows the quantification of metrics corresponding to each of the four interpretive phases, which are detailed in Table 1. These metrics allow us to understand whether a pathologist successfully detected a critical region, realized its relevance, recognized perceived histopathological features, and mapped features to a diagnosis. We made 9 hypotheses regarding how residency training year will affect interpretive behavior, centered around the interpretive phases, as detailed in Table 1.
Table 1.
Hypotheses for the four interpretive phases and overall viewing behavior, the features that will be assessed to test the hypotheses, and references to prior research that motivates the hypothesis.
| Interpretive Phase | Feature | Hypothesis |
|---|---|---|
| 1. Detect Critical Region | Binary detection or non-detection of cROI. | Training year is positively associated with detecting critical cROI.3,5,15–17 |
| Time to first cROI viewing. | Training year is negatively associated with the amount of time to first view the cROI.3,5,15–17 | |
| 2. Realize Relevance of Region | Proportion of time spent viewing the cROI relative to total viewing time. | Training year is positively associated with the proportion of cROI viewing time relative to total viewing time. |
| Overlap of drawn ROI with cROI (using Jaccard index). | Training year is positively associated with the overlap of drawn versus consensus ROIs. | |
| 3. Recognize Perceived Features | Accuracy of annotated descriptions. | Training year is positively associated with the accuracy of annotated descriptions.3 |
| 4. Map to Diagnostic Category | Diagnostic accuracy relative to consensus reference diagnosis. | Training year is positively associated with diagnostic accuracy relative to consensus reference diagnosis. |
| Overall | Viewing time in seconds. | Training year is negatively associated with viewing time.6 |
| Image coverage (proportion). | Training year is negatively associated with the proportion of the case attended to.3,5,15–17 | |
| Proportion of time spent at relatively high zoom levels. | Training year is negatively associated with the proportion of time spent at relatively high zoom levels.3,5,15–17 |
Methods
Human Research Participants Protection
All study procedures were approved by the Institutional Review Board (IRB) at the University of California, Los Angeles (Protocol #18–000327), and participants provided written informed consent.
Participating Trainees
Data were collected from 155 participants enrolled in pathology residency training at one of 10 major U.S. academic medical centers. Each trainee participated in 1–3 annual sessions during their 4-year residency training program; note that 13 participants began the study in residency and then extended participation into one year of a fellowship program.
Test Set Development
We selected a subset of images from a large set of 240 hematoxylin and eosin (H&E) stained digital whole slide images (WSI) that has been extensively validated in prior research, including the expert determination of consensus diagnoses and ROIs (hereafter we refer to these as consensus ROI, or cROI).7,8,15 The cROI was the region most supportive of the consensus diagnosis, and contained the necessary and sufficient features to reach the consensus diagnosis. From the larger set, we selected 32 WSI to use herein. The selected cases varied in consensus diagnostic category, and were divided into three test sets of 24 cases, each including 4 benign, 6 atypia, 10 ductal carcinoma in situ (DCIS), and 4 invasive cases.16 In this manner, each participant was exposed to a set of cases with the same distribution of consensus diagnoses; each test set was also similar in overall pathologist-rated difficulty based on prior research with a large sample of pathologists (N = 115). We also included one practice case, which had consensus diagnosis of invasive carcinoma.
Each case included a single H&E-stained image and a consensus reference diagnosis, representing the most clinically advanced lesion on the image (most severe diagnosis), and one or more cROI. These cROIs were never presented to participants, and were only used during data analysis. Note that cROIs generally provide less discriminatory power with cases diagnosed as benign or invasive; specifically, the features found in a benign or invasive cROI are often very similar to features found elsewhere on the image. In the case of benign, there are no concerning or atypical features anywhere on these images; in the case of invasive, supportive features are often found across nearly all imaged tissue space. We return to this topic in the Data Analysis section.
Study Equipment
Study equipment varied as a function of whether data collection was done on-site at participants’ institutions (208 sessions) versus remotely (72 sessions) due to SARS-CoV-2 access and social distancing restrictions.
For on-site data collections, a Dell Precision M4800 laptop was used to display images on a color-calibrated 22” liquid crystal display (LCD) at 1920 × 1080 resolution. WSI viewing software was developed using the Microsoft, Inc. DeepZoom Silverlight application. This software provided the capability for WSI to be viewed full screen while allowing users to view, zoom (up to 60× magnification) and pan the images, and to draw a region of interest (hereafter we refer to these as pROI; participant ROI) rectangle over an image area. Critically, this software tool continuously logged zooming and panning information alongside Cartesian image coordinates to indicate where the viewer tool was positioned in the image and the extent of zoom at any given time; these data were logged locally on the laptop using a SQL database. A SensoMotoric Instruments (Boston, MA; Model RED250; ≤ 0.5˚ accuracy at 250Hz) eye-tracking camera was attached to the display and used to collect binocular eye movement data following a 9-point calibration process for each participant; eye-tracking data are not considered herein given the restricted number of sessions available with this data type.
For remote data collections, participants were asked to use a desktop computer display to access WSI viewing software hosted on the PathcoreFlow (Pathcore Inc., Toronto, ON) platform. Similarly to the on-site data collection software, this software allowed users to display images at full screen, zoom and pan the images, and draw a pROI over an image area. It also continuously logged zooming and panning information alongside Cartesian image coordinates to indicate where the viewer tool was positioned in the image and the extent of zoom at any given time; these data were logged to the PathcoreFlow cloud-based SQL database.
In addition to recording image viewing behavior, we also used the Qualtrics web-based platform to record verbal annotations and diagnostic decisions on a histology form. The form asked participants to annotate their pROI in an open-ended text box, asking “What are the critical histopathological features in the ROI you drew? Be specific.”. It also asked participants to provide a categorical diagnosis ranging from benign to invasive (non-proliferative changes only, proliferative changes, atypia, (ductal or lobular) carcinoma in situ, and invasive carcinoma). Upon selecting one category (e.g., atypia), the form would request more specificity on the diagnosis (e.g., flat epithelial atypia, atypical ductal hyperplasia, etc). The form also asked participants to rate the case’s nuclear grade and presence of necrosis (absent, present focal, present central), and to provide Likert-based difficulty and confidence ratings.17 Ancillary tests, including immunohistochemistry (IHC) stains, were not available for the cases.
Study Locations and Procedures
Each participant remotely provided written consent and completed a demographic survey prior to their first data collection session. Participants were randomly assigned to one of the three test sets of cases. Data collection sessions were held remotely or in-person, as previously described, in a private room free from distraction. Participants would first view a practice case and be instructed on how to zoom and pan the image, draw a rectangular ROI around a critical image region, and then use the histology form to log their written annotations and categorical diagnoses. This same procedure was then followed for all 14 cases, presented in random order one at a time; after each case, the participant completed the histology form. During the first session, participants would interpret 14 novel cases. When a participant returned for a second visit as part of this study, they interpreted 9 of the same cases plus 5 novel cases; 5 new cases replaced those 5 novel cases when a participant returned for a third visit. Each successive visit was separated by at least a 9-month washout period. Details of the overall study design are in Supplementary Figure 1. After reviewing all 14 cases, participants were compensated for their time ($50USD gift card), and if eligible, invited to participate again the following year.
Data Scoring
Time series viewing behavior data, including zooming and panning information and ROI drawing information, were downloaded from the SQL database and PathcoreFlow database. Written annotations and diagnostic data were downloaded from Qualtrics, and participant diagnoses were compared to the consensus reference diagnosis to determine accuracy. As detailed in our previous research,18 we calculated a series of nine features based on viewing behavior, all of which were critical to testing the hypotheses outlined in Table 1. In Table 2, we detail descriptive statistics for each of the nine features.
Table 2.
The nine calculated features, their units, and summary statistics. For categorical features we provide the mean, and for continuous features the mean and standard deviation.
| Feature | Units | Mean or Mean (SD) |
|---|---|---|
| Detection or non-detection of cROI. | Binary (non-detection, detection) | 77% detection |
| Time to first cROI viewing. | Time in seconds | 26.2 (26.4) |
| Proportion of time spent viewing the cROI relative to total viewing time. | Proportion of time | 34% (25%) |
| Overlap of drawn ROI with cROI (using Jaccard index). | Proportion of pixels | 12% (20%) |
| Accuracy of annotated descriptions. | Binary (inaccurate, accurate) | 58% accurate |
| Diagnostic accuracy relative to consensus reference diagnosis. | Binary (inaccurate, accurate) | 50% accurate |
| Viewing time in seconds. | Time in seconds | 109.8 (72.7) |
| Image coverage. | Proportion of image pixels | 24% (17%) |
| Proportion of time spent at relatively high (quartiles 3 and 4) zoom levels. | Proportion of time in seconds | 43% (22%) |
To assess the accuracy of annotated descriptions, we asked four board-certified anatomic and clinical pathologists with sub-specialty expertise in breast pathology to manually score the accuracy of participant annotations. There were 1,583 annotations to score, corresponding to case reviews with at least a partially overlapping participant-drawn pROI versus the cROI; this method ensured that at least some of the cROI was contained within the pROI, and any extent of overlap (i.e., 1–100%) was acceptable. Note that the pROI was restricted in size. Three of the experts (authors JM, MRK, and LCL) independently reviewed subsets of the annotations, instructed to assess whether the annotation sufficiently described the most critical features in the ROI; this resulted in two experts scoring each annotation. Figure 2 depicts a sample cROI with example correct and incorrect participant-provided annotations.
Figure 2.
Example case with a consensus reference diagnosis of DCIS, with an expanded cROI and two example participant-provided annotations (one correct, one incorrect).
The experts agreed at an overall rate of 75.8% (1,200 of the 1,583 cases). For the minority of cases (24.2%, 383 cases) where the experts did not agree, a fourth expert (author DLW) independently reviewed and adjudicated the case. This resulted in a single binary metric (inaccurate, accurate) for the accuracy of each of the 1,583 written annotations.
To evaluate diagnostic decision-making errors, we compared participants’ categorical diagnoses to the consensus reference diagnosis for each case. An error was identified when a participant’s response did not match the single consensus diagnosis, which was determined from four possible diagnostic categories (i.e., benign, atypia, DCIS, invasive). Additional details about how viewing behavior features were calculated can be found in our prior publication.18
Data Analysis
To investigate associations between year of residency training (i.e., years 1–5) and each of the interpretive phases outlined in Figure 1, we fit generalized estimating equation (GEE) models using SPSS v21 (IBM, Inc., Armonk, NY). Each model used one of the nine features in Table 2 as the analytic outcome. In the case of binary outcomes (i.e., accuracy), we used a logistic model. We clustered by participant, with session number (i.e., 1, 2, 3) and trial number (i.e., 1, 2, … 14) as repeated measures; year of training was the explanatory variable of interest, and case consensus diagnosis was included as a categorical covariate. The latter was included to account for known differences in complexity and difficulty of various diagnostic categories, with atypia and DCIS tending to be more complex and difficult to diagnose relative to benign and invasive cases.7,15 For logistic models, we provide odds ratios (OR), and for models with continuous outcomes we provide beta coefficients in the original units.
Note that because cROIs on benign and invasive cases are less informationally meaningful than other cases, as previously discussed, we excluded benign and invasive cases for any analyses including the cROI-related features in Table 2. Thus, while most analyses included all 3,861 interpretations, these analyses included 2,573 interpretations.
Results
Participant demographics and the number of participants per year of residency are provided in Table 3. In total, 155 pathology residents participated, resulting in a total of 276 experimental sessions and 3,861 interpretations. Sixty-six (66) residents participated once, fifty-seven (57) participated twice, and thirty-two (32) participated three times.
Table 3.
Descriptive statistics of key participant demographic variables (age, gender, ethnicity) and participant counts by the number of years of participation in our study.
| Demographic Variable | Descriptive Statistics |
|---|---|
| Age | M = 33, SD = 4.5, Range 26–48 years |
| Gender | Male: N = 68 Female: N = 81 Other or decline to answer: N = 6 |
| Ethnicity | White: N = 95 Asian: N = 33 Black or African American: N = 7 American Indian, Alaska Native, Native Hawaiian, or Other Pacific Islander: N = 1 More than one race: N = 8 Don’t know, not sure, or decline to answer: N = 11 |
| 1 Year of Participation | N = 66 |
| 2 Years of Participation | N = 57 |
| 3 Years of Participation | N = 32 |
Phase 1. Outcomes Related to Detecting Critical Regions
Year of residency training was not significantly associated with whether participants detected the cROI, OR = 1.01, 95% CI: 0.95, 1.08, p = 0.73, nor was it associated with the time it took to first detect the cROI, β = 0.78, 95% CI: 0.58, 8.58, p = 0.25.
Phase 2. Outcomes Related to Realizing the Relevance of Critical Regions
Year of residency training was negatively associated with the proportion of time spent viewing the cROI relative to other image regions, β = −0.013, 95% CI: −0.02, −0.002, p = 0.025. Specifically, for each additional year of residency training, residents spent on average 1.3% less interpretation time viewing the cROI. Year of residency training was not, however, associated with the extent of overlap between the drawn ROI relative to cROI, β = −0.006, 95% CI: −0.013, 0.002, p = 0.13.
Phase 3. Outcome Related to Recognizing Perceived Features
Year of residency training was positively associated with the accuracy of written annotations. On average, residents with an additional year of residency training had an average 19% higher odds of accurately describing features, OR = 1.19, 95% CI: 1.07, 1.32, p < 0.001.
Phase 4: Outcomes Related to Mapping Features to Diagnostic Category
Year of residency training was positively associated with diagnostic accuracy. On average, residents with an additional year of residency training had 18% higher odds of accurately diagnosing, OR = 1.18, 95% CI: 1.1, 1.26, p < 0.001.
Overall Outcomes
Written annotation accuracy and diagnostic accuracy relative to the consensus reference diagnosis, showed substantial variation as a function of case consensus diagnosis, as detailed in Table 4. Diagnostic accuracy levels are similar to those found in other studies examining both trainees and practicing pathologists.
Table 4.
Accuracy of written annotations and categorical diagnoses by consensus diagnostic category. For example, 81% of interpretations by residents accurately annotated cases that the consensus panel classified as benign, with 89% diagnosing them accurately.
| Benign | Atypia | DCIS | Invasive | |
|---|---|---|---|---|
| Annotation Accuracy | 81% | 50% | 50% | 82% |
| Diagnostic Accuracy | 89% | 22% | 29% | 93% |
Testing our hypotheses, year of residency training was not significantly associated with viewing time in seconds, β = 2.88, 95% CI: −1.19, 6.95, p = 0.17. It was, however, positively associated with image coverage, with higher training years associated with broader attention over the full image, β = 0.01, 95% CI: 0.002, 0.02, p = 0.01. Specifically, each year of residency training was associated with an average 1% more image coverage in their review of cases. Finally, year of residency training was not significantly associated with the proportion of time spent at relatively high zoom levels, β = 0.003, 95% CI: −0.006, 0.012, p = 0.5.
Exploratory Analysis
An exploratory analysis investigated associations between eight features (i.e., all Table 2 features other than diagnostic accuracy) and diagnostic accuracy, while adjusting for year of residency training. One feature was significantly associated with diagnostic accuracy: comparing interpretations with an accurate vs. inaccurate written annotation for residents in the same year of training, the odds of an accurate diagnosis were more than 8 times higher when the written annotation was accurate, OR = 8.63, 95% CI: 6.28, 11.85, p < 0.001.
As a sensitivity analysis, we explored whether residents’ number of weeks of breast pathology training influenced any of our primary findings. This exploratory analysis was motivated by the fact that pathology residency programs are structured differently across institutions, and residents have highly variable experience interpreting breast pathology over the course of their residency (overall mean weeks of breast pathology training was 2.97, with a standard deviation of 0.98). To examine this potential confounding variable, we included the number of weeks of breast pathology training (self-reported data collected at each session) as a covariate in all analyses. The results are presented in Supplementary Table 1. Most of our primary results were similar with the directionality, effect sizes, and significance levels of relationships remaining stable when accounting for residents’ experience interpreting breast pathology. One outcome, annotation accuracy, continued to show a positive relationship with year of residency training, but the association was no longer statistically significant (p = 0.08). Importantly, the direction and overall magnitude of the effect remained similar (odds ratio = 1.19 vs. 1.13). Note that subjective measures, such as self-reported training duration, are inherently limited in precision and reliability when compared to objective, performance-based metrics.
Discussion
This is the first study to follow a large number of pathology residents from multiple training programs over the course of their training to assess the effects of residency training on the diagnostic interpretive process. We found mixed support for our hypotheses. To better understand how training influences the interpretive process, we examined its effects across the four interpretive phases: detecting critical regions, realizing their relevance, recognizing perceived histopathological features, and mapping to a diagnosis (Figure 1). For the first phase, the year in residency training was not significantly associated with the likelihood of detecting a cROI, nor with the latency to first view a cROI. This suggests that the ability to identify critical histopathological regions may develop early in training and remain relatively stable across residency. Indeed, overall rates of detecting the cROI were high, approximating 73% (similar to prior research)3, and did not vary significantly by year of residency.
For the second phase, residency training year was negatively associated with the time spent viewing the cROI relative to other regions, suggesting that as residents progress in their training they develop relatively broad visual search strategies.1,5,6,19–21 This suggests that trainees may increasingly use a broader approach to viewing an image, complemented by the finding that training year is also associated with attentional coverage of more of the image. Gaining expertise may partly involve an ability to focus on and then dynamically shift attention away from the cROI to gain a more holistic understanding of the image. This converges with extant theory that suggests that expertise is associated with developing global-focal-global scanning patterns to help detect abnormalities while maintaining holistic awareness of the entire image.22,23 However, we also found that residents do not necessarily improve in precisely delineating the most diagnostically relevant regions. Indeed, residents’ pROI only overlapped the cROI about 43% of the time, and the extent of overlap was only 12% overall and did not vary significantly by year of residency. The generally low extent of overlap is likely due to the restricted size of both cROIs and pROIs, and the fact that cROI-adjacent regions may contain sufficient information to reach the consensus diagnosis, despite not being included in the cROI. It is also worth considering that the observed lack of strong training effects on certain visual metrics, such as time spent on cROI, could reflect unique characteristics of WSI navigation. Compared to traditional microscope-based workflows, WSI may afford broader, more flexible image viewing strategies that reduce the need for narrowing focus on specific regions for extended periods of time. This flexibility may, in turn, flatten the trajectory of change in visual search behavior through training. As digital pathology continues to become more central in diagnostic pathology, training research may need to explicitly account for differences in how visual expertise develops in digital versus analog environments.
For the third phase, residency training year was strongly and positively associated with the accuracy of written descriptions of cROI features. This finding indicates that residents become better at verbalizing critical histopathological features as they gain experience, supporting models of expertise development which suggest that structured exposure and feedback help refine cognitive processing and the ability to articulate complex diagnostic criteria.24–26 Finally, for the fourth phase, residency training year was strongly and positively associated with diagnostic accuracy, suggesting that experience enhances the ability to integrate visual information with clinical knowledge to arrive at the correct diagnosis. Notably, our exploratory analyses showed that the written annotation accuracy showed a strong association with diagnostic accuracy, highlighting the potential importance of accurate feature recognition and description in arriving at an accurate diagnosis.
Overall, results suggest that while training supports some aspects of the interpretive process, particularly feature description and diagnostic decision-making, other aspects such as visual search and attention allocation may not follow a linear trajectory of change during training.
Theoretical and Clinical Implications
Our findings contribute to theoretical models of medical competency development by suggesting that interpretive expertise in pathology does not uniformly improve across all phases of the diagnostic process. The observed increases in annotation accuracy and diagnostic accuracy with training supports stage-based models of expertise, which posit that structured exposure and feedback refine perceptual and cognitive processing skills.24,27 However, the generally high rates of detecting critical regions, and the lack of significant improvement in detecting them, suggests that visual search strategies may develop early in training and remain relatively stable thereafter. It could also be the case that clinically relevant regions also tend to be visually salient on H&E staining, automatically attracting attention even in the absence of expertise.28 However, the decline in relative time spent on critical regions suggests that trainees may adopt a broader scanning strategy as their experience develops, potentially balancing depth of analysis with gathering a “Gestalt” of the entire image.20,21 These findings underscore the importance of competency-based training models that assess and target specific aspects of the interpretive process rather than assuming uniform progression across all skills.
Recent advancements in artificial intelligence (AI) have introduced large language models (LLMs) that can generate descriptive text for pathology images.29,30 Researchers are now exploring whether combining these text descriptions, and/or the descriptions produced by pathologists, with visual data can enhance the performance of AI algorithms in accurately classifying pathology cases. This emerging approach aligns with our finding that improved feature recognition and description are strongly associated with diagnostic accuracy. Beyond augmenting diagnosis, such tools offer clear potential for educational applications. For instance, AI-based systems could provide real-time feedback on written annotations, suggest diagnostic regions of interest for comparison, or serve as natural language-based diagnostic tutors that help residents articulate and refine their reasoning. Incorporating text descriptions into training workflows could offer an additional means of improving residents’ ability to identify and articulate key diagnostic features.31 Future research could explore whether integrating expert-provided or AI-assisted descriptions into computer-aided diagnosis (CAD) systems may improve diagnostic outcomes.
From a clinical perspective, our results highlight the need for training interventions that emphasize critical bottlenecks in the interpretive process. The strong relationship between annotation accuracy and diagnostic accuracy suggests that structured verbalization and feature description exercises could enhance diagnostic precision.32–34 Residency programs may benefit from incorporating targeted feedback mechanisms that reinforce not only correct diagnoses but also effective recognition and description of histopathological features.
Limitation and Future Directions
To our knowledge, this study represents the largest longitudinal examination of diagnostic and interpretive behavior of pathology residents. Strengths of our study include the large sample size, the inclusion of residents from 10 different training programs, the use of a standardized set of digital whole slide images with consensus reference diagnoses and defined ROIs, and the comprehensive quantitative assessment of all four interpretive phases. However, future research will benefit from a few methodological changes. First, while we tracked residents across multiple years in a longitudinal manner, not all participants completed multiple sessions, limiting the ability to study within-subject changes over time. Second, due to limited site access during the SARS-CoV-2 pandemic, eye movement data were sparsely collected within and across participants; more fine-grained visual search metrics derived from eye-tracking data may reveal patterns that viewport tracking (i.e., zooming and panning behavior) cannot. Third, data collection occurred both on-site and remotely, which could introduce variability in how participants interacted with images, despite our efforts to standardize the data collection experience. Fourth, while we examined a specialty of pathology with critical implications for patient health, future research should examine the generalizability of our finding to other tissue types. Each of these limitations presents an opportunity for continuing research. Furthermore, given that diagnostic errors can have serious consequences for patients, future studies should also evaluate the effectiveness of training interventions designed to improve the accuracy and consistency of feature recognition and description in pathology interpretation.
Conclusion
This study expands our understanding of how residency training influences the various phases of pathology image interpretation. Overall, training appears to increase the breadth of visual focus on images and improve both feature description and diagnostic accuracy. These findings underscore the need for targeted training approaches that emphasize optimal interpretive strategies, particularly for the phases of feature recognition and description. Refining training methods based on empirical evidence can help residency programs better prepare pathologists for the demands of clinical practice and ultimately enhance diagnostic reliability and patient outcomes.
Supplementary Material
Acknowledgements
We wish to thank Ventana Medical Systems, Inc., a member of the Roche Group, for use of iScan Coreo Au™ whole slide imaging system, and HD View SL for the source code used to build our digital viewer. For a full description of HD View SL please see http://hdviewsl.codeplex.com/. We also wish to thank the staff, faculty, and trainees at the various pathology training programs across the U.S. for their participation and assistance in this study.
Funding
Research reported in this publication was supported by the National Cancer Institute of the National Institutes of Health under award number R01 CA225585, R01 CA172343, R01 CA140560. The content is solely the responsibility of the authors and does not necessarily represent the views of the National Cancer Institute or the National Institutes of Health. Researchers are independent from funders, and the funding agency played no role in study design, conduct, analysis or interpretation.
Footnotes
Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
Ethics Approval and Consent to Participate
All participants provide written informed consent in accordance with Institutional Review Board approvals granted by the University of California Los Angeles.
Data Availability Statement
Due to the highly specialized nature of participant expertise and therefore increasing risk of identifiable data, we have decided not to make our data available in a repository. In the interest of minimizing the risk of participant identification, we will distribute study data on a case-by-case basis. Interested parties may contact Hannah Shucard at the University of Washington with data requests: hshucard@uw.edu
References
- 1.Brunyé T, Drew T, Weaver DL & Elmore JG A review of eye tracking for understanding and improving diagnostic interpretation. Cogn. Res. Princ. Implic 4, 1–16 (2019). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Kundel HL & Nodine CF Studies of eye movements and visual search in radiology. in Eye Movements and the Higher Psychological Processes (eds. Senders JW, Fisher DF & Monty RA) 317–327 (Lawrence Erlbaum Associates, Hillsdale, NJ, 1978). [Google Scholar]
- 3.Brunyé TT, Balla A, Drew T et al. From Image to Diagnosis: Characterizing Sources of Error in Histopathologic Interpretation. Mod. Pathol 36, 100162 (2023). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Elstein AS & Schwarz A Clinical problem solving and diagnostic decision making: selective review of the cognitive literature. BMJ 324, 729–732 (2002). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Krupinski EA, Tillack AA, Richter L et al. Eye-movement study and human performance using telepathology virtual slides. Implications for medical education and differences with experience. Hum. Pathol 37, 1543–1556 (2006). [DOI] [PubMed] [Google Scholar]
- 6.Krupinski EA, Graham AR & Weinstein RS Characterizing the development of visual search experience in pathology residents viewing whole slide images. Hum. Pathol 44, 357–364 (2013). [DOI] [PubMed] [Google Scholar]
- 7.Elmore JG Longton GM, Carney PA et al. Diagnostic Concordance Among Pathologists Interpreting Breast Biopsy Specimens. JAMA 313, 1122 (2015). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Oster N, Carney PA, Allison KH et al. Development of a diagnostic test set to assess agreement in breast pathology: Practical application of the Guidelines for Reporting Reliability and Agreement Studies (GRRAS). BMS Womens Health 13, 3 (2013). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Al-Moteri MO, Symmons M, Plummer V & Cooper S Eye tracking to investigate cue processing in medical decision-making: A scoping review. Comput. Hum. Behav 66, 52–66 (2017). [Google Scholar]
- 10.Logothetis NK & Sheinberg DL Visual object recognition. Annu. Rev. Neurosci 19, 577–621 (1996). [DOI] [PubMed] [Google Scholar]
- 11.Gauthier I & Tarr MJ Visual Object Recognition: Do We (Finally) Know More Now Than We Did? Annu. Rev. Vis. Sci 2, 377–396 (2016). [DOI] [PubMed] [Google Scholar]
- 12.Sox HC, Blatt MA, Higgins MC & Marton KI Medical Decision Making. (Butterworths, Boston, MA, 1988). doi: 10.1002/9781118341544. [DOI] [Google Scholar]
- 13.Ledley RS & Lusted LB Reasoning foundations of medical diagnosis. Science 130, 9–21 (1959). [DOI] [PubMed] [Google Scholar]
- 14.Kassirer JP, Kopelman RI & Wong JB Learning Clinical Reasoning. 332 (Williams & Wilkins, Baltimore, MD, 1991). [Google Scholar]
- 15.Allison KH Reisch LM, Carney PA et al. Understanding diagnostic variability in breast pathology: lessons learned from an expert consensus review panel. Histopathology 65, 240–51 (2014). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Onega T, Weaver DL< frederick PD et al. The diagnostic challenge of low-grade ductal carcinoma in situ. Eur. J. Cancer 80, 39–47 (2017). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Likert R A technique for the measurement of attitudes. Arch. Psychol 22 140, 55–55 (1932). [Google Scholar]
- 18.Brunyé TT et al. Machine learning classification of diagnostic accuracy in pathologists interpreting breast biopsies. J. Am. Med. Inform. Assoc 31, 552–562 (2024). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Nodine CF & Kundel HL Using eye movements to study visual search and to improve tumor detection. Radiographics 7, 1241–1250 (1987). [DOI] [PubMed] [Google Scholar]
- 20.Kundel HL, Nodine CF, Conant EF & Weinstein SP Holistic component of image perception in mammogram interpretation: gaze-tracking study. Radiology 242, 396–402 (2007). [DOI] [PubMed] [Google Scholar]
- 21.Kundel HL, Nodine CF, Krupinski EA & Mello-Thoms C Using gaze-tracking data and mixture distribution analysis to support a holistic model for the detection of cancers on mammograms. Acad. Radiol 15, 881–6 (2008). [DOI] [PubMed] [Google Scholar]
- 22.Brunyé TT, Mercan E, Weaver DLDL & Elmore JGJG Accuracy is in the eyes of the pathologist: The visual interpretive process and diagnostic accuracy with digital whole slide images. J. Biomed. Inform 66, 171–179 (2017). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Manning D, Ethell S, Donovan T & Crawford T How do radiologists do it? The influence of experience and training on searching for chest nodules. Radiography 12, 134–142 (2006). [Google Scholar]
- 24.Schmidt HG & Boshuizen HPA On acquiring expertise in medicine. Educ. Psychol. Rev 5, 205–221 (1993). [Google Scholar]
- 25.Higgs J, Jones MA, Loftus S & Christensen N Clinical Reasoning in the Health Professions E-Book: Clinical Reasoning in the Health Professions E-Book. (Elsevier Health Sciences, 2008). [Google Scholar]
- 26.Ericsson KA Deliberate Practice and the Acquisition and Maintenance of Expert Performance in Medicine and Related Domains. Acad. Med (2004) doi: 10.1097/00001888-200410001-00022. [DOI] [PubMed] [Google Scholar]
- 27.Norman G Research in clinical reasoning: Past history and current trends. Med. Educ 39, 418–427 (2005). [DOI] [PubMed] [Google Scholar]
- 28.Brunyé TT, Carney PA, Allison KH et al. Eye movements as an index of pathologist visual expertise: A pilot study. PLoS ONE 9, (2014). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29.Sun Y, Zhang Y Si Y et al. PathGen-1.6M: 1.6 Million Pathology Image-text Pairs Generation through Multi-agent Collaboration. Preprint at 10.48550/arXiv.2407.00203 (2024). [DOI] [Google Scholar]
- 30.Cheng J Applications of Large Language Models in Pathology. Bioengineering 11, 342 (2024). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 31.Lu MY, Chen B, Williamson DFK et al. A multimodal generative AI copilot for human pathology. Nature 634, 466–473 (2024). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 32.Ericsson KA & Simon HA Verbal Reports as Data. Psychol. Rev 87, 215–251 (1980). [Google Scholar]
- 33.Chi MTH, De Leeuw N, Chiu M-H & Lavancher C Eliciting self-explanations improves understanding. Cogn. Sci 18, 439–477 (1994). [Google Scholar]
- 34.Sluijter CE, van Lonkhuijzen LRCW, van Slooten H-J, Nagtegaal ID & Overbeek LIH The effects of implementing synoptic pathology reporting in cancer diagnosis: a systematic review. Virchows Arch 468, 639–649 (2016). [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Data Availability Statement
Due to the highly specialized nature of participant expertise and therefore increasing risk of identifiable data, we have decided not to make our data available in a repository. In the interest of minimizing the risk of participant identification, we will distribute study data on a case-by-case basis. Interested parties may contact Hannah Shucard at the University of Washington with data requests: hshucard@uw.edu