Teaching search patterns to medical trainees in an educational laboratory to improve perception of pulmonary nodules

Abstract.

The goal of this research is to demonstrate that teaching healthcare trainees a formal search or scan pattern for evaluation of the lungs improves their ability to identify pulmonary nodules on chest radiographs (CXRs). A group of physician assistant trainees were randomly assigned to control and experimental groups. Each group was shown two sets of CXRs, each set with a nodule prevalence of approximately 50%. The experimental group received search pattern training between case sets, whereas the control group did not. Both groups were asked to mark nodules when present and indicate their diagnostic confidence. Subject performance at nodule detection was quantified using changes in area under the localization receiver operating characteristic curve ($\mathrm{\Delta }\mathrm{AUC}$). There was no significant improvement in performance between case sets for the control group. There was a significant improvement in subject performance after training for the experimental group, $\mathrm{\Delta }\mathrm{AUC}=0.1539$, $p=0.0012$. These results demonstrate that teaching a search pattern to trainees improves their ability to identify nodules and decreases the number of perceptual errors in nodule identification, and suggest that our knowledge of medical image perception may be used to develop rational tools for the education of healthcare trainees.

1. Introduction

Scientists studying medical image perception have learned much about how human observers perceive medical images and arrive at diagnostic decisions.¹ To date, there has been limited translation of this knowledge into developing practical educational tools for teaching skills of image perception and interpretation to healthcare trainees. However, it is likely that an understanding of human image perception may facilitate the rational development of educational tools to help improve medical providers’ skills at image perception. The presented research addresses one particular aspect of image perception that may benefit from targeted training. The goal of this project is to demonstrate that teaching healthcare trainees a comprehensive search pattern for evaluation of the lungs during an educational laboratory will improve their ability to identify pulmonary nodules on chest radiographs (CXRs).

Medical errors occur for a variety of reasons; a subset are due to errors in perception and identification of an image finding as abnormal. One system for categorizing errors, developed by Kundel et al.,² divides perceptual errors into scan errors, recognition errors, and decision errors. It was shown that scanning errors, failure to fixate on an abnormality, accounted for almost one-third of all perceptual errors. It has been further shown that novices cover less of an image with their useful central field-of-view when compared with expert observers.³ It has been noted that experts tend to view images using a free search pattern,⁴ and not the directed systematic search patterns that are often taught to trainees. Part of the reason experts may be better at perceiving abnormalities is believed to be due to their ability to form a gestalt impression of an image.^5,6 However, it may be argued that novices without the ability to generate a rapid and accurate gestalt impression of an image may benefit from a more comprehensive search pattern when compared with experts. This raises the question of whether or not teaching a comprehensive search pattern to healthcare professionals with less image-interpretation experience would lead to improved ability to identify abnormalities.

Prior research has shown that teaching a search pattern for comprehensive evaluation of the lungs to medial trainees on an individual basis improved their ability to identify pulmonary nodules on CXR.⁷ After individual search pattern training (SPT) at a simulated radiology workstation (SRW), medical trainees demonstrated an improved ability to identify pulmonary nodules. In addition, trainees felt that the training was more effective when compared with conventional educational methods. While the individual training was effective for the education of trainees, it was not very time-efficient on the part of the radiologist facilitating the educational session, as roughly only one trainee could undergo the educational protocol per faculty-hour.

Similar to educational/simulation laboratories used in other fields,⁸ an educational laboratory may be an effective way to teach and assess skills related to image perception. The goal of this research is to demonstrate that teaching a search pattern to trainees in an educational computer laboratory decreased the number of errors in nodule identification.

2. Methods

2.1. Subjects

This study received an “Exemption of Human Subjects Research” from our institutional review board. Subjects consisted of physician assistant (PA) students at our institution. Participation in this study was voluntary. This population was selected because we have a large number of PA students with a similar level of medical training and experience who were able and willing to participate in this study. Prior to participation, all subjects provided written informed consent. Forty-eight PA students participated, 24 in the control group and 24 in the experimental group. At our institution, all PA students receive education on radiology in general, and chest radiography in particular, as a part of their standard curriculum.

2.2. Database of Images

Fifty normal frontal CXRs were selected from our institution’s clinical image database. Each digital imaging and communication in medicine (DICOM) format image was anonymized prior to use. Each CXR had a normal computed tomography scan completed within 1 month of the radiograph’s acquisition, and each scan was reviewed by a board-certified and fellowship-trained cardiothoracic radiologist to confirm that the study was normal.

To achieve the best level of case difficulty for our study population, simulated pulmonary nodules were used. All code for insertion of simulated nodules were written in MATLAB (MathWorks, Natick, Massachusetts). Half of the CXR cases contained nodules ($n=25$), with only one nodule per case. Nodules were simulated as the projection of a sphere on a blank background and added to the CXRs. Prior studies have shown that nodules simulated using analogous methods are indistinguishable from true nodules.⁹ The location of each nodule was determined randomly, but a cardiothoracic radiologist visually confirmed that each nodule was in an anatomically valid location overlying the lungs and indistinguishable from a true pulmonary nodule. The diameters of the nodules were randomly distributed between 10 and 20 mm. The simulated nodule served as the ground truth for confirming correct lesion localization during data analysis.

2.3. Educational Computer Laboratory

The study was conducted in two computer classrooms, one room for the control group and the other for the experimental group. There was one study proctor per room; both were cardiothoracic radiologists. In each room, subjects received training at their own SRW. SRWs were configured to be similar to clinical radiology picture archiving and communication system (PACS) workstations. Monitor resolution was approximately 2.1 Mpixels, compared with diagnostic radiology monitors with a resolution of 3 Mpixels or greater. Monitors were standard desktop computer monitors that were not DICOM grayscale standard display function calibrated. The PACS software was simulated using the ViewDEX software program (Sahlgrenska University Hospital, Goteborg, Sweden).¹⁰ This software has display control functions such as window, pan, and zoom, and records the location and confidence of lesions marked as potential nodules. Ambient light levels in the educational laboratories were not measured but were lower than those in a typical office or classroom and higher than those in a typical radiology reading room. The key aspects of the SRW user interface are shown in Fig. 1.

Fig. 1.

Diagram highlighting key aspects of the simulated radiology workstation. The white “+” represents the marking of a suspected nodule.

2.4. Study Protocol

Subjects were randomly divided into control and experimental groups. Subjects received training as part of their respective control or experimental groups at their own SRWs. The entire educational protocol lasted 3 h. Educational materials for this study consisted of static educational slides without audio component.

All subjects were first given a computer presentation providing a basic introduction to the study, the SRW software, and chest radiography lasting approximately 15 min. Subjects were told that their task was to identify and mark the center of any suspected pulmonary nodules using the SRW software, and provide an estimate of their diagnostic confidence using the following 5-point scale. Confidence scale (“$p$” represents the estimated probability of a nodule being present): 0: no nodule present, $p=0\%$; 1: probably not a nodule, $p=1\%$ to 25%; 2: possibly a nodule, $p=26\%$ to 50%; 3: probably a nodule, $p=51\%$ to 75%; 4: definitely a nodule, $p=76\%$ to 100%. Note that not marking any suspected nodule was equivalent to assigning a score of 0.

Subjects were not informed about the prevalence of nodules in each set, but were told that none of the CXRs contained more than one nodule. Images were presented without clinical information.

Five practice cases were provided to subjects to help them become familiar with the study software. Subjects were given the first set of 20 CXRs, case set 1 (CS1). Approximately half of the cases presented contained nodules. Subjects were given 45 min to mark pulmonary nodules if present and rate their diagnostic confidence.

2.4.1. Search Pattern Training

After the first set of cases, the experimental group was given 15 min to complete SPT, but the control group received no training. Search pattern training materials instructed the subjects to use specific eye movements when evaluating the CXR image. The suggested eye movement for the search pattern was to sweep the eyes from the medial aspect of each intercostal space toward the lateral chest wall, starting at the first right intercostal space, moving down one intercostal space at a time, and repeating the process on the left. Training materials included a written description of the scan pattern as well as an image showing the proposed scan pattern, similar to the image shown in Fig. 2. This systematic review ensured full coverage of the lungs with the central part of the field-of-view. Note that there are other scan patterns which would likely have also been valid, but only one pattern could be selected for this study. This pattern is essentially a modification of the boustrophedon scan pattern.¹¹ During SPT for the experimental group, the control group received a lecture on unrelated topics in radiology for a comparable amount of time.

Fig. 2.

Proposed lung search pattern. The suggested eye movement for the search pattern was to sweep the eyes from the medial aspect of each intercostal space toward the lateral chest wall, starting at the first right intercostal space, moving down one intercostal space at a time, and repeating the process on the left.

Subjects then were given a second set of CXRs, case set 2 (CS2). The prevalence of nodules and other image characteristics were approximately the same as for CS1. The subjects were given 45 min to mark suspected nodules and give a rating of their diagnostic confidence. After CS2, the control group was given the opportunity to undergo SPT for educational purposes, and all subjects elected to participate. Subjects received additional educational materials and the answer key to the cases. Subject review of additional educational and study materials after CS2 took about 1 h. The study protocol is summarized in Fig. 3.

Fig. 3.

Diagram summarizing the study protocol. S, subject; CS, case set; SPT, search pattern training; Ctrl, control group; Exp, experimental group; and SQ, subject questionnaire.

2.5. SPT Questionnaire

After completing the study, every participant was given an institutional review board–approved questionnaire to assess their opinions about SPT as a useful adjunct to their prior training in CXR interpretation. The poststudy questionnaire included two items:

• Q1: The SPT was helpful for learning the skills needed to identify pulmonary nodules on CXRs.

• Q2: The SPT has helped me feel more confident about my ability to identify pulmonary nodules on CXR.

Subjects were asked to indicate their level of agreement with the questionnaire statements using the following 5-point Likert response format: R1: strongly disagree; R2: disagree; R3: neutral; R4: agree; and R5: strongly agree.

2.6. Statistical Analysis

All software for data analysis was written in MATLAB (MathWorks, Natick, Massachusetts). As the images contained no or at most one nodule, data were analyzed using localization receiver operating characteristic (LROC) analysis.¹² For a lesion to be localized correctly, a subject must place his or her marking within the boundaries of the simulated nodule. Trapezoidal integration was used to estimate the area under the LROC curves (AUC). Differences in area under the LROC curves ($\mathrm{\Delta }\mathrm{AUC}$) between control and experimental groups were statistically analyzed using the bootstrap,¹³ similar to the Dorfman–Berbaum–Metz method.¹⁴ $p$-values less than 0.05 were considered statistically significant.

To determine significance for questionnaire responses, the Wilcoxon signed-rank test was used.¹⁵ Multiple comparisons were corrected for using the Bonferroni method.¹⁶ $p$-values less than 0.025 (0.05 type 1 error/2 questions) were considered statistically significant.

3. Results

3.1. Control Group

There was no statistically significant improvement in subject performance between CS1 and CS2, $\mathrm{\Delta }\mathrm{AUC}=0.0559$, 95% $\text{confidence interval}=[-0.040,0.159]$, $p\text{-}\text{value}=0.1253$. Respective LROC curves are shown in Fig. 4.

Fig. 4.

Average localization receiver operating characteristic (LROC) curves for control and experimental groups.

3.2. Experimental Group

There was a statistically significant improvement in subjects’ ability to identify nodules after laboratory SPT, $\mathrm{\Delta }\mathrm{AUC}=0.1539$, 95% $\text{confidence interval}=[0.051,0.256]$, $p\text{-}\text{value}=0.0012$. Respective LROC curves are shown in Fig. 4.

3.3. Questionnaire

Subjects indicated that they felt SPT was a useful adjunct to training for both questions, mean response: $\mathrm{Q}1=4.0$, $\mathrm{Q}2=3.6$, $p$-values for both questions $<0.001$. Histograms for subject responses are shown in Fig. 5.

Fig. 5.

Histograms of questionnaire responses. R1: strongly disagree; R2: disagree; R3: neutral; R4: agree; R5: strongly agree.

3.4. Training Efficiency

The total time needed to conduct this protocol was 3 h. Two faculty radiologists conducted the training of 48 PA students. The number of students trained per radiologist-hour using group training was 8. The number of students who were trained per radiologist-hour by individual training in our prior study was approximately 1.⁷

4. Discussion

To date, there has been limited translation of our knowledge of image perception into practical educational tools for teaching image interpretation to healthcare trainees to improve their skills at image interpretation. A scan pattern for lung evaluation was developed based on prior studies showing that novices evaluate less of an image with their useful field-of-view when compared with experts.³ Our study showed that teaching healthcare trainees a specific search pattern for evaluation of CXRs resulted in improved ability to identify pulmonary nodules.

Search pattern training has been included in the educational curriculum for the training programs of many health professionals. However, the more common search patterns are typically based on lists of structures to include in evaluation, essentially ordered checklists.^17,18 Search patterns suggesting specific eye movements have not been adequately emphasized in radiology education to date.⁷

The study proctors queried subjects regarding their prior CXR training after completion of the study. All subjects had received prior training in chest radiography and had been given search algorithms that were essentially checklists of structures to evaluate. None of the study subjects reported having been shown a specific scan pattern (with eye movements) for evaluation of the lungs. In fact, none of the trainees admitted having been shown a specific scan pattern for evaluating any type of medical image. The survey results show that trainees felt a training-specific scan pattern for evaluation of a CXR was a useful adjunct to their training.

A prior study⁷ showed that teaching healthcare trainees a lung search pattern on an individual basis improved a subject’s ability to identify pulmonary nodules. However, the student to instructor ratio was one-to-one, and not practical as a generalized educational model. This study shows that teaching search patterns in an educational laboratory is feasible and makes this model more generally applicable to a variety of education environments in which instructor availability may be limited. A more comprehensive comparison of subject performance using these two training models, including comparative AUCs, time spent on each case, and distance from the subjects’ markings to the nodule center, is beyond the scope of this publication, but is a topic worthy of further study.

Perception scientists have learned much about the way in which humans perceive medical images.¹ To date, this knowledge has had little practical application to the development of new educational tools for the instruction of healthcare trainees. It is likely that additional potential educational tools can be developed based on our knowledge of human image perception.

This study had several limitations. It showed that training novices in a specific lung scan pattern to ensure full coverage of an image improves their ability to identify pulmonary nodules. It is important to note that the usefulness of such training for individuals with greater experience in image interpretation, such as radiologists, is unclear. Further study is needed to determine which groups of healthcare practitioners would benefit most from SPT. In this study, subjects were tested immediately after training, but the duration of the training effect remains unknown. Further study is needed to determine if the SPT taught to trainees is retained days, weeks, or even longer after training. This study did not enquire about prior practical experience in CXR interpretation, data that may be useful to collect in future studies. A prior study demonstrated that extraction of actual nodules from a chest CT and insertion into a CXR may produce more realistic pulmonary nodules than simulated spherical nodules.¹⁹ Future studies and educational protocols may benefit from inclusion of nodules simulated using analogous methods. Another limitation is that the lack of eye tracking limits our knowledge about exactly how subject behavior changed after training. As subjects’ performance improved after training, it is not unreasonable to assume that their eye movements changed to be more effective. However, repeating this study with tracking of eye movements would be a useful addition to our knowledge of how SPT improves trainee behavior.

In conclusion, SPT in an educational laboratory has been shown to be an effective way to teach healthcare trainees practical image perception skills. Trainees indicated that they felt the training was beneficial and enhanced their ability to effectively evaluate CXRs. The proposed lung scan/search pattern was based on prior research in medical image perception and suggests that our knowledge of perception may be helpful in the rational development of new educational tools to facilitate education of healthcare trainees in image evaluation and interpretation.

Biographies

William F. Auffermann is a cardiothoracic radiologist and assistant professor of radiology and imaging sciences at the Emory University School of Medicine. He received his MD/PhD from the University of Minnesota, Twin Cities Campus, Minneapolis, Minnesota, USA, in 2004. His research interests include medical image perception, medical education, and the application of perception research to new educational methods.

Brent P. Little is a cardiothoracic radiologist and assistant professor of radiology and imaging sciences at the Emory University School of Medicine. He received his MD from the Yale School of Medicine in 2004. He is an associate director of the radiology residency program at Emory University, and has an interest in improving teaching methods in radiology education.

Srini Tridandapani is a cardiothoracic radiologist and associate professor of radiology and imaging sciences at the Emory University School of Medicine. He received his MD from the University of Michigan, Ann Arbor, USA, in 2001, and his PhD from the University of Washington, Seattle, Washington, USA, in 1994. His research interests focus on ways to use engineering to improve the practice of radiology.