Understanding Visual Search Patterns of Dermatologists Assessing Pigmented Skin Lesions Before and After Online Training

Abstract

The goal of this investigation was to explore the feasibility of characterizing the visual search characteristics of dermatologists evaluating images corresponding to single pigmented skin lesions (PSLs) (close-ups and dermoscopy) as a venue to improve training programs for dermoscopy. Two Board-certified dermatologists and two dermatology residents participated in a phased study. In phase I, they viewed a series of 20 PSL cases ranging from benign nevi to melanoma. The close-up and dermoscopy images of the PSL were evaluated sequentially and rated individually as benign or malignant, while eye position was recorded. Subsequently, the participating subjects completed an online dermoscopy training module that included a pre- and post-test assessing their dermoscopy skills (phase 2). Three months later, the subjects repeated their assessment on the 20 PSLs presented during phase I of the study. Significant differences in viewing time and eye-position parameters were observed as a function of level of expertise. Dermatologists overall have more efficient search than residents generating fewer fixations with shorter dwells. Fixations and dwells associated with decisions changing from benign to malignant or vice versa from photo to dermatoscopic viewing were longer than any other decision, indicating increased visual processing for those decisions. These differences in visual search may have implications for developing tools to teach dermatologists and residents about how to better utilize dermoscopy in clinical practice.

Introduction

Skin cancer, the most common cancer in the USA, varies clinically. Assessment of pigmented skin lesions represents a challenging aspect in the secondary prevention of melanoma, the most serious and potentially deadly form. Malignant melanoma (MM) is a serious form of skin cancer, ranked as the sixth most common cancer in the USA and the most common fatal malignancy among young adults. [1] The American Cancer Society estimated that there will be 76,250 new cases of MM and more than 9,180 deaths in 2012 [1]. Between 1950 and 2000, the National Cancer Institute (NCI) Surveillance, Epidemiology and End Results (SEER) database has documented a 619 % increase in the annual incidence of MM and a 165 % increase in the annual mortality, clearly underscoring the need for new early detection strategies. [2]

The early detection of MM is essential to patient survival, as diagnosis and excision of thinner MM is associated with better prognosis. This concept is well represented by the current tumor–node–metastasis (TNM) staging system and the corresponding survival rates for the different stages of disease [3, 4]. The 5-year survival associated with MMs diagnosed and excised when <1 mm thick (stage Ia) is 95 %, whereas for ulcerated MM >4 mm (stage IIb) and distant metastatic disease (stage IV), the survival rate is 45 and 12 %, respectively. It was estimated that more than 60,000 new MM cases per year in 2005 resulted in $60B of future projected-year-of-life lost in the USA alone. This is a significant burden to society for a malignancy avoidable by early detection and treatment [5]. Furthermore, there is also a significant morbidity associated with the excision of benign skin lesions when screening for melanoma. Therefore, increased is diagnostic specificity when evaluating PSLs has the potential to result in significant benefit from the morbidity and mortality perspective.

The main method of skin cancer detection continues to be direct examination of the skin by a trained professional [6]. Dermatologists are the most skilled at skin examination [7], but there is a shortage of dermatologists in the USA and other countries [8], a situation that could influence timely diagnosis and management of skin cancer. Patients with potentially urgent problems, such as pigmented changing lesions, have reported wait times as long as patients with routine problems [9]. Results of a survey conducted by the American Academy of Dermatology (AAD) in 2007 revealed that the mean wait time for dermatology appointments was 33 days. The AAD survey also found that one third of dermatology practices continued to seek an additional dermatologist or other health-care provider such as an NP or physician assistant (PA). Also evident was an increasing trend for dermatology time spent in cosmetic and surgical dermatology over medical dermatology [8]. These factors could have implications for timely diagnosis of melanoma, particularly in melanoma high-risk patients with multiple moles, atypical moles, or a personal or family history of melanoma [10].

The visual nature of skin cancer lends itself well to digital lesion imaging and teledermatology [11–16]. Skin cancer detection is enhanced using dermoscopy, which is a widely accepted imaging technique for recognizing, assessing, and managing skin lesions [17, 18]. A dermatoscope provides a ×10 magnification and illuminates a skin lesion without reflected light, allowing morphological classification based on accepted dermoscopic features. Dermoscopy enables more efficient monitoring, particularly of pigmented lesions, over time [19]. Use of dermoscopy by trained PCPs has been shown to improve detection of skin cancers and reduce the excision or referral of benign pigmented lesions by one half [20, 21]. While a significant effort has been vested in developing accurate and reproducible decision-making algorithms for dermoscopy assessment of skin lesions, there is limited understanding as to the visual pattern and cognitive aspects associated with this diagnostic process.

Visual search characteristics often function as predictors of training and diagnostic accuracy [22–35] in other clinical specialties such as radiology and pathology; thus, one would expect to find differences in other specialties where images are utilized in the diagnostic process. The objective of this study was to evaluate the feasibility of extending the use of eye-tracking methodology to the dermatology field, specifically to the evaluation of individual pigmented lesions through stored-and-forward close-up and dermoscopy images. To achieve this objective, we performed an assessment of expert and dermatology residents using a predetermined set of pigmented lesions representing a variety of pigmented skin lesions.

Materials and Methods

Images

A series of 20 cases was compiled. Twenty was chosen as (a) many medical image search studies typically use about this number to avoid fatigue during recording, and (b) it is the visual search data we are primarily concerned with and that yields thousands of data points which results in sufficient statistical power for the analyses. Each case was composed of a close-up photo of a single pigmented skin lesion (PSL) that was either malignant melanoma or a benign lesion that had characteristics of a MM but turned out to be benign and its corresponding dermoscopy image. Each case was rated on 1–10 scale by two expert dermatologists (RHW, CCL), where 10 = highly suspicious and 1 = benign (Fig. 1). For the analyses, 1–5 ratings were considered to be benign (n = 10), and the 6–10 ratings were highly atypical (n = 10). The cases spanned the range of ratings.

Fig. 1.

Close-up photo (left) and dermoscopic (right) images of malignant case rated 10 (top) and benign case rated 2 (bottom)

The images were formatted so the lesion of interest was centered and surrounded by a perimeter of normal skin. The study mages were saved in non-compressed jpeg format, and a PowerPoint presentation was created in which each slide contained a single image (all the same width and height, 672 × 448 pixels) centered in the slide with a black background. For each case, the close-up photograph was displayed first, followed by the dermoscopy image. Before each case, a nine-dot calibration pattern was displayed for use in the eye-tracking system.

Display

All images were displayed on a ViewSonic VA2703 LCD color monitor (1,920 × 1,080; 27″ screen; contrast ratio, 1200:1; max luminance, 300 cd/m²; sRGB gamma, 2.2). Ambient room lights were set to 40 lx. Subjects were seated with their eyes at about the level of the center of the display at about 18″ away from the display surface.

Subjects

The study was IRB approved by the University of Arizona, and all subjects provided written consent. Two board-certified dermatologists with dermoscopy experience and two 1st year dermatology residents were recruited to participate and were financially compensated for their time. The mean age of dermatologists (both male) was 57, and the mean age of the residents (1 male and 1 female) was 28. The dermatologists each had over 20 years of clinical experience, and the residents were second year. All had knowledge of both photographic and dermatoscopic image assessment; however, the residents had limited experience. All were given Nishihara’s test for color blindness prior to the start of the study and all passed. All subjects wore corrective lenses (glasses or contacts).

Protocol

The study consisted of five phases. Phase I was a baseline test in which each subject viewed the set of 20 PSL cases, and their diagnostic performance and visual search parameters were measured. Each session took approximately 30 min to complete. For each case, the subject was presented with the close-up photographic image first. He/she had to first report whether the case was benign or malignant, then reported their confidence in that decision as definite, probable or possible. After reporting a decision, the dermoscopy image was presented, and the subject could keep their previous decision and confidence or change either one. Viewing time was unlimited for both images. Subjects were not provided with feedback on their performance.

Within 1 week of phase I, the participants completed an online dermoscopy program (phase II) in which they first reviewed 35 PSL dermoscopy cases assessing them for the presence of asymmetry, atypical network, and blue and white veil (3-point check list). These were different images than those used in the eye-tracking portions of the study. No feedback on their performance was provided. The scores for each image for each criterion were stored automatically. In phase III, they each completed the online tutorial portion of the program titled “Introduction to Dermoscopy” that reviews the three assessment criteria in detail with example images provided. An interaction session was included in the program where the subjects rated images on the three criteria and receive feedback. These scores were automatically recorded. After approximately 1 week, they repeated the 35 case assessments (phase IV) without feedback, and scores were automatically recorded. Finally, in phase V, after approximately 12 weeks, each subject repeated the test delivered in step I (20 cases plus eye-tracking session), but the images were shown in a different random order. There was no eye tracking during phases II–IV as these were done on their own time.

Eye Tracking

The ASL SU4000 Eye-Tracker system (Applied Science Labs, Bedford, MA, USA) was used to record visual search. Prior to viewing any of the test cases, each subject is calibrated to the eye-tracking system by fixating each of the dots in the nine-point pattern, and the system uses these data in an automatic calibration procedure. During calibration, the subject is asked to fixate each dot without moving their head, but once calibrated, they are allowed to move their head since the system also has a head tracker that monitors head movements and takes such movements into account when generating the eye-position data.

The ASL system computes line of gaze and dwell time based on pupil and corneal reflection parameters. An infrared (IR) light-emitting diode and phototransistor detector are mounted on the headband. IR light is emitted and reflects off a reflective visor into the left eye, reflecting back off the pupil and cornea to the visor, which then reflects it back to a charge-coupled device camera. The eye-position data were analyzed using standard methods. Briefly, the accuracy of the system (spatial error between true eye position and computed measurements) is <1°. The SU4000 samples eye positions every 1/60 of a second to generate raw x-, y-coordinate eye-position data. Fixations are formed by grouping x- and y-coordinates of the raw data using a running mean distance calculation having a 0.58 radius threshold. Dwell time can be calculated for each fixation, summed across fixations, then associated with a given region of interest or location in the stimulus image. For this study, we analyzed the number of fixations generated during the search of each image and the dwell times associated with each fixation.

Results

Training (Online Course Phases II–IV)

Tables 1, 2, and 3 show the percentage of cases rated correctly for presence as asymmetry (1), atypical network (2), and blue and white veil (3) pre-training, during interactive training, and post-training. Overall, there were no significant differences between pre-training, interactive training, and post-training test scores for asymmetry (X² = 1.95) or atypical network (X² = 0.30), but there was for blue and white veil (X² = 4.35, p < 0.05).

Table 1.

Training results for asymmetry

	Training pre-test (%)	Interactive training (%)	Training post-test (%)
Dermatologist 1	71	94	71
Dermatologist 2	86	83	86
Resident 1	83	94	74
Resident 2	66	91	86
Overall	76	91	79

Table 2.

Training results for atypical network

	Training pre-test (%)	Interactive training (%)	Training post-test (%)
Dermatologist 1	91	91	94
Dermatologist 2	77	91	83
Resident 1	80	91	80
Resident 2	71	69	80
Overall	80	86	84

Table 3.

Training results for blue and white veil

	Training pre-test (%)	Interactive training (%)	Training post-test (%)
Dermatologist 1	71	86	77
Dermatologist 2	60	94	66
Resident 1	89	91	86
Resident 2	40	74	66
Overall	65	86	74

Diagnostic Accuracy Phases I and V

Overall, there was no significant difference (X² = 0.31, p = 0.96) in percent correct diagnoses for the photos from baseline (73 %) to post-training (74 %), or for dermoscopy (X² = 1.30, p = 0.73; 73–70 %). Overall, 48 % of decisions/confidence stayed the same from close-up photograph to dermoscopy viewing; in 24 %, confidence increased; in 10 %, confidence decreased; in 18 %, decision changed from benign to malignant or malignant to benign. There were no significant differences between dermatologists and residents (X² = 1.10, p = 0.778). Table 4 shows the decisions for the pre- and post-training sessions (X² = 3.46, p = 0.328).

Table 4.

Decisions pre- and post-training

	Pre-training	Post-training
Benign to malignant/vice versa	17.50	18.75
No change	53.75	42.50
Confidence up	22.50	25.00
Confidence down	6.25	13.75

Visual Search Phases I and V

Total viewing time for close-up photographs did not differ as a function of session (pre- vs post-training) (F = 0.935, p = 0.335), or whether a decision/confidence changed or not (F = 1.452, p = 0.230), but did for expertise level (p = 6.642, p = 0.0188) with dermatologists taking less time than residents (mean = 12.56 s, SD = 5.68 vs 15.61, SD = 7.55). There were no differences as a function of whether decisions were correct or not.

Total viewing time for dermoscopy did not differ as a function of session (F = 0.141, p = 0.708), but did for decision/confidence changed or not (F = 3.338, p = 0.0212) with benign to malignant taking longer (mean = 15.39 s, SD = 6.77) than other decisions (mean = 11.27, SD = 5.78 no change, mean = 11.92, SD = 6.79 confidence up, mean = 11.75, SD = 6.32 confidence down), and expertise level (F = 4.819, p = 0.0298) with dermatologists taking less time than residents (mean = 10.23, SD = 5.93 s vs 14.21, SD = 6.25). There were no differences as a function decisions whether they were correct or not.

For number of fixations generated on photos, there were no significant differences for decision/confidence change or whether decision was correct or not. For mean dwell times of fixations, there was significant for expertise level only (F = 3.953, p = 0.0487) with dermatologists having shorter (mean = 168.80 ms, SD = 66.37 vs mean = 198.09, SD = 75.87) dwells than residents. Figure 2 shows a typical scanning pattern on the photo (left) and dermatoscopic image (right).

Fig. 2.

A typical scanning pattern on the photo (left) and dermatoscopic image (right). The circles represent fixations or where the eye lands with foveal vision. The size of the circle reflects the dwell time with larger circles indicating longer dwells. The lines indicate saccades or jumps between fixations

For number of fixations on dermoscopy as function of change decision/confidence, there were significant differences on expertise level (F = 7.771, p = 0.006) with dermatologists generating fewer fixations (mean = 17.73, SD = 10.68 vs mean = 24.94, SD = 10.67) than residents; and for decision change (F = 4.490, p = 0.0048) with benign to malignant/vice versa being longer (mean = 27.24, SD = 9.85) than no change (mean = 20.01, SD = 10.48), confidence up (mean = 20.21, SD = 11.97) and down (mean = 19.63, SD = 11.05). For decision correct or not, expertise level (F = 6.735, p = 0.0104) was significant with dermatologists having fewer than residents.

For mean dwell times on dermatoscopic images for change in decision, there was significant difference for training (F = 5.215, p = 0.0239) with dermatologists having shorter dwells (mean = 129.38, SD = 67.96 ms vs mean = 178.18, SD = 69.30); and change in decision (F = 3.549, p = 0.0161) with benign to malignant/vice versa being longer (mean = 188.69, SD = 61.91) than no change (mean = 144.78, SD = 70.91), confidence up (mean = 146.32, SD = 79.07) and down (mean = 151.50, SD = 70.69). For decision correct or not, expertise level (F = 6.635 p = 0.0110) was significant with dermatologists having shorter mean dwells than residents.

Discussion

The training data suggest that all of the subjects benefited to some extent from the training as their interactive scores generally increased from the pre-test scores at least for some lesion characteristics, especially for the blue and white veil parameter. Although some effects of training were maintained over time, in general, performance in the post-test without feedback decreased although not always back to pre-test levels. A single training session appears to be useful, although as it has been previously shown, it will likely require significantly more time and cases to truly educate residents and even experienced dermatologists on the correct interpretation of dermoscopy features. [36, 37] It will be interesting in future studies to determine exactly how many cases it might take (with and/or without feedback during training) for someone to retain the lessons learned during training and then generalize those lessons and maintain them during daily interpretation of image-based cases. It may not, however, be simply a matter of more cases since expertise in interpreting medical images is more complex [38, 39].

As it will be expected, a single training session on dermoscopy also does not appear to impact diagnostic accuracy or visual search parameters. A consistent finding was that the dermatologists overall had more efficient search than residents generating fewer fixations with lower dwells. This has been observed in radiology and pathology and is considered one of the hallmarks of expertise [22–30].

It is interesting that the fixations and dwells associated with decisions changing from benign to malignant or vice versa from close-up photographs (CUP) to dermatoscopic viewing were longer than any other decision, indicating increased visual processing for those decisions. Basically, when someone is fairly certain of a diagnosis using the photo and the dermoscopic image readily confirms that diagnosis, it does not take much time to search the dermatoscopic image. When there is a shift in confidence but not the actual benign vs malignant decision, again, the dermoscopic image serves more as a confirmatory source of data and does not require extensive search/scanning. When the dermoscopic image does present new information however that contradicts the first decision made with the photo, it does seem to require more in-depth scanning and processing to decide that there truly is a different diagnosis than initially made. This clearly requires more investigation as we attempted a limited feasibility study with 20 cases and the majority of them resulted in no decision change or a confidence change rather than a decision change. Larger data sets with more observers at different levels of expertise may yield relevant data and improve our understanding of what factors and features impact decisions. We also did not ask the subjects what features in the dermoscopic images made them reconsider their decisions (i.e., asymmetry, atypical network, and blue and white veil). Future analyses would be useful to correlate dwell points in dermoscopy with specific dermoscopy features to understand what the physicians are looking at more often and spending more time analyzing.

These differences in visual search may have implications for developing tools to teach dermatologists and residents about how to better interpret dermoscopy features in clinical practice and how to better design training programs. Future studies should correlate dwell points with specific visual parameters if possible and then tie these to decisions.

The study does have some limitations. As a feasibility study into the specialty of dermatology, only 4 readers were used and only 20 cases for the eye-tracking assessment component of the study. In future studies, we propose to include a larger number of readers spanning a greater range of experience levels (e.g., include senior residents and fellows) as well as larger image sets. There was only a single training module, so in the future, we would like to investigate other available training programs that might be better suited to training that would be more in-depth or comprehensive thus leading to more improvements in performance and/or longer retention of the skills developed during training. Finally, there is the possibility that the subjects remembered the images from phase I to V (recall they saw different images in the training modules). This seems unlikely, however, as there was a 12-week separation between those two phases, and we used a different random order. Since there were overall no significant changes in the search parameters between phase I and V, it seems that there was no significant memory or educational effects (even though they had additional training between sessions via the training module).

Conclusions

Overall, this study revealed some very interesting findings on the impact of dermoscopic training on performance accuracy. It also revealed, perhaps for the first time, how photographic and dermoscopic images are searched during the interpretation process. Understanding these visual search strategies will help us better comprehend the diagnostic interpretation process and could, in the future, help us design better training tools that capitalize on the ways that residents and dermatologists process the information contained in close-up and dermoscopic images.