Novel Diagnostic Educational Resource: Use of a web-based adaptive learning module to teach inflammatory reaction patterns in dermatopathology to medical students, residents, and fellows

Abstract

Background

Perceptual and adaptive learning modules (PALM’s) provide a large number of visual examples for evaluation and accommodate to learner performance by actively adjusting the module parameters.

Methods

We developed a module for discriminating 5 inflammatory reaction patterns using the Novel Diagnostic Educational Resource (NDER) platform. The module included a 20 question pre-test, a 200 question training section, and a 20 question post-test. During the pre-test and post-test, images were displayed for an indefinite period of time with no feedback given. In the training section, images were displayed for a duration inverse to learner performance, and after submitting their response learners were immediately shown the correct answer. The performance of module participants was compared to a control group who completed pre-test and post-test only.

Results

26 pathology and dermatology residents completed the module and were included in analysis. Pre-test and post-test scores showed an average increase of 17.1 percentage points (95% CI 13.0 to 21.2, P < 0.001). When performance on pre-test and post-test was compared between the module and control groups, module group performance increased more than control group performance by an average of 10.1 percentage points (95% CI -2.5 to 17.8, P = 0.0119). 84% (37) of participants found the module somewhat useful or very useful and 68% (30) of participants would be pretty likely or very likely to recommend to another trainee.

Conclusions

Our findings validate the use of NDER for teaching inflammatory reaction patterns. Participants generally had favorable feedback regarding the interface and teaching potential of the module. Including a late re-test as part of the module would be beneficial in further validating future iterations. Next steps include optimizing module performance and developing module content for more advanced learners.

Background

Learning dermatopathology consists of training in both “slow” thinking – deliberate, requiring conscious effort and analysis to complete, and fast thinking – intuitive, using pattern recognition and reflex to quickly arrive at a cognitive endpoint.¹ Most current utilized methods for teaching dermatopathology – discussion at the multiheaded microscope, didactic lectures, and textbooks – are methods that target slow thinking. Whereas trainees primarily rely on the feature-based analytical processing reinforced during these teaching methods, expert clinicians are able to use pattern recognition to rapidly identify meaningful findings while ignoring superfluous features.² The only current strategy for teaching fast thinking or rapid pattern recognition consists of mentally assimilating large volume of cases during residency and/or fellowship, which requires years of clinical experience.

Perceptual and adaptive learning modules (PALM) offer profound potential to improve medical education through training fast thinking.² These technologies are perceptual in that they present a high volume of relevant visual examples in series, and adaptive in that they adjust or accommodate to learner performance. PALM take advantage of pre-attentive processing, the phenomenon of forming a quick impression of an image, and attempt to train rapid pattern recognition more efficiently. Through efficient and strategic presentation of a large number of relevant examples in series, PALM have been found to be effective tools in training medical students to more accurately recognize basic histological patterns and tissue types, descriptive terminology for skin lesions, common dermatologic diagnoses, and EKG interpretations.3, 4, 5, 6 Previous studies using electrocardiograms have shown that a combination of pattern-based and feature-based instruction performs better than either method alone.⁷

Novel Diagnostic Educational Resource (NDER) is an open source tool designed specifically for training rapid pattern recognition with annotated WSI.^4,8 NDER modules are adaptive, adjusting to each learner’s unique performance by changing the display time of an image based on cumulative user accuracy. A series of unique images is used to avoid recall bias and encourage the learner to extract knowledge on common features and patterns rather than memorize specific examples. We hypothesized that NDER would be an effective modality for improving the recognition of cutaneous inflammatory reaction patterns with a primary target learner audience of dermatology and pathology residents.⁴

Methods

Module development

Slides representative of 5 different inflammatory reaction patterns (spongiotic, psoriasiform, vasculitis, bullous, and lichenoid) were selected from slide archives at the University of Washington and Oregon Health & Science University. Slides were scanned at 20x and annotated using Aperio ImageScope (Leica Biosystems Imaging, Inc., Vista, CA, USA). The regions of interest for use in the module were selected by an attending dermatopathologist. A total of 100 slides were scanned for use in the module with 2 representative images at 7.5x magnification taken from each slide, for a total of 200 images. Still images at a resolution of 2000x2000 pixels were selected for use in the module.

The technical features of NDER have been previously described.⁸ A brief survey to record participant characteristics and familiarity was administered prior to completing the module (see below). The module itself consisted of a 20 question pre-test, 200 training slides, and a 20 question post-test. The pre-test and post-test question images were displayed in the same order to all participants. During the pre-test and post-test question sessions, users were given an unlimited amount of time to identify the reaction pattern demonstrated in the image. During the training portion of the module, slides were displayed for a variable amount of time ranging from 1.5 to 10 seconds, with the adaptive algorithm modifying the display time depending on user accuracy (Fig. 1). Participants were then asked to select the correct inflammatory reaction pattern and given immediate feedback on their response. At the conclusion of the module, users were given the opportunity to give feedback regarding the module. The full module can be found at https://pcs-webtest0.pathology.washington.edu/academics/pattern/.

Fig. 1.

Representative image of the NDER interface during the training portion of the module. Participants saw images for 1.5–10 s before the image disappeared, at which time participants had to select an answer in order to proceed with the module.

Participants

Email invitations to participate in the study were sent to dermatology and pathology residency programs across the United States. No compensation was provided for module completion. All participants in the study who opened the NDER module were invited to complete a post-module survey assessing opinions about the user’s experience with the NDER module. Survey responses were recorded via the Research Electronic Data Capture (REDCap) web application hosted at the University of Washington.⁹ Participants who did not answer all survey questions had their responses excluded.

A control group was comprised of residency trainees who completed only the pre- test and post-test without completing the NDER training module.

This study was determined to be exempt from institutional review board review by the University of Washington Human Subjects Division.

Statistical analyses

Pre-test and post-test scores for participants who completed the entire module were compared using a paired t-test. Cohen’s d was used to measure effect size. The average improvement in scores from the module group and the control group were compared using a two sample t-test with equal variances. Prior adaptive learning studies have demonstrated a 15–25% increase in accuracy amongst learners, and a 15% improvement was estimated as the expected benefit of the module for our study.^3,4

Results

Ninety-three participants completed the pre-module survey and began the NDER module, while 41 participants completed the entire NDER study including the post-test (44%). Responses from 1 dermatology faculty member and 14 medical students were excluded from analysis, leaving 26 participants in the study (intervention) group. The final study group included 13 pathology residents, 3 dermatology residents, and 10 residents who did not specify whether they were specializing in dermatology or pathology.

The mean (standard deviation) test scores for the intervention group were 75.8 (11.3) on the pre-test and 92.9 (7.5) on the post-test. Scores increased on average by 17.1 (95% CI: 13.0 to 21.2) percentage points. This result was statistically significantly different than no change in scores (zero percentage point increase) (two-sided P < 0.001, Paired t-test, Cohen’s d = 1.68). When compared to the hypothesized improvement between pre-test and post-test scores of 15%, the observed 17.1% increase in post-test scores was not statistically significantly different, demonstrating non-inferiority of observed results as compared to our hypothesis (Fig. 2A).

Fig. 2.

Boxplot demonstrating differences between (A) pre- and post-test scores after completing NDER module for inflammatory reaction patterns and (B) Pre- and post-test performance as compared between control and intervention groups.

For the control group (n=15; 10 pathology residents and 5 dermatology residents), mean (standard deviation) test scores were 73.7 (16.8) on the pre-test and 80.7 (20.9) on the post-test. Scores increased on average by 7.0 (95% CI: -0.7 to 14.7) percentage points.

Scores for the NDER intervention group increased, on average, by 10.1 percentage points more (95% CI: 2.5 percentage points less to 17.8 percentage points more) than the control group (Fig. 2B). This result was statistically significantly different than no difference in the change in scores between groups (two-sided P = 0.011, Two-sample t-test with equal variances, Cohen’s d = 0.83).

The post-module usability survey was completed in whole by 44 participants. Individual module performances were not able to be linked to usability survey responses. All complete usability survey responses were included, regardless of whether participants reported completing the entire module.

Of the respondents who completed the entire usability survey, 27% (12/44) were pathology trainees and 73% (32/44) were dermatology trainees. Just over half (57%, 25/44) reported that they had completed a dermatopathology rotation, while 77% (34/44) reported that they had not used a PALM before and 7% (3/44) reported using a PALM prior to our module.

The majority (84%; 37/44) of respondents responded that NDER was “very useful” or “somewhat useful” for learning inflammatory reaction patterns, and 68% (30/44) responded that they would be “very likely” or “pretty likely” to recommend NDER to another trainee. Two-thirds (66%, 29/44) felt that NDER was “not very difficult” or “not difficult at all” to use.

Conclusions

The NDER adaptive learning module leads to statistically significant increases in diagnostic accuracy of dermatologic inflammatory reaction patterns amongst resident learners of at least 15%, which has been reported as a threshold in prior studies.^3,4 Furthermore, when compared to a control group that did not complete the module, there was a statistically significant increase in performance. Our findings are in line with other studies which have validated the use of PALM for a variety of topics across medical disciplines.3, 4, 5, 6 Our study is the first to validate this technique at the graduate medical education level in dermatopathology, demonstrating that it can be useful as trainees progress through later phases of medical training.

The NDER module was largely well received by participants, with participants reporting the NDER module easy to use. In an open response section of the survey, numerous participants specifically commented on using such modules as a tool to study for certification exams, indicating a need for diversifying the tools available to prepare for the dermatopathology portion of such exams. Learners also expressed interest in using such a module to gain the ability to discriminate between more similar diagnoses and more complex diagnoses, showing that new and improved modules may be well received by resident learners.

Future NDER modules may benefit from several changes to maximize user satisfaction. Rapid pattern recognition training lends itself well to short training sessions, so integration with mobile phones and tablets of all sizes is important to ensure usability across learners, and to allow for convenient training with NDER in any place with an internet connection.⁸ However, numerous participants reported technical problems during the training module: images displaying for less than half a second, only seeing a portion of the image, or not seeing the image at all, which may have contributed to some participants who started the study not completing the training module. As such, designing the interface such that the user can manually display the image for the pre-set amount of time, or ensuring that the timer does not start until the image is fully loaded will be critical features for future modules regardless of the device on which they are displayed.

We found a significant improvement in scores between the pre-test and post-test even among the control group. The NDER module does not allow for randomization of the pre-test and post-test images, possibly contributing to testing effect/test-enhanced learning. Despite this, we saw greater improvements in the intervention group, supporting that the training module itself led to more substantial increases in post-test performance.

Future studies should include a late re-test to help assess the durability of the NDER as a learning technique. The current interface does not allow for linking of a participant’s email address with their NDER module performance, making late re-testing not feasible. Though not assessed in this study, other modules have demonstrated the durability of the learned material over time when utilizing PALMs.¹⁰

The authors anticipate that this specific module would be of most benefit to junior resident trainees in both pathology and dermatology residencies, or those with little dermatopathology exposure in their training programs. The algorithm for evaluating inflammatory dermatoses is highly dependent on accurate recognition of the representative reaction pattern. The skill to quickly recognize these patterns would help learners to narrow the list of potential diagnoses and begin evaluating the more specific differences between different conditions represented by a given reaction pattern. Using such a module following an initial explanation of these different patterns could help reinforce the knowledge by early learners. Expansion of the modules to include more nuanced differential diagnoses within a set of reaction patterns – for an example psoriasis, pityriasis rubra pilaris, and acrodermatitis enteropathica amongst the psoriasiform dermatoses – would be appropriate for more advanced learners.

We demonstrate the ability of the NDER adaptive learning module to improve diagnostic accuracy of dermatologic inflammatory reaction patterns in resident trainees. Our findings support that the NDER platform can be a beneficial learning tool for graduate medical education.

Competing Interests

The authors declare no competing interests.