A Randomized Trial on the Efficacy of Mastery Learning for Primary Care Provider Melanoma Opportunistic Screening Skills and Practice

Abstract

Background

Early detection of melanoma represents an opportunity to reduce the burden of disease among people at increased risk for melanoma.

Objective

To develop and demonstrate the efficacy of online training.

Design

Randomized educational trial.

Participants

Primary care providers (PCPs).

Intervention

Mastery learning course with visual and dermoscopic assessment, diagnosis and management, and deliberate practice with feedback to reach a minimum passing standard.

Main Measures

Pre-test/post-test diagnostic accuracy. Referral of concerning lesions for 3 months before and after the educational intervention.

Key Results

Among the 89 PCPs, 89.8% were internal medicine physicians, and the remainder were physician assistants embedded in internists’ practices. There were no differences between control and intervention groups regarding gender, age, race, or percentage of full-time PCPs. The control group had more PCPs who reported less than 5 years of practice (n = 18) than the intervention group (n = 6) (χ² [6, n = 89] = 14.34, p = 0.03). PCPs in the intervention group answered more melanoma detection questions correctly on the post-test (M = 10.05, SE = 1.24) compared to control group PCPs (M = 7.11, SE = 0.24), and had fewer false-positive and no false-negative melanoma diagnoses (intervention, M = 1.09, SE = 0. 20; control, M = 3.1, SE = 0.23; ANCOVA, F[1,378] =27.86, p < 0.001; η_p² = 0.26). PCPs who underwent training referred fewer benign lesions, including nevi, seborrheic keratoses, and dermatofibromas, than control PCPs (F[1,79] = 72.89, p < 0.001; η_p² = 0.489; F[1,79] = 25.82, p < 0.001; η_p² = 0.246; F[1,79] = 34.25, p < 0.001; η_p² = 0.302; respectively). Those receiving training referred significantly more melanomas than controls (F[1,79] = 24.38, p < 0.001; η_p² = 0.236). Referred melanomas (0.8  ± 0.07 per month for intervention, 0.17 ± 0.06 for control) were mostly located on the head and neck.

Conclusions

Mastery learning improved PCPs’ ability to detect melanoma on a standardized post-test and may improve referral of patients with suspected melanoma. Further studies are needed to confirm this finding.

ClinicalTrials.gov NCT02385253.

Electronic supplementary material

The online version of this article (10.1007/s11606-018-4311-3) contains supplementary material, which is available to authorized users.

INTRODUCTION

The incidence of melanoma in the United States is projected to more than double by 2030.^1,2 Melanoma detection by health care providers represents an opportunity to reduce the burden of disease and is recommended for at-risk patients by the US Preventive Services Task Force.³ Physician-detected melanomas tend to be thinner (earlier stage) and more curable than those found by untrained patients.^4–9 Training primary care providers (PCPs) in the detection of melanoma during the delivery of customary medical care for patients—i.e., opportunistic screening—may improve patient outcomes.

Previous studies in France, Germany, and the US have demonstrated improvement in diagnostic skills with educational training focused on unaided visual inspection.^10–12 Dermoscopy, a noninvasive in vivo technique commonly used by dermatologists, provides greater discriminatory power than unaided visual inspection for the detection of melanoma.¹³ However, this technique is not included in PCP training.¹⁴ Most PCPs consider melanoma screening important, but lack confidence in their skills. Effective, easily disseminated melanoma screening training is needed.^15,16

This report describes a mastery learning (ML) course to educate PCPs in the early detection of melanoma by unaided visual inspection (customary) and dermoscopy (in vivo 10× magnification). Mastery learning¹⁷ is an exacting variety of competency-based education¹⁸ in which learners acquire essential knowledge and skills, measured rigorously against fixed achievement standards, but without regard to the time needed to reach the outcome. In mastery learning, educational results are uniform, with little or no variation, while educational time may vary among trainees. The randomized trial assessed the efficacy of a self-paced mastery learning program to improve PCP skills in visual inspection and dermoscopy, measured using pre- and post-tests. Secondary outcomes involved examining the translational science (i.e., downstream) consequences of training on PCPs’ patient care practices by assessing referral of concerning lesions.^19–21

METHODS

Course

PCPs underwent training on the identification of at-risk patients and lesions suspicious for melanoma,^22,23 consisting of three units: (a) visual and dermoscopic assessment, (b) diagnosis and management, and (c) deliberate practice. The course was developed over 11 months by a team of dermatologists, PCPs, and medical educators (online supplementary Table 1). The following features of mastery learning were incorporated: (a) baseline testing; (b) clear learning objectives, with units sequenced by increasing difficulty; (c) engagement in educational activities with deliberate practice focused on reaching a minimum passing standard (MPS); (d) a set MPS for each unit; (e) formative testing to gauge unit completion against a preset MPS; (f) advancement to the next unit after mastery achievement; and (g) continued practice until the MPS is reached.¹⁷ The 450 clinical and dermoscopic images were verified with pathological diagnosis. Two dermatologists concurred on a single clinical diagnosis for each image. Since seborrheic keratoses and common nevi are frequently confused with melanoma,²⁴ a three-point dermoscopic algorithm was developed by two dermatologists (JKR and AAM) to aid in clinical assessment and triage (Fig. 1). The algorithm was field-tested concurrently with 25 PCPs (JKR) and 41 PCPs (AAM).²⁵ The algorithm was available as a pop-up throughout the online program.

Figure 1.

Triage algorithm for dermoscopic assessment of skin lesions that appeared to be pigmented on visual inspection.

Each unit was completed within 3 weeks, which was the interval used by Kerfoot and colleagues to improve prostate cancer screening²⁶ (supplementary Table 1). If a PCP failed to complete a unit, email reminders were sent every 2 days for the next 2 weeks. The triage algorithm was provided in all units (Fig. 1). All three educational program phases required an MPS for each feature assessed during visual inspection (border, color, and diameter) and dermoscopic assessment (asymmetry, network pattern, and blue-black-gray-white color), diagnosis and management (reassure, refer), and deliberate practice with feedback to achieve mastery. Given the importance of correctly diagnosing a melanoma, a patient safety approach was adopted with the pass standard at 85% correct responses for each of the six features of visual inspection and dermoscopic assessment.²⁷

PCP Sample and Procedure

Two senior physicians (JKR and GJM) presented the study details at Northwestern Department of Medicine meetings (January 2016–May 2016). Meeting attendee names were obtained from the sign-up sheets, which also provided gender and race/ethnicity. Criteria for inclusion were as follows: learner was in practice for at least 1 year, practice restricted to Northwestern Medicine, practice at least 20 h/week, and a patient panel with over 80% non-Hispanic whites, who are at greatest risk for melanoma. Exclusion criteria were participation in the development of the program or unwillingness to have electronic medical records (EMR) reviewed by a physician (JKR). Recruitment extended from January to August 2016 (Fig. 2). The 3-month period of follow-up for referrals ended in February 2017. Compensation included $125 for completion of the pre-test and baseline survey and $125 for the post-test, and those randomized to receive education received an additional $250 upon completion of the final unit. The institutional review board of Northwestern University approved the study with online consent.

Figure 2.

CONSORT diagram.

A random number sequence was generated by the statistician to assign PCPs to the intervention and control groups (1:1 ratio). Recruiting personnel and participants were blinded to the randomization until after the pre-test and baseline survey were completed. Other members of the research team were blinded to the group assignment of PCPs. Assignment to the control or intervention group was made after a PCP provided consent, completed a 12-lesion pre-test by deciding whether a lesion was benign or malignant for six clinical and six dermoscopic images, and completed a baseline survey that assessed demographic characteristics, personal relevance of melanoma, and attitudes about patient care. PCPs assigned to the control group were contacted in 3 months to complete the post-test. PCPs assigned to receive the intervention were given a personal identification number and a link to the program, and their progress in completing units was monitored. Melanoma educational history and diagnostic outcomes were assessed with a pre-test and post-test.

To determine the effects of the course on patient care (translational science),¹⁹ the electronic medical records (EMR) of the patients of each PCP were examined for referrals in the 3 months before and after study participation. One physician (JKR) recorded referrals from PCPs to dermatology, surgical oncology, head and neck surgery, or plastic surgery for concerning lesions and the anatomical location of the referred lesion. The diagnoses by the consultants, which were confirmed by biopsy if needed, were recorded as atypical nevi, benign nevi, seborrheic keratosis, actinic keratosis, dermatofibroma, non-melanoma skin cancer, and melanoma. The ML course did not include training for diagnosis of actinic keratosis or non-melanoma skin cancer. Actinic keratosis and non-melanoma skin cancer were expected to be prevalent in a patient panel of non-Hispanic whites with a history of sun exposure, and were included to assess increased awareness of skin lesions. All pathological specimens were interpreted by a dermatopathologist with expertise in melanoma diagnosis (PG).

Measures

Demographics

All PCPs reported gender, age, race, and ethnicity, as well as years in practice, full- or part-time practice, and personal or family history of melanoma.

Selection of pre/post-test cases

Melanoma diagnosis education history and diagnostic outcomes were assessed with a pre-test and post-test. Each test consisted of six clinical and six dermoscopic images of lesions with equivalent difficulty. The test images were selected from 80 pathology-confirmed lesions that were presented to 45 internal medicine residents, who determined whether each lesion was benign or malignant. Ten images that no one assessed correctly were removed from the panel. The 70 remaining clinical and dermoscopic images were assessed by 15 board-certified dermatologists to rank the degree of difficulty. Twenty-four images with a range of difficulty (20% difficult, 60% intermediate, and 20% not difficult) were selected to form a panel of 70% benign lesions and 30% melanomas (n = 24). The test and re-test about 5 days later were done by 15 dermatologists and dermatology residents (pre- test reliability [KR-21] = 0.81; post-test [KR-21] = 0.79)

Pre- and post-tests

Twelve different pre-test images were paired with 12 distinct post-test images of equivalent difficulty. Correct responses on the pre- and post-tests were summed to create diagnostic scores.

Patient care attitudes

Attitudes toward patient care were assessed with five items on a 10-point Likert scale: (a) How important is it to you to know the difference between a melanoma and benign moles? (b) How confused are you about deciding if a skin lesion is benign or if it could be malignant? (c) How confident are you that you know the difference between a melanoma and other types of moles? (d) What priority do you give to evaluate the patients’ moles during a focused physical examination? and (e) How likely are you to check skin lesions exposed during a focused physical examination? Each was assessed as a separate patient care attitude.

Personal relevance of melanoma was assessed using a 10-point Likert scale indicating degree of agreement with the statement, “I am at risk of developing a melanoma.” A second item asked PCPs to indicate their level of concern about developing a new melanoma at some point in their lives. These two items were summed to create a composite score of personal relevance (α = 0.71).

Willingness to learn

One item was used to assess PCPs’ level of willingness to learn how to check skin lesions for possible melanomas, with response options on a 10-point Likert scale.

Performance compared to other PCPs

PCPs were asked to report how they felt their performance in checking skin lesions compared to that of other PCPs on a 10-point Likert scale ranging from “not as good as others” to “better than others.”

Clinical proficiency in referral of patients for concerning lesions

Percentages for each type of lesion (atypical nevi, benign nevi, seborrheic keratosis, actinic keratosis, dermatofibroma, non-melanoma skin cancer, and melanoma) referred by PCPs to dermatology, head and neck surgery, plastic surgery, and surgical oncology were constructed for each PCP. These percentages were created for referrals made 3 months before and 3 months after the educational intervention for all PCPs.

Statistical Analysis

The sample was determined using a difference-in-difference (DID) approach to test the hypothesis that the course would improve knowledge, perceived importance of skin exam, competence, confidence, and diagnostic performance. The DID estimator compared outcomes between pre-tests and post-tests between PCPs who received the educational intervention and those who did not. A sample of 89 PCPs completing the trial with the post-test was expected to provide power of > 0.9.

PCPs who participated and those who declined to participate were compared on gender and race/ethnicity. PCPs in the intervention and control groups were compared on baseline demographics, practice information, and personal and family history of melanoma using chi-square analyses. Intervention and control group PCPs were compared with regard to history of prior education in melanoma diagnosis and baseline psychosocial variables related to personal melanoma relevance, patient care, willingness to learn to check for melanomas, and personal assessment of skin check performance compared to other PCPs, using two-sided t tests.

Analysis of covariance (ANCOVA) was conducted using SPSS version 18 (SPSS Inc., Chicago, IL) software to determine the efficacy of the intervention. The ANCOVA tested for differences between the groups in post-test diagnosis scores, controlling for pre-test scores and baseline variables that differed between the control and intervention groups.

Mixed ANOVAs were used to test changes for the seven types of concerning lesions to assess the efficacy of the intervention with regard to clinical proficiency in patient referral. The main effects for time (baseline vs. follow up) and group (control vs. intervention) and the interaction effects of time × group were assessed. Baseline variables that were significantly different between the intervention and control groups were included as covariates in the analyses.

RESULTS

Population

Ninety PCPs were enrolled (50% accrual). Of the 89 PCPs completing the study, 89.8% were internal medicine physicians, and the others were physician assistants embedded in the practices of internists. All participants reported attending at least one lecture about visual diagnosis of melanoma as a student or resident; however, none had been trained in dermoscopy. There were no differences in gender or race/ethnicity between those who participated and those who declined to participate (Fig. 2). There were no baseline differences between the control and intervention groups in gender, age, race, or percentage of full-time PCPs (Table 1). There were significantly more Hispanics in the control group (χ² [1, n = 89] = 4.10, p = 0.04). The control group had more PCPs who reported less than 5 years of practice (n = 18) than did the intervention group (n = 6) (χ² [6, n = 89] = 14.34, p = 0.03). The intervention group had more PCPs who reported 11–15 years, 26–30 years, and 31 or more years of practice. More PCPs in the control group reported a family history of melanoma (n = 7) compared to the intervention group (n = 1). There were no differences between the groups in baseline scores of patient care attitudes, personal melanoma relevance, willingness to learn, or comparison of self-performance to other PCPs (all ps > 0.05). The intervention and control groups did not differ on pre-test scores (t = −0.14, p = 0.910).

Table 1.

Characteristics of the population (N = 89)

Variable	Control (n = 45)	Intervention (n = 44)	Statistical comparison (n = 85)
Demographics
Gender			χ2 (1) = 2.19, p = 0.14
Male	19	12
Female	26	32
Age (years)			χ2 (4) = 5.65, p = 0.23
<30	3	1
31–40	20	17
41–50	16	14
51–60	6	8
62+	0	4
Race			χ2 (4) = 2.48, p = 0.65
Non-Hispanic white	29	29
Black/African American	1	2
Asian	14	12
Multiracial	0	1
Other	1	0
Ethnicity			χ2 (1) = 4.10, p = 0.04
Non-Hispanic	41	44
Hispanic	4	0
Certification
Internal medicine	40	40
Physician assistant	5	4
Years in practice			χ2 (6) = 14.34, p = 0.03
<5	18	6
5 to 10	8	11
11–15	3	7
16–20	11	8
21–25	4	4
26–30	1	4
31+	0	4
Full-time or part-time practice			X2 (1) = 0.90, p = 0.76
Full-time	13	14
Part-time	32	30
Personal history of melanoma			N/A
No	45	44
Family history of melanoma			χ2 (1) = 4.80, p = 0.03
No	38	43
Yes	7	1
Psychosocial variables
Patient care
Importance of knowing difference between melanoma and benign nevi	8.02 (1.32)	7.70 (2.01)	t = 0.88, p = 0.38
Confusion about diagnosing	6.38 (1.71)	6.05 (2.18)	t = 0.80, p = 0.43
Confident in diagnosis	3.38 (2.09)	3.70 (2.04)	t = −0.75, p = 0.46
Priority given to evaluate moles	4.29 (2.57)	5.20 (2.39)	t = −1.74, p = 0.09
Likely to check moles	1.84 (2.58)	2.57 (2.12)	t = −1.45, p = 0.152
Personal relevance	0.18 (5.06)	0.41 (4.86)	t = −0.22, p = 0.83
Willing to learn	3.24 (2.01)	3.89 (1.67)	t = −1.63, p = 0.106
Assessment of performance in comparison with other primary care physicians	0.16 (1.830	0.70 (1.86)	t = −1.40, p = 0.165

Efficacy Analyses: Translation of Knowledge to Improve Skills

The first phase of translational science (T1),¹⁹ knowledge and skill acquisition, demonstrated greater post-test accuracy with visual inspection (85/135 correct) than with dermoscopy (52/135 correct; online supplementary Table 3). Four PCPs who received training failed to achieve 85% proficiency on the second set of deliberate practice examples. Two were unable to correctly identify color on visual inspection, and the other two had difficulty with blue-black-gray-white identification on dermoscopy.

There was a significant difference between PCPs in the control and intervention groups in post-test diagnosis scores (ANCOVA, F[1,378] =27.86, p < 0.001; η_p² = 0.26; Fig. 3). PCPs in the intervention group answered more questions correctly on the post-test (M = 10.05, SE = 1.24) compared to the PCPs in the control group (M = 7.11, SE = 0.24), controlling for pre-test diagnosis scores, ethnicity, years of practice, and family history of melanoma (p < 0.001). Importantly, PCPs in the intervention group had no false-negative identification of melanomas in the post-test and had fewer false-positives (M = 1.09, SE = 0. 20) than the control group (M = 3.1, SE = 0.23).

Figure 3.

Percentage of correct responses on diagnostic assessment of control and intervention groups at pre- and post-tests.

Patient Care Practices: Referral for Concerning Lesions Before and After Training

The second phase of translational science (T2),¹⁹ which seeks to produce clinical efficacy at the level of the patient, showed significant effects, with trained PCPs referring fewer benign lesions than control PCPs. The mixed ANOVA results, controlling for baseline differences in ethnicity, years of practice, and family history, showed a significant interaction of time and group for the following benign conditions: benign nevi, seborrheic keratosis (Fig. 4), and dermatofibroma (F[1,79] = 72.89, p < 0.001; η_p² = 0.489; F[1,79] = 25.82, p < 0.001; η_p² = 0.246; F[1,79] = 34.25, p < 0.001; η_p² = 0.302; respectively). The findings also revealed a significantly greater number of melanoma referrals by those who received training (F[1,79] = 24.38, p < 0.001; η_p² = 0.236; Fig. 5). The melanomas were stage 0 (melanoma in situ) or stage 1A (<1.00 mm). The anatomical locations of the referred lesions were the head and neck (55%), upper extremities (25%), back (15%), and chest (5%).

Figure 4.

Number of referrals for seborrheic keratosis per month in the 3 months prior to training and 3 months after training for the control and intervention groups.

Figure 5.

Number of referrals for melanoma per month in the 3 months prior to training and 3 months after training for the control and intervention groups.

No differences between groups were observed for referral of atypical nevi (F[1,79] = 0.00, p = 0.94), actinic keratosis (F[1,79] = 0.01, p = 0.91), or non-melanoma skin cancer (F[1,79] = 0.42, p = 0.52). Actinic keratosis and non-melanoma skin cancer, which are two common conditions in sun-exposed patients, were excluded from the course and were counted to assess general increased awareness of skin lesions due to the course. There was no effect of the intervention on referrals of these two conditions.

DISCUSSION

The melanoma mastery learning online educational course improved PCPs’ ability to diagnose pigmented lesions, and patient care improved as a consequence of the ML educational intervention. Since trained PCPs overwhelmingly referred patients with lesions on readily visible body locations with high sun exposure, such as the head and neck and the arms and hands, it appears that opportunistic screening rather than total-body skin examination was performed. Improving skills and limiting skin examination to areas visible during the focused clinical examination among at-risk patients may improve skin examination performance from the 8.9–59.6% performing full-body skin examination reported by family practitioners and 5.5–56.4% by internists.^28–30

Patient care was improved by the training, which increased referral of melanomas and reduced referral of benign lesions. A reduction in referrals for benign lesions by ML course-trained PCP can potentially reduce health care costs, decrease patient anxiety, and reduce the burden of physician visits. Additional possible patient benefits include decreased anxiety that may arise while waiting for a dermatology appointment for a definitive diagnosis and reduced burden on frail elderly patients and their caregivers in having an additional physician visit. For example, seborrheic keratoses, which are common lesions in the elderly and which can sometimes have an atypical clinical appearance, constitute 22% of seemingly pigmented lesions referred to specialists by PCPs in the United Kingdom and Australia.³¹ In Australia, the use of dermoscopy reduced the number of excisions of benign lesions by PCPS by 63%.³² Thus, it seems likely that performing opportunistic screening in at-risk patients will improve the detection of early melanoma without incurring the risks of unnecessary procedures or additional financial and time burdens for the patient.^14,33

The study has several limitations. Since it is not possible to ascertain the number of melanomas and atypical nevi that were not considered clinically concerning by PCPs, there is a danger of underestimating the false-negative diagnosis, which may have biased the findings. To address the concern regarding failure to refer patients with concerning lesions, we intend to evaluate all patients cared for by PCPs during the study to assess the development of melanoma in the 12 months following the post-test. It is possible that those who participated in the study perceived this education as more important than those who declined; thus the results may be biased by participant attitudes. Retention of skills was not ascertained. The time needed for PCPs to perform opportunistic screening was not evaluated.

While this study suggests that online learning is feasible and dissemination may be possible, adoption will depend upon health care systems providing the training for PCPs, and PCPs having reason to participate in the training. Mastery learning improved PCPs’ ability to detect melanoma on a standardized post-test and may improve referral of patients with suspected melanoma. Further studies are needed to confirm this finding and to determine whether the training can be easily disseminated and whether the PCPs will retain these skills.³⁴

Compliance with Ethical Standards

Conflict of Interest

The authors declare that they have no conflict of interest.