Impact of Dermoscopy Training on Diagnostic Accuracy, and Its Association With Biopsy and Referral Patterns Among Primary Care Providers: A Retrospective and Prospective Educational Intervention Study

Abstract

Background:

Proper diagnosis in primary care is crucial due to the large number of skin cancer diagnoses each year and its associated growing economic burden. Understanding how primary care providers can be best trained in dermoscopy is instrumental in helping primary care providers differentiate benign and malignant cutaneous lesions so that appropriate action can be taken (eg, biopsy/referral to dermatology or reassurance).

Objective:

To assess the success of concise dermoscopy training among primary care providers.

Design:

A cohort study was conducted in primary care at Mayo Clinic, Rochester, including primary care physicians, nurse practitioners, and physician assistants in internal medicine and family medicine. Diagnostic accuracy was measured on standardized testing through image-based recognition and in clinical practice through observation of referral and biopsy patterns both before and after dermoscopy education, focused on the TADA plus method.

Results:

Forty-three primary care providers completed the image-based recognition tests, with significant improvement in mean score measured after dermoscopy training workshop (20%). Among the thirteen primary care providers who continued to use dermoscopy in clinical practice, a significant improvement (31%) in mean clinical diagnostic accuracy was observed comparing 1 year of practice data before versus after dermoscopy training.

Conclusion:

Improvement in diagnostic accuracy with utilization of the TADA plus method translates into clinical practice. Therefore, training primary care providers in dermoscopy may improve the dermatologic care patients receive in primary care, especially in rural or medically underserved areas where access to dermatology is limited.

Introduction

In the United States, skin cancer is the most common type of cancer, making accurate skin lesion assessment crucial.^1-3 The economic burden of this disease is large and increasing, further emphasizing the importance of early detection in primary care. ¹ Experts recommend that all primary care providers (PCPs) be proficient using dermoscopy, ⁴ which can reduce unnecessary biopsies and referrals of benign lesions, ⁵ but only 8% to 9% of primary care physicians utilize dermoscopy, with lack of training being a frequently cited barrier, while little is know about advanced practice providers specifically.^6,7 Existing literature evaluating diagnostic accuracy for skin cancer amongst both dermatology and primary care providers relies heavily on the number needed to biopsy (NNB), which has numerous limitations. ⁸ While attending a single dermoscopy workshop focused on the Triage Amalgamated Dermoscopic Algorithm (TADA; Figure 1) is known to improve diagnostic accuracy on image recognition tests and decrease the NNB,^9-14 existing evidence does not show if and how this translates to actual primary care clinician behavior/patient care outside of a NNB measure alone. TADA is a 3-step triage process that can be used to aid a clinician in discerning between benign and malignant lesions. The triage tool is outlined in detail in Figure 1.

Figure 1.

Triage Amalgamated Dermoscopic Algorithm (TADA).

Source: Reproduced with permission from Rogers et al. ¹⁵

*Exception: Lesions on palms, soles, or nails.

^aDisorganized or asymmetric distribution of colors and/or structures.

^bPatient should continue self monitoring and any changes or new symptoms (ie, itching, bleeding) brought to the attention of their physician.

Dermoscopy training has the ability to lower biopsies/referrals long-term and improve overall diagnostic accuracy for PCPs and Dermatology providers. For the PCP, dermoscopy utilization can assist in better triaging lesions to determine if they are benign versus malignant so that the most appropriate next step can be taken. In rural or medically underserved areas, increased PCP skin lesion knowledge and skill is especially important with literature showing that increased melanoma mortality is directly correlated with the distance traveled to the diagnosing clinician. Given the current gap in the literature and importance of dermoscopy, we sought to evaluate the potential associations between test-based diagnostic accuracy and clinical diagnostic accuracy, by training a cohort of PCPs to use dermoscopy and comparing changes of their image recognition abilities to referral and biopsy patterns (both before and after an educational intervention).

Methods

All PCPs from the Divisions of Family Medicine and Community Internal Medicine at Mayo Clinic in Rochester, MN were invited via email to participate in an educational study on dermoscopy. At this institution, a PCP is defined as a physician (MD/DO) or advanced practice provider (Nurse Practitioner or NP and Physician Assistant or PA). Enrollees (n = 55) completed a pre-test of 30 dermoscopy images before undergoing a 2-h asynchronous course, in which 2 experienced dermoscopists (HTM & DLS) taught the TADA Plus method where the TADA method was taught along with additional training on common benign lesions, AK/SCC spectrum, and benign nevi versus melanoma lesions. Participants had a period of 2 weeks to complete the online training that was delivered via recorded presentations. An image recognition post-test was administered after enrollees completed the online education (n = 43), and participants were taught how to physically operate a dermatoscope during a 30-min hands-on workshop (n = 23). Of note, the 30-question pre- and post-test was developed by the research team using expert or dermatopathology verified images. Only PCPs who completed both didactic and technical training activities (n = 23) were included for further study (Figure 2).

Figure 2.

Study design.

A 1-year retrospective review of participants’ billing data and visit notes was performed to calculate the pre-training diagnostic accuracy of each individual. Diagnoses unlikely to be aided by dermoscopy were excluded (eg, rashes, warts, cysts). Lesions were excluded where PCPs’ assessments were unclear or diagnoses could not be confirmed (if no biopsy or dermatology referral was completed). Each PCP’s pre-training diagnostic accuracy was calculated as the percent of agreement between their clinical impressions and those of consulting dermatologists (or pathology results, if biopsied).

After TADA plus training, participants’ dermoscopy-related practices were prospectively observed for 1 year to evaluate potential changes to PCP biopsy and referral patterns. Skin lesions examined by PCPs were photographed (including with dermoscopy), reported as benign or malignant, and managed accordingly. Post-training PCP diagnostic accuracy was calculated as the percent agreement between their clinical impressions and biopsy results or a dermatologist’s impression based on image review (DLS or JCS). Only PCPs who recorded at least 5 lesions (n = 13) were included in further analyses.

Changes in participants’ post-training standardized test scores and clinical diagnostic accuracy were compared using a paired t-test. Differences between each PCP’s clinical diagnostic accuracy before and after training were compared using Fisher exact tests, and compared to dermatologist impressions using a paired t-test. Analyses were completed using SAS (SAS Institute Inc., version 9.4). The study was approved by the Mayo Clinic Institutional Review Board on October 10, 2022. See Figure 1 for flow-chart of methodology

Results

Forty-three PCPs completed both the pre- and post-tests, including 17 physicians (46%), 17 NPs (38%), and 9 PAs (16%). Most participants had over 3 years of primary care experience (87%), and 98% had little or no prior dermoscopy training. Total mean scores on written tests improved significantly after the training workshop (20.0%), as did scores for distinguishing benign from malignant lesions (19.4%). Significant improvement was noted for all but 2 diagnoses: actinic keratosis and benign nevi (Table 1).

Table 1.

Pre/Post Results for Those Who Completed Both Tests.

	Pre (N = 43)	Post (N = 43)	P-value
Overall score (out of 30)			<.001 a
Mean (SD)	12.6 (3.54)	17.4 (3.83)
Median	12.0	18.0
Range	7.0, 20.0	9.0, 24.0
Benign score (out of 17)			<.001 a
Mean (SD)	7.6 (2.79)	10.4 (2.86)
Median	7.0	11.0
Range	3.0, 13.0	2.0, 15.0
Malignant score (out of 13)			<.001 a
Mean (SD)	5.0 (1.38)	7.0 (1.77)
Median	5.0	7.0
Range	2.0, 7.0	4.0, 11.0
Percent correct by question type (Mean (SD))
Malignant melanoma	55.3% (15.0)	73.5% (17.3)	<.001 a
Malignant—basal cell carcinoma	16.9% (20.2)	32.6% (24.7)	<.001 a
Malignant—squamous cell carcinoma	38.4% (34.2)	64.0% (35.1)	0.001 a
Pre-malignant—actinic keratosis	39.5% (38.7)	34.9% (35.4)	0.499 a
Benign—dermatofibroma	40.3% (36.8)	70.5% (30.2)	<.001 a
Benign–typical melanocytic nevus	64.5% (33.3)	47.1% (30.0)	0.001 a
Benign—angioma	51.2% (25.6)	72.7% (17.1)	<.001 a
Benign—seborrheic keratosis	29.5% (25.2)	58.5% (19.0)	<.001 a

^aPaired t-test P-value.

Thirteen PCPs (1 physician, 10 NPs, and 2 PAs), submitted 5 or more lesions prospectively and were included in final analyses. PCPs recorded a collective total of 249 skin lesions, which were ultimately diagnosed clinically by a dermatologist (84.7%) or histologically by a dermatopathologist (15.3%). The low count of dermatopathology verified lesions is expected in primary care where the majority of lesions examined are benign and do not require biopsy. The most common diagnoses were benign nevus (45.8%), seborrheic keratosis (24.5%), and melanoma (5.6%). A significant improvement (31%) in mean clinical diagnostic accuracy was observed comparing 1 year of practice data before versus after dermoscopy training (Table 2). Both retrospective and prospective datasets were analyzed to determine the diagnostic concordance between PCPs and dermatologists. When PCPs categorized a lesion as benign or malignant, their impressions were compared to those of dermatologists (or histology reports, if biopsied). Five participants showed statistically significant improvement in classifying lesions post-training; eight showed changes that did not reach statistical significance (five improved, three decreased; Table 2). For all 13 clinicians, their concordance with the dermatology team was tabulated to explore potential trends (Table 3).

Table 2.

Percent Correct for Each Provider in the Retrospective and Prospective Samples.

Provider	Retrospective sample		Prospective sample		P-value
	N samples	Diagnostic accuracy a (%)	N samples	Diagnostic accuracy a (%)
1	17	41.2	14	85.7	.025 b
2	15	41.2	30	76.7	.026 b
3	5	40.0	15	73.3	.290 b
4	8	50.0	14	85.7	.137 b
5	25	68.0	88	65.9	.999 b
6	20	45.0	6	83.3	.170 b
7	16	31.3	6	16.7	.634 b
8	11	45.5	11	81.8	.183 b
9	14	21.4	9	77.8	.013 b
10	25	24.0	12	83.3	.001 b
11	24	45.8	11	36.4	.721 b
12	10	10.0	27	81.5	<.001 b
13	10	30.0	6	66.7	.302 b
All PCPs	13	39.4	13	70.4	.003 c

^aDiagnositic accuracy = agreement by dermatology.

^bFisher’s exact test P-value.

^cPaired t-test P-value.

Table 3.

Further Analysis of Retrospective and Prospective Clinical Performance.

Those in red columns signify disagreement between PCP and dermatologist/pathology.

- Green highlighted columns represent correctly identified lesions.

- Red highlighted columns represent incorrectly identified lesions.

Discussion

To our knowledge, no studies have been conducted to analyze the impact of TADA with PCPs beyond standardized testing and one study evaluating NNB. The purpose of this study was to implement concise dermoscopy training focused on the TADA plus method and determine if subsequent PCP dermoscopy use translated into meaningful diagnostic improvement.

Our study demonstrated that after TADA training, PCPs significantly improved their ability to assess dermoscopy images of benign and malignant lesions, outside of benign melanocytic nevi and actinic keratoses where they fared slightly worse. With the TADA algorithm centered around the following 3 common benign lesions: angioma, dermatofibroma, and seborrheic keratosis, it is unsurprising that PCPs performed well for these 3 lesion types specifically with significant improvements measured on testing. There was a significant increase in diagnostic accuracy for all 3 types of malignant lesions, with the best performance on melanomas. We hypothesize that PCPs did not perform well with actinic keratoses due to the fact that no actinic keratoses case studies were utilized during training. Rather, only the basics of actinic keratosis lesions were taught. PCPs may have low risk tolerance when evaluating melanocytic lesions, erring in favor of biopsies/referrals to avoid missing melanomas if even slight ambiguity was present.

When measuring clinical diagnostic accuracy for the PCP group (N = 13) 1-year post training, a significant improvement was measured overall, demonstrating that the TADA plus method can effectively assist the clinician in evaluating skin lesions in the primary care setting. While only 5 PCPs had statistically significant improvement in diagnostic accuracy in clinical practice, improvement was observed for 10/13 (76.9%) of the clinicians. Further analysis shows that the lack of improvement observed amongst the 3 PCPs with decline in diagnostic accuracy was due largely to an observed behavior of referring/biopsying benign lesions, not missed malignant lesions. Of note, two of the PCPs did not see any malignant lesions and one PCP did see seventeen malignant lesions, correctly diagnosing 70.6% of them, over the course of 1 year following training. A closely mirrored scenario is observed amongst the 5 clinicians who had improvement that was not statistically significant. A similar hypothesis as above is made that the PCP was being overly cautious in the evaluation of the benign skin lesions and referring due to lack of absolute confidence in its benign quality. Increase in biopsy or referral for the first year after dermoscopy training has been documented in the literature and is expected due to the increase of information that dermoscopy provides. Therefore, it is unsurprising that this trend was observed amongst several providers. However, with continued dermoscopy use, biopsy and referral rate has been shown to improve further past the first year of utilization. ¹⁶

Despite this observed cautious referring trend amongst some individuals, TADA seems to solidify the PCP’s overall confidence for benign lesions when comparing clinical diagnostic accuracy before and after dermoscopy training as a group (Table 3). One year before training, PCPs appropriately offered reassurance with 19.3% of lesions to dermatology (true negative) but referred or biopsied 58.9% of benign lesions (false positive). Conversely, 1 year after training, PCPs offered reassurance with 62.3% of benign lesions (true negative), and reduced biopsy/referral of benign lesions to 22.9%. Improvement in benign lesion recognition was the most significant improvement noted in the study, which can potentially reduce unnecessary referrals, dermatology wait time, healthcare cost, and patient anxiety. For malignant lesions, PCPs appropriately referred/biopsied 62% (23/41) of malignant lesions, with 9.2% of assessed lesions being correctly identified as malignant (true positive) and 5.6% of lesions not being appropriately referred/biopsied (false negative), compared to 20.3% and 1.5% retrospectively. One clear limitation with the retrospective data is that all false negatives could not potentially be included in the dataset because no referral or biopsy was done, thus no review of the PCP’s final diagnosis. This could explain the slightly higher false negative rate after dermoscopy training. However, the PCP diagnosis was in concordance with the dermatology for 71.5% of the lesions after dermoscopy training, which was increased from only 39.5% before training. These findings support concise dermoscopy training and utilization centered around the TADA plus method in the primary care setting by both physicians and advanced practice providers. Further research is recommended to observe the diagnostic accuracy beyond 1-year post training to identify if continued improvements are noted as previously documented in the literature.

Strengths of the study include the dermoscopy educational program evaluating both learner acquisition of knowledge and specific clinical outcomes. Limitations include a small number of participants in the clinical component of the study (N = 13) compared to initial enrollment in the pre- and post-test (N = 43). The study included only one physician in the final analysis but all primary care providers in the department were invited including physicians (MD/DO) and advanced practice providers (NP/PA). The study was only completed at one academic institution. The study categorized lesions as either benign or malignant, but it should be recognized that borderline or dysplastic lesions are common. Therefore, supplementing good dermoscopy use with a thorough history, patient symptoms, and naked eye-exam is critical. Retrospectively, the research team was unable to verify skin lesions that were not referred to dermatology or biopsied, and prospectively, skin lesions that were not consented by the patient could not be included in the study per the IRB. In other words, this limitation potentially introduces the opportunity to miss a correctly identified benign lesion retrospectively. Prospectively, we were unable to precisely conclude that there were no overlooked early melanomas, as the patients were not observed beyond 1 year, and therefore, an early melanoma that didn’t have any worrisome features at the time of evaluation could have been missed. A longer study duration of 1+ years would be needed to identify if any patient presented back to the clinic with a melanoma previously thought to be benign. ¹⁷

Conclusion

Improvement in diagnostic accuracy with utilization of the TADA plus method translates into clinical practice. Training PCPs in dermoscopy may improve the dermatologic care patients receive in primary care. This is especially important in rural and medically underserved areas with limited access to dermatologists, because the stage of melanoma at time of diagnosis can be associated with distance traveled to their diagnosing clinician.^18,19 Future research could explore utilization of dermoscopy and teledermoscopy in primary care and its impact on the healthcare cost associated with treating cutaneous malignancies, access to needed dermatologic care, and patient satisfaction.