Abstract
Introduction:
Early detection of melanoma requires timely access to medical care. In this study, we examined the feasibility of using artificial intelligence (AI) to flag possible melanomas in self-referred patients concerned that a skin lesion might be cancerous.
Methods:
Patients were recruited for the study through advertisements in 2 hospitals in Halifax, Nova Scotia, Canada. Lesions of concern were initially examined by a trained medical student and if the study criteria were met, the lesions were then scanned using the FotoFinder System®. The images were analyzed using their proprietary computer software. Macroscopic and dermoscopic images were evaluated by 3 experienced dermatologists and a senior dermatology resident, all blinded to the AI results. Suspicious lesions identified by the AI or any of the 3 dermatologists were then excised.
Results:
Seventeen confirmed malignancies were found, including 10 melanomas. Six melanomas were not flagged by the AI. These lesions showed ambiguous atypical melanocytic proliferations, and all were diagnostically challenging to the dermatologists and to the dermatopathologists. Eight malignancies were seen in patients with a family history of melanoma. The AI’s ability to diagnose malignancy is not inferior to the dermatologists examining dermoscopic images.
Conclusion:
AI, used in this study, may serve as a practical skin cancer screening aid. While it does have technical and diagnostic limitations, its inclusion in a melanoma screening program, directed at those with a concern about a particular lesion would be valuable in providing timely access to the diagnosis of skin cancer.
Keywords: artificial intelligence, convoluted neural networks, deep learning, dermoscopy, melanoma, melanoma detection, melanoma screening, machine learning
Introduction
Melanoma is the leading cause of skin cancer mortality but has an excellent prognosis when diagnosed early. While thick primary cutaneous melanoma is often clinically obvious, early-stage melanoma may be difficult to diagnose. The use of computer-based learning (deep learning) has introduced a melanoma detection strategy that is equal to experienced dermatologists assessing the same dermoscopic image. 1 The artificial intelligence (AI) system used in this study, FotoFinder Moleanalyzer Pro® Version 6.0 (FotoFinder Systems GmbH, Bad Birnbach, Germany), hereafter referred to as FotoFinder®, has been shown to identify melanoma, basal cell carcinoma, and most squamous cell carcinomas including actinic keratoses. Our group and others have shown that this AI system has high sensitivity and specificity in diagnosing melanoma in selected, suspicious pigmented lesions.2-5
A significant barrier in the detection of early-stage melanoma relates to the lack of access to both primary healthcare providers and specialists. Nova Scotia has one of the highest rates of melanoma in Canada. 6 In the current healthcare model, a patient with a lesion of concern must first see a primary care provider. As of February 1st, 2023, 133,595 or 13.5% of the population of Nova Scotia were without access to primary healthcare. 7 Many primary care physicians have limited training and competence in diagnosing skin lesions. The most common referral to our pigmented lesion clinic is for seborrheic keratoses. The referral and subsequent wait time to see a specialist can delay the diagnosis and management of melanoma.
We have found that 80% of the melanomas seen in our clinic had been identified by the patient or a family member. Other studies have also reported that a high proportion of melanomas were initially detected by the patient.8,9
The aim of this study was to assess the potential use of AI as a diagnostic tool for individuals concerned that a pigmented lesion may be cancerous.
Methods
A single-centre study was undertaken in Nova Scotia with a population mainly originating from Scotland and Ireland predominantly with Fitzpatrick Skin Types I, II, and III.
Patients were recruited through advertisements posted in 2 hospitals and 4 family practice clinics in Halifax. Individuals concerned about a pigmented lesion were invited to contact our research team. The inclusion criterion was a lesion the participant had identified as being of concern for cancer. Lesions were excluded if the FotoFinder® camera could not take an image, if the lesion was ulcerated, occurred on the lips or genitals, the hairy scalp or was close to the eye (Table 1). The participants were seen by 1 of 4 medical students who had spent time (30-40 sessions) in the pigmented lesion clinic in the Division of Clinical Dermatology at Dalhousie University. They were trained to recognize common benign skin lesions, such as melanocytic nevus, seborrheic keratosis, actinic keratosis, angioma, dermatofibroma, porokeratosis, and viral wart. They were also trained on when to suspect melanoma. Benign lesions were excluded; the students provided participants with information about the nature of the lesion including an illustrated pamphlet. Where applicable, a letter was also sent to the family physician. When the student diagnosed a non-melanoma skin cancer, the patient was asked to contact their family doctor or to get a referral to a specialist.
Table 1.
Inclusion and Exclusion Criteria of Excluded Lesions.
| Technical exclusions (AI limitations) | Clinical assessment exclusions* |
|---|---|
| Lesion inaccessible to the AI camera | Obvious seborrheic keratosis |
| Lesions in hairy areas | Cherry angioma |
| Lesions on the lips or genitalia | Actinic keratosis |
| Lesions on the face | Ecchymosis |
| Lesions close to tattoos | Obvious dermatofibroma |
| Lesions close to other skin disorders | Porokeratosis |
| Lesions in those with a Fitzpatrick skin type IV, V, and VI | Other common skin lesions, including non-melanoma skin cancer |
| Previously excised lesions |
Abbreviations: AI, artificial intelligence.
Any lesion in which there was clinical uncertainty was included in the study.
If patients met the study criteria, a brief history was taken about the lesion of concern and any personal and family history of melanoma and other cancers. Pigmented lesions were imaged using a FotoFinder® Medicam 800HD and the dermoscopic images were analyzed using the same version of the MoleAnalyzer Pro. A probability of 0.8 was used to indicate possible malignancy.
Abbreviated histories and the macroscopic and dermoscopic images were independently examined by 3 experienced dermatologists (all with more than 10 years experience in dermoscopy), a senior dermatology resident, and later an experienced community-based dermatologist. Another dermatologist, not working at our centre, and blinded to the pathology findings, also reviewed all the excised lesions. The dermatologists were asked to provide a diagnosis and to indicate if the lesions should be excised or monitored.
If the AI, a dermatologist, or the resident flagged a lesion for an excision, the lesion was excised. If a repeat evaluation was suggested, this was done after an interval of 3 months when the lesion was again scanned by the FotoFinder® and the images were compared and scored by the dermatologists and the senior dermatology resident.
Pathological Diagnosis
Two dermatopathologists reviewed each biopsy, blinded to the official report. If they disagreed on the diagnosis, a third dermatopathologist also examined the case.
Statistical Evaluation
For statistical evaluation of sensitivity and specificity, a true positive was recorded where a malignancy (melanoma, basal cell carcinoma, or squamous cell carcinoma) was confirmed on pathology. A false positive was documented where a diagnosis of malignancy was suggested, but on pathological examination the lesion was benign. A false negative included all cases where a benign diagnosis was clinically suspected, but the pathology was a malignancy. Positive and negative likelihood ratios, positive and negative predictive values, and accuracy were calculated. Accuracy is defined as sensitivity × prevalence + specificity × (1 − prevalence). The number needed to biopsy (NNB) was calculated by dividing the total number of biopsies by the number of skin cancers.
Receiver operator characteristic (ROC) curves were constructed and the area under the curve (AUC) calculated. The statistics were calculated using IBM® SPSS® version 28.0.11(15). Inter-rater variability between the dermatologists was assessed using Fleiss’ kappa.
This study was reviewed and approved by the Nova Scotia Health Authority Research Ethics Board (REB file #1027718).
Results
Patients were enrolled in this study between June 1st and August 12th 2022. Four hundred and eighty-two individuals responded to the study advertisements. One hundred sixty-seven lesions were excluded from the study (Table 2). One patient was excluded because the lesion was on the hairy scalp. This lesion was diagnosed clinically as a melanoma by one of the medical students, confirmed on excision, having a Breslow depth of 0.9 mm. The remaining 318 participants had 381 lesions of concern that met the inclusion and exclusion criteria.
Table 2.
Categorization of the 167 Lesions Excluded.
| Technical exclusions | Clinical exclusions | ||
|---|---|---|---|
| Reason for the exclusion | Number (%) | Clinical diagnosis | Number (%) |
| Facial | 10 (6%) | Seborrheic keratosis | 82 (49.1%) |
| Scalp | 6 (3.6%) | Dermatofibroma | 20 (12%) |
| Dermatosis | 4 (2.4%) | Angioma | 8 (4.8%) |
| Too large | 2 (1.2%) | Actinic keratosis | 8 (4.8%) |
| Genital | 2 (1.2%) | Compound nevus (Unna type) | 6 (3.6%) |
| Lip | 1 (0.6%) | Basal cell carcinoma | 5 (3%) |
| Previous excision | 1 (0.6%) | Squamous cell carcinoma | 4 (2.4%) |
| Solar lentigo | 3 (1.8%) | ||
| Acrochordon | 1 (0.6%) | ||
| Comedone | 1 (0.6%) | ||
| Ecchymosis | 1 (0.6%) | ||
| Wart | 1 (0.6%) | ||
| Porokeratosis | 1 (0.6%) | ||
The mean age of the patients was 41.1 years, with more females (72.9%) self-referring than males (Table 3). Of the 318 patients, 15 (4.7%) had a prior history of melanoma, while 102 (32%) had a first-degree relative with a history of melanoma. It is to be expected that this group may be more aware of possible skin cancer; 8 of the 17 malignancies were found in those with a family history of melanoma (Table 4).
Table 3.
Demographics of the 318 Eligible Participants.
| Dermographic Criteria | Total participants | Female | Male |
|---|---|---|---|
| Paticipants | 318 | 232 (73.0%) | 86 (27.0%) |
| Mean age | 42.5 | 41.1 | 46.4 |
| Median age | 40 | 38 | 46 |
| Age range | 19 to 80 | 19 to 80 | 19 to 78 |
| Personal history of melanoma | 14 (4.4%) | 12 (5.2%) | 2 (2.3%) |
| Family history of melanoma | 92 (28.9%) | 70 (30.2%) | 22 (25.6%) |
Table 4.
Participants With a Personal or Family history of Melanoma.
| Description | Past history | Family history |
|---|---|---|
| Total lesions | 16 | 102 |
| Met criteria | 16 (100%) | 95 (93%) |
| Lesion excised | 6 | 19 |
| Malignancy | 2 (33%) | 8 (42%) |
| Melanoma | 0 | 5 |
| BCC | 1 | 2 |
| PSCC | 1* | 1* |
Abbreviations: BCC, basal cell carcinoma; PSCC, pigmented squamous cell carcinoma in situ.
Both a personal and family history of melanoma.
Evaluation of the 381 Lesions of Concern
The anatomical site of the lesions of concern is shown in Figure 1. Eighty-three lesions were flagged for excision, of which 17 were malignant, including 10 melanomas, 6 pigmented basal cell carcinomas and a pigmented squamous cell carcinoma in situ. The AI flagged 27 lesions for excision and identified 11 cancers and the dermatologists identified 8 to 14 of them (Table 5). Where monitoring was suggested, a recommendation for excision occurred in 2 cases by a dermatologist, and both showed malignancy.
Figure 1.
Distribution of the 381 lesions of concern and the skin cancers excised. BCC, basal cell carcinoma; SCC, squamous cell carcinoma in situ.
Table 5.
The Pathological Diagnosis and the Dermatologist’s Suggested Clinical Diagnosis.
| Pathological diagnosis | AI, 11/17* | Derm 1, 9/17* | Derm 2, 8/17* | Derm 3, 13/17* | Derm 4, 14/17* | Res, 8/17* | Ext Derm, 12/17* |
|---|---|---|---|---|---|---|---|
| Melanoma pT1b Breslow depth 1.0 mm | F | MM | MM | MM | BCC or MM | MM | MM |
| Melanoma Breslow 0.5 mm ** | NF | MN | SK | MM | AMN | SK | MN |
| Unusual Spitzoid nevus vs. thin melanoma 0.5 mm ** | NF | MN | MN | MN | MM | MN | MM |
| Melanoma Breslow depth 0.2 mm | F | MN | MN | MM | AMN or MM | MN | MM |
| Melanoma in situ ** | NF | MN | MN | MN | MN | AMN | MN |
| Melanoma in situ ** | NF | MM | MM | MM | MM | MM | MM |
| Melanoma in situ ** | NF | AMN | MN | MN # | AMN | AMN | MN |
| Melanoma in situ ** | NF | AMN | SK | MN | AMN | MN | MN |
| Melanoma in situ | F | SL | AMN | SCC | SL | SL | MM |
| Melanoma in situ | F | MN | MM | MM | AMN or MM | MN | MM |
| BCC | F | BCC | MM | SCC | BCC | MM | MM or BCC |
| BCC | F | AK | DF | MN | BCC | SK | MM |
| BCC | F | BCC | SK | MM # | MM | MM | BCC |
| BCC | F | MM | MM | MM | MM | SK | MM |
| BCC | F | BCC | BCC | BCC | BCC or MM | SCC | BCC |
| BCC | F | BCC | BCC | BCC | BCC | BCC | BCC |
| Pigmented SCC in situ | F | SL | SL | SL | SL | SL | SL |
When an excision was suggested, the diagnosis is highlighted in bold.
Abbreviations: AI, artificial intelligence; AK, actinic keratosis; AMN, atypical melanocytic nevus; BCC, basal cell carcinoma; Derm, dermatologist; DF, dermatofibroma; Ext Derm, external dermatologist; F, flagged; MM, melanoma; MN, melanocytic nevus; NF, not flagged; Res, dermatology resident; SCC, squamous cell carcinoma in situ; SK, seborrheic keratosis; SL, solar lentigo.
Number of cancer diagnoses of the 17 cancers identified.
** Pathologically atypical melanocytic peoliferations where melanoma could not be excluded.
Excision suggested after review.
Table 6 shows the statistical comparison between the AI and the clinicians based on the relevant histopathological findings. The community dermatologist had the highest sensitivity with 57 suggested excisions and the corresponding lowest specificity and statistical accuracy. The AI had comparable sensitivity and specificity to 2 dermatologists suggesting a similar number of excisions. The AI had comparable statistical accuracy (overall probability that the lesion was correctly classified) to the external dermatologist at 73.6% and 74.7% respectively. Similarly, the AI’s NNB (2.25) was the lowest followed by the external dermatologist (2.3).
Table 6.
Statistical Evaluation for the 83 Excised Lesions.
| Excisions, Reviews, Statisitical Tests | AI | Derm 1 | Derm 2 | Derm 3 | Derm 5 | Res | Ext Derm |
|---|---|---|---|---|---|---|---|
| Excisions N (%)* | 27 (32.5%) | 28 (33.7%) | 27 (32.5%) | 39 (47.0%) | 57 (68.7%) | 24 (28.9%) | 28 (33.7%) |
| Reviews recommended | N/A | 10 (12%) | 1 (1%) | 11 (13%) | 15 (18%) | 8 (10%) | 14 (17%) |
| Sensitivity | 64.7% | 52.9% | 47% | 76.5% | 82.4% | 47% | 70.6% |
| Specificity | 75.76% | 69.7% | 71.2% | 48.5% | 31.8% | 75.8% | 76.8% |
| PLR | 2.67 | 1.75 | 1.63 | 1.48 | 1.21 | 1.94 | 2.91 |
| NLR | 0.47 | 0.68 | 0.74 | 0.49 | 0.55 | 0.70 | 0.39 |
| PPV | 40.0% | 30.4% | 29% | 27% | 23.2% | 32.7% | 42.1% |
| NPV | 89.6% | 85.6% | 83.3% | 89.1% | 87.8% | 83.2% | 91.1% |
| NNB | 2.25 | 3.2 | 3.4 | 3.6 | 4.2 | 3.0 | 2.3 |
| Accuracy | 73.56% | 66.4% | 66.4% | 54% | 41.9% | 70% | 74.72% |
Abbreviations: AI, artificial intelligence; Derm, dermatologist; Ext Derm, external dermatologist; NLR, negative likelihood ratio; NNB, number needed to biopsy; NPV, negative predictive value; PLR, positive likelihood ratio; PPV, positive predictive value; Res, dermatology resident.
Percentages of excisions recommended of the total number of excisions performed.
ROC curves were constructed (Figure 2) to compare the recommendation for excision made by the dermatologists and the AI with the pathological result. The closer to the central line, the less accurate the observers. The AUC or C-statistic is shown in Table 7. The AUC is an effective, alternative way of summarizing the overall accuracy of an assessment. The external dermatologist showed the greatest diagnostic accuracy (0.732) closely followed by the AI (0.702). Scores between 0.7 and 0.8 are acceptable.
Figure 2.
Receiver operator curve (ROC). The reference line indicated when sensitivity and 1-specificity have equal values.
Table 7.
Area Under the Receiver Operator Curve.
| Reviewer | Area | Std error | Asymptotic sig | 95% CI | |
|---|---|---|---|---|---|
| AI | 0.702 | 0.074 | .010 | 0.556 | 0.848 |
| Derm 1 | 0.628 | 0.079 | .104 | 0.474 | 0.783 |
| Derm 2 | 0.591 | 0.080 | .247 | 0.435 | 0.748 |
| Derm 3 | 0.648 | 0.074 | .060 | 0.504 | 0.793 |
| Derm 4 | 0.571 | 0.075 | .370 | 0.424 | 0.717 |
| Resident | 0.614 | 0.080 | .149 | 0.457 | 0.771 |
| External derm | 0.732 | 0.071 | .003 | 0.592 | 0.871 |
Scores above 0.7 are bolded to show an acceptable diagnostic accuracy.
Abbreviations: AI, artificial intelligence; CI, confidence interval; derm, dermatologist; sig, significance; Std, standard.
Considering the lesions excised and the decision by each dermatologist to excise or not, the inter-rater reliability based on Fleiss’ kappa had a value of .22 (95% CI: 0.16-0.28). The interpretation of ĸ values is notoriously difficult but based on the proposal by Landis and Koch, the agreement about action (excise or not) is fair. 10
Both dermatopathologists concurred in the diagnosis of all lesions. They reported that 38 (50%) lesions were diagnostic challenges (36 and 21, respectively). These challenging cases were reviewed by a third dermatopathologist, who agreed with the final diagnosis in all cases.
Discussion
Cancer screening programs aim to detect cancer at an early, more treatable stage. Population-based cancer screening programs, such as for breast and bowel cancer, focus on asymptomatic patients. Success depends on the incidence of that cancer and the ability of the screening method to detect it, balanced by any harm associated with the detection procedures. Population-based screening for melanoma would require whole-body skin examination, which is not feasible and is a poor use of resources. Population screening for melanoma is not recommended in the United States, Canada, Australia, or New Zealand.11,12 However, total skin examination, focused on individuals at high risk for skin cancer, may improve the benefit-to-harm balance. 13
Our study recruited 482 members of the public who were concerned about the possibility of skin cancer, highlighting that people are concerned about their skin. This was particularly relevant in patients with a family history of melanoma. As expected, many lesions were benign, with 46% clinically diagnosed as seborrheic keratoses. However, of the 381 lesions scanned, 17 were skin cancers.
The AI proved to have a diagnostic accuracy that was non-inferior to experienced dermatologists as assessed by most statistical parameters including accuracy, ROC, the area under the curve, and the NNB ratio. The latter is often cited as one of the most useful metrics for measuring accuracy in melanoma detection. 14 The NNB does not consider false negatives and is more a measure of specificity. 15
We have shown the need for easier access to expert advice for patients concerned about a skin lesion. AI screening can be practical, and its accuracy in selected cases is non-inferior to that of an expert dermoscopist. However, the AI missed 6 cases of in situ melanoma. Interestingly, the dermatopathologists found all these lesions to be diagnostically challenging. The ability of these atypical melanocytic proliferations to evolve and spread locally or metastasize is largely unknown. Vermariëen-Wang et al 16 have presented evidence that the atypical superficial melanocytic proliferations of uncertain significance do not metastasize. While reassuring, we feel that it would be prudent to repeat AI screening after an interval of 6 months or sooner if the patient notices a change in a lesion.
We suggest that AI could be incorporated into cancer screening programs for patients needing an urgent assessment of a pigmented lesion.
Limitations
The AI system used was unable to scan all lesions of concern for technical reasons, mainly due to the location of the lesion (Table 1). Ambiguous melanocytic proliferations may be missed. This study did not compare the overall clinical accuracy of dermatologists with AI and focused on a single lesion of concern and the interpretation of the dermoscopic findings.
Conclusion
We have demonstrated that AI could provide an accurate diagnostic spot-check tool to assess pigmented lesions of concern. This could be particularly helpful for patients with poor access to a healthcare provider suitably trained to diagnose pigmented lesions.
Footnotes
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding: The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: Dalhousie Medical Research Foundation Allan and Leslie Shaw Research Fund, Dalhousie Faculty of Medicine Sandy Murray RIM Studentship, Faculty of Medicine Professor Murray Macneill, Summer Medical Research Studentship.
IRB Approval Status: Reviewed and approved by the Nova Scotia Health Authority Research Ethics Board, Halifax, Nova Scotia. NSHA REB ROMEO FILE #102499.
ORCID iDs: Rachel A. Dorey 
https://orcid.org/0000-0001-8400-6714
Michael L. MacGillivary
https://orcid.org/0000-0001-8339-257X
Peter R. Hull
https://orcid.org/0000-0002-3896-9373
References
- 1. Haenssle HA, Fink C, Toberer F, et al. Man against machine reloaded: performance of a market-approved convolutional neural network in classifying a broad spectrum of skin lesions in comparison with 96 dermatologists working under less artificial conditions. Ann Oncol. 2020;31(1):137-143. [DOI] [PubMed] [Google Scholar]
- 2. Fink C, Blum A, Buhl T, et al. Diagnostic performance of a deep learning convolutional neural network in the differentiation of combined naevi and melanomas. J Eur Acad Dermatol Venereol. 2020;34(6):1355-1361. [DOI] [PubMed] [Google Scholar]
- 3. Winkler JK, Sies K, Fink C, et al. Melanoma recognition by a deep learning convolutional neural network—performance in different melanoma subtypes and localisations. Eur J Cancer. 2020;127:21-29. [DOI] [PubMed] [Google Scholar]
- 4. MacLellan AN, Price EL, Publicover-Brouwer P, et al. The use of noninvasive imaging techniques in the diagnosis of melanoma: a prospective diagnostic accuracy study. J Am Acad Dermatol. 2021;85(2):353-359. [DOI] [PubMed] [Google Scholar]
- 5. Sies K, Winkler JK, Fink C, et al. Past and present of computer-assisted dermoscopic diagnosis: performance of a conventional image analyser versus a convolutional neural network in a prospective data set of 1,981 skin lesions. Eur J Cancer. 2020;135:39-46. [DOI] [PubMed] [Google Scholar]
- 6. Conte S, Ghazawi FM, Le M, et al. Population-based study detailing cutaneous melanoma incidence and mortality trends in Canada. Front Med. 2022;9:830254. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7. Nova Scotia Health Authority. Finding a primary care provider in Nova Scotia—February 2023. 2023. Accessed March 29, 2023. https://www.nshealth.ca/documentsandreports/finding-primary-care-provider-nova-scotia-february-2023.
- 8. Tromme I, Sacré L, Hammouch F, Richez P, Degryse JM, Speybroeck N. Melanoma diagnosis: predictive value of macroscopic changes observed by the patient. Melanoma Res. 2018;28(6):611-617. [DOI] [PubMed] [Google Scholar]
- 9. Avilés-Izquierdo JA, Molina-López I, Rodríguez-Lomba E, Marquez-Rodas I, Suarez-Fernandez R, Lazaro-Ochaita P. Who detects melanoma? Impact of detection patterns on characteristics and prognosis of patients with melanoma. J Am Acad Dermatol. 2016;75(5):967-974. [DOI] [PubMed] [Google Scholar]
- 10. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977;33(1):159-174. [PubMed] [Google Scholar]
- 11. Johansson M, Brodersen J, Gøtzsche PC, Jørgensen KJ. Screening for reducing morbidity and mortality in malignant melanoma. Cochrane Database Syst Rev. 2019;6(6):CD012352. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12. Johnson MM, Leachman SA, Aspinwall LG, et al. Skin cancer screening: recommendations for data-driven screening guidelines and a review of the US Preventive Services Task Force controversy. Melanoma Manag. 2017;4(1):13-37. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13. Omara S, Wen D, Ng B, et al. Identification of incidental skin cancers among adults referred to dermatologists for suspicious skin lesions. JAMA Network Open. 2020;3(12):e2030107. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14. Argenziano G, Cerroni L, Zalaudek I, et al. Accuracy in melanoma detection: a 10-year multicenter survey. J Am Acad Dermatol 2012;67(1):54-59. [DOI] [PubMed] [Google Scholar]
- 15. Marchetti MA, Yu A, Nanda J, et al. Number needed to biopsy ratio and diagnostic accuracy for melanoma detection. J Am Acad Dermatol. 2020;83(3):780-787. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16. Vermariën-Wang J, Doeleman T, van Doorn R, Mooyaart AL, Blokx WAM, Schrader AMR. Ambiguous melanocytic lesions: a retrospective cohort study of incidence and outcome of melanocytic tumor of uncertain malignant potential (MELTUMP) and superficial atypical melanocytic proliferation of uncertain significance (SAMPUS) in the Netherlands. J Am Acad Dermatol. 2023;88(3):602-608. [DOI] [PubMed] [Google Scholar]