Dermoscopy, with and without visual inspection, for diagnosing melanoma in adults

Abstract

Background

Melanoma has one of the fastest rising incidence rates of any cancer. It accounts for a small percentage of skin cancer cases but is responsible for the majority of skin cancer deaths. Although history‐taking and visual inspection of a suspicious lesion by a clinician are usually the first in a series of ‘tests’ to diagnose skin cancer, dermoscopy has become an important tool to assist diagnosis by specialist clinicians and is increasingly used in primary care settings. Dermoscopy is a magnification technique using visible light that allows more detailed examination of the skin compared to examination by the naked eye alone. Establishing the additive value of dermoscopy over and above visual inspection alone across a range of observers and settings is critical to understanding its contribution for the diagnosis of melanoma and to future understanding of the potential role of the growing number of other high‐resolution image analysis techniques.

Objectives

To determine the diagnostic accuracy of dermoscopy alone, or when added to visual inspection of a skin lesion, for the detection of cutaneous invasive melanoma and atypical intraepidermal melanocytic variants in adults. We separated studies according to whether the diagnosis was recorded face‐to‐face (in‐person), or based on remote (image‐based), assessment.

Search methods

We undertook a comprehensive search of the following databases from inception up to August 2016: CENTRAL; MEDLINE; Embase; CINAHL; CPCI; Zetoc; Science Citation Index; US National Institutes of Health Ongoing Trials Register; NIHR Clinical Research Network Portfolio Database; and the World Health Organization International Clinical Trials Registry Platform. We studied reference lists and published systematic review articles.

Selection criteria

Studies of any design that evaluated dermoscopy in adults with lesions suspicious for melanoma, compared with a reference standard of either histological confirmation or clinical follow‐up. Data on the accuracy of visual inspection, to allow comparisons of tests, was included only if reported in the included studies of dermoscopy.

Data collection and analysis

Two review authors independently extracted all data using a standardised data extraction and quality assessment form (based on QUADAS‐2). We contacted authors of included studies where information related to the target condition or diagnostic threshold were missing. We estimated accuracy using hierarchical summary receiver operating characteristic (SROC),methods. Analysis of studies allowing direct comparison between tests was undertaken. To facilitate interpretation of results, we computed values of sensitivity at the point on the SROC curve with 80% fixed specificity and values of specificity with 80% fixed sensitivity. We investigated the impact of in‐person test interpretation; use of a purposely developed algorithm to assist diagnosis; observer expertise; and dermoscopy training.

Main results

We included a total of 104 study publications reporting on 103 study cohorts with 42,788 lesions (including 5700 cases), providing 354 datasets for dermoscopy. The risk of bias was mainly low for the index test and reference standard domains and mainly high or unclear for participant selection and participant flow. Concerns regarding the applicability of study findings were largely scored as ‘high’ concern in three of four domains assessed. Selective participant recruitment, lack of reproducibility of diagnostic thresholds and lack of detail on observer expertise were particularly problematic.

The accuracy of dermoscopy for the detection of invasive melanoma or atypical intraepidermal melanocytic variants was reported in 86 datasets; 26 for evaluations conducted in person (dermoscopy added to visual inspection), and 60 for image‐based evaluations (diagnosis based on interpretation of dermoscopic images). Analyses of studies by prior testing revealed no obvious effect on accuracy; analyses were hampered by the lack of studies in primary care, lack of relevant information and the restricted inclusion of lesions selected for biopsy or excision. Accuracy was higher for in‐person diagnosis compared to image‐based evaluations (relative diagnostic odds ratio (RDOR) 4.6, 95% confidence interval (CI) 2.4 to 9.0; P < 0.001).

We compared accuracy for (a), in‐person evaluations of dermoscopy (26 evaluations; 23,169 lesions and 1664 melanomas),versus visual inspection alone (13 evaluations; 6740 lesions and 459 melanomas), and for (b), image‐based evaluations of dermoscopy (60 evaluations; 13,475 lesions and 2851 melanomas),versus image‐based visual inspection (11 evaluations; 1740 lesions and 305 melanomas). For both comparisons, meta‐analysis found dermoscopy to be more accurate than visual inspection alone, with RDORs of (a), 4.7 (95% CI 3.0 to 7.5; P < 0.001), and (b), 5.6 (95% CI 3.7 to 8.5; P < 0.001). For a), the predicted difference in sensitivity at a fixed specificity of 80% was 16% (95% CI 8% to 23%; 92% for dermoscopy + visual inspection versus 76% for visual inspection), and predicted difference in specificity at a fixed sensitivity of 80% was 20% (95% CI 7% to 33%; 95% for dermoscopy + visual inspection versus 75% for visual inspection). For b) the predicted differences in sensitivity was 34% (95% CI 24% to 46%; 81% for dermoscopy versus 47% for visual inspection), at a fixed specificity of 80%, and predicted difference in specificity was 40% (95% CI 27% to 57%; 82% for dermoscopy versus 42% for visual inspection), at a fixed sensitivity of 80%.

Using the median prevalence of disease in each set of studies ((a), 12% for in‐person and (b), 24% for image‐based), for a hypothetical population of 1000 lesions, an increase in sensitivity of (a), 16% (in‐person), and (b), 34% (image‐based), from using dermoscopy at a fixed specificity of 80% equates to a reduction in the number of melanomas missed of (a), 19 and (b), 81 with (a), 176 and (b), 152 false positive results. An increase in specificity of (a), 20% (in‐person), and (b), 40% (image‐based), at a fixed sensitivity of 80% equates to a reduction in the number of unnecessary excisions from using dermoscopy of (a), 176 and (b), 304 with (a), 24 and (b), 48 melanomas missed.

The use of a named or published algorithm to assist dermoscopy interpretation (as opposed to no reported algorithm or reported use of pattern analysis), had no significant impact on accuracy either for in‐person (RDOR 1.4, 95% CI 0.34 to 5.6; P = 0.17), or image‐based (RDOR 1.4, 95% CI 0.60 to 3.3; P = 0.22), evaluations. This result was supported by subgroup analysis according to algorithm used. We observed higher accuracy for observers reported as having high experience and for those classed as ‘expert consultants’ in comparison to those considered to have less experience in dermoscopy, particularly for image‐based evaluations. Evidence for the effect of dermoscopy training on test accuracy was very limited but suggested associated improvements in sensitivity.

Authors' conclusions

Despite the observed limitations in the evidence base, dermoscopy is a valuable tool to support the visual inspection of a suspicious skin lesion for the detection of melanoma and atypical intraepidermal melanocytic variants, particularly in referred populations and in the hands of experienced users. Data to support its use in primary care are limited, however, it may assist in triaging suspicious lesions for urgent referral when employed by suitably trained clinicians. Formal algorithms may be of most use for dermoscopy training purposes and for less expert observers, however reliable data comparing approaches using dermoscopy in person are lacking.

Plain language summary

How accurate is dermoscopy compared to visual inspection of the skin for diagnosing skin cancer (melanoma) in adults?

What is the aim of the review?

The aim of this Cochrane Review was to find out the accuracy of dermoscopy for the diagnosis of melanoma in comparison to visual inspection of the skin with the naked eye. The Review also investigated whether diagnostic accuracy using dermoscopy on a patient in person differed to the accuracy of diagnosis using dermoscopic images of the skin. Researchers in Cochrane included 104 studies to answer this question.

Why is improving the diagnosis of melanoma important?

Melanoma is one of the most dangerous forms of skin cancer. Not recognising a melanoma when it is present (a false‐negative test result), delays surgery to remove it, risking cancer spreading to other organs in the body, and possibly death. Diagnosing a skin lesion (a mole or area of skin with an unusual appearance in comparison with the surrounding skin) as a melanoma when it is not (a false‐positive result), may result in unnecessary surgery, further investigations and patient anxiety. Visual inspection of suspicious skin lesions by a clinician using the naked eye is usually the first of a series of ‘tests’ to diagnose melanoma. Magnification techniques can be used by skin cancer specialists to allow a more detailed examination of suspicious skin lesions than can be achieved using the naked eye alone.

What was studied in the review?

A dermatoscope is a handheld device using visible light (such as from incandescent or LED bulbs), that can be used as part of the clinical examination of suspicious skin lesions. Dermoscopy has become an important tool to assist diagnosis by specialist clinicians and is also increasingly used in primary care settings. Knowing the diagnostic accuracy of dermoscopy added to visual inspection alone is important to understanding who it should be used by and in which healthcare settings.

Researchers sought to find out the diagnostic accuracy of dermoscopy of suspicious skin lesions on a patient in person and using dermoscopic images compared to visual inspection alone. Researchers also sought to find out whether diagnostic accuracy was improved by use of a dermoscopy checklist or by an increase in level of clinical expertise.

What are the main results of the review?

The review included 104 studies reporting data for people with lesions suspected of melanoma. The main results for the diagnosis of melanoma (including very early melanomas), are based on 86 of the studies, 26 of which provide information on the accuracy of dermoscopy added to in‐person visual inspection of a skin lesion and 60 provide information based on examination of dermoscopic images without the patient being present.

The 26 in‐person studies provide the most relevant data for the use of dermoscopy in practice and their results are summarised here. A total of 23,169 suspicious skin lesions were included in the 26 studies and 13 of them also provided information on the accuracy of visual inspection of a lesion without the use of dermoscopy. The results suggest that dermoscopy is more accurate than visual inspection on its own, both for identifying melanoma correctly and excluding things that are not melanoma.

The studies used different ways of deciding whether a skin lesion was a melanoma or not, which means that we cannot be exactly sure about how much better dermoscopy is compared to visual inspection alone. Instead we can give an illustrative example of the expected effect of the increase in accuracy using a group of 1000 lesions, of which 120 (12%), are melanoma. In order to see how much better dermoscopy is in identifying melanoma correctly when compared to just looking at the skin, we have to assume that both lead to the same number of lesions being falsely diagnosed as melanoma (we assumed that 176 of the 880 lesions without melanoma would have an incorrect diagnosis of melanoma). In this fixed situation, adding dermoscopy to visual inspection would correctly identify an extra 19 melanomas (110 compared with 91), that would have been missed by just looking at the skin alone. In other words, more melanomas would be correctly identified.

In order to see how much better dermoscopy is in deciding if a skin lesion is not a melanoma when compared to just looking at the skin, we have to assume that both lead to the same number of melanomas being correctly diagnosed (in this case we assumed that 96 out of the 120 melanomas would be correctly diagnosed). In this situation, adding in dermoscopy to visual inspection would reduce the number of lesions being wrongly diagnosed as being melanoma by 176 (a reduction from 220 in the visual inspection group to 44 lesions in the dermoscopy group). In other words, more lesions that were not melanoma would be correctly identified and fewer people would end up being sent for surgery.

Value of visual inspection checklists and effect of observer expertise

There was no evidence that use of a checklist to help dermoscopy interpretation changed diagnostic accuracy. Accuracy was better (with fewer missed melanomas and fewer people having unnecessary surgery), when the diagnosis was made by people with more clinical expertise and training.

How reliable are the results of the studies of this review?

In the majority of included studies, the diagnosis of melanoma was made by lesion biopsy and the absence of melanoma was confirmed by biopsy or by follow‐up over time to make sure the skin lesion remained negative for melanoma, both of which are likely to have been a reliable method for deciding whether patients really had melanoma*. In a few studies, the absence of melanoma was made by expert diagnosis, which is unlikely to have been a reliable method for deciding whether patients really had melanoma. Poor reporting of study conduct made assessment of the reliability of studies difficult. Selective participant recruitment and lack of detail regarding the threshold for deciding on a positive test result were particularly problematic.

Who do the results of this review apply to?

Sixty‐six studies were undertaken in Europe (77%), with the remainder undertaken in North America (6 studies), Asia (4), Oceania (4), or were multicentre (7). Mean age ranged from 30 to 58 years (reported in 26 studies). The percentage of individuals with melanoma ranged between 1% and 41% for dermoscopy in‐person studies (median 12%), and between 3% and 61% in studies using dermoscopy images (median 24%). Almost all of the studies were carried out in referral settings rather than in primary care. In the majority of studies the lesions were unlikely to be representative of the range of those seen in practice, for example only including skin lesions of a certain size or with a specific appearance. In addition variation in the expertise of clinicians performing visual inspection and the definition used for a positive dermoscopy test result across studies makes it unclear as to how dermoscopy should be carried out and by people with different levels of clinical expertise in order to achieve the accuracy observed in studies.

What are the implications of this review?

When used by specialists, dermoscopy is better at diagnosing melanoma compared to inspection of a suspicious skin lesion using the naked eye alone. Dermoscopy is more accurate when interpreted with the patient present rather than using dermoscopy images. Dermoscopy might help general practitioners to correctly identify people with suspicious lesions who need to be seen by a specialist. Checklists to help interpret dermoscopy might improve the accuracy of people with less expertise and training. Further, well‐reported studies assessing the diagnostic accuracy of dermoscopy when used in primary care and to identify the best way of delivering dermoscopy training are needed.

How up‐to‐date is this review?

The review authors searched for and used studies published up to August 2016.

*In these studies biopsy, clinical follow‐up or specialist clinician diagnosis were the reference standards (means of establishing the final diagnosis).

Summary of findings

Summary of findings'. 'Summary of findings table.

Question	What is the diagnostic accuracy of dermoscopy, in comparison to visual inspection, for the detection of cutaneous invasive melanoma and atypical intraepidermal melanocytic variantsin adults?
Population	Adults with lesions suspicious for melanoma, including: those with limited prior testing (presenting in primary, community or private dermatology settings), and referred populations (presenting in secondary care or specialist skin cancer clinics).
Index test	Dermoscopy with or without the use of any established algorithms or checklist to aid diagnosis, including: in‐person evaluations (face‐to‐face diagnosis), and image‐based evaluations (diagnosis based on assessment of a clinical image).
Comparator test	Visual inspection
Target condition	Cutaneous invasive melanoma and atypical intraepidermal melanocytic variants
Reference standard	Histology with or without follow‐up to confirm absence of malignancy in benign‐appearing lesions
Action	If accurate, positive results ensure melanoma lesions are not missed but are appropriately excised (or referred), and those with negative results can be safely reassured and discharged.
	Number of studies		Total lesions			Total melanomas
Quantity of evidence	104		42,788			5700
Limitations
Risk of bias (in‐person; image‐based)	Potential risk for participant selection from use of case‐control type design (19 image‐based), inappropriate exclusion criteria (8; 25), or lack of detail (17; 27). All dermoscopy interpretation was blinded to reference standard diagnosis. Dermoscopy thresholds were clearly pre‐specified (25; 50). Low risk for reference standard (29; 63); high risk from use of expert diagnosis or > 20% of benign lesions with no histology (5; 11). Blinding of reference standard to clinical diagnosis reported only in one image‐based evaluation. High risk for participant flow (15; 26), due to differential verification (6; 15), and exclusions following recruitment (10; 16). Timing of tests was not mentioned in 23 (18).
Applicability of evidence to question (in‐person; image‐based)	Participants restricted to those with melanocytic lesions only (10; 35), or other narrowly defined groups (5 image‐based), or to those with histopathology results (29; 57), and included multiple lesions per participant (8 in‐person). High concern for dermoscopy (16; 57), with no description of diagnostic thresholds (8; 25), or reporting of average or consensus diagnoses (9; 35). Dermoscopic image interpretation blinded to clinical images (51 image‐based). Little information given concerning the expertise of the histopathologist (28; 50).
Findings
We included 104 study publications (providing data for 103 cohorts of lesions). We separated a priori 83 publications providing 86 datasets for evaluation of the primary target condition into in‐person (n = 26), and image‐based (n = 60), evaluations. Subsequent analysis confirmed differences in accuracy according to the different approaches to diagnosis (P < 0.0001). Analyses of studies by degree of prior testing revealed no obvious effect on accuracy; the study publications provided insufficient relevant information, and the majority of studies were apparently conducted in referred populations, which hampered our analyses. The findings presented are based on results for all studies regardless of position on the clinical pathway. Sensitivities at fixed specificities and specificities at fixed sensitivities are given for illustrative purposes only and should not be taken as indicative of actual test performance.
Test	In‐person visual inspection alone versus visual inspection plus dermoscopy: any algorithm or threshold
Data analysed	Visual inspection				13 datasets; 6740 lesions; 459 cases
	Dermoscopy				26 datasets; 23,169 lesions; 1664 cases
Resultsa	Sensitivity (95% CI) %		Fixed specificity		Fixed sensitivity		Specificity (95% CI) %
Visual inspection	76% (66 to 85)		80%		80%		75% (57 to 87)
Dermoscopy	92% (87 to 95)						95% (90 to 98)
Numbers applied to a hypothetical cohort of 1000 lesionsb
	TP	FN	FP	TN	TP	FN	FP	TN
At a prevalence of 5%	VI: 38 D: 46 ↑ 8	VI: 12 D: 4 ↓ 8	190	760	40	10	VI: 238 D: 47 ↓191	VI: 713 D: 904 ↑191
At a prevalence of 12%	VI: 91 D: 110 ↑19	VI: 29 D: 10 ↓ 19	176	704	96	24	VI: 220 D: 44 ↓176	VI: 660 D: 836 ↑176
At a prevalence of 21%	VI: 160 D: 193 ↑ 33	VI: 50 D: 17 ↓ 33	158	632	168	42	VI: 198 D: 40 ↓158	VI: 5935 D: 750 ↑158
Test:	Image‐based visual inspection alone versus visual inspection plus dermoscopy: any algorithm or threshold
Data analysed	Visual inspection				11 datasets; 1740 lesions; 305 cases
	Dermoscopy				60 datasets; 13475 lesions; 2851 cases
Results	Sensitivity (95% CI) %		Fixed specificity		Fixed sensitivity		Specificity (95% CI) %
Visual inspection	47% (34 to 59)		80%		80%		42% (28 to 58)
Dermoscopy	81% (76 to 86)						82% (75 to 87)
Numbers applied to a hypothetical cohort of 1000 lesionsc
	TP	FN	FP	TN	TP	FN	FP	TN
At a prevalence of 18%	VI: 85 D: 146 ↑ 61	VI: 95 D: 34 ↓ 61	164	656	144	36	VI: 476 D: 148 ↓328	VI: 344 D: 672 ↑328
At a prevalence of 24%	VI: 113 D: 194 ↑81	VI: 127 D: 46 ↓ 81	152	608	192	48	VI: 441 D: 137 ↓304	VI: 319 D: 623 ↑304
At a prevalence of 39%	VI: 183 D: 316 ↑ 133	VI: 207 D: 74 ↓ 133	122	488	312	78	VI: 354 D: 110 ↓244	VI: 256 D: 500 ↑244
Test	Results according to algorithm used to assist dermoscopy interpretation
	Datasets	Lesions; cases	Sensitivity (95% CIs) %	Specificity (95% CI) %	Numbers in a cohort of 1000 lesionsd
					TP	FN	FP	TN
In‐person					At median prevalence of 12%
No algorithm	8	4704; 849	88% (75 to 95)	87% (80 to 92)	106	14	114	766
Pattern analysis	6	4307; 296	92% (87 to 95)	92% (68 to 98)	110	10	70	810
ABCD at > 5.45 (or likely)	5	1438; 160	81% (62 to 92)	92% (82 to 97)	97	235	70	810
Image‐based					At median prevalence of 24%
No algorithm	24	4498; 941	76% (70 to 82)	79% (71 to 85)	182	58	61	699
Pattern analysis	20	4621; 989	83% (76 to 88)	87% (80 to 92)	199	41	99	661
ABCD at > 5.45	7	2471; 406	81% (60 to 92)	81% (69 to 89)	194	46	144	616
7PCL at ≥ 3	11	3408; 798	80% (63 to 91)	67% (51 to 80)	192	48	251	509
3PCL	7	1505; 363	74% (61 to 85)	60% (42 to 76)	178	62	304	456
3PCL: three‐point checklist; 7PCL: seven‐point checklist; ABCD(E): asymmetry, border, colour, differential structures (enlargement); CI: confidence interval; D: dermoscopy; FN: false‐negative; FP: false‐positive; TN: true‐negative; TP: true‐positive; VI: visual inspection

^aNumbers for a hypothetical cohort of 1000 lesions are presented for two illustrative examples of points on the SROC curves: firstly for the sensitivities of tests at fixed specificities of 80%; and secondly for the specificities of tests at fixed sensitivities of 80%.
 ^bNumbers estimated at 25th, 50th (median), and 75% percentiles of invasive melanoma or atypical intraepidermal melanocytic variants prevalence observed across 26 datasets reporting in‐person evaluations of dermoscopy added to visual inspection.
 ^cNumbers estimated at 25th, 50th (median), and 75% percentiles of invasive melanoma or atypical intraepidermal melanocytic variants prevalence observed across 60 datasets reporting diagnosis using dermoscopic images
 ^dNumbers estimated at median prevalence (50th percentile), of invasive melanoma or atypical intraepidermal melanocytic variants observed across 26 datasets reporting in‐person evaluations of dermoscopy added to visual inspection and then for 60 datasets reporting diagnosis using dermoscopic images

Background

This review is one of a series of Cochrane Diagnostic Test Accuracy (DTA) Reviews on the diagnosis and staging of melanoma and keratinocyte skin cancers conducted for the National Institute for Health Research (NIHR) Cochrane Systematic Reviews Programme. Appendix 1 shows the content and structure of the programme.

Target condition being diagnosed

Melanoma is one of the most aggressive forms of skin cancer, with the potential to metastasise to other parts of the body via the lymphatic system and blood stream. It accounts for a small percentage of skin cancer cases but is responsible for up to 75% of skin cancer deaths (Boring 1994; Cancer Research UK 2017a).

Melanoma arises from uncontrolled proliferation of melanocytes ‐ the epidermal cells that produce pigment or melanin. It most commonly arises in the skin but can occur in any organ that contains melanocytes, including mucosal surfaces, the back of the eye, and lining around the spinal cord and brain. Cutaneous melanoma refers to a skin lesion with malignant melanocytes present in the dermis, and includes superficial spreading, nodular, acral lentiginous, and lentigo maligna melanoma variants (see Figure 1). Melanoma 'in situ' refers to malignant melanocytes that are contained within the epidermis and have not yet invaded the dermis, but are at risk of progression to melanoma if left untreated. Lentigo maligna, a subtype of melanoma in situ in chronically sun‐damaged skin, denotes another form of proliferation of abnormal melanocytes. Lentigo maligna can progress to invasive melanoma if its growth breaches the dermo‐epidermal junction during a vertical growth phase (when it becomes known as 'lentigo maligna melanoma'), however its rate of malignant transformation is both lower and slower than for melanoma in situ (Kasprzak 2015). Melanoma in situ and lentigo maligna are both atypical intraepidermal melanocytic variants.

1.

Sample photographs of superficial spreading melanoma (left) and nodular melanoma (right). Copyright © 2010 Dr Rubeta Matin: reproduced with permission.

The incidence of melanoma rose to over 200,000 newly diagnosed cases worldwide in 2012 (Erdmann 2013; Ferlay 2015), with an estimated 55,000 deaths (Ferlay 2015). The highest incidence is observed in Australia with 13,134 new cases of melanoma of the skin in 2014 (ACIM 2017), and in New Zealand with 2341 registered cases in 2010 (HPA and MelNet NZ 2014). For 2014 in the USA, the predicted incidence was 73,870 per annum and the predicted number of deaths 9940 (Siegel 2015). The highest rates in Europe are seen in north‐western Europe and the Scandinavian countries, with highest incidence reported in Switzerland of 25.8 per 100,000 in 2012. Rates in England have tripled from 4.6 and 6.0 per 100,000 in men and women, respectively, in 1990, to 18.6 and 19.6 per 100,000 in 2012 (EUCAN 2012). Indeed in the UK, melanoma has one of the fastest rising incidence rates of any cancer, and has the biggest projected increase in incidence between 2007 and 2030 (Mistry 2011). In the decade leading up to 2013, age‐standardised incidence increased by 46%, with 14,500 new cases in 2013 and 2459 deaths in 2014 (Cancer Research UK 2017b). While overall incidence rates are higher in women than in men, the rate of incidence in the latter is increasing faster than in women (Arnold 2014).

The rising incidence in melanoma is thought to be primarily related to an increase in recreational sun exposure and use of tanning beds, and an increasingly ageing population with higher lifetime ultraviolet (UV), exposure, in conjunction with possible earlier detection (Belbasis 2016; Linos 2009). Putative risk factors are reviewed in detail elsewhere (Belbasis 2016), but can be broadly divided into host or environmental factors. Host factors include fair skin and light hair or eye colour; older age (Geller 2002); male sex (Geller 2002); previous skin cancer history (Tucker 1985); predisposing skin lesions, for example, high melanocytic naevus counts (Gandini 2005), clinically atypical naevi (Gandini 2005), or large congenital naevi (Swerdlow 1995); genetically inherited skin disorders, such as xeroderma pigmentosum (Lehmann 2011), and a family history of melanoma (Gandini 2005). Environmental factors include recreational and occupational exposure to sunlight, both cumulative and episodic burning (Armstrong 2017; Gandini 2005); artificial tanning (Boniol 2012); and immunosuppression, for example, in organ transplant recipients or HIV‐positive individuals (DePry 2011). Lower socioeconomic class may be associated with delayed presentation and thus more advanced disease at diagnosis (Reyes‐Ortiz 2006).

A database of over 40,000 US patients from 1998 onwards, which assisted the development of the 8th American Joint Committee on Cancer (AJCC) Staging System indicated a five‐year survival of 99% for very early stage melanoma, dropping to anything between 32% and 93% in stage III disease depending on tumour thickness, the presence of ulceration and number of involved nodes (Gershenwald 2017). Before the advent of targeted and immunotherapies, disseminated melanoma (to distant sites/visceral organs), was associated with median survival of six to nine months, one‐year survival rate of 25%, and three‐year survival of 15% (Balch 2009; Korn 2008).

Between 1975 and 2010, five‐year relative survival for melanoma (i.e. not including deaths from other causes), in the USA increased from 80% to 94%, with survival for localised, regional, and distant disease estimated at 99%, 70%, and 18%, respectively in 2010 (Cho 2014). However, mortality rates showed little change, at 2.1 per 100,000 deaths in 1975 and 2.7 per 100,000 in 2010 (Cho 2014). Increasing incidence in localised disease over the same period (from 5.7 to 21 per 100,000), suggests that much of the observed improvement in survival may be due to earlier detection and heightened vigilance (Cho 2014). New targeted therapies for advanced (stage IV), melanoma (e.g. BRAF inhibitors), have improved survival, and immunotherapies are evolving such that long‐term survival is being documented (Pasquali 2018; Rozeman 2017). No new data regarding the survival prospects for patients with stage IV disease were analysed for the AJCC 8 staging guidelines due to lack of contemporary data (Gershenwald 2017).

Treatment of melanoma

For primary melanoma, the mainstay of definitive treatment is early detection and excision of the lesion, to remove both the tumour and any malignant cells that might have spread into the surrounding skin (Garbe 2016; Marsden 2010; NICE 2015a; SIGN 2017; Sladden 2009). Recommended surgical margins vary according to tumour thickness (Garbe 2016), and stage of disease at presentation (NICE 2015a).

Index test(s)

For the purposes of our series of reviews, we consider each component of the diagnostic process, including visual inspection or clinical examination, a diagnostic or index ‘test', the accuracy of which can be established in comparison with a reference standard of diagnosis, either alone or in combination with other available technologies that may assist the diagnostic process. In this review, although dermoscopy is the primary focus, two index tests are in fact under consideration, namely visual inspection and dermoscopy, both of which can be undertaken in person (face‐to‐face with the patient), or as an image‐based examination (remote from the patient using images). As dermoscopy is added to visual inspection of a skin lesion when it is undertaken in person, we effectively have three index tests: visual inspection alone (in person or using images), visual inspection + dermoscopy (in‐person dermoscopy), and dermoscopy alone (image‐based dermoscopy).

As visual inspection of a lesion is always undertaken first in a face‐to‐face patient consultation, in this section we first consider visual inspection alone before going on to describe the addition of dermoscopy.

Visual inspection

Clinical history‐taking to identify risk factors and visual inspection of the lesion, surrounding skin and comparison with other lesions on the rest of the body, is fundamental to the diagnosis of skin cancer. In the UK, clinical examination is typically done at two decision points – first in the general practice (GP) surgery where a decision is made to refer or not to refer, and then a second time by a dermatologist or other secondary care clinician where a decision is made to biopsy or not.

Visual inspection of a lesion relies on both non‐analytical and analytical pattern recognition strategies (Elstein 2002; Norman 1989; Norman 2009). Non‐analytical pattern recognition formulates an initial hypothesis hidden from the conscious view of the diagnostician, while analytical pattern recognition uses more explicit rules based on conscious analytical reasoning (Norman 2009). The balance between non‐analytical and analytical reasoning varies between clinicians, according to factors such as constitutional reasoning style preference, experience and familiarity with the diagnostic question. Various attempts have been made to formalise the 'mental rules' involved in analytical pattern recognition for melanoma, ranging from setting out criteria that should be considered (e.g. ‘pattern analysis’; Friedman 1985; Sober 1979), to formal scoring systems with explicit numerical thresholds (MacKie 1985; MacKie 1990). These variants on visual inspection strategies, and their comparative accuracy, are reviewed in detail in a separate systematic review in this series (Dinnes 2018a). We have included in this review data on the accuracy of visual inspection only where both visual inspection and dermoscopy were evaluated in the same lesions in order to robustly estimate the comparative accuracy of adding dermoscopy to visual inspection compared to visual inspection alone, so that the benefit of dermoscopy can be quantified.

Visual inspection of a digital photograph (or ‘macroscopic’ image), of a suspicious skin lesion can also be undertaken as part of a teledermatology consultation, whereby photographs, dermoscopic images, or both are taken by non‐specialist clinicians and forwarded to a dermatologist to obtain a specialist opinion (Chuchu 2018a). Images can also be encompassed in a store‐and‐forward smartphone application, whereby a photograph of a concerning lesion is taken by the smartphone user and forwarded for an assessment of skin cancer risk by a specialist clinician (Chuchu 2018b). Images are often accompanied by a summary of the medical history and demographic information as part of a consultation package (Ndegwa 2010). According to UK guidelines, both clinical and dermoscopic images must be sent for ‘full dermatology’, that is, as a replacement for a face‐to‐face consultation, whereas for ‘triage teledermatology’ dermoscopic images should be sent where facilities permit (BAD 2013).

Dermoscopy

Dermoscopy (also referred to as dermatoscopy or epiluminescence microscopy or ELM), has become a widely used tool for the specialist clinician and is increasingly being used in primary care settings. It uses a hand‐held microscope and incident light (with or without oil immersion), to reveal subsurface images of the skin at increased magnification of x 10 to x 100 (Kittler 2011). Used alongside clinical examination, dermoscopy has been shown in some studies to increase the sensitivity of clinical diagnosis of melanoma from around 60% to as much as 90% (Bono 2006; Carli 2002a; Kittler 1999; Stanganelli 2000), with much smaller effects in others (Benelli 1999; Bono 2002a).

The visual nature of dermoscopic interpretation means that, when used on an in‐person basis, dermoscopy is essentially added to visual inspection of a skin lesion, and similar non‐analytical and analytical pattern recognition strategies are also employed to reach a diagnosis. Pattern analysis (Pehamberger 1993; Steiner 1987a), is thought to be the most specific and reliable technique to aid dermoscopy interpretation when used by specialists (Maley 2014); however, dermoscopic histological correlations have been established and diagnostic algorithms developed based on colour, aspect, pigmentation pattern, and skin vessels. One of the first formal scoring systems was the ABCD rule for dermoscopy (Nachbar 1994; Stolz 1994a), which includes 21 different features to be considered and scored (two based on asymmetry of the lesion, eight on lesion border, six related to lesion colour and five to differential structures), and has reported sensitivity ranging between 84% and 93% (Nachbar 1994; Stolz 1994a). Subsequently published algorithms attempt to simplify assessment without missing melanomas, for example, the Menzies tool (Menzies 1996), the seven‐point dermoscopy checklist (Annessi 2007; Argenziano 1998; Argenziano 2001; Gereli 2010, amongst others), and the three‐point checklist (Gereli 2010). However, dermoscopy can fail to diagnose atypical or early or featureless melanomas (Skvara 2005). These and other identified algorithms are described in detail in Appendix 2.

In modern practice, dermoscopic images are almost always obtained for skin lesions that are recommended for excision and are also obtained for lesions that have not yet met the diagnostic threshold for excision but are to be monitored over time in case of any further suspicious changes. Dermoscopic images are also a key component of teledermatology consultations, usually accompanied by digital photographs and other pertinent information (Chuchu 2018a), as discussed above.

The accuracy of dermoscopy has been suggested to vary with examiner experience (Kittler 2011), and results when used by untrained or less experienced examiners are potentially no better than clinical inspection alone (Binder 1997; Kittler 2002). Training in dermoscopy use can vary from a single one‐hour lecture (Benvenuto‐Andrade 2006), to an intensive course lasting a week or more (De Giorgi 2011), often supplemented with web‐based learning or using textbooks or CD‐ROMs (Carli 2003a; Menzies 2009; Tan 2009). The most effective means of training health professionals in dermoscopy remains to be established. Evidence from Australia suggests that it takes time to train non‐expert clinicians in the use of dermoscopy, and dropout rates from training programmes may be up to 40% (Menzies 2009).

Clinical pathway

The diagnosis of melanoma can take place in primary, secondary, and tertiary care settings by both generalist and specialist healthcare providers. In the UK, people with concerns about a new or changing lesion will usually present first to their GP or less commonly, directly to a specialist in secondary care, which could include a dermatologist, plastic surgeon, general surgeon or other specialist surgeon (such as an ear, nose, and throat (ENT) specialist or maxillofacial surgeon), or ophthalmologist (Figure 2). Current UK guidelines recommend that all suspicious pigmented lesions presenting in primary care should be assessed by taking a clinical history and visual inspection using the weighted seven‐point checklist (MacKie 1990); lesions suspected to be melanoma should be referred for appropriate specialist assessment within two weeks (Chao 2013; Marsden 2010; NICE 2015a). There are currently no recommendations promoting the use of dermoscopy in primary care in the UK, although the 2015 NICE suspected cancer recognition and referral guidelines state that people should be referred “using a suspected cancer pathway referral (for an appointment within 2 weeks), if dermoscopy suggests melanoma of the skin” (NICE 2015a). Studies from France (Chappuis 2016), and the Netherlands (Ahmadi 2017), suggest that around 8% of GPs use dermoscopy, compared to as many as 40% of GPs in Australia reported to use a dermoscope in their routine practice (Youl 2007a).

2.

Current clinical pathway for people with skin lesions.

Theoretically, teledermatology consultations may aid appropriate triage of lesions into urgent referral; non‐urgent secondary care referral (e.g. for suspected basal cell carcinoma); or where available, referral to an intermediate care setting, such as clinics run by GPs with a special interest in dermatology. The distinction between setting and examiner qualifications and experience is important, as specialist clinicians might work in primary care settings (e.g. in the UK, GPs with a special interest in dermatology and skin surgery who have undergone appropriate training), and generalists might practice in secondary care settings (e.g. plastic surgeons who do not specialise in skin cancer). The level of skill and experience in skin cancer diagnosis will vary for both generalist and specialist care providers and will also have an impact on test accuracy.

Following referral, a specialist clinician will also use history‐taking and visual inspection of the lesion (in comparison with other lesions on the skin), usually in conjunction with dermoscopic examination, to inform a clinical decision. If melanoma is suspected, then urgent excision biopsy is recommended; for suspected cutaneous squamous cell carcinoma (cSCC), urgent excision with predetermined surgical margins. Other lesions such as basal cell carcinoma (BCC), suspected dysplastic naevi or pre‐malignant lesions such as lentigo maligna may also be referred for a diagnostic biopsy, followed by appropriate treatment, further surveillance or reassurance and discharge.

Prior test(s)

Although smartphone applications and community‐based teledermatology services can increasingly be directly accessed by people who have concerns about a skin lesion, visual inspection of a suspicious lesion by a clinician is usually the first in a series of tests to diagnose skin cancer. In the UK this usually takes place in primary care setting, however in some countries people with suspicious lesions can present directly to a specialist setting (NICE 2015b). Dermoscopy is likely to be added to visual inspection of a lesion in secondary care and referral settings, however, it is increasingly used in primary care, particularly in countries such as Australia (Youl 2007a).

Consideration of the degree of prior testing that study participants have undergone is key to interpretation of test accuracy indices, as these are known to vary according to the disease spectrum (or case‐mix), of included participants (Lachs 1992; Moons 1997; Leeflang 2013; Usher‐Smith 2016). Spectrum effects are often observed when tests that are developed further down the referral pathway have lower sensitivity and higher specificity when applied in settings with participants with limited prior testing (Usher‐Smith 2016). Studies of individuals with suspicious lesions at the initial clinical presentation stage ('test naïve'), are likely to have a wider range of differential diagnoses and include a higher proportion of people with benign diagnoses compared with studies of participants who have been referred for a specialist opinion on the basis of visual inspection (with or without dermoscopy), by a generalist practitioner. Furthermore, studies in more specialist settings may focus on equivocal or difficult‐to‐diagnose lesions rather than lesions with a more general level of clinical suspicion. However this direction of effect is not consistent across tests and diseases, the mechanisms in action often being more complex than prevalence alone and can be difficult to identify (Leeflang 2013). A simple categorisation of studies according to primary, secondary or specialist setting may not always adequately reflect this difference in disease spectrum.

Role of index test(s)

Although visual inspection and history‐taking are key to diagnosing skin cancer and are always undertaken as part of a clinical examination, dermoscopy has become an important tool to assist diagnosis by specialist clinicians and is increasingly used in primary care settings. For the majority of generalist practitioners, the primary goal is to identify people with benign lesions and appropriately reassure them, thereby minimising the proportion of people who are referred unnecessarily, while still identifying those lesions that require referral and expert assessment. For the specialist, the aim is not only to identify those in need of urgent excision due to invasive cancer, but also to identify high‐risk lesions with considerable potential to progress to invasive disease, such as those with severe dysplasia or in situ disease, such as lentigo maligna, for example.

When diagnosing potentially life‐threatening conditions such as melanoma, the consequences of falsely reassuring a person that they do not have skin cancer can be serious and potentially fatal, as the resulting delay to diagnosis means that the window for successful early treatment may be missed. To minimise such false‐negative diagnoses, a good diagnostic test will demonstrate high sensitivity and a high negative predictive value (NPV), where very few of those with a negative test result will actually have a melanoma. False‐positive test results from a test with poor specificity will result in the removal of many benign lesions. Unneccessary surgery is arguably less of an error than missing a potentially fatal melanoma, but is costly: false‐positive diagnoses not only cause unnecessary scarring from the biopsy or excision procedure, but also increase patient anxiety whilst they await the definite histology results and increase healthcare costs as the number needed to remove to yield one melanoma diagnosis increases.

The additive value of dermoscopy over and above visual inspection alone is likely to vary with differences in setting, prior testing and selection of participants, and observer qualifications, experience and training. Furthermore dermoscopic images of lesions are increasingly taken by non‐expert clinicians or by non‐clinicians, sometimes using mobile phone applications, and are forwarded to specialist clinics or to commercial organisations for interpretation, sometimes accompanied by a clinical image of the lesion with varying amounts of patient information (such as age, gender, and location of the lesion). With skin cancer rates continuing to rise, the increasing availability of dermoscopy for generalist use, and with a growing number of other high‐resolution image‐analysis techniques, particularly for specialist use, it is important to understand the relative accuracy and appropriate place of available tests in the diagnostic pathway (whether as replacements for dermoscopy, or as add‐on diagnostic tools).

Although this review examines the accuracy of image‐based dermoscopy interpretation, studies conducted specifically in a teledermatology context are the subject of a separate systematic review (Chuchu 2018a). Similarly, studies of mobile phone applications, where the intended users are members of the general public rather than clinicians are the subject of another review (Chuchu 2018b).

Alternative test(s)

As part of our series of systematic reviews, we have reviewed a number of other tests that may have a role in the diagnosis of melanoma in a specialist setting, including reflectance confocal microscopy (RCM) (Dinnes 2018b), optical coherence tomography (OCT) (Ferrante di Ruffano 2018a), and computer‐aided diagnosis (CAD) techniques applied to various types of images, including those generated by dermoscopy, diffuse reflectance spectrophotometry (DRS) and electrical impedance spectroscopy (EIS) (Ferrante di Ruffano 2018b), and high‐frequency ultrasound (Dinnes 2018c). Other tests reviewed include teledermatology (Chuchu 2018a), and mobile phone applications (Chuchu 2018b). Evidence permitting, we plan to compare the accuracy of available tests in an overview review, exploiting within‐study comparisons of tests and allowing the analysis and comparison of commonly used diagnostic strategies where tests may be used singly or in combination.

We also considered and excluded a number of tests from this review, such as tests used for monitoring people (e.g. total body photography of those with large numbers of typical or atypical naevi). We also did not assess histopathological confirmation following lesion excision because it is the established reference standard for melanoma diagnosis and one of the standards against which the index tests are evaluated in these reviews.

Rationale

This series of reviews of diagnostic tests used to assist clinical diagnosis in either clinical practice or in a research setting, aims to identify the most accurate approaches to diagnosis and provide clinical and policy decision‐makers with the highest possible standard of evidence on which to base diagnostic and treatment decisions. With increasing rates of melanoma and a trend to adopt the use of dermoscopy and other high‐resolution image analysis in primary care, the anxiety around missing early cases needs to be balanced against the risk of over referrals, to avoid sending too many people with benign lesions for a specialist opinion. It is questionable whether all skin cancers identified by sophisticated techniques contribute to morbidity and mortality or whether newer technologies run the risk of increasing false‐positive diagnoses. The full impact of use of these technologies cannot be understood without an understanding of the accuracy of more established techniques such as dermoscopy, in comparison to visual inspection. It is also possible that widespread use of dermoscopy in primary care with inadequate training could result in harm from missed melanomas, particularly if used as a replacement for traditional history‐taking and clinical examination of the entire skin. Many branches of medicine have noted the danger of such "gizmo idolatry" amongst doctors (Leff 2008). The trend towards remote interpretation of clinical images (whether macroscopic or dermoscopic images of lesions), and the use of remote technologies that do not involve clinicians without substantive evidence could further disrupt clinical pathways and healthcare payments as they may attract custom from the worried well, leaving an ever decreasing pool of qualified doctors to pick up any resulting problems.

There are a number of available systematic reviews in the field. Some are limited by now out‐of‐date search periods, for example searches in Rajpara 2009 were carried out up to 2007, and in Vestergaard 2008 up to 2008. Others are focused on specific clinical questions, for example, selected healthcare professionals (Corbo 2012 including only direct comparisons of the accuracy of primary care physicians versus dermatologists, and Loescher 2011 reviewing the skin cancer detection skills of advanced practice nurses), or settings (Herschorn 2012 including direct comparisons of visual inspection versus dermoscopy in primary care). More recently, Harrington and colleagues (Harrington 2017), published a systematic review of clinical prediction rules (or published algorithms), to assist the diagnosis of melanoma (both for clinical examination and for dermoscopy), and included studies published up to May 2015. This review did not consider whether diagnoses were made based on images or were conducted in person, nor did it consider variations in the definition of the target condition, and furthermore it did not compare diagnosis with and without the use of an algorithm.

The critical question about the accuracy of dermoscopy in addition to visual inspection and the impact of examiner, prior patient testing, underlying risk status and the use of images for diagnosis needs to be answered before the potential contribution of other diagnostic tests can be set in context and appropriately placed in the diagnostic pathway.

This review follows a generic protocol that covers the full series of Cochrane DTA Reviews for the diagnosis of melanoma (Dinnes 2015a). The Background and Methods sections of this review therefore use some text that was originally published in the protocol (Dinnes 2015a), and text that overlaps some of our other reviews (Dinnes 2018a; Dinnes 2018b). Appendix 3 provides a glossary of terms used.

Objectives

To determine the diagnostic accuracy of dermoscopy alone, or when added to visual inspection of a skin lesion, for the detection of cutaneous invasive melanoma and atypical intraepidermal melanocytic variants in adults.

Accuracy was estimated separately, according to the prior testing undergone by study participants, comparing those with limited prior testing with those referred for further evaluation of a suspicious lesion. We originally aimed to estimate the effect on accuracy of diagnosis based on a face‐to‐face (in‐person), encounter versus a remote (image‐based), assessment as a secondary objective, however given the considerable difference in nature of an in‐person consultation compared to the viewing of an image, we estimated accuracy separately for each approach to diagnosis. We therefore aimed to compare tests in the following way:

To estimate incremental accuracy for the diagnosis of invasive melanoma and atypical intraepidermal melanocytic variants in adults, a), from dermoscopy added to in‐person visual inspection of a skin lesion, or b), from dermoscopic image‐based assessment in comparison to visual inspection of a clinical photograph.

Secondary objectives

For the identification of cutaneous invasive melanoma and atypical intraepidermal melanocytic variants:

• to compare the accuracy of dermoscopy to visual inspection alone, where both tests have been evaluated in the same studies (direct test comparisons);

• to determine the diagnostic accuracy of individual algorithms used to assist dermoscopy;

• to determine the effect of observer experience on diagnostic accuracy;

• to determine the effect of dermoscopy training on diagnostic accuracy

For the alternative definitions of the target condition:

• to determine the diagnostic accuracy of dermoscopy alone, or added to visual inspection of a skin lesion, for the detection of invasive melanoma only in adults, and to estimate incremental accuracy a), from dermoscopy added to in‐person visual inspection of a skin lesion, or b), from dermoscopic image‐based assessment in comparison to visual inspection of a clinical photograph;

• to determine the diagnostic accuracy of dermoscopy alone, or added to visual inspection of a skin lesion, for the detection of any skin cancer or skin lesion with a high risk of progression to melanoma in adults, and to estimate incremental accuracy a), from dermoscopy added to in‐person visual inspection of a skin lesion, or b), from dermoscopic image‐based assessment in comparison to visual inspection of a clinical photograph.

Investigation of sources of heterogeneity

We set out to address a range of potential sources of heterogeneity for investigation across our series of reviews, as outlined in our generic protocol (Dinnes 2015a), and described in Appendix 4, however our ability to investigate these was necessarily limited by the available data on each individual test reviewed.

The sources of heterogeneity that we investigated for dermoscopy were:

• prior testing: comparing those at initial presentation versus referred patients

• in‐person versus image‐based evaluations

• type of reference standard: histology alone versus histology + clinical follow‐up or other reference standard

• use of a diagnostic algorithm: no algorithm reported versus any named algorithm used

• lesion type: pigmented versus melanocytic lesions

• number of observers making the diagnosis: single observer versus consensus of two or more

• disease prevalence: 0% to 5%; 5% to 10%, 10% to 20%, more than 20%

Methods

Criteria for considering studies for this review

Types of studies

We included test accuracy studies that allowed comparison of the result of the index test with that of a reference standard, including the following:

• studies where all participants received a single index test and a reference standard;

• studies where all participants received more than one index test(s) and reference standard;

• studies where participants were allocated (by any method), to receive different index tests or combinations of index tests and all received a reference standard (between‐person comparative studies (BPC));

• studies that recruited series of participants unselected by true disease status (referred to as case series for the purposes of this review);

• diagnostic case‐control studies that separately recruited diseased and non‐diseased groups (see Rutjes 2005);

• both prospective and retrospective studies; and

• studies where previously acquired clinical or dermoscopic images were retrieved and prospectively interpreted for study purposes.

We excluded studies from which we could not extract 2x2 contingency data or if they included fewer than five melanoma cases or fewer than five benign lesions. The size threshold of five is arbitrary. However such small studies are unlikely to add precision to estimate of accuracy.

Studies available only as conference abstracts were excluded; however, attempts were made to identify full papers for potentially relevant conference abstracts (Searching other resources).

Participants

We included studies in adults with pigmented skin lesions or lesions suspicious for melanoma or those at high risk of developing melanoma, including those with a family history or previous history of melanoma skin cancer, atypical or dysplastic naevus syndrome, or genetic cancer syndromes.

We excluded studies that recruited only participants with malignant diagnoses and studies that compared test results in participants with malignancy compared with test results based on 'normal' skin as controls, due to the inherent bias in such comparisons (Rutjes 2006).

We excluded studies conducted in children or that clearly reported inclusion of more than 50% of participants aged 16 and under.

Index tests

Studies reporting accuracy data for dermoscopy, with diagnosis made either in person (face‐to‐face diagnosis), or image‐based (diagnosis based on dermoscopic images, remotely from the study participant), were eligible for inclusion. We included all established algorithms or checklists to assist diagnosis.

We included studies developing new algorithms or methods of diagnosis (i.e. derivation studies), if they:

• used a separate independent 'test set' of participants or images to evaluate the new approach; or

• investigated lesion characteristics that had previously been suggested as associated with melanoma and the study reported accuracy based on the presence or absence of specific combinations of characteristics.

We excluded studies if they:

• used a statistical model to produce a data‐driven equation, or algorithm based on multiple diagnostic features, with no separate test set;

• used cross‐validation approaches such as 'leave‐one‐out' cross‐validation (Efron 1983);

• evaluated the accuracy of the presence or absence of individual lesion characteristics or morphological features, with no overall diagnosis of malignancy;

• reported accuracy data for ‘clinical diagnosis’ with no clear description as to whether the reported data related to visual inspection alone or included dermoscopy in all study participants;

• were based on the experience of a skin cancer‐specific clinic, where dermoscopy may or may not have been used on an individual participant basis.

Although primary care clinicians can have a specialist interest in skin cancer, for the purposes of this review we considered primary care physicians as generalist practitioners and dermatologists as specialists. Within each group, we extracted any reporting of special interest or accreditation in skin cancer.

Target conditions

We defined the primary target condition as the detection of:

• any form of invasive cutaneous melanoma, or atypical intraepidermal melanocytic variants (i.e. including melanoma in situ, or lentigo maligna, which has a risk of progression to invasive melanoma).

We considered two additional definitions of the target condition in secondary analyses, namely the detection of:

• any form of invasive cutaneous melanoma alone;

• any skin lesion requiring excision. This latter definition includes other forms of skin cancer, such as basal cell carcinoma (BCC) and cutaneous squamous cell carcinoma (cSCC), as well as melanoma in situ, lentigo maligna, and lesions with severe melanocytic dysplasia.

The diagnosis of the keratinocyte skin cancers, basal cell carcinoma, and squamous cell carcinoma as primary target conditions using visual inspection and/or dermoscopy are the subject of a separate review (Dinnes 2018d).

Reference standards

The ideal reference standard is histopathological diagnosis in all eligible lesions. A qualified pathologist or dermatopathologist should perform histopathology. Ideally, reporting should be standardised, detailing a minimum dataset to include the histopathological features of melanoma to determine the American Joint Committee on Cancer (AJCC) Staging System (e.g. Slater 2014). We did not apply this as a necessary inclusion criterion, but extracted any pertinent information.

Partial verification (applying the reference test only to a subset of those undergoing the index test), was of concern given that lesion excision or biopsy are unlikely to be carried out for all benign‐appearing lesions within a representative population sample. Therefore to reflect what happens in reality, we accepted clinical follow‐up of benign‐appearing lesions as an eligible reference standard, whilst recognising the risk of differential verification bias (as misclassification rates of histopathology and follow‐up will differ).

Additional eligible reference standards included cancer registry follow‐up and 'expert opinion' with no histology or clinical follow‐up. Cancer registry follow‐up is considered less desirable than active clinical follow‐up, as follow‐up is not carried out within the control of the study investigators. Furthermore, if participant‐based analyses as opposed to lesion‐based analyses are presented, it may be difficult to determine whether the detection of a malignant lesion during follow‐up is the same lesion that originally tested negative on the index test.

All of the above were considered eligible reference standards with the following caveats:

• all study participants with a final diagnosis of the target disorder must have had a histological diagnosis, either subsequent to the application of the index test or after a period of clinical follow‐up; and

• at least 50% of all participants with benign lesions must have had either a histological diagnosis or clinical follow‐up to confirm benignity.

Search methods for identification of studies

Electronic searches

The Information Specialist (SB) carried out a comprehensive search for published and unpublished studies. A single large literature search was conducted to cover all topics in the programme grant (see Appendix 1 for a summary of reviews included in the programme grant). This allowed for the screening of search results for potentially relevant papers for all reviews at the same time. A search combining disease related terms with terms related to the test names, using both text words and subject headings was formulated. The search strategy was designed to capture studies evaluating tests for the diagnosis or staging of skin cancer. As the majority of records were related to the searches for tests for staging of disease, a filter using terms related to cancer staging and to accuracy indices was applied to the staging test search, to try to eliminate irrelevant studies, for example, those using imaging tests to assess treatment effectiveness. A sample of 300 records that would be missed by applying this filter was screened and the filter adjusted to include potentially relevant studies. When piloted on MEDLINE, inclusion of the filter for the staging tests reduced the overall numbers by around 6000. The final search strategy, incorporating the filter, was subsequently applied to all bibliographic databases as listed below (Appendix 5). The final search result was cross‐checked against the list of studies included in five systematic reviews; our search identified all but one of the studies, and this study was not indexed on MEDLINE. The Information Specialist (SB) devised the search strategy, with input from the Information Specialist from Cochrane Skin. No additional limits were used.

We searched the following bibliographic databases to 29 August 2016 for relevant published studies:

• MEDLINE via OVID (from 1946);

• MEDLINE In‐Process & Other Non‐Indexed Citations via OVID; and

• Embase via OVID (from 1980).

We searched the following bibliographic databases to 30 August 2016 for relevant published studies:

• the Cochrane Central Register of Controlled Trials (CENTRAL; 2016, Issue 7), in the Cochrane Library;

• the Cochrane Database of Systematic Reviews (CDSR; 2016, Issue 8), in the Cochrane Library;

• Cochrane Database of Abstracts of Reviews of Effects (DARE; 2015, Issue 2);

• CRD HTA (Health Technology Assessment), database 2016, Issue 3; and

• CINAHL (Cumulative Index to Nursing and Allied Health Literature via EBSCO from 1960).

We searched the following databases for relevant unpublished studies using a strategy based on the MEDLINE search:

• CPCI (Conference Proceedings Citation Index), via Web of Science™ (from 1990; searched 28 August 2016); and

• SCI Science Citation Index Expanded™ via Web of Science™ (from 1900, using the 'Proceedings and Meetings Abstracts' Limit function; searched 29 August 2016).

We searched the following trials registers using the search terms 'melanoma', 'squamous cell', 'basal cell' and 'skin cancer' combined with 'diagnosis':

• Zetoc (from 1993; searched 28 August 2016).

• The US National Institutes of Health Ongoing Trials Register (www.clinicaltrials.gov); searched 29 August 2016.

• NIHR Clinical Research Network Portfolio Database (www.nihr.ac.uk/research‐and‐impact/nihr‐clinical‐research‐network‐portfolio/); searched 29 August 2016.

• The World Health Organization International Clinical Trials Registry Platform (apps.who.int/trialsearch/); searched 29 August 2016.

We aimed to identify all relevant studies regardless of language or publication status (published, unpublished, in press, or in progress). We applied no date limits.

Searching other resources

We screened relevant systematic reviews identified by the searches for their included primary studies, and included any missed by our searches. We checked the reference lists of all included papers, and subject experts within the author team reviewed the final list of included studies. No electronic citation searching was conducted.

Data collection and analysis

Selection of studies

At least one review author (JDi or NC), screened titles and abstracts, with any queries discussed and resolved by consensus. A pilot screen of 539 MEDLINE references showed good agreement (89% with a kappa of 0.77), between screeners. We included at initial screening primary test accuracy studies and test accuracy reviews (for scanning of reference lists), of any test used to investigate suspected melanoma, BCC, or cSCC. Both a clinical reviewer (from one of a team of 12 clinician reviewers), and a methodologist reviewer (JDi or NC), applied inclusion criteria independently to all full‐text articles (Appendix 6). We resolved disagreements by consensus or by consulting a third party (JDe, CD, HW, and RM). We contacted authors of eligible studies when insufficient data were presented to allow for the construction of 2x2 contingency tables.

Data extraction and management

One clinical (as detailed above), and one methodologist reviewer (JDi, NC or LFR), independently extracted data concerning details of the study design, participants, index test(s) or test combinations and criteria for index test positivity, reference standards, and data required to populate a 2x2 diagnostic contingency table for each index test using a piloted data extraction form. We extracted data at all available index test thresholds. We resolved disagreements by consensus or by involving a third party (JDe, CD, HW, and RM).

Where information related to final lesion diagnoses or diagnostic thresholds were missing, we contacted authors of included studies. In particular, invasive cSCC (included as disease‐positive for one of our secondary objectives), is not always differentiated from 'in situ' variants such as Bowens disease (which we did not consider as disease‐positive for any of our definitions of the target condition). We contacted authors of conference abstracts published from 2013 to 2015 to ask whether full data were available. If no full paper was identified, we marked conference abstracts as 'pending' and will revisit them in a future review update.

Dealing with multiple publications and companion papers

Where we identified multiple reports of a primary study, we maximised yield of information by collating all available data. Where there were inconsistencies in reporting or overlapping study populations, we contacted study authors for clarification in the first instance. If this contact with study authors was unsuccessful, we used the most complete and up‐to‐date data source where possible.

Assessment of methodological quality

We assessed risk of bias and applicability of included studies using the QUADAS‐2 checklist (Whiting 2011), tailored to the review topic (see Appendix 7). We piloted the modified QUADAS‐2 tool on a small number of included full‐text articles. One clinical (as detailed above), and one methodologist reviewer (JDi, NC or LFR), independently assessed quality for the remaining studies; we resolved any disagreement by consensus or by involving a third party where necessary (JDe, CD, HW, and RM).

Statistical analysis and data synthesis

We planned separate analyses according to the point that study participants had reached in the clinical pathway (numbered from 1 to 7 in Figure 3), the clarity with which the pathway could be determined (clear or unclear), and the evaluation of in‐person versus image‐based diagnosis.

3.

Clinical pathway

Our unit of analysis was the lesion rather than the participant. This is because firstly, in skin cancer initial treatment is directed to the lesion rather than systemically (thus it is important to be able to correctly identify cancerous lesions for each person), and secondly, it is the most common way in which the primary studies reported data. Although there is a theoretical possibility of correlations of test errors when the same people contribute data for multiple lesions, most studies included very few people with multiple lesions and any potential impact on findings was likely to be very small, particularly in comparison with other concerns regarding risk of bias and applicability. For each analysis, we included only one dataset per study to avoid multiple counting of lesions. Where an individual study assessed multiple algorithms, we selected datasets on the following preferential basis:

• ‘no algorithm’ reported; data presented for clinician’s overall diagnosis or management decision;

• pattern analysis or pattern recognition;

• ABCD algorithm (or derivatives of);

• seven‐point checklist (7PCL; also referred to as Glasgow/Mackie checklist);

• Menzies algorithm;

• three‐point checklist (3PCL).

For each index test, algorithm or checklist under consideration, we plotted estimates of sensitivity and specificity on coupled forest plots and in receiver operating characteristic (ROC) space. For tests where commonly used thresholds were reported we estimated summary operating points (summary sensitivities and specificities), with 95% confidence and prediction regions using the bivariate hierarchical model (Chu 2006; Reitsma 2005). Where inadequate data were available for the model to converge, we simplified it, first by assuming no correlation between estimates of sensitivity and specificity and secondly by setting estimates of near zero variance terms to zero (Takwoingi 2015). Where all studies reported 100% sensitivity (or 100% specificity), we summed the number with disease (or no disease), across studies and used it to compute a binomial exact 95% confidence interval. Where missing or indeterminate results were reported, study authors usually did not provide sufficient details to allow us to include these data in our analyses. Where study authors reported missing or indeterminate results in more detail, these results were excluded by us for consistency.

We included data on the accuracy of visual inspection, to allow comparisons of tests, only if reported in the studies of dermoscopy, due to the known substantial unexplained heterogeneity in all studies of the accuracy of visual inspection (Dinnes 2018a). We made comparisons between visual inspection results with dermoscopy data from all dermoscopy studies, and then only using dermoscopy data from studies that also reported visual inspection data for the same participants, to enable a robust direct comparison (Takwoingi 2013).

We made comparisons between tests by comparing summary ROC curves using the hierarchical summary receiver‐operator curves (HSROC) model (Rutter 2001), rather than by estimating average operating points, as this approach allows incorporation of data at different thresholds and from different algorithms or checklists. We used a HSROC model that assumed a constant SROC shape between tests and subgroups (allowing for asymmetry in shape), and modelled differences in threshold and accuracy by addition of covariates. We assessed the significance of the differences between tests by the likelihood ratio test (LR test), assessing differences in both accuracy and threshold. We fitted simpler models when convergence was not achieved due to small numbers of studies, first assuming symmetric SROC curves (setting the shape term to zero), and then setting random effects variance estimates to zero.

We have presented estimates of accuracy from HSROC models as diagnostic odds ratios (estimated where the SROC curve crosses the sensitivity=specificity line), with 95% confidence intervals. We have presented differences between tests and subgroups from HSROC analyses as relative diagnostic odds ratios with 95% confidence intervals. To facilitate interpretation in terms of rates of false‐positive and false‐negative diagnoses, we computed values of sensitivity at the point on the SROC curve with 80% specificity and of specificity at the point on the SROC curve with 80% sensitivity, choosing these 80% values because they lie within the estimates for the majority of analyses. These results should only be considered as illustrative examples of possible sensitivities (and specificities), and differences in sensitivities (and specificities), that could be expected. We computed confidence intervals for these estimates of sensitivity and specificity assuming normal distribution of sampling error on logit scales; and computed confidence intervals for differences in sensitivity and specificity assuming normal distributions of sampling error on untransformed scales.

For computation of likely numbers of true‐positive, false‐positive, false‐negative and true‐negative findings in the 'Summary of findings' tables, we applied these indicative values to lower, median and upper quartiles of the prevalence observed in the study groups.

We fitted bivariate models using the 'xtmelogit' command in STATA 15 and HSROC models using the 'NLMIXED' procedure in the SAS statistical software package (SAS 2012), and the metadas macro (Takwoingi 2010).

Investigations of heterogeneity

We also investigated heterogeneity, and made comparisons between algorithms and according to observer experience and qualifications by comparing summary ROC curves using the HSROC model (Rutter 2001), with additional covariates for differences in threshold and accuracy as used for comparing tests. We omitted small subgroups from models where parameter estimates could not be obtained due to convergence problems.

Sensitivity analyses

We planned sensitivity analyses, restricting analyses to studies where:

• the same study evaluated both dermoscopy (added to visual inspection), and visual inspection alone (direct test comparisons as discussed above);

• partial verification was avoided (restricting to studies including follow‐up of benign lesions);

• for studies using follow‐up of benign‐appearing lesions, the interval between the index test and the reference standard was at least three months;

• for direct test comparisons, the period of application between the index tests was within one month;

• concerns around applicability for participant selection were low;

• there was low risk of bias for the index test;

• there was low risk of bias for the reference standard.

Assessment of reporting bias

Because of uncertainty about the determinants of publication bias for diagnostic accuracy studies and the inadequacy of tests for detecting funnel plot asymmetry (Deeks 2005), we did not perform tests to detect publication bias.

Results

Results of the search

We identified and screened a total of 34,517 unique references for inclusion. Of these, we reviewed 1051 full‐text papers for eligibility for any one of the suite of reviews of tests to assist in the diagnosis of melanoma or keratinocyte skin cancer. Of the 1051 full‐text papers assessed, we excluded 848 from all the reviews in our series (see Figure 4, PRISMA flow diagram of search and eligibility results; Moher 2009).

4.

PRISMA flow diagram.

Of the 340 studies tagged as potentially eligible for this review of dermoscopy, we included 104 publications. Exclusions were mainly due to the inability to construct a 2x2 contingency table based on the data presented (n = 43); the use of ineligible index tests (n = 19) (for example, reporting of data for ‘clinical diagnosis’ or for serial use of the index test in a follow‐up context); assessment of individual lesion characteristics (n = 28); or derivation type studies, developing new algorithms or checklists without a separate training and test set of lesions (n = 31). Other reasons for exclusion included not meeting our requirements for an eligible reference standard (n = 15), ineligible study populations (n = 21), (for example, recruiting only malignant or only benign lesions), inadequate sample size (n = 22), ineligible definition of the target condition (n = 18), or with test interpretation by medical students or laypersons (n = 3). A list of the 236 publications excluded from this review with reasons for exclusion is provided in Characteristics of excluded studies, with a list of all studies excluded from the full series of reviews available as a separate PDF (please contact skin.cochrane.org for a copy of the pdf).

We contacted the authors of 17 publications for further data to allow study inclusion in the review and received responses from four authors with regard to seven publications. Two authors provided additional data but these were insufficient to allow inclusion of the studies (Cabrijan 2008; Warshaw 2009a; Warshaw 2009b; Warshaw 2010), one replied indicating that dermoscopy was not necessarily used in all study participants (Youl 2007a; Youl 2007b), and one replied but was unable to access the data needed (Fabbrocini 2008). We contacted the authors of a further 20 included studies for further details of study methods and received responses in regard to 10 studies, eight providing further information regarding the diagnostic thresholds used (Blum 2003a; Blum 2004a; Bono 2006; Bourne 2012; Carrera 2016; Durdu 2011; Kittler 1999; Stanganelli 2000), one providing full anonymised study data (Rosendahl 2011), and one unable to provide the information requested, although the study could still be included (Menzies 2009).

Of the 104 included study publications, two provide data for two separate cohorts of lesions: Guitera et al reports data for one cohort of lesions recruited in Modena, Italy (denoted Guitera 2009a (Modena)), and one cohort recruited in Sydney (denoted Guitera 2009b (Sydney)); Haenssle 2010 reports data for one cohort of lesions examined on participants' first visit (denoted Haenssle 2010a (FV)), and one cohort of lesions identified during participant follow‐up (denoted Haenssle 2010b (FU)). Four different publications report data on one further cohort; we included data from one publication (Blum 2004b), in the primary analyses, with data from Blum 2003a, Blum 2003b and Blum 2004a providing results for different algorithms or thresholds for the same set of lesions. The total number of cohorts of lesions described in the 104 study publications is therefore 103 (104 + 2 minus 3). The 104 study publications provided a total of 354 dermoscopy datasets (each publication often providing more than one 2x2 contingency table according to the use of different algorithms, different test thresholds or different observers), for 42,788 lesions and 5700 malignancies. The total number of study participants with suspicious lesions cannot be estimated due to lack of reporting in study publications (reported in only 44 studies with 9591 participants). A third of study publications (n = 31; 30%), also reported accuracy data for diagnosis using visual inspection; these provided 61 datasets for 9025 lesions and 959 malignancies. A systematic review of the accuracy of visual inspection per se is reported in Dinnes 2018a. A further 29 of the 104 included study publications reported data for tests other than dermoscopy or visual inspection including: teledermatology (n = 3), reflectance confocal microscopy (RCM), (n = 7), exfoliative cytology (n = 1), and computer‐assisted diagnosis (CAD) techniques (n = 18).

Methodological quality of included studies

We have summarised the overall methodological quality of all included studies (regardless of target condition), according to in‐person or image‐based approaches to dermoscopy or to visual inspection. Figure 5 shows a total of 35 in‐person evaluations, with results per study presented in Figure 6; and Figure 7 shows a total of 74 image‐based evaluations of dermoscopy, with results per study presented in Figure 8. The total number of entries in Figure 6 and Figure 8 sums to 109 (35 + 74), instead of 103 (as per the number of included cohorts), for the following reasons: a), three publications (Carli 2002a; Dummer 1993; Unlu 2014), reported both in‐person and image‐based data and therefore appear in both the in‐person and image‐based plots (making 106 entries), and b), one cohort was reported on in four papers (Blum 2003a; Blum 2003b; Blum 2004a; Blum 2004b), which all contributed data to the review analyses and were therefore quality assessed four times (making 109 entries).

5.

Risk of bias and applicability concerns graph for in‐person evaluations: review authors' judgements about each domain presented as percentages across included studies

6.

Risk of bias and applicability concerns for in‐person evaluations summary: review authors' judgements about each domain for each included study

7.

Risk of bias and applicability concerns graph for image‐based evaluations: review authors' judgements about each domain presented as percentages across included studies

8.

Risk of bias and applicability concerns for image‐based evaluations summary: review authors' judgements about each domain for each included study

In‐person evaluations

We judged risk of bias to be low for the majority of studies in only two of five quality domains that we assessed (dermoscopy index test, reference standard); the majority of studies were at high or unclear risk of bias for the remaining three domains (participant selection, visual inspection index test flow and timing; Figure 5). We scored applicability of study findings as of high or unclear concern in all four domains that we assessed (participant selection, dermoscopy and visual inspection index tests, reference standards).

For participant selection, we judged 11 studies (31%), at low risk of bias (Carli 1994; Dreiseitl 2009; Duff 2001; Grimaldi 2009; Langley 2007; Menzies 2009; Morales Callaghan 2008; Nachbar 1994; Soyer 2004; Stanganelli 2000; Unlu 2014); and we considered eight (23%), at high risk (Figure 6), due to exclusion of lesions by size (Bono 2002b; Bono 2006; Kittler 1999), or type (Ahnlide 2016; Cristofolini 1994; Guitera 2009a (Modena); Haenssle 2010a (FV); Haenssle 2010b (FU)). The study by Haenssle and colleagues excluded participants showing melanoma development on pre‐existing pigmented lesions during the 12 months after the analysed time frame. Twelve studies (34%), did not report the method of participant selection and 15 (43%), did not clearly describe exclusions from the study. We considered almost all cohorts (91%; n = 32), at high concern for applicability of participants. In the majority of cases (n = 30), this was due to restricted study populations, such as inclusion of only melanocytic lesions (n = 10), or inclusion of lesions selected for excision based on the clinical or dermoscopic diagnosis (n = 28). We judged only four cohorts (11%), to have included a representative patient population (Dreiseitl 2009; Grimaldi 2009; Menzies 2009; Stanganelli 2000). Eight cohorts (23%), also included multiple lesions per participant (Durdu 2011; Gokdemir 2011; Grimaldi 2009; Haenssle 2010b (FU); Haenssle 2010a (FV); Kittler 1999; Morales Callaghan 2008; Stanganelli 2000), and 12 others (34%), did not clearly report number of included participants.

For the index test domain, there are 33 evaluations of in‐person dermoscopy and 16 evaluations of in‐person visual inspection (Figure 5). For dermoscopy, we considered 24 evaluations (73%), at low risk of bias (Ahnlide 2016; Argenziano 2006; Ascierto 2010; Bauer 2000; Benelli 1999; Bono 2002a; Bono 2002b; Bono 2006; Broganelli 2005; Carli 1994; Carli 2002a; Coras 2003; Cristofolini 1994; Durdu 2011; Feldmann 1998; Grimaldi 2009; Guitera 2009a (Modena); Haenssle 2010a (FV); Haenssle 2010b (FU); Langley 2007; Menzies 2009; Morales Callaghan 2008; Stanganelli 2000; Viglizzo 2004), and we judged two evaluations (6%), at high risk (Kittler 1999; Nachbar 1994); seven studies (21%), did not provide sufficient information to allow us to make a full 'Risk of bias' judgement. We judged all studies to have made the diagnosis blinded to the reference standard result, given that this is always undertaken prior to histology; 25 (76%), also clearly reported pre‐specification of the diagnostic threshold, 20 using named algorithms or pattern analysis (Ahnlide 2016; Argenziano 2006; Ascierto 2010; Benelli 1999; Bono 2006; Broganelli 2005; Carli 1994; Carli 2002a; Coras 2003; Cristofolini 1994; Durdu 2011; Feldmann 1998; Grimaldi 2009; Guitera 2009a (Modena); Haenssle 2010a (FV); Haenssle 2010b (FU); Langley 2007; Morales Callaghan 2008; Soyer 1995; Stanganelli 2000), and five (15%), describing the process by which they reached their diagnosis (Bauer 2000; Bono 2002a; Bono 2002b; Menzies 2009; Viglizzo 2004). Two studies (6%), developed new algorithms (Nachbar 1994), or evaluated multiple thresholds for test positivity (Kittler 1999).

We considered that all 16 visual inspection evaluations also made the diagnosis blinded to the reference standard result. One (6%), was at high risk of bias due to evaluation of several different ABCDE algorithm thresholds (Benelli 1999), and we judged nine studies (56%), unclear as to the diagnostic thresholds used.

We recorded high concern for the applicability of the index tests for 16 in‐person evaluations of dermoscopy (48%; Figure 5), primarily due to a lack of description of the diagnostic thresholds used (n = 8), but also as a result of presentation of average (Argenziano 2006), or consensus diagnoses (Bauer 2000; Benelli 1999; Carli 1994; Carli 2002a; Haenssle 2010b (FU); Haenssle 2010a (FV); Morales Callaghan 2008), as opposed to the diagnosis of a single observer. Six studies (18%), did not provide sufficient information to allow us to judge the clinical applicability of the dermoscopy diagnosis, and we could not fully judge observer expertise in dermoscopy in five evaluations.

We recorded high concern for the applicability of the index tests for 14 of the 16 (88%), visual inspection evaluations Argenziano 2006; Benelli 1999; Bono 2002a; Bono 2002b; Bono 2006; Carli 2002a; Dummer 1993; Grimaldi 2009; Krahn 1998; Menzies 2009; Morales Callaghan 2008; Soyer 1995; Unlu 2014; Viglizzo 2004), due to the threshold for diagnosis not being detailed in 12 studies (75%), reporting of average (Argenziano 2006), or consensus diagnoses (Benelli 1999; Carli 2002a; Morales Callaghan 2008), or diagnosis by non‐expert observers (Grimaldi 2009; Menzies 2009).

Of the 35 included in‐person evaluations, we judged 29 (83%), at low risk of bias for the reference standard due to the use of an acceptable reference standard (Ahnlide 2016; Argenziano 2006; Ascierto 2010; Bauer 2000; Benelli 1999; Bono 2002a; Bono 2002b; Bono 2006; Broganelli 2005; Carli 1994; Carli 2002a; Coras 2003; Cristofolini 1994; Duff 2001; Dummer 1993; Durdu 2011; Feldmann 1998; Gokdemir 2011; Guitera 2009a (Modena); Kittler 1999; Krahn 1998; Langley 2007; Morales Callaghan 2008; Nachbar 1994; Piccolo 2000; Soyer 1995; Soyer 2004; Unlu 2014; Viglizzo 2004; Figure 5). Five (14%), did not meet our criteria for an acceptable reference standard, with more than 20% of the benign lesions undergoing follow‐up rather than excision (Grimaldi 2009; Haenssle 2010b (FU); Haenssle 2010a (FV); Menzies 2009; Stanganelli 2000), and we judged one study at unclear risk of bias due to lack of reporting of the number of participants with a histological reference standard and number with follow‐up (Dreiseitl 2009). Blinding of the reference standard to the index test (in this case the pathology referral diagnosis), was recorded but did not contribute to the overall risk of bias for this domain. Menzies 2009 did not implement any blinding of the reference standard, and 34 studies (97%), did not describe blinding. The applicability of the reference standard was of low concern in seven evaluations (20%; Argenziano 2006; Duff 2001; Feldmann 1998; Krahn 1998; Langley 2007; Nachbar 1994; Unlu 2014), high in one (Menzies 2009), and unclear for 27 (77%). In Menzies 2009, high concern was due to the use of expert opinion for classifying the final diagnosis of some lesions. Only seven studies reported histopathology interpretation by an experienced histopathologist or by a dermatopathologist (Argenziano 2006; Duff 2001; Feldmann 1998; Krahn 1998; Langley 2007; Nachbar 1994; Unlu 2014).

In terms of flow and timing, we judged 15 of the 35 cohorts at high risk of bias (43%), (Ahnlide 2016; Argenziano 2006; Coras 2003; Dreiseitl 2009; Dummer 1993; Durdu 2011; Feldmann 1998; Grimaldi 2009; Guitera 2009a (Modena); Haenssle 2010a (FV); Haenssle 2010b (FU); Kittler 1999; Langley 2007; Menzies 2009; Stanganelli 2000), six (17%), at low risk (Ascierto 2010; Bauer 2000; Benelli 1999; Carli 1994; Morales Callaghan 2008; Soyer 2004), and 13 (37%), did not provide enough information on which to judge this domain (Figure 5). Of those at high risk, six evaluations did not use the same reference standard for all participants (differential verification), (Dreiseitl 2009; Grimaldi 2009; Haenssle 2010a (FV); Haenssle 2010b (FU); Menzies 2009; Stanganelli 2000), and 10 did not include all participants in the analysis (Ahnlide 2016; Argenziano 2006; Coras 2003; Dreiseitl 2009; Dummer 1993; Feldmann 1998; Guitera 2009a (Modena); Kittler 1999; Langley 2007; Menzies 2009). A further 23 (66%) cohorts were unclear on the interval between the application of the index test and excision for histology, with only 12 (34%), reporting consecutive diagnosis and excision or biopsy (Ahnlide 2016; Ascierto 2010; Benelli 1999; Carli 1994; Durdu 2011; Feldmann 1998; Guitera 2009a (Modena); Haenssle 2010a (FV); Haenssle 2010b (FU); Langley 2007; Morales Callaghan 2008; Soyer 2004).

Image‐based evaluations

Across the 74 image‐based dermoscopy evaluations, we judged risk of bias to be high or unclear in all domains apart from the dermoscopy index test domain (Figure 7; Figure 8). We scored applicability of study findings as being of high concern in almost all studies for three out of four domains that we assessed. Only the reference standard domain raised few concerns about applicability.

For participant selection, we judged 38 of the 74 evaluations (51%), at high risk of bias (Arevalo 2008; Argenziano 2011; Benelli 2000a; Binder 1994; Blum 2003b; Blum 2004a; Carli 2003b; Carrera 2016; di Meo 2016; Feci 2015; Ferrari 2015; Friedman 2008; Gereli 2010; Guitera 2009b (Sydney); Hauschild 2014; Kittler 2001; Malvehy 2014; Menzies 2005; Menzies 2008; Menzies 2013; Piccolo 2014; Pizzichetta 2002; Pizzichetta 2004; Pupelli 2013; Rosendahl 2011; Rubegni 2012; Sboner 2004; Seidenari 1998; Skvara 2005; Stanganelli 1998a; Stanganelli 1999; Stolz 1994a; Tan 2009; Tenenhaus 2010; Troyanova 2003; Wells 2012; Westerhoff 2000; Winkelmann 2016), and 27 (36%), did not provide sufficient information to judge this domain (Figure 7). Nineteen evaluations (26%), implemented a case‐control type design with separate sampling of melanoma and non‐melanoma lesions, and 25 (34%), excluded lesions on the basis of size or thickness (n = 6); type of lesion (n = 8); lesion site (n = 3); equivocal pathology (n = 4); or inadequate image quality (n = 8). Twenty‐nine evaluations (39%), did not report the method of participant selection and 31 (42%), did not clearly describe exclusions from the study. We considered all evaluation cohorts at high concern for applicability of participants. In the majority of cases, this was due to restricted study populations such as inclusion of only melanocytic (n = 35), amelanotic (n = 2), nodular (n = 1), regressing (n = 1), or acral (n = 1), lesions, or inclusion of lesions selected for excision based on the clinical or dermoscopic diagnosis (n = 57). Nineteen evaluations clearly reported including similar numbers of participants and lesions, seven reported inclusion of multiple lesions per participant and 48 did not report the number of participants.

For the index test domain, there are 74 evaluations of image‐based dermoscopy and 15 evaluations of visual inspection of clinical images (Figure 7). For dermoscopy, we considered 50 evaluations (68%), at low risk of bias (Annessi 2007; Arevalo 2008; Argenziano 1998; Argenziano 2011; Benelli 2000a; Benelli 2001; Binder 1994; Binder 1995; Binder 1999; Blum 2004a; Blum 2004b; Bourne 2012; Carli 2002a; Carli 2003a; Carli 2003b; Carrera 2016; Dal Pozzo 1999; di Meo 2016; Dolianitis 2005; Dummer 1993; Feci 2015; Ferrari 2015; Friedman 2008; Gereli 2010; Guitera 2009b (Sydney); Kreusch 1992; Lorentzen 1999a; Lorentzen 2000; Lorentzen 2008; Malvehy 2014; Menzies 1996; Menzies 2008; Menzies 2013; Nilles 1994; Pagnanelli 2003; Piccolo 2014; Pizzichetta 2002; Pizzichetta 2004; Pupelli 2013; Rao 1997; Rosendahl 2011; Rubegni 2012; Rubegni 2016; Seidenari 2005; Skvara 2005; Stanganelli 2015; Stolz 1994a; Tan 2009; Unlu 2014; Zalaudek 2006), and we judged two evaluations high risk, both appearing to report new algorithms or lesion scoring based on their own study data (Blum 2003a; Blum 2003b). Twenty‐two evaluations (30%), did not provide sufficient information to allow to make a full 'Risk of bias' judgement. We judged all studies to have made the diagnosis blinded to the reference standard result; 50 (68%), also clearly reported pre‐specification of the diagnostic threshold (40 using named algorithms or pattern analysis, four reporting new algorithms developed using training and test sets (Dal Pozzo 1999; Menzies 1996; Nilles 1994; Stolz 1994a), and six providing an indication as to how the diagnosis was to be reached (Binder 1995; Carli 2003b; Carrera 2016; Friedman 2008; Lorentzen 1999a; Malvehy 2014)).

We considered that all 15 image‐based visual inspection evaluations also made the diagnosis blinded to the reference standard result. We considered three at low risk of bias due to the use of named algorithms with pre‐specified thresholds (Benelli 2000a; Benelli 2001; Rao 1997), and two (Pizzichetta 2004; Rosendahl 2011), provided some prior indication as to how the diagnosis was to be reached in the study. We judged the remaining 10 unclear as to pre‐specification of the diagnostic thresholds used.

We recorded high concern for the applicability of the index tests for 67 (91%), image‐based evaluations of dermoscopy (Alarcon 2014; Annessi 2007; Arevalo 2008; Argenziano 1998; Argenziano 2011; Benelli 2000a; Benelli 2001; Binder 1994; Binder 1995; Binder 1999; Blum 2003a; Blum 2003b; Blum 2004a; Bourne 2012; Carli 2002a; Carli 2002b; Carli 2003a; Carli 2003b; Carrera 2016; Dal Pozzo 1999; di Meo 2016; Dolianitis 2005; Feci 2015; Ferrari 2015; Ferris 2015; Friedman 2008; Gereli 2010; Gilmore 2010; Glud 2009; Guitera 2009b (Sydney); Hauschild 2014; Kittler 1998; Kittler 2001; Kreusch 1992; Lorentzen 1999a; Lorentzen 2000; Lorentzen 2008; Malvehy 2014; Menzies 1996; Menzies 2005; Menzies 2008; Menzies 2013; Nilles 1994; Pagnanelli 2003; Piccolo 2002a; Piccolo 2014; Pizzichetta 2002; Rigel 2012; Rubegni 2012; Rubegni 2016; Sboner 2004; Seidenari 1998; Seidenari 2005; Seidenari 2007; Skvara 2005; Stanganelli 1998a; Stanganelli 1999; Stanganelli 2005; Stanganelli 2015; Stolz 1994a; Tan 2009; Troyanova 2003; Unlu 2014; Wells 2012; Westerhoff 2000; Winkelmann 2016), primarily due to blinded interpretation of dermoscopic images without reference to a macro photograph or other patient information (n = 51), or the presentation of average or consensus diagnoses as opposed to for a single observer (n = 35). Twenty‐five evaluations did not provide sufficient detail regarding the diagnostic threshold used, and we judged four to have reported data for non‐expert observers. The seven evaluations that we judged as having unclear concern for the applicability of dermoscopy reported data for single observers, and provided the clinical image of the lesion alongside the dermoscopic image (Blum 2004b; Lorentzen 2000; Pizzichetta 2004; Pupelli 2013; Rao 1997; Rosendahl 2011; Tenenhaus 2010). All except Tenenhaus 2010 also detailed the diagnostic thresholds used and four clearly described image interpretation by an expert observer (Blum 2004b; Lorentzen 2000; Rosendahl 2011; Tenenhaus 2010). See Figure 7.

We recorded high concern for the applicability of the index tests for all 15 visual inspection evaluations due to the image‐based nature of test interpretation; only three of these clearly reported diagnosis by a single observer (Pizzichetta 2004; Rao 1997; Rosendahl 2011), the remaining 12 reported average (n = 10), or consensus (n = 2), diagnoses. Thirteen evaluations also did not detail the threshold for diagnosis (all apart from Benelli 2000a and Benelli 2001). Eight evaluations clearly described diagnosis by expert observers (Benelli 2001; Carli 2002b; Carli 2003b; Lorentzen 1999a; Rao 1997; Rosendahl 2011; Stanganelli 2005; Troyanova 2003).

Of the 74 included image‐based evaluations, we judged 63 (85%), at low risk of bias for the reference standard due to the use of an acceptable reference standard (Alarcon 2014; Annessi 2007; Arevalo 2008; Argenziano 1998; Benelli 2000a; Benelli 2001; Binder 1994; Binder 1995; Binder 1999; Blum 2003a; Blum 2003b; Blum 2004a; Carli 2002a; Carli 2002b; Carli 2003a; Carli 2003b; Dal Pozzo 1999; di Meo 2016; Dummer 1993; Feci 2015; Ferrari 2015; Ferris 2015; Friedman 2008; Gereli 2010; Gilmore 2010; Glud 2009; Guitera 2009b (Sydney); Hauschild 2014; Kittler 1998; Lorentzen 1999a; Lorentzen 2000; Lorentzen 2008; Malvehy 2014; Menzies 1996; Nilles 1994; Pagnanelli 2003; Piccolo 2002a; Piccolo 2014; Pizzichetta 2002; Pizzichetta 2004; Pupelli 2013; Rao 1997; Rigel 2012; Rosendahl 2011; Rubegni 2012; Rubegni 2016; Sboner 2004; Seidenari 1998; Seidenari 2005; Seidenari 2007; Skvara 2005; Stanganelli 1998a; Stanganelli 1999; Stanganelli 2005; Stanganelli 2015; Stolz 1994a; Tan 2009; Troyanova 2003; Unlu 2014; Wells 2012; Westerhoff 2000; Winkelmann 2016; Zalaudek 2006). Seven evaluations were at high risk of bias, having more than 20% of the benign lesions undergoing follow‐up rather than excision (Argenziano 2011; Blum 2004b; Kittler 2001; Menzies 2005), or including some lesions with expert diagnosis only and no follow‐up (Bourne 2012; Dolianitis 2005; Menzies 2005; Tenenhaus 2010), or both. We recorded blinding of the reference standard to the index test (in this case the pathology referral diagnosis), but this did not contribute to the overall risk of bias for this domain. Only one study implemented blinding of the reference standard to the original clinical diagnosis (Friedman 2008), and the remaining studies did NR it. See Figure 7.

The applicability of the reference standard was of low concern in 20 evaluations (27%), all of which reported histopathology interpretation by an experienced histopathologist or by a dermatopathologist (Alarcon 2014; Annessi 2007; Carli 2003b; di Meo 2016; Ferrari 2015; Ferris 2015; Friedman 2008; Gilmore 2010; Glud 2009; Hauschild 2014; Lorentzen 2000; Malvehy 2014; Piccolo 2002a; Pupelli 2013; Rao 1997; Rubegni 2012; Rubegni 2016; Stanganelli 2015; Unlu 2014; Wells 2012), was of high concern in four (5%), due to the use of expert opinion for classifying the final diagnosis of some lesions (Bourne 2012; Dolianitis 2005; Menzies 2005; Tenenhaus 2010), and unclear for 50 (68%). In terms of flow and timing, we judged 26 (35%), cohorts at high risk of bias (Alarcon 2014; Arevalo 2008; Argenziano 2011; Blum 2004b; Bourne 2012; Carrera 2016; di Meo 2016; Dolianitis 2005; Dummer 1993; Feci 2015; Ferrari 2015; Guitera 2009b (Sydney); Kittler 2001; Kreusch 1992; Lorentzen 1999a; Lorentzen 2008; Malvehy 2014; Menzies 2005; Menzies 2008; Menzies 2013; Pizzichetta 2004; Rosendahl 2011; Stanganelli 2005; Tenenhaus 2010; Westerhoff 2000; Zalaudek 2006), 16 (22%), at low risk (Annessi 2007; Binder 1995; Binder 1999; Blum 2003a; Blum 2003b; Blum 2004a; Carli 2002b; Ferris 2015; Glud 2009; Hauschild 2014; Lorentzen 2000; Pizzichetta 2002; Rao 1997; Skvara 2005; Stanganelli 2015; Wells 2012), and 32 (43%), did not provide enough information on which to judge this domain (Figure 7). Of those at high risk, 15 evaluations did not use the same reference standard for all participants (differential verification), and 16 did not include all participants in the analysis. Eighteen cohorts (24%), were unclear on the interval between the application of the index test and lesion excision with only eight (11%), considered to report consecutive diagnosis and excision or biopsy.

Findings

Unless otherwise stated, we undertook all analyses using HSROC models.

1. Target condition: invasive melanoma and atypical intraepidermal melanocytic variants

Eighty‐three study publications reported accuracy data for dermoscopy for the detection of primary target condition, invasive melanoma and atypical intraepidermal melanocytic variants. Two study publications each reported data for two different sets of lesions (Guitera 2009a (Modena); Guitera 2009b (Sydney); Haenssle 2010a (FV); Haenssle 2010b (FU)); and one study (Carli 2002a), provided one dataset for in‐person dermoscopy and one for image‐based interpretation of dermoscopic images. We selected a total of 86 datasets for the primary analyses; 26 for evaluations conducted in person and 60 for image‐based evaluations. Twenty‐four of the 83 study publications provided direct comparisons of dermoscopy with visual inspection alone (i.e. data for both tests reported for the same study population). Eleven studies compared in‐person visual inspection with in‐person visual inspection + dermoscopy; 11 studies compared diagnosis based on clinical images with diagnosis based on dermoscopic images of the same lesions; and two studies compared in‐person visual inspection with image‐based dermoscopy.

Analyses by clinical pathway and in‐person versus image‐based design

We have provided summary details of the in‐person and image‐based studies in Appendix 8 and Appendix 9 and have presented results for the primary analyses in Table 2 and Table 3, with heterogeneity investigations presented in Table 4. Forest plots of study data for each analysis are in Figure 9 and Figure 10; summary estimates for in‐person comparisons are in Figure 11 and Figure 12, and for image‐based comparisons in Figure 13 and Figure 14.

1. Investigation of effect of pathway positions for detection of invasive melanoma or atypical intraepidermal melanocytic variants.

Test Position on pathway a, b	Studies	Lesions (cases)		DOR (95% CI)	Specificity at 80% sensitivity (95% CI) %	Sensitivity at 80% specificity (95% CI) %	RDOR (95% CI)	P value (LR)
a. Pathway (in‐person evaluations)
Limited prior testing (all lesions included) (position 2 on clinical pathway)
Clear	2	566 (37)		15.2 (1.8 to 128)	78% (20 to 98)	79% (38 to 96)	0.41 (0.03 to 5.9)	0.001
Referred (all lesions included) (position 4 on clinical pathway)
Clear	2	3830 (82)		494 (58 to 4218)	100% (94 to 100)	98% (91 to 100)	13.4 (1.06 to 169)
Unclear	2	8764 (82)		111 (16.4 to 765)	98% (80 to 100)	95% (76 to 99)	3.0 (0.25 to 36.0)
Referred (selected on reference standard) (position 5 on clinical pathway)
Clear	5	3247 (767)		36.9 (9.1 to 150)	91% (64 to 98)	88% (73 to 96)	1.0 (comparator)
Unclear	12	3847 (539)		77.0 (34.0 to 174)	96% (90 to 99)	93% (86 to 97)	2.1 (0.41 to 10.7)
Referred (equivocal lesions only) (position 5* on clinical pathway)
Clear	2	227 (70)		74.2 (6.4 to 859)	96% (53 to 100)	93% (66 to 99)	2.0 (0.14 to 29.3)
Lesions undergoing follow‐up (position 7 on clinical pathway)
Unclear	1	2688 (87)		8.3 (0.63 to 111)	63% (6 to 98)	69% (21 to 95)	0.23 (0.01 to 4.7)
b. Pathway (image‐based evaluations)
Limited prior testing (selected on reference standard) (position 3 on clinical pathway)
Clear	1	45 (9)	0.39 (0.02 to 8.2)		5% (0 to 68)	14% (1 to 69)	0.02 (0.001 to 0.43)	0.007
Unclear	1	463 (29)	7.5 (0.61 to 92.8)		61% (8 to 97)	67% (19 to 95)	0.33 (0.02 to 5.0)
Referred (all lesions included) (position 4 on clinical pathway)
Clear	1	134 (31)	11.6 (0.94 to 142)		73% (13% to 98%)	75% (25 to 96)	0.51 (0.03 to 7.7)
Unclear	4	1619 (248)	15.1 (4.2 to 54.0)		78% (45% to 94%)	79% (55 to 92)	0.66 (0.13 to 3.5)
Referred (selected on reference standard) (position 5 on clinical pathway)
Clear	6	1336 (304)	22.7 (8.0 to 64.6)		85% (63% to 95%)	84% (68 to 93)	1.0 (comparator)
Unclear	35	7436 (1680)	16.0 (10.2 to 25.0)		79% (70 to 87)	80% (73 to 85)	0.70 (0.23 to 2.1)
Referred (equivocal lesions only) (position 5* on clinical pathway)
Clear	3	1210 (139)	84.0 (16.2 to 436)		96% (79 to 99)	94% (80 to 99)	3.7 (0.52 to 26.1)
Unclear	6	956 (326)	49.4 (16.4 to 149)		93% (79 to 98)	91% (80 to 96)	2.2 (0.47 to 10.0)
Lesions undergoing follow‐up (position 7 on clinical pathway)
Unclear	3	276 (85)	2.3 (0.50 to 10.4)		29% (6 to 72)	42% (16 to 73)	0.10 (0.02 to 0.63)
CI: confidence interval; DOR: diagnostic odds ratio; LR: likelihood ratio test; RDOR: relative diagnostic odds ratio

^apositions on the clinical pathway described in Figure 3
 ^bclear or unclear position on the clinical pathway

2. Comparison of visual inspection and dermoscopy for detection of invasive melanoma or atypical intraepidermal melanocytic variants.

Test	Studies	Lesions (cases)	DOR (95% CI)	Specificity at 80% sensitivity (95% CI) %	Sensitivity at 80% specificity (95% CI) %	RDOR (95% CI)	P value (LR)
In‐person evaluations
Visual inspection	13	6740 (459)	13.1 (7.0 to 24.5)	75% (57 to 87)	76% (66 to 85)	4.7 (3.0 to 7.5)	< 0.001
Visual inspection + dermoscopy	26	23,169 (1664)	61.7 (34.9 to 109)	95% (90 to 98)	92% (87 to 95)
Change with adding dermoscopy to visual inspection (95% CI)	‐	‐	‐	+20% (+7 to +33)	+16% (+8 to +23)	‐	‐
In‐person evaluations (direct studies)
Visual inspection	11	5854 (412)	13.7 (5.9 to 31.8)	75% (49 to 90)	77% (63 to 87)	4.8 (2.8 to 8.1)	< 0.001
Visual inspection + dermoscopy	11	5854 (412)	65.7 (27.0 to 160)	96% (87 to 99)	92% (84 to 96)
Change with adding dermoscopy to visual inspection (95% CI)	‐	‐	‐	+21% (+2 to +39)	+15% (+7 to +23)	‐	‐
Image‐based evaluations
Clinical (macro) images	11	1740 (305)	3.2 (1.9 to 5.4)	42% (28 to 58)	47% (34 to 59)	5.6 (3.7 to 8.5)	< 0.001
Dermoscopic images	60	13,475 (2851)	17.8 (12.3 to 25.7)	82% (75 to 87)	81% (76 to 86)
Change replacing visual inspection with dermoscopy (95% CI)	‐	‐	‐	+40% (+27 to +57)	+34% (+24 to +46)	‐	‐
Image‐based evaluations (direct studies)
Clinical (macro) images	11	1740 (305)	3.6 (1.7 to 7.6)	48% (25 to 73)	47% (30 to 64)	5.3 (3.5 to 8.0)	< 0.001
Dermoscopic images	11	1735 (306)	19.2 (8.7 to 42.0)	83% (70 to 91)	83% (68 to 92)
Change replacing visual inspection with dermoscopy (95% CI)	‐	‐	‐	+34% (+15 to +53)	+36% (+20 to +52)	‐	‐
CI: confidence interval; DOR: diagnostic odds ratio; LR: likelihood ratio test; RDOR: relative diagnostic odds ratio

3. Investigations of sources of heterogeneity in person studies positions for detection of invasive melanoma or atypical intraepidermal melanocytic variants.

Test	Studies	Lesions (cases)	RDOR (95% CI)	Specificity at 80% sensitivity (95% CI) %	Sensitivity at 80% specificity (95% CI) %	RDOR (95% CI)	P value (LR)
Difference between in‐person and image‐based studies
In‐person	26	23,169 (1664)	73.2 (41.2 to 130)	95% (92 to 98)	94% (90 to 97)	4.6 (2.4 to 9.0)	< 0.001
Image	60	13,475 (2851)	15.8 (10.7 to 23.3)	79% (72 to 86)	80% (73 to 85)
Difference (95% CI)	‐	‐	‐	+16% (+9 to +23)	+14% (+8 to +21)	‐	‐
Differences in reference standard (in‐person studies)
Histology	18	5105 (767)	51.4 (24.6 to 107)	94% (86 to 98)	91% (84 to 95)	0.27 (0.06 to 1.22)	0.23
Histology+FU	7	17,733 (865)	188 (50.8 to 697)	99% (93 to 100)	97% (90 to 99)
Difference (95% CI)	‐	‐	‐	+5% (‐1 to +10)	+6% (‐0 to +12)	‐	‐
Use of an algorithm (in‐person studies)
No algorithm	16	9302 (1159)	72.6 (30.1 to 175)	96% (88 to 99)	93% (86 to 97)	1.4 (0.34 to 5.6)	0.17
Any algorithm	10	13,867 (505)	52.3 (18.1 to 151)	94% (82 to 98)	91% (80 to 96)
Difference (95% CI)	‐	‐	‐	‐2% (‐10 to +7)	‐2% (‐11 to +7)	‐	‐
Lesion type (in‐person studies)
Melanocytic	8	2460 (416)	38.2 (11.7 to 124)	91% (72 to 98)	89% (74 to 96)	0.48 (0.12 to 2.0)	0.60
Pigmented	18	20,709 (1248)	79.1 (35.7 to 175)	96% (90 to 99)	94% (88 to 97)
Difference (95% CI)	‐	‐	‐	+5% (‐7 to +16)	+5% (‐6 to +15)	‐	‐
Single or multiple individuals making diagnosis (in‐person studies)
Single	13	8436 (1044)	60.3 (21.7 to 168)	95% (83 to 99)	92% (83 to 96)	1.0 (0.18 to 5.8)	0.30
Consensus	7	12,377 (294)	59.2 (14.9 to 236)	95% (78 to 99)	92% (76 to 98)
Difference (95% CI)	‐	‐	‐	0% (‐10 to +10)	0% (‐11 to +11)	‐	‐
Prevalence (in‐person studies)
0%‐5%	6	15,392 (206)	99.1 (24.6 to 400)	97% (87 to 99)	94% (82 to 98)	5.4 (0.80 to 36.6)	0.008
> 5%‐10%	6	1718 (117)	18.3 (4.7 to 71.9)	81% (43 to 96)	81% (58 to 93)	1.0 (comparator)
> 10%‐20%	6	2089 (312)	49.4 (14.0 to 175)	94% (75 to 99)	90% (77 to 96)	2.7 (0.42 to 17.3)
> 20%	8	3970 (1029)	92.1 (27.5 to 309)	96% (86 to 99)	93% (86 to 97)	5.0 (0.78 to 32.4)
CI: confidence interval; DOR: diagnostic odds ratio; FU: follow‐up; LR: likelihood ratio test; RDOR: relative diagnostic odds ratio

9.

In‐person evaluations of the accuracy of dermoscopy added to visual inspection grouped by pathway categorisation for detecting invasive melanoma or atypical intraepidermal melanocytic variants

10.

Image‐based evaluations of the accuracy of dermoscopy grouped by pathway categorisation for detecting for detecting invasive melanoma or atypical intraepidermal melanocytic variants

11.

Comparison of the accuracy of visual inspection with visual inspection (VI) + dermoscopy for detection of invasive melanoma or atypical intraepidermal melanocytic variants from in‐person studies

12.

Paired comparisons of the accuracy of visual inspection with visual inspection (VI) + dermoscopy for detection of invasive melanoma or atypical intraepidermal melanocytic variants from in‐person studies

13.

Comparison of the accuracy of visual inspection with dermoscopy for detection of invasive melanoma or atypical intraepidermal melanocytic variants from image‐based studies

14.

Paired comparison of the accuracy of visual inspection versus dermoscopy for detection of invasive melanoma or atypical intraepidermal melanocytic variants from paired image‐based studies

We noted clear differences in accuracy between studies undertaken in person and those that evaluated images, with the accuracy of diagnosis using dermoscopic images and visual inspection of photographs being significantly lower in image‐based studies. For dermoscopy, the diagnostic odds ratio for in‐person diagnosis was more than four times that of image‐based diagnosis (RDOR 4.6, 95% CI 2.40 to 9.0; P < 0.001; Table 4; Figure 15). The high magnitude and importance of this observed difference drove our decision to undertake all analyses separately for in‐person and image‐based analyses as a primary objective of the review.

15.

Comparison of the accuracy of dermoscopy for detection of invasive melanoma or atypical intraepidermal melanocytic variants between in‐person (visual inspection + dermoscopy) and image‐based (dermoscopy) studies

Of the 26 evaluations conducted on an in‐person basis, 11 contained enough information to describe where on the clinical pathway they had assessed their participants (coded as ‘clear’ on pathway), and we considered that 15 did not provide sufficient information to allow us to identify the pathway (coded ‘unclear’ on pathway). We coded pathway positions between 1 (test‐naïve participants), and 7 (participants identified as high risk for developing melanoma with lesions undergoing follow‐up surveillance). See Figure 3 for a diagram of the clinical pathway. For the 60 image‐based evaluations, we coded 11 as clear on the pathway and 49 as unclear. Across both sets of studies, we considered only 5% (4 of 86 studies), to have included participants who were presenting for a first structured clinical assessment of a suspicious lesion, the remaining datasets came from studies in participants referred for specialist assessment.

Although there were significant differences between studies undertaken at different points on the pathway, for both in‐person (Table 2a; Figure 9), and image‐based approaches (Table 2b; Figure 10), there was no clear trend in the estimates of accuracy of dermoscopy according to the degree of prior testing of study participants (as represented by study position on the pathway). Accuracy did appear to be lowest (in terms of DORs), in studies with limited prior testing of the participants (Bourne 2012; Grimaldi 2009; Menzies 2009; Rosendahl 2011), and in those with lesions undergoing follow‐up (Haenssle 2010b (FU); Kittler 2001; Skvara 2005; Stanganelli 2015), however, the data were too scarce to draw any firm conclusions. We did not give any further analytical consideration to classification of evaluations by position on the clinical pathway.

Dermoscopy added to visual inspection of a skin lesion (in‐person evaluations)

Of the 26 in‐person evaluations of dermoscopy (Appendix 8 and Figure 9), 11 compared visual inspection alone to visual inspection + dermoscopy, including two that compared both tests to a CAD‐based test (Bono 2002a), and one that reported data for a teledermatology consultation (Grimaldi 2009), and 15 presented data only for dermoscopy in addition to visual inspection (with no data for visual inspection alone), including four that compared in‐person dermoscopy to the accuracy of other tests, including RCM (Langley 2007; Guitera 2009a (Modena)), exfoliative cytology (Durdu 2011), and CAD (Bauer 2000). Two studies compared the accuracy of different dermoscopy algorithms (Kittler 1999; Menzies 2009).

Two evaluations were conducted in limited prior testing populations (Grimaldi 2009; Menzies 2009). Of those in referred populations, we considered two to have been conducted in participants with equivocal lesions (Carli 1994; Soyer 1995), and one in participants at high risk for developing melanoma with lesions undergoing surveillance (Haenssle 2010b (FU)). The latter study also reported data separately for the same participants at their first visit for lesion assessment (Haenssle 2010a (FV)). Seventeen evaluations were prospective case series, five were retrospective (Ahnlide 2016; Bono 2006; Carli 2002a; Duff 2001; Stanganelli 2000), and four did not clearly report the design (Bauer 2000; Carli 1994; Gokdemir 2011; Soyer 1995). One study included all melanomas observed across the recruitment period but only a random sample of 50% of observed benign naevi (Guitera 2009a (Modena)). Eighteen evaluations included only pigmented lesions and eight restricted inclusion to lesions considered to be melanocytic in nature. Eighteen of the 26 evaluations (69%), clearly reported including in situ melanomas as disease‐positive, the remaining eight describing only 'melanomas' not broken down by invasive or in situ (Broganelli 2005; Cristofolini 1994; Durdu 2011; Gokdemir 2011; Grimaldi 2009; Morales Callaghan 2008; Nachbar 1994; Stanganelli 2000). The prevalence of invasive melanoma and atypical intraepidermal melanocytic variants ranged from less than 1% (Haenssle 2010a (FV)), to 41% (Guitera 2009a (Modena); Soyer 1995); median 12% (IQR 5, 21%).

Twenty‐four evaluations (89%), clearly reported that they had conducted diagnosis on an in‐person basis and we assumed in‐person diagnosis in three studies that did not clearly report the use of images or face‐to‐face diagnosis (Broganelli 2005; Gokdemir 2011; Stanganelli 2000). Diagnosis was recorded by primary care physicians in two studies (7%), (Grimaldi 2009; Menzies 2009), by dermatology residents (trainees), under the supervision of a senior dermatologist (Haenssle 2010b (FU); Haenssle 2010a (FV)), or by a mixed group of dermatology residents and consultants (Ahnlide 2016), in three (11%), by dermatologists or presumed‐to‐be dermatologists (based on author’s institutions), in 17 (63%), by plastic surgeons (Duff 2001), or oncologists (Bono 2002a; Bono 2002b; Bono 2006), in four (15%), or was NR (4%), (Feldmann 1998). Where reported (n = 22), the number of observers ranged from 1 to 63 (median 2, IQR 1.25 to 4). Thirteen evaluations (48%), reported test accuracy for a single observer, eight (30%), for a consensus of two or three observers, and we could not derive this information for the remaining six evaluations. Eight evaluations (30%), did not report any formal algorithm to assist diagnosis and nine (33%), reported using pattern analysis. The remaining evaluations used formal algorithms to assist diagnosis: the ABCD algorithm (n = 5), the seven‐point checklist (n = 3), the Menzies criteria (n = 1), and seven features for melanoma (n = 1). See Appendix 2 for details of the algorithms used.

Across the 27 evaluations the sensitivity of dermoscopy ranged from 53% to 100% and specificity from 28% to 100% (Figure 9). The low specificities of 28% (Guitera 2009a (Modena), and 56% (Carli 1994), appeared as outliers, all other studies having specificities of 69% or above. Guitera 2009a (Modena) included a relatively high proportion of Spitz naevi in the disease‐negative group than might be expected in routine clinical practice (19%), while Carli 1994 primarily aimed to distinguish atypical from typical melanocytic lesions and reported accuracy for the decision to excise a lesion as opposed to accurate diagnosis of melanoma.

We pooled results across algorithms and thresholds as a summary ROC curve (23,487 lesions and 1737 melanomas; Figure 11). Estimates of accuracy obtained from the curve suggest that the specificity of dermoscopy would be 95% (95% CI 90% to 98%), at a fixed threshold of 80% sensitivity, and sensitivity would be 92% (95% CI 87% to 95%), at a fixed threshold of 80% specificity (Table 3). We chose these 80% fixed values as they lie within the estimates for the majority of analyses and should only be considered as illustrative examples of the values that might be achieved based on the observed data (see Statistical analysis and data synthesis).

Incremental accuracy from dermoscopy added to in‐person visual inspection alone

Of the 13 available in‐person evaluations of visual inspection, 11 were reported in these in‐person dermoscopy studies and two (Dummer 1993; Unlu 2014), compared in‐person visual inspection to image‐based dermoscopy (see results for image‐based dermoscopy below). Of the 13 evaluations, 77% (n = 10), reported using no algorithm to assist visual inspection diagnosis and three used the ABCD (Stanganelli 2000), or ABCDE algorithm (Benelli 1999; Cristofolini 1994).

Sensitivities for visual inspection ranged from 38% to 100%; specificities ranged from 45% to 99% (Appendix 10; Figure 11). We compared the accuracy of visual inspection with the accuracy of dermoscopy estimated from (a), all 26 dermoscopy studies (23,169 lesions and 1664 melanomas), and all 13 in‐person visual inspection studies (6740 lesions and 459 melanomas; Figure 11), and (b), estimated from direct comparisons in the subset of 11 studies that evaluated both visual inspection and dermoscopy on an in‐person basis (5854 lesions and 412 melanomas; Figure 12). In both comparisons the accuracy of dermoscopy in addition to visual inspection exceeded that of visual inspection alone (Table 3). In (a), the diagnostic odds ratio (DOR), for dermoscopy was 4.7 (95% CI 3.0 to 7.5; P < 0.001), times that of visual inspection alone, in (b), it was 4.8 (95% CI 2.8 to 8.1; P < 0.001), times that of visual inspection alone. These effects correspond to predicted differences in specificity of (a), 20% (95% CI 7% to 33%), (based on specificity with dermoscopy of 95% vs 75% for visual inspection), and (b), 21% (95% CI 2% to 39%), (based on specificity with dermoscopy of 96% vs 75% for visual inspection), at a fixed sensitivity of 80% (Table 3), and predicted differences in sensitivity of (a), 16% (95% CI 8% to 23%), (based on sensitivity with dermoscopy of 92% vs 76% for visual inspection), and (b), 15% (95% CI 7% to 23%), (based on sensitivity with dermoscopy of 92% vs 77% for visual inspection,), at a fixed specificity of 80% (Table 3).

Dermoscopic images (image‐based evaluations)

Of the 60 image‐based evaluations of dermoscopy (Appendix 9 and Figure 10), 30 presented data only for dermoscopy, 14 compared diagnosis based on clinical images to diagnosis based on dermoscopic images, 19 compared dermoscopy to the accuracy of other tests including RCM‐ (Alarcon 2014; Ferrari 2015; Guitera 2009b (Sydney); Pupelli 2013; Stanganelli 2015), and CAD‐based tests (Binder 1994; Blum 2004b; Ferris 2015; Friedman 2008; Glud 2009; Hauschild 2014; Malvehy 2014; Menzies 2005; Piccolo 2002a; Piccolo 2014; Rigel 2012; Stanganelli 2005; Wells 2012; Winkelmann 2016). Studies that evaluated dermoscopy images rather than using real‐time in‐person dermoscopy tended to have been undertaken for reasons of efficiency and not as evaluations of a remote‐imaging service, for example, 18 (30%), evaluations compared the accuracy of different dermoscopy algorithms and 13 (22%), compared the accuracy of different observers (see Table 5, Analysis by observer experience).

4. Analysis by observer experience for detection of invasive melanoma or atypical intraepidermal melanocytic variants.

Test	Studies	Lesions (cases)	DOR (95% CI)	Specificity at 80% sensitivity (95% CI) %	Sensitivity at 80% specificity (95% CI) %	RDOR (95% CI)	P‐value (LR)
Experience: in‐person
NR	10	8390 (1015)	97.7 (35.6 to 268)	97% (90 to 99)	94% (87 to 98)	1.9 (0.49 to 7.1)	0.64
High	14	14,213 (612)	52.4 (21.6 to 127)	94% (84 to 98)	91% (83 to 96)	1.00 (comparator)
Trained	2	566 (37)	19.2 (1.6 to 226)	82% (19 to 99)	82% (36 to 97)	0.37 (0.03 to 5.1)
Experience: image‐based
NR	11	2777 (465)	35.4 (15.9 to 78.7)	90% (80 to 96)	89% (80 to 95)	2.0 (0.8 to 4.9)	< 0.001
High	34	8933 (1956)	17.2 (11.8 to 26.5)	82% (74 to 87)	81% (75 to 86)	1.00 (comparator)
Moderate	5	678 (193)	11.3 (5.9 to 21.3)	73% (58 to 85)	74% (61 to 84)	0.64 (0.37 to 1.1)
Low	6	448 (123)	5.3 (2.6 to 10.8)	55% (35 to 73)	58% (41 to 74)	0.30 (0.15 to 0.58)
Mixed	5	473 (117)	4.4 (1.4 to 13.5)	50% (23 to 77)	54% (29 to 78)	0.25 (0.07 to 0.81)
Trained	11	1087 (240)	9.0 (4.5 to 17.9)	68% (51 to 81)	70% (55 to 82)	0.15 (0.25 to 1.02)
CI: confidence interval; DOR: diagnostic odds ratio; LR: likelihood ratio test; NR: not reported; RDOR: relative diagnostic odds ratio

Two evaluations recruited participants from limited prior testing populations (Bourne 2012; Rosendahl 2011). Of those in referred populations, we considered that nine had been conducted in participants with equivocal lesions (Alarcon 2014; Annessi 2007; Carli 2003b; Dummer 1993; Ferrari 2015; Kittler 1998; Pupelli 2013; Rubegni 2012; Stolz 1994a), and three in participants with lesions undergoing follow‐up (Kittler 2001; Skvara 2005; Stanganelli 2015). Seven (12%), evaluations were prospective case series, 33 (55%), were retrospective case series, 17 (28%), used a case‐control type design and in two (3%), the design was not clearly reported. All studies prospectively re‐interpreted previously acquired dermoscopic images for the purposes of the study. The majority of studies recruited either pigmented (26; 43%), or melanocytic (30; 50%), lesions, including one restricted to melanocytic acral lesions only (Rubegni 2012). Two studies (3%), recruited any lesion selected for excision (Malvehy 2014; Zalaudek 2006), and two (3%), included only amelanotic (Pizzichetta 2004), or amelanotic or hypomelanotic (Menzies 2008), lesions. Forty‐four of the 60 evaluations (73%), clearly reported including in situ melanomas as disease‐positive, the remaining 16 describing only 'melanomas' not broken down by invasive or in situ (Binder 1994; Binder 1995; Ferrari 2015; Gilmore 2010; Kittler 1998; Malvehy 2014; Pagnanelli 2003; Piccolo 2002a; Pizzichetta 2002; Rigel 2012; Rubegni 2016; Seidenari 1998; Seidenari 2005; Stanganelli 1998a; Stanganelli 2005; Unlu 2014). The prevalence of disease ranged from 3% (Dummer 1993), to 61% (Stolz 1994a), (median 24%, IQR 18% to 39%). Prevalence was generally higher in case‐control type studies (median 37%, IQR 25% to 50%), compared to other designs (median 23%, IQR 18% to 33%).

Diagnosis was recorded by dermatologists or assessors presumed to be dermatologists in 80% of studies (n = 48), by dermatology residents in one (Carli 2003a), and by observers with mixed qualifications in 17% (n = 10), including one where all observers were primary care‐based (three GPS and a clinical nurse in Bourne 2012). Observer qualifications were NR in one study (Stolz 1994a). Where reported (n = 56), the number of observers ranged from 1 to 179 (median 3, IQR 2 to 8). Twenty‐five evaluations (42%), reported test accuracy for a single observer, nine (15%), for a consensus of two or three observers, one study for a consensus of at least 50% of all observers (Carrera 2016), and 19 (32%), for the median or average across observers. We could not derive this information for the remaining six evaluations. Half of all evaluations (n = 30), blinded dermoscopic image interpretation; a further third of evaluations (n = 20), provided the associated clinical (n = 17), RCM (n = 2), or baseline dermoscopy image (n = 1), to assist diagnosis. Four evaluations provided information on lesion site, or patient age or gender and the remaining six did not describe the provision of additional information. Twenty‐three studies (38%), did not report any formal algorithm to assist diagnosis, and 19 (32%), reported using pattern analysis. The remaining 18 studies used formal algorithms to assist diagnosis: the ABCD algorithm (n = 6), the seven‐point checklist (n = 3), or a revised version thereof (n = 1), the three‐point checklist (n = 3), the Menzies criteria (n = 1), and seven features for melanoma (n = 3), or the observers' own choice of algorithm (n = 1; Appendix 9).

Across the 60 image‐based dermoscopy evaluations, the sensitivity ranged from 22% to 100% and specificity from 31% to 99%. We pooled results across algorithms and thresholds as a summary ROC curve (13,475 lesions and 2851 melanomas; Figure 13). Estimates of accuracy obtained from the curve suggest that the specificity of dermoscopy would be 82% (95% CI 75% to 87%), at a fixed threshold of 80% sensitivity and sensitivity would be 81% (95% CI 76% to 86%), at a fixed threshold of 80% specificity (Table 3).

Incremental accuracy of dermoscopic image‐based diagnosis compared to visual inspection of images

Of the 11 visual inspection evaluations based on interpretation of clinical images, nine (82%), reported using no algorithm to assist image interpretation, and two used the ABCD algorithm (Benelli 2000a; Benelli 2001). Seven studies reported blinded interpretation of the clinical image with no further patient or lesion information provided, one study allowed observers to view both the clinical and dermoscopic image simultaneously (Pizzichetta 2004), and three did not clearly describe blinding between the clinical and dermoscopic images (Benelli 2000a; Stanganelli 2005; Winkelmann 2016).

Sensitivities for image‐based visual inspection ranged from 21% to 80%; specificities ranged from 53% to 97% (Figure 13). We compared the accuracy of visual inspection with the accuracy of dermoscopy estimated from (a), all 60 dermoscopy studies (13,475 lesions and 2851 melanomas), and the 11 image‐based visual inspection studies (1740 lesions and 305 melanomas), (Figure 13), and estimated from direct comparisons in (b), the subset of 11 studies that evaluated both clinical and dermoscopic images (1740 lesions and 305 melanomas; Figure 14). In both comparisons, the accuracy of dermoscopy exceeded that of visual inspection alone (Table 3). In (a), the diagnostic odds ratio (DOR), for dermoscopy was 5.6 (95% CI 3.7 to 8.5; P < 0.001), times that of visual inspection alone, in (b), it was 5.3 (95% CI 3.5 to 8.0; P < 0.001), times that of visual inspection alone. These effects correspond to predicted differences in specificity of (a), 40% (95% CI 27% to 57%), (based on specificity with dermoscopy of 82% vs 42% for visual inspection), and (b), 34% (95% CI 15% to 53%), (based on specificity with dermoscopy of 83% vs 48% for visual inspection), at a fixed sensitivity of 80% (Table 3), and predicted differences in sensitivity of (a), 34% (95% CI 24% to 46%), based on sensitivity with dermoscopy of 81% vs 47% for visual inspection and (b), 36% (95% CI 20% to 52%), based on sensitivity with dermoscopy of 83% vs 47% for visual inspection at a fixed specificity of 80% (Table 3).

Secondary analyses for the detection of invasive melanoma and melanocytic intra‐epidermal variants

Covariate investigations

Table 4 and Table 6 report the results of the heterogeneity investigations. Given the large difference in accuracy for in‐person evaluations compared to those based on the assessment of dermoscopic images, we elected to undertake all subsequent covariate investigations for in‐person (Table 4), and image‐based (Table 6), studies separately. In four of the covariate investigations (apart from that by disease prevalence), we dropped subgroups with small numbers of studies to allow a comparison between the two larger subgroups.

5. Investigations of sources of heterogeneity in image‐based studies for detection of invasive melanoma or atypical intraepidermal melanocytic variants.

Test	Studies	Lesions (cases)	DOR (95% CI)	Specificity at 80% sensitivity (95% CI) %	Sensitivity at 80% specificity (95% CI) %	RDOR (95% CI)	P‐value (LR)
Differences in reference standard (image‐based studies)
Histology	48	10,267 (2210)	20.8 (13.6 to 31.9)	84% (77 to 89)	84% (77 to 89)	2.8 (0.92 to 8.9)	0.19
Histology+FU	8	2762 (549)	7.3 (2.6 to 20.9)	64% (36 to 84)	65% (41 to 84)
Difference (95% CI)	‐	‐	‐	‐20% (‐47 to +6)	‐18% (‐42 to +5)	‐	‐
Use of an algorithm (image‐based studies)
No algorithm	42	8762 (1834)	18.9 (11.8 to 30.3)	83% (74 to 88)	82% (75 to 88)	1.4 (0.60 to 3.3)	0.22
Any algorithm	18	4713 (1017)	13.4 (6.7 to 27.0)	77% (62 to 87)	77% (63 to 87)
Difference (95% CI)	‐	‐	‐	‐6% (‐20 to +9)	‐5% (‐19 to +9)	‐	‐
Lesion type (image‐based studies)
Melanocytic	30	6980 (1710)	18.1 (9.8 to 33.4)	82% (70 to 90)	82% (71 to 89)	1.10 (0.49 to 2.50)	0.16
Pigmented	26	4062 (733)	16.4 (9.6 to 27.9)	80% (70 to 88)	80% (71 to 87)
Difference (95% CI)	‐	‐	‐	‐2% (‐15 to 12)	‐1% (‐13 to +10)	‐	‐
Single or multiple individuals making diagnosis (image‐based studies)
Single	26	5877 (1030)	27.2 (14.5 to 51.2)	88% (78 to 93)	87% (78 to 92)	1.9 (0.80 to 4.4)	0.30
Consensus	28	5232 (1350)	14.4 (8.1 to 25.7)	78% (66 to 87)	78% (68 to 86)
Difference (95% CI)	‐	‐	‐	‐10% (‐23 to +4)	‐8% (‐20 to +3)	‐	‐
Prevalence (image‐based studies)
0%‐20%	14	4855 (519)	422 (65.2 to 2726)	99% (94 to 100)	98% (93 to 100)	30.7 (1.51 to 6.24)	0.12
> 20%‐30%	17	3893 (901)	13.7 (1.2 to 162)	78% (25 to 97)	77% (20 to 98)	1.0 (comparator)
> 30%‐40%	9	974 (330)	19.5 (8.8 to 42.8)	83% (69 to 91)	83% (68 to 92)	1.4 (0.11 to 18.8)
> 40%	14	1387 (630)	15.5 (0.2 to 23.3)	79% (72 to 85)	79% (71 to 86)	1.1 (0.09 to 13.9)
CI: confidence interval; DOR: diagnostic odds ratio; FU: follow‐up; LR: likelihood ratio test; RDOR: relative diagnostic odds ratio

In‐person evaluations

Further analysis of the 26 in‐person evaluations found no clearly significant relationships between accuracy and the five covariates that we considered. We noted some evidence of differences for choice of reference standard and disease prevalence (Table 4).

• Choice of reference standard: observed accuracy was lower in studies that relied on a histological reference standard (n = 18), as opposed to those (n = 7), that included follow‐up of some benign lesions although the difference was not statistically significant (RDOR 0.27, 95% CI 0.06 to 1.22; P = 0.23). Theoretically, the inclusion of a follow‐up reference standard: has the potential to lower sensitivity (as any melanomas missed on the index clinic visit that are identified on follow‐up would be considered as false negatives), and increase specificity (as lesions considered benign and not recommended for excision on the index clinic visit and that do not show any changes on follow‐up will increase the number of true‐negative results). The data observed did demonstrate the anticipated effect on specificity (with specificities at 80% sensitivity increasing from 94% in histology‐only studies to 99% in histology or follow‐up evaluations), however the effect on sensitivity at 80% fixed specificities was the opposite to that anticipated (sensitivity was 6% higher in the histology or follow‐up group compared to histology alone). Three of the six in‐person evaluations using histology or follow‐up as a reference standard reported sensitivities of over 95% (Duff 2001; Dreiseitl 2009; Grimaldi 2009). Of the nine false‐negative cases in Duff 2001, eight melanomas were identified during follow‐up (between 5 to 41 months after the initial diagnosis), however with high overall prevalence of disease, sensitivity remained high at 98%. Dreiseitl 2009 and Grimaldi 2009 did not report any melanomas picked up during follow‐up (at 1 year and 6 months' follow‐up, respectively). The perfect sensitivity in Grimaldi 2009 is likely due to lesions classified as positive if they were ‘suspicious for melanoma’ as opposed to being a likely or definite melanoma. For Dreiseitl 2009 the high sensitivity is likely explained by diagnosis by an expert clinician, and with more than six lesions examined per patient assisting diagnosis.

• Disease prevalence: observed accuracy was somewhat higher where disease prevalence of melanoma was 5% or less (RDOR 5.4, 95% CI 0.80 to 36.6), and where prevalence was greater than 20% (RDOR 5.0; 95% CI 0.78 to 32.4), compared to those with disease prevalence between 5% and 10% (likelihood ratio (LR) test for differences between groups: P = 0.008). No obvious explanation for these results could be derived from the study characteristics (Table 4).

• Other investigations: the RDOR for use of no algorithm to aid diagnosis compared to a named algorithm was 1.4 (95% CI 0.34 to 5.6; P = 0.17), for a single observer compared to a consensus of two or more observers was 1.0 (95% CI 0.18 to 5.8; P = 0.30), and for evaluations including only melanocytic lesions compared to any pigmented lesion was 0.48 (95% CI 0.12 to 2.0; P = 0.60), (Table 4).

Image‐based evaluations

For the 60 image‐based evaluations, we noted no clearly significant relationships between accuracy and the five covariates. The choice of reference standard showed an effect in the opposite direction to that observed for the in‐person evaluation (Table 6). Observed accuracy was higher in studies that relied on a histological reference standard (n = 48), as opposed to those (n = 8), that included follow‐up of some benign lesions (RDOR 2.8; 95% CI 0.92 to 8.9; P = 0.19). At a fixed specificity of 80%, observed sensitivity in studies using a follow‐up reference standard was lower (65%), compared to those using histology alone (84%), as might be expected, however at a fixed sensitivity of 80%, specificities in studies that included follow‐up of some benign lesions was also lower (64%), compared to those using histology alone (84%). This effect is likely due to a combination of reasons that cannot be derived from the data due to heterogeneity in participants, tests and observers.

• Disease prevalence: disease prevalence was higher in image‐based studies than in in‐person studies and a different grouping for prevalence was used. Observed accuracy appeared highest where disease prevalence of melanoma was 20% or less (RDOR 30.7, 95% CI 1.51 to 6.24), compared to prevalence more than 20% to 30% and higher.

• Other investigations: for the other characteristics investigated, we observed similar results to those obtained for in‐person evaluations for use of a named algorithm, the effect from restriction to melanocytic lesions only was in the opposite direction although non‐significant (P = 0.16), and we observed results to a greater order of magnitude for diagnosis by a single observer compared to a consensus of two or more observers.

1.1.1. Analyses by algorithms used to assist dermoscopy

Appendix 2 has details of the algorithms used to assist diagnosis and Table 7 shows results by algorithm used (or not used), for each of the target conditions under consideration in this review. We undertook all analyses in this section using the bivariate normal model.

6. Algorithm and threshold analysis for each definition of the target condition.

Target condition Testa	Datasets (n)	Lesions (cases)	Pooled sensitivity (95% CI)	Pooled specificity (95% CI)	Datasets (n)	Lesions (cases)	Pooled sensitivity (95% CI)	Pooled specificity (95% CI)
a. Invasive melanoma and atypical intraepidermal melanocytic variants	In‐person				Image‐based
No algorithm: any threshold	8	4707 (849)	0.88 (0.75 to 0.95)	0.87 (0.80 to 0.92)	24	4498 (941)	0.76 (0.70 to 0.82)	0.79 (0.71 to 0.85)
No algorithm: correct diagnosis	‐	‐	‐	‐	18	4118 (795)	0.77 (0.69 to 0.83)	0.84 (0.76 to 0.89)
No algorithm: excise decision	‐	‐	‐	‐	10	831 (263)	0.79 (0.69 to 0.86)	0.55 (0.50 to 0.61)
Pattern: any threshold or NR	6	4307 (296)	0.92 (0.87 to 0.95)	0.92 (0.68 to 0.98)	20	4621 (989)	0.83 (0.76 to 0.88)	0.87 (0.80 to 0.92)
Pattern: at ≥ 1 characteristics present	1	220 (33)	0.88 (0.72 to 0.97)	0.79 (0.73 to 0.85)	‐	‐	‐	‐
Pattern: at ≥ 3 characteristics present	1	68 (5)	1.00 (0.48 to 1.00)	0.56 (0.42 to 0.68)	‐	‐	‐	‐
Pattern: correct diagnosis	‐	‐	‐	‐	19	4095 (896)	0.81 (0.73 to 0.87)	0.87 (0.80 to 0.92)
Pattern: excise decision	‐	‐	‐	‐	3	933 (227)	0.97 (0.68 to 1.00)	0.72 (0.60 to 0.81)
ABCD at NR (likely > 5.45)	1	235 (5)	1.00 (0.48 to 1.00)	0.90 (0.85 to 0.93)	‐	‐	‐	‐
ABCD at > 5.45	4	1203 (155)	0.78 (0.58 to 0.90)	0.93 (0.79 to 0.98)	7	2471 (406)	0.81 (0.60 to 0.92)	0.81 (0.69 to 0.89)
ABCD at or likely > 5.45 (2 previous groups combined)	5	1438 (160)	0.81 (0.62 to 0.92)	0.92 (0.82 to 0.97)	‐	‐	‐	‐
ABCD at > 4.75	1	309 (73)	0.83 (0.69 to 0.92)	0.45 (0.39 to 0.51)	10	4242 (816)	0.81 (0.67 to 0.90)	0.72 (0.93 to 0.80)
Revised ABCD at ≥ 4	‐	‐	‐	‐	1	269 (84)	0.87 (0.78 to 0.93)	0.89 (0.83 to 0.93)
ABCD at 60% specificity	1	356 (73)	0.90 (0.81 to 0.96)	0.60 (0.54 to 0.66)	‐	‐	‐	‐
ABCD at 70% specificity	1	356 (73)	0.85 (0.75 to 0.92)	0.70 (0.64 to 0.75)	‐	‐	‐	‐
ABCD at 75% specificity	1	356 (73)	0.85 (0.75 to 0.92)	0.75 (0.69 to 0.80)	‐	‐	‐	‐
ABCD at 80% specificity	1	356 (73)	0.77 (0.65 to 0.86)	0.80 (0.75 to 0.84)	‐	‐	‐	‐
ABCD at 85% specificity	1	356 (73)	0.71 (0.59 to 0.81)	0.85 (0.80 to 0.89)	‐	‐	‐	‐
ABCD at 90% specificity	1	356 (73)	0.64 (0.52 to 0.75)	0.90 (0.86 to 0.93)	‐	‐	‐	‐
ABCDE at > 1.3	1	356 (73)	1.00 (0.95 to 1.00)	0.15 (0.11 to 0.20)	‐	‐	‐	‐
ABCDE at > 2.65	1	356 (73)	0.97 (0.90 to 1.00)	0.39 (0.33 to 0.45)	‐	‐	‐	‐
ABCDE at > 3.05	1	356 (73)	0.95 (0.87 to 0.98)	0.57 (0.51 to 0.62)	‐	‐	‐	‐
ABCDE at > 3.6	1	356 (73)	0.90 (0.81 to 0.96)	0.70 (0.64 to 0.75)	‐	‐	‐	‐
ABCDE at > 4.25	1	356 (73)	0.82 (0.71 to 0.90)	0.82 (0.77 to 0.86)	‐	‐	‐	‐
ABCDE at > 4.9	1	356 (73)	0.74 (0.62 to 0.84)	0.90 (0.86 to 0.93)	‐	‐	‐	‐
ABCDE at ≥ 4	‐	‐	‐	‐	1	269 (84)	0.90 (0.82 to 0.96)	0.87 (0.81 to 0.92)
7FFM at ≥ 2	1	401 (60)	0.80 (0.68 to 0.89)	0.89 (0.85 to 0.92)	4	2200 (340)	0.89 (0.76 to 0.96)	0.84 (0.78 to 0.89)
7PCL at ≥ 2	1	638 (108)	0.93 (0.86 to 0.97)	0.98 (0.97 to 0.99)	‐	‐	‐	‐
7PCL at ≥ 3	2	11137 (127)	0.67 (0.46 to 0.83)	0.96 (0.88 to 0.99)	11	3408 (798)	0.80 (0.63 to 0.91)	0.67 (0.51 to 0.80)
7PCL at ≥ 5	‐	‐	‐	‐	1	322 (70)	0.67 (0.55 to 0.78)	0.83 (0.78 to 0.87)
7PCL at NR	‐	‐	‐	‐	4	1936 (360)	0.72 (0.56 to 0.84)	0.79 (0.61 to 0.90)
Revised 7PCL at NR (likely ≥ 1)	‐	‐	‐	‐	1	1678 (238)	0.61 (0.54 to 0.67)	0.88 (0.86 to 0.89)
Revised 7PCL at ≥ 1	‐	‐	‐	‐	1	300 (100)	0.88 (0.80 to 0.94)	0.51 (0.44 to 0.58)
Revised 7PCL for FU: major change	‐	‐	‐	‐	1	70 (12)	0.67 (0.35 to 0.90)	0.60 (0.47 to 0.73)
Menzies at 2 negative and ≥ 1 positive	1	206 (23)	0.83 95)	0.69 (0.62 to 0.75)	4	1856 (317)	0.78 (0.38 to 0.96)	0.63 (0.39 to 0.81)
Menzies at NR	‐	‐	‐	‐	2	60 (26)	0.77 (0.57 to 0.89)	0.82 (0.66 to 0.92)
3PCL at ≥ 2	‐	‐	‐	‐	7	1505 (363)	0.74 (0.61 to 0.85)	0.60 (0.42 to 0.76)
4‐point (scored 3PCL) at > 2	‐	‐	‐	‐	1	75 (32)	0.84 (0.67 to 0.95)	0.81 (0.67 to 0.92)
Hofman algorithm at NR	‐	‐	‐	‐	1	254 (75)	0.87 (0.77 to 0.93)	0.88 (0.82 to 0.92)
CASH at ≥ 6	‐	‐	‐	‐	1	477 (119)	0.78 (0.70 to 0.85)	0.51 (0.46 to 0.56)
CASH at ≥ 8	‐	‐	‐	‐	2	190 (56)	0.97 (0.79 to 1.00)	0.69 (0.60 to 0.76)
Chaos/Clues at = 2	‐	‐	‐	‐	2	940 (148)	0.82 (0.75 to 0.87)	0.53 (0.36 to 0.70)
Acral 3‐step	‐	‐	‐	‐	1	107 (25)	0.96 (0.80 to 1.00)	0.91 (0.83 to 0.96)
b. Invasive melanoma	In‐person				Image‐based
No algorithm: threshold NR	3	190 (62)	0.87 (0.76 to 0.93)	0.96 (0.91 to 0.98)	6	683 (202)	0.77 (0.59 to 0.88)	0.79 (0.63 to 0.90)
Pattern analysis: threshold NR	1	45 (16)	0.81 (0.54 to 0.96)	0.97 (0.82 to 1.00)	1	119 (24)	1.00 (0.86 to 1.00)	0.97 (0.91 to 0.99)
ABCD at > 4.2	1	495 (23)	0.88 (0.69 to 0.97)	0.64 (0.60 to 0.68)	‐	‐	‐	‐
ABCD at > 4.75	‐	‐	‐	‐	2	330 (85)	0.76 (0.66 to 0.84)	0.84 (0.73 to 0.91)
ABCD at > 5.45	2	832 (242)	0.79 (0.74 to 0.84)	0.90 (0.58 to 0.98)	1	258 (64)	0.45 (0.33 to 0.58)	0.94 (0.89 to 0.97)
7PCL at NR	‐	‐	‐	‐	1	332 (217)	0.90 (0.85 to 0.94)	0.79 (0.71 to 0.86)
Menzies at 2 negative and ≥ 1 positive	‐	‐	‐	‐	4	4184 (715)	0.91 (0.83 to 0.96)	0.71 (0.68 to 0.74)
3PCL at > NR	‐	‐	‐	‐	1	332 (217)	0.82 (0.77 to 0.87)	0.40 (0.31 to 0.50)
Kenet (modified) at MM likely	1	54 (12)	1.00 (0.74 to 1.00)	0.95 (0.84 to 0.99)	1	258 (64)	0.75 (0.63 to 0.85)	0.94 (0.89 to 0.97)
Kenet (modified) at MM possible	1	54 (12)	1.00 (0.74 to 1.00)	0.45 (0.30 to 0.61)	1	258 (64)	0.89 (0.79 to 0.95)	0.87 (0.82 to 0.91)
CASH at ≥ 8	‐	‐	‐	‐	1	332 (217)	0.82 (0.76 to 0.86)	0.72 (0.63 to 0.80)
Kreusch algorithm	‐	‐	‐	‐	1	265 (96)	0.98 (0.93 to 1.00)	0.83 (0.77 to 0.89)
Menzies for amelanotic at 1	‐	‐	‐	‐	1	332 (217)	0.91 (0.87 to 0.95)	0.70 (0.61 to 0.79)
Menzies for amelanotic at 0	‐	‐	‐	‐	1	332 (217)	1.00 (0.98 to 1.00)	0.52 (0.43 to 0.62)
c. Any skin cancer or lesion with high risk of progression to melanoma	In‐person				Image‐based
No algorithm at NR	1	231 (77)	0.90 (0.81 to 0.95)	0.94 (0.89 to 0.97)	2	83 (32)	0.78 (0.61 to 0.89)	0.75 (0.61 to 0.85)
Pattern analysis: threshold NR	1	3372 (98)	0.90 (0.82 to 0.95)	1.00 (0.99 to 1.00)	1	119 (37)	1.00 (0.91 to 1.00)	0.96 (0.90 to 0.99)
ABCD at > 5.45	1	200 (46)	0.98 (0.88 to 1.00)	0.98 (0.94 to 1.00)	‐	‐	‐	‐
3PCL at ≥ 2	1	77 (39)	0.85 (0.69 to 0.94)	0.26 (0.13 to 0.43)	1	150 (44)	0.91 (0.78 to 0.97)	0.72 (0.62 to 0.80)
3PCL: three‐point checklist; 7FFM: seven features for melanoma; 7PCL: seven‐point checklist; ABCD(E): asymmetry, border, colour, differential structures (enlargement);CASH: colour, architecture, symmetry and homogeneity; CI: confidence interval; FU: follow‐up; MM: malignant melanoma; NR: not reported

^aAll analyses by algorithm were undertaken using the bivariate normal model (BVN).

In‐person evaluations of dermoscopy added to visual inspection

The 26 in‐person evaluations of dermoscopy added to visual inspection of a lesion provide a total of 39 datasets using different algorithms or diagnostic thresholds for the detection of invasive melanoma or atypical intraepidermal melanocytic variants. Eight of the datasets did not report the use of any algorithm to assist diagnosis, eight reported data for pattern analysis, and the remaining 23 datasets used one or more of seven different formally developed algorithms (Table 7a).

We estimated a pooled sensitivity of 88% (95% CI 75% to 95%), and specificity of 87% (95% CI 80% to 92%), for observer diagnosis without the use of a formal algorithm (n = 8 datasets; 4707 lesions, 849 melanomas). The approach to diagnosis was not well described; however, most studies in this dataset reported accuracy for the clinician’s correct diagnosis of melanoma rather than the decision to biopsy or excise a lesion (Appendix 8). Pooled results for studies using pattern analysis were similar but with narrower confidence intervals for sensitivity (sensitivity 92%, 95% CI 87% to 95%; specificity 92%, 95% CI 68% to 98%; 6 datasets with 4307 lesions and 296 melanomas).

Of the more formal algorithms for melanoma diagnosis, we could pool results for only two. Five datasets (1438 lesions and 160 melanomas), using the ABCD algorithm at a threshold of above 5.45 produced a sensitivity of 81% (95% CI 62% to 92%), and specificity of 92% (95% CI 82% to 97%). Two evaluations (11,137 lesions and 127 melanomas), reported data for the seven‐point checklist (7PCL), at a threshold of 3 or above, giving a sensitivity of 67% (95% CI 46% to 83%), and specificity of 96% (95% CI 88% to 99%). The latter result is based on a single study publication, which reports results for 8449 lesions detected on a participant's first clinic visit (Haenssle 2010a (FV)), and separately for 2373 lesions examined during follow‐up (Haenssle 2010b (FU)). The ABCDE algorithm, seven features for melanoma (7FFM), and the Menzies criteria were each assessed in a single study on an in‐person basis; results were generally similar to those observed above (Table 7a).

Image‐based evaluations of dermoscopic images

The 60 evaluations of dermoscopic images provide a total of 113 datasets using different algorithms or diagnostic thresholds for the detection of invasive melanoma or atypical intraepidermal melanocytic variants. Twenty‐eight of the datasets did not report the use of any algorithm to assist diagnosis (4 studies reporting data at two thresholds), 22 report data for pattern analysis (2 studies reporting data at two thresholds), and the remaining 63 datasets used one or more of 14 different formally developed algorithms (Table 7a).

For observer diagnosis without the use of a formal algorithm, diagnostic thresholds (i.e. the clinical decision that was recorded by the clinician concerned), were poorly reported; however, we attempted to differentiate between those studies reporting the observer’s correct diagnosis of melanoma from those reporting the decision to excise a lesion. Pooling all data regardless of threshold (24 datasets; 4498 lesions and 941 melanomas), gave a sensitivity of 76% (95% CI 70% to 82%), and specificity of 79% (95% CI 71% to 85%). Restricting the analysis to the 18 datasets reporting data for observers correctly diagnosing melanoma (4118 lesions; 795 melanomas), gave a sensitivity of 77% (95% CI 69% to 83%), and specificity of 84% (95% CI 76% to 89%). For the 10 datasets that reported data for the decision to excise a lesion (831 lesions; 263 melanomas), sensitivity was similar at 79% (95% CI 69% to 86%), but specificity reduced to 55% (95% CI 50% to 61%). Pooled results for 20 evaluations (4621 lesions and 989 melanomas), reporting use of pattern analysis resulted in higher sensitivity (83%, 95% CI 76% to 88%), and specificity (87%, 95% CI 80% to 92%), compared to the no‐algorithm‐reported studies but results were both lower in comparison to the in‐person evaluations.

Sufficient data were available to allow pooling for seven different formal algorithms to assist diagnosis (Table 7a); all summary estimates showed either lower sensitivity or lower specificity, or both, in comparison to either the no‐algorithm or pattern‐analysis datasets. The ABCD checklist at a threshold of above 5.45 (7 datasets; 2471 lesions and 406 melanomas), had a sensitivity of 81% (95% CI 60% to 92%), and specificity of 81% (95% CI 69% to 89%). At the lower threshold of above 4.75 for diagnosis of melanoma, sensitivity remained at 81% (95% CI 67% to 90%), with narrower confidence intervals with a lower specificity of 72% (95% CI 93% to 80%), (10 datasets; 4242 lesions and 816 cases).

Eleven datasets evaluated the 7PCL at a threshold of 3 or above (3408 lesions and 798 melanomas), pooled sensitivity was 80% (95% CI 63% to 91%), and specificity 67% (95% CI 51% to 80%). Four evaluations that did not report the threshold used with the 7PCL demonstrated lower sensitivity (72%, 95% CI 56% to 84%), but higher specificity (79%, 95% CI 61% to 90%).

Four datasets, with 2200 lesions and 340 melanomas assessed the 7FFM tool. Sensitivity was 89% (95% CI 76% to 96%), with specificity 84% (95% CI 78% to 89%). Four datasets evaluated the Menzies criteria using the method described in the original Menzies 1996 paper, pooled sensitivity was 78% (95% CI 38% to 96%), and specificity 63% (95% CI 39% to 81%), (1856 lesions and 317 melanomas).

We pooled seven evaluations of the 3PCL at a threshold of 2 or above (1505 lesions and 363 melanomas), summary sensitivity was 74% (95% CI 61% to 85%), and specificity 60% (95% CI 42% to 76%). Sixteen additional datasets reporting data for other algorithms or at different thresholds are reported in Table 7a, however study numbers are too small to describe results in any detail.

1.1.2. Analyses by observer experience and qualifications

Table 5 and Table 8 report results for the effect of observer experience and qualifications. Observer experience was generally poorly described in the study reports (see Appendix 8 and Appendix 9); however, we attempted broad classifications by expertise in dermoscopy and reported qualifications with the ‘consultant’ category in the latter analysis separated into ‘Expert consultant’ (for any study describing observers as expert or experienced), and ‘Consultant’ where experience or expertise was not otherwise reported (for example, for those that described observers as dermatologists). We have described results separately for in‐person (Figure 16; Figure 17), and image‐based evaluations (Figure 18; Figure 19).

7. Analysis by observer qualifications for detection of invasive melanoma or atypical intraepidermal melanocytic variants.

Test	Studies	Lesions (cases)	DOR (95% CI)	Specificity at 80% sensitivity (95% CI) %	Sensitivity at 80% specificity (95% CI) %	RDOR (95% CI)	P value (LR)
Qualifications: in‐person
Consultant expert*	11	2767 (439)	52.4 (21.6 to 127)	94% (84 to 98)	91% (83 to 96)	1.00 (comparator)	0.33
Consultant*	10	8390 (1015)	97.7 (35.6 to 268)	97% (90 to 99)	95% (87 to 98)	1.86 (0.949 to 7.11)
GP	2	566 (37)	19.2 (1.6 to 226)	82% (19 to 99)	82% (36 to 97)	0.37 (0.03 to 5.08)
Resident/registrara	2	11137 (127)	51.6 (2.9 to 927)	93% (42 to 100)	93% (42 to 100)	Not estimable within model	‐
Mixed (secondary care)a	1	309 (46)	29.6 (13.5 to 64.8)	88% (77 to 94)	88% (77 to 94)	Not estimable within model	‐
Qualifications: image based
Consultant expert*	33	8664 (1854)	19.4 (13.1 to28.8)	83% (76 to 88)	83% (77 to 88)	1.0 (comparator)	< 0.001
Consultant*	25	4589 (955)	11.9 (7.6 to 18.6)	74% (65 to 82)	75% (66 to 82)	0.61 (0.40 to 0.92)
Resident*	5	927 (138)	6.0 (2.6 to 14.0)	59% (37 to 78)	61% (41 to 78)	0.31 (0.14 to 0.71)
Mixed (other)	4	867 (229)	15.1 (4.0 to 57.0)	79% (48 to 94)	79% (not estimable)	0.78 (0.20 to 3.1)
GP/Primary care	3	288 (55)	1.9 (0.7 to 5.0)	30% (12 to 57)	34% (51 to 93)	0.10 (0.04 to 0.25)
Mixed (secondary care)b	4	399 (111)	10.3 (3.0 to 35.3)	72% (43 to 90)	72% (43 to 90)	Not estimable within model	‐
Physician assistantb	1	65 (25)	3.6 (1.1 to 11.5)	47% (22 to 74)	47% (22 to 74)	Not estimable within model	‐
CI: confidence interval; DOR: diagnostic odds ratio; LR: likelihood ratio test; NR: not reported; RDOR: relative diagnostic odds ratio

^*Consultants were usually dermatologists but could also be plastic surgeons or oncologists.
 ^aIn‐person model could not be fitted including the small number of studies in these groups. Estimates for these groups are obtained from computed the DOR for the individual study, or random effects meta‐analyses of DORs where there is more than one study. Estimates at the 80% sensitivity and specificity values are computed assuming symmetric SROC curves.
 ^bImage‐based model could not be fitted including the small number of studies in these groups. Estimates for these groups are obtained from computed the DOR for the individual study, or random effects meta‐analyses of DORs where there is more than one study. Estimates at the 80% sensitivity and specificity values are computed assuming symmetric SROC curves.

16.

Comparison of the accuracy of visual inspection + dermoscopy for detection of invasive melanoma or atypical intraepidermal melanocytic variants from in‐person studies according to reported observer experience

17.

Comparison of the accuracy of dermoscopy for detection of invasive melanoma or atypical intraepidermal melanocytic variants from image‐based studies according to observer experience

18.

Comparison of the accuracy of visual inspection + dermoscopy for detection of invasive melanoma or atypical intraepidermal melanocytic variants from in‐person studies according to observer qualifications (summary receiver operating characteristic (ROC) curves were not estimable from the model for 'resident/registrar' and 'mixed (secondary care‐based)' experience groups)

19.

Comparison of the accuracy of dermoscopy for detection of invasive melanoma or atypical intraepidermal melanocytic variants from image‐based studies according to observer qualification. (Hierarchical summary receiver operating characteristic (HSROC) curves could not be estimated for 'mixed (secondary care‐based)' and 'physician assistant' groups)

The in‐person evaluations classified the majority of observers as having high dermoscopy experience (n = 14), or as experience NR (n = 10). Two studies reported data for GPs provided with some dermoscopy training for the purposes of the study (Grimaldi 2009; Menzies 2009). We found no statistically significant differences between groups (Table 5), although we noted that the poorest performance was in the GP training group.

The 60 image‐based evaluations provided 77 datasets according to observer experience; 13 evaluations providing data for more than one observer (Argenziano 1998; Benelli 2001; Binder 1995; Ferris 2015; Hauschild 2014; Menzies 2005; Pagnanelli 2003; Piccolo 2002a; Piccolo 2014; Seidenari 1998; Seidenari 2005; Stanganelli 1999; Tan 2009). The LR test for differences between groups was statistically significant (P < 0.001). Using the high‐experience group as the reference (34 datasets; 8933 lesions and 1956 melanomas), the RDOR for the observers where experience was NR (11 datasets; 2777 lesions and 465 melanomas), was 2.0 (95% CI 0.8 to 4.9), while the RDORs for the lower experience groups all suggested lower accuracy (Table 5; Figure 17). The RDORs for each group in comparison to the high‐experience group were: moderate experience 0.64 (95% CI 0.37 to 1.1; 5 datasets, 678 lesions and 193 melanomas); low experience 0.30 (95% CI 0.15 to 0.58; 6 datasets; 448 lesions and 123 melanomas); ‘mixed’ experience 0.25 (95% CI 0.07 to 0.81; 5 datasets, 473 lesions and 117 melanomas); and for the 'trained' group 0.15 (95% CI 0.25 to 1.02; 11 datasets, 1087 lesions and 240 melanomas).

We observed similar trends when we subgrouped evaluations according to reported observer qualifications, however data for clinicians other than consultant or ‘consultant experts’ were relatively sparse, especially for the in‐person evaluations where no statistically significant differences between groups was determined (Table 8; Figure 18). For the image‐based evaluations accuracy was highest for the ‘Expert consultant’ group (DOR 19.4, 95% CI 13.1 to 28.8; 33 datasets, 8664 lesions and 1854 melanomas; Figure 19). RDORs in comparison to the ‘expert’ group were 0.61 (95% CI 0.40 to 0.92), for observers described as ‘dermatologists’ (25 datasets; 4589 lesions and 955 melanomas), 0.31 (95% CI 0.14 to 0.71 for registrar (trainee), or resident‐level observers (5 datasets; 927 lesions and 138 observers), and 0.10 (95% CI 0.04 to 0.25), for the GP group (3 datasets; 288 lesions and 55 melanomas). Results for the GP group may simply be attributed to small sample sizes; however, we observed the lowest sensitivity for detection of melanoma (22%), for Bourne 2012, in which seven of nine malignancies were melanomas in situ or lentigo maligna, and we observed the lowest specificity (44%), for Piccolo 2014, which included a relatively high percentage of Spitz naevi (14% of the disease‐negative group), which may have been more difficult to differentiate from melanomas. Both studies also implemented blinded dermoscopy image interpretation whereas the third study in this group (Menzies 2005), also provided the clinical image and information on patient history to the interpreting clinicians.

1.1.3. Results of sensitivity analyses

In our generic protocol we planned a number of sensitivity analyses. We discussed one of these, restricting comparisons between dermoscopy and visual inspection alone to studies where the same study evaluated both tests (direct comparisons), alongside the main test comparisons above (Table 3). For completeness, we have included the results of these in Table 9 (in‐person evaluations), and Table 10 (image‐based evaluations), along with the results of all other sensitivity analyses.

8. Sensitivity analyses for in‐person visual inspection and dermoscopy added to visual inspection for the detection of invasive melanoma or atypical intraepidermal melanocytic variants.

Test	Studies	Lesions (cases)	DOR (95% CI)	Specificity at 80% sensitivity (95% CI) %	Sensitivity at 80% specificity (95% CI) %	RDOR (95% CI)
All in‐person evaluations
Visual inspection	13	6740 (459)	13.1 (7.0 to 24.5)	75% (57 to 87)	76% (66 to 85)	4.7 (3.0 to 7.5)
Visual inspection + dermoscopy	26	23,169 (1664)	61.7 (34.9 to 109)	95% (90 to 98)	92% (87 to 95)	‐
Change with adding dermoscopy to visual inspection (95% CI)	‐	‐	‐	+20% (+7 to +33)	+16% (+8 to +23)	‐
In‐person evaluations: direct comparison
Visual inspection	11	5854 (412)	13.7 (5.9 to 31.8)	75% (49 to 90)	77% (63 to 87)	4.8 (2.8 to 8.1)
Visual inspection + dermoscopy	11	5854 (412)	65.7 (27.0 to 160)	95% (87 to 99)	92% (84 to 96)	‐
Change with adding dermoscopy to visual inspection (95% CI)	‐	‐	‐	+21% (+2 to +39)	+15% (+7 to +23)	‐
In‐person evaluations: with histology and follow‐up for those not having surgery
Visual inspection	2	3607 (60)	18.4 (2.63 to 128)	82% (39 to 97)	82% (40 to 97)	14.4 (4.4 to 47.6)
Visual inspection + dermoscopy	6	17,574 (800)	265 (49 to 1428)	99% (91 to 100)	98% (87 to 100)	‐
Change with adding dermoscopy to visual inspection (95% CI)	‐	‐	‐	+16% (‐23 to 56)	+16% (‐20 to 53)	‐
In‐person evaluations with low risk of bias for the index test
Visual inspection	4	3957 (176)	16.9 (6.1 to 46.8)	80% (52 to 94)	80% (63 to 91)	3.1 (1.3 to 7.4)
Visual inspection + dermoscopy	20	19182 (831)	53.0 (25.8 to 109)	94% (87 to 98)	91% (84 to95)	‐
Change with adding dermoscopy to visual inspection (95% CI)	‐	‐	‐	+14% (‐6 to +34)	+11% (‐1 to +23)	‐
In‐person evaluations with low risk of bias for the reference test
Visual inspection	10	2802 (367)	13.8 (7.3 to 26.3)	76% (59 to 87)	77% (67 to 85)	4.2 (2.5 to 7.1)
Visual inspection + dermoscopy	20	7636 (1418)	57.8 (32.2 to 104)	95% (89 to 97)	92% (87 to 95)	‐
Change with adding dermoscopy to visual inspection (95% CI)	‐	‐	‐	+19% (+5 to +32)	+15% (+6 to +23)	‐
In‐person evaluations with low risk of bias for flow and timing
Visual inspection	2	601 (66)	11.0 (2.7 to 44.4)	61% (26 to 87)	73% (55 to 85)	5.1 (1.2 to 20.9)
Visual inspection + dermoscopy	4	984 (113)	55.7 (24.4 to 127)	95% (85 to 98)	88% (79 to 94)	‐
Change with adding dermoscopy to visual inspection (95% CI)	‐	‐	‐	+34% (‐45 to +100)	+16% (‐28 to +60)	‐
In‐person evaluations excluding case‐control studies
Visual inspection	13	6740 (459)	13.1 (7.0 to 24.5)	75% (57 to 87)	76% (66 to 85)	4.7 (3.0 to 7.5)
Visual inspection + dermoscopy	26	23,169 (1664)	61.7 (34.9 to 109)	95% (90 to 98)	92% (87 to 95)	‐
Change with adding dermoscopy to visual inspection (95% CI)	‐	‐	‐	+20% (+7 to +33)	+16% (+8 to +23)	‐
CI: confidence interval; DOR: diagnostic odds ratio; RDOR: relative diagnostic odds ratio

9. Sensitivity analyses for image‐based visual inspection or dermoscopy for the detection of invasive melanoma or atypical intraepidermal melanocytic variants.

Test	Studies	Lesions (cases)	DOR (95% CI)	Specificity at 80% sensitivity (95% CI) %	Sensitivity at 80% specificity (95% CI) %	RDOR (95% CI)
All image‐based evaluations
Clinical (macro) images	11	1740 (305)	3.2 (1.9 to 5.4)	42% (28 to 58)	47% (34 to 59)	5.6 (3.7 to 8.5)
Dermoscopic images	60	13 to 475 (2851)	17.8 (12.3 to 25.7)	82% (75 to 87)	81% (76 to 86)
Change replacing visual inspection with dermoscopy (95% CI)	‐	‐	‐	+40% (+27 to 57)	+34% (+24 to +46)	‐
Image‐based evaluations: direct studies
Clinical (macro) images	11	1740 (305)	3.6 (1.7 to 7.6)	48% (25 to 73)	47% (30 to 64)	5.3 (3.5 to 8.0)
Dermoscopic images	11	1735 (306)	19.2 (8.7 to 42.0)	83% (70 to 91)	83% (68 to 92)
Change replacing visual inspection with dermoscopy (95% CI)	‐	‐	‐	+34% (+15 to +53)	+36% (+20 to +52)	‐
Image‐based evaluations: with histology and follow‐up for those not having surgery
Clinical (macro) images	0	‐	‐	‐	‐	‐
Dermoscopic images	7	2612 (523)	7.4 (4.5 to 12.0)	67% (58 to 75)	57% (39 to 74)
Change replacing visual inspection with dermoscopy (95% CI)	‐	‐	‐	‐	‐	‐
Image‐based evaluations with low risk of bias for the index test
Clinical (macro) images	3	1113 (117)	1.9 (0.91 to 4.0)	+32% (17 to 52)	32% (17 to 52)	10.4 (5.7 to 19.0)
Dermoscopic images	40	11,194 (2318)	19.8 (12.4 to 31.7)	83% (76 to 89)	83% (76 to 89)
Change replacing visual inspection with dermoscopy (95% CI)	‐	‐	‐	+51% (+35 to +68)	+51% (+34 to +67)	‐
Image‐based evaluations with low risk of bias for the reference test
Clinical (macro) images	9	1650 (276)	3.2 (1.9 to 5.4)	42% (28 to 58)	47% (34 to 59)	5.6 (3.7 to 8.5)
Dermoscopic images	51	10,894 (2359)	17.8 (12.3 to 25.8)	82% (75 to 87)	81% (76 to 86)
Change replacing visual inspection with dermoscopy (95% CI)	‐	‐	‐	+40% (+26 to +53)	+34% (+24 to +46)	‐
Image‐based evaluations with low risk of bias for flow and timing
Clinical (macro) images	1	53 (10)	15.9 (1.6 to 161)	79% (21 to 98)	80% (34 to 97)	0.54 (0.05 to 5.5)
Dermoscopic images	11	1391 (410)	8.6 (4.4 to 16.7)	65% (42 to 83)	69% (56 to 80)
Change replacing visual inspection with dermoscopy (95% CI)	‐	‐	‐	‐14% (‐67 to +39)	‐10% (‐48 to +28)	‐
Image‐based evaluations excluding case‐control studies
Clinical (macro) images	7	964 (183)	7.2 (3.5 to 14.8)	62% (40 to 80)	66% (50 to 78)	3.4 (1.8 to 6.4)
Dermoscopic images	37	10,270 (1923)	24.3 (15.2 to 39.0)	86% (79 to 91)	85% (79 to 90)
Change replacing visual inspection with dermoscopy (95% CI)	‐	‐	‐	+24% (+4 to +44)	+20% (+7 to +32)	‐
CI: confidence interval; DOR: diagnostic odds ratio; RDOR: relative diagnostic odds ratio

In‐person evaluations

Analyses restricting studies to those avoiding partial verification (including only those that allowed histology or follow‐up), increased the relative benefit from adding dermoscopy to visual inspection from an RDOR of 4.7 (95% CI 3.0 to 7.5), to 14.4 (95% CI 4.4 to 47.6), however study numbers were small and the increase in sensitivity at 80% specificity and in specificity at 80% sensitivity remained similar (Table 9). We observed limited differences for the analyses restricting studies to those with low risk of bias for the index test or low risk of bias for the reference standard. An additional post hoc analysis restricting studies to those with low risk of bias for flow and timing resulted in small study numbers and did not appear to have a large impact on accuracy. Planned analyses restricting to studies with at least a three‐month interval between the index test and the reference standard, and where concerns around applicability for participant selection were low, were not possible due to lack of studies.

Image‐based evaluations

There were small study numbers for visual inspection in comparison to the numbers for dermoscopy, so it was more difficult to interpret sensitivity analyses for image‐based evaluations in terms of the differences between diagnosis using dermoscopic images versus visual inspection of images, for example, for restriction to those that allowed histology or follow‐up as a reference standard (7 datasets for dermoscopy compared to 0 for visual inspection), for low risk of bias for the index test (40 for dermoscopy versus 3 for visual inspection), and for low risk of bias for flow and timing (11 datasets for dermoscopy versus 1 for visual inspection; Table 10). Restriction to studies with low risk of bias for the reference standard made very little difference to the accuracy of either test or to the RDOR for dermoscopy versus visual inspection.

Again, due to lack of studies, we could not carry out planned analyses restricting to studies with at least a three‐month interval between the index test and the reference standard, and where concerns around applicability for participant selection were low.

An additional post hoc sensitivity analysis restricting studies to those that did not use a case‐control design increased the accuracy of visual inspection of images from a DOR of 3.2 (95% CI 1.9 to 5.4), to DOR 7.2 (95% CI 3.5 to 14.8), for the seven remaining datasets, and increased the DOR for diagnosis using dermoscopic images from 17.8 (95% CI 12.3 to 25.7) for all 60 datasets to 24.3 (95% CI 15.2 to 39.0) for the remaining 37 datasets; the RDOR between tests reduced from 5.6 (95% CI 3.7 to 8.5), to 3.4 (95% CI 1.8 to 6.4) (Table 10). From the sensitivities and specificities estimated from SROC curves, this fall in RDOR appears to be primarily related to an increase in accuracy for diagnosis based on visual inspection of images rather than a fall in accuracy for dermoscopic examination, due to the exclusion of case‐control studies. The direction of this finding is contrary to the standard expectation that case‐control studies overestimate test accuracy compared to other designs (Rutjes 2006).

2. Target condition: invasive melanoma only

In this section we present the results for studies of dermoscopy for the identification of invasive melanoma, according to the approach taken for diagnosis: in‐person or image‐based evaluations. Summary characteristics of studies are presented in Appendix 11, with forest plots of study data in Appendix 12 and results of meta‐analyses in Table 11, and Figure 20 and Figure 21.

10. Comparison of visual inspection and dermoscopy for the detection of invasive melanoma.

Test	Studies	Lesions (cases)	DOR (95% CI)	Specificity at 80% sensitivity (95% CI) %	Sensitivity at 80% specificity (95% CI) %	RDOR (95% CI)	P value (LR)
In‐person evaluations
Visual inspection	2	147 (51)	20.8 (6.0 to 72.5)	84% (66 to 93)	84% (57 to 95)	6.2 (1.5 to 26.6)	0.015
Visual inspection + dermoscopy	6	789 (115)	129 (19.2 to 870)	97% (94 to 98)	97% (46 to 100)
Difference (95% CI)	‐	‐	‐	+13% (‐1 to +27)	+13% (‐0 to +27)	‐	‐
In‐person evaluations: direct studies
Visual inspection	2	147 (51)	20.1 (4.0 to 101)	75% (23to 97)	78% (64 to 88)	11.3 (1.4 to 689.8)	0.015
Visual inspection + dermoscopy	2	147 (51)	226 (21.7 to 2358)	99% (54 to 100)	94% (72 to 99)
Difference (95% CI)	‐	‐	‐	+24% (‐21 to +69)	+15% (+2 to +29)	‐	‐
Image‐based evaluations
Clinical (macro) images	4	454 (145)	11.0 (4.1 to 29.3)	74% (52 to 88)	72% (49 to 88)	2.5 (1.2 to 5.1)	0.032
Dermoscopic images	13	5618 (1092)	27.5 (12.2 to 61.7)	87% (75 to 94)	88% (75 to 94)
Difference (95% CI)	‐	‐	‐	+13% (‐1 to +28)	+15% (‐1 to +30)	‐	‐
Image‐based evaluations: direct studies
Clinical (macro) images	4	454 (145)	11.9 (3.4 to 40.9)	45% (5 to 92)	72% (59 to 82)	3.4 (1.0 to 11.1)	0.049
Dermoscopic images	4	454 (145)	40.4 (8.2 to 198)	89% (47 to 99)	83% (72 to 90)
Difference (95% CI)	‐	‐	‐	+44% (‐20 to +100)	+11% (+1 to +22)	‐	‐
CI: confidence interval; DOR: diagnostic odds ratio; LR: likelihood ratio test; RDOR: relative diagnostic odds ratio

20.

Comparison of the accuracy of visual inspection with visual inspection (VI) + dermoscopy for detection of invasive melanoma from in‐person studies

21.

Comparison of the accuracy of visual inspection with dermoscopy for detection of invasive melanoma from image‐based studies

Dermoscopy added to visual inspection of a skin lesion (in‐person evaluations)

Six studies evaluated the accuracy of in‐person dermoscopy for the detection of invasive melanoma only, one of which also reported data for the primary target condition (Feldmann 1998), and two of which presented data for visual inspection (Krahn 1998; Viglizzo 2004). All studies were case series based in secondary care or specialist units apart from Coras 2003, which was based in a private dermatology clinic. All the studies recruited participants with pigmented lesions, Viglizzo 2004 restricting to melanocytic lesions only. Four studies did not report using any formal algorithm to assist dermoscopy diagnosis (Coras 2003; Krahn 1998; Piccolo 2000; Viglizzo 2004); Feldmann 1998 used the ABCD checklist and Ascierto 2010 used a modified version of the Kenet risk stratification approach (referenced to Ascierto 1998). The prevalence of melanoma ranged from 5% (Feldmann 1998), to 49% (Krahn 1998). All studies used a histological reference standard.

The sensitivity of in‐person dermoscopy ranged from 64% to 100% and specificities ranged from 93% to 98% (Appendix 12). In meta‐analysis the DOR was 129 (95% CI 19.2 to 870; 789 lesions and 115 melanomas). The specificity of in‐person dermoscopy at 80% fixed sensitivity was 97% (95% CI 94% to 98%), and sensitivity at 80% fixed specificity was also 97% (95% CI 46% to 100%; Table 11). Again, these sensitivities and specificities at fixed values should be taken as illustrative of the data observed.

In Feldmann 1998, the sensitivity for the detection of invasive melanoma alone was 11% higher compared to sensitivity for the detection of invasive melanoma or atypical intraepidermal melanocytic variants (64% vs 53%), because the 5 included melanoma in situ lesions were all classified as negative for melanoma on dermoscopy and were classed as true‐negative results for the detection of invasive melanoma alone.

Incremental accuracy from dermoscopy added to in‐person visual inspection alone

The two studies providing direct comparisons of visual inspection alone and visual inspection + dermoscopy reported using no algorithm to assist visual inspection diagnosis (Krahn 1998; Viglizzo 2004). Based on reported authors' institutions, we assumed that observers in both studies were dermatologists.

Sensitivities for visual inspection were 79% (Krahn 1998), and 67% (Viglizzo 2004); specificities were 78% and 95%, respectively (Appendix 12). We compared the accuracy of visual inspection with the accuracy of dermoscopy estimated from (a), all six dermoscopy studies (789 lesions and 115 melanomas), and both in‐person visual inspection studies (147 lesions and 51 melanomas), and estimated from (b), direct comparisons in the subset of two studies that evaluated both visual inspection and dermoscopy on an in‐person basis (147 lesions and 51 melanomas). In both comparisons the accuracy of dermoscopy added to visual inspection exceeded that of visual inspection alone (Table 11). In (a), the DOR for dermoscopy was 6.2 (95% CI 1.5 to 26.6; P = 0.015), times that of visual inspection alone, in (b), it was 11.3 (95% CI 1.4 to 689.8; P = 0.015), times that of visual inspection alone. These effects correspond to predicted differences in specificity of (a), 13% (95% CI ‐1% to 27%), based on sensitivity with dermoscopy of 97% vs 84% for visual inspection and (b), 24% (95% CI ‐21% to 69%), based on sensitivity with dermoscopy of 99% vs 75% for visual inspection at a fixed sensitivity of 80% (Table 11); and predicted differences in sensitivity of (a), 13% (95% CI ‐0% to 27%), based on specificity with dermoscopy of 97% vs 84% for visual inspection and (b), 15% (95% CI 2% to 29%), based on specificity with dermoscopy of 94% vs 78% for visual inspection at a fixed specificity of 80% (Table 11).

Dermoscopic images (image‐based evaluations)

Thirteen datasets reported the accuracy of image‐based dermoscopy for the detection of invasive melanoma, none of which reported data for the primary target condition. Eight evaluations included series of lesions observed in secondary care or specialist clinic settings (prevalence 10% to 36%). The remaining five evaluations used a case‐control type design, with separate sampling of melanoma and benign lesion images. Prevalence ranged from 27% to 65%. Studies used the ABCD checklist (Lorentzen 2000; Menzies 2013), the Menzies algorithm (Arevalo 2008; Menzies 1996; Westerhoff 2000), or their own algorithm (Kreusch 1992; Nilles 1994), to assist dermoscopic diagnosis. Six evaluations did not report using any algorithm to assist diagnosis.

Five evaluations presented only the dermoscopic image with no further patient information (Arevalo 2008; Lorentzen 2008; Menzies 1996; Nilles 1994; Troyanova 2003), five presented observers with a concurrent clinical image of the lesion (Hauschild 2014; Lorentzen 1999a; Lorentzen 2000; Rao 1997; Westerhoff 2000); two provided only lesion site (Kreusch 1992), or site, age and gender (Friedman 2008), and one did not describe any further information (Menzies 2013). Images were interpreted by dermatologists or assumed‐to‐be dermatologists in 10 studies, by dermatologists or melanoma fellows in Rao 1997, by GPs in Westerhoff 2000 and by mixed secondary care clinicians in Friedman 2008.

Sensitivities ranged from 48% to 100%, specificities ranged from 49% to 97% (Appendix 12). In meta‐analysis the DOR was 27.5 (95% CI 12.2 to 61.7; 5618 lesions and 1092 melanoma cases). Specificity at 80% fixed sensitivity was 87% (95% CI 75% to 94%), and sensitivity at 80% fixed specificity was 88% (95% CI 75% to 94%), (Table 11).

Incremental accuracy of dermoscopic image‐based diagnosis compared to visual inspection of images

The four studies providing direct comparisons of diagnosis based on clinical images and diagnosis based on dermoscopic images reported using no algorithm to assist visual inspection diagnosis (Lorentzen 1999a; Troyanova 2003; Westerhoff 2000), or use of the ABCD algorithm (Rao 1997). Observers were dermatologists (Lorentzen 1999a; Troyanova 2003), a melanoma fellow (Rao 1997), or GPs (Westerhoff 2000).

Sensitivities for visual inspection ranged from 62% to 86%; and specificities from 54% to 89%, respectively (Appendix 12). We compared the accuracy of visual inspection with the accuracy of dermoscopy estimated from (a), all 13 dermoscopy studies (5618 lesions and 1092 melanomas), and the four visual inspection studies (454 lesions and 145 melanomas), and estimated from direct comparisons in (b), with the subset of four studies that evaluated both visual inspection and dermoscopy on an image‐based basis (454 lesions and 145 melanomas). In both comparisons the accuracy of diagnosis based on dermoscopic images exceeded that based on clinical images (Table 11). In (a), the DOR for dermoscopy was 2.5 (95% CI 1.2 to 5.1; P = 0.032), times that of visual inspection alone, in (b), it was 3.4 (95% CI 1.0 to 11.1; P = 0.049), times that of visual inspection alone. These effects correspond to predicted differences in specificity of (a), 13% (95% CI ‐1% to 28%), based on sensitivity with dermoscopy of 87% vs 74% for visual inspection and (b), 44% (95% CI ‐20% to 100%), based on sensitivity with dermoscopy of 89% vs 45% for visual inspection at a fixed sensitivity of 80% and predicted differences in sensitivity of (a), 15% (95% CI ‐1% to 30%), based on specificity with dermoscopy of 88% vs 72% for visual inspection and (b), 11% (95% CI 1% to 22%), based on specificity with dermoscopy of 83% vs 72% for visual inspection at a fixed specificity of 80% (Table 11).

3. Target condition: any skin lesion requiring excision

In this section we present the results for studies of visual inspection for the identification of any skin lesion requiring excision, according to the approach taken for diagnosis: in‐person or image‐based evaluations. For each study we could only extract data for the detection of any skin cancer. We have presented summary characteristics of studies in Appendix 13, with forest plots of study data in Figure 22 and Figure 23, and results of meta‐analyses in Table 12. Heterogeneity was too high and data too sparse to allow us to make formal statistical comparisons between tests, thus the analysis focuses on describing the observed accuracy. Only meta‐analytical models assuming underlying symmetric SROC curves could be fitted to these data.

22.

Forest plot of tests. 9 Visual inspection ‐ in‐person (any skin cancer), 10 VI+dermoscopy ‐ in‐person (any skin cancer)

23.

Forest plot of tests. 11 Visual inspection ‐ image‐based (any skin cancer), 12 dermoscopy alone ‐ image‐based (any skin cancer)

11. Comparison of visual inspection and dermoscopy for the detection of any skin lesion requiring excision.

Testa	Studies	Lesions (cases)	DOR (95% CI)	Specificity at 80% sensitivity (95% CI) %	Sensitivity at 80% specificity (95% CI) %
In‐person evaluations
Visual inspection	2	3457 (151)	38.4 (2.5 to 582)	91% (39 to 99)	91% (39 to 99)
Visual inspection + dermoscopy	4	3880 (260)	232 (16.0 to 3354)	98% (80 to 100)	98% (80 to 100)
In‐person evaluations: direct studies
Visual inspection	2	3457 (151)	15.0 (0.18 to 1225)	79% (4 to 100)	79% (4 to 100)
Visual inspection + dermoscopy	2	3449 (137)	88.1 (1.1 to 7338)	96% (21 to 100)	96% (21 to 100)
Image‐based evaluations
Clinical (macro) images	3	547 (138)	21.7 (4.8 to 98.9)	84% (54 to 96)	84% (54 to 96)
Dermoscopic images	5	815 (217)	37.5 (8.8 to 161)	90% (69 to 98)	90% (69 to 98)
Image‐based evaluations: direct studies
Clinical (macro) images	3	547 (138)	12.1 (5.4 to 26.7)	75% (58 to 87)	75% (58 to 87)
Dermoscopic images	3	546 (136)	18.4 (8.1 to 41.7)	82% (67 to 91)	82% (67 to 91)
CI: confidence interval; DOR: diagnostic odds ratio

^aEstimates are based on fitting models with symmetric receiver operating characteristic (ROC) curves, and no formal comparisons between tests are made due to paucity of data. It is noted that the estimates for the visual inspection studies change between the all data and paired data analyses for both in‐person and image‐based analyses. This is driven by differences in the heterogeneity in accuracy between the models, which affects all parameters in the analyses.

Dermoscopy added to visual inspection of a skin lesion (in‐person evaluations)

Four datasets evaluated the accuracy of in‐person dermoscopy for the detection of any skin lesion requiring excision (Argenziano 2006; Durdu 2011; Soyer 2004; Stanganelli 2000), one of which also reported data for the primary outcome (Durdu 2011), and two reported data for visual inspection alone (Argenziano 2006; Stanganelli 2000). Studies were based in primary care, with diagnosis by GPs (Argenziano 2006), or secondary care or specialist referral clinics, with diagnosis by dermatologists. The prevalence of skin cancer ranged from 3% in Stanganelli 2000 to 51% in Argenziano 2006. Studies used the ABCD algorithm (Durdu 2011), the 3PCL (Argenziano 2006), pattern analysis (Stanganelli 2000), or no algorithm (Soyer 2004), to assist diagnosis. Stanganelli 2000 supplemented a histological reference standard with clinical follow‐up, and the others reported data compared to histology alone.

Sensitivities ranged from 85% to 98%; specificities ranged from 26% to 100% (Figure 22). In meta‐analysis the DOR was 232 (95% CI 16.0 to 3354; 3880 lesions and 260 skin cancer cases) (Table 12). We could not make any formal comparison with in‐person visual inspection due to heterogeneity and sparsity of data; however, the DOR for the two studies reporting data for visual inspection alone (3457 lesions and 151 skin cancers), was 15.0 (95% CI 0.18 to 1225; Argenziano 2006; Stanganelli 2000), compared to 88.1 (95% CI 1.1 to 7338), for in‐person dermoscopy in these same two studies (3449 lesions and 137 skin cancers; the total number of lesions and melanomas differs because Argenziano 2006 was a between‐person comparison study with a different number of lesions randomised to each arm). Sensitivities at 80% fixed specificity and specificities at 80% fixed sensitivity were both 17% higher using dermoscopy (both 96% with dermoscopy compared to 79% for visual inspection alone), due to the use of symmetric ROC curves for these analyses.

We observed the lowest sensitivity and specificity for dermoscopy in Argenziano 2006, however we could only include 2x2 data for the GP diagnosis using the 3PCL for the 77 lesions selected for excision by an expert dermatologist, as the remaining 1126 for which GP diagnosis was recorded did not have an adequate reference standard for inclusion in our review. In Durdu 2011 specificity estimates were not affected by the wider definition of the target condition; however, sensitivity increased from 80% for detection of melanoma or atypical intraepidermal melanocytic variants to 98% for detection of any lesion requiring excision, as all 34 BCCs were correctly identified.

Dermoscopic images (image‐based evaluations)

Five datasets reported the accuracy of image‐based visual inspection for the detection of any skin lesion requiring excision (Carli 2002b; Lorentzen 2008; Rosendahl 2011; Stanganelli 1998a; Zalaudek 2006), all of which also reported data for the primary target condition or for the detection of invasive melanoma alone (Lorentzen 2008), and three of which reported data for diagnosis based on clinical images (Carli 2002b; Rosendahl 2011; Stanganelli 1998a). Studies selected images from secondary care clinics or specialist units (Carli 2002b; Lorentzen 2008; Stanganelli 1998a; Zalaudek 2006), or from a primary care practice (Rosendahl 2011). The prevalence of lesions suitable for excision ranged from 22% (Rosendahl 2011), to 47% (Stanganelli 1998a); the latter selecting images for use in a dermoscopy training study. Diagnosis was based on the 3PCL (Zalaudek 2006), pattern analysis (Rosendahl 2011), or no formal algorithm. Data were presented for a single dermatologist (Rosendahl 2011), for a consensus of two dermatologists (Carli 2002b), for the average across 20 dermatologists (Stanganelli 1998a), or 150 dermatologists (Zalaudek 2006), or was not clearly reported (Lorentzen 2008). Observers were also provided with the clinical image for the same lesion (Rosendahl 2011; Stanganelli 1998a), with lesion site, and patient age and gender (Zalaudek 2006), or with no further clinical information to assist diagnosis (Carli 2002b; Lorentzen 2008).

Sensitivities ranged from 78% to 100%; specificities ranged from 72% to 96% (Figure 23). In meta‐analysis the DOR was 37.5 (95% CI 8.8 to 161; 815 lesions and 217 skin cancer cases). We could not make any formal comparison with diagnosis based on clinical images due to heterogeneity and sparsity of data, however the DOR for the three studies reporting image‐based visual inspection (547 lesions and 138 skin cancers), was 12.1 (95% CI 5.4 to 26.7; Carli 2002b; Rosendahl 2011; Stanganelli 1998a), compared to 18.4 (95% CI 8.1 to 41.7), for image‐based dermoscopy in these same three studies. Sensitivities at 80% fixed specificity and specificities at 80% fixed sensitivity were 7% higher using dermoscopy (both 82% with dermoscopy compared to 75% for visual inspection of clinical images).

The wider definition of the target condition to include any skin lesion requiring excision led to increased sensitivities and lower specificities in three studies due to classification of BCCs as true positives rather than false negatives (Carli 2002b; Rosendahl 2011; Stanganelli 1998a). We also extracted data from Rosendahl 2011 and Stanganelli 1998a for the correct diagnosis of any malignancy rather than correct diagnosis of each individual type of skin cancer, which led to considerable increased in sensitivity in both studies.

4. Evaluations of dermoscopy training

Six studies evaluated observer accuracy using dermoscopy before and after a dermoscopy training intervention. Two studies reported data for detection of invasive melanoma alone (Troyanova 2003; Westerhoff 2000), and four reported data for the detection of invasive melanoma and atypical intraepidermal melanocytic variants (Pagnanelli 2003; Piccolo 2014; Stanganelli 1999; Tan 2009). A further 14 studies reported the delivery of some form of dermoscopy training, either prior to the study commencing (Kittler 1998; Seidenari 2007), or within the context of the study itself (Argenziano 1998; Argenziano 2006; Binder 1999; Carli 2003a; Dolianitis 2005; Grimaldi 2009; Kittler 1998; Menzies 2008; Menzies 2009; Seidenari 2007; Stanganelli 1998a; Zalaudek 2006). Six of the latter group of studies compared the accuracy of diagnosis based on visual inspection alone (pre‐dermoscopy training), to visual inspection and dermoscopy (post‐dermoscopy training); we have incorporated these data into the visual inspection versus dermoscopy comparisons reported above.

We have shown details of the training interventions provided in the six eligible studies in Appendix 14 and reported results of the analyses in Table 13 and Figure 24. All studies were image‐based evaluations.

12. Accuracy of dermoscopy before versus after dermoscopy training (all image‐based).

Test	Studies	Lesions (cases)	DOR (95% CI)	Specificity at 80% sensitivity (95% CI) %	Sensitivity at 80% specificity (95% CI) %	RDOR (95% CI)	P‐value (LR)
Detection of invasive melanoma or atypical intraepidermal melanocytic variants
Before training	4	245 (65)	6.3 (1.68 to 23.5)	62% (27 to 88)	60% (30 to 84)	1.4 (0.38 to 5.3)	< 0.001
After training	4	245 (65)	8.9 (2.4 to 33.3)	69% (40 to 88)	69% (33 to 91)
Change with training (95% CI)	‐	‐	‐	+8% (‐24 to +40)	+8% (‐19 to +36)	‐	‐
Detection of invasive melanoma
Before training	2	150 (75)	5.2 (0.95 to 28.7)	50% (9 to 91)	60% (25 to 87)	3.2 (0.94 to 10.6)	0.051
After training	2	150 (75)	16.4 (2.6 to 103)	80% (32 to 97)	80% (47 to 95)
Change with training (95% CI)	‐	‐	‐	+29% (‐24 to +82)	+20% (‐5 to +45)	‐	‐
CI: confidence interval; DOR: diagnostic odds ratio; LR: likelihood ratio test; RDOR: relative diagnostic odds ratio

24.

Forest plot of tests. Accuracy of dermoscopy before and after dermoscopy training (MM and invasive melanoma or atypical intraepidermal melanocytic variants)

For the detection of invasive melanoma or atypical intraepidermal melanocytic variants, all four evaluations (Pagnanelli 2003; Piccolo 2014; Stanganelli 1999; Tan 2009), demonstrated an increase in the average sensitivity of dermoscopy of between 13% and 15% (pre‐training sensitivity ranged from 36% to 80% and post‐training from 73% to 93%). We observed no change in average specificity following dermoscopy training for three studies, and specificity fell from 70% pre‐training to 44% post‐training in Piccolo 2014. The pooled analysis showed no impact on accuracy from dermoscopy training (RDOR 1.4, 95% CI 0.38 to 5.3).

Three of the four studies reported the training of dermatologists (n = 83 in Stanganelli 2000), or of a mixed group of dermatologists, registrars or residents (n = 16 in Pagnanelli 2003; n = 6 in Tan 2009). Pagnanelli 2003 also included three medical students in their group of "16 trainees". These three studies provided web‐based interactive training (Pagnanelli 2003; Tan 2009), with an expectation of a time commitment of one hour per day for two weeks (Pagnanelli 2003), or with a dermatoscope provided for use in clinical practice for 10 months between tests (Tan 2009), or in‐person dermoscopy training workshops (Stanganelli 1999). In Piccolo 2014 however, the ‘trainee’ was a single GP who undertook similar online training using an interactive atlas of dermoscopy, which may explain the outlying result for specificity.

For the detection of invasive melanoma, both evaluations demonstrated an increase in the average sensitivity of dermoscopy in the order of 16% to 18% following dermoscopy training (from 76% to 92% in Troyanova 2003’s study of 32 dermatologists and from 58% to 76% in Westerhoff 2000’s study of 74 GPs), with minimal impact on specificity (84% before and after training in Troyanova 2003 and 56% before and 58% after training in Westerhoff 2000). The pooled analysis showed a non‐statistically significant increase in accuracy after training of 3.2 times that before training (95% CI 0.94 to 10.6, P = 0.05; 150 lesions and 75 cases). As well as the differences in clinician qualifications, the content and duration of the training programmes also varied. Troyanova 2003 provided six hours of in‐person teaching daily for two consecutive days; the test using clinical and dermoscopic images of 50 lesions was undertaken at the beginning and at the end of the course. In Westerhoff 2000, GPs were provided with a pictorial atlas outlining the Menzies approach to dermoscopic diagnosis and given a one‐hour presentation on the method; the pre‐ and post‐tests were undertaken at the leisure of the individual GPs.

Discussion

Summary of main results

The included studies evaluated dermoscopy to assist the diagnosis of melanoma in a range of study populations, on an in‐person basis added to visual inspection of a skin lesion and using dermoscopic images, and both with and without the use of published algorithms to assist diagnosis. We observed wide variations in both sensitivity and specificity for dermoscopy use for all definitions of the target condition. In terms of methodological quality, many studies were at high or unclear risk of bias for participant selection and for timing of diagnosis in relation to reference standard diagnosis, but were at low risk of bias for the index test and reference standard. Concern around the applicability of studies was almost universally high due to restricted inclusion of lesions (for example inclusion of only melanocytic lesions or of lesions selected for excision based on the clinical or dermoscopic diagnosis), and lack of reproducibility of diagnostic thresholds. Poor reporting in the primary studies hindered attempts to analyse studies according to their position on the clinical pathway and to fully assess sources of heterogeneity and methodological quality.

In this review we have estimated the incremental accuracy of dermoscopy in comparison to visual inspection using summary ROC curves rather than by estimating average sensitivity and specificity operating points. We have reported points from the fitted SROC curves (the sensitivity at 80% specificity, and the specificity at 80% sensitivity), however these are for illustrative purposes and should not be quoted as the actual performance of dermoscopy. Whilst it may not be possible to estimate the absolute accuracy of dermoscopy, nor to make any clear recommendations to ensure that dermoscopy is used in such a way as to maximise sensitivity, we can make a strong comparison between dermoscopy and visual inspection alone despite the limitations and heterogeneity of included studies, particularly from the studies that make within‐patient comparisons between diagnostic strategies of visual inspection alone, and visual inspection supplemented by dermoscopy. We have also presented results separately for in‐person and image‐based studies, as we observed clear differences in their findings. We chose to emphasise the in‐person findings over the image‐based studies as these are more applicable to typical practice.

Thus, whilst we cannot answer the overall question of how accurate dermoscopy is, we are able to assess the incremental gain in accuracy of using dermoscopy, and identify some characteristics that increase or decrease its accuracy.

Five main findings can be drawn from our review:

1) On average, the addition of dermoscopy to in‐person visual inspection of a lesion increases both sensitivity and specificity by a considerable margin.

Approximately one third of eligible studies presented data for in‐person dermoscopy (26 of 86), for the primary target condition of invasive melanoma or atypical intraepidermal melanocytic variants. Studies included a range of study populations and used a number of different algorithms to assist interpretation, such that we observed considerable heterogeneity in both sensitivity and specificity for both visual inspection alone and for visual inspection + dermoscopy. The Table 1 presents key results and translates summary estimates to a hypothetical cohort of 1000 lesions.

Sensitivity: at a fixed specificity of 80%, the use of dermoscopy increased the sensitivity of in‐person visual inspection by 16%, from 76% to 92%. Assuming melanoma or atypical intraepidermal melanocytic variant prevalences of 5%, 12% and 21%, a test sensitivity of 92% with the added use of dermoscopy would reduce the number of melanomas missed in comparison to using visual inspection alone by 8, 19 and 33 (resulting in 4, 10 and 17 melanomas missed). An assumed test specificity of 80% (for both visual inspection and visual inspection + dermoscopy), would result in 190, 176 and 158 false‐positive test results (or unnecessary excisions).

Specificity: at a fixed sensitivity of 80%, the use of dermoscopy increased the specificity of in‐person visual inspection by 20%, from 75% to 95%. Applying these results to a cohort of 1000 lesions at the same three prevalences of disease, both tests would miss between 10 and 42 melanomas, with the addition of dermoscopy reducing false positives (or reducing the number of excisions that would be performed), by 191, 176 and 158 per 1000 (compared with 238, 220 and 198 unnecessary excisions with visual inspection alone).

We noted very similar findings between the analysis of all studies, and the analyses restricted to studies that made within‐person comparisons of strategies of visual inspection alone and visual inspection aided by dermoscopy. The same difference was evident for our secondary analyses for the detection of invasive melanoma alone and for the detection of any skin lesion requiring excision.

2) In‐person dermoscopy is substantially more accurate than image‐based assessments

Much of the available evidence for the diagnostic accuracy of dermoscopy is based on the interpretation of dermoscopic images (60 of 86), as opposed to ‘real time’ diagnosis, face‐to‐face with the patient concerned. Formal comparison of test accuracy found in‐person dermoscopy to be substantially more accurate compared to diagnosis based on dermoscopic images (RDOR 4.6, 95% CI 2.4 to 9.0; P < 0.001). Although there may be a number of contributing factors, including differences in study populations, different algorithms to assist test interpretation and differences in observer experience, it is likely that, as for visual inspection of a clinical image (Dinnes 2018a), remote test interpretation cannot approximate a physical, face‐to‐face patient to clinician interaction. In particular, total body skin examination is likely to have a significant impact on the decision to excise a lesion suspected to be melanoma (Argenziano 2012; Aldridge 2013; Grob 1998). Across the 60 image‐based evaluations, half (30 of 60), were blinded to all other patient information and only 17 (28%), provided observers with the clinical image of the same lesion to assist test interpretation.

Nevertheless, given the increasing trend towards remote test interpretation (or teledermatology), it is important to try to understand the potential impact from image‐based assessments. From the data observed, at a fixed specificity of 80%, diagnosis based on dermoscopic images was 34% more sensitive than diagnosis based on clinical images alone (an increase from 47% to 81% sensitivity). Assuming melanoma or atypical intraepidermal melanocytic variant prevalences of 18%, 24% and 39%, these results translate to 164, 152 and 122 false‐positive test results, with 34, 46 and 74 melanomas missed (false negatives), using dermoscopic images (a reduction of 61, 81 and 133 compared to diagnosis based on clinical images alone). At a fixed sensitivity of 80%, test specificity for diagnosis based on dermoscopic images would be 40% higher compared to that based on clinical images (specificity of 82% compared to 42%). Applying these results to a cohort of 1000 lesions would miss between 36 and 78 melanomas, with 148, 137 and 110 false‐positive results based on dermoscopic image interpretation (a reduction of 328, 304 and 244 in comparison to the evaluation of clinical images alone).

A post hoc analysis restricting study inclusion to those that did not use a case‐control design appeared to increase the accuracy of image‐based visual inspection and, to a lesser extent, the accuracy of diagnosis based on dermoscopic images. Nevertheless the observed accuracy of in‐person dermoscopy was still greater than that using dermoscopic images. It is also important to note that none of the included image‐based dermoscopy evaluations purported to be an evaluation of teledermatology. Such evaluations are included in a separate systematic review of teledermatology for the diagnosis of skin cancer (Chuchu 2018a). Although the results for image‐based dermoscopy from this review have some bearing on the accuracy that might be achieved by the remote assessment of dermoscopic images, we suggest that future studies should not be undertaken that evaluate dermoscopic images to approximate to in‐person evaluation. We have retained the image‐based studies in the review as they do enable comparisons of different aspects of dermoscopic diagnosis (see below), but they could potentially be excluded from future reviews.

3) We could determine no effect from prior testing of participants or study position on the clinical pathway, and there is insufficient evidence to assess the accuracy of dermoscopy in a primary care setting.

Less than half of in‐person evaluations (42%; 11 of 26), and only 18% of image‐based evaluations (11 of 60), contributing to analyses for the primary target condition contained enough information to describe the position of participants on the clinical pathway. This figure is lower than for our review of visual inspection for the detection of melanoma, where two‐thirds of in‐person evaluations were clearly positioned on the clinical pathway (Dinnes 2018a). The majority of evaluations of dermoscopy however appear to have been conducted in referral settings, with only four eligible studies conducted in primary care populations; two in‐person evaluations and two image‐based, thus our planned comparison between initial presentation versus referred patients is underpowered. Within the referred population studies there was some (largely non‐significant), indication of higher accuracy in equivocal lesions and lower accuracy in studies of patients with lesions undergoing follow‐up, particularly in image‐based studies. The classification of study populations was dependent on the terminology used by the study authors and the groupings may not fully reflect differences between study populations.

4) There is no clear evidence that accuracy is improved by the use of any named or published algorithm to assist diagnosis.

The use of a named or published algorithm to assist dermoscopy interpretation (as opposed to no reported algorithm or reported use of pattern analysis), had no significant impact on accuracy either for in‐person (RDOR 1.4, 95% CI 0.34 to 5.6; P = 0.17), or image‐based (RDOR 1.4, 95% CI 0.60 to 3.3; P = 0.22), evaluations. This result was supported by subgroup analysis according to algorithm used. Although the vast majority of data comparing algorithms came from image‐based evaluations there is no reason to suggest that the relative accuracy of different approaches to diagnosis would vary according to whether the evaluation was image‐based as opposed to in person, even if in absolute terms accuracy was higher for the latter group of studies.

In this instance, we were able to pool data separately according to algorithm and threshold used, therefore we used the bivariate normal model rather than the summary ROC approach. For in‐person evaluations most of the data related to no algorithm (8 datasets), to pattern analysis (6 datasets), or to the ABCD approach at a threshold of above 5.45 (5 datasets). Test sensitivities and specificities were broadly similar for no algorithm (88% (95% CI 75% to 95%), and 87% (95% CI 80% to 92%)), and for pattern analysis (92% (95% CI 87% to 95%), and 92% (95% CI 68% to 98%)); use of the ABCD algorithm produced similar specificity (92%, 95% CI 82% to 97%), but lower sensitivity (81%, 95% CI 62% to 92%), although confidence intervals were wide and overlapping. At the median prevalence of melanoma of 12% observed across the in‐person evaluations, the number of melanomas missed per 1000 lesions tested ranged between 10 and 23 with false‐positive results of 70 to 114 (Table 1). For image‐based evaluations, test sensitivities and specificities were again broadly similar for no algorithm (76% (95% CI 70% to 82%), and 79% (95% CI 71% to 85%)), and for pattern analysis (83% (95% CI 76% to 88%), and 87% 95% CI 80% to 92%)). The formal algorithms with the most data included ABCD at above 5.45 (7 datasets), the seven‐point checklist at 3 or above (11 datasets), and the three‐point checklist (7 datasets). Sensitivities were broadly similar with overlapping confidence intervals (ranging from 74% to 81%), with generally lower specificities but again with overlapping confidence intervals (summary estimates ranging from 60% to 81%). At the median prevalence of melanoma of 24% observed across the image‐based evaluations, the number of melanomas missed per 1000 lesions tested ranged between 41 and 62 with false‐positive results of 61 for no algorithm to 304 for the three‐point checklist (Table 1).

The lack of reporting of diagnostic thresholds in the studies that did not use algorithms to assist diagnosis ('no algorithm' studies), means that we have not been able to clearly compare accuracy for the diagnosis of melanoma in comparison to a clinician's decision to excise a skin lesion; the latter perhaps being more clinically relevant in practice. Data from image‐based studies appear to show similar sensitivity for correct diagnosis of melanoma and for the decision to excise a lesion but considerably lower specificity for the decision to excise a lesion, when the target condition was defined as melanoma or atypical intraepidermal melanocytic variants. For the target condition of any skin cancer or lesion with a high risk of progression to melanoma, sensitivities and specificities were both over 90% in three of the four studies reporting data for dermoscopy added to in‐person visual inspection, suggesting that clinicians may be better at identifying skin lesions that require some intervention than at correctly identifying melanomas, however the data are too limited to allow us to draw strong conclusions.

5) Observer expertise and training in dermoscopy improves diagnostic accuracy

Observer experience and expertise in using dermoscopy to assess pigmented lesions is likely to have an impact on test accuracy, however this information was often not provided in great detail, particularly for the in‐person evaluations. We made broad classifications of reported experience in dermoscopy and by observer qualifications which, on the whole, led to statistically significantly higher accuracy for observers reported as having high experience and for those classed as ‘expert consultants’ in comparison to those considered to have less experience in dermoscopy. Much of the evidence for the effect of observer expertise was again provided by image‐based dermoscopy interpretations as opposed to those conducted in person, however similar patterns were observed for both sets of studies. Only two in‐person and three image‐based studies evaluated dermoscopy in the hands of GPs; these showed lower accuracy (RDOR 0.21 (95% CI 0.01 to 3.12), for in‐person and RDOR 0.09 (95% CI 0.04 to 0.24), for image‐based studies), than expert consultants.

Six studies assessed the effect of dermoscopy training on test accuracy in a limited number of participants. Despite differences in the type and length of training interventions, all of the six eligible evaluations resulted in increased sensitivity following training with limited effects on specificity in five of the six studies.

Strengths and weaknesses of the review

The strengths of this review include an in‐depth and comprehensive electronic literature search, systematic review methods including double extraction of papers by both clinicians and methodologists, and contact with study authors to allow study inclusion or clarify data. We adopted a clear analysis structure focusing on estimating incremental gains in accuracy. We undertook a detailed and replicable analysis of methodologic quality.

For our main analyses however, we estimated summary ROC curves rather than average sensitivity and specificity operating points. We took this approach to facilitate pooling across the heterogeneous mixture of thresholds and scoring systems, however it does mean that quoted sensitivities and specificities are at best illustrative and do not reflect the actual performance of dermoscopy. As a result, although we can assess the incremental gain from dermoscopy added to visual inspection, we cannot make any clear recommendations regarding how dermoscopy should be used in order to ensure that melanomas are not missed.

In comparison to other available systematic reviews, our review extends the time period searched for eligible studies, and includes all eligible studies regardless of availability of a direct comparison with visual inspection alone (Vestergaard 2008), requirement for an algorithm or ‘clinical prediction rule’ (Harrington 2017), or focus on specific healthcare professionals or study settings (Corbo 2012; Herschorn 2012; Loescher 2011). Our review of a single large literature search and concurrent systematic review of a number of other tests for the diagnosis of melanoma has led to the identification of additional dermoscopy datasets and inclusion of a much greater of number of studies (i.e. 104 compared to 23 in Rajpara 2009; nine in Vestergaard 2008; and 43 in Harrington 2017). We also explicitly considered whether diagnoses were made based on dermoscopic images or were conducted in person and considered variations in the definition of the target condition. Most importantly perhaps, our review considers the accuracy of dermoscopy both in comparison to visual inspection and for diagnosis with and without the use of a formal algorithm. As for considerations of the accuracy of visual inspection of a lesion per se (Dinnes 2018a), unless the accuracy of diagnostic decisions made without the use of a formal algorithm can be established, the added contribution of such algorithms cannot be fully understood.

Our stringent application of review inclusion criteria meant that we excluded some studies included in previous reviews. For example, those reporting accuracy data for ‘clinical diagnosis’, where dermoscopy may or may not have been used to assist diagnosis, were not included. Of the nine studies included in the Vestergaard 2008 review, we excluded two due to the inclusion of fewer than five melanomas (Carli 2003c; Carli 2004a), and of the 23 in Rajpara 2009 we excluded one due lack of clarity on the 2x2 contingency table (Ascierto 2000). We also excluded seven of the 43 studies included in the Harrington review due to lack of clear data to construct a 2x2 contingency table (Argenziano 2003), or reporting of data in brief letter format (Blum 2004c; Strumia 2003), the serial use of the algorithm in the context of lesion follow‐up (Buhl 2012), the derivation aspect to the study (Henning 2008; MacKie 2002), or diagnosis by laypersons (Luttrell 2012).

The main concerns for the review are a result of the poor reporting of primary studies, in particular limiting assessment of methodological quality, and limiting both the assessment of studies by prior testing of participants and by observer expertise in dermoscopy. Our review of visual inspection alone for the diagnosis of melanoma identified a general trade‐off between sensitivity and specificity along the clinical pathway, with higher sensitivity and lower specificity in limited prior testing studies compared to those in referred populations (Dinnes 2018a). The lack of data from limited prior testing populations in this review and the lack of detailed information on the prior testing of participants included in referred populations meant that we could not derive any clear patterns in sensitivity or specificity. We did identify some evidence of higher accuracy by more specialist or experienced observers, however, better study descriptions of observers would assist such investigations.

Applicability of findings to the review question

There are clear concerns regarding the clinical applicability of studies included in this review. Approximately three‐quarters of studies only provided data from evaluations of dermoscopic images (with or without data from visual inspection of clinical photographs), such that resulting accuracy estimates cannot be extrapolated to in‐person assessments of skin lesions. Furthermore, almost all in‐person evaluations of dermoscopy used in conjunction with visual inspection had high concerns for the applicability of the included population and half had high concern for the applicability of the test. The restriction of including only excised lesions and the small number of studies conducted in a limited prior testing population mean that our results cannot be extrapolated to a primary care population.

Authors' conclusions

Implications for practice.

Due to methodological limitations of the included studies and heterogeneity in study methods and results, we cannot explicitly estimate the sensitivity and specificity of dermoscopy, either with or without visual inspection, however, we can conclude that the incremental benefit of dermoscopy over and above visual inspection alone is consistent and considerable. Dermoscopy is therefore a valuable tool to support visual inspection of a suspicious skin lesion for the detection of melanoma and atypical intraepidermal melanocytic variants, particularly in referred populations and in the hands of experienced users. Data to support its use in a primary care population are limited; however, it is likely to be of some benefit for triaging suspicious lesions for urgent referral when employed by suitably trained clinicians. Overall, the use of formal algorithms to assist diagnosis does not appear to improve accuracy, however, neither is there sufficient evidence to suggest that the ‘no algorithm’ approach should be preferred in all settings. Formal algorithms may be more useful for dermoscopy training purposes and for less expert observers, however reliable data from in‐person evaluations of dermoscopy are lacking.

Implications for research.

Given the vast volume of research that has been funded to evaluate dermoscopy, further research into the added value of established dermoscopy algorithms per se is unlikely to be warranted. Further evaluation of dermoscopy use in the primary care setting and to identify the optimal approach to dermoscopy training may be warranted, however. Such evaluations should be conducted on an in‐person basis with prospective recruitment of consecutive series of participants and with systematic follow‐up of non‐excised lesions to avoid over‐reliance on a histological reference standard. A clear identification of the level of training and experience required to achieve good results is required. Any future research study needs to be clear about the diagnostic pathway followed by study participants prior to study enrolment, and should conform to the updated Standards for Reporting of Diagnostic Accuracy (STARD) guideline (Bossuyt 2015).

What's new

Date	Event	Description
18 December 2018	Amended	Affiliations, Disclaimer and Sources of support updated.

Appendices

Appendix 1. Current content and structure of the Programme Grant

	LIST OF REVIEWS	Number of studies
	Diagnosis of melanoma
1	Visual inspection	49
2	Dermoscopy +/‐ visual inspection	104
3	Teledermatology	22
4	Smartphone applications	2
5a	Computer‐aided diagnosis – dermoscopy‐based techniques	42
5b	Computer‐aided diagnosis – spectroscopy‐based techniques	Review amalgamated into 5a
6	Reflectance confocal microscopy	18
7	High frequency ultrasound	5
	Diagnosis of keratinocyte skin cancer (BCC and cSCC)
8	Visual inspection +/‐ Dermoscopy	24
5c	Computer‐aided diagnosis – dermoscopy‐based techniques	Review amalgamated into 5a
5d	Computer‐aided diagnosis – spectroscopy‐based techniques	Review amalgamated into 5a
9	Optical coherence tomography	5
10	Reflectance confocal microscopy	10
11	Exfoliative cytology	9
	Staging of melanoma
12	Imaging tests (ultrasound, CT, MRI, PET‐CT)	38
13	Sentinel lymph node biopsy	160
	Staging of cSCC
	Imaging tests review	Review dropped; only one study identified
13	Sentinel lymph node biopsy	Review amalgamated into 13 above (n = 15 studies)

Appendix 2. Content of algorithms used to assist melanoma diagnosis using dermoscopy

Pattern analysis Pehamberger 1987	ABCD Stolz 1994a	ABCD (revised) Blum 2003a	ABCDE Kittler 1999	Seven‐point checklist Argenziano 1998Argenziano 2011(revised)
Irregular and multicomponent pigmentary network pattern Peripheral dark network patches Sharp network margin Pseudopods Radial streaming Blue‐grey areas Pigment dots (blotches, black dots, brown globules) Black dots at periphery Whitish veil Depigmentation and hypopigmented areas Erythema Telangiectasia Comedo‐like openings, milia‐like cysts Red‐blue areas	Asymmetry score x 1.3. Calculated according to the colours and structures present within the lesion and not only with respect to the contour of the lesion + Border score x 0.1. For each of 8 lesion segments presenting with an abrupt cut‐off of pigment pattern the score was increased by 1 point. Maximum border score 8 + Colour score x 0.5. Up to 6 different colours counted: white, red, blue‐grey light‐brown, dork‐brown, and black + Differential structure score x 0.5. 5 main structural features: homogeneous areas network, streaks, dots, and 'globules according to size Thresholds > 5.45 or > 4.75	Asymmetry of the outer shape in at least 1 axis (+1); + asymmetry of the differential structures inside the lesion in at least 1 axis (+1) Border: abrupt cutoff of network at the border of the lesion in at least 1 quarter of the circumference (+1) Colour: ≥ 3 colours (+1) Differential structures: ≥ 3 differential structures (+1) Threshold ≥ 4	As for ABCD but with addition of ‘E’ for enlargement or change Patient self‐report of change in lesion size, colour or shape within the last year, or whether they experienced any sign of ulceration or spontaneous bleeding. De novo appearance of a lesion within the last year was regarded as change in size. ABCD‐E score calculated by adding 1.2 to the standard ABCD score for changing lesions and subtracting 0.8 from the standard ABCD score for non‐changing lesions 6 thresholds tested, no single one recommended	Major criteria: atypical network blue‐white veil atypical vascular pattern Minor criteria irregular dots/globules irregular streaks irregular blotches regression structures Major criteria score 2 points each; minor criteria score 1 point each. Threshold for excision ≥ 3. For the revised seven‐point checklist, each criterion is given a score of 1 point, and the threshold for excision is ≥ 1 point, rather than ≥ 3 points.
Seven‐point checklist (for lesion FU) Stanganelli 2015	Three‐point checklist Soyer 2004	Four‐point checklist di Meo 2016	Risk stratification Kenet 1994	Risk stratification (modified) Ascierto 1998, Ascierto 2003, Ascierto 2010
A score of ‘no change’ was assigned if all variables remained constant, with a tolerance of major axis change of 2 mm (Beer 2011; Terushkin 2012); ‘minor change’ if there was only symmetrical change in structural or chromatic pattern; ‘moderate change’ if either structural or chromatic changes were asymmetrical, but there were no melanoma‐specific criteria; and ‘major change’ if there were asymmetrical structural and chromatic changes, or the appearance of melanoma‐specific criteria (i.e. major or minor criteria on original 7‐point checklist: blue‐white veil, atypical or negative pigment network, atypical vascular patterns, irregular dots and globules, streaks, irregular blotches, peripheral pigmented structureless areas and regression.)	Asymmetry ‐ in colour and/or structure in one or two axes, Atypical pigment network ‐ pigmented network with thickened lines and irregular distribution Blue‐white structures ‐ any blue and/or white colour within the lesion The presence of two or three criteria is suggestive for melanoma.	Same as three‐point checklist but asymmetry given 2 points instead of 1 Asymmetry of colour and structure 1st axis 1 point Asymmetry of colour and structure 2nd axis 1 point Irregular or thick pigmented network 1 point Blue‐white structure 1 point A total score > 2 was used as cut‐off	Stratum 1 (probable MM): pseudopods radial streaming heterogeneity of pigment network with thick dark extensions at the edge blue‐grey areas white scar‐like areas and presence of pigment network Stratum 2 (possible MM): marked irregular network with irregular pigment confluence eccentricity of pigment network with darkest regions near edge	Very high risk Pigment network and any classical ELM features specific for melanoma: pseudopods radial streaming blue‐grey veil High risk Pigment network and “subtle new ELM features that may suggest melanoma but often are also seen in atypical naevi”, e.g. irregular brown globules at periphery irregular black dots at periphery hypopigmentation at lesion periphery
Menzies’ checklist Menzies 1996	Seven features for melanoma (7FFM) Dal Pozzo 1999	Chaos and clues Rosendahl 2011	CASH Dolianitis 2005; Henning 2007; Henning 2008	‐
Negative features Point and axial symmetry of pigmentation Presence of only a single colour Positive features Multiple (5‐6) colours Blue‐white veil Multiple brown dots Multiple blue/grey Peripheral black dots or globules A broadened network Pseudopods Radial streaming Scar‐like depigmentation Threshold: both negative features absent and ≥ 1 positive features present	Major features Regression erythema Radial streaming Grey‐blue veil Irregularly distributed pseudopods Minor features score 1 each Unhomogeneity Irregular pigment network Sharp margin Major features score 2 points each; minor features score 1 point each. Threshold: ≥ 2	Chaos Asymmetry of colour or structure (defined by basic principles of pattern analysis as revised by Kittler 2007. Clues Eccentric structureless zone (any colour except skin colour) Grey or blue structures Peripheral black dots or clods Segmental radial lines or pseudopods Polymorphous vessels White lines, thick reticular or branched lines, and parallel lines on ridges (acral lesions) Clues searched for in presence of chaos; both present for test‐positive	Colour: light brown, dark brown, black, red, white, blue, each receive 1 point. Architectural disorder: non‐uniformity of structures and their distribution in the lesion; benign melanocytic lesions having uniform structures and distribution. Absent/mild, moderate and marked architectural disorder receive 0, 1, and 2 points, respectively. Symmetry: biaxial symmetry scores 0 monoaxial symmetry scores 1 biaxial asymmetry scores 2 Homogeneity/heterogeneity: 7 structures each score 1 network dots/globules streaks/pseudopods blue‐whitish veil regression structures (blue‐grey areas with or without peppering; scarring) blotches (structureless regions of any colour occupying > 10% of the area of the lesion) polymorphous blood vessels (dotted and irregular linear) A total CASH score (TCS) ≥ 8 is suggestive for melanoma.	‐
ELM: epiluminescence microscopy; MM: malignant melanoma

Appendix 3. Glossary of terms

Term	Definition
Atypical intraepidermal melanocytic variant	Unusual area of darker pigmentation contained within the epidermis that may progress to an invasive melanoma; includes melanoma in situ and lentigo maligna
Atypical naevi	Unusual looking but non‐cancerous mole or area of darker pigmentation of the skin
BRAF V600 mutation	BRAF is a human gene that makes a protein called B‐Raf which is involved in the control of cell growth. BRAF mutations (damaged DNA) occur in around 40% of melanomas, which can then be treated with particular drugs
BRAF inhibitors	Therapeutic agents that inhibit the serine‐threonine protein kinase BRAF mutated metastatic melanoma
Breslow thickness	A scale for measuring the thickness of melanomas by the pathologist using a microscope, measured in mm from the top layer of skin to the bottom of the tumour.
Congenital naevi	A type of mole found on infants at birth
Dermoscopy	Whereby a handheld microscope is used to allow more detailed, magnified, examination of the skin compared to examination by the naked eye alone
False‐negative	An individual who is truly positive for a disease, but whom a diagnostic test classified as disease‐free
False‐positive	An individual who is truly disease‐free, but whom a diagnostic test classified as having the disease
Histopathology/histology	The study of tissue, usually obtained by biopsy or excision, for example under a microscope
Incidence	The number of new cases of a disease in a given time period
Index test	A diagnostic test under evaluation in a primary study
Lentigo maligna	Unusual area of darker pigmentation contained within the epidermis that includes malignant cells but with no invasive growth. May progress to an invasive melanoma
Lymph node	Lymph nodes filter the lymphatic fluid (clear fluid containing white blood cells) that travels around the body to help fight disease; they are located throughout the body often in clusters (nodal basins)
Melanocytic naevus	An area of skin with darker pigmentation (or melanocytes) also referred to as ‘moles’
Meta‐analysis	A form of statistical analysis used to synthesise results from a collection of individual studies
Metastases/metastatic disease	Spread of cancer away from the primary site to somewhere else through the bloodstream or the lymphatic system
Micrometastases	Micrometastases are metastases so small that they can only be seen under a microscope
Mitotic rate	Microscopic evaluation of number of cells actively dividing in a tumour
Morbidity	Detrimental effects on health
Mortality	Either (1) the condition of being subject to death; or (2) the death rate, that reflects the number of deaths per unit of population in relation to any specific region, age group, disease, treatment or other classification, usually expressed as deaths per 100, 1000, 10,000 or 100,000 people
Multidisciplinary team	A team with members from different healthcare professions and specialties (e.g. urology, oncology, pathology, radiology, and nursing). Cancer care in the National Health Service (NHS) uses this system to ensure that all relevant health professionals are engaged to discuss the best possible care for that patient
Prevalence	The proportion of a population found to have a condition
Prognostic factors/indicators	Specific characteristics of a cancer or the person who has it which might affect the patient’s prognosis
Receiver operating characteristic (ROC) plot	A plot of the sensitivity and 1 minus the specificity of a test at the different possible thresholds for test positivity; represents the diagnostic capability of a test with a range of binary test results
Receiver operating characteristic (ROC) analysis	The analysis of a ROC plot of a test to select an optimal threshold for test positivity
Recurrence	Recurrence is when new cancer cells are detected following treatment. This can occur either at the site of the original tumour or at other sites in the body.
Reference Standard	A test or combination of tests used to establish the final or ‘true’ diagnosis of a patient in an evaluation of a diagnostic test
Reflectance confocal microscopy (RCM)	A microscopic technique using infrared light (either in a handheld device or a static unit) that can create images of the deeper layers of the skin
Sensitivity	In this context the term is used to mean the proportion of individuals with a disease who have that disease correctly identified by the study test
Specificity	The proportion of individuals without the disease of interest (in this case with benign skin lesions) who have that absence of disease correctly identified by the study test
Staging	Clinical description of the size and spread of a patient’s tumour, fitting into internationally agreed categories
Subclinical (disease)	Disease that is usually asymptomatic and not easily observable, e.g. by clinical or physical examination
Systemic treatment	Treatment, usually given by mouth or by injection, that reaches and affects cancer cells throughout the body rather than targeting one specific area

Appendix 4. Proposed sources of heterogeneity

i. Population characteristics

• general versus higher risk populations

• patient population: Primary /secondary/specialist unit

• lesion suspicion: general suspicion/atypical/equivocal/NR

• lesion type: any pigmented; melanocytic

• inclusion of multiple lesions per participant

• ethnicity

ii. Index test characteristics

• the nature of and definition of criteria for test positivity

• observer experience with the index test

• approaches to lesion preparation (e.g., the use of oil or antiseptic gel for dermoscopy)

iii. Reference standard characteristics

• reference standard used

• whether histology‐reporting meets pathology‐reporting guidelines

• use of excisional versus diagnostic biopsy

• whether two independent dermatopathologists reviewed histological diagnosis

iv. Study quality

• consecutive or random sample of participants recruited

• index test interpreted blinded to the reference standard result

• index test interpreted blinded to the result of any other index test

• presence of partial or differential verification bias (whereby only a sample of those subject to the index test are verified by the reference test or by the same reference test with selection dependent on the index test result)

• use of an adequate reference standard

• overall risk of bias

Appendix 5. Final search strategies

Melanoma search strategies to August 2016

Database: Ovid MEDLINE(R) 1946 to August week 3 2016

Search strategy:

1 exp melanoma/

2 exp skin cancer/

3 exp basal cell carcinoma/

4 basalioma$1.ti,ab.

5 ((basal cell or skin) adj2 (cancer$1 or carcinoma$1 or mass or masses or tumour$1 or tumor$1 or neoplasm$1 or adenoma$1 or epithelioma$1 or lesion$1 or malignan$ or nodule$1)).ti,ab.

6 (pigmented adj2 (lesion$1 or mole$ or nevus or naevi or naevus or naevi or skin)).ti,ab.

7 (melanom$1 or nonmelanoma$1 or non‐melanoma$1 or melanocyt$ or non‐melanocyt$ or nonmelanocyt$ or keratinocyt$).ti,ab.

8 nmsc.ti,ab.

9 (squamous cell adj2 (cancer$1 or carcinoma$1 or mass or masses or tumor$1 or tumour$1 or neoplasm$1 or adenoma$1 or epithelioma$1 or epithelial or lesion$1 or malignan$ or nodule$1) adj2 (skin or epiderm$ or cutaneous)).ti,ab.

10 (BCC or CSCC or NMSC).ti,ab.

11 keratinocy$.ti,ab.

12 Keratinocytes/

13 or/1‐12

14 dermoscop$.ti,ab.

15 dermatoscop$.ti,ab.

16 photomicrograph$.ti,ab.

17 exp epiluminescence microscopy/

18 (epiluminescence adj2 microscop$).ti,ab.

19 (confocal adj2 microscop$).ti,ab.

20 (incident light adj2 microscop$).ti,ab.

21 (surface adj2 microscop$).ti,ab.

22 (visual adj (inspect$ or examin$)).ti,ab.

23 ((clinical or physical) adj examin$).ti,ab.

24 3 point.ti,ab.

25 three point.ti,ab.

26 pattern analys$.ti,ab.

27 ABCD$.ti,ab.

28 Menzies.ti,ab.

29 7 point.ti,ab.

30 seven point.ti,ab.

31 (digital adj2 (dermoscop$ or dermatoscop$)).ti,ab.

32 artificial intelligence.ti,ab.

33 AI.ti,ab.

34 computer assisted.ti,ab.

35 computer aided.ti,ab.

36 neural network$.ti,ab.

37 exp diagnosis, computer‐assisted/

38 MoleMax.ti,ab.

39 image process$.ti,ab.

40 automatic classif$.ti,ab.

41 image analysis.ti,ab.

42 SIAscop$.ti,ab.

43 Aura.ti,ab.

44 (optical adj2 scan$).ti,ab.

45 MelaFind.ti,ab.

46 SIMSYS.ti,ab.

47 MoleMate.ti,ab.

48 SolarScan.ti,ab.

49 VivaScope.ti,ab.

50 (high adj3 ultraso$).ti,ab.

51 (canine adj2 detect$).ti,ab.

52 ((mobile or cell or cellular or smart) adj ((phone$1 adj2 app$1) or application$1)).ti,ab.

53 smartphone$.ti,ab.

54 (DermoScan or SkinVision or DermLink or SpotCheck).ti,ab.

55 Mole Detective.ti,ab.

56 Spot Check.ti,ab.

57 (mole$1 adj2 map$).ti,ab.

58 (total adj2 body).ti,ab.

59 exfoliative cytolog$.ti,ab.

60 digital analys$.ti,ab.

61 (image$1 adj3 software).ti,ab.

62 (teledermatolog$ or tele‐dermatolog$ or telederm or tele‐derm or teledermoscop$ or tele‐dermoscop$ or teledermatoscop$ or tele‐dermatoscop$).ti,ab.

63 (optical coherence adj (technolog$ or tomog$)).ti,ab.

64 (computer adj2 diagnos$).ti,ab.

65 exp sentinel lymph node biopsy/

66 (sentinel adj2 node).ti,ab.

67 naevisense.mp. or HFUS.ti,ab.

68 electrical impedance spectroscopy.ti,ab.

69 history taking.ti,ab.

70 patient history.ti,ab.

71 (naked eye adj (exam$ or assess$)).ti,ab.

72 (skin adj exam$).ti,ab.

73 physical examination/

74 ugly duckling.mp. or UD.ti,ab.

75 ((physician$ or clinical or physical) adj (exam$ or triage or recog$)).ti,ab.

76 ABCDE.mp. or VOC.ti,ab.

77 clinical accuracy.ti,ab.

78 Family Practice/or Physicians, Family/or clinical competence/

79 (confocal adj2 microscop$).ti,ab.

80 diagnostic algorithm$1.ti,ab.

81 checklist$.ti,ab.

82 virtual imag$1.ti,ab.

83 volatile organic compound$1.ti,ab.

84 dog$1.ti,ab.

85 gene expression analy$.ti,ab.

86 reflex transmission imag$.ti,ab.

87 thermal imaging.ti,ab.

88 elastography.ti,ab.

89 or/14‐88

90 (CT or PET).ti,ab.

91 PET‐CT.ti,ab.

92 (FDG or F18 or Fluorodeoxyglucose or radiopharmaceutical$).ti,ab.

93 exp Deoxyglucose/

94 deoxy‐glucose.ti,ab.

95 deoxyglucose.ti,ab.

96 CATSCAN.ti,ab.

97 exp Tomography, Emission‐Computed/

98 exp Tomography, X‐ray computed/

99 positron emission tomograph$.ti,ab.

100 exp magnetic resonance imaging/

101 (MRI or fMRI or NMRI or scintigraph$).ti,ab.

102 exp echography/

103 Doppler echography.ti,ab.

104 sonograph$.ti,ab.

105 ultraso$.ti,ab.

106 doppler.ti,ab.

107 magnetic resonance imag$.ti,ab.

108 or/90‐107

109 (stage$ or staging or metasta$ or recurrence or sensitivity or specificity or false negative$ or thickness$).ti,ab.

110 "Sensitivity and Specificity"/

111 exp cancer staging/

112 or/109‐111

113 108 and 112

114 89 or 113

115 13 and 114

Database: Ovid MEDLINE(R) In‐Process & Other Non‐Indexed Citations 29 August 2016

Search strategy:

1 basalioma$1.ti,ab.

2 ((basal cell or skin) adj2 (cancer$1 or carcinoma$1 or mass or masses or tumour$1 or tumor$1 or neoplasm$1 or adenoma$1 or epithelioma$1 or lesion$1 or malignan$ or nodule$1)).ti,ab.

3 (pigmented adj2 (lesion$1 or mole$ or nevus or naevi or naevus or naevi or skin)).ti,ab.

4 (melanom$1 or nonmelanoma$1 or non‐melanoma$1 or melanocyt$ or non‐melanocyt$ or nonmelanocyt$ or keratinocyt$).ti,ab.

5 nmsc.ti,ab.

6 (squamous cell adj2 (cancer$1 or carcinoma$1 or mass or masses or tumor$1 or tumour$1 or neoplasm$1 or adenoma$1 or epithelioma$1 or epithelial or lesion$1 or malignan$ or nodule$1) adj2 (skin or epiderm$ or cutaneous)).ti,ab.

7 (BCC or CSCC or NMSC).ti,ab.

8 keratinocy$.ti,ab.

9 or/1‐8

10 dermoscop$.ti,ab.

11 dermatoscop$.ti,ab.

12 photomicrograph$.ti,ab.

13 (epiluminescence adj2 microscop$).ti,ab.

14 (confocal adj2 microscop$).ti,ab.

15 (incident light adj2 microscop$).ti,ab.

16 (surface adj2 microscop$).ti,ab.

17 (visual adj (inspect$ or examin$)).ti,ab.

18 ((clinical or physical) adj examin$).ti,ab.

19 3 point.ti,ab.

20 three point.ti,ab.

21 pattern analys$.ti,ab.

22 ABCD$.ti,ab.

23 Menzies.ti,ab.

24 7 point.ti,ab.

25 seven point.ti,ab.

26 (digital adj2 (dermoscop$ or dermatoscop$)).ti,ab.

27 artificial intelligence.ti,ab.

28 AI.ti,ab.

29 computer assisted.ti,ab.

30 computer aided.ti,ab.

31 neural network$.ti,ab.

32 MoleMax.ti,ab.

33 image process$.ti,ab.

34 automatic classif$.ti,ab.

35 image analysis.ti,ab.

36 SIAscop$.ti,ab.

37 Aura.ti,ab.

38 (optical adj2 scan$).ti,ab.

39 MelaFind.ti,ab.

40 SIMSYS.ti,ab.

41 MoleMate.ti,ab.

42 SolarScan.ti,ab.

43 VivaScope.ti,ab.

44 (high adj3 ultraso$).ti,ab.

45 (canine adj2 detect$).ti,ab.

46 ((mobile or cell or cellular or smart) adj ((phone$1 adj2 app$1) or application$1)).ti,ab.

47 smartphone$.ti,ab.

48 (DermoScan or SkinVision or DermLink or SpotCheck).ti,ab.

49 Mole Detective.ti,ab.

50 Spot Check.ti,ab.

51 (mole$1 adj2 map$).ti,ab.

52 (total adj2 body).ti,ab.

53 exfoliative cytolog$.ti,ab.

54 digital analys$.ti,ab.

55 (image$1 adj3 software).ti,ab.

56 (teledermatolog$ or tele‐dermatolog$ or telederm or tele‐derm or teledermoscop$ or tele‐dermoscop$ or teledermatoscop$ or tele‐dermatoscop$).ti,ab.

57 (optical coherence adj (technolog$ or tomog$)).ti,ab.

58 (computer adj2 diagnos$).ti,ab.

59 (sentinel adj2 node).ti,ab.

60 naevisense.mp. or HFUS.ti,ab.

61 electrical impedance spectroscopy.ti,ab.

62 history taking.ti,ab.

63 patient history.ti,ab.

64 (naked eye adj (exam$ or assess$)).ti,ab.

65 (skin adj exam$).ti,ab.

66 ugly duckling.mp. or UD.ti,ab.

67 ((physician$ or clinical or physical) adj (exam$ or triage or recog$)).ti,ab.

68 ABCDE.mp. or VOC.ti,ab.

69 clinical accuracy.ti,ab.

70 (Family adj (Practice or Physicians)).ti,ab.

71 (confocal adj2 microscop$).ti,ab.

72 clinical competence.ti,ab.

73 diagnostic algorithm$1.ti,ab.

74 checklist$.ti,ab.

75 virtual imag$1.ti,ab.

76 volatile organic compound$1.ti,ab.

77 dog$1.ti,ab.

78 gene expression analy$.ti,ab.

79 reflex transmission imag$.ti,ab.

80 thermal imaging.ti,ab.

81 elastography.ti,ab.

82 or/10‐81

83 (CT or PET).ti,ab.

84 PET‐CT.ti,ab.

85 (FDG or F18 or Fluorodeoxyglucose or radiopharmaceutical$).ti,ab.

86 deoxy‐glucose.ti,ab.

87 deoxyglucose.ti,ab.

88 CATSCAN.ti,ab.

89 positron emission tomograph$.ti,ab.

90 (MRI or fMRI or NMRI or scintigraph$).ti,ab.

91 Doppler echography.ti,ab.

92 sonograph$.ti,ab.

93 ultraso$.ti,ab.

94 doppler.ti,ab.

95 magnetic resonance imag$.ti,ab.

96 or/83‐95

97 (stage$ or staging or metasta$ or recurrence or sensitivity or specificity or false negative$ or thickness$).ti,ab.

98 96 and 97

99 82 or 98

100 9 and 99

Database: Embase 1974 to 29 August 2016

Search strategy:

1 *melanoma/

2 *skin cancer/

3 *basal cell carcinoma/

4 basalioma$.ti,ab.

5 ((basal cell or skin) adj2 (cancer$1 or carcinoma$1 or mass or masses or tumour$1 or tumor$1 or neoplasm$ or adenoma$ or epithelioma$ or lesion$ or malignan$ or nodule$)).ti,ab.

6 (pigmented adj2 (lesion$1 or mole$ or nevus or naevi or naevus or naevi or skin)).ti,ab.

7 (melanom$1 or nonmelanoma$1 or non‐melanoma$1 or melanocyt$ or non‐melanocyt$ or nonmelanocyt$ or keratinocyt$).ti,ab.

8 nmsc.ti,ab.

9 (squamous cell adj2 (cancer$1 or carcinoma$1 or mass or tumor$1 or tumour$1 or neoplasm$1 or adenoma$1 or epithelioma$1 or epithelial or lesion$1 or malignan$ or nodule$1) adj2 (skin or epiderm$ or cutaneous)).ti,ab.

10 (BCC or cscc).mp. or NMSC.ti,ab.

11 keratinocyte.ti,ab.

12 keratinocy$.ti,ab.

13 or/1‐12

14 dermoscop$.ti,ab.

15 dermatoscop$.ti,ab.

16 photomicrograph$.ti,ab.

17 *epiluminescence microscopy/

18 (epiluminescence adj2 microscop$).ti,ab.

19 (confocal adj2 microscop$).ti,ab.

20 (incident light adj2 microscop$).ti,ab.

21 (surface adj2 microscop$).ti,ab.

22 (visual adj (inspect$ or examin$)).ti,ab.

23 ((clinical or physical) adj examin$).ti,ab.

24 3 point.ti,ab.

25 three point.ti,ab.

26 pattern analys$.ti,ab.

27 ABCD$.ti,ab.

28 Menzies.ti,ab.

29 7 point.ti,ab.

30 seven point.ti,ab.

31 (digital adj2 (dermoscop$ or dermatoscop$)).ti,ab.

32 artificial intelligence.ti,ab.

33 AI.ti,ab.

34 computer assisted.ti,ab.

35 computer aided.ti,ab.

36 neural network$.ti,ab.

37 MoleMax.ti,ab.

38 exp diagnosis, computer‐assisted/

39 image process$.ti,ab.

40 automatic classif$.ti,ab.

41 image analysis.ti,ab.

42 SIAscop$.ti,ab.

43 (optical adj2 scan$).ti,ab.

44 Aura.ti,ab.

45 MelaFind.ti,ab.

46 SIMSYS.ti,ab.

47 MoleMate.ti,ab.

48 SolarScan.ti,ab.

49 VivaScope.ti,ab.

50 confocal microscop$.ti,ab.

51 (high adj3 ultraso$).ti,ab.

52 (canine adj2 detect$).ti,ab.

53 ((mobile or cell$ or cellular or smart) adj ((phone$1 adj2 app$1) or application$1)).ti,ab.

54 smartphone$.ti,ab.

55 (DermoScan or SkinVision or DermLink or SpotCheck).ti,ab.

56 Spot Check.ti,ab.

57 Mole Detective.ti,ab.

58 (mole$1 adj2 map$).ti,ab.

59 (total adj2 body).ti,ab.

60 exfoliative cytolog$.ti,ab.

61 digital analys$.ti,ab.

62 (image$1 adj3 software).ti,ab.

63 (optical coherence adj (technolog$ or tomog$)).ti,ab.

64 (teledermatolog$ or tele‐dermatolog$ or telederm or tele‐derm or teledermoscop$ or tele‐dermoscop$ or teledermatoscop$).mp. or tele‐dermatoscop$.ti,ab.

65 (computer adj2 diagnos$).ti,ab.

66 *sentinel lymph node biopsy/

67 (sentinel adj2 node).ti,ab.

68 naevisense.ti,ab.

69 HFUS.ti,ab.

70 electrical impedance spectroscopy.ti,ab.

71 history taking.ti,ab.

72 patient history.ti,ab.

73 (naked eye adj (exam$ or assess$)).ti,ab.

74 (skin adj exam$).ti,ab.

75 *physical examination/

76 ugly duckling.ti,ab.

77 UD sign$.ti,ab.

78 ((physician$ or clinical or physical) adj (exam$ or recog$ or triage)).ti,ab.

79 ABCDE.ti,ab.

80 clinical accuracy.ti,ab.

81 *general practice/

82 (confocal adj2 microscop$).ti,ab.

83 clinical competence/

84 diagnostic algorithm$.ti,ab.

85 checklist$1.ti,ab.

86 virtual image$1.ti,ab.

87 volatile organic compound$1.ti,ab.

88 VOC.ti,ab.

89 dog$1.ti,ab.

90 gene expression analys$.ti,ab.

91 reflex transmission imaging.ti,ab.

92 thermal imaging.ti,ab.

93 elastography.ti,ab.

94 dog$1.ti,ab.

95 gene expression analys$.ti,ab.

96 reflex transmission imaging.ti,ab.

97 thermal imaging.ti,ab.

98 elastography.ti,ab.

99 or/14‐93

100 PET‐CT.ti,ab.

101 (CT or PET).ti,ab.

102 (FDG or F18 or Fluorodeoxyglucose or radiopharmaceutical$).ti,ab.

103 exp Deoxyglucose/

104 CATSCAN.ti,ab.

105 deoxyglucose.ti,ab.

106 deoxy‐glucose.ti,ab.

107 *positron emission tomography/

108 *computer assisted tomography/

109 positron emission tomograph$.ti,ab.

110 *nuclear magnetic resonance imaging/

111 (MRI or fMRI or NMRI or scintigraph$).ti,ab.

112 *echography/

113 Doppler.ti,ab.

114 sonograph$.ti,ab.

115 ultraso$.ti,ab.

116 magnetic resonance imag$.ti,ab.

117 or/100‐116

118 (stage$ or staging or metasta$ or recurrence or sensitivity or specificity or false negative$ or thickness$).ti,ab.

119 "Sensitivity and Specificity"/

120 *cancer staging/

121 or/118‐120

122 117 and 121

123 99 or 122

124 13 and 123

Database: Cochrane Library (Wiley) 2016 searched 30 August 2016 CDSR Issue 8 of 12 2016 CENTRAL Issue 7 of 12 2016 HTA Issue 3 of 4 July 2016 DARE Issue 3 of 4 2015

Search strategy:

#1 melanoma* or nonmelanoma* or non‐melanoma* or melanocyt* or non‐melanocyt* or nonmelanocyt* or keratinocyte*

#2 MeSH descriptor: [Melanoma] explode all trees

#3 "skin cancer*"

#4 MeSH descriptor: [Skin Neoplasms] explode all trees

#5 skin near/2 (cancer* or carcinoma* or mass or masses or tumour* or tumor* or neoplasm* or adenoma* or epithelioma* or lesion* or malignan* or nodule*)

#6 nmsc

#7 "squamous cell" near/2 (cancer* or carcinoma* or mass or masses or tumour* or tumor* or neoplasm* or adenoma* or epithelioma* or lesion* or malignan* or nodule*) near/2 (skin or epiderm* or cutaneous)

#8 "basal cell" near/2 (cancer* or carcinoma* or mass or masses or tumour* or tumor* or neoplasm* or adenoma* or epithelioma* or lesion* or malignan* or nodule*)

#9 pigmented near/2 (lesion* or nevus or mole* or naevi or naevus or naevi or skin)

#10 #1 or #2 or #3 or #4 or #5 or #6 or #7 or #8 or #9

#11 dermoscop*

#12 dermatoscop*

#13 Photomicrograph*

#14 MeSH descriptor: [Dermoscopy] explode all trees

#15 confocal near/2 microscop*

#16 epiluminescence near/2 microscop*

#17 incident next light near/2 microscop*

#18 surface near/2 microscop*

#19 "visual inspect*"

#20 "visual exam*"

#21 (clinical or physical) next (exam*)

#22 "3 point"

#23 "three point"

#24 "pattern analys*"

#25 ABDC

#26 Menzies

#27 "7 point"

#28 "seven point"

#29 digital near/2 (dermoscop* or dermatoscop*)

#30 "artificial intelligence"

#31 "AI"

#32 "computer assisted"

#33 "computer aided"

#34 AI

#35 "neural network*"

#36 MoleMax

#37 "computer diagnosis"

#38 "image process*"

#39 "automatic classif*"

#40 SIAscope

#41 "image analysis"

#42 "optical near/2 scan*"

#43 Aura

#44 MelaFind

#45 SIMSYS

#46 MoleMate

#47 SolarScan

#48 Vivascope

#49 "confocal microscopy"

#50 high near/3 ultraso*

#51 canine near/2 detect*

#52 Mole* near/2 map*

#53 total near/2 body

#54 mobile* or smart near/2 phone*

#55 cell next phone*

#56 smartphone*

#57 "mitotic index"

#58 DermoScan or SkinVision or DermLink or SpotCheck

#59 "Mole Detective"

#60 "Spot Check"

#61 mole* near/2 map*

#62 total near/2 body

#63 "exfoliative cytolog*"

#64 "digital analys*"

#65 image near/3 software

#66 teledermatolog* or tele‐dermatolog* or telederm or tele‐derm or teledermoscop* or tele‐dermoscop* or teledermatoscop* or tele‐dermatolog*

#67 "optical coherence" next (technolog* or tomog*)

#68 computer near/2 diagnos*

#69 sentinel near/2 node*

#70 #11 or #12 or #13 or #14 or #15 or #16 or #17 or #18 or #19 or #20 or #21 or #22 or #23 or #24 or #25 or #26 or #27 or #28 or #29 or #30 or #31 or #32 or #33 or #34 or #35 or #36 or #37 or #38 or #39 or #40 or #41 or #42 or #43 or #44 or #45 or #46 or #47 or #48 or #49 or #50 or #51 or #52 or #53 or #54 or #55 or #56 or #57 or #58 or #59 or #60 or #61 or #62 or #63 or #64 or #65 or #66 or #67 or #68 or #69

#71 ultraso*

#72 sonograph*

#73 MeSH descriptor: [Ultrasonography] explode all trees

#74 Doppler

#75 CT or PET or PET‐CT

#76 "CAT SCAN" or "CATSCAN"

#77 MeSH descriptor: [Positron‐Emission Tomography] explode all trees

#78 MeSH descriptor: [Tomography, X‐Ray Computed] explode all trees

#79 MRI

#80 MeSH descriptor: [Magnetic Resonance Imaging] explode all trees

#81 MRI or fMRI or NMRI or scintigraph*

#82 "magnetic resonance imag*"

#83 MeSH descriptor: [Deoxyglucose] explode all trees

#84 deoxyglucose or deoxy‐glucose

#85 "positron emission tomograph*"

#86 #71 or #72 or #73 or #74 or #75 or #76 or #77 or #78 or #79 or #80 or #81 or #82 or #83 or #84 or #85

#87 stage* or staging or metasta* or recurrence or sensitivity or specificity or "false negative*" or thickness*

#88 MeSH descriptor: [Neoplasm Staging] explode all trees

#89 #87 or #88

#90 #89 and #86

#91 #70 or #90

#92 #10 and #91

#93 BCC or CSCC or NMCS

#94 keratinocy*

#95 #93 or #94

#96 #10 or #95

#97 naevisense

#98 HFUS

#99 "electrical impedance spectroscopy"

#100 "history taking"

#101 "patient history"

#102 naked next eye near/1 (exam* or assess*)

#103 skin next exam*

#104 "ugly duckling" or (UD sign*)

#105 MeSH descriptor: [Physical Examination] explode all trees

#106 (physician* or clinical or physical) near/1 (exam* or recog* or triage*)

#107 ABCDE

#108 "clinical accuracy"

#109 MeSH descriptor: [General Practice] explode all trees

#110 confocal near microscop*

#111 "diagnostic algorithm*"

#112 MeSH descriptor: [Clinical Competence] explode all trees

#113 checklist*

#114 "virtual image*"

#115 "volatile organic compound*"

#116 dog or dogs

#117 VOC

#118 "gene expression analys*"

#119 "reflex transmission imaging"

#120 "thermal imaging"

#121 elastography

#122 #97 or #98 or #99 or #100 or #101 or #102 or #103 or #104 or #105 or #106 or #107 or #108 or #109 or #110 or #111 or #112 or #113 or #114 or #115 or #116 or #117 or #118 or #119 or #120 or #121

#123 #70 or #122

#124 #96 and #123

#125 #96 and #90

#126 #125 or #124

#127 #10 and #126

Database: CINAHL +(EBSCO) 1937 to 30 August 2016

Search strategy:

S1 (MH "Melanoma") OR (MH "naevi and Melanomas+")

S2 (MH "Skin Neoplasms+")

S3 (MH "Carcinoma, Basal Cell+")

S4 basalioma*

S5 (basal cell) N2 (cancer* or carcinoma* or mass or masses or tumor* or tumour* or neoplasm* or adenoma* or epithelioma* or lesion* or malignan* or nodule*)

S6 (pigmented) N2 (lesion* or mole* or nevus or naevi or naevus or naevi or skin)

S7 melanom* or nonmelanoma* or non‐melanoma* or melanocyt* or non‐melanocyt* or nonmelanocyt*

S8 nmsc

S9 TX BCC or cscc or NMSC

S10 (MH "Keratinocytes")

S11 keratinocyt*

S12 S1 OR S2 OR S3 OR S4 OR S5 OR S6 OR S7 OR S8 OR S9 OR S10 OR S11

S13 dermoscop* or dermatoscop* or photomicrograph* or (3 point) or (three point) or ABCD* or Menzies or (7 point) or (seven point) or AI or Molemax or SIASCOP* or Aura or MelaFind or SIMSYS or MoleMate or SolarScan or smartphone* or DermoScan or SkinVision or DermLink or SpotCheck

S14 (epiluminescence or confocal or incident or surface) N2 (microscop*)

S15 visual N1 (inspect* or examin*)

S16 (clinical or physical) N1 (examin*)

S17 pattern analys*

S18 (digital) N2 (dermoscop* or dermatoscop*)

S19 (artificial intelligence)

S20 (computer) N2 (assisted or aided)

S21 (neural network*)

S22 (MH "Diagnosis, Computer Assisted+")

S23 (image process*)

S24 (automatic classif*)

S25 (image analysis)

S26 SIAScop*

S27 (optical) N2 (scan*)

S28 (high) N3 (ultraso*)

S29 elastography

S30 (mobile or cell or cellular or smart) N2 (phone*) N2 (app or application*)

S31 (mole*) N2 (map*)

S32 total N2 body

S33 exfoliative cytolog*

S34 digital analys*

S35 image N3 software

S36 teledermatolog* or tele‐dermatolog* or telederm or tele‐derm or teledermoscop* or tele‐dermoscop* or teledermatoscop* or tele‐dermatoscop* teledermatolog* or tele‐dermatolog* or telederm or tele‐derm or teledermoscop*

S37 (optical coherence) N1 (technolog* or tomog*)

S38 computer N2 diagnos*

S39 sentinel N2 node

S40 (MH "Sentinel Lymph Node Biopsy")

S41 naevisense or HFUS or checklist* or VOC or dog*

S42 electrical impedance spectroscopy

S43 history taking

S44 "Patient history"

S45 naked eye

S46 skin exam*

S47 physical exam*

S48 ugly duckling

S49 UD sign*

S50 (physician* or clinical or physical) N1 (exam*)

S51 clinical accuracy

S52 general practice

S53 (physician* or clinical or physical) N1 (recog* or triage)

S54 confocal microscop*

S55 clinical competence

S56 diagnostic algorithm*

S57 checklist*

S58 virtual image*

S59 volatile organic compound*

S60 gene expression analys*

S61 reflex transmission imag*

S62 thermal imaging

S63 S13 or S14 or S15 OR S16 OR S17 OR S18 OR S19 OR S20 OR S21 OR S22 OR S23 OR S24 OR S25 OR S26 OR S27 OR S28 OR S29 OR S30 OR S31 OR S32 OR S33 OR S34 OR S35 OR S36 OR S37 OR S38 OR S39 OR S40 OR S41 OR S42 OR S43 OR S44 OR S45 OR S46 OR S47 OR S48 OR S49 OR S50 OR S51 OR S52 OR S53 OR S54 OR S55 OR S56 OR S57 OR S58 OR S59 OR S60 OR S61 OR S62

S64 CT or PET

S65 PET‐CT

S66 FDG or F18 or Fluorodeoxyglucose or radiopharmaceutical*

S67 (MH "Deoxyglucose+")

S68 deoxy‐glucose or deoxyglucose

S69 CATSCAN

S70 CAT‐SCAN

S71 (MH "Deoxyglucose+")

S72 (MH "Tomography, Emission‐Computed+")

S73 (MH "Tomography, X‐Ray Computed")

S74 positron emission tomograph*

S75 (MH "Magnetic Resonance Imaging+")

S76 MRI or fMRI or NMRI or scintigraph*

S77 echography

S78 doppler

S79 sonograph*

S80 ultraso*

S81 magnetic resonance imag*

S82 S64 OR S65 OR S66 OR S67 OR S68 OR S69 OR S70 OR S71 OR S72 OR S73 OR S74 OR S75 OR S76 OR S77 OR S78 OR S79 OR S80 OR S81

S83 stage* or staging or metasta* or recurrence or sensitivity or specificity or (false negative*) or thickness

S84 (MH "Neoplasm Staging")

S85 S83 OR S84

S86 S82 AND S85

S87 S63 OR S86

S88 S12 AND S87

Database: Science Citation Index SCI Expanded (Web of Science) 1900 to 30 August 2016

Conference Proceedings Citation Index (Web of Science) 1900 to 1 September 2016

Search strategy:

#1 (melanom* or nonmelanom* or non‐melanoma* or melanocyt* or non‐melanocyt* or nonmelanocyt* or keratinocyt*)

#2 (basalioma*)

#3 ((skin) near/2 (cancer* or carcinoma or mass or masses or tumour* or tumor* or neoplasm* or adenoma* or epithelioma* or lesion* or malignan* or nodule*))

#4 ((basal) near/2 (cancer* or carcinoma* or mass or masses or tumour* or tumor* or neoplasm* or adenoma* or epithelioma* or lesion* or malignan* or nodule*))

#5 ((pigmented) near/2 (lesion* or mole* or nevus or naevi or naevus or naevi or skin))

#6 (nmsc or BCC or NMSC or keratinocy*)

#7 ((squamous cell (cancer* or carcinoma* or mass or masses or tumour* or tumor* or neoplasm* or adenoma* or epithelioma* or lesion* or malignan* or nodule*))

#8 (skin or epiderm* or cutaneous)

#9 #8 AND #7

#10 #9 OR #6 OR #5 OR #4 OR #3 OR #2 OR #1

#11 ((dermoscop* or dermatoscop* or photomicrograph* or epiluminescence or confocal or "incident light" or "surface microscop*" or "visual inspect*" or "physical exam*" or 3 point or three point or pattern analy* or ABCDE or Menzies or 7 point or seven point or dermoscop* or dermatoscop* or AI or artificial or computer aided or computer assisted or neural network* or Molemax or image process* or automatic classif* or image analysis or siascope or optical scan* or Aura or melafind or simsys or molemate or solarscan or vivascope or confocal microscop* or high ultraso* or canine detect* or cellphone* or mobile* or phone* or smartphone or dermoscan or skinvision or dermlink or spotcheck or spot check or mole detective or mole map* or total body or exfoliative psychology or digital or image software or optical coherence or teledermatology or telederm* or teledermoscop* or teledermatoscop* or computer diagnos* or sentinel))

#12 ((naevisense or HFUS or impedance spectroscopy or history taking or patient history or naked eye or skin exam* or physical exam* or ugly duckling or UD sign* or physician* exam* or physical exam* or ABCDE or clinical accuracy or general practice or confocal microscop* or clinical competence or diagnostic algorithm* or checklist* or virtual image* or volatile organic or VOC or dog* or gene expression or reflex transmission or thermal imag* or elastography))

#13 #11 or #12

#14 ((PET or CT or FDG or deoxyglucose or deoxy‐glucose or fluorodeoxy* or radiopharma* or CATSCAN or positron emission or computer assisted or nuclear magnetic or MRI or FMRI or NMRI or scintigraph* or echograph* or Doppler or sonograph* or ultraso* or magnetic reson*))

#15 ((stage* or staging or metast* or recurrence or sensitivity or specificity or false negative* or thickness*))

#16 #14 AND #15

#17 #16 OR #13

#18 #10 AND #17

Refined by: DOCUMENT TYPES: (MEETING ABSTRACT OR PROCEEDINGS PAPER)

Appendix 6. Full text inclusion criteria

Criterion	Inclusion	Exclusion
Study design	For diagnostic and staging reviews Any study for which a 2×2 contingency table can be extracted, e.g. diagnostic case‐control studies 'cross‐sectional' test accuracy study with retrospective or prospective data collection studies where estimation of test accuracy was not the primary objective but test results for both index and reference standard were available RCTs of tests or testing strategies where participants were randomised between index tests and all undergo a reference standard (i.e. accuracy RCTs)	< 5 melanoma cases (diagnosis reviews) < 10 participants (staging reviews) Studies developing new criteria for diagnosis unless a separate 'test set' of images were used to evaluate the criteria (mainly digital dermoscopy) Studies using 'normal' skin as controls Letters, editorials, comment papers, narrative reviews Insufficient data to construct a 2×2 table
Target condition	Melanoma Keratinocyte skin cancer (or non‐melanoma skin cancer) BCC or epithelioma cSCC	Studies exclusively conducted in children Studies of non‐cutaneous melanoma or SCC
Population	For diagnostic reviews Adults with a skin lesion suspicious for melanoma, BCC, or cSCC (other terms include PSL/naevi, melanocytic, keratinocyte, etc.) Adults at high risk of developing melanoma skin cancer, BCC, or cSCC For staging reviews Adults with a diagnosis of melanoma or cSCC undergoing tests for staging of lymph nodes or distant metastases or both	People suspected of other forms of skin cancer Studies conducted exclusively in children
Index tests	For diagnosis Visual inspection/clinical examination Dermoscopy/dermatoscopy Teledermoscpoy Smartphone/mobile phone applications Digital dermoscopy/artificial intelligence Confocal microscopy Ocular coherence tomography Exfoliative cytology High‐frequency ultrasound Canine odour detection DNA expression analysis/gene chip analysis Other For staging CT PET PET‐CT MRI Ultrasound +/fine needle aspiration cytology FNAC SLNB +/high‐frequency ultrasound Other Any test combination and in any order Any test positivity threshold Any variation in testing procedure (e.g. radioisotope used)	Sentinel lymph biopsy for therapeutic rather than staging purposes Tests to determine melanoma thickness Tests to determine surgical margins/lesion borders Tests to improve histopathology diagnose LND
Reference standard	For diagnostic studies Histopathology of the excised lesion Clinical follow‐up of non‐excised/benign appearing lesions with later histopathology if suspicious Expert diagnosis (studies should not be included if expert diagnosis is the sole reference standard) For studies of imaging tests for staging Histopathology (via LND or SLMB) Clinical/radiological follow‐up A combination of the above For studies of SLNB accuracy for staging LND of both SLN+ and SLn participants to identify all diseased nodes LND of SLN+ participants and follow‐up of SLN participants to identify a subsequent nodal recurrence in a previously investigated nodal basin	For diagnostic studies Exclude if any disease positive participants have diagnosis unconfirmed by histology Exclude if > 50% of disease‐negative participants have diagnosis confirmed by expert opinion with no histology or follow‐up Exclude studies of referral accuracy, i.e. comparing referral decision with expert diagnosis, unless evaluations of teledermatology or mobile phone applications
BCC: basal cell carcinoma; cSCC: cutaneous squamous cell carcinoma; CT: computed tomography; FNAC: fine needle aspiration cytology; lND: lymph node dissection; MRI: magnetic resonance imaging; PET: positron emission tomography; PET‐CT: positron emission tomography computed tomography; RCT: randomised controlled trial; SCC: squamous cell carcinoma; SLN+: positive sentinel lymph node; SLn: negative sentinel lymph node; SLNB: sentinel lymph node biopsy.

Appendix 7. Quality assessment (based on QUADAS‐2)

Patient selection domain (1)

Selective recruitment of study participants can be a key influence on test accuracy. In general terms, all participants eligible to undergo a test should be included in a study, allowing for the intended use of that test within the context of the study. We considered studies that separately sampled malignant and benign lesions to have used a case‐control design; and those that supplemented a series of suspicious lesions with additional malignant or benign lesions to be at unclear risk of bias

In terms of exclusions, we considered studies that excluded particular lesion types (e.g. lentigo maligna), particular lesion sites, or excluded lesions with lack of observer agreement (e.g. on histopathology) to be at high risk of bias. For image‐based evaluations, some studies excluded lesions on the basis of image quality as an a priori exclusion criterion while others excluded lesions with inadequate images post hoc. In order to judge studies consistently, we considered all exclusions due to image quality in the Flow and timing domain (Were all participants included in the analysis?).

In judging the applicability of patient populations to the review question, we considered restriction to particular lesion populations, such as melanocytic, nodular, high risk or restrictions by size to be of high concern for applicability. Studies that included only lesions with histopathology results were also considered of high concern for applicability on the basis that in usual practice, whether in primary, secondary or specialist care, a greater or lesser proportion of patients will have lesions with low levels of suspicion of malignancy such that they can be reassured and discharged, or followed up over a period of time. The restriction of a study sample to those with lesions undergoing biopsy or excision will therefore not adequately reflect a usual‐care setting. Furthermore, due to the invasive nature of sampling lesions for histology, studies are not likely to mandate biopsy or excision as a study requirement regardless of the index of suspicion (in which case restriction to those with histology would not be of concern in terms of having a representative population).

Given that diagnosis of skin cancer is primarily lesion‐based, there is the potential for study participants with multiple lesions to contribute disproportionately to estimates of test accuracy, especially if they are at particular risk of having skin cancer. We considered studies that included a high number of lesions in relation to the number of study to be less representative than studies conducted in a more general population participants (i.e. if the difference between the number of included lesions and number of included participants was greater than 5%).

Index test domain (2)

Given the potential for subjective differences in test interpretation for melanoma, the interpretation of the index test blinded to the result of the reference standard is a key means of reducing bias. For prospective studies and retrospective studies that used the original index test interpretation, the diagnosis will by nature be interpreted and recorded before the result of the reference standard is known; however, studies using previously acquired images could be particularly susceptible to information bias. For these studies to be at low risk of bias, we required a clear indication that observers were unaware of the reference standard diagnosis at time of test interpretation. We also added an item to assess the presence of blinding between interpretations of different algorithms, however this item was not included in the overall assessment of risk of bias.

Pre‐specification of the index test threshold was considered present if the study clearly reported that the threshold used was not data driven, that is, was not based on study results. Studies that did not clearly describe the threshold used but that required clinicians to record a diagnosis or management decision for a lesion were considered to be unclear on this criterion. Studies reporting accuracy for multiple numeric thresholds, where ROC analysis was used to select the threshold, or that reported accuracy for the presence of independently significant lesion characteristics with no separate test set of lesions were considered at high risk of bias.

In terms of applicability of the index test to the review question, we required the test to be applied and interpreted as it would be in a clinical practice setting, that is, in person or face‐to‐face with the study participant, and by a single observer as opposed to a consensus decision or average across multiple observers. Image‐based studies were considered to be of high concern.

Despite the often subjective nature of test interpretation, it is also important for study authors to outline the particular lesion characteristics that were considered to be indicative for melanoma, particularly where established algorithms or checklists were not used. Studies were considered of low concern if the threshold used was established in a prior study or sufficient threshold details were presented to allow replication.

The experience of the examiner will also impact on the applicability of study results. We required studies to describe the test interpreter as ‘experienced’ or ‘expert’ to have low concern about applicability.

Reference standard domain (3)

In an ideal study, consecutively recruited participants should all undergo incisional or excisional biopsy of the skin lesion regardless of level of clinical suspicion of melanoma. In reality, both partial and differential verification bias are likely. Partial verification bias may occur where histology is the only reference standard used, and only those participants with a certain degree of suspicion of malignancy based on the result of the index test undergo verification, the others either being excluded from the study or defined as being disease‐negative without further assessment or follow‐up, as discussed under ‘Patient selection domain’.

Differential verification bias will be present where other reference standards are used in addition to histological verification of suspicious lesions. A typical example of verification bias in skin cancer occurs when investigators do not biopsy people with benign‐appearing lesions but instead follow them up for a period of time to determine whether any malignancy subsequently develops (these would be false‐negatives on the index test). We defined an 'adequate' reference standard as: all disease‐positive individuals having a histological reference standard either at the time of application of the index test or after a period of clinical follow‐up; and at least 80% of disease‐negative participants have received a histological diagnosis, with up to 20% undergoing at least three months' follow‐up of benign‐appearing lesions.

A further challenge is the potential for incorporation bias, that is, where the result of the index test is used to help determine the reference standard diagnosis. It is normal practice for the clinical diagnosis (usually by visual inspection or dermoscopy) to be included on pathology request forms and for the histopathologist to use this diagnosis to help with the pathology interpretation. Although inclusion of such clinical information on the histopathology request form is theoretically a form of incorporation bias, blinded interpretation of the histopathology reference standard is not normal practice, and enforcement of such conditions would significantly limit the generalisability of the study results. Blinding to the index test (visual inspection or clinical diagnosis) was therefore recorded but did not contribute to our overall assessment of risk of bias for the reference standard domain.

In judging the applicability of the reference standard to our review question, we scored studies as high concern around applicability if they used expert diagnosis (with no follow‐up) as a reference standard in any participant, or did not report histology interpretation by a dermatopathologist.

Flow and timing domain (4)

In the ideal study, the diagnosis based on the index test and reference standard should be made consecutively or as near to each other in time as possible to avoid changes in lesion over time. For lesions with a histological reference standard, we have defined a one‐month period as an appropriate interval between application of the index test and the reference standard. Studies reporting biopsy or excision ‘following’, ‘after’ or ‘subsequent to’ the visual inspection diagnosis (or using similar descriptors) were considered to have met this criterion. For studies using clinical follow‐up, a minimum three‐month follow‐up period has been defined as at low risk of bias for detecting false‐negatives. This interval was chosen based on a study showing that most false‐negative melanomas will be diagnosed within three months of the initial negative index test although a small number will be diagnosed up to 12 months subsequently (Altamura 2008).

In assessing whether all participants were included in the analysis, we considered studies at high risk of bias if participants were excluded following recruitment. As discussed in the ‘Patient selection domain’, a priori exclusion of images on the basis of image quality was also considered under this item.

Comparative domain

For studies reporting accuracy data for both visual inspection and dermoscopy, a comparative domain has been added to record blinding between tests and the time interval between tests. Given that visual inspection is an essential component of in‐person dermoscopy and that dermoscopic image interpretation should really be done alongside a clinical image of a suspicious skin lesion, responses to this item have been recorded but do not contribute to overall risk of bias. For the time interval between tests, in‐person dermoscopy will usually be conducted subsequent to visual inspection and clinical photographs and dermoscopic images acquired at the same consultation; responses to this item therefore largely reflect study reporting quality rather than risk of bias.

QUADAS‐2 tool

The following tables use text that was originally published in the QUADAS‐2 tool by Whiting and colleagues (Whiting 2011).

Item	Response (delete as required)
Participant selection (1) risk of bias
1) Was a consecutive or random sample of participants or images enrolled?	Yes – if paper states consecutive or random No – if paper describes other method of sampling Unclear – if participant sampling not described
2) Was a case‐control design avoided?	Yes – if consecutive or random or case‐control design clearly not used No – if study described as case‐control or describes sampling specific numbers of participants with particular diagnoses Unclear – if not described
3) Did the study avoid inappropriate exclusions, e.g., 'difficult to diagnose' lesions not excluded lesions not excluded on basis of disagreement between evaluators	Yes ‐ if inappropriate exclusions were avoided No – if lesions were excluded that might affect test accuracy, e.g., 'difficult to diagnose' lesions, or where disagreement between evaluators was observed Unclear – if not clearly reported but there is suspicion that difficult to diagnose lesions may have been excluded
4) For between‐person comparative studies only (i.e., allocating different tests to different study participants): A) were the same participant selection criteria used for those allocated to each test? B) was the potential for biased allocation between tests avoided through adequate generation of a randomised sequence? C) was the potential for biased allocation between tests avoided through concealment of allocation prior to assignment?	For A) Yes – if same selection criteria were used for each index test, No – if different selection criteria were used for each index test, Unclear – if selection criteria per test were not described, N/A – if only 1 index test was evaluated or all participants received all tests For B) Yes – if adequate randomisation procedures are described, No – if inadequate randomisation procedures are described, Unclear – if the method of allocation to groups is not described (a description of 'random' or 'randomised' is insufficient), N/A – if only 1 index test was evaluated or all participants received all tests For C) Yes – if appropriate methods of allocation concealment are described, No – if appropriate methods of allocation concealment are not described, Unclear – if the method of allocation concealment is not described (sufficient detail to allow a definite judgement is required), N/A – if only 1 index test was evaluated
Could the selection of participants have introduced bias? For non‐comparative and within‐person‐comparative studies If answers to all of questions 1), 2), and 3) 'Yes': If answers to any 1 of questions 1), 2), or 3) 'No': If answers to any 1 of questions 1), 2), or 3) 'Unclear': For between‐person comparative studies If answers to all of questions 1), 2), 3), and 4) 'Yes': If answers to any 1 of questions 1), 2), 3), or 4) 'No': If answers to any 1 of questions 1), 2), 3), or 4) 'Unclear':	For non‐comparative and within‐person‐comparative studies Risk is low Risk is high Risk unclear For between‐person comparative studies Risk is low Risk is high Risk unclear
Participant selection (1) concerns regarding applicability
1) Are the included participants and chosen study setting appropriate to answer the review question, i.e., are the study results generalisable? This item is not asking whether exclusion of certain participant groups might bias the study's results (as in Risk of Bias above), but is asking whether the chosen study participants and setting are appropriate to answer our review question. Because we are looking to establish test accuracy in both primary presentation and referred participants, a study could be appropriate for 1 setting and not for the other, or it could be unclear as to whether the study can appropriately answer either question For each study assessed, please consider whether it is more relevant for A) participants with a primary presentation of a skin lesion or B) referred participants, and respond to the questions in either A) or B) accordingly. If the study gives insufficient details, please respond Unclear to both parts of the question	A) For studies that will contribute to the analysis of participants with a primary presentation of a skin lesion (i.e., test naive) Yes – if participants included in the study appear to be generally representative of those who might present in a usual practice setting No – if study participants appear to be unrepresentative of usual practice, e.g., in terms of severity of disease, demographic features, presence of differential diagnosis or comorbidity, setting of the study, and previous testing protocols Unclear – if insufficient details are provided to determine the generalisability of study participants B) For studies that will contribute to the analysis of referred participants (i.e., who have already undergone some form of testing) Yes – if study participants appear to be representative of those who might be referred for further investigation. If the study focuses only on those with equivocal lesions, for example, we would suggest that this is not representative of the wider referred population No – if study participants appear to be unrepresentative of usual practice, e.g., if a particularly high proportion of participants have been self‐referred or referred for cosmetic reasons. Other factors to consider include severity of disease, demographic features, presence of differential diagnosis or comorbidity, setting of the study, and previous testing protocols Unclear – if insufficient details are provided to determine the generalisability of study participants
2) Did the study avoid including participants with multiple lesions?	Yes – if the difference between the number of included lesions and number of included participants is less than 5% No – if the difference between the number of included lesions and number of included participants is greater than 5% Unclear – if it is not possible to assess
Is there concern that the included participants do not match the review question? If the answer to question 1) or 2) 'Yes': If the answer to question 1) or 2) 'No': If the answer to question 1) or 2) 'Unclear':	Concern is low Concern is high Concern is unclear
Index test (2) risk of bias (to be completed per test evaluated)
1) Was the index test or testing strategy result interpreted without knowledge of the results of the reference standard?	Yes – if index test described as interpreted without knowledge of reference standard result or, for prospective studies, if index test is always conducted and interpreted prior to the reference standard No – if index test described as interpreted in knowledge of reference standard result Unclear – if index test blinding is not described
2) Was the diagnostic threshold at which the test was considered positive (i.e., melanoma present) prespecified?	Yes – if threshold was prespecified (i.e., prior to analysing study results) No – if threshold was not prespecified Unclear – if not possible to tell whether or not diagnostic threshold was prespecified
3) For within‐person comparisons of index tests or testing strategies (i.e., > 1 index test applied per participant): was each index test result interpreted without knowledge of the results of other index tests or testing strategies?	Yes – if all index tests were described as interpreted without knowledge of the results of the others No – if the index tests were described as interpreted in the knowledge of the results of the others Unclear – if it is not possible to tell whether knowledge of other index tests could have influenced test interpretation N/A – if only 1 index test was evaluated
Could the conduct or interpretation of the index test have introduced bias? For non‐comparative and between‐person comparison studies If answers to questions 1) and 2) 'Yes': If answers to either questions 1) or 2) 'No': If answers to either questions 1) or 2) 'Unclear': For within‐person comparative studies If answers to all questions 1), 2), for any index test and 3) 'Yes': If answers to any 1 of questions 1) or 2) for any index test or 3) 'No': If answers to any 1 of questions 1) or 2) for any index test or 3) 'Unclear':	For non‐comparative and between‐person comparison studies Risk is low Risk is high Risk is unclear For within‐person comparative studies Risk is low Risk is high Risk is unclear
Index test (2) concern about applicability
1) Was the diagnostic threshold to determine presence or absence of disease established in a previously published study? E.g., previously evaluated/established algorithm/checklist used lesion characteristics indicative of melanoma used objective (usually numerical) threshold used	Yes – if a previously evaluated/established tool to aid diagnosis of melanoma was used or if the diagnostic threshold used was established in a previously published study No – if an unfamiliar/new tool to aid diagnosis of melanoma was used, if no particular algorithm was used, or if the objective threshold reported was chosen based on results in the current study Unclear – if insufficient information was reported
2) Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication? Study results can only be reproduced if the diagnostic threshold is described in sufficient detail. This item applies equally to studies using pattern recognition and those using checklists or algorithms to aid test interpretation	Yes – If the criteria for diagnosis of melanoma were reported in sufficient detail to allow replication No – if the criteria for diagnosis of melanoma were NR in sufficient detail to allow replication Unclear – If some but not sufficient information on criteria for diagnosis to allow replication were provided
3) Was the test interpretation carried out by an experienced examiner?	Yes – if the test was interpreted by 1 or more speciality‐accredited dermatologists, or by examiners of any clinical background with special interest in dermatology and with any formal training in the use of the test No – if the test was not interpreted by an experienced examiner (see above) Unclear – if the experience of the examiner(s) was NR in sufficient detail to judge or if examiners were described as 'Expert' with no further detail given N/A – if system‐based diagnosis, i.e., no observer interpretation
Is there concern that the index test, its conduct, or interpretation differ from the review question? If answers to questions 1), 2), and 3) 'Yes': If answers to questions 1), 2), or 3) 'No': If answers to questions 1), 2), or 3) 'Unclear':	Concern is low Concern is high Concern is unclear
Reference standard (3) risk of bias
1) Is the reference standard likely to correctly classify the target condition? A) Disease‐positive – 1 or more of the following: histological confirmation of melanoma following biopsy or lesion excision clinical follow‐up of benign‐appearing lesions for at least 3 months following the application of the index test, leading to a histological diagnosis of melanoma B) Disease‐negative – 1 or more of the following: histological confirmation of absence of melanoma following biopsy or lesion excision in at least 80% of disease‐negative participants clinical follow‐up of benign‐appearing lesions for a minimum of 3 months following the index test in up to 20% of disease‐negative participants	A) Disease‐positive Yes – if all participants with a final diagnosis of melanoma underwent 1 of the listed reference standards No – If a final diagnosis of melanoma for any participant was reached without histopathology Unclear – if the method of final diagnosis was NR for any participant with a final diagnosis of melanoma or if the length of clinical follow‐up used was not clear or if a clinical follow‐up reference standard was reported in combination with a participant‐based analysis and it was not possible to determine whether the detection of a malignant lesion during follow‐up is the same lesion that originally tested negative on the index test B) Disease‐negative Yes – If at least 80% of benign diagnoses were reached by histology and up to 20% were reached by clinical follow‐up for a minimum of 3 months following the index test No – if more than 20% of benign diagnoses were reached by clinical follow‐up for a minimum of 3 months following the index test or if clinical follow‐up period was less than 3 months Unclear – if the method of final diagnosis was NR for any participant with benign or non‐melanoma diagnosis
2) Were the reference standard results interpreted without knowledge of the results of the index test? Please score this item for all studies even though histopathology interpretation is usually conducted with knowledge of the clinical diagnosis (from visual inspection or dermoscopy or both). We will deal with this by not including the response to this item in the 'Risk of bias' assessment for these tests. For reviews of all other tests, this item will be retained	Yes – if the reference standard diagnosis was reached blinded to the index test result No – if the reference standard diagnosis was reached with knowledge of the index test result Unclear – if blinded reference test interpretation was not clearly reported
Could the reference standard, its conduct, or its interpretation have introduced bias? For visual inspection/dermoscopy evaluations If answer to question 1) 'Yes': If answer to question 1) 'No': If answer to question 1) 'Unclear': For all other tests If answers to questions 1) and 2) 'Yes': If answers to questions 1) or 2) 'No': If answers to questions 1) or 2) 'Unclear':	For visual inspection/dermoscopy evaluations Risk is low Risk is high Risk is unclear For all other tests Risk is low Risk is high Risk is unclear
Reference standard (3) concern about applicability
1) Are index test results presented separately for each component of the target condition (i.e., separate results presented for those with invasive melanoma, melanoma in situ, lentigo maligna, severe dysplasia, BCC, and cSCC)?	Yes – if index test results for each component of the target condition can be disaggregated No – if index test results for the different components of the target condition cannot be disaggregated Unclear – if not clearly reported
2) Expert opinion (with no histological confirmation) was not used as a reference standard 'Expert opinion' means diagnosis based on the standard clinical examination, with no histology or lesion follow‐up ***do not complete this item for teledermatology studies	Yes – if expert opinion was not used as a reference standard for any participant No – if expert opinion was used as a reference standard for any participant Unclear – if not clearly reported
3) Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Yes – if histology interpretation was reported to be carried out by an experienced histopathologist or dermatopathologist No – if histology interpretation was reported to be carried out by a less experienced histopathologist Unclear – if the experience/qualifications of the pathologist were NR
Is there concern that the target condition as defined by the reference standard does not match the review question? If answers to all questions 1), 2), and 3) 'Yes': If answers to any 1 of questions 1), 2), or 3) 'No': If answers to any 1 of questions 1), 2), or 3) 'Unclear': ***For teledermatology studies only If answers to all questions 1) and 3) 'Yes': If answers to questions 1) or 3) 'No': If answers to questions 1) or 3) 'Unclear':	Concern is low Concern is high Concern is unclear ***For teledermatology studies only Concern is low Concern is high Concern is unclear
Flow and timing (4) risk of bias
1) Was there an appropriate interval between index test and reference standard? A) For histopathological reference standard, was the interval between index test and reference standard ≤ 1 month? B) If the reference standard includes clinical follow‐up of borderline/benign‐appearing lesions, was there at least 3 months' follow‐up following application of index test(s)?	A) Yes – if study reports ≤ 1 month between index and reference standard No – if study reports > 1 month between index and reference standard Unclear – if study does not report interval between index and reference standard B) Yes – if study reports ≥ 3 months' follow‐up No – if study reports < 3 months' follow‐up Unclear – if study does not report the length of clinical follow‐up
2) Did all participants receive the same reference standard?	Yes – if all participants underwent the same reference standard No – if more than 1 reference standard was used Unclear – if not clearly reported
3) Were all participants included in the analysis?	Yes – if all participants were included in the analysis No – if some participants were excluded from the analysis Unclear– if not clearly reported
4) For within‐person comparisons of index tests Was the interval between application of index tests ≤ 1 month?	Yes – if study reports ≤ 1 month between index tests No – if study reports > 1 month between index tests Unclear – if study does not report the interval between index tests
Could the participant flow have introduced bias? For non‐comparative and between‐person comparison studies If answers to questions 1), 2), and 3) 'Yes': If answers to any 1 of questions 1), 2), or 3) 'No': If answers to any 1 of questions 1), 2), or 3) 'Unclear': For within‐person comparative studies If answers to all questions 1), 2), 3), and 4) 'Yes': If answers to any 1 of questions 1), 2), 3), or 4) 'No': If answers to any 1 of questions 1), 2), 3), or 4) 'Unclear':	For non‐comparative and between‐person comparison studies Risk is low Risk is high Risk is unclear For within‐person comparative studies Risk is low Risk is high Risk is unclear
BCC: basal cell carcinoma; cSCC: cutaneous squamous cell carcinoma

Appendix 8. Summary study details: in‐person evaluations for detection of invasive melanoma and atypical intraepidermal melanocytic variants

Study author Pathway (clear/unclear) Other target condition assessed	Study type Setting Country Participants/lesions	Inclusion criteria	Index tests (algorithm) Diagnostic approach	Threshold	Observer qualifications (number) Experience	Reference standard Final diagnoses Prevalence	Exclusions
Ahnlide 2016 Referred (selected on reference) (u)	WPC‐algorithms R‐CS Secondary Sweden NR/309	Excised melanocytic skin lesions with recorded dermoscopy ABCD score and clinician's preliminary diagnosis. Preliminary diagnosis of LM or SN excluded	Dermoscopy 1. no algorithm 2. ABCD In‐person	1. Subjective impression (diagnosis of MM) 2. > 4.75; > 5.45	Dermatology registrar or consultants (n = 13; experienced unit; dermoscopy training); visiting residents data excluded	Histology MM 23; MiS 23 BN: 263	57 lesions with missing scoring; 5 non‐melanocytic diagnosis; 5 with pre‐op diagnosis of LM or SN; 1 with ambiguous histopathological diagnosis
Bauer 2000 Referred (c)	WPC NR‐CS Secondary Italy 311/315	PSL examined during a campaign for the early diagnosis of cutaneous melanoma (CM)	Dermoscopy (no algorithm) (Also evaluated CAD‐Dermoscopy)	NR Subjective impression (diagnosis of malignancy)	Dermatologist (n = 3; trained in recognition of PSLs) Consensus of 3 (expert consult for disagreements)	Histology MM 30; MiS 12 'Atypical' dysplastic 25; BN 212; NML 36	‐
Benelli 1999 Referred (selected on reference) (u)	WPC P‐CS Secondary Italy NR/401	All PSL observed and excised at the Dermatologic Surgery Department	1. VI (ABCDE) 2. Dermoscopy (7FFM) In‐person	1. ≥ 1 to all 5 characteristics present 2. Score ≥ 2	Dermatologist (n = 2; exp NR) Consensus of 2	Histology MM 54; MiS 6 BCC 1 BN 337; LS 5; SK 1 60/401; 15%	NR
Bono 2002a Referred (selected on reference) (c)	WPC P‐CS Specialist clinic Italy 298/313	PSL with a more or less important suspicion for MM on VI and/or dermoscopy	1 .VI (no algorithm) 2. Dermoscopy (no algorithm) In‐person (also evaluates CAD‐Dermoscopy)	VI ‐ subjective impression Dermosc ‐ ≥ 1 characteristic present	Surgical oncologist (n = 4; high) Single observer	Histology MM 55; MiS 11 BCC 6; 8 SK; 3 SN; BN 230 66/313; 21%	NR
Bono 2002b Referred (selected on reference) (c)	WPC P‐CS Specialist clinic Italy 157/161	PSL ≤ 6 mm requiring surgical biopsy for diagnosis based on clinical or dermoscopic suspicion of MM	1. VI (no algorithm) 2. Dermoscopy (no algorithm) In‐person	VI ‐ subjective impression Dermoscopy‐ ≥ 1 characteristic present	Surgical oncologist (n = 2; high) Single observer	Histology MM 10; MiS 3 BCC 2; SK 4; SN 5; BN 124 13/161; 8%	NR
Bono 2006 Referred (selected on reference) (c)	WPC R‐CS Specialist clinic Italy 204/206	PSL ≤ 3 mm undergoing excision due to a more or less important suspicion for MM on VI and/or dermoscopy	1. VI (no algorithm) 2. Dermoscopy (Menzies) In‐person	VI ‐ subjective impression Dermoscopy‐ NR	NR; assumed surgical oncologist as per Bono 2002a; Bono 2002b (n = 4; exp NR) Single observer	Histology MM 19; MiS 4 SN 3; BN 169; Other 11 23/206; 11%	NR
Broganelli 2005 Referred (selected on reference) (u)	NC P‐CS Secondary Italy NR/638	PSL undergoing excision; 2x2 for melanocytic only included	Dermoscopy (7PCL) Unclear if in‐person or image‐based	> 1 change in minor criteria or ≥ 1 major characteristic present	Dermatologist (assumed) (n = NR; exp NR)	Histology MEL 108 ‘Non‐melanoma’ 530	‐
Carli 1994 Equivocal (selected on reference) (c)	NC NR‐CS Secondary Italy 67/67	Clinically suspicious melanocytic lesions undergoing excision for diagnostic purposes (obvious MM excluded)	Dermoscopy (pattern analysis) In person	Irregular pigmented network + ≥ 1 other listed characteristic	Dermatologist (n = 2; exp High) Consensus of 2	Histology MM 3; MiS 2 BN 62 5/68; 7%	‐
Carli 2002a Referred (selected on reference) (u)	WPC R‐CS Secondary Italy NR/256	Clinically equivocal and suspicious PSL subjected to excisional biopsy at the Institute of Dermatology	1. VI (no algorithm) 2. Dermoscopy (pattern) In‐person (Dermoscopy – image‐based)	Subjective impression	Dermatologist (n = 2; High exp – “extensive experience in both clinical and dermoscopic diagnosis”) Consensus of 2	Histology MM 40; MiS 14 BCC 5 BN 177; SN 16; SK 4 54/256; 21%	NR
Cristofolini 1994 Referred (selected on reference) (u)	WPC P‐CS Secondary Italy NR/220	Patients with PSL presenting during a campaign for the early diagnosis of cutaneous melanoma at the Dermatology Department	1. VI (ABCDE) 2. Dermoscopy (pattern) In‐person	1. ≥ 2 characteristics present 2. ≥ 1 characteristic present	Dermatologist (n = 4; High exp ‐ dermatologists had all been trained in the recognition of pigmented lesions) Unclear observer interpretation	Histology MEL 33 BCC 0 BN 181; SK 4; 2 other 33/220; 15%	NR
Dreiseitl 2009 Referred (c)	NC P‐CS Specialist clinic Austria 458/3021	Patients presenting at PSL clinic that serves as a secondary and tertiary referral centre	Dermoscopy (no algorithm)	NR Subjective impression (diagnosis of MM)	Dermatologist (n = 1; ‘expert’) Single observer	Histology or FU (6 months) MEL 31 (27 participants) 'Benign': 2990 (431 participants) 27/458; 6%	806 lesions (53 participants) with inadequate follow‐up
Duff 2001 Referred (selected on reference) (c)	NC R‐CS Specialist clinic UK NR/2372	Excised lesions recorded on PSL database	Dermoscopy (no algorithm) In person	Subjective assessment (decision to excise)	Plastic surgeon (n = 1; exp NR) Single observer	Histology or FU MM 400; MiS 186; BCC: 316; cSCC: 97 Dysplastic 195; "other" 14; 'Benign' (not excised): 1164 586/2372; 25%	NR Results for BCC; SCC not disaggregated from benign lesions
Durdu 2011 Referred (selected on reference) (u) Any	WPC P‐CS Secondary Turkey 176/200	PSL that could not be diagnosed with only dermatologic physical examination; 2x2 included for melanocytic subset	Dermoscopy (ABCD; non‐melanocytic excluded first) (Also evaluated exfoliative cytology) In person	NR	Dermatologist (n = 1; exp NR) Single observer	Histology MEL: 10; BCC: 34; Other malignant 2 SK 24; BN 100; DF 12; warts 16; dirt 1; other 1 10/200; 5%	‐
Feldmann 1998 Referred (selected on reference) (u) Invasive melanoma	NC P‐CS Secondary Austria NR/500	Melanocytic lesions examined by dermatoscopy prior to excision	Dermoscopy (ABCD) In person	> 5.45	NR (n = NR; exp NR) Unclear observer interpretation	Histology MM 25; MiS 5 BN 272; dysplasia 190; lentigines 7; lentigo naevi 1 30/500; 6%	NR
Gokdemir 2011 Referred (selected on reference) (u)	NC NR‐CS Secondary Turkey 362/449	Patients with melanocytic and non‐melanocytic skin lesions with dermoscopic and histologic diagnoses	Dermoscopy (no algorithm) Unclear if in‐person or image‐based	Subjective assessment (diagnosis of MM)	Dermatologist (n = NR; exp High “at least 2 years’ experience with Molemax II”) Unclear observer interpretation	Histology MEL: 13; BCC: 45 Benign: 390 13/448; 3%	Bham team: 1 BCC moved from FP to TN)
Grimaldi 2009 Limited prior testing (c)	WPC P‐CS Primary Italy 197/235	Cutaneous PSL requiring confirmation of diagnosis by teledermatology	1. VI (no algorithm) 2. Dermoscopy (no algorithm) In‐person (single) (Also evaluated Teledermatology)	Subjective impression (‘suspicious for malignancy’)	GP (n = 13) Assumed Low (expertise NR; simple protocols for diagnosis provided for study purposes)	Histology/clinical FU (6 months) MEL: 5; BCC 0; benign 230 (NR) 5/235; 2%	NR
Guitera 2009a (Modena) Referred (selected on reference) (c)	WPC P‐CS Secondary Italy 195/195	Lesions excised on the basis of clinical suspicion (history, dermoscopy examination, and/or digital monitoring)	Dermoscopy (pattern analysis) (Also evaluated RCM) In‐person	Subjective assessment (diagnosis of MM)	Dermatologist (n = 2; exp High) Single observer	Histology MM 61; MiS 18 BN 116 (including 22 SN) 79/198; 41%	Only 50% of imaged naevi were included (randomly selected from the image database prior to analysis) to reduce the MM/naevus ratio
Haenssle 2010a (FV) Referred (u)	NC P‐CS Secondary Germany 688/11137 FV: 8449	Participants at increased risk for melanoma: > 50 common and/or ≤ 3 atypical naevi; atypical mole syndrome (AMS); or familial atypical mole and multiple melanoma syndrome. (First visit data included here)	Dermoscopy (7PCL)	≥ 3	Dermatology residents (n = 13; formally trained in dermoscopy and supervised by experienced dermatologist) Consensus of 2	Histology or FU (every 3, 6, or 12 months) Full sample: MM 77; MiS 50; BCC 2 BN 1047; SN 16; SK 12; Other benign 9935 40/8449; 0.005%	‐
Haenssle 2010b (FU) Follow‐up (u)	NC P‐CS Secondary Germany Full sample; 688/11137 FU: 2688 lesions	Participants at increased risk for melanoma: > 50 common and/or ≤ 3 atypical naevi; atypical mole syndrome (AMS); or familial atypical mole and multiple melanoma syndrome (FU data only included here)	Dermoscopy (7PCL) In person	≥ 3	Dermatology residents (n = 13; formally trained in dermoscopy and supervised by experienced dermatologist) Consensus of 2	Histology or FU (every 3, 6, or 12 months) Full sample: MM 77; MiS 50; BCC 2 BN 1047; SN 16; SK 12; Other benign 9935 87/2688; 3%	‐
Kittler 1999 Referred (selected on reference) (u)	WPC‐algorithms P‐CS Secondary Austria 352/373	Melanocytic PSL < 1 cm in diameter, consecutively excised	Dermoscopy 1. ABCD 2. ABCDE (developed in this study) In‐person	1. Sensitivity at range of specificities (randomly sampled 75% spec) (author communication suggests > 4.75 used) 2. cutoffs between 1.30‐7.35	Dermatologist (assumed) (n = NR; exp NR) Unclear observer interpretation	Histology MM 55; MiS 18 SK 4; BN 126; atypical naevi 113; congenital naevi 3, SN 13; blue naevi 7; solar lentigines 14; DF 1; combined naevi 2 73/356; 21%	Non‐melanocytic lesions (n = 17; including angiomatous tumours, pigmented SK, DF, and pigmented BCC) "easily distinguished by standard ELM criteria and pattern analysis"
Langley 2007 Referred (selected on reference) (u)	WPC P‐CS Specialist clinic Canada 125/125	Patients with lesions scheduled for excision at the PLC to either remove atypical naevi or to rule out melanoma or for cosmetic reasons; excluded if lesion not amenable to RCM or prior diagnosis biopsy	Dermoscopy (pattern analysis) (Also evaluated RCM) In‐person	Subjective assessment (diagnosis of MM)	Dermatologist (assumed); (n = 1; exp NR)	Histology MM 22; MiS 15 BN 88 37/125; 30%	Technical difficulties with imaging (n = 2)
Menzies 2009 Limited prior testing (c) Any	WPC‐algorithms P‐CS Primary Australia NR/374	PSL that would be biopsied or referred on after routine naked‐eye examination	1 .VI (no algorithm) 2. Dermoscopy (no algorithm) In‐person (single)	Subjective impression ("correct diagnosis of melanoma"; excise decision)	GP (n = 62) Assumed Low (trained in dermoscopy for study; required history of excision or referral of at least 10 PSLs over the previous 12‐month period but no prior dermoscopy use)	Histology/clinical FU (3‐6 months)/expert diagnosis MEL: 32; BD 2; benign 323; unknown 9 4%	6 BCC and 2 BD excluded by authors, 43 excluded as both VI + Dermoscopic diagnoses not available
Morales Callaghan 2008 Referred (selected on reference) (u)	WPC P‐CS Secondary Spain 166/200	Randomly selected melanocytic lesions; melanocytic on both clinical and dermoscopic criteria	1. VI (no algorithm) 2. Dermoscopy (no algorithm) In‐person	NR	Dermatologist (n = 2; high exp – “experience in dermoscopy”) Consensus of 2	Histology MEL: 6 BN 184; SN 1; other 9 6/200; 3%	NR
Nachbar 1994 Referred (selected on reference) (u)	NC P‐CS Secondary Germany NR/194	Pigmented melanocytic skin lesions consecutively excised	Dermoscopy (ABCD) In person (excluded VI data as dermoscopy also used for VI)	> 5.45	Dermatologist (assumed) (n = NR; exp High)	Histology MEL: 69; BCC 3 BN 103; SK 19 69/194; 36%	‐
Soyer 1995 Equivocal (selected on reference) (c) Any	WPC NR‐CS Austria NR/159	PSL difficult to diagnose on clinical grounds alone	1. VI (no algorithm) 2. Dermoscopy (pattern) In‐person	NR	Dermatologist (n = 2; exp High; "the examination was performed by a dermatologist expert in dermoscopy") Single observer	Histology MM 50; MiS 15 BCC 3; SK 18; AK 4; BN 61; other 7 65/159; 41%	NR
Stanganelli 2000 Referred (c) Any	WPC R‐CS Specialist clinic Italy NR/3372	PSL referred by dermatologists and general practitioners either for pre‐surgical assessment or consultation	1. VI (ABCD) 2. Dermoscopy (no algorithm) In‐person (single)	NR Subjective impression	NR (assumed dermatologist ‐ described as one of the co‐authors; n = 1)	Histology/registry FU MEL: 55 BCC 43; benign 3274 55/3372; 2%	NR BCC: 3 BCCs considered to be MM were classed as TN rather than FP for review purposes
3PCL: three‐point checklist; 4PCL: four‐point checklist; 7FFM: seven features for melanoma; 7PCL: seven‐point checklist; ABCD(E): asymmetry, border, colour, differential structures (enlargement); AJCC: American Joint Committee on Cancer; AK: actinic keratosis; AMN; acral melanocytic naevi; BCC: basal cell carcinoma; BD: Bowen’s disease; Bham: Birmingham; BN: benign naevi; BPC: between person comparison (of tests); c: clearly positioned on clinical pathway; CCD: compact disc; CAD: computer‐assisted diagnosis; CASH: colour, architecture, symmetry and homogeneity; CCS: case‐control study; CM: cutaneous melanoma; CMM: cutaneous malignant melanoma;CS: case series; cSCC: cutaneous squamous cell carcinoma; DF: dermatofibroma; ELM: epiluminescence microscopy; FP: false positive; FU: follow‐up; GP: general practitioner; IDS: International Dermoscopy Society; IQR: interquartile range; LK: lichen sclerosis; LM: lentigo maligna; LP: lichen planus; LS: lentigo simplex; MEL: invasive melanoma or melanoma in situ; MM: malignant (invasive) melanoma; MiS: melanoma in situ; MN: melanocytic naevi; N/A: not applicable; NC: non‐comparative; NML: non melanocytic lesion; NR: not reported; P: prospective; PLC: pigmented lesion clinic; PSL: pigmented skin lesion; R: retrospective; RCM: reflectance confocal microscopy; RCT: randomised controlled trial; SCC: squamous cell carcinoma; SD: standard deviation; SK: seborrhoeic keratosis; SN: Spitz naevi; SSM: superficial spreading melanoma; TN: true negative; u: unclear position on clinical pathway; VI: visual inspection; WPC: within person comparison (of tests)

Appendix 9. Summary study details: image‐based evaluations for detection of invasive melanoma and atypical intraepidermal melanocytic variants

Study author Pathway (clear/unclear) Other target condition assessed	Study type Country Setting Participants/lesions	Inclusion criteria	Index tests (algorithm) Diagnostic approach	Threshold	Observer qualifications (number) Experience				Reference standard Final diagnoses Prevalence	Exclusions Comments
Alarcon 2014 Equivocal (selected on reference) (c)	WPC P‐CS Specialist clinic Spain 264/264	Dermoscopically equivocal pigmented lesions, assumed to be melanocytic	Dermoscopy (no algorithm) Image‐based (RCM, site, age) (also evaluated RCM)	NR; diagnosis of MM	Dermatologist (n = 3; exp NR) described as expert in RCM Single observer				Histology or FU; 79 followed‐up MEL 92; BCC: 12 BN 107; 53 SK and AK 92/343; 27%	79 lesions without criteria of malignancy on RCM were scheduled for clinical or digital FU
Annessi 2007 Equivocal (selected on reference) (u)	WPC‐algorithms NR‐CS Specialist clinic Italy 195/198	Atypical macular melanocytic lesions; all > 5 mm diameter, with a flat or barely elevated surface and at least 3 of the following features: (a) asymmetry, (b) irregular margins, (c) ill‐defined borders, and (d) colour variegation	Dermoscopy (pattern analysis; 7PCL; ABCD) Image‐based (blinding NR)	NR ‐ likely 'standard'; ABCD ≥ 4.75	Dermatologist (n = 2; exp High) ELM‐experienced dermatologists" Consensus of 2				Histology MM 72; MiS 24 BN 102 96/198; 48%	NR
Argenziano 1998 Referred (selected on reference) (u)	WPC‐algorithms; observer R‐NR Secondary Italy NR/342	Atypical melanocytic skin lesions with dermoscopic images that had undergone biopsy due to clinician suspicion	Dermoscopy (pattern analysis; ABCD; 7PCL) Image‐based (blinded)	Overall diagnosis MM; ABCD > 4.75; 7PCL ≥ 3	Dermatologist (n = 3 experienced; n = 2 less experienced who underwent training) Consensus of 2 (expert) Single (less experienced)				Histology MM 99; MiS 18 BN 225 117/342; 34%	NR
Argenziano 2011 Referred (selected on reference) (c)	WPC‐algorithms CCS Secondary Italy NR/300	Randomly sampled 100 melanomas; 100 excised BN 100 BN that showed no relevant changes to warrant excision during the FU period; all ≤ 15 mm	Dermoscopy (pattern analysis; 7PCL; 7PCL revised) Image‐based (blinded)	Pattern – diagnosis of MM and excise decision; 7PCL ≥ 3; revised ≥ 1)	Dermatologist (n = 8; exp NR) average				Histology or FU MEL 100 57 Clark naevi, 28 SN, 10 congenital naevi, 5 blue naevi; 100 not excised 100/300; 33%	NR
Benelli 2000a Referred (selected on reference) (u)	WPC CCS Secondary Italy NR/600	All small (≤ 6 mm) melanomas and melanocytic naevi consecutively excised over 2 different time periods	1. VI (ABCD) 2. Dermoscopy (7FFM) Image‐based (blinding NR)	Both ≥ 2	Dermatologist (assumed) (n = 3; exp NR) evaluated by 3 different observers; in case of disagreement , the majority view prevailed Consensus of 3				Histology alone MEL 76 BN 524 76/600; 13%	NR
Benelli 2001 Referred (selected on reference) (u)	WPC R‐CS Italy Training images NR/50	Slides of PSL selected for evaluation during a training course on dermoscopy. Lesions not located on head, palms or soles	1. VI (ABCDE) 2. Dermoscopy (7FFM) Image‐based (blinded)	1. ≥ 3 & ≥ 2 2. ≥ 2	Expert author (n = 1); dermatologists (n = 65) Single author ‐ High exp; average result for dermatologist group; exp NR				Histology MM 10, MiS 2 BCC 2 BN 25, SN 5, SK 3, other 2 (1 missing) 12/50; 24%	NR
Binder 1994 Referred (selected on reference) (u)	WPC RCS Secondary Austria NR/200	Images of PSL randomly selected from a image database.	Dermoscopy (pattern analysis) (Also evaluates CAD dermoscopy) Image‐based (blinded)	Subjective impression (diagnosis of MM)	Dermatologist (n = 3; exp High) Consensus of 2				Histology MEL 40 BN 60 40/100; 40%	NR
Binder 1995 Referred (selected on reference) (u)	WPC‐observer RCS Secondary Austria NR/240	PSL with available dermoscopy images, both with and without oil immersion, and histological confirmation of diagnosis.	Dermoscopy (no algorithm) (Dermoscopy with/without oil immersion) Image‐based (blinded)	Subjective impression (diagnosis of MM)	Dermatologist (n = 6 expert; n = 13 non‐expert); Average				Histology MEL 57; BCC: 8 Severe dysplasia: 42; other 'Benign' : 133 57/240; 24%	NR Test results not disaggregated for BCC
Binder 1999 Referred (selected on reference) (u)	WPC‐algorithms RCS Secondary Austria NR/250	Randomly selected, histologically proven PSL with digital dermoscopy images	Dermoscopy (pattern analysis; ABCD) Image‐based (blinded)	Subjective impression (diagnosis of MM); ABCD at > 4.75; > 5.45	Mixed (n = 17; exp mixed) dermatology residents ‐ 5; dermatologist (board‐certified) ‐ 12 Average result				Histology MM 34; MiS 7 BN 182; 13 SN; 14 lentigines 41/250; 16%	NR
Blum 2003a Referred (selected on reference) (u)	WPC‐alg R‐CS Specialist clinic Germany NR/269	Melanocytic skin lesions to be excised because of clinically and/or dermoscopically clear or suspicious malignancy, or by the wish of the patient after clear benign diagnosis*	ABCD (modified); ABCDE (modified) Image‐based (unclear)	NR	Dermatologist (assumed) (n = NR; exp NR) NR				Histology MM 71; MiS 9; lM 4 'Benign': 185 84/269; 31%	*dataset overlaps Blum 2004b so not included in primary analysis, only algorithm comparisons (recruited November 1998‐March 2000)
Blum 2003b Referred (selected on reference) (u)	NC R‐CS Specialist clinic Germany 205/254	All lesions of patients with multiple atypical naevi excised due to suspicious clinical or dermoscopic features, or both, were included*	New (based on Hofmann‐Wellenhof 2001) Image‐based (blinded)	Presence of reticular, globular and homogeneous structures	Dermatologist (assumed) (n = 2; exp NR) Consensus of 2				Histology MM 63; MiS 12 BN 64; dysplastic 96; other nevus 19 75/254; 30%	*dataset overlaps Blum 2004b so not included in primary analysis, only algorithm comparisons (recruited September 1998 to December 1999)
Blum 2004a Referred (selected on reference) (u)	WPC‐observer R‐CS Specialist clinic Germany 157/157	PSL excised due to suspicious clinical and/or dermoscopic features	Pattern analysis Image‐based (blinded)	Level of suspicion 'roughly 50% or more'.	Dermatologist (assumed) (n = 3; with “different experiences in dermoscopy: excellent (A), average (B) and beginner (C)." Single				Histology MM 29;MiS 2 BN 53; dysplastic 59 'epithelial benign’ 13 32/157; 20%	*dataset overlaps Blum 2004b so not included in primary analysis, only observer comparisons (recruited September 1998 to March 1999)
Blum 2004b Referred (u)	WPC‐algorithms P‐CS Specialist clinic Germany NR/837	Melanocytic skin lesions imaged prospectively at the PLC	Dermoscopy (ABCD; 7PCL; 7FFM; Menzies) Image‐based (blinded) (also evaluated CAD‐Dermoscopy)	NR ‐ author confirms "published standards used"	Dermatologist (assumed); n = 1 Single observer				Histology or FU (568 benign examined 2‐3 times in 6 months) MM 71; MiS 9; lM 4 'Benign' 766 84/837; 10%	NR
Bourne 2012 3–Limited testing (selected on reference) (c)	WPC‐algorithms; algorithms R‐CS Australia Primary 46/50	All skin lesions excised to exclude skin cancer (and 3 examples common lesions assessed as clearly benign and not biopsied)	VI (no algorithm) Dermoscopy (3‐point; Menzies; BLINCK*) Image‐based (blinded)	NR	GP (n = 3) Clinical nurse (n = 1) Mixed exp “varying levels of dermatoscopic experience” Average				Histology/clinical FU/expert diagnosis MM 1; MiS 8 BCC 6; SK 5; BN 11; other 19 9/45; 20%	5 non‐pigmented specimens (not further identified) in the set of 50 were excluded from dermoscopic evaluations *data for BLINCK excluded as derivation
Carli 2002a Referred (selected on reference) (u)	WPC NR‐CS Secondary Italy NR/256	Clinically equivocal and suspicious PSL	1. VI (no algorithm) (in‐person) 2. Dermoscopy (pattern analysis) (in‐person and image‐based) Image‐based (age, site provided)	Subjective impression (diagnosis of MM)	Dermatologist (n = 2; exp High; ‘extensive experience in both clinical and dermoscopic diagnosis of PSLs’) Consensus of 2				Histology alone MM 40; MiS 14 BCC: 5 SK 4; BN 168; 9 blue naevi; 16 SN 54/256; 21%	None
Carli 2002b Referred (selected on reference) (u) Any	WPC R‐CS Italy Secondary NR/57	Clinically suspicious or equivocal PSL undergoing excision for diagnostic purposes; all ≤ 14 mm diameter	1. VI (NR) 2. Dermoscopy (NR) Image‐based (blinded)	NR	Dermatologists (n = 2) High exp ('with experience in the field of '); consensus of 2				Histology MM 6, MiS 5 BCC 10 BN 31, SK 1; other 4 11/57; 19%	4 ‘not evaluables’ excluded (NB these differ between clinical images and dermoscopic images (1 MM excluded from VI analysis)
Carli 2003a Referred (selected on reference) (u)	WPC‐algorithms RCS Secondary Italy NR/200	Melanocytic lesions < 14 mm in diameter, excised because they were clinically suspicious or equivocal	Dermoscopy (pattern analysis; ABCD; 7PCL) Image‐based (blinded)	Subjective impression (diagnosis of MM); ABCD > 5.45; 7PCL ≥ 3	Dermatology registrar (n = 5; exp low) Single observer				Histology MM 30; MiS 14 BN 156 44/200; 22%	NR
Carli 2003b Equivocal (selected on reference) (u)	WPC R‐CS Italy Secondary NR/200	Clinically difficult to diagnose or equivocal melanocytic lesions randomly selected; all melanomas < 1 mm thickness.	1. VI (no algorithm) 2. Dermoscopy (own choice) Image‐based (blinding NR)	Subjective impression	Dermatology registrar (n = 2); dermatologists (senior experts n = 2; practicing dermatologists n = 4) Average result				Histology MM 40; MiS 24 BN 136 64/200; 32%	NR
Carrera 2016 Referred (u)	WPC‐algorithms CCS Specialist clinic Multicentre NR/477*	Images of melanocytic lesions including MM with unequivocal histology, and histologically verified naevi or naevi demonstrating stability under sequential dermoscopic imaging over time	Dermoscopy (ABCD; 7PCL; CASH; Menzies; 3PCL; Chaos/Clues) Image‐based (clinical image also provided)	> 4.75; ≥ 3; ≥ 6; 2 negative and 1 positive characteristic; ≥ 2; both present	GP 24; dermatology registrar 25; dermatologist 73; 1 medical student and 7 'other'; Mean 12 years (SD 8.7) dermatology exp; 93.8% "comfortable" using dermoscopy Consensus (≥ 50%)				Histology or FU (sequential dermoscopic imaging over time; n = NR) MEL 119 BN: 358 119/477; 25%	NR *Up to 50 lesions per PLC (1:3 ratio of MEL to BN; 1:1 polarised or nonpolarised images); randomised into 12 image sets of 39 (n = 8) or 40 (n = 7) unique lesions and 5 non‐unique lesion images (2 MEL, 3 BN) repeated in all sets
Dal Pozzo 1999 Referred (selected on reference) (u)	NC PCS Secondary Italy NR/713	PSL observed clinically and dermoscopically	Dermoscopy (7FFM) Image‐based (blinded)	≥ 2	Dermatologist (assumed) (n = 3; exp NR) Consensus of 3				Histology MM 139; MiS 29; BCC: 1 SK 3; BN 536; other 5 168/713; 24%	None All BCC considered TN
di Meo 2016 Referred (selected on reference) (u)	WPC‐algorithms RCS Secondary Italy 125/125	Melanocytic skin lesions that underwent excision (*accuracy data excludes the dysplastic naevi)	Dermoscopy (3PCL; CASH; 4PCL) Image‐based (blinded)	≥ 2 characteristics present; ≥ 8; ≥ 2	Dermatologist (n = 2; exp High) NR				Histology; MEL 32 BN 43 32/75; 43%	50 lesions with mild/moderate dysplasia excluded
Dolianitis 2005 Referred (selected on reference) (u)	WPC‐algorithms R‐CS Multi‐centre Training images NR/40	Melanocytic skin lesions randomly selected from a collection of dermoscopic images belonging to one author	1. VI (no algorithm) 2. Dermoscopy (pattern analysis; Menzies criteria; 7‐point; ABCD) Image‐based (blinded)	1. subjective impression 2. subjective impression; NR; NR; > 4.75	Dermatologists (n = 16); dermatology trainees (n = 16); GPs (n = 35) Mixed exp (“range of experience levels with assessment of skin lesions”); Average result				Histology (n = 39); expert diagnosis (n = 1) MM 18, MiS 2 BN 12; SN 3; other 4 20/20; 50%	NR; poor‐quality images exclusion criterion
Dummer 1993 Equivocal (selected on reference) (c)	WPC P‐CS Secondary Germany NR/771	Patients with skin lesions difficult to diagnose clinically	1. VI (no algorithm) 2. Dermoscopy (pattern analysis) Image‐based (blinding NR)	Unclear (German language); diagnosis of MM	Dermatologist (assumed) (n = 2; exp unclear) limited detail; German paper Single				Histology MM 23 BN 706; SK 4; BMN 32 23/771; 3%	Further 53 non‐melanocytic lesions not included prior to examination (no melanomas present in this group)
Feci 2015 Referred (selected on reference) (u)	BPC RCS Secondary Italy 321/321	PSL suspicious for melanoma and excised; observers randomly allocated to observation with different "stressors"*	Dermoscopy (pattern analysis) Image‐based (blinded)	NR; diagnosis of MM	Dermatologist (n = 3; exp High) 'expert dermatologists' "with at least 10 years’ exp in dermoscopy" NR				Histology MM 99; MiS 33 BN 219 34/107; 32%	NR *Data pooled across arms for primary analysis
Feldmann 1998 Referred (selected on reference) (u) MM	NC P‐CS Secondary Austria NR/500	Melanocytic lesions examined by dermatoscopy prior to excision	Dermoscopy (ABCD) In person	> 5.45; > 4.2	NR (n = NR; exp NR) Unclear observer interpretation				Histology MM 25; MiS 5 BN 272; mild/moderate dysplasia 190; lentigines 7; lentigo naevi 1 30/500; 6%	NR
Ferrari 2015 Equivocal (selected on reference) (u)	WPC R‐CS Secondary Italy NR/322	Melanocytic lesions with equivocal clinical and/or dermoscopic features that underwent excision	Dermoscopy (7‐point) Image‐based (RCM, image) (also evaluated RCM in subgroup)	≥ 3; diagnosis of MM	Dermatologist (n = 1; exp NR) Single				Histology MEL 70 'Benign' naevi: 252 (including 15 SN) 70/322; 22%	90 "positive‐clear cut" lesions scoring 5 or more were excluded from RCM evaluation
Ferris 2015 Referred (selected on reference) (u)	WPC‐observer R‐NR Secondary US NR/65	Dermoscopic images of skin lesions excised on the basis of clinical suspicion of malignancy, with available histologic diagnoses	Dermoscopy (no algorithm) Image‐based (blinded) (Also evaluates CAD‐Dermoscopy)	NR; excise decision	Dermatologist (n = 2 board certified); dermatology residents (n = 10); physician assistants practicing in dermatology (n = 8) Average per group				Histology MM 15; MiS 10 BN 20, blue naevi 2, lentigines 4 , SK 4 25/65; 38%	NR
Friedman 2008 Referred (selected on reference) (u) MM	WPC CCS Secondary/private USA 94/99	An industry database of images of PSL ≤ 6 mm was used to sample images of melanoma and non‐melanoma lesions; high‐grade dysplastic naevi were excluded.	Dermoscopy (no algorithm) Image‐based (site, age, gender) (Also evaluates CAD‐Spectroscopy)	Correct diagnosis; excise decision	Mixed ‐ secondary care (n = 10; exp High) Average result (reports mean and median; mean used)				Histology MM 21; MiS 28; BCC: BN 34; SK 2; 14 other benign 49/99; 49%	NR
Gereli 2010 Referred (selected on reference) (u)	WPC‐algorithms CCS Secondary Turkey NR/96	Lesions considered clinically atypical *before dermoscopic examination and excisional biopsy.	Dermoscopy (3PCL; 7PCL) Image‐based (blinded)	≥ 2 characteristics present; ≥ 3	Dermatologist (assumed) (n = 3; exp mixed) "two experienced and one inexperienced observers" Average result				Histology MM 44 MiS 4 SK 2; blue naevi 2; BN 44 48/96; 50%	NR (*determined by ≥ 3 of: diameter > 5 mm, ill‐defined borders, irregular margins, presence of papular and macular components)
Gilmore 2010 Referred (selected on reference) (u)	NC R‐CS Secondary Austria NR/69	Polarised dermoscopic images of atypical melanocytic lesions	Dermoscopy (no algorithm) Image‐based (blinded)	NR; excise decision	Dermatologist (assumed) (n = 1; exp High) Single observer				Histology MEL 36 BN (dysplastic): 33 36/69; 52%	130 in derivation set of lesions
Glud 2009 Referred (selected on reference) (u)	WPC P‐CS Secondary Denmark 65/83	Patients referred for excision biopsy of where the diagnosis of melanoma could not be excluded on clinical investigation	Dermoscopy (no algorithm) Image‐based (blinded) (Also evaluates CAD spectroscopy)	NR; diagnosis of MM	Dermatologist (n = 1; exp High) Single observer				Histology MM 7; MiS 5; 1 melanoma metastases (included as disease negative) SK 1; BN 57; BD 1; DF 6; other 5 12/83; 14%	NR
Guitera 2009b (Sydney) Referred (selected on reference) (u)	WPC P‐CS Specialist clinic Australia 131/131	Lesions excised on the basis of clinical suspicion (history, dermoscopy examination, and/or digital monitoring)	Dermoscopy (pattern analysis) Image‐based (age, site) (Also evaluates RCM)	NR; diagnosis of MM	Dermatologist (n = 2; exp High; ‘expert’) Single observer				Histology MM 28; MiS 16 BN 87 (including 3 SN) 44/131; 34%	(25 lesions out of 156 were rejected for poor‐quality dermoscopy image, blinded to the diagnostician)
Hauschild 2014 Referred (selected on reference) (u) MM	WPC‐observer CCS* Secondary/private US 130/130	Subset of PSL evaluated in a MelaFind study (Monheit 2011); 65 melanoma and 65 non‐melanoma randomly selected. Excluded ulcerated, non‐pigmented, or located on excluded anatomic sites	Dermoscopy (no algorithm) (Also evaluates CAD spectroscopy) Image‐based (clinical image, patient history)	NR; excise decision	Dermatologists (n = 202; randomised between 2 arms); PSL experts (n = 9) Single observer				Histology MM 36; MiS 29 'Benign' diagnoses: 65 65/130; 50%	*RCT of diagnosis based on clinical/dermoscopic images versus same + MelaFind, with observers randomised between arms
Kittler 1998 Equivocal (selected on reference) (u)	NC NR‐CS Secondary Austria NR/50	PSL images selected on image quality and difficulty of diagnosis; all melanomas has “only subtle ELM features as clues to the malignancy of the lesion .. difficult to differentiate from benign"	Dermoscopy (no algorithm); compared photographic slides and compressed digital images; latter used for review Image‐based (blinded	Subjective impression; diagnosis of MM	Dermatologist (n = 8; exp NR) described as "pre‐trained in ELM" Single (randomly sampled one for inclusion)				Histology MEL: 23 SK 1; BN 26 23/50; 46%	NR
Kittler 2001 Follow‐up (u)	NC CCS Secondary Austria 20/80	Images retrieved from a PSL database; melanocytic skin lesions from patients with multiple atypical naevi and with digital dermoscopy follow‐up*	Dermoscopy (no algorithm) Image based (blinded)	NR; excise	Dermatologist (n = 24; exp mixed – including basic dermoscopy experience (n = 9), dermoscopy training but basic experience (n = 10), experienced and trained dermatologists (n = 5) Average result reported				Histology or FU MM 5, MiS 5 BN 70 10/80; 13%	NR *10 patients with early melanomas and 10 other patients randomly selected; benign melanocytic skin lesions taken at random from these 20 participants.
Malvehy 2014 Referred (selected on reference) (u)	WPC P‐CS Multicentre 1611/1943	Patients with skin lesions selected for total excision to rule out melanoma; dermatologists were encouraged to enrol a mix of lesions with an even distribution of low‐, medium‐ and high‐risk lesions	Dermoscopy (no algorithm; ABCD; 7PCL; 7PCLrev) Image‐based (clinical image?) (Also evaluates CAD ‐ Nevsiense)	Dx of malignancy; > 4.75; > 5.45; 7PCL NR	Dermatologist (n = 3; exp NR) dermatologists with 2–5 years of experience in dermoscopy assessment Unclear				Histology VI/dermoscopy only – MM 126; MiS 112 Breakdown of non‐diseased not provided for VI/dermoscopy sample (Full sample of 1942: MM 153; MiS 112; BCC 48, cSCC 1; MCC 1 BN 1497; 5 SN, 51 SK, 6 SCC in situ; 8 AK; 61 other) 238/1678; 14%	473 excluded from total sample – mainly due to investigator oversight or inability to render a final histopathological diagnosis; 74 were device‐related (60 with inadequate reference measurement quality and 14 to device failure) 242 excluded from VI/dermoscopy analysis due to image quality
Menzies 2005 Referred (u)	WPC‐observer R‐CS Specialist clinic Multicentre Australia, US, Germany NR/786*	PSL imaged using SolarScan at 9 different clinical centres including specialist referral centres and private skin cancer clinics	Dermoscopy (no algorithm) Image‐based (clinical image and pt history provided) (also evaluated CAD Dermoscopy)	Subjective impression (diagnosis of MM)	Dermatologist; (n = 3 international experts); dermatologists (n = 4); dermatology registrars (n = 3); GPs (n = 3) Average reported per group				Histology or FU (26% of full sample FU; 3% expert diagnosis) Sydney Melanoma Unit only (n = 78) MM 5;MiS 6; lM 2 BN 65 13/78; 17%	*Only the 78 lesions from the Sydney Melanoma Unit included in the VI/Dermoscopy evaluation
Menzies 2008 Referred (selected on reference) (u)	WPC‐algorithms CCS Multicentre NR/497	Dermoscopic amelanotic (with no melanin pigmentation) or hypomelanotic (a melanin pigmentation area of < 25% of the total surface area or slightly pigmented but with no dark brown, deep blue, or black pigmentation) lesions*	Dermoscopy (7PCL; Menzies; 3PCL) Image‐based (blinding NR) (also developed new algorithm on 80% of sample and tested on 20% but number disease positive NR for the test set to allow 2x2 to be estimated)	≥ 3; standard threshold; ≥ 2	Dermatologist (assumed) (n = 12; exp in dermoscopic evaluation scored 99 individual morphological features in approximately equal sample sizes Single observer				Histology and FU (numbers NR; some naevi included that showed no changes following consecutive digital monitoring) MM 91; 14 MiS; 126 BCC; 4 cSCC BN 159; SN 11; SK 22; DF 17; BD 7; KA 1; AK 8; other 37 105/497; 21%	NR *All melanomas included, and a random selection of melanocytic and nonmelanocytic lesions on a non‐melanoma to melanoma ratio of 3:1
Pagnanelli 2003 Referred (selected on reference) (u)	WPC‐algorithms R‐NR Setting NR Italy NR/20	Images of PSL from the training set of the Consensus Net Meeting on Dermoscopy (CNMD), selected by 2 experts*	Dermoscopy (pattern analysis; Menzies; 7PCL; ABCD) Image‐based (clinical image)	Subjective impression; correct diagnosis; algorithm NR	Mixed ‐ sec (n = 16; exp NR) Average result				Histology MEL 6; BCC: 2 SK 2; CN 8; SN 2 6/20; 30%	NR Data not disaggregated for BCC *pre‐ and post‐ dermoscopy training data presented for each algorithm
Piccolo 2002a Referred (selected on reference) (u)	WPC‐observer R‐CS Secondary Italy 289/341	PSL excised because of equivocal dermoscopic findings or at the patient’s request	Dermoscopy (no algorithm) Image‐based (clinical image) (Also evaluates CAD dermoscopy)	NR; diagnosis of MM	Dermatologist (n = 1 expert); dermatology resident (n = 1) Single observer				Histology MEL 13 SK 3; BN 316; DF 7; angiomas 2 13/341; 4	NR
Piccolo 2014 Referred (selected on reference) (u)	WPC‐observer R‐CS Secondary Italy 165/165	Dermoscopically atypical PSL *	Dermoscopy (ABCD) Image‐based (blinded) (Also evaluates CAD dermoscopy)	> 4.74	Dermatologists (n = 3; 1 expert, 2 non‐expert); GP (n = 1; underwent dermoscopic training by studying an interactive atlas of dermoscopy between time periods T0 and T1). Single (results per observer)				Histology MM 23; MiS 10 BN 105; CN; 19 SN; 5 blue naevi; 3 dermal naevi. 33/165; 20%	NR *Images assessed at T0 and at 6 months (T1)
Pizzichetta 2002 Referred (selected on reference) (u)	WPC‐algorithms R‐CS Specialist clinic Italy 123/129	Small (≤ 5 mm) melanocytic skin lesions surgically excised	Dermoscopy (pattern analysis; ABCD) Image‐based (blinded)	Dx of MM; > 4.75; > 5.45	Dermatologist (assumed) (n = 2; exp NR) Single observer				Histology MEL 5 lesions BN 124 lesions 5/129; 4%	NR
Pizzichetta 2004 Referred (selected on reference) (u)	WPC R‐CS US/Italy Secondary 151/151	Clinical and/or dermoscopic hypomelanotic (extent of pigmentation ≤ 30%) and amelanotic skin lesions	1. VI (no algorithm) 2. Dermoscopy (pattern) Image‐based (clinical image)	Subjective impression	NR (presume dermatologist; n = 1) Exp NR; single observer				Histology AHM 34, MiS 5 BCC 25, SCC 5 BN 47, SN 5, SK 8, other 18 39/108; 36% (analysed)	23 lesions excluded due to image quality; further 43 lesions were not available for evaluation by clinical images ("mainly benign melanocytic lesions"
Pupelli 2013 Equivocal (selected on reference) (c)	WPC CCS Specialist clinic Italy 96/96	Melanomas < 5 mm consecutively excised; + 3 histologically proven small‐diameter naevi per included melanoma	Dermoscopy (7‐point) Image‐based (RCM, site, age) (also evaluated RCM)	≥ 3; diagnosis of MM	Dermatologist (assumed) (n = NR; exp NR)				Histology MM 13; MiS 11 BN 72 (including 7 SN) 24/96; 25%	NR
Rigel 2012 Referred (selected on reference) (u)	WPC R‐NR Unclear US NR/24	PSL analysed as part of a prior study using a MSDSLA system (Monheit 2011)	Dermoscopy (no algorithm) Image‐based (clinical image) (Also evaluates CAD Spectroscopy)	NR; excise decision	Dermatologist (n = 179; exp mixed) Average result				Histology MEL 5; 'Benign' diagnoses: 19 5/24; 21%	‐
Rosendahl 2011 3–Limited testing (selected on reference) (u) Any	WPC‐alg R‐CS Australia Primary 389/463	PSL submitted for histology from the primary care skin cancer practice of one author	1. VI (no algorithm) 2. Dermoscopy (pattern; Chaos and Clues)	1. subjective impression 2. NR; both characteristics present		Dermatologist (n = 1) High exp (confirmed by study author) Single observer			Histology MM 9; MiS 20 BCC 72; SCC 5 BN 217; BD 18; AK 14*; BNM 140 29/463; 6%	3 poor‐quality images excluded *AK considered malignant by study authors
Rubegni 2012 Equivocal (selected on reference) (u)	WPC‐algorithms R‐CS Secondary Italy 107/107	Palmoplantar (acral) PSL excised over a 3‐year period. All with clinical/dermoscopic suspicious features in the absence of any clear benignity pattern	Dermoscopy (pattern analysis; 3‐step algorithm (Koga 2011)) Image‐based (blinded)	diagnosis of MM; excise decision (3‐step)			Dermatologist (n = 2; exp High ‐ 20 years’ experience in dermoscopy) Single observer data		Histology MM 21; MiS 4 'Benign' diagnoses: 82 25/107; 23%	NR
Rubegni 2016 Referred (selected on reference) (u)	NR R‐CS Secondary NR‐Italy 95/95	Melanocytic skin lesions showing clear‐cut dermoscopic features of regression and excised for suspected malignancy	Dermoscopy (pattern analysis) Image‐based (blinding NR)	NR; diagnosis of MM	Dermatologist (n = 3; exp High) experienced dermoscopists Single observer				Histology MEL 45 BN 50 45/95; 47%	NR
Sboner 2004 Referred (selected on reference) (u)	NC R‐CS Secondary NR‐Italy NR/152	Melanocytic lesion images acquired consecutively	Dermoscopy (no algorithm) Image‐based (blinded)	NR; diagnosis of MM			Dermatologist (n = 8; exp NR) Single observer		Histology; MM 31; MiS 11 BN 110 42/152; 28%	NR
Seidenari 1998 Referred (selected on reference) (u)	WPC‐Obs CCS Secondary Italy NR/90	Patients referred by dermatologists or general physicians with ≥ 1 PSL difficult to interpret on clinical grounds alone, numerous PSLs, or because the patients were at increased risk for melanoma or prior malignancy	Dermoscopy (no algorithm) Image‐based (blinded)	Subjective impression; diagnosis of MM			Dermatologist (n = 2; 1 expert, routinely used videomicroscopy; 1 non‐expert) Single observer		Histology MEL 31 59 "nonmelanomas” including dysplastic naevi" 31/90; 34%	‐
Seidenari 2005 Referred (selected on reference) (u)	WPC R‐CS Specialist clinic Italy NR/603	Melanocytic lesions, which had undergone surgical excision for clinical, dermoscopic, or cosmetic reasons after referral by a dermatologist for examination of a particular lesion or of the whole skin	Dermoscopy (pattern analysis) Image‐based (blinded)	Correct diagnosis of MM (atypia grade 3); excise decision (atypia grade 2 and above)			Dermatologist (n = 2) Consensus of 2		Histology MEL 112 BN 491 112/603; 19%	NR
Seidenari 2007 Referred (selected on reference) (u)	NC R‐CS Setting NR Italy NR/243	Dermoscopic images of melanocytic lesion that had undergone excision	Dermoscopy (no algorithm) Image‐based (blinded)	NR; diagnosis of MM			Mixed (n = 4; exp mixed) Single observer		Histology MM 35; MiS 8 BN 200 43/243; 18%	‐
Skvara 2005 Follow‐up (u)	WPC‐alg CCS Secondary Austria NR/126	Consecutive lesions showing changes over time during digital dermoscopy follow‐up that were excised at 2 clinics	Dermoscopy (ABCD; 7PCL) Image‐based (blinded)	> 4.75; ≥ 3			Dermatologist (n = 2; exp High)		Histology MEL 63 BN 63 63/126; 50%	NR
Stanganelli 1998a Referred (selected on reference) (u) Any	WPC R‐CS Italy Training images Italy NR/30	PSL images selected from computerised files of the skin cancer clinic	1. VI (no algorithm) 2. Dermoscopy (no algorithm) Image‐based (clinical image)	NR; clinical diagnosis			Dermatologists (n = 20) Exp NR (“experience in ELM but (with) no formal training”) Average result		Histology MEL 10 BCC 4 BN 10, SK 3, other 3 10/30; 33%	NR BCC results not disaggregated
Stanganelli 1999 Referred (selected on reference) (u)	WPC‐Obs CCS Specialist clinic Italy NR/30	PSL images selected from database for training study	Dermoscopy (no algorithm) Image‐based (clinical image)	Correct diagnosis MM			Dermatologist (assumed) (n = 83; exp mixed) Median result pre‐ and post‐ dermoscopy training		Histology MM 10; MiS 1 14 BN; 5 BNM 11/30; 37%	NR
Stanganelli 2005 Referred (u)	WPC R‐CS Italy Specialist clinic NR/477	Melanocytic lesions referred to Skin Cancer Unit for clinical and dermoscopic evaluation.	1. VI (no algorithm) 2. Dermoscopy (no algorithm) Image‐based (clinical image also provided) (also evaluated CAD Dermoscopy)	NR (diagnosis of MM)		Dermatologist (n = 3); GP (n = 3) Dermatologists – High exp (“2 years dermoscopy experience”); exp NR for GPs, assumed Low Average reported			Histology/registry FU MEL 31 BN 103 31/134; 23%	NR
Stanganelli 2015 Follow‐up (u)	WPC R‐CS Specialist clinic Italy 70/70	Lesions excised on the basis of clinical and/or dermoscopic changes at follow‐up suggesting a malignancy	Dermoscopy (7‐point revised ‐ FU) Image based (baseline image provided) (also evaluated RCM)	‘major change’ (diagnosis of MM)				Dermatologist (assumed) (n = NR; exp NR)	Histology MM 11; MiS 1 BN 55; BNM 3 12/70; 17%	NR
Stolz 1994a Equivocal (selected on reference) (u)	NC R‐CS Secondary Germany NR/157 (79 in test set included)	Equivocal PSLs with size < 9 x13 mm, melanoma tumour thickness of 1 mm and melanoma Clark's level ≤ III	Dermoscopy (ABCD) Image‐based (blinded)	> 5.45; diagnosis of MM				NR (n = 1; exp NR) Single	Histology MEL: 48 (test set only) BN 31 48/79; 61%	NR
Tan 2009 Referred (selected on reference) (u)	WPC‐Obs CCS Training images UK NR/30	Test series of images of melanomas and benign lesions	Dermoscopy (pattern analysis modified) Image‐based (clinical image)	Excise decision				Mixed (n = 6; exp mixed) Average result; pre‐ and post‐ dermoscopy training	Histology MEL 15 Other: 11 BN; 3 SK; 1 vascular 15/30; 50%	NR
Tenenhaus 2010 Referred (u)	NC CCS Secondary France NR/227	Dermoscopic images of all melanoma lesions recorded on two databases, + 227 randomly selected benign lesions	Dermoscopy (no algorithm; based on ABCD and others) Image‐based (clinical image also provided)	NR; subjective impression (diagnosis of MM; excise decision)			Dermatologist (n = 5; exp High) Single observer		Histology + other (65/227 benign not excised; assume expert diagnosis) MM 28; lM 4 BN (excised) 165; 'benign’ not excised: 62 32/27; 14%	NR
Unlu 2014 Referred (selected on reference) (u)	WPC‐algorithms R‐CS Specialist clinic Turkey 115/115	Melanocytic lesions excised at PLC	1. VI (no algorithm) 2. Dermoscopy (ABCD; 7PCL; 3PCL; CASH) Image‐based (blinded)	1. NR; diagnosis of MM 2. > 5.44; ≥ 3; ≥ 2; ≥ 8				Dermatologist (assumed) (n = 3; exp High) VI appears to be in clinic diagnosis (single observer); derm images scored by 3 other ‘expert’ dermatoscopists Consensus of 3	Histology alone MEL 24 BN 91 (including 6 SN) 24/115; 21%	NR
Wells 2012 Referred (selected on reference) (u)	WPC CCS Industry database US NR/47	PSL selected from a repository of lesions amassed during an acquisition study conducted by MELA Sciences Inc for the US Food and Drug Administration	Dermoscopy (no algorithm) Image‐based (clinical image, patient history) (Also evaluates CAD spectroscopy)	NR; MM or not				Dermatologist (n = 39; exp NR). Average	Histology MEL 23 'Benign' diagnoses: 24 23/47; 49%	‐
Winkelmann 2016 Referred (selected on reference) (u)	WPC CCS Unclear Training images NR/12	Selected images previously analysed by MSDSLA	1. VI (no algorithm) 2. Dermoscopy (no algorithm) Image‐based (clinical image)	NR				Dermatologists (n = 70) Exp NR; average	Histology MM 3; MiS 2 BN 7 5/12; 42%	NR
Zalaudek 2006 Referred (selected on reference) (u) Any	NC R‐CS Specialist clinic Italy NR/165	Random sample of excised, equivocal and nonequivocal, PSL and non‐PSLs with melanin or haemoglobin pigmentation in all or part of the lesion	Dermoscopy (3PCL) Image‐based (age, site, gender)	≥ 2 characteristics present				Mixed (n = 150; exp NR) Average result	Histology Full sample: MM 18; MiS 11 BCC: 18 79 BN; 26 SK; 8 vascular; 3 DF 26/150; 17%	15 used for training purposes 5 BCC moved from FP to TN
3PCL: three‐point checklist; 4PCL: four‐point checklist; 7FFM: seven features for melanoma; 7PCL: seven‐point checklist; ABCD(E): asymmetry, border, colour, differential structures (enlargement); AJCC: American Joint Committee on Cancer; AK: actinic keratosis; AMN: acral melanocytic naevi; BCC: basal cell carcinoma; BD: Bowen’s disease; Bham: Birmingham; BN: benign naevi; BPC: between person comparison (of tests); c: clearly positioned on clinical pathway; CCD: compact disc; CAD: computer‐assisted diagnosis; CASH: colour, architecture, symmetry and homogeneity; CCS: case‐control study; CM: cutaneous melanoma; CMM: cutaneous malignant melanoma;CS: case series; cSCC: cutaneous squamous cell carcinoma; DF: dermatofibroma; ELM: epiluminescence microscopy; exp: experience; FP: false positive; FU: follow‐up; GP: general practitioner; IDS: International Dermoscopy Society; IQR: interquartile range; LK: lichen sclerosis; LM: lentigo maligna; LP: lichen planus; LS: lentigo simplex; MEL: invasive melanoma or melanoma in situ; MM: malignant (invasive) melanoma; MiS: melanoma in situ; MN: melanocytic naevi; MSDSLA: multispectral digital skin lesion analysis device; N/A: not applicable; NC: non‐comparative; NML: non melanocytic lesion; NR: not reported; P: prospective; PLC: pigmented lesion clinic; PSL: pigmented skin lesion; R: retrospective; RCM: reflectance confocal microscopy; RCT: randomised controlled trial; SCC: squamous cell carcinoma; SD: standard deviation; SK: seborrhoeic keratosis; SN: Spitz naevi; SSM: superficial spreading melanoma; TN: true negative; u: unclear position on clinical pathway; VI: visual inspection; WPC: within person comparison (of tests)

Appendix 10. Forest plots of sensitivity and specificity for visual inspection for the detection of invasive melanoma and atypical intraepidermal melanocytic variants (invasive melanoma or atypical intraepidermal melanocytic variants)

Figure 139

25.

Forest plot of tests. 5 Visual inspection ‐ in‐person (invasive melanoma or atypical intraepidermal melanocytic variants), 7 visual inspection ‐ image‐based (invasive melanoma or atypical intraepidermal melanocytic variants)

Appendix 11. Summary study details for detection of invasive melanoma

Study author Other target conditions also assessed	Study type Country Setting Participants/lesions	Inclusion criteria	Index tests (algorithm) Diagnostic approach	Threshold		Observer qualifications (number) Experience	Reference standard Final diagnoses Prevalence (invasive melanoma)	Exclusions Comments
In‐person evaluations
Ascierto 2010	WPC P‐CS Specialist clinic Italy 54/54	Clinically relevant cutaneous pigmented lesions, undergoing dermoscopy and excision	Dermoscopy (risk stratification; modified Kenet 2001) In‐person (Also evaluates CAD Spectroscopy)	Very high risk; high or very high risk; correct diagnosis of MM		Dermatologist (n = NR; exp High) Unclear observer interpretation	Histology MM 12 'Benign' 42 12/42; 22%	‐
Coras 2003	WPC NR‐CS Private Germany NR/45	PSLs undergoing excision due to diagnosis of melanoma or atypical nevus, to rule out melanoma or at the patient's request	Dermoscopy (no details; diagnosis based on clinical exam, dermoscopy, medical history ) In‐person (Also evaluates teledermatology assessment of clinical/dermoscopic images)	NR; Correct diagnosis of MM		Dermatologist (n = 3; exp High) participating experts with great experience in dermatoscopy Single observer	Histology MM 16; 'Benign': 29 16/45; 36%	10 excluded due to poor image quality; 45 did not undergo excision
Feldmann 1998 Invasive melanoma or atypical intraepidermal melanocytic variants	NC P‐CS Secondary Austria NR/500	Melanocytic lesions examined by dermatoscopy prior to excision	Dermoscopy (ABCD) In‐person	> 5.45; > 4.2		NR (n = NR; exp NR) Unclear observer interpretation	Histology MM 25; MiS 5 BN 272; dysplasia 190; lentigines 7; lentigo naevi 1 30/500; 6%	NR
Krahn 1998	WPC P‐CS Secondary Germany 80/80	Excised PSLs	1. VI (no algorithm) 2. Dermoscopy (no algorithm) In‐person	NR; clinical diagnosis of MM		Dermatologist (assumed) (n = 1; exp NR) Single observer	Histology MM 39 BN 37; dysplastic 2; SN 1 39/80; 49%	None
Piccolo 2000	NC NR‐CS Multicentre Austria 40/43	PSLs selected because of their diagnostic difficulty	Dermoscopy (no algorithm) In‐person observer (Also evaluates teledermatology assessment of clin/dermoscopic images)	NR; correct diagnosis of MM		Dermatologists (n = 1; exp High) Single observer	Histology MM 11; BCC 3 SK 2; BN 23; other 4 11/43; 26%	NR; poor‐quality index test image. The digital images were assigned an image‐quality rating (1, excellent; 2, good; 3, sufficient; 4, poor). All images scoring 4 were excluded from the study
Viglizzo 2004	WPC NR‐CS Specialist clinic Italy NR/79	PSLs examined at the Dermoscopy Service and undergoing excisions; high and medium risk on dermoscopy were selected for excision and 2x2 can be estimated only for melanocytic subgroup	1. VI (no algorithm) 2. Dermoscopy (no algorithm) In‐person	NR; correct diagnosis of MM		Dermatologist (assumed) (n = NR; exp NR) Single observer	Histology MM 12 MN: 67 12/67; 18%	None
Image‐based
Arevalo 2008	NC RP‐CS Specialist clinic Australia NR/3367	Melanocytic lesions imaged at the Sydney Melanoma Unit with a histopathologic diagnosis or that remained unchanged following short‐term (2.5‐4.5 months) digital monitoring (diagnosed as benign)	Dermoscopy (Menzies criteria) Image‐based (blinded)		Absence of negative characteristics + ≥ 1 positive characteristic present; correct diagnosis of MM	Dermatologist (assumed) (n = 2; exp NR) Consensus of 2; referral to a 3rd observer if disagreement	Histology or FU MM 341 'Benign' 3026 341/3367; 10%	None
Friedman 2008 Invasive melanoma or atypical intraepidermal melanocytic variants	WPC CCS Secondary/private US 94/99	An industry database of images of PSL ≤ 6 mm was used to sample images of melanoma and non‐melanoma lesions; high‐grade dysplastic naevi were excluded	Dermoscopy (no algorithm) Image‐based (site, age, gender) (Also evaluates CAD‐Spectroscopy)		Correct diagnosis; excise decision	Mixed ‐ secondary (n = 10; exp High) Average result (reports mean and median; mean used)	Histology MM 21; MiS 28; BCC: BN 34; SK 2; 14 other benign 21/99; 21%	NR
Hauschild 2014 Invasive melanoma; Invasive melanoma or atypical intraepidermal melanocytic variants	WPC CCS Secondary/private US 130/130	Subset of PSL evaluated in a MelaFind study (Monheit 2011); 65 melanoma and 65 non‐melanoma randomly selected. Excluded ulcerated, non‐pigmented, or located on excluded anatomic sites.	Dermoscopy (no algorithm) (Also evaluates CAD spectroscopy) Image‐based (clinical image, pt history)		NR; excise decision	Dermatologist (n = 101; exp High) Single observer	Histology MM 36; MiS 29 'Benign' diagnoses: 65 36/130; 28%	‐
Kreusch 1992	NC RP‐CS Secondary Germany Full sample: 858/1506 (265 melanocytic included)	Pigmented lesions suspected to be malignant melanoma with adequate photo‐documentation and histology results	Dermoscopy (from Kreusch 1991) Image‐based (slides labelled only with patient code and lesion localisation)		≥ 9; correct diagnosis of MM	Dermatologist (assumed) (n = 1; ‘experienced') (Also presents results for inexperienced student – data not included) Single observer	Histology MM 96; BN 169	52 NML excluded from second‐step evaluation
Lorentzen 1999a	WPC P‐CS Specialist clinic Denmark 232/232	Patients with lesions suspicious for CMM referred to outpatients clinic	1. VI (no algorithm) 2. Dermoscopy (no algorithm) Image‐based (clinical image)		Subjective impression; correct diagnosis of MM	Dermatologist (n = 4; exp High) Average	Histology MM 49; BCC 16 SK 12; BN 137 other: 18 (SN, BD + others) 49/232; 21%	Poor‐quality index test image 10 cases excluded
Lorentzen 2000	WPC‐alg RP‐CS Specialist clinic Denmark 258/258	PSL from patients consecutively referred to the skin cancer outpatient clinic with available clinical photographs, dermatophotographs and a subsequent excision biopsy were included	Dermoscopy (ABCD; Kenet risk stratification) Image‐based (clinical image)		> 4.75; Kenet – probable melanoma; possible/probable melanoma	Dermatologist (n = 3; exp High; 3 senior dermatologists with > 5 years' daily experience in dermatoscopy Single observer (reported per observer)	Histology MM 64; BCC 25 SK 14; BN 135; dysplastic 3; other: 16 64/258; 25%	‐
Lorentzen 2008	WPC NR‐CS Specialist clinic Denmark 119/119	Patients referred to the specialist naevus clinic; compared classic dermoscopy to acrylic globe magnifier	Dermoscopy (Kenet risk stratification) Image‐based (blinded)		NR	Dermatologist (n = NR) Average	Histology MM 24; BCC 13 BN 69; mild/moderate dysplasia 2; SK 9; other 2 24/119; 20%	1 DF
Menzies 1996	NC RP‐Unclear Image libraries Multicentre NR/385	PSL from the Sydney Melanoma Unit with dermoscopic images and histological diagnoses; melanomas and randomly selected clinically atypical non‐melanoma lesions were included	Dermoscopy (Menzies criteria) Image‐based (blinded)		2 characteristics absent and ≥ 1 characteristic present; correct diagnosis of MM	Dermatologist (assumed) (n = NR; exp NR) NR	Histology MM 107; BCC: 18 SK 23; acquired BN 58; dysplastic 105; blue naevi 11; ephelis/lentigo 17; SN 6; spindle cell nevus 2; DF 2; hemangioma 13; solar keratosis 9; other 14 107/385; 28%	‐
Menzies 2013	WPC‐algorithms CCS Secondary Mixed NR/467	Nodular malignant melanoma* and a random selection of non‐nodular invasive primary melanoma, benign nodular melanocytic lesions, and nodular nonmelanocytic lesions at a ratio of nodular melanoma to other subgroups of 1:2. Nodular benign melanocytic lesions and nodular nonmelanocytic lesions were identified by the clinical appearance of a solitary nodule and confirmed using dermoscopic examination.	Dermoscopy (ABCD; Menzies, CASH; 7PCL; 3PCL; Menzies (amelanotic)) Image‐based (NR)		ABCD > 5.45; CASH > 8; Menzies amelanotic > 0 and > 1; others at standard thresholds	Dermatologist (n = 1; exp NR). 12 scorers blinded to the lesion diagnosis scored 99 individual features in each lesion of approximately equal sample sizes, as previously described. Following the review of the article for publication, an additional feature (blue‐black structures) was scored for all lesions by one observer (E.C.). Single observer	Histology or FU ("some" benign melanocytic naevi showed no change over time compared with baseline photographs). NM 83; 134 MM BN 115; 217/332; 65%	135 NML excluded from second step evaluation *an invasive melanoma without an in situ (junctional) component beyond 3 rete ridges of the dermal invasive component
Nilles 1994	NC RP‐CS Secondary Germany NR/209	Melanocytic skin lesions that underwent excision	Dermoscopy (new algorithm) Image‐based (blinded)		Any characteristic present?; correct diagnosis of MM	Dermatologist (assumed) (n = 1; exp NR) S ingle observer	Histology MM 41 BN168 41/209; 20%	260 lesions used to identify best model; accuracy for overall diagnosis reported for 209 lesions investigated in year 1990
Rao 1997	WPC‐Obs RP‐CS Specialist clinic US 63/72	Patients with atypical melanocytic lesions or suspected early malignant melanoma	1. VI (no algorithm) 2. Dermoscopy (no algorithm) Image‐based (clinical image)		Subjective impression; diagnosis	Dermatologist (n = 2); melanoma fellow (n = 2) Single observer	Histology MM 21 Atypical MN 51 21/72; 29%	None
Troyanova 2003	WPC‐tests CCS Specialist clinic NR NR/50	Images of PSLs selected for a dermoscopy training study	1. VI (no algorithm) 2. Dermoscopy (no algorithm) Image‐based (blinded)		NR; correct diagnosis of MM	Dermatologist (n = 32; exp High) Average	Histology MM 25 'Benign' 50 25/50; 50%	NR
Westerhoff 2000	WPC‐Obs CCS Specialist clinic Australia NR/100	Clinically atypical PSLs randomly selected from PSL image database	1. VI (no algorithm) 2. Dermoscopy (no algorithm; Menzies criteria) Image‐based (blinded)		NR; diagnosis of MM	GPs (n = 74; no formal training in dermoscopy, randomised to dermoscopy education intervention (n = 37) or not (n = 37) Average reported	Histology or FU MM 50 'Benign' 50 50/100; 50%	*Diagnoses recorded for both groups of GPs at baseline (pre‐test) and after training of one arm (post‐test); post‐test data for the intervention group of GPs was used for the Visual Inspection analysis
AK: actinic keratosis; alg: algorithm; BD: Bowen’s disease; BCC: basal cell carcinoma; BN: benign naevi; BPC: between person comparison (of tests);CAD: computer‐assisted diagnosis; CCS: case‐control study; CS: case series; CMM: cutaneous malignant melanoma;cSCC: cutaneous squamous cell carcinoma; DF: dermatofibroma; FU: follow‐up; LS: lentigo simplex; MiS: melanoma in situ (or lentigo maligna); MM: malignant melanoma; NC: non‐comparative; NR: not reported; Obs: observer; P: prospective; PLC: pigmented lesion clinic; PSL: pigmented skin lesion; R: retrospective; RCM: reflectance confocal microscopy; SK: seborrhoeic keratosis; SN: Spitz naevi; WPC: within person comparison (of tests)

Appendix 12. Forest plots of sensitivity and specificity for visual inspection and for visual inspection plus dermoscopy for the detection of invasive melanoma

Figure 140; Figure 141

26.

Forest plot of tests. 1 Visual inspection ‐ in‐person (invasive melanoma), 2 VI+dermoscopy ‐ in‐person (invasive melanoma)

27.

Forest plot of tests. 3 Visual inspection ‐ image‐based (invasive melanoma), 4 dermoscopy alone ‐ image‐based (invasive melanoma)

Appendix 13. Summary study details for detection of any skin lesion requiring excision

Study author Other target conditions reported	Study type Country Setting	Inclusion criteria	Index tests (algorithm) Diagnostic approach	Threshold	Observer qualifications (number) Experience	Reference standard Final diagnoses Prevalence (any skin cancer)	Exclusions Comments (marked *)
In‐person evaluations
Argenziano 2006	RCT Italy, Spain Primary NR/85	Patients asking for screening or exhibiting ≥ 1 skin tumours as seen during routine physical examination (patient‐finding screening). Participating PCPs randomised to either visual inspection alone or visual inspection + dermoscopy; only excised lesions can be included for each arm.	VI (ABCD) Dermoscopy (3‐point checklist) In‐person (single observer)	Subjective impression; diagnosis of malignancy	GPs (n = 37) All trained in ABCD rule	Histology MEL 6 BCC 37; SCC 10 Benign 32 53/85; 62%	*Only those patients who were considered to have lesions suggestive of skin cancer had histology and could be included; rest had expert diagnosis (making full dataset ineligible for this review)
Durdu 2011 Invasive melanoma or atypical intraepidermal melanocytic variants	WPC P‐CS Secondary Turkey 176/200	PSL that could not be diagnosed with only dermatologic physical examination; 2x2 included for melanocytic subset	Dermoscopy (ABCD; nonmelanocytic excluded first) (Also evaluated exfoliative cytology) In‐person	NR	Dermatologist (n = 1; exp NR) Single observer	Histology MEL: 10; BCC: 34; other malignant 2 SK 24; BN 100; DF 12; warts 16; dirt 1; other 1 10/200; 5%	‐
Soyer 2004	NC R‐CS Specialist unit Italy 225/231	Lesions at pigmented lesion clinic considered by experienced dermatologists to merit excision on clinical grounds	Dermoscopy (no algorithm) In‐person	NR	Dermatologist (n = 1; exp High)* Single	Histology MEL: 68; BCC 9 'Benign' 154 77/154; 33%	*Also reports data for 6 inexperienced observers interpretation of the acquired dermoscopic images; data excluded as includes 3 medical students
Stanganelli 2000 Invasive melanoma or atypical intraepidermal melanocytic variants	WPC R‐CS Italy Specialist clinic NR/3372	PSL referred by dermatologists and general practitioners either for pre‐surgical assessment or consultation	VI (ABCD) Dermoscopy (no algorithm) In person (single)	NR Subjective impression	NR (assumed dermatologist ‐ described as one of the co‐authors; n = 1)	Histology/registry FU MEL 55 BCC 43; benign 3274 98/3372; 3%	NR
Image‐based evaluations
Carli 2002b Invasive melanoma or atypical intraepidermal melanocytic variants Any	WPC R‐CS Italy Secondary NR/57	Clinically suspicious or equivocal PSL undergoing excision for diagnostic purposes; all ≤ 14 mm diameter	1. VI (NR) 2. Dermoscopy (NR) Image‐based (blinded)	NR	Dermatologists (n = 2) High exp ('with experience in the field of '); consensus of 2	Histology MM 6, MiS 5 BCC 10 BN 31, SK 1; other 4 11/57; 19%	4 ‘not evaluables’ excluded (NB these differ between clinical images and dermoscopic images (1 MM excluded from VI analysis)
Lorentzen 2008 Invasive melanoma	WPC NR‐CS Specialist clinic Denmark 119/119	Patients referred to the specialist naevus clinic; compared classic dermoscopy to acrylic globe magnifier	Dermoscopy (Kenet risk stratification) Image‐based (blinded)	NR	Dermatologist (n = NR) Average	Histology MM 24; BCC 13 BN 69; mild/moderate dysplasia 2; SK 9; other 2 24/119; 20%	1 DF
Rosendahl 2011 Invasive melanoma or atypical intraepidermal melanocytic variants	WPC‐algorithms R‐CS Australia Primary 389/463	PSL submitted for histology from the primary care skin cancer practice of 1 author	1. VI (no algorithm) 2. Dermoscopy (pattern; Chaos and Clues)	1. Subjective impression 2. NR; both characteristics present	Dermatologist (n = 1) High exp (confirmed by author); single obs	Histology MM 9; MiS 20 BCC 72; SCC 5 BN 217; BD 18; AK 14*; BNM 140 *considered malignant by study authors 29/463; 6%	3 poor‐quality images excluded
Stanganelli 1998a Invasive melanoma or atypical intraepidermal melanocytic variants Any	WPC R‐CS Italy Training images Italy NR/30	PSL images selected from computerised files of the skin cancer clinic	1. VI (no algorithm) 2. Dermoscopy (no algorithm) Image‐based (clinical image)	NR; clinical diagnosis	Dermatologists (n = 20) Exp NR (“experience in ELM but (with) no formal training”) Average result	Histology MEL 10 BCC 4 BN 10, SK 3, other 3 10/30; 33%	NR BCC results not disaggregated
Zalaudek 2006 Invasive melanoma or atypical intraepidermal melanocytic variants	NC R‐CS Specialist clinic Italy NR/165	Random sample of excised, equivocal and nonequivocal, PSL and non‐PSLs with melanin or haemoglobin pigmentation in all or part of the lesion	Dermoscopy (3PCL) Image‐based (age, site, gender)	≥ 2 characteristics present	Mixed (n = 150; exp NR) Average result	Histology Full sample: MM 18; MiS 11 BCC: 18 79 BN; 26 SK; 8 vascular; 3 DF 26/150; 17%	15 used for training purposes 5 BCC moved from FP to TN
3PCL: three‐point checklist; AK: actinic keratosis; alg: algorithm; BD: Bowen’s disease; BCC: basal cell carcinoma; BN: benign naevi; BPC: between person comparison (of tests);CAD: computer‐assisted diagnosis; CCS: case‐control study; CS: case series; cSCC: cutaneous squamous cell carcinoma; DF: dermatofibroma; ELM: epiluminescence microscopy; exp: experience; FP: false positive; FU: follow‐up; LS: lentigo simplex; MiS: melanoma in situ (or lentigo maligna); MM: malignant melanoma; NC: non‐comparative; NR: not reported; Obs: observer; P: prospective; PCP: primary care provider; PLC: pigmented lesion clinic; PSL: pigmented skin lesion; R: retrospective; RCM: reflectance confocal microscopy; SK: seborrhoeic keratosis; SN: Spitz naevi; TN: true negative; VI: visual inspection; WPC: within person comparison (of tests)

Appendix 14. Dermoscopy training interventions

Study author Outcomes reported	Inclusion criteria Numberof lesions; cases Algorithm used In‐person/image‐based	Clinicians recruited for training	Pre‐training	Training approach	Post‐training
Detection of Invasive melanoma or atypical intraepidermal melanocytic variants
Pagnanelli 2003 Pathway – unclear	Clinical and dermoscopic images of PSL from the training set of the Consensus Net Meeting on Dermoscopy (CNMD), selected by 2 experts N = 20; MEL 6 Dermoscopy (pattern analysis; Menzies; 7PCL; ABCD) Image‐based (clinical image)	Recruited 16 ‘colleagues’, including medical Students (n = 3), dermatology residents (n = 9) and dermatologists (n = 4) All reported limited personal experience of dermoscopy, no formal training and did not use dermoscopy in daily professional practice	After the 1‐hour lecture at the beginning of the study, lesion images were provided on CD‐Rom; participants asked to complete electronic data sheet listing criteria for diagnosing PSLs by pattern analysis and by the various algorithms and to offer a dermoscopic diagnosis for each case within 20 days	1‐hour lecture on basic principles of dermoscopy, the dermoscopic features of PSLs, pattern analysis and the diagnostic algorithms (ABCD rule, seven‐point checklist, Menzies’ method). + a web‐based tutorial (http://www.dermoscopy.org); participants requested to devote 1 hour per day, 5 days per week for 2 consecutive weeks.	Post‐training evaluation 5 weeks after initial evaluation Participants re‐evaluated the same 20 cases, again over a 20‐day period.
Piccolo 2014 Pathway – unclear	Dermoscopically atypical PSL N = 165; MM 23; MiS 10 Dermoscopy (ABCD) Image‐based (blinded) (Also evaluates CAD dermoscopy)	3 dermatologists and 1 GP scored according to number of years specializing in dermoscopy, number of PSLs assessed by dermoscopy on a daily basis, number of relevant workshops/seminars attended, and number of authored publications on dermoscopy: highly experienced (observer 1), moderately experienced, (observers 2 and 3); and minimally experienced (observer 4).	Digital dermoscopic images assessed by each observer using ABCD at T0	Between T0 and T1, Observer 4 underwent dermoscopic training by studying an interactive atlas of dermoscopy (Argenziano 2003; appears to be same as for Pagnanelli 2003)	The same digital dermoscopic images were assessed by each of the 4 observers using ABCD after 6 months (T1)
Stanganelli 1999 Pathway – unclear	PSL images selected from database for training study N = 30; MM 10; MiS 1 Dermoscopy (no algorithm) Image‐based (clinical image)	Of 223 dermatologists who participated in one of the six workshops, 83 (37%) were reported on; average of 10 years of general experience in dermatology (range 1‐22) with routine use of ELM by 52 individuals (conventional dermatoscope for 43 and digital equipment for 9)	Pre‐training test conducted after the opening lecture of each workshop (clinical classification of PSLs). Images projected onto a screen in pairs (clinical and ELM image); classified by as CMM, MN, NML, unclassifiable or equivocal; approximately 2.5 min per lesion	Nationwide educational programme in ELM; one‐day meetings and workshops (duration: 6 hours) held with free registration. Topics included: clinical classification and diagnosis of PSLs management of patients with PSLs; basic principles of ELM; ELM criteria ELM diagnosis; limitations of ELM	Same set of slides re‐evaluated at the end of the workshop. Slides and respective correct diagnosis were discussed only after the second test.
Tan 2009 Pathway – unclear	Test series of images of melanomas and benign lesions N = 30; MEL 15 Dermoscopy (pattern analysis modified) Image‐based (clinical image)	3 consultant dermatologists and 3 specialist registrars; none had routinely used a dermatoscope	Assessed 30 test cards consisting of 1 macroscopic and 1 dermatoscopic image of each lesion; printed on A4 laminated paper Participants classified images as ‘benign’, ‘malignant’ or ‘not known’, gave a diagnosis if known, and indicated whether they would excise the lesion.	Participants received an online tutorial (www.dermatoscopy.org) teaching the MPADA (Modified Pattern Analysis Diagnostic Algorithm), which could be referred to during the study period. Also each given a dermatoscope to use in clinical practice for 10 months.	10 months later, the test‐card questionnaire was repeated (test 2)
Detection of invasive melanoma alone
Troyanova 2003	Patients with atypical melanocytic lesions or suspected early malignant melanoma N = 50 1. VI (no algorithm) 2. Dermoscopy (no algorithm) Image‐based (clinical image)	Volunteer dermatologists (n = 32); experienced in clinical diagnosis of PSLs, but had no formal training in dermoscopy. ELM qualification based on good theoretical knowledge of the literature and on personal experience by trial and error.	50 clinical images displayed individually using slide projector; scored as melanoma or "not‐melanoma". 50 dermoscopy slides then presented and ELM diagnoses recorded. Each image shown for 30 seconds; no discussion of assumed diagnosis was permitted. None of the test slides used for training	6 hours of teaching daily for 2 consecutive days. Training was based on presentation of several hundred slides with oral explanation of the ELM criteria.	Tests were performed in the beginning and in the end of the teaching course Same test performed with slides of 50 different PSLs
Westerhoff 2000	Images of PSL selected for a dermoscopy training study N = 50; 50 MM 1. VI (no algorithm) 2. Dermoscopy (no algorithm; Menzies criteria) Image‐based (blinded)	GPs (n = 74) recruited by telephone from a list of current practitioners. Required to have no formal training in dermoscopy and did not use dermoscopy in their clinical practice. Participants randomised into an education intervention group or non‐education intervention group (each n = 37)	Lesions presented with the clinical photograph first, followed by dermoscopy image Participants given 4 options: melanoma; benign melanocytic lesion, benign non‐melanocytic lesion, 'other' (specify) Clinical diagnosis recorded prior to observation of dermoscopic image. Tests completed at participants' leisure	Supplied with pictorial atlas by Menzies 1996b and a 1‐hour presentation on dermoscopy that specifically reviewed the Menzies method and included a quiz with images of 25 different PSL (not used in test)	As for pre‐test
ABCD: asymmetry, border, colour, dimensions;AK: actinic keratosis; alg: algorithm; BD: Bowen’s disease; BCC: basal cell carcinoma; BN: benign naevi; BPC: between person comparison (of tests);CAD: computer‐assisted diagnosis; CCS: case‐control study; CMM: cutaneous malignant melanoma; CS: case series; cSCC: cutaneous squamous cell carcinoma; DF: dermatofibroma; ELM: epiluminescence microscopy; FU: follow‐up; LS: lentigo simplex; MEL: melanoma; MiS: melanoma in situ (or lentigo maligna); MM: malignant melanoma; NC: non‐comparative; NML: non melanocytic lesion; NR: not reported; Obs: observer; P: prospective; PLC: pigmented lesion clinic; PSL: pigmented skin lesion; R: retrospective; RCM: reflectance confocal microscopy; SK: seborrhoeic keratosis; SN: Spitz naevi; WPC: within person comparison (of tests)

Data

Presented below are all the data for all of the tests entered into the review.

Tests. Data tables by test.

Test	No. of studies	No. of participants
1 Visual inspection ‐ in‐person (invasive melanoma)	2	147
2 VI+dermoscopy ‐ in‐person (invasive melanoma)	6	789
3 Visual inspection ‐ image‐based (invasive melanoma)	4	454
4 Dermoscopy alone ‐ image‐based (invasive melanoma)	13	5618
5 Visual inspection ‐ in‐person (invasive melanoma or atypical intraepidermal melanocytic variants)	13	6740
6 VI+dermoscopy ‐ in‐person (invasive melanoma or atypical intraepidermal melanocytic variants)	26	23169
7 Visual inspection ‐ image‐based (invasive melanoma or atypical intraepidermal melanocytic variants)	11	1740
8 Dermoscopy alone ‐ image‐based (invasive melanoma or atypical intraepidermal melanocytic variants)	60	13475
9 Visual inspection ‐ in‐person (any)	2	3457
10 VI+Dermoscopy ‐ in‐person (Any)	4	3880
11 Visual inspection ‐ image‐based (any)	3	547
12 Dermoscopy alone ‐ image‐based (any)	5	815
13 MM2‐ Dermoscopy ‐ no algorithm ‐ threshold NR ‐ in‐person	8	4707
14 MM2‐ Dermoscopy ‐ pattern ‐ at ≥ 1 char present ‐ in‐person	1	220
15 MM2‐ Dermoscopy ‐ pattern ‐ at ≥ 3 characteristics present ‐ in‐person	1	68
16 MM2‐ Dermoscopy ‐ pattern ‐ threshold NR ‐ in‐person	6	4307
17 MM2‐ Dermoscopy ‐ ABCD at NR (likely > 5.45) ‐ in‐person	1	235
18 MM2‐ Dermoscopy ‐ ABCD at > 5.45 ‐ in‐person	4	1203
19 MM2‐ Dermoscopy ‐ ABCD at > 4.75 ‐ in‐person	1	309
20 MM2‐ Dermoscopy ‐ ABCD at 60% specificity ‐ in‐person	1	356
21 MM2‐ Dermoscopy ‐ ABCD at 80% specificity ‐ in‐person	1	356
22 MM2‐ Dermoscopy ‐ ABCD at 70% specificity ‐ in‐person	1	356
23 MM2‐ Dermoscopy ‐ ABCD at 75% specificity ‐ in‐person	1	356
24 MM2‐ Dermoscopy ‐ ABCD at 85% specificity ‐ in‐person	1	356
25 MM2‐ Dermoscopy ‐ ABCD at 90% specificity ‐ in‐person	1	356
26 MM2‐ Dermoscopy ‐ ABCDE at > 1.3 ‐ in‐person	1	356
27 MM2‐ Dermoscopy ‐ ABCDE at > 2.65 ‐ in‐person	1	356
28 MM2‐ Dermoscopy ‐ ABCDE at > 3.05 ‐ in‐person	1	356
29 MM2‐ Dermoscopy ‐ ABCDE at > 3.6 ‐ in‐person	1	356
30 MM2‐ Dermoscopy ‐ ABCDE at > 4.25 ‐ in‐person	1	356
31 MM2‐ Dermoscopy ‐ ABCDE at > 4.9 ‐ in‐person	1	356
32 MM2‐ Dermoscopy ‐ 7FFM at ≥ 2 ‐ in‐person	1	401
33 MM2‐ Dermoscopy ‐ 7‐point at ≥ 2 ‐ in‐person	1	638
34 MM2‐ Dermoscopy ‐ 7‐point at ≥ 3 ‐ in‐person	2	11137
35 MM2‐ Dermoscopy ‐ Menzies at 2 negative and ≥ 1 positive ‐ in‐person	1	206
36 MM2‐ Dermoscopy ‐ no algorithm ‐ any threshold ‐ image‐based	24	4498
37 MM2‐ Dermoscopy ‐ no algorithm ‐ correct diagnosis ‐ image‐based	18	4118
38 MM2‐ Dermoscopy ‐ no algorithm ‐ excise decision ‐ image‐based	10	831
39 MM2‐ Dermoscopy ‐ pattern ‐ any threshold ‐ image‐based	20	4621
40 MM2‐ Dermoscopy ‐ pattern ‐ correct diagnosis ‐ image‐based	19	4095
41 MM2‐ Dermoscopy ‐ pattern ‐ excise decision ‐ image‐based	3	933
42 MM2‐ Dermoscopy ‐ ABCD at > 4.75 ‐ image‐based	10	4242
43 MM2‐ Dermoscopy ‐ ABCD at > 5.45 ‐ image‐based	7	2471
44 MM2‐ Dermoscopy ‐ revised ABCD at ≥ 4 ‐ image‐based	1	269
45 MM2‐ Dermoscopy ‐ ABCDE at ≥ 4 ‐ image‐based	1	269
46 MM2‐ Dermoscopy ‐ 7‐point at NR ‐ image‐based	4	1936
47 MM2‐ Dermoscopy ‐ 7‐point at ≥ 3 ‐ image‐based	11	3408
48 MM2‐ Dermoscopy ‐ 7‐point at ≥ 5 ‐ image‐based	1	322
49 MM2‐ Dermoscopy ‐ revised 7‐point at NR (likely ≥ 1) ‐ image‐based	1	1678
50 MM2‐ Dermoscopy – revised 7‐point at ≥ 1 ‐ image‐based	1	300
51 MM2‐ Dermoscopy ‐ revised 7‐point for FU ‐ major change ‐ image‐based	1	70
52 MM2‐ Dermoscopy ‐ 7FFM at ≥ 2 ‐ image‐based	4	2200
53 MM2‐ Dermoscopy ‐ Menzies at 2 negative and ≥ 1 positive ‐ image‐based	4	1856
54 MM2‐ Dermoscopy ‐ Menzies at NR ‐ image‐based	2	60
55 MM2‐ Dermoscopy ‐ 3‐point at ≥ 2 ‐ image‐based	7	1505
56 MM2‐ Dermoscopy ‐ 4‐point (scored 3‐point) at > 2 ‐ image‐based	1	75
57 MM2‐ Dermoscopy ‐ Hofman algorithm at NR ‐ image‐based	1	254
58 MM2‐ Dermoscopy CASH at ≥ 6 ‐ image‐based	1	477
59 MM2‐ Dermoscopy CASH at ≥ 8 ‐ image‐based	2	190
60 MM2‐ Dermoscopy Chaos/Clues at = 2 ‐ image‐based	2	940
61 MM2‐ Dermoscopy ‐ Acral 3‐step ‐ image‐based	1	107
62 VI+dermoscopy (in‐person) ‐ observer experience NR (invasive melanoma or atypical intraepidermal melanocytic variants)	10	8390
63 VI+dermoscopy (in‐person) ‐ high experience (invasive melanoma or atypical intraepidermal melanocytic variants)	14	14213
65 VI+dermoscopy (in‐person) ‐ trained observer (invasive melanoma or atypical intraepidermal melanocytic variants)	2	566
66 Dermoscopy (image‐based) ‐ observer experience NR (invasive melanoma or atypical intraepidermal melanocytic variants)	11	2777
67 Dermoscopy (image‐based) ‐ high experience (invasive melanoma or atypical intraepidermal melanocytic variants)	34	8933
68 Dermoscopy (image‐based) ‐ moderate experience (invasive melanoma or atypical intraepidermal melanocytic variants)	5	678
69 Dermoscopy (image‐based) ‐ low experience (invasive melanoma or atypical intraepidermal melanocytic variants)	6	448
70 Dermoscopy (image‐based) ‐ mixed experience (invasive melanoma or atypical intraepidermal melanocytic variants)	5	473
71 Dermoscopy (image‐based) ‐ trained observer (invasive melanoma or atypical intraepidermal melanocytic variants)	11	1087
72 VI+dermoscopy (in‐person) ‐ consultant expert (invasive melanoma or atypical intraepidermal melanocytic variants)	11	2767
73 VI+dermoscopy (in‐person) ‐ consultant (invasive melanoma or atypical intraepidermal melanocytic variants)	10	8390
74 VI+dermoscopy (in‐person) ‐ resident/registrar (invasive melanoma or atypical intraepidermal melanocytic variants)	2	11137
75 VI+dermoscopy (in‐person) ‐ mixed (secondary care‐based) (invasive melanoma or atypical intraepidermal melanocytic variants)	1	309
76 VI+dermoscopy (in‐person) ‐ GP (invasive melanoma or atypical intraepidermal melanocytic variants)	2	566
77 Dermoscopy (image‐based) ‐ consultant expert (invasive melanoma or atypical intraepidermal melanocytic variants)	33	8664
78 Dermoscopy (image‐based) ‐ consultant (invasive melanoma or atypical intraepidermal melanocytic variants)	24	3986
79 Dermoscopy (image‐based) ‐ resident (invasive melanoma or atypical intraepidermal melanocytic variants)	5	927
80 Dermoscopy (image‐based) ‐ mixed (secondary care based) (invasive melanoma or atypical intraepidermal melanocytic variants)	4	399
81 Dermoscopy (image‐based) ‐ mixed (other) (invasive melanoma or atypical intraepidermal melanocytic variants)	4	867
82 Dermoscopy (image‐based) ‐ GP/primary care (invasive melanoma or atypical intraepidermal melanocytic variants)	3	288
83 Dermoscopy (image‐based) ‐ physician assistant (invasive melanoma or atypical intraepidermal melanocytic variants)	1	65
84 Dermoscopy ‐ before training (invasive melanoma or atypical intraepidermal melanocytic variants)	4	245
85 Dermoscopy ‐ after training (invasive melanoma or atypical intraepidermal melanocytic variants)	4	245
86 Dermoscopy ‐ before training (invasive melanoma)	2	150
87 Dermoscopy ‐ after training (invasive melanoma)	2	150
88 MM1‐ Dermoscopy ‐ no algorithm ‐ threshold NR ‐ in‐person	3	190
89 MM1‐ Dermoscopy ‐ pattern analysis ‐ threshold NR ‐ in‐person	1	45
90 MM1‐ Dermoscopy ‐ ABCD at > 4.2 ‐ in‐person	1	495
91 MM1‐ Dermoscopy ‐ ABCD at > 5.45 ‐ in‐person	2	832
92 MM1‐ Dermoscopy ‐ Kenet (modified) at melanoma possible ‐ in‐person	1	54
93 MM1‐ Dermoscopy ‐ Kenet (modified) at melanoma likely ‐ in‐person	1	54
94 MM1‐ Dermoscopy ‐ no algorithm ‐ threshold NR ‐ image‐based	6	683
95 MM1‐ Dermoscopy ‐ no algorithm ‐ decision to excise ‐ image‐based (paired data only)	1	99
96 MM1‐ Dermoscopy ‐ pattern analysis ‐ threshold NR ‐ image‐based	1	119
97 MM1‐ Dermoscopy ‐ ABCD at > 4.75 ‐ image‐based	2	330
98 MM1‐ Dermoscopy ‐ ABCD at > 5.45 ‐ image‐based	1	258
99 MM1‐ Dermoscopy ‐ 7‐point at NR ‐ image‐based	1	332
100 MM1‐ Dermoscopy ‐ Menzies at 2 negative and ≥ 1 positive ‐ image‐based	4	4184
101 MM1‐ Dermoscopy ‐ 3‐point at > NR ‐ image‐based	1	332
102 MM1‐ Dermoscopy ‐ Kenet at melanoma likely ‐ image‐based	1	258
103 MM1‐ Dermoscopy ‐ Kenet at melanoma possible ‐ image‐based	1	258
104 MM1‐ Dermoscopy CASH at ≥ 8 ‐ image‐based	1	332
105 MM1‐ Dermoscopy ‐ Kreusch algorithm ‐ image‐based	1	265
106 MM1‐ Dermoscopy ‐ Menzies for amelanotic at 1 ‐ image‐based	1	332
107 MM1‐ Dermoscopy ‐ Menzies for amelanotic at 0 ‐ image‐based	1	332
108 MM3‐ Dermoscopy – no algorithm at NR ‐ in‐person	1	231
109 MM3‐ Dermoscopy ‐ pattern analysis ‐ threshold NR ‐ in‐person	1	3372
110 MM3‐ Dermoscopy – ABCD at > 5.45 ‐ in‐person	1	200
111 MM3‐ Dermoscopy – 3‐point at ≥ 2 ‐ in‐person	1	77
112 MM3‐ Dermoscopy – no algorithm at NR ‐ image‐based	2	83
113 MM3‐ Dermoscopy ‐ pattern analysis ‐ threshold NR ‐ image‐based	1	119
114 MM3‐ Dermoscopy – 3‐point at ≥ 2 ‐ image‐based	1	150
115 MM2 ‐ VI ‐ in‐person (with image‐based dermoscopy)	2	886

1. Test.

Visual inspection ‐ in‐person (invasive melanoma).

2. Test.

VI+dermoscopy ‐ in‐person (invasive melanoma).

3. Test.

Visual inspection ‐ image‐based (invasive melanoma).

4. Test.

Dermoscopy alone ‐ image‐based (invasive melanoma).

5. Test.

Visual inspection ‐ in‐person (invasive melanoma or atypical intraepidermal melanocytic variants).

6. Test.

VI+dermoscopy ‐ in‐person (invasive melanoma or atypical intraepidermal melanocytic variants).

7. Test.

Visual inspection ‐ image‐based (invasive melanoma or atypical intraepidermal melanocytic variants).

8. Test.

Dermoscopy alone ‐ image‐based (invasive melanoma or atypical intraepidermal melanocytic variants).

9. Test.

Visual inspection ‐ in‐person (any).

10. Test.

VI+Dermoscopy ‐ in‐person (Any).

11. Test.

Visual inspection ‐ image‐based (any).

12. Test.

Dermoscopy alone ‐ image‐based (any).

13. Test.

MM2‐ Dermoscopy ‐ no algorithm ‐ threshold NR ‐ in‐person.

14. Test.

MM2‐ Dermoscopy ‐ pattern ‐ at ≥ 1 char present ‐ in‐person.

15. Test.

MM2‐ Dermoscopy ‐ pattern ‐ at ≥ 3 characteristics present ‐ in‐person.

16. Test.

MM2‐ Dermoscopy ‐ pattern ‐ threshold NR ‐ in‐person.

17. Test.

MM2‐ Dermoscopy ‐ ABCD at NR (likely > 5.45) ‐ in‐person.

18. Test.

MM2‐ Dermoscopy ‐ ABCD at > 5.45 ‐ in‐person.

19. Test.

MM2‐ Dermoscopy ‐ ABCD at > 4.75 ‐ in‐person.

20. Test.

MM2‐ Dermoscopy ‐ ABCD at 60% specificity ‐ in‐person.

21. Test.

MM2‐ Dermoscopy ‐ ABCD at 80% specificity ‐ in‐person.

22. Test.

MM2‐ Dermoscopy ‐ ABCD at 70% specificity ‐ in‐person.

23. Test.

MM2‐ Dermoscopy ‐ ABCD at 75% specificity ‐ in‐person.

24. Test.

MM2‐ Dermoscopy ‐ ABCD at 85% specificity ‐ in‐person.

25. Test.

MM2‐ Dermoscopy ‐ ABCD at 90% specificity ‐ in‐person.

26. Test.

MM2‐ Dermoscopy ‐ ABCDE at > 1.3 ‐ in‐person.

27. Test.

MM2‐ Dermoscopy ‐ ABCDE at > 2.65 ‐ in‐person.

28. Test.

MM2‐ Dermoscopy ‐ ABCDE at > 3.05 ‐ in‐person.

29. Test.

MM2‐ Dermoscopy ‐ ABCDE at > 3.6 ‐ in‐person.

30. Test.

MM2‐ Dermoscopy ‐ ABCDE at > 4.25 ‐ in‐person.

31. Test.

MM2‐ Dermoscopy ‐ ABCDE at > 4.9 ‐ in‐person.

32. Test.

MM2‐ Dermoscopy ‐ 7FFM at ≥ 2 ‐ in‐person.

33. Test.

MM2‐ Dermoscopy ‐ 7‐point at ≥ 2 ‐ in‐person.

34. Test.

MM2‐ Dermoscopy ‐ 7‐point at ≥ 3 ‐ in‐person.

35. Test.

MM2‐ Dermoscopy ‐ Menzies at 2 negative and ≥ 1 positive ‐ in‐person.

36. Test.

MM2‐ Dermoscopy ‐ no algorithm ‐ any threshold ‐ image‐based.

37. Test.

MM2‐ Dermoscopy ‐ no algorithm ‐ correct diagnosis ‐ image‐based.

38. Test.

MM2‐ Dermoscopy ‐ no algorithm ‐ excise decision ‐ image‐based.

39. Test.

MM2‐ Dermoscopy ‐ pattern ‐ any threshold ‐ image‐based.

40. Test.

MM2‐ Dermoscopy ‐ pattern ‐ correct diagnosis ‐ image‐based.

41. Test.

MM2‐ Dermoscopy ‐ pattern ‐ excise decision ‐ image‐based.

42. Test.

MM2‐ Dermoscopy ‐ ABCD at > 4.75 ‐ image‐based.

43. Test.

MM2‐ Dermoscopy ‐ ABCD at > 5.45 ‐ image‐based.

44. Test.

MM2‐ Dermoscopy ‐ revised ABCD at ≥ 4 ‐ image‐based.

45. Test.

MM2‐ Dermoscopy ‐ ABCDE at ≥ 4 ‐ image‐based.

46. Test.

MM2‐ Dermoscopy ‐ 7‐point at NR ‐ image‐based.

47. Test.

MM2‐ Dermoscopy ‐ 7‐point at ≥ 3 ‐ image‐based.

48. Test.

MM2‐ Dermoscopy ‐ 7‐point at ≥ 5 ‐ image‐based.

49. Test.

MM2‐ Dermoscopy ‐ revised 7‐point at NR (likely ≥ 1) ‐ image‐based.

50. Test.

MM2‐ Dermoscopy – revised 7‐point at ≥ 1 ‐ image‐based.

51. Test.

MM2‐ Dermoscopy ‐ revised 7‐point for FU ‐ major change ‐ image‐based.

52. Test.

MM2‐ Dermoscopy ‐ 7FFM at ≥ 2 ‐ image‐based.

53. Test.

MM2‐ Dermoscopy ‐ Menzies at 2 negative and ≥ 1 positive ‐ image‐based.

54. Test.

MM2‐ Dermoscopy ‐ Menzies at NR ‐ image‐based.

55. Test.

MM2‐ Dermoscopy ‐ 3‐point at ≥ 2 ‐ image‐based.

56. Test.

MM2‐ Dermoscopy ‐ 4‐point (scored 3‐point) at > 2 ‐ image‐based.

57. Test.

MM2‐ Dermoscopy ‐ Hofman algorithm at NR ‐ image‐based.

58. Test.

MM2‐ Dermoscopy CASH at ≥ 6 ‐ image‐based.

59. Test.

MM2‐ Dermoscopy CASH at ≥ 8 ‐ image‐based.

60. Test.

MM2‐ Dermoscopy Chaos/Clues at = 2 ‐ image‐based.

61. Test.

MM2‐ Dermoscopy ‐ Acral 3‐step ‐ image‐based.

62. Test.

VI+dermoscopy (in‐person) ‐ observer experience NR (invasive melanoma or atypical intraepidermal melanocytic variants).

63. Test.

VI+dermoscopy (in‐person) ‐ high experience (invasive melanoma or atypical intraepidermal melanocytic variants).

65. Test.

VI+dermoscopy (in‐person) ‐ trained observer (invasive melanoma or atypical intraepidermal melanocytic variants).

66. Test.

Dermoscopy (image‐based) ‐ observer experience NR (invasive melanoma or atypical intraepidermal melanocytic variants).

67. Test.

Dermoscopy (image‐based) ‐ high experience (invasive melanoma or atypical intraepidermal melanocytic variants).

68. Test.

Dermoscopy (image‐based) ‐ moderate experience (invasive melanoma or atypical intraepidermal melanocytic variants).

69. Test.

Dermoscopy (image‐based) ‐ low experience (invasive melanoma or atypical intraepidermal melanocytic variants).

70. Test.

Dermoscopy (image‐based) ‐ mixed experience (invasive melanoma or atypical intraepidermal melanocytic variants).

71. Test.

Dermoscopy (image‐based) ‐ trained observer (invasive melanoma or atypical intraepidermal melanocytic variants).

72. Test.

VI+dermoscopy (in‐person) ‐ consultant expert (invasive melanoma or atypical intraepidermal melanocytic variants).

73. Test.

VI+dermoscopy (in‐person) ‐ consultant (invasive melanoma or atypical intraepidermal melanocytic variants).

74. Test.

VI+dermoscopy (in‐person) ‐ resident/registrar (invasive melanoma or atypical intraepidermal melanocytic variants).

75. Test.

VI+dermoscopy (in‐person) ‐ mixed (secondary care‐based) (invasive melanoma or atypical intraepidermal melanocytic variants).

76. Test.

VI+dermoscopy (in‐person) ‐ GP (invasive melanoma or atypical intraepidermal melanocytic variants).

77. Test.

Dermoscopy (image‐based) ‐ consultant expert (invasive melanoma or atypical intraepidermal melanocytic variants).

78. Test.

Dermoscopy (image‐based) ‐ consultant (invasive melanoma or atypical intraepidermal melanocytic variants).

79. Test.

Dermoscopy (image‐based) ‐ resident (invasive melanoma or atypical intraepidermal melanocytic variants).

80. Test.

Dermoscopy (image‐based) ‐ mixed (secondary care based) (invasive melanoma or atypical intraepidermal melanocytic variants).

81. Test.

Dermoscopy (image‐based) ‐ mixed (other) (invasive melanoma or atypical intraepidermal melanocytic variants).

82. Test.

Dermoscopy (image‐based) ‐ GP/primary care (invasive melanoma or atypical intraepidermal melanocytic variants).

83. Test.

Dermoscopy (image‐based) ‐ physician assistant (invasive melanoma or atypical intraepidermal melanocytic variants).

84. Test.

Dermoscopy ‐ before training (invasive melanoma or atypical intraepidermal melanocytic variants).

85. Test.

Dermoscopy ‐ after training (invasive melanoma or atypical intraepidermal melanocytic variants).

86. Test.

Dermoscopy ‐ before training (invasive melanoma).

87. Test.

Dermoscopy ‐ after training (invasive melanoma).

88. Test.

MM1‐ Dermoscopy ‐ no algorithm ‐ threshold NR ‐ in‐person.

89. Test.

MM1‐ Dermoscopy ‐ pattern analysis ‐ threshold NR ‐ in‐person.

90. Test.

MM1‐ Dermoscopy ‐ ABCD at > 4.2 ‐ in‐person.

91. Test.

MM1‐ Dermoscopy ‐ ABCD at > 5.45 ‐ in‐person.

92. Test.

MM1‐ Dermoscopy ‐ Kenet (modified) at melanoma possible ‐ in‐person.

93. Test.

MM1‐ Dermoscopy ‐ Kenet (modified) at melanoma likely ‐ in‐person.

94. Test.

MM1‐ Dermoscopy ‐ no algorithm ‐ threshold NR ‐ image‐based.

95. Test.

MM1‐ Dermoscopy ‐ no algorithm ‐ decision to excise ‐ image‐based (paired data only).

96. Test.

MM1‐ Dermoscopy ‐ pattern analysis ‐ threshold NR ‐ image‐based.

97. Test.

MM1‐ Dermoscopy ‐ ABCD at > 4.75 ‐ image‐based.

98. Test.

MM1‐ Dermoscopy ‐ ABCD at > 5.45 ‐ image‐based.

99. Test.

MM1‐ Dermoscopy ‐ 7‐point at NR ‐ image‐based.

100. Test.

MM1‐ Dermoscopy ‐ Menzies at 2 negative and ≥ 1 positive ‐ image‐based.

101. Test.

MM1‐ Dermoscopy ‐ 3‐point at > NR ‐ image‐based.

102. Test.

MM1‐ Dermoscopy ‐ Kenet at melanoma likely ‐ image‐based.

103. Test.

MM1‐ Dermoscopy ‐ Kenet at melanoma possible ‐ image‐based.

104. Test.

MM1‐ Dermoscopy CASH at ≥ 8 ‐ image‐based.

105. Test.

MM1‐ Dermoscopy ‐ Kreusch algorithm ‐ image‐based.

106. Test.

MM1‐ Dermoscopy ‐ Menzies for amelanotic at 1 ‐ image‐based.

107. Test.

MM1‐ Dermoscopy ‐ Menzies for amelanotic at 0 ‐ image‐based.

108. Test.

MM3‐ Dermoscopy – no algorithm at NR ‐ in‐person.

109. Test.

MM3‐ Dermoscopy ‐ pattern analysis ‐ threshold NR ‐ in‐person.

110. Test.

MM3‐ Dermoscopy – ABCD at > 5.45 ‐ in‐person.

111. Test.

MM3‐ Dermoscopy – 3‐point at ≥ 2 ‐ in‐person.

112. Test.

MM3‐ Dermoscopy – no algorithm at NR ‐ image‐based.

113. Test.

MM3‐ Dermoscopy ‐ pattern analysis ‐ threshold NR ‐ image‐based.

114. Test.

MM3‐ Dermoscopy – 3‐point at ≥ 2 ‐ image‐based.

115. Test.

MM2 ‐ VI ‐ in‐person (with image‐based dermoscopy).

Characteristics of studies

Characteristics of included studies [ordered by study ID]

Ahnlide 2016.

Study characteristics
Patient sampling	Study design: case series Data collection: retrospective Period of data collection: 7 March 2013‐28 April 2014 Country: Sweden
Patient characteristics and setting	Inclusion criteria: excised melanocytic skin lesions with recorded dermoscopy ABCD score and clinician's preliminary diagnosis Setting: secondary (general dermatology) Prior testing: clinical and/or dermatoscopic suspicion Setting for prior testing: secondary (general dermatology) Exclusion criteria: previously biopsied lesions and wide excisions not included; other exclusion prior to enrolment included: invalid report or missing data (n = 34); visiting residents' data (n = 66); non‐melanocytic on histology or benign melanocytic lesions with special patterns (e.g. papillomatous, congenital naevi and mucosal lesions) (n = 658) Sample size (participants): NR Sample size (lesions): number eligible: 1135/number included: 309 Participant characteristics: NR Lesion characteristics: NR
Index tests	Dermoscopy: no algorithm (clinician's preliminary diagnosis); ABCD Method of diagnosis: in‐person diagnosis Prior test data: clinical examination and/or case notes Diagnostic threshold: ABCD > 4.75 or > 5.45 (calculated automatically based on clinician scoring presence/absence of ABCD criteria into computerised patient file) Preliminary preoperative diagnosis was based on physical examination and dermoscopic assessment (including application of ABCD algorithm) Diagnosis based on: single observer (n = 13) Observer qualifications: dermatology residents (n = 6; "residents were encouraged to consult the specialists in difficult cases"); dermatologists (n = 7) Experience in practice: not described Experience with dermoscopy: NR per observer, but assumed High given training described; describe use of dermoscopy and ABCD at department for > 10 years; reports "repeated joint feedback sessions evaluating the preoperative dermoscopy photographs of excised lesions, enrolment in dermoscopy courses for both residents and senior consultants and daily continuous education in dermoscopy for residents."
Target condition and reference standard(s)	Reference standard: histological diagnosis alone; histopathological diagnosis was recorded postoperatively in the patient file by a nurse. Disease‐positive: 46; disease‐negative: 263 Target condition (final diagnoses) Melanoma (invasive): 23; melanoma (in situ): 23 Benign naevus: 263
Flow and timing	Excluded participants: missing scoring (n = 57); wrongly scored due to pre‐op non‐melanocytic diagnosis (n = 5); lesions with preliminary diagnosis of lentigo maligna or SN (n = 5); ambiguous histology (n = 1) Time interval to reference test: NR ‐ but likely consecutively as dermoscopy was used preoperatively
Comparative
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Yes
Was a case‐control design avoided?	Yes
Did the study avoid inappropriate exclusions?	No
Are the included patients and chosen study setting appropriate?	No
Did the study avoid including participants with multiple lesions?	Unclear
		High	High
DOMAIN 2: Index Test Dermoscopy ‐ in‐person
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Yes
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?	No
Was the test applied and interpreted in a clinically applicable manner?	Yes
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	Yes
Was the test interpretation carried out by an experienced examiner?	Yes
		Low	Low
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	Yes
Were the reference standard results interpreted without knowledge of the results of the index tests?	Unclear
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Unclear
		Low	Unclear
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Yes
Did all patients receive the same reference standard?	Yes
Were all patients included in the analysis?	No
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?	Yes
		High

Alarcon 2014.

Study characteristics
Patient sampling	Study design: case series Data collection: prospective; dermoscopic images assessed remotely from the patient Period of data collection: 1 June 2011‐30 May 2012; 1 year Country: Spain
Patient characteristics and setting	Inclusion criteria: dermoscopically equivocal pigmented lesions, assumed to be melanocytic, seen at Melanoma Unit Setting: specialist unit (skin cancer/PLC), Melanoma Unit of the Hospital Clinic of Barcelona Prior testing: dermatoscopic suspicion in all cases Setting for prior testing: specialist unit (skin cancer/PLC) Exclusion criteria: non‐melanocytic appearance Sample size (participants): number eligible: unclear/number included: 264 Sample size (lesions): number eligible: 343/number included: 264 Participant characteristics: median age (years): 54.7 (8‐89 years); 51.5% male Lesion characteristics: Fitzpatrick phototype: I‐II, 42%; III‐IV, 50%; lesion site: head/neck: 73; 27.7%; trunk: 135; 51.1%; limbs: 49; 18.6%; describe if other 7; 7% (acral). Lesion thickness: ≤ 1 mm: 86 of 92 melanoma
Index tests	Dermoscopy: no algorithm Method of diagnosis: dermoscopic images Prior test data: clinical examination and/or case notes ‐ lesion site and age provided + RCM images. Dermoscopy and RCM interpretation appear to have been conducted by same observer with no indication of blinding Diagnostic threshold: NR; no details Diagnosis based on: single observer (n = 3) Observer qualifications: dermatologist. All the images were interpreted independently by 1/3 dermatologists with expertise in RCM Experience in practice: not described Experience with dermoscopy: assumed High experience; 3 dermatologists with expertise in RCM Any other detail: all of the lesions were imaged with a digital camera (Canon PowerShot G10; Canon, Tokyo, Japan) and a high‐resolution dermatoscope dermatoscope (DermLite Photo; 3Gen LLC, Dana Point, CA, USA)
Target condition and reference standard(s)	Reference standard: histological diagnosis + follow‐up Histology (n = 264); follow‐up (n = 79); selection for excision based on RCM diagnosis otherwise all would have been excised Target condition (final diagnoses) Melanoma (in situ and invasive, or NR): 92; BCC: 12 Benign naevus: 107; 53 SK and AK
Flow and timing	Excluded participants: none reported Time interval for reference: appears consecutive; "Data regarding age, sex, anatomical location, melanoma risk factors and dermoscopic diagnosis were collected before the RCM examination and histopathological analyses were performed" Time interval between index test(s): not specified but appears consecutive application of dermoscopy and RCM
Comparative
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Yes
Was a case‐control design avoided?	Yes
Did the study avoid inappropriate exclusions?	Yes
Are the included patients and chosen study setting appropriate?	No
Did the study avoid including participants with multiple lesions?	Yes
		Low	High
DOMAIN 2: Index Test Dermoscopy ‐ image‐based
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Unclear
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	No
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	No
Was the test interpretation carried out by an experienced examiner?	Yes
		Unclear	High
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	Yes
Were the reference standard results interpreted without knowledge of the results of the index tests?	Unclear
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Yes
		Low	Low
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Yes
Did all patients receive the same reference standard?	No
Were all patients included in the analysis?	Yes
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?
		High

Annessi 2007.

Study characteristics
Patient sampling	Study design: case series Data collection: NR Period of data collection: December 2004‐June 2006 Country: Italy
Patient characteristics and setting	Inclusion criteria: consecutive atypical macular melanocytic lesions; all > 5 mm in diameter, with a flat or barely elevated surface and at least 3 of the following features: (a) asymmetry, (b) irregular margins, (c) ill‐defined borders, and (d) colour variegation. Setting: specialist unit (skin cancer/PLC) Prior testing: clinical suspicion of malignancy without dermatoscopic suspicion Setting for prior testing: specialist unit (skin cancer/PLC) Exclusion criteria: none reported Sample size (participants): number included: 195 Sample size (lesions): number included: 198 Participant characteristics: mean age: 43 years; male: (106 men) 54% Lesion characteristics: all ≤ 1 mm thickness; mean 0.3 mm; all > 5 mm diameter
Index tests	Dermoscopy: pattern analysis; 7PCL; ABCD Method of diagnosis: dermoscopic images Prior test data: unclear; clinical and ELM digital images taken but unclear what was actually presented to observers Diagnostic threshold: reported only for ABCD; melanocytic lesions with ABCD scores 4.76‐5.45 (suspect lesions) were considered test‐positive Diagnosis based on: consensus (n = 2) Observer qualifications: described as "ELM‐experienced dermatologists" Experience in practice: high experience or 'expert’ Experience with dermoscopy: high experience /‘expert’ users
Target condition and reference standard(s)	Reference standard: histological diagnosis alone; conducted in dermatopathology laboratory Disease‐positive: 96; disease‐negative: 102 Target condition (final diagnoses) Melanoma (invasive): 72; melanoma (in situ): 24 Benign naevi: 102 ‐ described as Clark's melanocytic naevi (68 junctional and 34 compound)
Flow and timing	Excluded participants: none described Time interval to reference test: appears consecutive; "After ELM assessment, all lesions were excised and processed for routine histopathologic examination"
Comparative
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Yes
Was a case‐control design avoided?	Yes
Did the study avoid inappropriate exclusions?	Yes
Are the included patients and chosen study setting appropriate?	No
Did the study avoid including participants with multiple lesions?	Yes
		Low	High
DOMAIN 2: Index Test Dermoscopy ‐ image‐based
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Yes
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?	Unclear
Was the test applied and interpreted in a clinically applicable manner?	No
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	Yes
Was the test interpretation carried out by an experienced examiner?	Yes
		Low	High
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	Yes
Were the reference standard results interpreted without knowledge of the results of the index tests?	Unclear
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Yes
		Low	Low
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Yes
Did all patients receive the same reference standard?	Yes
Were all patients included in the analysis?	Yes
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?	Unclear
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?
		Low

Arevalo 2008.

Study characteristics
Patient sampling	Study design: case series Data collection: retrospective image selection/prospective interpretation Period of data collection: no time period given just states lesions evaluated since 1991 Country: Australia
Patient characteristics and setting	Inclusion criteria: melanocytic lesions imaged at the Sydney Melanoma Unit with a histopathologic diagnosis or that remained unchanged following short‐term (4.5‐5 months) digital monitoring (diagnosed as benign) Setting: specialist unit (skin cancer/PLC) Prior testing: selected for excision (no further detail); changes on digital monitoring Setting for prior testing: NR Exclusion criteria: lentigo maligna and lentigo malignant melanoma Sample size (participants): NR Sample size (lesions): number eligible: 3367 melanocytic lesions/number included: 3367 Participant characteristics: NR Lesion characteristics NR
Index tests	Dermoscopy: Menzies criteria Method of diagnosis: dermoscopic images Prior test data: no further information used Diagnostic threshold: lesion must have none of the 2 negative features of symmetry of pattern or single colour, and must have ≥ 1 of the following 9 positive features of melanoma; blue‐white veil, pseudopods, radial streaming, peripheral black dots or globules, multiple brown dots, multiple blue‐grey dots, scar‐like depigmentation, broadened network and multiple colours. Diagnosis based on: unclear; appears to be consensus (n = 2); all lesions scored independently by 2 observers blinded to the diagnosis, with referral to a 3rd observer if there was a disagreement. Observer qualifications: NR; likely dermatologists Experience in practice: not described Experience with dermoscopy: not described Any other detail: the images were obtained using a dermoscopic camera (Dermaphot; Heine Ltd) or a digital imaging device (Solarscan).
Target condition and reference standard(s)	Reference standard: histological diagnosis + follow‐up Further details: not described in detail; only included lesions with histopathology or those that remained unchanged following short‐term (2.5‐4.5 months) Target condition (final diagnoses) Melanoma (invasive): 341 'Benign' diagnoses: 3026
Flow and timing	Excluded participants: poor‐quality index test image as exclusion criterion Time interval to reference test: NR
Comparative
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Unclear
Was a case‐control design avoided?	Yes
Did the study avoid inappropriate exclusions?	No
Are the included patients and chosen study setting appropriate?	No
Did the study avoid including participants with multiple lesions?	Unclear
		High	High
DOMAIN 2: Index Test Dermoscopy ‐ image‐based
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Yes
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	No
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	Yes
Was the test interpretation carried out by an experienced examiner?	Unclear
		Low	High
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	Yes
Were the reference standard results interpreted without knowledge of the results of the index tests?	Unclear
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Unclear
		Low	Unclear
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Unclear
Did all patients receive the same reference standard?	No
Were all patients included in the analysis?	No
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?	No
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?
		High

Argenziano 1998.

Study characteristics
Patient sampling	Study design: unclear Data collection: retrospective image selection/prospective interpretation Period of data collection: NR Country: Italy Test set derived: 342 lesions were randomly divided into a training set of 57 CMs and 139 MN and a test set of 60 CMs and 86 MN
Patient characteristics and setting	Inclusion criteria: atypical melanocytic skin lesions with dermoscopic images that had undergone biopsy due to clinician suspicion Setting: NR Prior testing: dermatoscopic suspicion in all cases Setting for prior testing: secondary (general dermatology) Exclusion criteria: none reported Sample size (participants): NR Sample size (lesions): number eligible: 342/number included: 342 Participant characteristics: NR Lesion characteristics: lesion thickness ≤ 1 mm: 28%; 68 CMs < 0.76 mm; 49 CMs > 0.75 mm
Index tests	Dermoscopy: pattern analysis (not described but classified by clinical reviewer); 7PCL (derived and evaluated in this study); ABCD Method of diagnosis: dermoscopic images; in vivo photography as x10 magnification with special photography equipment after being covered in immersion oil Prior test data: no further information used; "In a blind study"‐ implies no information beyond the dermoscopic images available. Diagnostic threshold: pattern analysis ‐ 'overall ELM diagnosis; 'ABCD ‐ Score > 4.75; 7PCL ‐ Score of 3 or more. Diagnosis based on: single (n = 2; less experienced observers) and Consensus (2 observers) (n = 3; ELM‐experienced observers) Observer qualifications: dermatologist Experience in practice: not described. Experience with dermoscopy: high experience ‐ 3 ELM‐experienced; moderate/trained ‐ less experienced dermatologists (who underwent "short formal ELM training of 9 hours") Any other detail: Training set used to derive 7PCL. Initially two models were developed. One using multivariate analysis to create a formula for calculating the probability of each lesion belonging to the group of melanomas but was deemed too complex for clinical use. The second model used the odds ratios (ORs) from the multivariate analysis to create a simpler diagnostic method based on identification of major and minor ELM criteria. A score of 2 was given to the 3 criteria with ORs> 5 (major criteria) and a score of 1 was given to the 4 criteria with OR< 5 (minor criteria); a total score of 3 or more set to identify melanoma. Major criteria included atypical pigment network (presence of an irregular and prominent pigment network), grey‐blue areas and atypical vascular pattern. Minor criteria: streaks, blotches, irregular dots and globules, and regression pattern (presence of white areas or peppering).
Target condition and reference standard(s)	Reference standard: histological diagnosis alone (not further described) Disease‐positive: 117; disease‐negative: 225 Target condition (final diagnoses) Melanoma (invasive): 99; melanoma (in situ): 18 'Benign' diagnoses: 114 atypical naevi; 111 common naevi
Flow and timing	Excluded participants: none reported Time interval to reference test: NR
Comparative
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Unclear
Was a case‐control design avoided?	Unclear
Did the study avoid inappropriate exclusions?	Unclear
Are the included patients and chosen study setting appropriate?	No
Did the study avoid including participants with multiple lesions?	Unclear
		Unclear	High
DOMAIN 2: Index Test Dermoscopy ‐ image‐based
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Yes
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?	Unclear
Was the test applied and interpreted in a clinically applicable manner?	No
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	Yes
Was the test interpretation carried out by an experienced examiner?	Yes
		Low	High
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	Yes
Were the reference standard results interpreted without knowledge of the results of the index tests?	Unclear
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Unclear
		Low	Unclear
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Unclear
Did all patients receive the same reference standard?	Yes
Were all patients included in the analysis?	Yes
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?	Yes
		Unclear

Argenziano 2006.

Study characteristics
Patient sampling	Study design: RCT allocating primary care physicians to use either VI alone or VI + dermoscopy (only excised lesions can be included for each arm) Data collection: prospective Period of data collection: May 2003‐September 2004 Country: Italy and Spain
Patient characteristics and setting	Inclusion criteria: patients asking for screening or exhibiting ≥ 1 skin tumours as seen during routine physical examination (patient‐finding screening) were considered for inclusion; those undergoing excision were included in this review (i.e. those deemed sufficiently suspicious by the expert evaluation). PCPs were invited to participate in the trial; only those who attended the training sessions and who then screened patients and referred them to the PLCs were randomised Setting: primary Prior testing: no prior testing Setting for prior testing: N/A Exclusion criteria: NR Sample size (participants): number eligible: 3271 patients screened; 1325 participants allocated to 'Naked Eye' observation and 1197 participants allocated to dermoscopy observation; number included: 162 received histology after expert evaluation at the PLC Sample size (lesions): 85 in VI arm and 77 in dermoscopy arm underwent excision Participant characteristics: based on full sample: mean age 40, range 2‐90 (VI group)/41, range 3‐94 (dermoscopy group). Male: 498 (38%) VI group/451 (38%) dermoscopy Lesion characteristics NR
Index tests	VI: ABCD (control arm of RCT) Method of diagnosis: in‐person diagnosis Prior test data: N/A in‐person diagnosis Diagnostic threshold: qualitative NR; described in Intro as: simple morphologic features summarised by the asymmetry, border irregularity, colour variegation, and diameter 5 mm (ABCD) Diagnosis based on: average (n = 37) Observer qualifications: primary care physicians Experience in practice: not described Experience with dermoscopy: not described Other detail: pre‐randomisation all participating PCPs underwent training in ABCD rule for clinical diagnosis and 3PCL for dermoscopy (see below) Dermoscopy: 3‐point rule (intervention arm of RCT) Method of diagnosis: in‐person diagnosis Prior test data: N/A in‐person diagnosis Diagnostic threshold: ≥ 2 characteristics present (algorithm is based on the recognition of only 3 individual features: dermoscopic asymmetry (in colour and/or structure, not in shape), atypical network (pigmented network with thick lines and irregular distribution), and blue‐white structures (presence of any blue and/or white colour within the lesion). Each PCP in both groups examined the individual lesions and scored the patient outcome, as banal or suggestive of skin cancer Diagnosis based on: average (n = 36) Observer qualifications: primary care physicians Experience in practice: not described Experience with dermoscopy: not described Dermoscopy training: all PCPs received training (2 hour session) on the clinical ABCD rule for diagnosis of melanoma, basic recognition of non‐melanoma skin cancers including BCC and SCC + a 2 hour session describing the dermoscopy 3PCL.
Target condition and reference standard(s)	Reference standard: histological diagnosis alone All lesions considered suggestive of skin cancer at the PLC were excised and subsequently diagnosed histopathologically. Equivocal lesions by histopathologic examination were reviewed by a second independent pathologist and a final diagnosis made. Disease‐positive: 92 malignant tumours; disease‐negative: 70 benign tumours Target condition (final diagnoses) Melanoma (in situ and invasive, or NR): 12; BCC: 66; cSCC: 14 Sebhorrheic keratosis: 13; melanocytic naevi = 51; other: 6
Flow and timing	Excluded participants: only those participants who were considered to have lesions suggestive of skin cancer had histology and were included. All the rest had expert diagnosis (not included in the final 2x2 data extracted) Time interval to reference test: NR Time interval between index test(s): N/A (RCT)
Comparative	RCT examining effect of making dermoscopy available to primary care practitioners Blinding between tests: randomised comparison Time interval between index test(s): N/A tests used in different participants
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Yes
Was a case‐control design avoided?	Yes
Did the study avoid inappropriate exclusions?	Unclear
Are the included patients and chosen study setting appropriate?	No
Did the study avoid including participants with multiple lesions?	Unclear
		Unclear	High
DOMAIN 2: Index Test Visual Inspection ‐ in‐person
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Yes
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	No
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	Yes
Was the test interpretation carried out by an experienced examiner?	Unclear
		Low	High
DOMAIN 2: Index Test Dermoscopy ‐ in‐person
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Yes
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	No
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	Yes
Was the test interpretation carried out by an experienced examiner?	Unclear
		Low	High
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	Yes
Were the reference standard results interpreted without knowledge of the results of the index tests?	Unclear
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Yes
		Low	Low
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Unclear
Did all patients receive the same reference standard?	Yes
Were all patients included in the analysis?	No
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?
		High
DOMAIN 5: Comparative
Was each index test result interpreted without knowledge of the results of other index tests or testing strategies?	Yes
Was the interval between application of the index tests less than one month?	Yes
Were all tests applied and interpreted in a clinically applicable manner?	Yes
		Low	Low

Argenziano 2011.

Study characteristics
Patient sampling	Study design: case‐control Data collection: retrospective Period of data collection: 2006‐2008 Country: Naples, Italy
Patient characteristics and setting	Inclusion criteria: randomly sampled 100 melanomas and 100 excised melanocytic naevi from a digital collection of lesions screened 2006‐2008 at the Department of Dermatology of the Second University of Naples; also randomly sampled 100 melanocytic naevi that showed no relevant changes to warrant excision during the follow‐up period from a larger database of monitored naevi Setting: secondary (general dermatology) Prior testing: retrospective study of a random sample of dermoscopic images collected in departmental database. 100/349 excised melanomas 100/1512 excised naevi Setting for prior testing: NR Exclusion criteria: excluded non‐melanocytic lesions, lesions on certain anatomical sites (facial, acral, mucosal and nail lesions), lesions > 15 mm, and lesions with conflicting histopathological features Sample size (participants): NR Sample size (lesions): number included: 300 Participant characteristics: NR Lesion characteristics: NR
Index tests	Dermoscopy: pattern analysis; 7PCL; revised 7PCL Method of diagnosis: dermoscopic images Prior test data: no further information used; "No additional information was provided, to avoid the possible bias that clinical information may give to the assessment on morphological criteria." Diagnostic threshold: pattern analysis ‐ classify as naevus/melanoma/or lesion to be excised. 7PCL ‐ individual criteria scored. Original 7‐point‐score ≥ 3 merits excision (based on 3 major criteria with 2 points each (atypical network, blue‐white veil and atypical vascular pattern) and 4 minor criteria with 1 point each (irregular dots/globules, irregular streaks, irregular blotches and regression structures). Revised 7PCL: score ≥ 1 merits excision (each criterion is given a score of 1 point) Diagnosis based on: average; (n = 8) Observer qualifications: dermatologist Experience in practice: high; "Experienced dermatologists" Experience with dermoscopy: high; dermatologists specifically trained in dermoscopy
Target condition and reference standard(s)	Reference standard: histological diagnosis + follow‐up; 200/300 had histology. 100/300 were naevi that had been followed up 1‐3 years (median 22 months; range 1‐3 years) Target condition (final diagnoses) Melanoma (in situ and invasive, or NR): 100; not clear if in situ included Excised naevi included: 57 Clark naevi, 28 SN, 10 small congenital naevi and 5 blue naevi The remaining 100 monitored lesions were reported as 74 reticular naevi and 26 globular naevi
Flow and timing	Excluded participants: none reported Time interval to reference test: unknown
Comparative
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Yes
Was a case‐control design avoided?	No
Did the study avoid inappropriate exclusions?	No
Are the included patients and chosen study setting appropriate?	No
Did the study avoid including participants with multiple lesions?	Unclear
		High	High
DOMAIN 2: Index Test Dermoscopy ‐ image‐based
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Yes
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?	Unclear
Was the test applied and interpreted in a clinically applicable manner?	No
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	Yes
Was the test interpretation carried out by an experienced examiner?	Yes
		Low	High
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	No
Were the reference standard results interpreted without knowledge of the results of the index tests?	Unclear
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Unclear
		High	Unclear
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Unclear
Did all patients receive the same reference standard?	No
Were all patients included in the analysis?	Yes
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?	Yes
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?	Unclear
		High

Ascierto 2010.

Study characteristics
Patient sampling	Study design: case series Data collection: prospective Period of data collection: NR (states in a period of 1 year) Country: Italy
Patient characteristics and setting	Inclusion criteria: clinically relevant cutaneous pigmented lesions, undergoing dermoscopy and excision; only melanocytic lesions meeting ≥ 2 clinical ABCDE criteria underwent dermoscopy Setting: specialist unit (skin cancer/PLC) Prior testing: clinical examination with ABCDE Setting for prior testing: specialist unit (skin cancer/PLC) Exclusion criteria: none reported Sample size (participants): number eligible: 54/number included: 54 Sample size (lesions): NR Participant characteristics: median age 41 (19‐73 years); 19 men Lesion characteristics NR
Index tests	Dermoscopy: risk stratification (modified Kenet 1994) Method of diagnosis: in‐person diagnosis; all participants underwent total body skin examination Prior test data: clinical examination and/or case notes Diagnostic threshold: very high risk; lesion with a pigment network and any of the classical ELM features specific for melanoma (pseudopods, radial streaming, blue‐grey veil, atypical vessel, etc.). High risk: lesion with a pigment network and subtle new ELM features that may suggest melanoma but often are also seen in atypical naevi. Diagnosis based on: unclear, assumed single observer per participant (n = 3) Observer qualifications: dermatologist Experience in practice: high; "evaluations made by expert dermatologists (at least 3 years of experience)" Experience with dermoscopy: assumed high
Target condition and reference standard(s)	Reference standard: histological diagnosis alone (not further described) Disease‐positive: 12 MM; disease‐negative: 42 Target condition (final diagnoses) Melanoma (invasive) 12 'Benign' diagnoses: 42
Flow and timing	Exclusions: none reported Time interval to reference test: "Before surgery, all patients were investigated by clinical and epiluminescence microscopy (ELM) screenings"
Comparative
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Unclear
Was a case‐control design avoided?	Yes
Did the study avoid inappropriate exclusions?	Unclear
Are the included patients and chosen study setting appropriate?	No
Did the study avoid including participants with multiple lesions?	Yes
		Unclear	High
DOMAIN 2: Index Test Dermoscopy ‐ in‐person
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Yes
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	Yes
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	Yes
Was the test interpretation carried out by an experienced examiner?	Yes
		Low	Low
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	Yes
Were the reference standard results interpreted without knowledge of the results of the index tests?	Unclear
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Unclear
		Low	Unclear
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Yes
Did all patients receive the same reference standard?	Yes
Were all patients included in the analysis?	Yes
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?
		Low

Bauer 2000.

Study characteristics
Patient sampling	Study design: case series Data collection: NR. Appears retrospective Period of data collection: January 1996‐February 1997 Country: Italy
Patient characteristics and setting	Inclusion criteria: PSLs examined and excised during a campaign for the early diagnosis of cutaneous melanoma (CM) Setting: secondary (general dermatology); from authors' institution Prior testing: NR "campaign for the early diagnosis of cutaneous melanoma (CM)" Setting for prior testing: NR Exclusion criteria: none reported Sample size (participants): number included: 311 Sample size (lesions): number included: 315 Participant characteristics: NR Lesion characteristics: thickness: 14 < 0.75 mm, 10 0.75‐1.5 mm, and 6 > 1.5 mm (n = 42 melanoma)
Index tests	Dermoscopy: no algorithm; possibly based on pattern analysis Method of diagnosis: in‐person diagnosis Prior test data: clinical examination based on ABCD Diagnostic threshold: presence of malignancy; ELM parameters considered included irregular and multi component pigmentary network pattern, peripheral dark network patches, sharp network margin, pseudopods, radial streaming, blue‐grey areas, pigment dots (blotches, black dots, brown globules), black dots at periphery, whitish veil, depigmentation and hypopigmented areas, erythema, telangiectasia, comedo‐like openings, milia‐like cysts, red‐blue areas Diagnosis based on: consensus (3 observers) "diagnosis was made by consensus amongst the dermatologists (Stanganelli 2005) ... when they disagreed a fourth dermatologist, an expert in the diagnosis of PSLs, was consulted."; n = 4 Observer qualifications: dermatologist Experience in practice: not described Experience with dermoscopy: assumed high ‐ all dermatologists were "trained in the recognition of PSLs during a training course on the clinical diagnosis of naevi and melanomas"; with referral of disagreements to PSL expert Also evaluates a CAD‐based test not included in this review
Target condition and reference standard(s)	Reference standard: histological diagnosis alone (not further described) Disease‐positive: 42; disease‐negative: 273 Target condition (final diagnoses) Melanoma (invasive): 30; melanoma (in situ): 12 Severe dysplasia: 25 'atypical' dysplastic; benign naevus: 212; 36 nonmelanocytic
Flow and timing	Particpant exclusions: none reported Index test to reference standard interval: after diagnosis, "all lesions were then excised and examined histologically
Comparative
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Unclear
Was a case‐control design avoided?	Yes
Did the study avoid inappropriate exclusions?	Unclear
Are the included patients and chosen study setting appropriate?	No
Did the study avoid including participants with multiple lesions?	Yes
		Unclear	High
DOMAIN 2: Index Test Dermoscopy ‐ in‐person
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Yes
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	No
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	Yes
Was the test interpretation carried out by an experienced examiner?	Yes
		Low	High
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	Yes
Were the reference standard results interpreted without knowledge of the results of the index tests?	Unclear
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Unclear
		Low	Unclear
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Yes
Did all patients receive the same reference standard?	Yes
Were all patients included in the analysis?	Yes
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?
		Low

Benelli 1999.

Study characteristics
Patient sampling	Study design: case series Data collection: prospective Period of data collection: 1 September 1997‐30 September 1998 Country: Italy
Patient characteristics and setting	Inclusion criteria: all PSLs observed and excised at the Dermatologic Surgery Department Setting: Dermatologic Surgery Department Prior testing: selected for excision (no further detail) Setting for prior testing: Dermatologic Surgery Department Exclusion criteria: none reported Sample size (participants): NR Sample size (lesions): number included: 401 Participant characteristics: NR Lesion characteristics: thickness 42, < 0.75 mm thick; 80, 0.76 to 1.5 mm thick; 4, 1.5 to 4 mm thick (mean 0.60 mm, median 0.55 mm. max 1.9 mm, min 0.10 mm, SD 0.45)
Index tests	VI: ABCDE Method of diagnosis: in‐person diagnosis Prior test data: N/A in‐person diagnosis Diagnostic threshold: data given for accuracy of each potential score (1‐5); score estimation described in detail Diagnosis based on: consensus (2 observers) n = 2 Observer qualifications: dermatologist Experience in practice: not described Experience with dermoscopy: not described Dermoscopy 7FFM Method of diagnosis: in‐person diagnosis Prior test data: clinical and dermoscopic evaluations made in person by 2 dermatologists prior to excision. Decision to excise the lesions was taken prior to this by 3 different dermatologists. Diagnostic threshold: 2x2 available for 77FM on its own, and for 77FM + each of 5 clinical features, and also for 77FM + each of 5 clinical scores (1‐5); score estimation described in detail Test observers as described for VI (above)
Target condition and reference standard(s)	Reference standard: histological diagnosis alone Disease‐positive: 60 (15%) lesions; disease‐negative: 340 (non‐melanoma) + 1 BCC Target condition (final diagnoses) Melanoma (invasive): 54 (13.5%); melanoma (in situ): 6 (1.5%); BCC: 1 (0.4%) Sebhorrheic keratosis: 1 (0.4%); melanocytic naevi: 316; epithelioid and/or spindle cell naevi: 18 (4.5%); LS: 5 (1.2%)
Flow and timing	Excluded participants: NR Time interval to reference test: same day
Comparative	Blinding between tests: clinical and dermoscopic evaluations made in person by 2 dermatologists prior to excision. Time interval between index test(s): same day
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Yes
Was a case‐control design avoided?	Yes
Did the study avoid inappropriate exclusions?	Unclear
Are the included patients and chosen study setting appropriate?	No
Did the study avoid including participants with multiple lesions?	Unclear
		Unclear	High
DOMAIN 2: Index Test Visual Inspection ‐ in‐person
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	No
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	No
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	Yes
Was the test interpretation carried out by an experienced examiner?	Unclear
		High	High
DOMAIN 2: Index Test Dermoscopy ‐ in‐person
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Yes
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	No
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	Yes
Was the test interpretation carried out by an experienced examiner?	Unclear
		Low	High
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	Yes
Were the reference standard results interpreted without knowledge of the results of the index tests?	Unclear
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Unclear
		Low	Unclear
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Yes
Did all patients receive the same reference standard?	Yes
Were all patients included in the analysis?	Yes
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?
		Low
DOMAIN 5: Comparative
Was each index test result interpreted without knowledge of the results of other index tests or testing strategies?	No
Was the interval between application of the index tests less than one month?	Yes
Were all tests applied and interpreted in a clinically applicable manner?	No
		Low	High

Benelli 2000a.

Study characteristics
Patient sampling	Study design: case‐control Data collection: retrospective image selection/prospective interpretation Period of data collection: January 1993‐December 1998 (melanomas); September 1997‐September 1999 (melanocytic naevi) Country: Italy
Patient characteristics and setting	Inclusion criteria: all small (≤ 6 mm) melanomas and melanocytic naevi consecutively excised over 2 different time periods Setting: secondary (general dermatology) Prior testing: NR; all excised Setting for prior testing: NR Exclusion criteria: size > 6 mm Sample size (participants): NR Sample size (lesions): 600 Participant characteristics: mean age 44 years (range 20‐79) Lesion characteristics: NR
Index tests	VI: ABCDE Method of diagnosis: image‐based Prior test data: unclear whether dermoscopic image also shown at same time Diagnostic threshold: ≥ 2 characteristics present Diagnosis based on: consensus of 3 (evaluated by 3 different observers; in case of disagreement, the majority view prevailed) Observer qualifications: dermatologist (assumed from authors' institution) Experience in practice: NR Experience with dermoscopy: NR Dermoscopy: 7FFM Method of diagnosis: image‐based Prior test data: unclear whether clinical image also shown at same time Diagnostic threshold: ≥ 2 Test observers: as described for VI (above)
Target condition and reference standard(s)	Reference standard: histology alone; no further details Target condition (final diagnoses) Melanoma (invasive or in situ) 76 (8/468 melanomas in full sample were in situ; NR for ≤ 6 mm group) Benign naevi 524
Flow and timing	Excluded participants: none reported Time interval to reference test: NR
Comparative	Blinding between tests: unclear whether images shown at same time Time interval between index test(s): image capture not described
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Yes
Was a case‐control design avoided?	No
Did the study avoid inappropriate exclusions?	No
Are the included patients and chosen study setting appropriate?	No
Did the study avoid including participants with multiple lesions?	Unclear
		High	High
DOMAIN 2: Index Test Visual inspection ‐ image‐based
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Yes
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	No
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	Yes
Was the test interpretation carried out by an experienced examiner?	Unclear
		Low	High
DOMAIN 2: Index Test Dermoscopy ‐ image‐based
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Yes
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	No
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	Yes
Was the test interpretation carried out by an experienced examiner?	Unclear
		Low	High
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	Yes
Were the reference standard results interpreted without knowledge of the results of the index tests?	Unclear
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Unclear
		Low	Unclear
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Unclear
Did all patients receive the same reference standard?	Yes
Were all patients included in the analysis?	Yes
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?
		Unclear
DOMAIN 5: Comparative
Was each index test result interpreted without knowledge of the results of other index tests or testing strategies?	Unclear
Was the interval between application of the index tests less than one month?	Unclear
Were all tests applied and interpreted in a clinically applicable manner?
		Unclear

Benelli 2001.

Study characteristics
Patient sampling	Study design: case series Data collection: retrospective image selection/prospective interpretation Period of data collection: NR ‐ only dates of training course and agreement study given (April‐May 1999) Country: Italy
Patient characteristics and setting	Inclusion criteria: slides of pigmented skin tumours were selected for evaluation during a training course on dermoscopy. Lesions not located on head, palms or soles histological slide available Setting: training images; authors' institution. Institute of Dermatologic Sciences, University of Milan Prior testing: slides of pigmented skin tumours were selected for evaluation during a training course on dermoscopy Setting for prior testing: unspecified Exclusion criteria: none reported Sample size (participants): NR Sample size (lesions): number included: 49 (paper reports 50 but only 49 accounted for in text) Participant characteristics: none reported Lesion characteristics: none reported
Index tests	VI: ABCDE Method of diagnosis: clinical photographs Prior test data: no further information used Diagnostic threshold: ABCDE Score ≥ 2; presence of 2 criteria; ABCDE Score ≥ 3; presence of 3 criteria. All criteria described in full Diagnosis based on: single (n = 1); average (n = 65; attending 1/3 courses in dermoscopy held to inform dermatologists about a new dermatoscopic diagnostic method (7FFM)) Observer qualifications: dermatologists Experience in practice: expert author; not described for participating dermatologists Experience with dermoscopy: expert author; prior experience not described for participating dermatologists; all underwent dermoscopy training for study purposes Dermoscopy: 7FFM Method of diagnosis: dermoscopic images Prior test data: no further information used although clinicians had evaluated clinical images for the same 50 lesions earlier the same day Diagnostic threshold: malignant if 7FFM Score ≥ 2; i.e. presence of 1 major feature or concurrent presence of 2 minor features. All criteria described in full Test observers: as described for VI (above) Dermoscopy training: 3 one‐day dermatoscopy courses held to inform dermatologists about authors' own new dermoscopy algorithm (7FFM). Each course lasted 6 hours. Morning session participants executed pre‐test interpretation of clinical images using ABCDE. Then principles of dermatoscopy were presented during the course and as post‐test, participants evaluated 50 dermoscopic slides of same lesions using 7FFM Length of training 1 day (6 hours) Post‐training experience: < 6months Training format: in‐person teaching
Target condition and reference standard(s)	Reference standard: histological diagnosis alone Disease‐positive: 12/49 melanomas (paper reports 50 but only 49 accounted for in text) Target condition (final diagnoses) Melanoma (invasive): 10; melanoma (in situ): 2; BCC: 2 pigmented BCC 3 seborrhoeic keratoses, 2 pigmented BCC, 1 blue naevus, 2 angiokeratoma, 5 SN, 5 junctional naevi, 9 compound naevi, 10 naevi undergoing regression
Flow and timing	Excluded participants: none reported Time interval to reference test: unclear
Comparative	Blinding between tests: clinical images interpreted in the morning and dermoscopic images in the afternoon Time interval between index test(s): image capture NR
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Unclear
Was a case‐control design avoided?	Yes
Did the study avoid inappropriate exclusions?	Unclear
Are the included patients and chosen study setting appropriate?	No
Did the study avoid including participants with multiple lesions?	Unclear
		Unclear	High
DOMAIN 2: Index Test Visual inspection ‐ image‐based
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Yes
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	No
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	Yes
Was the test interpretation carried out by an experienced examiner?	Yes
		Low	High
DOMAIN 2: Index Test Dermoscopy ‐ image‐based
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Yes
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?	Unclear
Was the test applied and interpreted in a clinically applicable manner?	No
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	Yes
Was the test interpretation carried out by an experienced examiner?	Yes
		Low	High
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	Yes
Were the reference standard results interpreted without knowledge of the results of the index tests?	Unclear
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Unclear
		Low	Unclear
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Unclear
Did all patients receive the same reference standard?	Yes
Were all patients included in the analysis?	Yes
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?
		Unclear
DOMAIN 5: Comparative
Was each index test result interpreted without knowledge of the results of other index tests or testing strategies?	Yes
Was the interval between application of the index tests less than one month?	Unclear
Were all tests applied and interpreted in a clinically applicable manner?	No
		Unclear	High

Binder 1994.

Study characteristics
Patient sampling	Study design: case‐control Data collection: retrospective image selection/prospective interpretation Period of data collection: NR Country: Austria Test set derived: from a sample of 200 PSL, 2 databases were randomly created for learning and testing purposes. The database was also provided with the histological diagnosis.
Patient characteristics and setting	Inclusion criteria: images of PSLs randomly selected from a PSL image database Setting: secondary (general dermatology) Prior testing: selected for excision (no further detail) Setting for prior testing: NR Exclusion criteria: none reported Sample size (participants): NR Sample size (lesions): NR Participant characteristics: NR Lesion characteristics: NR
Index tests	Dermoscopy: (modified) pattern analysis Method of diagnosis: dermoscopic images Prior test data: no further information used; no additional clinical information was provided Diagnostic threshold: observer correct diagnosis of melanoma; presence/absence of 8 ELM criteria were judged (pigment network, brown globules, radial streaming, pseudopods, black dots, margin regularity, pigmentation, depigmentation) and individual diagnosis made Diagnosis based on: consensus (2 observers); n = 3. Images were examined independently by each observer; presence/absence of each ELM criterion decided by agreement of at least 2/3 observers Observer qualifications: dermatologist Experience in practice: high experience or 'Expert' Experience with dermoscopy: high experience ‐ described as "ELM experienced dermatologists" Any other detail: the images were obtained by photographing the PSL on 24 x 36 mm colour slide film, with oil immersion, using a Wild binocular stereomicroscope M 650 (Wild Heerbrugg AG, Switzerland) at a final magnification of x16 using flashlight illumination. Also evaluates a CAD‐based test not included in this review.
Target condition and reference standard(s)	Reference standard: histological diagnosis alone (no further details) Disease‐positive: 40; disease‐negative: 60 Target condition (final diagnoses) Melanoma (in situ and invasive, or NR): 40 Benign naevus: 60
Flow and timing	Excluded participants: none reported Time interval to reference test: NR Time interval between index test(s): NR
Comparative
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Yes
Was a case‐control design avoided?	No
Did the study avoid inappropriate exclusions?	Unclear
Are the included patients and chosen study setting appropriate?	No
Did the study avoid including participants with multiple lesions?	Unclear
		High	High
DOMAIN 2: Index Test Dermoscopy ‐ image‐based
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Yes
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	No
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	Yes
Was the test interpretation carried out by an experienced examiner?	Yes
		Low	High
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	Yes
Were the reference standard results interpreted without knowledge of the results of the index tests?	Unclear
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Unclear
		Low	Unclear
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Unclear
Did all patients receive the same reference standard?	Yes
Were all patients included in the analysis?	Yes
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?
		Unclear

Binder 1995.

Study characteristics
Patient sampling	Study design: case series Data collection: retrospective image selection/prospective interpretation Period of data collection: NR Country: Austria
Patient characteristics and setting	Inclusion criteria: PSLs with available dermoscopy images, both with and without oil immersion, and histological confirmation of diagnosis. Setting: secondary (general dermatology) Prior testing: selected for excision (no further detail) Setting for prior testing: NR Exclusion criteria: none reported Sample size (participants): NR Sample size (lesions): number included: 240 Participant characteristics: NR Lesion characteristics: median thickness 0.7 mm, IQR 0.48‐0.76 mm; all < 1 cm diameter
Index tests	Dermoscopy: no algorithm Method of diagnosis: dermoscopic images of lesions with and without oil immersion (results with oil immersion used for primary analysis); images randomly presented to prevent consecutive presentation of slides for the same lesion. Each image was shown for 20 seconds with a 20‐minute break after 240 slides Prior test data: no further information presented Diagnostic threshold: correct diagnosis of melanoma. For each PSL image only one diagnosis was allowed (MM or not MM) Diagnosis based on: average (n = 19); 6 ELM experts and 13 randomly picked dermatologist 'nonexperts' Observer qualifications: dermatologist Experience in practice: high ‐ all certified dermatologists, experienced in clinical diagnosis Experience with dermoscopy: mixed. 'Nonexperts' had no formal ELM training; ‘expert’ users had been working scientifically in the development of ELM for at least 3 years Any other detail: images were obtained by photographing the PSLs on 24 x 36‐mm colour‐slide film with ELM and without oil immersion (surface microscopy ISM) using a binocular stereomicroscope (M 650, Wild AG, Heerbrugg, Switzerland) at a final magnification of x 16 using flashlight illumination
Target condition and reference standard(s)	Reference standard: histological diagnosis alone (not further described) Disease‐positive: 57; disease‐negative: 183 Target condition (final diagnoses) Melanoma (in situ and invasive, or NR): 57; BCC: 8 Severe dysplasia: 42; other 'Benign' : 133
Flow and timing	Reference interval: appears consecutive; "After photographing, all lesions were excised" Excluded participants: none reported
Comparative
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Yes
Was a case‐control design avoided?	Yes
Did the study avoid inappropriate exclusions?	Unclear
Are the included patients and chosen study setting appropriate?	No
Did the study avoid including participants with multiple lesions?	Unclear
		Unclear	High
DOMAIN 2: Index Test Dermoscopy ‐ image‐based
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Yes
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?	Yes
Was the test applied and interpreted in a clinically applicable manner?	No
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	No
Was the test interpretation carried out by an experienced examiner?	Yes
		Low	High
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	Yes
Were the reference standard results interpreted without knowledge of the results of the index tests?	Unclear
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Unclear
		Low	Unclear
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Yes
Did all patients receive the same reference standard?	Yes
Were all patients included in the analysis?	Yes
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?
		Low

Binder 1999.

Study characteristics
Patient sampling	Study design: case series Data collection: retrospective image selection/prospective interpretation Period of data collection: NR Country: Austria
Patient characteristics and setting	Inclusion criteria: randomly selected, histologically proven PSLs with digital dermoscopy images Setting: secondary (general dermatology) Prior testing: NR Setting for prior testing: NR Exclusion criteria: none reported Sample size (participants): NR Sample size (lesions): number included: 250 Participant characteristics: NR Lesion characteristics: thickness; 7 (17%) of the 41 melanomas were in situ lesions, 24 (59%) < 0.75 mm, and 10 (24%) ranged from 0.76‐1.8 mm; lesion size: all ≤ 8 mm diameter
Index tests	Dermoscopy: ABCD; pattern analysis/no algorithm Method of diagnosis: dermoscopic images Prior test data: no further information used. Computer presented images in random order. Diagnostic threshold: ABCD classification (score > 5.45, > 4.75); sensitivity and specificity also estimated at Q* (point where sensitivity=specificity). Subjective diagnosis (based on certainty of melanoma between 1 and 5) also recorded using pattern analysis (experts) or subjective rating (1st‐year residents) Diagnosis based on: average (n = 17) Observer qualifications: dermatology residents = 5; dermatologist (board‐certified) = 12 Experience in practice: mixed. 1st‐year residents (n = 5); practicing board‐certified dermatologists with experience ranging from 4‐15 years (n = 8), and 4 board‐certified recognised as experts mainly working at PSL units (n = 4) Experience with dermoscopy: mixed experience "Ten of the 17 raters (58.8%) reported on previous usage of the ABCD score, at least for testing purposes of the method." Dermoscopy training: written materials "Before testing all readers were instructed how to apply the ABCD criteria according to the literature published"
Target condition and reference standard(s)	Reference standard: histological diagnosis alone Disease‐positive: 41 (16.4%) lesions; disease‐negative: 209 (83.3%) lesions Target condition (final diagnoses) Melanoma (invasive): 34 lesions; melanoma (in situ): 7 lesions Benign naevus: 96 nevocellular naevi of the compound type, 62 junctional type, 24 dermal type, 13 SN; 14 lentigines
Flow and timing	Participant exclusions: none reported Index to reference interval: consecutive; "After photography all lesions were excised" Time interval between algorithms: same time; image‐based
Comparative
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Yes
Was a case‐control design avoided?	Yes
Did the study avoid inappropriate exclusions?	Unclear
Are the included patients and chosen study setting appropriate?	No
Did the study avoid including participants with multiple lesions?	Unclear
		Unclear	High
DOMAIN 2: Index Test Dermoscopy ‐ image‐based
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Yes
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?	Unclear
Was the test applied and interpreted in a clinically applicable manner?	No
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	Yes
Was the test interpretation carried out by an experienced examiner?	Yes
		Low	High
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	Yes
Were the reference standard results interpreted without knowledge of the results of the index tests?	Unclear
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Unclear
		Low	Unclear
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Yes
Did all patients receive the same reference standard?	Yes
Were all patients included in the analysis?	Yes
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?	Yes
		Low

Blum 2003a.

Study characteristics
Patient sampling	Study design: case series Data collection: retrospective image selection/prospective interpretation Period of data collection: November 1998‐March 2000; lesions overlap with Blum 2004b; data only included in algorithm comparison and not in primary analysis. Country: Germany Test set derived: study developed a simplified version of ABCD algorithm; described full data set "randomly divided into 2 groups (N0 and N1)" but new algorithm development was based on full dataset
Patient characteristics and setting	Inclusion criteria: melanocytic skin lesions to be excised because of clinically and/or dermoscopically clear or suspicious malignancy, or by the wish of the patient after clear benign diagnosis Setting: secondary (general dermatology) Prior testing: clinical and/or dermatoscopic suspicion; patient request for evaluation/excision Setting for prior testing: NR Exclusion criteria: consecutive images of 1 lesion and external recorded images were not included. Images from all parts of the bodies were taken except of subungual and mucosal sites. Sample size (participants): 269 Sample size (lesions): 269 Participant characteristics: male: (45/84) Lesion characteristics: median Breslow thickness 0.96 mm (SD 0.70 mm) for all melanomas
Index tests	Dermoscopy modified ABCD (with and without 'E' for evolution); denoted by study authors as ABC‐point list; + 7FFM; 7PCL; Menzies criteria; original ABCD not included due to lesion overlap with Blum 2004b Method of diagnosis: dermoscopic images Prior test data: unclear; study described image acquisition and storage but did not describe image interpretation Diagnostic threshold: NR for established algorithms "performed according to the criteria given in literature" For ABC‐point list: ≥ 4 points. A ‐ asymmetry of the outer shape in at least 1 axis (+1) (as per (Stolz; Nachbar); (A) ‐ asymmetry of the differential structures inside the lesion in at least 1 axis (+1) (new item); B ‐ abrupt cutoff of network at the border of the lesion in at least 1 quarter of the circumference (+1); C ‐ ≥ 3 colours (+1); D ‐ ≥ 3 differential structures (+1); E ‐ evolution/change noticed by the patient during the last 3 months (+1); no or uncertain information +0; no change in the last 3 months (‐1) Diagnosis based on: unclear (n = NR) Observer qualifications: NR; likely dermatologists Experience in practice: not described Experience with dermoscopy: not described
Target condition and reference standard(s)	Reference standard: histological diagnosis alone Target condition (final diagnoses) Melanoma (invasive): 71; melanoma (in situ): 9; lentigo maligna 4 'Benign' diagnoses: 185
Flow and timing	Excluded participants: none reported Time interval to reference test: appears consecutive; consent given "for the recording and the following operation under local anaesthesia"
Comparative
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Yes
Was a case‐control design avoided?	Yes
Did the study avoid inappropriate exclusions?	Yes
Are the included patients and chosen study setting appropriate?	No
Did the study avoid including participants with multiple lesions?	Yes
		Low	High
DOMAIN 2: Index Test Dermoscopy ‐ image‐based
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	No
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?	Yes
Was the test applied and interpreted in a clinically applicable manner?	No
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	Yes
Was the test interpretation carried out by an experienced examiner?	Unclear
		High	High
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	Yes
Were the reference standard results interpreted without knowledge of the results of the index tests?	Unclear
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Unclear
		Low	Unclear
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Yes
Did all patients receive the same reference standard?	Yes
Were all patients included in the analysis?	Yes
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?	Unclear
		Low

Blum 2003b.

Study characteristics
Patient sampling	Study design: case series Data collection: retrospective image selection/prospective interpretation Period of data collection: September 1998‐December 1999; lesions overlap with Blum 2004b; data only included in algorithm comparison and not in primary analysis. Country: Germany
Patient characteristics and setting	Inclusion criteria: all lesions of patients with multiple atypical naevi excised due to suspicious clinical and/or dermoscopic features were included Setting: specialist unit (skin cancer/PLC) Prior testing: clinical and/or dermatoscopic suspicion Setting for prior testing: specialist unit (skin cancer/PLC) Exclusion criteria: lesions located on soles, palms, subungual and mucosal sites were excluded Sample size (participants): number included: 205 Sample size (lesions): number eligible: 254/number included: 254 Participant characteristics: median age: 39.2 (1.6‐86.4 years); male: 97 (47.3%) Lesion characteristics: NR
Index tests	Dermoscopy: new algorithm (based on criteria of Hofmann‐Wellenhof 2001) Method of diagnosis: dermoscopic images Prior test data: unclear; looks like blinded test interpretation Diagnostic threshold: lesions were classified into six different types according to morphological criteria of the new classification of atypical naevi (Clark naevi): reticular, globular and homogeneous or combinations of two of these types (Hofmann‐Wellenhof 2001). If reticular, globular and homogeneous structures were found in one melanocytic lesion, this lesion was classified as a 3‐structure type Diagnosis based on: consensus (2 observers); n = 2 Observer qualifications: NR; likely dermatologists, "All images were viewed by two investigators" Experience in practice: not described Experience with dermoscopy: not described
Target condition and reference standard(s)	Reference standard: histological diagnosis alone Disease‐positive: 75 MM; disease‐negative: 179 Target condition (final diagnoses) Melanoma (invasive): 63; melanoma (in situ): 12 Benign naevus: recurrent naevus 6; SN or Reed naevus 6; congenital naevus 4; blue naevus 3; naevus without dysplasia 64; dysplastic naevus 96
Flow and timing	Excluded participants: none reported Time interval between index and reference: assumed consecutive, "All patients gave written informed consent for the digital documentation and the following operation under local anaesthesia"
Comparative
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Yes
Was a case‐control design avoided?	Yes
Did the study avoid inappropriate exclusions?	No
Are the included patients and chosen study setting appropriate?	No
Did the study avoid including participants with multiple lesions?	No
		High	High
DOMAIN 2: Index Test Dermoscopy ‐ image‐based
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	No
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	No
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	Yes
Was the test interpretation carried out by an experienced examiner?	Unclear
		High	High
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	Yes
Were the reference standard results interpreted without knowledge of the results of the index tests?	Unclear
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Unclear
		Low	Unclear
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Yes
Did all patients receive the same reference standard?	Yes
Were all patients included in the analysis?	Yes
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?	Yes
		Low

Blum 2004a.

Study characteristics
Patient sampling	Study design: case series Data collection: retrospective Period of data collection: September 1998 to March 1999; lesions overlap with Blum 2004b; data only included in algorithm comparison and not in primary analysis. Country: Germany
Patient characteristics and setting	Inclusion criteria: PSLs excised due to suspicious clinical and/or dermoscopic features Setting: pigmented lesion clinic Prior testing: clinical and/or dermatoscopic suspicion Setting for prior testing: specialist unit (skin cancer/PLC) Exclusion criteria: "Consecutive (repeat) images of one lesion were not included"; malignant epithelial tumours (BCC, SCC) were excluded. Sample size (participants): number eligible: 157/number included: 157 Sample size (lesions): number eligible: 162/number included: 157 Participant characteristics: median age: 38.9 years (2 to 87 years); 45.2% male Lesion characteristics no change in the past 3 months was reported by 87 (55.4%) patients, followed by an observed change in 39 (24.8%) patients and no clear clinical history was given by 31 (19.7%) patients.Lesion site: Face/Ears: 9 (5.7%); trunk: 102 (65%); limbs: 38(24.2%); Acral 6(3.8%), mucosal sites 2(1.2%); lesion thickness ≤ 1 mm: 23 CMs (2 in‐situ, 29 invasive) median Breslow thickness 0.86 mm (standard deviation 0.54 mm; range 0.30‐40 mm)
Index tests	Dermoscopy: pattern analysis Method of diagnosis: dermoscopic images Prior test data: Images interpreted with and without clinical information (clinical history, age, sex of the patients and location of the tumour). Diagnostic threshold: diagnosis of suspect CM made when the level of suspicion was 'roughly 50% or more'. " Clinical history was scored as positive "when any morphological change was recognized by the patient in the past 3 months. Morphological changes included change in size, colour or shape or any sign of ulceration or spontaneous bleeding. Possible dermoscopic classifications were benign naevi, atypical naevi, cutaneous melanoma and other benign epithelial tumours (e.g. SK, angioma)" Diagnosis based on: single observer (n=3) Observer qualifications: not described; likely dermatologists Experience in practice: not described Experience with dermoscopy: high/Moderate/Low "Three investigators ... with different experiences in dermoscopy: excellent (A), average (B) and beginner (C)."
Target condition and reference standard(s)	Reference standard: histological diagnosis alone Target condition (final diagnoses) Melanoma (invasive): 29; melanoma (in situ): 2 Benign naevus: 53; 59 dysplastic naevi; 13 'epithelial benign tumours'
Flow and timing	Excluded participants: consecutive images of one lesion were not included ‐ assumed to be repeated images of same lesion; 162 images originally with 5 excluded to give a total study number of 157 lesion Index to reference interval: assumed consecutive; ""All patients gave their written consent for the digital documentation and the following operation under local anaesthesia"
Comparative
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Yes
Was a case‐control design avoided?	Yes
Did the study avoid inappropriate exclusions?	No
Are the included patients and chosen study setting appropriate?	No
Did the study avoid including participants with multiple lesions?	Yes
		High	High
DOMAIN 2: Index Test Dermoscopy ‐ image‐based
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Yes
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?	Unclear
Was the test applied and interpreted in a clinically applicable manner?	No
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	Yes
Was the test interpretation carried out by an experienced examiner?	Yes
		Low	High
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	Yes
Were the reference standard results interpreted without knowledge of the results of the index tests?	Unclear
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Unclear
		Low	Unclear
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Yes
Did all patients receive the same reference standard?	Yes
Were all patients included in the analysis?	Yes
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?
		Low

Blum 2004b.

Study characteristics
Patient sampling	Study design: case series Data collection: prospective; dermoscopic images assessed remotely from the patient Period of data collection: 11 November 1998‐2 March 2000 Country: Germany Test set derived: for validation of a new CAD procedure the complete collection (837 melanocytic lesions) was divided into 2 equal random subgroups n1 (training set) and n2 (test set)
Patient characteristics and setting	Inclusion criteria: melanocytic skin lesions imaged prospectively at the PLC of the Department of Dermatology, University of Tuebingen, Germany Setting: specialist unit (skin cancer/PLC) Prior testing: NR Setting for prior testing: specialist unit (skin cancer/PLC) Exclusion criteria: images from mucous membrane areas were excluded Sample size (participants): NR Sample size (lesions): number eligible: 837/number included: 837 Participant characteristics: NR Lesion characteristics: median Breslow thickness for all melanomas 0.78 mm (range 0.10‐3.50)
Index tests	Dermoscopy: 7FFM; 7PCL; ABCD; Menzies criteria Method of diagnosis: dermoscopic images Prior test data: not clearly reported; results using new CAD algorithm were, "compared with established dermoscopic classification rules applied to the same image material as the diagnostic computer algorithm." Diagnostic threshold: NR; original algorithms cited, "established dermoscopic classification rules"; authored confirmed published standard thresholds of the mentioned algorithms were used Diagnosis based on: single observer; n= 1 Observer qualifications: dermatologist Experience in practice: not described Experience with dermoscopy: not described, assumed high; "lesions were prospectively classified as benign or malignant melanocytic lesions by the principal investigator (A.B.)"
Target condition and reference standard(s)	Reference standard: histological diagnosis + follow‐up Disease‐positive: 84; disease‐negative: 185 Clinical follow‐up + histology of suspicious lesions: unexcised lesions were analysed independently by 2 of the investigators 2‐3 times in 6 months on the basis of dermoscopic criteria. These lesions were classified as benign without any suspicion of malignancy by dermoscopic criteria, and follow‐up records for at least 6 months showed no evidence of malignancy; n = 568 Target condition (final diagnoses) Melanoma (invasive): 71; melanoma (in situ): 9; lentigo maligna 4 'Benign' diagnoses: 766
Flow and timing	Excluded participants: none reported Time interval to reference test: appears consecutive; "After obtaining informed written patient consent, 269 melanocytic skin lesions were excised under local anaesthesia and the diagnosis was established by histopathology"
Comparative
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Yes
Was a case‐control design avoided?	Yes
Did the study avoid inappropriate exclusions?	Yes
Are the included patients and chosen study setting appropriate?	No
Did the study avoid including participants with multiple lesions?	Unclear
		Low	High
DOMAIN 2: Index Test Dermoscopy ‐ image‐based
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Yes
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?	Unclear
Was the test applied and interpreted in a clinically applicable manner?	Unclear
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	Yes
Was the test interpretation carried out by an experienced examiner?	Yes
		Low	Unclear
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	No
Were the reference standard results interpreted without knowledge of the results of the index tests?	Unclear
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Unclear
		High	Unclear
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Yes
Did all patients receive the same reference standard?	No
Were all patients included in the analysis?	Yes
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?	Yes
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?	Unclear
		High

Bono 2002a.

Study characteristics
Patient sampling	Study design: case series Data collection: prospective Period of data collection: June 1998‐March 2000 Country: Italy Test set derived: a training set was separately derived using data obtained from 237 previously studied lesions (Farina 2000)
Patient characteristics and setting	Inclusion criteria: cutaneous pigmented lesions with clinical and/or dermatoscopic features that suggested a more or less important suspicion for CM Setting: specialist unit (skin cancer/PLC) Prior testing: clinical and/or dermatoscopic suspicion Setting for prior testing: specialist unit (skin cancer/PLC) Exclusion criteria: location/site of lesion ‐ awkwardly situated lesions e.g. interdigital space, ears, nose or eyelids. Lesions on scalp excluded due to hair interference with reflectance; lesion size obvious large, thick melanomas Sample size (participants): number included: 298 Sample size (lesions): number included: 313 Participant characteristics: mean age: 40 years (10‐86 years); male: 122; 41% Lesion characteristics: lesion site: head/neck 3%; trunk 61%; limbs 36%; thickness ≤ 1 mm 70% (46/66); for 55 invasive MM: median thickness 0.64 mm, range 0.17‐3.24 mm. Median diameter: 11 mm (3‐31 mm)
Index tests	VI: no algorithm (training in the unit is based on ABCD but subjective experience of the clinician used for diagnosis) Method of diagnosis: in‐person diagnosis Prior test data: N/A in‐person diagnosis Diagnostic threshold: clinical diagnostic criteria based on subjective experience; emphasise lesion colour over dimensions. Diagnosis of suspect CM made when the level of suspicion was "roughly 50% or more". ABCD criteria have been the basis of training at the unit, but is not implemented in diagnosis; preferred emphasis on colour rather than dimensional character Diagnosis based on: single observer; (n = 1) Observer qualifications: surgical oncologists Experience in practice: high experience or ‘Expert’; > 5 years Experience with dermoscopy: assumed high experience; > 5 years Dermoscopy: no algorithm Method of diagnosis: in‐person diagnosis Prior test data: clinical examination and/or case notes Diagnostic threshold: presence of at least one of the following criterion: radial streaming, pseudopods, grey‐blue veil, regression and erythema, whitish veil, black dots at the periphery (if network present), thick irregular network or milky‐red background with red dots Test observers as described for VI (above) Dermatoscopy performed by a hand‐held monocular microscope equipped with an achromatic lens permitting a magnification of x10 (Heine Delta 10)
Target condition and reference standard(s)	Reference standard: histological diagnosis alone Target condition (final diagnoses) Melanoma (invasive): 55; melanoma (in situ): 11; BCC: 6 'Benign' diagnoses: 241;151 compound naevus, 24 junctional naevus, 12 dermal naevus, 12 LS, 10 dysplastic naevus, 8 spindle‐cell naevus, 8 SK, 5 blue naevus, 3 SN, 8 other
Flow and timing	Excluded participants: NR Interval between index and reference: NR
Comparative	Blinding between tests: same clinician undertook both diagnoses (in‐person) Time interval between index test(s): appears consecutive but not fully clear
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Yes
Was a case‐control design avoided?	Yes
Did the study avoid inappropriate exclusions?	Unclear
Are the included patients and chosen study setting appropriate?	No
Did the study avoid including participants with multiple lesions?	Yes
		Unclear	High
DOMAIN 2: Index Test Visual Inspection ‐ in‐person
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Unclear
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	Yes
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	No
Was the test interpretation carried out by an experienced examiner?	Yes
		Unclear	High
DOMAIN 2: Index Test Dermoscopy ‐ in‐person
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Yes
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	Yes
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	Yes
Was the test interpretation carried out by an experienced examiner?	Yes
		Low	Low
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	Yes
Were the reference standard results interpreted without knowledge of the results of the index tests?	Unclear
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Unclear
		Low	Unclear
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Unclear
Did all patients receive the same reference standard?	Yes
Were all patients included in the analysis?	Yes
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?
		Unclear
DOMAIN 5: Comparative
Was each index test result interpreted without knowledge of the results of other index tests or testing strategies?	No
Was the interval between application of the index tests less than one month?	Yes
Were all tests applied and interpreted in a clinically applicable manner?	No
		Low	High

Bono 2002b.

Study characteristics
Patient sampling	Study design: case series Data collection: prospective Period of data collection: December 2000‐Aug 2001 Country: Italy
Patient characteristics and setting	Inclusion criteria: consecutive cutaneous pigmented lesions that were ≤ 6 mm in diameter and required surgical biopsy for diagnosis based on clinical or dermoscopic suspicion of CMM Setting: specialist unit (skin cancer/PLC) Prior testing: clinical and/or dermatoscopic suspicion Setting for prior testing: NR Exclusion criteria: lesion size > 6 mm; non‐pigmented Sample size (participants): number eligible: 349/number included: 157 Sample size (lesions): number eligible: 375/number included: 161 Participant characteristics: mean age 38 years (14‐82); male: 61 (39%) Lesion characteristics: site: head/neck 14 (9%); trunk 88 (55%); limbs 59 (36%); lesion size: median: 5 mm (1 mm‐6 mm)
Index tests	VI: no algorithm ABCD criteria have been the basis of training at the unit, but is not implemented in diagnosis; preferred emphasis on colour rather than dimensional character Method of diagnosis: in‐person diagnosis Prior test data: N/A, in‐person diagnosis Diagnostic threshold: a diagnosis of suspect CM is made when the level of suspicion is roughly ≥ 50%; lesions at a lower index of suspicion were considered benign for the purposes of this study. ABCD criteria have been the basis of training at the unit, but is not implemented in diagnosis; preferred emphasis on colour rather than dimensional character Diagnosis based on: single observer diagnostic criteria based on the subjective experience of the single clinician examining the pigmented lesion (n = 2) Observer qualifications: surgical oncologists Experience in practice: high experience or ‘Expert’; described as “expert in the recognition of pigmented lesions" Experience with dermoscopy: high experience/‘Expert’ users Other detail: diagnostic criteria were based on the subjective experience of the single clinician examining the pigmented lesion, although the ABCD criteria have been the basis of training at the unit, they did not consider the ABCD mnemonic an essential formula for diagnosis of CM. They did not take into consideration the dimensional character and attributed great importance to the colour of a given lesion. Dermoscopy: no algorithm Method of diagnosis: in‐person diagnosis Prior test data: clinical examination and/or case notes in‐person; dermoscopy performed by the same 2 clinicians who firstly made and registered the clinical diagnosis Diagnostic threshold: dermatoscopic criteria for diagnosis of malignancy were radial streaming, pseudopods, grey‐blue veil, regression and erythema, whiteish veil, black dots at periphery (if network present), thick irregular network, or milky‐red background with red dots. A lesion was suspected for CM when positive for at least one criterion. Test observers as described for VI (above) Any other detail: this technique was performed by a hand‐held monocular microscope equipped with an achromatic lens permitting a magnification of 10x (Heine Delta 10)
Target condition and reference standard(s)	Reference standard: histological diagnosis alone Disease‐positive: 13 CM; disease‐negative: 148 Target condition (final diagnoses) Melanoma (invasive): 10; melanoma (in situ): 3; BCC: 2(1.2%) Mild/moderate dysplasia: 26 (16.1%); SK: 4 (5%); benign naevus: compound naevus 57 (35.4%), junctional naevus 38 (23.6%), spindle‐cell naevus 6 (3.7%), SN 5 (3.1%), blue naevus 2 (1.2%), other 6 (3.7%), LS 2 (1.2%)
Flow and timing	Excluded participants: none reported Time interval to reference test: NR
Comparative	Blinding between tests: dermoscopy performed by the same 2 clinicians who firstly made and registered the clinical diagnosis Time interval between index test(s): appears consecutive
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Yes
Was a case‐control design avoided?	Yes
Did the study avoid inappropriate exclusions?	No
Are the included patients and chosen study setting appropriate?	No
Did the study avoid including participants with multiple lesions?	Yes
		High	High
DOMAIN 2: Index Test Visual Inspection ‐ in‐person
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Unclear
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	Yes
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	No
Was the test interpretation carried out by an experienced examiner?	Yes
		Unclear	High
DOMAIN 2: Index Test Dermoscopy ‐ in‐person
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Yes
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	Yes
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	Yes
Was the test interpretation carried out by an experienced examiner?	Yes
		Low	Low
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	Yes
Were the reference standard results interpreted without knowledge of the results of the index tests?	Unclear
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Unclear
		Low	Unclear
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Unclear
Did all patients receive the same reference standard?	Yes
Were all patients included in the analysis?	Yes
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?
		Unclear
DOMAIN 5: Comparative
Was each index test result interpreted without knowledge of the results of other index tests or testing strategies?	No
Was the interval between application of the index tests less than one month?	Yes
Were all tests applied and interpreted in a clinically applicable manner?	Yes
		Low	Low

Bono 2006.

Study characteristics
Patient sampling	Study design: case series Data collection: retrospective Period of data collection: January 2003‐December 2004 Country: Italy
Patient characteristics and setting	Inclusion criteria: consecutive patients with PSLs with a maximum diameter of ≤ 3 mm undergoing excision. The decision for diagnostic excision was based on clinical and/or dermoscopic features suggesting a more or less important suspicion for CM Setting: specialist unit (skin cancer/PLC) Melanoma and Sarcoma Unit; Istituto Nazionale Tumori of Milan Prior testing: clinical and/or dermatoscopic suspicion Setting for prior testing: specialist unit (skin cancer/PLC) Exclusion criteria: ‐ lesion size > 3 mm Sample size (participants): number eligible: 204/number included: 204 Sample size (lesions): number eligible: 206/number included: 206 Participant characteristics: median age: 40 (6‐74); male: 71 (35%) Lesion characteristics: head/neck 8 (4%); trunk 84 (41%); limbs 114 (55%). Median size: 2 mm (1‐3 mm)
Index tests	VI: no algorithm Method of diagnosis: in‐person diagnosis Prior test data: N/A in‐person diagnosis Diagnostic threshold: a diagnosis of suspicious CM is made when the level of suspicion is roughly ≥ 50%; lesions at a lower index of suspicion were considered not CM; ABCD criteria have been the basis of training at the unit, but is not implemented in diagnosis; preferred emphasis on colour rather than dimensional character Diagnosis based on: single observer; n = 1 Observer qualifications: NR (assumed oncologist as per Bono 2002a and Bono 2002b); "single clinician examining the pigmented lesion" Experience in practice: not described Experience with dermoscopy: not described Dermoscopy: Menzies criteria Method of diagnosis: in‐person diagnosis Prior test data: clinical examination and/or case notes Diagnostic threshold: dermoscopic criteria for diagnosis of malignancy were those of Menzies 1996 and Menzies 2003 Test observers as described for VI (above) A hand‐held monocular microscope equipped with an achromatic lens permitting a magnification of 10x (Heine Delta 20 microscope; Heine Ltd, Herrsching, Germany)
Target condition and reference standard(s)	Reference standard: histological diagnosis alone Details: the slides were evaluated according to widely accepted criteria for the histopathological diagnosis of the various pigmented lesions. Disease‐positive: 23; disease‐negative: 183 Target condition (final diagnoses) Melanoma (invasive): 19 (9.2%); melanoma (in situ): 4 (0%) Mild/moderate dysplasia: dysplastic naevus 10 (4.9%); junctional naevus 76 (36.9%); compound naevus 50 (24.3%); dermal naevus 12 (5.8%); blue naevus 11 (5.3%); Reed naevus 7 (3.4%); SN 3 (1.5%); halo naevus 3 (1.5%); LS 7 (3.4%); other 4 (1.9%)
Flow and timing	Excluded participants: none Time interval to reference test: NR
Comparative	Blinding between tests: Single observer performed both tests Time interval between index test(s): NR
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Yes
Was a case‐control design avoided?	Yes
Did the study avoid inappropriate exclusions?	No
Are the included patients and chosen study setting appropriate?	No
Did the study avoid including participants with multiple lesions?	Yes
		High	High
DOMAIN 2: Index Test Visual Inspection ‐ in‐person
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Yes
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	Yes
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	No
Was the test interpretation carried out by an experienced examiner?	Unclear
		Low	High
DOMAIN 2: Index Test Dermoscopy ‐ in‐person
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Yes
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	Yes
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	Yes
Was the test interpretation carried out by an experienced examiner?	Unclear
		Low	Unclear
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	Yes
Were the reference standard results interpreted without knowledge of the results of the index tests?	No
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Unclear
		Low	Unclear
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Unclear
Did all patients receive the same reference standard?	Yes
Were all patients included in the analysis?	Yes
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?
		Unclear
DOMAIN 5: Comparative
Was each index test result interpreted without knowledge of the results of other index tests or testing strategies?	No
Was the interval between application of the index tests less than one month?	Yes
Were all tests applied and interpreted in a clinically applicable manner?	Yes
		Low	Low

Bourne 2012.

Study characteristics
Patient sampling	Study design: case series Data collection: retrospective image selection/prospective interpretation Period of data collection: June 1‐July 6 2009 Country: Australia
Patient characteristics and setting	Inclusion criteria: all skin lesions consecutively excised at a skin cancer practice to exclude skin cancer and common lesions assessed as clearly benign and not biopsied were included Setting: private; "a dedicated skin cancer practice in Brisbane, Australia" Prior testing: clinical and/or dermatoscopic suspicion. Prior testing to assemble the test set occurs in secondary care by an experienced skin cancer doctor, then the images are tested on primary care professionals Setting for prior testing: specialist unit (skin cancer/PLC) Exclusion criteria: clinically obvious BCCs, which could be easily diagnosed without dermoscopy were not included in the collection set Sample size (participants): number eligible: 46/number included: 46 Sample size (lesions): number eligible: 50/number included: 50 Participant characteristics: mean age: 58y (30‐60y); male: 22 Lesion characteristics: face = 8; neck = 1; chest = 3; back = 21; shoulder = 2; arm = 3; thigh = 4; leg = 7; foot plantar = 1
Index tests	VI: no algorithm Method of diagnosis: clinical photographs Prior test data: no further information used; image assessments were done on 4 occasions, each time using a different diagnostic approach. Diagnostic threshold: NR clinicians provided with Excel answer sheets for each method listing the various criteria used in that algorithm but no algorithm was cited for VI Diagnosis based on: average (n = 4) Observer qualifications: 3 GPs and 1 clinical nurse Experience in practice: mixed; described as varying levels of dermatoscopic experience Experience with dermoscopy: mixed; described as varying levels of dermatoscopic experience Dermoscopy 3‐point rule; Menzies criteria Method of diagnosis: dermoscopic images Prior test data: no further information used; image assessments were done on 4 occasions, each time using a different diagnostic approach. Diagnostic threshold: NR in paper; author communications states that standard thresholds were used, ≥ 2 for the 3PCL and Menzies method as described in original paper Test observers as described for VI (above)
Target condition and reference standard(s)	Reference standard: histological diagnosis + other Histopathological examination (n = 46); expert diagnosis as benign (n = 3); digital follow‐up (n = 1) Target condition (final diagnoses) Melanoma (invasive): 1; melanoma (in situ): 7; BCC: 6; lentigo maligna 1 Sebhorrheic keratosis: 5; 'Benign' diagnoses: banal naevus 10, blue naevus 1, naevus and SK/solar lentigo collision 3, solar lentigo 4, LP or LK 4, DF 1, psoriasis 1, solar keratosis 2, intraepidermal carcinoma 3, regressed keratoacanthoma 1
Flow and timing	Excluded participants: as 2 of the methods (Menzies and 3PCL) related to only pigmented lesions, the 5 non‐pigmented specimens in the set of 50 were excluded from the contingency tables for these methods. Time interval to reference test: "all skin lesions consecutively excised to exclude skin cancer were recorded"
Comparative	Blinding between tests: image assessments on 4 different occasions different algorithms Time interval between index test(s): same day; images acquired at time of face‐to‐face consultation
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Yes
Was a case‐control design avoided?	Yes
Did the study avoid inappropriate exclusions?	Yes
Are the included patients and chosen study setting appropriate?	No
Did the study avoid including participants with multiple lesions?	Yes
		Low	High
DOMAIN 2: Index Test Visual inspection ‐ image‐based
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Unclear
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	No
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	No
Was the test interpretation carried out by an experienced examiner?	Unclear
		Unclear	High
DOMAIN 2: Index Test Dermoscopy ‐ image‐based
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Yes
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?	Unclear
Was the test applied and interpreted in a clinically applicable manner?	No
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	Yes
Was the test interpretation carried out by an experienced examiner?	Unclear
		Low	High
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	No
Were the reference standard results interpreted without knowledge of the results of the index tests?	Yes
Expert opinion (with no histological confirmation) was not used as a reference standard	No
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Unclear
		High	High
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Yes
Did all patients receive the same reference standard?	No
Were all patients included in the analysis?	No
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?	Unclear
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?	Yes
		High
DOMAIN 5: Comparative
Was each index test result interpreted without knowledge of the results of other index tests or testing strategies?	Yes
Was the interval between application of the index tests less than one month?	Yes
Were all tests applied and interpreted in a clinically applicable manner?	No
		Low	High

Broganelli 2005.

Study characteristics
Patient sampling	Study design: case series Data collection: prospective Period of data collection: 1998‐2002 Country: Italy
Patient characteristics and setting	Inclusion criteria: PSLs excised at Dept of Dermatology; all lesions considered suspicious on clinical parameters (on at least one of ABCDE parameters apart from diameter) underwent dermoscopy; 2x2 for melanocytic only included Setting: secondary (general dermatology) Prior testing: clinical suspicion only; decision to excise "follows the dermoscopic diagnosis" Setting for prior testing: specialist unit (skin cancer/PLC) Exclusion criteria: none reported Sample size (participants): NR Sample size (lesions): number included: 638 melanocytic lesions Participant characteristics: age range: between 2 months and 90 years Lesion characteristics NR
Index tests	Dermoscopy; 7PCL Method of diagnosis: unclear. Study describes "day‐to‐day" office activity, but ELM interpretation referred to as evaluating "recorded images" to split into melanocytic and non‐melanocytic lesions. "Melanocytic lesions were investigated on the basis of a pattern analysis and those that revealed altered dermoscopic parameters were distinguished between minor and major criteria" Prior test data: unclear what additional information was available Diagnostic threshold: > 1 alteration in minor criteria or ≥ 1 major char present; not further described. Based on data in Argenziano 1998, this is akin to a score of ≥ 2 as major criteria score 2 points and minor ones score 1 each Diagnosis based on: unclear appears to be in clinic diagnoses n = NR Observer qualifications: NR likely dermatologists Experience in practice: not described Experience with dermoscopy: not described
Target condition and reference standard(s)	Reference standard: histological diagnosis alone Details: "lesions were fixed with formaline and included in paraffin for histological examination. For some of them serial sections were made" Target condition (final diagnoses) Melanoma (in situ and invasive, or NR): 108 'Benign' diagnoses: non‐melanomas = 530
Flow and timing	Excluded participants: none reported Time interval to reference test: NR
Comparative
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Unclear
Was a case‐control design avoided?	Yes
Did the study avoid inappropriate exclusions?	Unclear
Are the included patients and chosen study setting appropriate?	No
Did the study avoid including participants with multiple lesions?	Unclear
		Unclear	High
DOMAIN 2: Index Test Dermoscopy ‐ in‐person
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Yes
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	Unclear
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	Yes
Was the test interpretation carried out by an experienced examiner?	Unclear
		Low	Unclear
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	Yes
Were the reference standard results interpreted without knowledge of the results of the index tests?	Unclear
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Unclear
		Low	Unclear
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Unclear
Did all patients receive the same reference standard?	Yes
Were all patients included in the analysis?	Yes
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?
		Unclear

Carli 1994.

Study characteristics
Patient sampling	Study design: case series Data collection: unclear Period of data collection: November 1993‐May 1994 Country: Italy
Patient characteristics and setting	Inclusion criteria: clinically suspicious melanocytic lesions undergoing excision for diagnostic purposes Setting: secondary (general dermatology) Prior testing: clinical suspicion of malignancy based on: recent lesion changes or presence of at least two of: diameter > 6 mm, asymmetric, irregular feathery edges, uneven or "very" dark colour, "increased or disappearance of skin outline" Setting for prior testing: secondary (general dermatology) Exclusion criteria: clinically obvious melanomas excluded Sample size (participants): number included: 67 Sample size (lesions): number included: 67 Participant characteristics: mean age 36 years; median age 33; all > 20 years; male: 31% Lesion characteristics: none reported
Index tests	Dermoscopy: pattern analysis; criteria derived from a number of other studies (citations include Steiner 1993, Pehamberger 1987, Steiner 1987, Nachbar 1994, Bahmer 1990, Kenet 1993, Stolz 1989, Soyer 1987, Dal Pozzo 1994) Method of diagnosis: in‐person diagnosis Prior test data: clinical examination Diagnostic threshold: a pigment network that was irregular, accentuated, wide‐meshed, with distinct borders, + at least one of the following parameters: inhomogeneous depigmentation present at the periphery; presence of unevenly distributed black dots; uneven brown globules, with irregular distribution; presence of radial streaks; presence of pseudopods; the presence of grey‐blue veil Diagnosis based on: consensus (2 observers); n = 2 Observer qualifications: NR; likely dermatologist Experience in practice: high; described as "two experienced observers" Experience with dermoscopy: high; as above
Target condition and reference standard(s)	Reference standard: histology (not further described) Disease‐positive: 5; disease‐negative: 63 Target condition (final diagnoses) Melanoma (invasive): 3; melanoma (in situ): 2 'Benign' diagnoses: atypical melanocytic hyperplasia 2; naevi with architectural atypia 14; naevi with ‘cyto’‐architectural atypia 7; no atypia 40
Flow and timing	Participant exclusions: none reported Time interval to reference test: ELM performed at the time of excision of the lesion
Comparative
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Yes
Was a case‐control design avoided?	Yes
Did the study avoid inappropriate exclusions?	Yes
Are the included patients and chosen study setting appropriate?	No
Did the study avoid including participants with multiple lesions?	Yes
		Low	High
DOMAIN 2: Index Test Dermoscopy ‐ in‐person
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Yes
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	No
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	Yes
Was the test interpretation carried out by an experienced examiner?	Yes
		Low	High
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	Yes
Were the reference standard results interpreted without knowledge of the results of the index tests?	Unclear
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Unclear
		Low	Unclear
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Yes
Did all patients receive the same reference standard?	Yes
Were all patients included in the analysis?	Yes
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?
		Low

Carli 2002a.

Study characteristics
Patient sampling	Study design: case series Data collection: prospective for clinical examination and in‐vivo dermoscopy; retrospective image selection/prospective interpretation for ex‐vivo dermoscopic evaluation Period of data collection: June 1997‐December 1998 Country: Italy
Patient characteristics and setting	Inclusion criteria: clinically equivocal and suspicious PSLs subjected to excisional biopsy at the Institute of Dermatology Setting: secondary (not further specified) Prior testing: clinical and/or dermatoscopic suspicion Setting for prior testing: secondary Exclusion criteria: none reported Sample size (participants): NR Sample size (lesions): 256 Participant characteristics: none reported Lesion characteristics: of the cutaneous melanomas, 14 (25.9%) were in situ melanoma (Clark level I), 18 (33.3%) were invasive with < 0.75 mm thickness, 19 (35.3%) were of intermediate thickness (0.76–1.50 mm) and 3 (5.5%) were > 1.5 mm. The median thickness of invasive melanomas was 0.94 mm ± 0.5 (SD) (range 0.2–6)
Index tests	VI: no algorithm Method of diagnosis: in‐person diagnosis Prior test data: unclear Diagnostic threshold: NR Diagnosis based on: consensus (2 observers); final clinical diagnosis was based on agreement between the 2 observers. In case of disagreement, the opinion of a 3rd observer was considered to be the judge for the diagnosis Observer qualifications: dermatologist Experience in practice: high experience or ‘Expert’; described as “dermatologists with extensive experience in both clinical and dermoscopic diagnosis of pigmented skin lesions” Experience with dermoscopy: high experience /‘Expert’ users Dermoscopy: pattern analysis Method of diagnosis: in‐person diagnosis and image‐based diagnosis. Clinical examination and in vivo dermoscopy were performed before excision by 2 trained dermatologists and diagnosis reached. Dermoscopic images were re‐analysed by the same 2 observers at the end of the inclusion period (December 1998), blind to the previous clinical and histological diagnoses Prior test data: N/A for in‐person; for image‐based: slides of dermoscopic images were evaluated using a viewer that made it impossible to analyse the clinical features of the lesion; both observers had access to clinical information, including the age of the participant, the site of the lesion, the history of change over time as reported by the participant at the time of in vivo examination. Diagnostic threshold: dermoscopic diagnosis was based on the ELM pattern analysis criteria, using the same diagnostic categories used for clinical diagnosis; characteristics investigated included pigment network, pigmentation, hypopigmentation, brown globules, black dots, pseudopods, radial streaming, grey‐blue veil, atypical vascular pattern Test observers as described for VI (above)
Target condition and reference standard(s)	Reference standard: histological diagnosis alone Target condition (final diagnoses) Melanoma (invasive): 40; melanoma (in situ): 14 BCC: 5 Sebhorrheic keratosis: 4; Benign naevus: 90 common melanocytic naevi; 78 melanocytic naevi; 9 blue naevi; 16 SN/ Reed naevi
Flow and timing	Excluded participants: none reported Time interval to reference test: NR
Comparative	Blinding between tests: In‐person clinical examination and dermoscopy Time interval between index test(s): the interval between in vivo dermoscopy and re‐evaluation of dermoscopic images was reported as 1 year
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Yes
Was a case‐control design avoided?	Yes
Did the study avoid inappropriate exclusions?	Unclear
Are the included patients and chosen study setting appropriate?	No
Did the study avoid including participants with multiple lesions?	Unclear
		Unclear	High
DOMAIN 2: Index Test Visual Inspection ‐ in‐person
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Unclear
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	No
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	Yes
Was the test interpretation carried out by an experienced examiner?	Yes
		Unclear	High
DOMAIN 2: Index Test Dermoscopy ‐ in‐person
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Yes
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	No
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	Yes
Was the test interpretation carried out by an experienced examiner?	Yes
		Low	Low
DOMAIN 2: Index Test Dermoscopy ‐ image‐based
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Yes
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	No
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	Yes
Was the test interpretation carried out by an experienced examiner?	Yes
		Low	High
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	Yes
Were the reference standard results interpreted without knowledge of the results of the index tests?	Unclear
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Unclear
		Low	Unclear
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Unclear
Did all patients receive the same reference standard?	Yes
Were all patients included in the analysis?	Yes
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?
		Unclear
DOMAIN 5: Comparative
Was each index test result interpreted without knowledge of the results of other index tests or testing strategies?	No
Was the interval between application of the index tests less than one month?	Yes
Were all tests applied and interpreted in a clinically applicable manner?	No
		Low	High

Carli 2002b.

Study characteristics
Patient sampling	Study design: case series Data collection: NR Period of data collection: NR Country: Italy
Patient characteristics and setting	Inclusion criteria: clinically suspicious or equivocal PSLs undergoing excision for diagnostic purposes; only lesions with a diameter of ≤ 14 mm were included Setting: secondary (general dermatology) Prior testing: clinical suspicion of malignancy without dermatoscopic suspicion Setting for prior testing: secondary (general dermatology) Exclusion criteria: none reported Sample size (participants): number included: NR Sample size (lesions): number included: 57 Participant characteristics: none reported Lesion characteristics: thickness ≤ 1 mm: 11 cases (5 in situ 6 invasive); All ≤ 14 mm diameter
Index tests	VI: no algorithm Method of diagnosis: clinical photographs; fixed focus distance of 10 cm; images observed using a viewer in 2 separate diagnostic sessions Prior test data: no further information used; contact (dermoscopic) images viewed first and then distant images (clinical), without knowing the classification of the contact image of the individual lesions Diagnostic threshold: NR Diagnosis based on: consensus (2 observers); n = 2 Observer qualifications: dermatologist Experience in practice: high experience or ‘Expert’; states, "with experience in the field of PSL" Experience with dermoscopy: high experience /‘Expert’ users; "experienced in the field of PSLs" Other detail: used an auto focus micro Nikkon 60 lens objective mounted on a Nikon f50 camera, with a fixed focus distance of 10 cm Dermoscopy: no algorithm Method of diagnosis: dermoscopic images Prior test data: no further information used; contact (dermoscopic) images viewed first and then distant images (clinical), without knowing the classification of the contact image of the individual lesions Diagnostic threshold: NR Test observers as described for VI (above) Any other detail: Dermaphot device placed directly on the lesion without previous application of oil; only lesions with a diameter of ≤ 14 mm were included in the study. The image has an automatic, original magnification of x 10 .
Target condition and reference standard(s)	Reference standard: histology (not further described) Disease‐positive: 21; disease‐negative: 36 Target condition (final diagnoses) Melanoma (invasive): 6; melanoma (in situ): 5; BCC: 10 'Benign' diagnoses: 36
Flow and timing	Excluded participants: no exclusions reported Time interval to reference test: photographic procedures performed consecutively prior to surgery
Comparative	Blinding between tests: described as blinded Time interval between index test(s): photographic procedures performed consecutively prior to surgery
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Unclear
Was a case‐control design avoided?	Yes
Did the study avoid inappropriate exclusions?	Unclear
Are the included patients and chosen study setting appropriate?	No
Did the study avoid including participants with multiple lesions?	Unclear
		Unclear	High
DOMAIN 2: Index Test Visual inspection ‐ image‐based
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Unclear
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	No
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	No
Was the test interpretation carried out by an experienced examiner?	Yes
		Unclear	High
DOMAIN 2: Index Test Dermoscopy ‐ image‐based
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Unclear
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	No
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	No
Was the test interpretation carried out by an experienced examiner?	Yes
		Unclear	High
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	Yes
Were the reference standard results interpreted without knowledge of the results of the index tests?	Yes
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Unclear
		Low	Unclear
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Yes
Did all patients receive the same reference standard?	Yes
Were all patients included in the analysis?	Yes
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?
		Low
DOMAIN 5: Comparative
Was each index test result interpreted without knowledge of the results of other index tests or testing strategies?	Yes
Was the interval between application of the index tests less than one month?	Yes
Were all tests applied and interpreted in a clinically applicable manner?	No
		Low	High

Carli 2003a.

Study characteristics
Patient sampling	Study design: case series Data collection: retrospective image selection/prospective interpretation Period of data collection: NR Country: Italy (from authors' institution)
Patient characteristics and setting	Inclusion criteria: melanocytic lesions < 14 mm in diameter, excised because they were clinically suspicious or equivocal Setting: secondary (general dermatology) Prior testing: clinical and/or dermatoscopic suspicion Setting for prior testing: specialist unit (skin cancer/PLC) Exclusion criteria: non‐melanocytic lesions Sample size (participants): NR Sample size (lesions): number included: 200 Participant characteristics: none reported Lesion characteristics: all < 14 mm in diameter
Index tests	Dermoscopy: pattern analysis; 7PCL; ABCD Method of diagnosis: dermoscopic images Prior test data: no further information used; "Dermoscopic images were examined using a viewer" Diagnostic threshold: for ABCD > 5.45; for the 7PCL ≥ 3; pattern analysis: threshold not described Diagnosis based on: single observer (n = 5); average also presented Observer qualifications: dermatology residents working out of the PLC: 3 working predominantly in the inpatient units and in mycology laboratories, 1 working in dermato‐allergology and 1 in the general outpatient units of the dermatology clinic Experience in practice: low experience or recently qualified Experience with dermoscopy: low experience/novice users ‐ considered as 'Trained'; all had undergone training in dermoscopy; 1 had previously taken part in a study on dermoscopy based both on pattern analysis and on the ABCD rule while the others had had no previous experience in practical dermoscopy during work in other fields of dermatology Dermoscopy training: length of training 8‐h formal lessons + interactive CD of dermoscopy Post‐training experience: 4‐h practice at pigmented lesion clinic Training format: in‐person teaching; CD‐ROM tutorial Any other detail: images taken at x10 magnification using a Dermaphot (Heine Optotechnik, Germany) mounted on a Nikon F50 camera
Target condition and reference standard(s)	Reference standard: histology (not further described) Disease‐positive: 44; disease‐negative: 156 Target condition (final diagnoses) Melanoma (invasive): 30; melanoma (in situ): 14 Benign naevus: 156
Flow and timing	Participant exclusions: none reported Reference interval NR
Comparative
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Unclear
Was a case‐control design avoided?	Yes
Did the study avoid inappropriate exclusions?	Yes
Are the included patients and chosen study setting appropriate?	No
Did the study avoid including participants with multiple lesions?	Unclear
		Unclear	High
DOMAIN 2: Index Test Dermoscopy ‐ image‐based
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Yes
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?	Unclear
Was the test applied and interpreted in a clinically applicable manner?	No
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	Yes
Was the test interpretation carried out by an experienced examiner?	No
		Low	High
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	Yes
Were the reference standard results interpreted without knowledge of the results of the index tests?	Unclear
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Unclear
		Low	Unclear
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Unclear
Did all patients receive the same reference standard?	Yes
Were all patients included in the analysis?	Yes
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?	Yes
		Unclear

Carli 2003b.

Study characteristics
Patient sampling	Study design: case series Data collection: retrospective image selection/prospective interpretation Period of data collection: 1999‐2001 Country: Italy
Patient characteristics and setting	Inclusion criteria: clinically difficult to diagnose or equivocal melanocytic lesions randomly selected from image database; all melanomas < 1 mm thickness. Setting: secondary (general dermatology) Prior testing: clinical suspicion of malignancy without dermatoscopic suspicion Setting for prior testing: secondary (general dermatology) Exclusion criteria: ≥ 1 mm thick melanomas, non‐melanocytic lesions, easy to diagnose, dermoscopically peculiar lesions (e.g. blue naevi or SN) Sample size (participants): NR Sample size (lesions): number included: 200 Participant characteristics: none reported Lesion characteristics ≤ 1 mm thickness: 64; median thickness 0.3 mm, 25th‐75th centile 0.00‐0.58 mm; mean diameter 7.4 (SD 79) mm; median: 7 mm (2‐16 mm) Any other detail: same lesions appear to be reported in De Giorgi 2011 but with a different set of 8 observers (De Giorgi 2011 excluded from review on this basis)
Index tests	VI: no algorithm Method of diagnosis: clinical photographs Prior test data: no further information used Diagnostic threshold: NR Diagnosis based on: average; n = 8 Observer qualifications: dermatology registrar; 2 final‐year residents. Dermatologist 6 Experience in practice: mixed experience, 2 senior experts, 4 practicing dermatologists, 2 last‐year resident dermatologists. Both latter groups formally trained in dermoscopy. Experience with dermoscopy: classified as 'high' due to expertise/training in dermoscopy use Other detail: clinical photos using Nikon F40 with macro lens at 15 cm Dermoscopy: no algorithm (own choice) Method of diagnosis: clinical photographs and dermoscopic images Prior test data: unclear Diagnostic threshold: NR. All observers familiar with pattern analysis, ABCD and 7PCL, each was free to choose method of choice Test observers as described for VI (above)
Target condition and reference standard(s)	Reference standard: histological diagnosis alone Disease‐positive: 64; disease‐negative: 136 Target condition (final diagnoses) Melanoma (invasive): 40; melanoma (in situ): 24 Other: 136 melanocytic naevi
Flow and timing	Excluded participants: no exclusions reported Time interval to reference test: interval not described
Comparative	Blinding between tests: clinical diagnosis made and then clinical and dermoscopic images viewed together Time interval between index test(s): images obtained at time of excision
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Yes
Was a case‐control design avoided?	Yes
Did the study avoid inappropriate exclusions?	No
Are the included patients and chosen study setting appropriate?	No
Did the study avoid including participants with multiple lesions?	Unclear
		High	High
DOMAIN 2: Index Test Visual inspection ‐ image‐based
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Unclear
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	No
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	No
Was the test interpretation carried out by an experienced examiner?	Yes
		Unclear	High
DOMAIN 2: Index Test Dermoscopy ‐ image‐based
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Yes
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	No
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	Yes
Was the test interpretation carried out by an experienced examiner?	Yes
		Low	High
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	Yes
Were the reference standard results interpreted without knowledge of the results of the index tests?	Unclear
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Yes
		Low	Low
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Unclear
Did all patients receive the same reference standard?	Yes
Were all patients included in the analysis?	Yes
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?
		Unclear
DOMAIN 5: Comparative
Was each index test result interpreted without knowledge of the results of other index tests or testing strategies?	No
Was the interval between application of the index tests less than one month?	Yes
Were all tests applied and interpreted in a clinically applicable manner?	No
		Low	High

Carrera 2016.

Study characteristics
Patient sampling	Study design: case‐control Data collection: retrospective image selection/prospective interpretation. Each PLC provided up to 50 lesions with a 1:3 ratio of melanomas to naevi. Each contributor randomly selected either polarised or non‐polarised images based on 1:1 randomisation. Following exclusions, lesions were randomised into 12 image sets containing 39 (n = 8) or 40 (n = 7) unique lesions and 5 non‐unique lesion images (2 melanoma, 3 benign) that were repeated in all sets. Period of data collection: NR Country: multicentre (images contributed from PLCs in Australia, Austria, Germany, Italy, Spain, Switzerland, and the USA)
Patient characteristics and setting	Inclusion criteria: images of melanocytic lesions including melanomas with an unequivocal histopathologic diagnosis, and histopathologically verified naevi or naevi demonstrating stability under sequential dermoscopic imaging over time. Setting: specialist unit (skin cancer/PLC) 12 PLCs Prior testing: selected for excision (no further detail) Setting for prior testing: specialist unit (skin cancer/PLC) Exclusion criteria: acral, mucosal, or facial sites excluded; non‐melanocytic appearance; lesions with equivocal (final) diagnosis after review of the pathology report or sequential imaging Sample size (participants): number eligible: NR; number included: NR Sample size (lesions): number eligible: 580 lesion images were contributed; number included: 477 (103 excluded on review by Memorial Sloan Kettering Cancer Center investigators) Participant characteristics: none reported Lesion characteristics: none reported
Index tests	Dermoscopy 3‐point rule; 7PCL; ABCD; Menzies criteria; chaos and clues Method of diagnosis: dermoscopic images Prior test data: clinical image, evaluators examined the close‐up clinical image of each lesion before viewing the dermoscopic image; image contributors also asked to provide information on anatomical location, patient age and sex, and imaging modality (polarised vs non‐polarised) but unclear whether this information was provided to observers or not. Diagnostic threshold: observers asked to evaluate "a comprehensive list" of dermoscopic structures abstracted from various algorithms; overlapping criteria were merged into 1 criterion. Criteria were grouped into (1) global pattern, (2) pattern organization, (3) symmetry of contour, (4) symmetry of pattern, (5) architectural disorder, (6) abruptness of lesion border, (7) colours, and (8) melanocytic structures, including network and vascular structures. Algorithm performance was retrospectively assessed based on the following thresholds: 7PCL ≥ 3; CASH ≥ 6; Menzies NR; ABCD > 4.75; 3PCL NR; chaos and clues NR. For NR thresholds, author communications state "We assessed all of the algorithms. There isn't a threshold for these algorithms, there are just published rules of usage that determine the benign/malignant classification of the lesion" Diagnosis based on: consensus (≥ 50%); when ≥ 50% of the observers identified a dermoscopic feature for a given study lesion, the attribute was considered present; n = 130 (240 participants registered via the IDS website for the study; 103 completed all available images in their data sets and 130 evaluated ≥ 20 lesions) Observer qualifications: GP 24; dermatology registrar 25; dermatologist 73; 1 medical student and 7 'other' Experience in practice: mixed; mean 12 (SD 8.7) years of dermatology experience Experience with dermoscopy: mixed; 122 (93.8%) reported being comfortable using dermoscopy, and 121 (93.1%) were regular users of dermoscopy Dermoscopy training: algorithm tutorials were created and posted by dermoscopic experts through the IDS website; review of these was encouraged but not mandatory.
Target condition and reference standard(s)	Reference standard: histological diagnosis + follow‐up Histology: all melanomas (n = 119) and a proportion of benign lesions (n = NR) Clinical follow‐up + histology of suspicious lesions: sequential dermoscopic imaging over time; not further detailed; length of follow‐up NR; naevi required to be either histopathologically verified or to have demonstrated stability under sequential dermoscopic imaging over time. Target condition (final diagnoses) Melanoma (in situ and invasive, or NR): 119 Benign naevus: melanocytic naevus: 358
Flow and timing	Excluded participants: poor quality index test image as exclusion criterion Time interval to reference test: NR Time interval between index test(s): in‐person; sequential
Comparative
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Yes
Was a case‐control design avoided?	No
Did the study avoid inappropriate exclusions?	No
Are the included patients and chosen study setting appropriate?	No
Did the study avoid including participants with multiple lesions?	Unclear
		High	High
DOMAIN 2: Index Test Dermoscopy ‐ image‐based
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Yes
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?	Yes
Was the test applied and interpreted in a clinically applicable manner?	No
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	Yes
Was the test interpretation carried out by an experienced examiner?	Unclear
		Low	High
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	Unclear
Were the reference standard results interpreted without knowledge of the results of the index tests?	Unclear
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Unclear
		Unclear	Unclear
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Unclear
Did all patients receive the same reference standard?	No
Were all patients included in the analysis?	No
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?	Unclear
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?	Yes
		High

Coras 2003.

Study characteristics
Patient sampling	Study design: case series Data collection: prospective Period of data collection: 16‐month period. Date NR Country: Germany
Patient characteristics and setting	Inclusion criteria: PSLs undergoing excision due to diagnosis of melanoma or atypical naevus, to rule out melanoma or at the participant's request. Paper states, "Each of the three participating dermatologists in private practice sent their digital images via email attachment including anonymized identification to the department of dermatology. (Face‐to‐face diagnosis) Setting: secondary (general dermatology) (teledermatoscopy diagnosis); private care; face‐to‐face diagnosis Prior testing: NR Setting for prior testing: NR Exclusion criteria: none reported Sample size (participants): NR Sample size (lesions): number eligible: 90; number included: 45 Participant characteristics: none reported Lesion characteristics: none reported
Index tests	In‐person assessment (for those comparing face‐to‐face vs histology) Method of diagnosis: participating dermatologists with experience in dermatoscopy established a clinical diagnosis based on pattern analysis after personal consultation with the patient in their private practice clinics. Prior test data: NR Diagnostic threshold: NR Diagnosis based on: single Number of examiners: 3 Observer qualifications: dermatologist (experts with great experience in dermoscopy) Experience in practice: high Experience with index test: high Teledermatology Acquisition and transmission of images: each of the participating dermatologists acquired digital images after face‐to‐face consultation, and sent them via an email attachment with corresponding participant data and medical history. Nature of images used: clinical photographs and dermoscopic images Any additional patient information provided: clinical examination and/or case notes Observer qualifications (remote diagnosis): physician experienced in dermatoscopy Diagnosis based on: single observer Method of diagnosis: a physician evaluated the images and made a diagnosis based on the images and history of the participant Other detail: the participating dermatologists used the same technical equipment for the acquisition of digital images.
Target condition and reference standard(s)	Reference standard: histology Details: the histological diagnosis of majority of cases was performed at the Department of Dermatology Regensburg 45 participants; disease‐positive: 16; disease‐negative: 29 Target condition (final diagnoses) Melanoma (invasive): 16; 'Benign' diagnoses: 29
Flow and timing	Excluded participants: reported that many images were of poor quality (10) and that only 45 biopsies were done 50 participants who did not have histology excluded Time interval to reference test: unclear Time interval between index test(s): most likely days (email transmission of images for remote assessment)
Comparative
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Unclear
Was a case‐control design avoided?	Yes
Did the study avoid inappropriate exclusions?	Unclear
Are the included patients and chosen study setting appropriate?	No
Did the study avoid including participants with multiple lesions?	Unclear
		Unclear	High
DOMAIN 2: Index Test Dermoscopy ‐ in‐person
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Yes
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	Yes
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	Yes
Was the test interpretation carried out by an experienced examiner?	Yes
		Low	Low
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	Yes
Were the reference standard results interpreted without knowledge of the results of the index tests?	Unclear
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Unclear
		Low	Unclear
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Unclear
Did all patients receive the same reference standard?	Yes
Were all patients included in the analysis?	No
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?
		High

Cristofolini 1994.

Study characteristics
Patient sampling	Study design: case series Data collection: prospective Period of data collection: October 1990‐June 1991 Country: Italy
Patient characteristics and setting	Inclusion criteria: patients with pigmented lesions presenting during a campaign for the early diagnosis of cutaneous melanoma at the Dermatology Department in Trento Setting: secondary (general dermatology) Prior testing: NR Setting for prior testing: NR Exclusion criteria: lesions that were not taken into consideration included benign lesions, naevi of Unna and Miescher types and naevi that showed no inclusion criteria at the ABCDE clinical examination Sample size (participants): number eligible: 700 people; number included: NR Sample size (lesions): number eligible: 220; number included: 220 Participant characteristics: none reported Lesion characteristics: none reported
Index tests	VI: ABCDE Method of diagnosis: in‐person diagnosis Prior test data: N/A in‐person diagnosis Diagnostic threshold: lesions showing ≥ 2 of the ABCDE criteria all of which were shown the same diagnostic importance, were considered positive Diagnosis based on: unclear; n = 4 Observer qualifications: dermatologist Experience in practice: high experience or ‘Expert’; all trained in the recognition of pigmented lesions during a training course about the clinical diagnosis of naevi and melanomas; all working in a department where the early diagnosis of melanoma had been dealt with for > 10 years Experience with dermoscopy: high experience /‘Expert’ users Other detail: ABCDE criteria are (asymmetry in shape, border irregular and notched, colour mottled‐haphazard display, dimension > 6 mm, evolution changes in pigmentation) Dermoscopy: pattern analysis Method of diagnosis: in‐person diagnosis Prior test data: clinical evaluation directly followed by dermoscopy Diagnostic threshold: lesion positive for at least 1 criterion: irregular and multicomponent pigmentary network pattern, peripheral dark network patches, sharp network margins, pseudopods (if network present), radial streaming (if network present), black dots at periphery (if network present), blue‐grey areas (if network present) and whitish veil (milky way, if network present). Observers: as described above
Target condition and reference standard(s)	Reference standard: histological diagnosis alone Target condition (final diagnoses) Melanoma (in situ and invasive, or NR): 33 Mild/moderate dysplasia: 23 dysplastic naevi; SK: 4; benign naevus: 158 common naevus Other: 2 thrombosed angiomas
Flow and timing	Excluded participants: no exclusions reported Time interval to reference test: not described Time interval between index tests: clinical evaluation directly followed by dermoscopy
Comparative	Blinding between tests: clinical evaluation directly followed by dermoscopy Time interval between index test(s): same day
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Unclear
Was a case‐control design avoided?	Yes
Did the study avoid inappropriate exclusions?	No
Are the included patients and chosen study setting appropriate?	No
Did the study avoid including participants with multiple lesions?	Unclear
		High	High
DOMAIN 2: Index Test Visual Inspection ‐ in‐person
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Yes
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	Unclear
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	Yes
Was the test interpretation carried out by an experienced examiner?	Yes
		Low	Unclear
DOMAIN 2: Index Test Dermoscopy ‐ in‐person
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Yes
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	Unclear
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	Yes
Was the test interpretation carried out by an experienced examiner?	Unclear
		Low	Unclear
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	Yes
Were the reference standard results interpreted without knowledge of the results of the index tests?	Unclear
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Unclear
		Low	Unclear
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Unclear
Did all patients receive the same reference standard?	Yes
Were all patients included in the analysis?	Yes
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?
		Unclear
DOMAIN 5: Comparative
Was each index test result interpreted without knowledge of the results of other index tests or testing strategies?	No
Was the interval between application of the index tests less than one month?	Yes
Were all tests applied and interpreted in a clinically applicable manner?	Unclear
		Low	Unclear

Dal Pozzo 1999.

Study characteristics
Patient sampling	Study design: case series Data collection: prospective; dermoscopic images assessed remotely from the patient Period of data collection: January 1992‐June 1997 Country: Italy Test set derived: "Training set" 218 pigmented lesions classified as: 45 melanomas (19 of which in situ), 38 epithelioid and/or spindle cell naevi; 45 melanocytic naevi; 45 mainly dermal melanocytic naevi. "Test set"; 713 PSLs‐melanocytic in nature consecutively observed
Patient characteristics and setting	Inclusion criteria: PSLs observed clinically and dermoscopically at the Institute of Dermatology Sciences University of Milan; all excised Setting: secondary (general dermatology) Prior testing: NR Setting for prior testing: NR Exclusion criteria: none reported Sample size (participants): NR Sample size (lesions): number eligible: test set ‐ 713 PSLs; number included: 713 Participant characteristics: NR Lesion characteristics: NR
Index tests	Dermoscopy 7FFM (own new algorithm) Method of diagnosis: dermoscopic images Prior test data: no further information used Diagnostic threshold: the lesions where the sum of the features gave a score ≥ 2 were diagnosed as being malignant. Diagnosis based on: consensus (3 observers); n = 3 Observer qualifications: NR; appears to be the 3 co‐authors; likely expert dermatologists Experience in practice: not described Experience with dermoscopy: not described Any other detail: training set of 218 pigmented lesions used to develop new algorithm. All dermoscopic features recorded. Statistical significance of each feature assessed using Chi2 test and Fischer's exact test. Final features chosen according to reproducibility by different observers and relationships with histopathological criteria predictive of malignancy. Final algorithm: to diagnose melanoma the presence of one major feature or the concurrent presence of two minor features is regarded as sufficient. "We attributed a score of 2 to the major features and a score 1 to the minor features: major features are regression erythema, radial streaming, grey‐blue veil, irregularly distributed pseudopods; minor features are unhomogeneity, irregular pigment network, sharp margin."
Target condition and reference standard(s)	Reference standard: histological diagnosis alone Disease‐positive: 168; disease‐negative: 545 Target condition (final diagnoses) Melanoma (invasive): 139; melanoma (in situ): 29; BCC: 1 Sebhorrheic keratosis: 3; Benign naevus: junctional melanocytic naevi = 92; mainly junctional compound melanocytic naevi = 37; compound melanocytic naevi = 224; congenital melanocytic naevi = 20; melanocytic naevi showing regression and inflammatory infiltrate = 102; combined melanocytic naevi = 8 Epithelioid and/or spindle cell naevi = 53; LS = 3; black reticulated solar lentigo = 1; melanoacanthoma = 1
Flow and timing	Excluded participants: none reported Time interval to reference test: none reported
Comparative
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Yes
Was a case‐control design avoided?	Yes
Did the study avoid inappropriate exclusions?	Unclear
Are the included patients and chosen study setting appropriate?	No
Did the study avoid including participants with multiple lesions?	Unclear
		Unclear	High
DOMAIN 2: Index Test Dermoscopy ‐ image‐based
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Yes
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	No
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	Yes
Was the test interpretation carried out by an experienced examiner?	Unclear
		Low	High
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	Yes
Were the reference standard results interpreted without knowledge of the results of the index tests?	Unclear
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Unclear
		Low	Unclear
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Unclear
Did all patients receive the same reference standard?	Yes
Were all patients included in the analysis?	Yes
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?
		Unclear

di Meo 2016.

Study characteristics
Patient sampling	Study design: case series Data collection: retrospective image selection/prospective interpretation Period of data collection: February‐December 2014 Country: Italy
Patient characteristics and setting	Inclusion criteria: melanocytic skin lesions that underwent excision Setting: secondary (general dermatology) Prior testing: selected for excision (no further detail) Setting for prior testing: secondary (general dermatology) Exclusion criteria: acral and mucosal lesions; dysplastic naevi excluded; disagreement between evaluators on tumour histological classification ‐ lesions that did not meet at least 2 consents were excluded; poor‐quality index test image (considered under flow and timing) Sample size (participants): number included: 125 Sample size (lesions): number included: 125 Participant characteristics: mean age: men 44.6 years; women 50.0 years; male: 61; 58% Lesion characteristics: thickness ≤ 1 mm: all 32 melanomas
Index tests	Dermoscopy 3PCL; scored 3‐point '4‐point checklist' (authors' own scoring); CASH algorithm Method of diagnosis: dermoscopic images Prior test data: no further information used Diagnostic threshold: 3PCL ≥ 2 criteria present; CASH score > 7; 4‐point checklist > 2 Diagnosis based on: unclear; lesions were "randomly assessed by two independent dermatologists" not clear if average or consensus; n = 2 Observer qualifications: dermatologist Experience in practice: high Experience with dermoscopy: high; dermatologists with > 7 years of experience in dermoscopy Any other detail: The 3PCL criteria: asymmetry in colour and/or structures in 1/2 axes, pigmented network with thickened lines and irregular distribution, and any blue and/or white structure within the lesion CASH algorithm has 4 criteria: colour, architectural disorder, symmetry and homo/heterogeneity. Scoring described in detail 4‐point checklist; doubled all 3 criteria of the 3PCL and chose the one conferring more sensitivity, specificity and accuracy (symmetry parameter doubled)
Target condition and reference standard(s)	Reference standard: histological diagnosis alone All lesions were excised and independently analysed by 2 dermatopathologists. The diagnosis of dysplastic naevus was based on the histopathological diagnostic criteria set by the World Health Organization Melanoma Programme (Clemente 1991). It was considered as a benign lesion Disease‐positive: 32; disease‐negative: 93 Target condition (final diagnoses) Melanoma (in situ and invasive, or NR): 32 Mild/moderate dysplasia: 50; benign naevus: 43
Flow and timing	Excluded participants: dysplastic naevi (n = 50) excluded from 2x2; poor‐quality index test images ‐ exclusion criterion Interval between index and reference standard: not clearly described
Comparative
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Yes
Was a case‐control design avoided?	Yes
Did the study avoid inappropriate exclusions?	No
Are the included patients and chosen study setting appropriate?	No
Did the study avoid including participants with multiple lesions?	Yes
		High	High
DOMAIN 2: Index Test Dermoscopy ‐ image‐based
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Yes
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?	Unclear
Was the test applied and interpreted in a clinically applicable manner?	No
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	Yes
Was the test interpretation carried out by an experienced examiner?	Yes
		Low	High
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	Yes
Were the reference standard results interpreted without knowledge of the results of the index tests?	Unclear
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Yes
		Low	Low
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Unclear
Did all patients receive the same reference standard?	Yes
Were all patients included in the analysis?	No
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?	Yes
		High

Dolianitis 2005.

Study characteristics
Patient sampling	Study design: case series Data collection: retrospective image selection/prospective interpretation Period of data collection: July 2001‐June 2002 Country: Australia
Patient characteristics and setting	Inclusion criteria: dermoscopy training study using a CD with 5 test sets of images, each with 40 images of melanocytic skin lesions. Only good‐quality macroscopic and dermoscopic images were included. Setting: specialist unit; Victorian Melanoma Service, Department of Dermatology, University of Melbourne Prior testing: unclear Setting for prior testing: NR Exclusion criteria: non‐melanocytic lesions; poor‐quality index test image. Only good‐quality macroscopic and dermoscopic images were included, where the whole lesion was visible, including the entire periphery (considered under flow/timing) Sample size (participants): NR Sample size (lesions): number eligible: 40; number included: 40 Participant characteristics: none reported Lesion characteristics: ≤ 1 mm thickness: 14 invasive melanomas; median 0.50 mm
Index tests	VI: no algorithm Method of diagnosis: clinical photographs alone Prior test data: no further information used Other test data: dermoscopic images presented to observer subsequent to diagnosis using clinical images alone Diagnostic threshold: NR Diagnosis based on: average; 61 participants (invited to participate in a study comparing dermoscopic algorithms; advertised at several medical meetings and on a website for primary care physicians) Observer qualifications: 10 dermatologists, 16 dermatology trainees, 35 GPs Experience in practice: mixed. Participant (volunteers) "had a range of experience levels with assessment of skin lesions [outlined in detail in the paper] .. and a significant number were novices in dermoscopy”. Paper reports 82% of participants responded that they assessed at least 2‐4 PSL per week. Experience in dermoscopy: mixed (as above); some educational material provided Dermoscopy: pattern analysis; 7PCL; ABCD; Menzies criteria Method of diagnosis: dermoscopic images Prior test data: no further information used. Macroscopic image not shown Diagnostic threshold: ABCD rule; lesions scoring > 4.75 (i.e. lesions "of concern" were considered test‐positive along with those considered to be melanomas, scoring > 5.45); thresholds NR for the other algorithms (original studies referenced). Test observers as described for VI (above) Dermoscopy training: participants were given explanatory written material as well as 3 CDs. 2 CDs contained educational material on dermoscopy,1 from the American Academy of Dermatology and the other from the website dermoscopy.org. Participants were advised to work through all the educational material prior to assessing the test set of images. Length of training: not clear Post‐training experience: < 6 months Training format: online/written materials/CD‐ROM tutorial
Target condition and reference standard(s)	Reference standard: histological diagnosis + other (1 lesion described as having no biopsy performed). Histology not further described Disease‐positive: 20; disease‐negative: 19 Expert diagnosis: 1 Target condition (final diagnoses) Melanoma (invasive): 18; lentigo maligna 2 Benign naevus: 7 dysplactic naevi; 3 SN; 3 junctional naevi; 2 compound naevi; 4 other (ink‐spot lentigo, blue naevus, solar lentigo, ephelis)
Flow and timing	Excluded participants: none reported Time interval to reference test: NR Time interval between index test(s): NR
Comparative	Blinding between tests: dermoscopic images presented to observer subsequent to diagnosis using clinical images alone Time interval between index test(s): image acquisition NR
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Yes
Was a case‐control design avoided?	Yes
Did the study avoid inappropriate exclusions?	Unclear
Are the included patients and chosen study setting appropriate?	No
Did the study avoid including participants with multiple lesions?	Unclear
		Unclear	High
DOMAIN 2: Index Test Visual inspection ‐ image‐based
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Unclear
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	No
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	No
Was the test interpretation carried out by an experienced examiner?	Unclear
		Unclear	High
DOMAIN 2: Index Test Dermoscopy ‐ image‐based
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Yes
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?	Unclear
Was the test applied and interpreted in a clinically applicable manner?	No
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	Yes
Was the test interpretation carried out by an experienced examiner?	Unclear
		Low	High
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	No
Were the reference standard results interpreted without knowledge of the results of the index tests?	Yes
Expert opinion (with no histological confirmation) was not used as a reference standard	No
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Unclear
		High	High
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Unclear
Did all patients receive the same reference standard?	No
Were all patients included in the analysis?	Yes
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?	Unclear
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?
		High
DOMAIN 5: Comparative
Was each index test result interpreted without knowledge of the results of other index tests or testing strategies?	Yes
Was the interval between application of the index tests less than one month?	Unclear
Were all tests applied and interpreted in a clinically applicable manner?	No
		Unclear	High

Dreiseitl 2009.

Study characteristics
Patient sampling	Study design: case series Data collection: prospective Period of data collection: test set: February‐November 2004 Country: Austria Test set derived: study focuses on test set but gives detail of separate study in which classifier was trained
Patient characteristics and setting	Inclusion criteria: patients presenting at PSL clinic Setting: specialist unit (skin cancer/PLC) The PSL unit of the Department of Dermatology at the Medical University of Vienna serves as a secondary and tertiary referral centre Prior testing: NR Setting for prior testing: specialist unit (skin cancer/PLC) Exclusion criteria: none reported Sample size (participants): number eligible: 511; number included: 458 with complete information Sample size (lesions): number eligible: 3827; number included: 3021; however data reported on a per‐participant basis Participant characteristics: none reported Lesion characteristics: none reported
Index tests	Dermoscopy: no algorithm Method of diagnosis: in‐person diagnosis; physicians were instructed to perform an independent routine examination on the study participants Prior test data: clinical examination and/or case notes Diagnostic threshold: NR; decision to excise to rule out melanoma histopathologically Diagnosis based on: single observer (n = 1) Observer qualifications: dermatologist; data reported for 6 additional less experienced observers using MoleMax II system (reported in CAD review) Experience in practice: high experience; "Expert dermatologist" Experience with dermoscopy: high experience
Target condition and reference standard(s)	Reference standard: histological diagnosis + follow‐up Histology (excision); number patient/lesions: NR Clinical follow‐up + histology of suspicious lesions Length of follow‐up: 6 months; number participants: NR Target condition (final diagnoses) Melanoma (in situ and invasive, or NR): 27 participants; 31 lesions 'Benign' diagnoses: 431 participants; 2990 lesions
Flow and timing	Excluded participants: 806 lesions (53 participants) with inadequate follow‐up Index test to reference standard interval:
Comparative
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Yes
Was a case‐control design avoided?	Yes
Did the study avoid inappropriate exclusions?	Yes
Are the included patients and chosen study setting appropriate?	Yes
Did the study avoid including participants with multiple lesions?	Yes
		Low	Low
DOMAIN 2: Index Test Dermoscopy ‐ in‐person
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Unclear
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	Yes
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	No
Was the test interpretation carried out by an experienced examiner?	Yes
		Unclear	High
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	Unclear
Were the reference standard results interpreted without knowledge of the results of the index tests?	Unclear
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Unclear
		Unclear	Unclear
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Unclear
Did all patients receive the same reference standard?	No
Were all patients included in the analysis?	No
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?	Yes
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?
		High

Duff 2001.

Study characteristics
Patient sampling	Study design: case series Data collection: retrospective Period of data collection: January 1993‐December 1998 Country: UK
Patient characteristics and setting	Inclusion criteria: excised lesions recorded on PLC database with data supplemented with hospital patient administration system and pathology database Setting: rapid‐access PLC at Frenchay Hospital Prior testing: selected for excision (no further detail) Setting for prior testing: specialist unit (skin cancer/PLC) Exclusion criteria: none reported Sample size (participants): number eligible: 9968 attended clinic during time period; number included: NR Sample size (lesions): number included: 2372 (1256 undertaken immediately) Participant characteristics: male: 40% (n = 950) Lesion characteristics: mean thickness of melanomas reported graphically per annum (all estimates are approximate): 1993, 1.44 mm; 1994, 0.82 mm; 1995, 1.22 mm; 1996, 1.40 mm; 1997, 1.35; 1998, 0.90 mm
Index tests	Dermoscopy; no algorithm Method of diagnosis: in‐person diagnosis Prior test data: clinical examination and/or case notes Diagnostic threshold: NR; diagnosis of melanoma Diagnosis based on: single observer (n = 2 as reported in Kirkpatrick 1995) Observer qualifications: plastic surgeons Experience in practice: NR Experience with dermoscopy: not described; "A consultant examines all lesions with a dermatoscope."
Target condition and reference standard(s)	Reference standard: histology alone; histopathologist with special interest in melanoproliferative lesions Disease‐positive: 586; disease‐negative: 1786 Target condition (final diagnoses) Melanoma (invasive): 400; melanoma (in situ): 186 (128 in situ 58 LMs) BCC: 316; cSCC: 97 Atypical/dysplastic 195; "other" 14; 'Benign': 1164
Flow and timing	Participant exclusions: none reported Index test to reference standard interval: not all lesions were excised immediately (2372 excisions were undertaken, of which 1256 were done immediately
Comparative
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Yes
Was a case‐control design avoided?	Yes
Did the study avoid inappropriate exclusions?	Yes
Are the included patients and chosen study setting appropriate?	No
Did the study avoid including participants with multiple lesions?	Unclear
		Low	High
DOMAIN 2: Index Test Dermoscopy ‐ in‐person
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Unclear
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	Yes
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	No
Was the test interpretation carried out by an experienced examiner?	Unclear
		Unclear	High
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	Yes
Were the reference standard results interpreted without knowledge of the results of the index tests?	Unclear
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Yes
		Low	Low
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Unclear
Did all patients receive the same reference standard?	Yes
Were all patients included in the analysis?	Yes
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?
		Unclear

Dummer 1993.

Study characteristics
Patient sampling	Study design: case series Data collection: prospective; dermoscopic images assessed remotely from the patient Period of data collection: 12 month period (year/dates NR) Country: Germany
Patient characteristics and setting	Inclusion criteria: patients with skin lesions difficult to diagnose clinically Setting: secondary Prior testing: clinical suspicion of malignancy without dermatoscopic suspicion Setting for prior testing: specialist unit (skin cancer/PLC) a type of specialist care‐ dermatology based clinic Exclusion criteria: patients who had excisions performed in individual practices or where there was no histology or cases that were so obvious they didn't need to have further investigation (clearly benign) Sample size (participants): NR Sample size (lesions): number eligible: 824; number included: 771 Participant characteristics: none reported Lesion characteristics: none reported
Index tests	VI: no algorithm Method of diagnosis: in person Prior test data: in person Other test data: dermoscopic images viewed separately Diagnostic threshold: NR Diagnosis based on: single observer; (n = 2 or 3) Observer qualifications: unclear; clinician based in dermatology clinic (assumed dermatologist) Experience in practice: unclear Experience with index test: unclear Dermoscopy: pattern analysis Method of diagnosis: dermoscopic images Prior test data: unclear Diagnostic threshold: NR Observers: as described above
Target condition and reference standard(s)	Reference standard: histological diagnosis alone Disease‐positive: 23 MM; disease‐negative: 748 benign Target condition (final diagnoses) Invasive melanoma: 23 Benign naevus 706; SK 4; benign non‐melanocytic naevus 32
Flow and timing	Excluded participants: 53 non‐melanocytic lesions not included in the final analysis (no melanomas present in this group) Time interval to reference test: NR Time interval between index test(s): NR
Comparative
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Unclear
Was a case‐control design avoided?	Yes
Did the study avoid inappropriate exclusions?	Yes
Are the included patients and chosen study setting appropriate?	No
Did the study avoid including participants with multiple lesions?	Unclear
		Unclear	High
DOMAIN 2: Index Test Visual Inspection ‐ in‐person
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Unclear
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	Yes
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	No
Was the test interpretation carried out by an experienced examiner?	Unclear
		Unclear	High
DOMAIN 2: Index Test Dermoscopy ‐ image‐based
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Yes
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	No
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	Yes
Was the test interpretation carried out by an experienced examiner?	Unclear
		Low	Unclear
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	Yes
Were the reference standard results interpreted without knowledge of the results of the index tests?	No
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Unclear
		Low	Unclear
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Unclear
Did all patients receive the same reference standard?	Yes
Were all patients included in the analysis?	No
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?
		High

Durdu 2011.

Study characteristics
Patient sampling	Study design: case series Data collection: prospective Period of data collection: January 2006‐January 2009 Country: Turkey
Patient characteristics and setting	Inclusion criteria: PSLs that could not be diagnosed with only dermatologic physical examination Setting: secondary (general dermatology) Prior testing: clinical examination and dermoscopy Setting for prior testing: secondary (general dermatology) Exclusion criteria: none reported Sample size (participants): number included: 176 Sample size (lesions): number included: 200 Participant characteristics: mean age: 48 years (4‐85 years). male: 64; 36.4% Lesion characteristics: 9% nodulo‐ulcerative, 56% papular, 17% macular, 10% nodular, 8% plaque
Index tests	Dermoscopy: ABCD Method of diagnosis: in‐person diagnosis Prior test data: clinical examination Diagnostic threshold: 2‐step process: step 1 melanocytic and non‐melanocytic were differentiated (Braun 2005; Zalaudek 2008); step 2 ABCD applied to melanocytic lesions only (threshold > 5.45) Diagnosis based on: single observer; n = 2; 1 for dermoscopy diagnosis and 1 for Tzanck smear Observer qualifications: dermatologist Experience in practice: not described Experience with dermoscopy: not described
Target condition and reference standard(s)	Reference standard: histological diagnosis alone (excisional biopsies (n = 166) or punch biopsy (n = 34) Details: "Biopsy specimens were stained with hematoxylin and eosin. Immunohistochemical (anti‐S‐100 and human melanoma black [HMB]‐45) and histochemical (Fontana‐Masson) stains were also applied, if necessary"; interpretation by a "pathologist" Disease‐positive: 46; disease‐negative: 154 Target condition (final diagnoses) Melanoma (in situ and invasive, or NR): 10; BCC: 34; 1 pigmented mammary Paget disease; 1 pigmented metastatic mammary carcinoma Sebhorrheic keratosis: 24; benign melanocytic naevus: 100; DF 12; warts 16; 1 dirt; 1 hereditary hemorrhagic telangiectasia
Flow and timing	Participant exclusions: none reported Time interval to reference test: appears consecutive. Following dermoscopic examination and cytology "either a punch or an excisional biopsy specimen was taken from the lesions and was examined histopathologically"
Comparative
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Unclear
Was a case‐control design avoided?	Yes
Did the study avoid inappropriate exclusions?	Unclear
Are the included patients and chosen study setting appropriate?	Yes
Did the study avoid including participants with multiple lesions?	No
		Unclear	High
DOMAIN 2: Index Test Dermoscopy ‐ in‐person
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Yes
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	Yes
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	Yes
Was the test interpretation carried out by an experienced examiner?	Unclear
		Low	Unclear
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	Yes
Were the reference standard results interpreted without knowledge of the results of the index tests?	Unclear
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Unclear
		Low	Unclear
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Unclear
Did all patients receive the same reference standard?	Yes
Were all patients included in the analysis?	Yes
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?
		High

Feci 2015.

Study characteristics
Patient sampling	Study design: RCT of the effect of ambient stressors and time constraints on decision making; PSL images were randomised to control group, ambient stress group and time stress* group (*result included in main analysis) Data collection: retrospective image selection/prospective interpretation Period of data collection: January‐December 2013 Country: Italy
Patient characteristics and setting	Inclusion criteria: PSLs suspicious for melanoma and with histopathological diagnoses Setting: secondary (general dermatology) Prior testing: clinical and/or dermatoscopic suspicion of melanoma or atypical Setting for prior testing: secondary (general dermatology) Exclusion criteria: not clearly reported however only melanomas and atypical naevi included Sample size (participants): number included: appears to be 1 lesion per participant ‐ "consecutive PSL removed from different patients" Sample size (lesions): number included: 321; 102 in time stress group Participant characteristics: none reported Lesion characteristics: mean thickness 0.28 mm, range ‐ in situ to 1.88 mm
Index tests	Dermoscopy: pattern analysis Method of diagnosis: dermoscopic images Prior test data: no further information used; "dermatologists" knew neither the aim of the study nor the number of naevi and melanomas within each sample group Diagnostic threshold: NR Diagnosis based on: unclear; appears to be single (and different) observer per arm of the trial (n = 3). The time stress group "simulated clinical decision making by arbitrarily allowing a time of 10s for the evaluation of each PSL" using Microsoft PowerPoint slide show Observer qualifications: dermatologist Experience in practice: high experience Experience with dermoscopy: high experience; described as "expert dermatologists" "with at least 10 years' experience in dermoscopy" Any other detail: dermoscopic image acquisition was performed using DermLite ® II pro (3Gen; DermLite, San Juan Capistrano,Calif., USA) connected to a Cyber‐shot 7.2 megapixel camera (Sony Inc., Tokyo, Japan)
Target condition and reference standard(s)	Reference standard: histological diagnosis alone Details: diagnosis was based on AJCC guidelines (Balch 2001) and always made by the same pathologist Disease‐positive: 102 (34 per arm); disease‐negative: 219 (73 per arm) Target condition (final diagnoses) Melanoma (invasive): 69 (33 per arm); melanoma (in situ): 33 (11 per arm) Benign naevus: benign melanocytic naevi 219
Flow and timing	Excluded participants: appear to have excluded on image quality "Among 686 PSL dermoscopic images acquired during the study period, 321 were suitable for our study" Time interval to reference test: NR
Comparative
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Yes
Was a case‐control design avoided?	Yes
Did the study avoid inappropriate exclusions?	No
Are the included patients and chosen study setting appropriate?	No
Did the study avoid including participants with multiple lesions?	Yes
		High	High
DOMAIN 2: Index Test Dermoscopy ‐ image‐based
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Yes
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	No
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	Yes
Was the test interpretation carried out by an experienced examiner?	Yes
		Low	High
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	Yes
Were the reference standard results interpreted without knowledge of the results of the index tests?	Unclear
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Unclear
		Low	Unclear
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Unclear
Did all patients receive the same reference standard?	Yes
Were all patients included in the analysis?	No
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?
		High

Feldmann 1998.

Study characteristics
Patient sampling	Study design: case series Data collection: prospective Period of data collection: NR Country: Austria
Patient characteristics and setting	Inclusion criteria: melanocytic lesions examined by dermatoscopy prior to excision Setting: secondary (general dermatology) Prior testing: NR; "selection for excision was not exclusively based on the dermatoscopic findings but also according to the wishes of the patients." Setting for prior testing: NR Exclusion criteria: none reported Sample size (participants): NR Sample size (lesions): number included: 500 Participant characteristics: none reported Lesion characteristics: mean Breslow thickness 0.49 mm, range 0.12‐1.38 mm
Index tests	Dermoscopy: ABCD Method of diagnosis: in‐person diagnosis Prior test data: clinical examination Diagnostic threshold: > 5.45 (Nachbar 1994); from study results > 4.2 Diagnosis based on: unclear; n = unclear Observer qualifications: NR Experience in practice: not described Experience with dermoscopy: not described
Target condition and reference standard(s)	Reference standard: histological diagnosis alone Details: histology was performed with at least 3 incisions (naevi), and serial sections through the entire lesion (melanomas). The assessment was based on the generally accepted criteria for dysplasia and malignancy (1, 4). In the case of diagnostic uncertainties, the Austrian reference center for histopathological diagnostics carried out a second assessment. Disease‐positive: 30 MM; disease‐negative: 470 Target condition (final diagnoses) Melanoma (invasive): 25; melanoma (in situ): 5 Mild/moderate dysplasia: 190; benign naevus: 272; 7 lentigines 1 lentigo naevi
Flow and timing	Excluded participants: results not presented for 8 lesions Time interval to reference test: appears consecutive; dermoscopy described as used "prior to ... excision and histology)"
Comparative
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Unclear
Was a case‐control design avoided?	Yes
Did the study avoid inappropriate exclusions?	Unclear
Are the included patients and chosen study setting appropriate?	No
Did the study avoid including participants with multiple lesions?	Unclear
		Unclear	High
DOMAIN 2: Index Test Dermoscopy ‐ in‐person
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Yes
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	Unclear
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	Yes
Was the test interpretation carried out by an experienced examiner?	Unclear
		Low	Unclear
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	Yes
Were the reference standard results interpreted without knowledge of the results of the index tests?	Unclear
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Yes
		Low	Low
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Yes
Did all patients receive the same reference standard?	Yes
Were all patients included in the analysis?	No
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?	Unclear
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?
		High

Ferrari 2015.

Study characteristics
Patient sampling	Study design: case series Data collection: retrospective image selection/prospective interpretation Period of data collection: 2010 Country: Italy
Patient characteristics and setting	Inclusion criteria: melanocytic lesions with equivocal clinical and/or dermoscopic features that underwent excision and had a complete set of dermoscopy and RCM images with histopathology report. Only dermoscopically featureless (scoring 0‐2 on 7PCL) or equivocal lesions (those scoring 3‐4 on dermoscopy 7PCL) were included in RCM evaluation. Setting: secondary (general dermatology) Prior testing: clinical and/or dermatoscopic suspicion Setting for prior testing: secondary (general dermatology) Exclusion criteria: incomplete histopathology report; 90 "positive‐clear cut" lesions (scoring ≥ 5 on 7PCL) were excluded from RCM evaluation Poor‐quality index test image, "Only lesions with high quality dermoscopic images, a complete set of confocal images and histopathology report available were included in the study"; considered under flow and timing Sample size (participants): number included: NR Sample size (lesions): number eligible: 322; number included: 322 for dermoscopy; 232 for RCM Participant characteristics: none reported Lesion characteristics: overall mean thickness 1.05 +/‐ 16 mm, range 0–10 mm (70 melanomas); those scoring 0‐2 on 7PCL: mean 0.18 +/‐ 0.42 mm; range 0–0.94 mm) (6 melanomas). Those scoring 3‐4 on 7PCL: mean 0.36 +/‐0.42, range 0‐1.4 mm (17 melanomas)
Index tests	Dermoscopy: 7PCL Method of diagnosis: dermoscopic images Prior test data: RCM and dermoscopy images interpreted by same observer; no indication of randomisation or interpretation in isolation Diagnostic threshold: "featureless" lesions for score ranging between 0‐2, "positive‐borderline" lesions for score between 3‐4 and "positive‐ clear cut" lesions for score from 5‐10 Diagnosis based on: single observer (n = 1) Observer qualifications: dermatologist Experience in practice: not described Experience with dermoscopy: assumed to be high, described as "dermatologist trained in dermoscopy and RCM"
Target condition and reference standard(s)	Reference standard: histological diagnosis alone Details: histopathology was performed by a board‐certified pathologist Disease‐positive: 70; disease‐negative: 252 Target condition (final diagnoses) Melanoma (in situ and invasive, or NR): 70 'Benign' naevi: 252 (including 15 SN)
Flow and timing	Excluded participants: "Only lesions with high quality dermoscopic images, a complete set of confocal images and histopathology report available were included in the study" Time interval to reference test: images taken 'before excision', "Before excision, all lesions were recorded by means of digital dermoscopy and RCM"
Comparative
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Yes
Was a case‐control design avoided?	Yes
Did the study avoid inappropriate exclusions?	No
Are the included patients and chosen study setting appropriate?	No
Did the study avoid including participants with multiple lesions?	Unclear
		High	High
DOMAIN 2: Index Test Dermoscopy ‐ image‐based
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Yes
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	No
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	Yes
Was the test interpretation carried out by an experienced examiner?	Yes
		Low	High
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	Yes
Were the reference standard results interpreted without knowledge of the results of the index tests?	Unclear
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Yes
		Low	Low
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Yes
Did all patients receive the same reference standard?	Yes
Were all patients included in the analysis?	No
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?
		High

Ferris 2015.

Study characteristics
Patient sampling	Study design: unclear. Some dermoscopic images were collected prospectively and some were obtained from collection of existing images; selection process not described Data collection: retrospective image selection/prospective interpretation Period of data collection: NR Country: USA Test set derived: study developed a new CAD classifier using training/test set of images; plus a 'reader study'* conducted to compare accuracy with dermatologist interpretation of images (*reported here). Some dermoscopic images used to train the classifier were obtained from publicly available or purchased image libraries, these were not included in the reader study or used to test the performance of the classifier. The image set was randomly divided into 2 by diagnosis, with half used for training and half used for testing, with the exception that all high‐grade dysplastic naevi were exclusively assigned to the training set to increase the representation of dermoscopic features that could be present in melanoma. Results were extracted only for the test set.
Patient characteristics and setting	Inclusion criteria: dermoscopic images of skin lesions excised on the basis of clinical suspicion of malignancy, with available histologic diagnoses. Reader study included one melanoma that was misclassified as benign by the new CAD classifier + random sample of images determined to be of suitable quality for display on a computer screen. Setting: secondary (general dermatology) Prior testing: clinical suspicion (no further detail) Setting for prior testing: secondary (general dermatology) Exclusion criteria: high‐grade dysplastic naevi were not included in the test set or reader study Sample size (participants): number eligible: NR; number included: NR Sample size (lesions): number eligible: 473 (includes 273 randomised to training set and 27 non‐biopsied lesions); number included: CAD test set 173 lesions; dermoscopy‐ 65 lesions Participant characteristics: none reported Lesion characteristics: test set: mean lesion thickness 0.76 mm, median 0.5 mm, range 0.2‐98 mm); reader study: mean 0.93 mm, median 0.74 mm, range 0.2‐98 mm.
Index tests	Dermoscopy: no algorithm Method of diagnosis: dermoscopic images Prior test data: no further information used Diagnostic threshold: NR Diagnosis based on: average (n = 30); 35 invited to participate. Observer qualifications: 2 board‐certified dermatologists, 10 dermatology residents, and 8 physician assistants currently practicing dermatology Experience in practice: mixed Experience with dermoscopy: mixed; all observers self‐reported some training and experience with the use of dermoscopy. Among board‐certified dermatologists, 67% reported using dermoscopy ‘‘always/almost always’’ or ‘‘very frequently.’', compared to 90% of the dermatology residents and 75% of the physician assistants.
Target condition and reference standard(s)	Reference standard: histological diagnosis alone Details: all lesions were biopsied based on clinical suspicion of malignancy. All histologic diagnoses were rendered by at least 1 board‐certified dermatopathologist and were used as the reference standard for diagnosis Disease‐positive: dermoscopy 25 MM; CAD 39 MM/disease‐negative: dermoscopy 40 MM; CAD 134 MM Target condition (final diagnoses) For reader study only: Invasive melanomas 15; melanoma in situ 10 Low‐grade dysplastic naevi 16, benign naevi 14 , blue naevi 2, lentigines 4 , SK 4
Flow and timing	Excluded participants: none reported Time interval to reference test: "Dermoscopic images of skin lesions were collected before biopsy"
Comparative
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Unclear
Was a case‐control design avoided?	Unclear
Did the study avoid inappropriate exclusions?	Unclear
Are the included patients and chosen study setting appropriate?	No
Did the study avoid including participants with multiple lesions?	Unclear
		Unclear	High
DOMAIN 2: Index Test Dermoscopy ‐ image‐based
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Unclear
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	No
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	No
Was the test interpretation carried out by an experienced examiner?	Unclear
		Unclear	High
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	Yes
Were the reference standard results interpreted without knowledge of the results of the index tests?	Unclear
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Yes
		Low	Low
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Yes
Did all patients receive the same reference standard?	Yes
Were all patients included in the analysis?	Yes
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?
		Low

Friedman 2008.

Study characteristics
Patient sampling	Study design: case‐control Data collection: retrospective image selection/prospective interpretation Period of data collection: NR; lesions selected in July 2005 Country: USA Test set derived: MelaFind data randomly split into training and test sets however Melafind has previously been evaluated, the only difference here being that only small lesions were included. Full dataset included in review
Patient characteristics and setting	Inclusion criteria: a database of images of PSLs ≤ 6 mm was used to sample images of melanoma and non‐melanoma lesions; "approximately 80% of the lesions were biopsied to rule out melanoma, whereas the remaining lesions were biopsied mostly to rule out non‐melanoma skin cancer or because of patient concern." Setting: mixed (private and secondary); digital dermoscopic database acquired by Electro‐Optical Sciences Inc for the development and testing of MelaFind; 26 clinical sites have contributed (dermatologic hospital‐based clinics and private practice offices) Prior testing: selected for excision (no further detail). All lesions excised or underwent shave biopsy Setting for prior testing: NR Exclusion criteria: high‐grade dysplastic naevi were excluded. Previously biopsied, ulcerated, or bleeding lesions also excluded, as were those on mucosal surfaces and lesions that contained foreign matter (e.g. tattoos) Sample size (participants): number included: 94 Sample size (lesions): number eligible: 1977; number included: 99 Participant characteristics: none reported Lesion characteristics: 21 invasive MM: median thickness 0.32 mm (0.10‐1.40 mm). Lesion size: range: 2 mm‐22 mm
Index tests	Dermoscopy: no algorithm Method of diagnosis: dermoscopic images. Readers were provided with a CD‐ROM with colour dermoscopic images created using MelaFind multispectral image; for some cases standard dermoscopic images were also available. The equivalence of the 2 image types was assessed for a sample of 10 lesions by 3 readers. Prior test data: readers provided with participant gender, age, and lesion location; all evaluations were performed independently Diagnostic threshold: clinical diagnosis; “Is this lesion a melanoma?” and “Would you biopsy/excise this lesion?”. If readers indicated that they would biopsy the lesion because they were sure it was melanoma or to rule out melanoma, then the case was considered true‐positive Diagnosis based on: average; mean and median reported (n = 10); used mean value for review purposes Observer qualifications: 9 dermatologists; 1 nurse practitioner specialising in dermatology Experience in practice: high experience or ‘Expert’; "All 10 readers were expert dermoscopists (9 dermatologists and 1 nurse practitioner specialising in dermatology) Experience with dermoscopy: high experience /‘Expert’ users
Target condition and reference standard(s)	Reference standard: histological diagnosis alone Details: the original histology slides were evaluated by 2 out of 4 study dermatopathologists without knowledge of any additional clinical information; in cases of significant discordance in diagnoses, the slide was reviewed by a third study dermatopathologist. A lesion with at least 1 diagnosis of melanoma by the study dermatopathologists was considered melanoma. Dysplastic naevi with severe cytologic atypia were considered high grade, and those with mild to moderate atypia were considered low grade. Disease‐positive: 49; disease‐negative: 50 Target condition (final diagnoses) Melanoma (invasive): 21; melanoma (in situ): 28; BCC: 2 Mild/moderate dysplasia: 32 low‐grade dysplastic; SK: 2; 14 other benign
Flow and timing	Participant exclusions: none reported Index test to reference standard interval: timing between image acquisition and original histology NR
Comparative
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Yes
Was a case‐control design avoided?	No
Did the study avoid inappropriate exclusions?	No
Are the included patients and chosen study setting appropriate?	No
Did the study avoid including participants with multiple lesions?	Yes
		High	High
DOMAIN 2: Index Test Dermoscopy ‐ image‐based
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Yes
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	No
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	No
Was the test interpretation carried out by an experienced examiner?	Yes
		Low	High
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	Yes
Were the reference standard results interpreted without knowledge of the results of the index tests?	Yes
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Yes
		Low	Low
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Unclear
Did all patients receive the same reference standard?	Yes
Were all patients included in the analysis?	Yes
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?
		Unclear

Gereli 2010.

Study characteristics
Patient sampling	Study design: case‐control Data collection: retrospective image selection/prospective interpretation Period of data collection: NR Country: Turkey
Patient characteristics and setting	Inclusion criteria: images of melanoma and non‐melanoma PSLs; non‐melanoma lesions clinically considered to be atypical before dermoscopic examination and excisional biopsy. Atypicality was determined by the presence of ≥ 3 of the following features: a diameter > 5 mm, ill‐defined borders, irregular margins, and the presence of papular and macular components. Melanoma and non‐melanoma lesions separately sampled Setting: secondary (general dermatology). Authors' institution: Dept Dermatology, Istanbul, Turkey Prior testing: clinical suspicion of malignancy Setting for prior testing: NR Exclusion criteria: none reported Sample size (participants): NR Sample size (lesions): number included: 96 Participant characteristics: none reported Lesion characteristics: all > 5 mm diameter
Index tests	Dermoscopy: 3‐point rule; 7PCL Method of diagnosis: dermoscopic images Prior test data: no further information used Diagnostic threshold: 3‐point rule: ≥ 2 characteristics present (asymmetry, atypical pigment network, blue‐white structures); 7PCL: ≥ 3 characteristics present (atypical pigment network, blue‐whitish veil, atypical vascular pattern, irregular streaks, irregular dots/globules, irregular pigmentation, regression structures) Diagnosis based on: average (n = 3) Observer qualifications: NR; likely dermatologists (co‐authors based in Dept Dermatology) Experience in practice: mixed: "two experienced and one inexperienced observers" Experience with dermoscopy: mixed
Target condition and reference standard(s)	Reference standard: histological diagnosis alone (no further details) Disease‐positive: 48; disease‐negative: 48 Target condition (final diagnoses) Melanoma (invasive): 44 (14 superficial spreading, 12 nodular, 10 acral, 4 lentiginous, 4 without classification of tumour thickness); melanoma (in situ): 4 Sebhorrheic keratosis: 2; blue naevi 2; melanocytic naevi 44
Flow and timing	Participant exclusions: none reported Index test to reference standard interval: not described
Comparative
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Unclear
Was a case‐control design avoided?	No
Did the study avoid inappropriate exclusions?	Unclear
Are the included patients and chosen study setting appropriate?	No
Did the study avoid including participants with multiple lesions?	Unclear
		High	High
DOMAIN 2: Index Test Dermoscopy ‐ image‐based
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Yes
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?	Unclear
Was the test applied and interpreted in a clinically applicable manner?	No
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	Yes
Was the test interpretation carried out by an experienced examiner?	Unclear
		Low	High
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	Yes
Were the reference standard results interpreted without knowledge of the results of the index tests?	Unclear
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Unclear
		Low	Unclear
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Unclear
Did all patients receive the same reference standard?	Yes
Were all patients included in the analysis?	Yes
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?
		Unclear

Gilmore 2010.

Study characteristics
Patient sampling	Study design: case series Data collection: retrospective image selection/prospective interpretation Period of data collection: 2003‐2008 Country: Austria Test set derived: NR. Training set: 65 melanomas and 65 dysplastic naevi, test set: 36 melanomas and 33 dysplastic naevi (included in review)
Patient characteristics and setting	Inclusion criteria: atypical melanocytic lesions with polarised dermoscopic images; describes database as a "random, but representative, cohort" but does not describe method of selection Setting: secondary (general dermatology) Prior testing: unclear Setting for prior testing: NR Exclusion criteria: none reported Sample size (participants): number included: NR Sample size (lesions): number included: 199: derivation set n = 130; test set n = 69 Participant characteristics: none reported Lesion characteristics: none reported
Index tests	Dermoscopy: no algorithm Method of diagnosis: dermoscopic images Prior test data: no further information used; described as blinded assessment Diagnostic threshold: NR; subjective impression; excise or not Diagnosis based on: single observer (n = 1) Observer qualifications: dermatologist Experience in practice: not described Experience with dermoscopy: not described; implies high or expert assessment. Conducted by 1 of the co‐authors
Target condition and reference standard(s)	Reference standard: histological diagnosis alone Details: "lesions were excised and examined microscopically by expert dermatopathologists using standard: histopathologic diagnostic criteria" Disease‐positive: 36 = test set and 65 = derivation set; disease‐negative: 33 = test set and 65 = derivation set Target condition (final diagnoses) Melanoma (in situ and invasive, or NR): 36 test set and 65 derivation set Dysplastic naevi: 33 test set and 65 derivation set
Flow and timing	Participant exclusions: none reported Index test to reference standard interval: not described
Comparative
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Unclear
Was a case‐control design avoided?	Yes
Did the study avoid inappropriate exclusions?	Unclear
Are the included patients and chosen study setting appropriate?	No
Did the study avoid including participants with multiple lesions?	Unclear
		Unclear	High
DOMAIN 2: Index Test Dermoscopy ‐ image‐based
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Unclear
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	No
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	No
Was the test interpretation carried out by an experienced examiner?	Yes
		Unclear	High
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	Yes
Were the reference standard results interpreted without knowledge of the results of the index tests?	Unclear
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Yes
		Low	Low
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Unclear
Did all patients receive the same reference standard?	Yes
Were all patients included in the analysis?	Yes
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?
		Unclear

Glud 2009.

Study characteristics
Patient sampling	Study design: case series Data collection: prospective; dermoscopic images assessed remotely from the patient Period of data collection: January‐April 2007 Country: Denmark
Patient characteristics and setting	Inclusion criteria: patients referred for excision biopsy of pigmented lesions where the diagnosis of melanoma could not be excluded on clinical investigation Setting: secondary (other); Dept Plastic Surgery and Burn Unit Prior testing: clinical suspicion of malignancy Setting for prior testing: secondary (not further specified) Exclusion criteria: none reported Sample size (participants): number included: 65 Sample size (lesions): number included: 83 Participant characteristics: median age 47 years (18‐90 years); male = 29; 45% Lesion characteristics: melanoma thickness 0.29 mm‐18 mm
Index tests	Dermoscopy: no algorithm Method of diagnosis: dermoscopic images Prior test data: no further information used Diagnostic threshold: NR; diagnosis of melanoma Diagnosis based on: single observer (n = 1) Observer qualifications: dermatologist Experience in practice: high Experience with dermoscopy: high experience; "dermoscopic images were examined by an experienced dermatologist" Any other detail: the dermoscopic and SIAgraphic images were obtained by SIAscope II (Amon Clinica, Cambridge, UK) and stored using the proprietary Dermetrics software (Astron Clinica).
Target condition and reference standard(s)	Reference standard: histological diagnosis alone Details: following image acquisition "the excision biopsy was performed and an experienced histopathologist examined the tissue". Breslow thickness and Clark level were determined by standard: histopathologic examination. Tumour staging was performed as described by Balch et al according to the 2001 melanoma staging system (Balch 2001). Disease‐positive: 12; disease‐negative: 71 Target condition (final diagnoses) Melanoma (invasive): 7; melanoma (in situ): 5; 1 melanoma metastasis (included as benign) Sebhorrheic keratosis: 1; benign naevus: 57; 'Benign' diagnoses: BD 1, haemangioma 1, LS 2, epidermal naevi 2, DF 6
Flow and timing	Participant exclusions: none reported Index test to reference standard interval: following image acquisition "the excision biopsy was performed"
Comparative
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Yes
Was a case‐control design avoided?	Yes
Did the study avoid inappropriate exclusions?	Yes
Are the included patients and chosen study setting appropriate?	No
Did the study avoid including participants with multiple lesions?	No
		Low	High
DOMAIN 2: Index Test Dermoscopy ‐ image‐based
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Unclear
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	No
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	No
Was the test interpretation carried out by an experienced examiner?	Yes
		Unclear	High
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	Yes
Were the reference standard results interpreted without knowledge of the results of the index tests?	Unclear
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Yes
		Low	Low
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Yes
Did all patients receive the same reference standard?	Yes
Were all patients included in the analysis?	Yes
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?
		Low

Gokdemir 2011.

Study characteristics
Patient sampling	Study design: case series Data collection: NR Period of data collection: 2005‐2009 Country: Turkey
Patient characteristics and setting	Inclusion criteria: patients with melanocytic and non‐melanocytic skin lesions excised due to dermoscopic suspicion of malignancy or dysplasia Setting: secondary (general dermatology) Prior testing: NR Setting for prior testing: unspecified Exclusion criteria: none reported Sample size (participants): number eligible: 1264; number included: 362 Sample size (lesions): number included: 449 Participant characteristics: mean age 40.3 years (+/‐ 1.08), range 1‐89 years; male: 160; 44.2% Lesion characteristics: none reported
Index tests	Dermoscopy: no algorithm Method of diagnosis: unclear; appears to be in‐person diagnosis Prior test data: clinical examination Diagnostic threshold: NR; diagnosis of melanoma Diagnosis based on: unclear (n = NR) Observer qualifications: dermatologist Experience in practice: not described Experience with dermoscopy: high experience; at least 2 years' experience with Molemax II
Target condition and reference standard(s)	Reference standard: histological diagnosis alone; not further described Disease‐positive 13; disease‐negative 433 Target condition (final diagnoses) Melanoma (in situ and invasive, or NR): 13; BCC: 45 Benign: not described
Flow and timing	Participant exclusions: none reported Index test to reference standard interval: NR
Comparative
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Unclear
Was a case‐control design avoided?	Yes
Did the study avoid inappropriate exclusions?	Unclear
Are the included patients and chosen study setting appropriate?	No
Did the study avoid including participants with multiple lesions?	No
		Unclear	High
DOMAIN 2: Index Test Dermoscopy ‐ in‐person
Were the index test results interpreted without knowledge of the results of the reference standard?	Unclear
If a threshold was used, was it pre‐specified?	Unclear
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	Unclear
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	No
Was the test interpretation carried out by an experienced examiner?	Yes
		Unclear	High
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	Yes
Were the reference standard results interpreted without knowledge of the results of the index tests?	Unclear
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Unclear
		Low	Unclear
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Unclear
Did all patients receive the same reference standard?	Yes
Were all patients included in the analysis?	Yes
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?
		Unclear

Grimaldi 2009.

Study characteristics
Patient sampling	Study design: case series Data collection: prospective Period of data collection: October 2005‐March 2006 Country: Italy
Patient characteristics and setting	Inclusion criteria: cutaneous pigmented lesions with digital images forwarded by primary care physicians to a referral centre for confirmation of diagnosis Setting: primary; lesions selected for referral by GPs; accuracy of GP diagnosis assessed Prior testing: NR Setting for prior testing: NR Exclusion criteria: lesions whose removal had been explicitly demanded by the patients for aesthetic reasons, as well as those irritated or subjected to trauma Sample size (participants): number included: 197 Sample size (lesions): number included: 235 Participant characteristics: none reported Lesion characteristics: none reported
Index tests	VI: no algorithm Method of diagnosis: in‐person diagnosis Prior test data: N/A; in‐person diagnosis Other test data: "two‐step judgment (before and after dermoscopy) formulated by the sending physician, who labelled each lesion as ‘benign’ or ‘suspicious for malignancy’" Diagnostic threshold: NR "Each physician was asked to formulate a written first judgment of every lesion before digital acquisition and to re‐evaluate it after dermoscopy" Diagnosis based on: single observer; (n = 13) Observer qualifications: GP; from approximately 250 primary care clinicians attending a conference, 13 volunteered to participate Experience in practice: not clearly described; assumed to be low experience with pigmented lesions Experience in dermoscopy: unclear; classified as 'trained', “simple protocols for diagnosis were made up and given to the participants via e‐learning courses, direct meetings, and involving self assessment procedures” Dermoscopy: ABCD Method of diagnosis: in‐person diagnosis Prior test data: clinical examination and/or case notes Diagnostic threshold: NR; "The evaluation method followed the ABCD rule of dermoscopy" (Nachbar 1994); not fully clear whether this relates to GP in‐person diagnosis or telediagnosis at reference centre, "two‐step judgment (before and after dermoscopy) formulated by the sending physician, who labelled each lesion as ‘benign’ or ‘suspicious for malignancy’." Dermoscopy training: "During the first phase of the study, simple protocols for diagnosis were made up and given to the participants via e‐learning courses, direct meetings, and involving self‐assessment procedures (Pagnanelli 2003)." Length of training: NR Training format: online/in‐person teaching/self‐assessment procedures
Target condition and reference standard(s)	Reference standard: histological diagnosis + follow‐up (reference is expert diagnosis for teledermatology component of study) Histology (not further described): n = 16; disease‐positive: 5; disease‐negative: 11 Clinical follow‐up (6 months) + histology of suspicious lesions: n = 219; disease‐positive: 0; disease‐negative: 208 Target condition (final diagnoses) Melanoma (in situ and invasive, or NR): 5 Other: 230 benign
Flow and timing	Excluded participants: NR Time interval to reference test: NR Time interval between index test(s): NR
Comparative	Blinding between tests: in‐person without and with dermocopy Time interval between index test(s): same day
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Yes
Was a case‐control design avoided?	Yes
Did the study avoid inappropriate exclusions?	Yes
Are the included patients and chosen study setting appropriate?	Yes
Did the study avoid including participants with multiple lesions?	No
		Low	High
DOMAIN 2: Index Test Visual Inspection ‐ in‐person
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Unclear
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	Yes
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	No
Was the test interpretation carried out by an experienced examiner?	No
		Unclear	High
DOMAIN 2: Index Test Dermoscopy ‐ in‐person
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Yes
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	Yes
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	Yes
Was the test interpretation carried out by an experienced examiner?	No
		Low	High
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	No
Were the reference standard results interpreted without knowledge of the results of the index tests?	Unclear
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Unclear
		High	Unclear
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Unclear
Did all patients receive the same reference standard?	No
Were all patients included in the analysis?	Yes
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?	Yes
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?
		High
DOMAIN 5: Comparative
Was each index test result interpreted without knowledge of the results of other index tests or testing strategies?	No
Was the interval between application of the index tests less than one month?	Yes
Were all tests applied and interpreted in a clinically applicable manner?	Yes
		Low	Low

Guitera 2009a (Modena).

Study characteristics
Patient sampling	Study design: case series Data collection: prospective Period of data collection: September 2004‐August 2007 Country: Italy (and Australia ‐ see Guitera 2009b (Sydney))
Patient characteristics and setting	Inclusion criteria: lesions suspicious of melanoma based on dermatoscopic diagnostic criteria or lesion change; included only a random sample of 50% of benign naevi observed during time period Setting: secondary (general dermatology); Department of Dermatology, University of Modena, Italy Prior testing: clinical and/or dermatoscopic suspicion/changes on digital monitoring Setting for prior testing: secondary (general dermatology) Exclusion criteria: location/site of lesion lesions on soles/palms excluded; lentigo maligna excluded; lesions used in previous assessments or RCM model development Sample size (participants): number included: 195 Sample size (lesions): number included: 195 Participant characteristics: median age: 42 (7‐88 years); IQR 32y, 59y; male: 51.3% Lesion characteristics: pigmented: 92%; 8% amelanotic lesions or those with tan, light grey, or pale blue pigment only). Median thickness 0.65 mm (IQR 0.23mm, 0.98mm)
Index tests	Dermoscopy: pattern analysis Method of diagnosis: in‐person diagnosis; at time of first consultation and prior to RCM Prior test data: clinical examination Diagnostic threshold: NR Diagnosis based on: single observer (n = 1) Observer qualifications: dermatologist; not clearly reported, but is study co‐author Experience in practice: high experience Experience with dermoscopy: high experience; described as Modena expert based in Dermatology Dept Other detail: hand‐held dermoscope (Delta 10, Heine, Herrsching, Germany).
Target condition and reference standard(s)	Reference standard: histological diagnosis alone (not further described) Disease‐positive: 79; disease‐negative: 116 Target condition (final diagnoses) Melanoma (invasive): 61; melanoma (in situ): 18 Benign naevus: 116 (78 compound, 0 dermal, 16 junctional, and 22 Spitz)
Flow and timing	Excluded participants: only 50% of imaged naevi were included (randomly selected from the image database prior to analysis) to reduce the MM/naevus ratio Time interval to reference test: consecutive; imaged prior to biopsy
Comparative
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Yes
Was a case‐control design avoided?	Yes
Did the study avoid inappropriate exclusions?	No
Are the included patients and chosen study setting appropriate?	No
Did the study avoid including participants with multiple lesions?	Yes
		High	High
DOMAIN 2: Index Test Dermoscopy ‐ in‐person
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Yes
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	Yes
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	No
Was the test interpretation carried out by an experienced examiner?	Yes
		Low	High
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	Yes
Were the reference standard results interpreted without knowledge of the results of the index tests?	Unclear
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Unclear
		Low	Unclear
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Yes
Did all patients receive the same reference standard?	Yes
Were all patients included in the analysis?	No
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?	No
		High

Guitera 2009b (Sydney).

Study characteristics
Patient sampling	Study design: case series Data collection: prospective; dermoscopic images assessed remotely from the patient Period of data collection: September 2004‐August 2007 Country: Australia (and Italy ‐ see Guitera 2009a (Modena))
Patient characteristics and setting	Inclusion criteria: lesions suspicious of melanoma based on dermatoscopic diagnostic criteria or lesion change Setting: specialist clinic; Sydney Melanoma Diagnostic Centre, Australia Prior testing: clinical and/or dermatoscopic suspicion/changes on digital monitoring Setting for prior testing: specialist clinic Exclusion criteria: location/site of lesion lesions on soles/palms excluded. Lentigo maligna excluded; lesions used in previous assessments or RCM model development Sample size (participants): number included: 131 Sample size (lesions): number eligible 156 number included: 131 Participant characteristics: median age: 52 (19‐90years); IQR 40, 63y; male: 58.8% Lesion characteristics: pigmented: 84%; 16% amelanotic lesions or those with tan, light grey, or pale blue pigment only). Median thickness 0.40 mm (IQR 0, 0.84 mm)
Index tests	Dermoscopy: pattern analysis Method of diagnosis: dermoscopic images Prior test data: lesion site and age available to observer; dermoscopy diagnosis of Sydney lesions was made retrospectively on the images in a random order, blinded to RCM and pathological diagnosis but not to information of site and age, by a Modena expert (GP) using pattern analysis (Pehamberger 1993) Diagnostic threshold: NR Diagnosis based on: single observer (n = 1) Observer qualifications: assume dermatologist; described as Modena expert based in Dermatology Dept Experience in practice: high experience Experience with dermoscopy: high experience Other detail: Sydney; high‐resolution digital oil immersion dermoscopy camera (Sentry, Polartechnics Ltd, Sydney, NSW, Australia)
Target condition and reference standard(s)	Reference standard: histological diagnosis alone (no further details) Disease‐positive: 44; disease‐negative: 87 Target condition (final diagnoses) Melanoma (invasive): 26; melanoma (in situ): 16 Benign naevus: 87 (49 compound, 9 dermal, 26 junctional, and 3 Spitz)
Flow and timing	Excluded participants: 25 lesions out of 156 were rejected for poor‐quality dermoscopy image, blinded to the diagnostician Time interval to reference test: imaged prior to biopsy Time interval between index test(s): N/A
Comparative
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Yes
Was a case‐control design avoided?	Yes
Did the study avoid inappropriate exclusions?	No
Are the included patients and chosen study setting appropriate?	No
Did the study avoid including participants with multiple lesions?	Yes
		High	High
DOMAIN 2: Index Test Dermoscopy ‐ image‐based
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Yes
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	No
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	No
Was the test interpretation carried out by an experienced examiner?	Yes
		Low	High
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	Yes
Were the reference standard results interpreted without knowledge of the results of the index tests?	Unclear
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Unclear
		Low	Unclear
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Yes
Did all patients receive the same reference standard?	Yes
Were all patients included in the analysis?	No
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?
		High

Haenssle 2010a (FV).

Study characteristics
Patient sampling	Study design: case series Data collection: prospective Period of data collection: 1998‐2008 Country: Germany
Patient characteristics and setting	Inclusion criteria: participants at increased risk for melanoma: >50 common and/or ≤ 3 atypical naevi; atypical mole syndrome; or familial atypical mole and multiple melanoma syndrome Setting: secondary (dermatology) Prior testing: all identified as high risk Setting for prior testing: NR Exclusion criteria: patients showing melanoma development on pre‐existing pigmented lesions during the following 12 months after the analysed time frame Sample size (participants): 688 Sample size (lesions): 11,137 Participant characteristics: mean age 42 (range NR). 60% male. Group 1 (50 common and/or ≤ atypical naevi) 67%; Group 2 (atypical mole syndrome) 31.8%; Group 3 (familial atypical mole and multiple melanoma syndrome) 1.2%. Personal history of melanoma (29.2%); family history of melanoma (13.1%); high number (> 50) of naevi (56.4%) Lesion characteristics: NR
Index tests	Dermoscopy: 7PCL Method of diagnosis: in person Prior test data: also considered lesional history (e.g., increase in size, itching, scaling, change in colour, intermittent bleeding), and the ugly duckling sign (Grob 1998) and 'moles‐breed‐true' concept (Scope 2006). Lesions scoring < 3 on 7PCL were excised if these other factors were present at first visit. Lesions scoring < 3 with defined clinical or dermatoscopic criteria of atypia (e.g. asymmetry in shape, irregular margin, variegated colour, prominent pigment network) (Ascierto 2000) were marked on digital overview images and electronically stored by using two digital dermatoscopy systems for follow‐up Diagnostic threshold: ≥ 3 Diagnosis based on: consensus of 2 Observer qualifications: dermatology residents (n = 13); supervised by experienced dermatologist Experience in practice: NR Experience with dermoscopy: high; formally trained in dermoscopy
Target condition and reference standard(s)	Reference standard: histology or follow‐up (every 3, 6, or 12 months) Target condition (final diagnoses) Invasive melanoma 77; melanoma in situ 50; BCC 2 Benign naevi 1047; SN 16; SK 12; other benign 9935 (not excised)
Flow and timing	Excluded participants: none reported Time interval to reference test: consecutive
Comparative
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Yes
Was a case‐control design avoided?	Yes
Did the study avoid inappropriate exclusions?	No
Are the included patients and chosen study setting appropriate?	No
Did the study avoid including participants with multiple lesions?	No
		High	High
DOMAIN 2: Index Test Dermoscopy ‐ in‐person
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Yes
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	No
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	Yes
Was the test interpretation carried out by an experienced examiner?	Yes
		Low	High
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	No
Were the reference standard results interpreted without knowledge of the results of the index tests?	Unclear
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Unclear
		High	Unclear
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Yes
Did all patients receive the same reference standard?	No
Were all patients included in the analysis?	Yes
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?
		High

Haenssle 2010b (FU).

Study characteristics
Patient sampling	Study design: case series Data collection: prospective Period of data collection: 1998‐2008 Country: Germany
Patient characteristics and setting	Inclusion criteria: participants at increased risk for melanoma: > 50 common and/or ≤ 3 atypical naevi; atypical mole syndrome; or familial atypical mole and multiple melanoma syndrome Setting: secondary (dermatology) Prior testing: all identified as high risk Setting for prior testing: NR Exclusion criteria: patients showing melanoma development on pre‐existing pigmented lesions during the following 12 months after the analysed time frame Sample size (participants): 688 Sample size (lesions): 11,137 Participant characteristics: mean age 42 (range NR). 60% male. Mean age 42 (range NR). 60% male. Group 1 (50 common and/or ≤ atypical naevi) 67%; Group 2 (atypical mole syndrome) 31.8%; Group 3 (familial atypical mole and multiple melanoma syndrome) 1.2%. Personal history of melanoma (29.2%); family history of melanoma (13.1%); high number (> 50) of naevi (56.4%) Lesion characteristics: NR
Index tests	Dermoscopy: 7PCL Method of diagnosis: in person Prior test data: also considered lesional history (e.g. increase in size, itching, scaling, change in colour, intermittent bleeding), and the ugly duckling sign (Grob 1998) and 'moles‐breed‐true' concept (Scope 2006). Lesions scoring < 3 on 7PCL were excised if these other factors were present at first visit. Lesions scoring < 3 with defined clinical or dermatoscopic criteria of atypia (e.g. asymmetry in shape, irregular margin, variegated colour, prominent pigment network) (Ascierto 2000) were marked on digital overview images and electronically stored by using 2 digital dermatoscopy systems for follow‐up Diagnostic threshold: ≥ 3 Diagnosis based on: consensus of 2 Observer qualifications: dermatology residents (n = 13); supervised by experienced dermatologist Experience in practice: NR Experience with dermoscopy: high; formally trained in dermoscopy
Target condition and reference standard(s)	Reference standard: histology or follow‐up (every 3, 6, or 12 months); mean follow‐up 44.28 (range 2‐123) months Target condition (final diagnoses) Invasive melanoma 77; melanoma in situ 50; BCC 2 Benign naevi 1047; SN 16; SK 12; other benign 9935 (not excised)
Flow and timing	Excluded participants: none reported Time interval to reference test: consecutive
Comparative
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Yes
Was a case‐control design avoided?	Yes
Did the study avoid inappropriate exclusions?	No
Are the included patients and chosen study setting appropriate?	No
Did the study avoid including participants with multiple lesions?	No
		High	High
DOMAIN 2: Index Test Dermoscopy ‐ in‐person
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Yes
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	No
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	Yes
Was the test interpretation carried out by an experienced examiner?	Yes
		Low	High
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	No
Were the reference standard results interpreted without knowledge of the results of the index tests?	Unclear
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Unclear
		High	Unclear
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Yes
Did all patients receive the same reference standard?	No
Were all patients included in the analysis?	Yes
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?
		High

Hauschild 2014.

Study characteristics
Patient sampling	Study design: RCT of diagnosis based on clinical/dermoscopic images versus same + MelaFind, with observers randomised between arms. Lesions selected on a case‐control type basis with cases and controls sampled from a previous study (Monheit 2011). Data collection: retrospective image selection/prospective interpretation Period of data collection: NR Country: USA
Patient characteristics and setting	Inclusion criteria: subset of PSLs evaluated in Monheit 2011; melanoma and non‐melanoma randomly selected Setting: mixed secondary/private; lesions sampled from Monheit 2011 trial: "Seven clinical sites with 23 investigators participated in this trial. Three sites were academic institutions (University of Pittsburgh, Duke University, and Northwestern University), and 4 sites were dermatologic practices highly experienced in managing PLs." Prior testing: selected for excision (no further detail) Setting for prior testing: NR Exclusion criteria: ulcerated or non‐pigmented lesions, or located on excluded anatomic sites. Lesions with prebiopsy clinical diagnoses of melanoma were excluded from Monheit 2011 Sample size (participants): number included: 130 Sample size (lesions): number eligible: 1632 lesions in Monheit trial; number included: 130 Participant characteristics: none reported Lesion characteristics: head/neck 23%; trunk 41.5%; upper limbs/shoulder 20%; lower limbs/hip 16.2%. Median thickness (melanomas) 0.39 mm (range 0.12‐1.2 mm)
Index tests	Dermoscopy: no algorithm Method of diagnosis: clinical photographs and dermoscopic images (Arm 1 and Arm 3 of trial; Arm 2 included MelaFind images) Prior test data: clinical images (overview and close up); + 24 items regarding patient demographics and risk factors for melanoma such as: personal or family history of melanoma, number of atypical naevi, Fitzpatrick skin type, number of severe sunburns before and after age 20, etc Diagnostic threshold: biopsy decision Diagnosis based on: average. Board‐certified dermatologists who were members of a public dermatology list volunteered to participate in the trial. Selection was made on a first‐come basis with randomisation between 2 study arms until at least 65 dermatologists participated in each Arm. Of the 227 dermatologists registered, 211 completed at least 78 cases and therefore were considered eligible. Finally included 101/108 dermatologists in Arm 1 and 101/108 dermatologists in Arm 2 (MelaFind). A 3rd arm included 9/12 PSL experts "prospectively identified by the Principal Investigator based on field standing prior to participant recruitment" Observer qualifications: dermatologists Experience in practice: high; all board‐certified, in Arm 1 > 90% had > 10 years' experience in practice; Arm 3 consisted of PSL experts Experience with dermoscopy: high; for Arm 1 all except 6 were trained in dermoscopy use and 80/101 always or almost always used dermoscopy for PSLs; Arm 3 consisted of PSL experts
Target condition and reference standard(s)	Reference standard: histological diagnosis alone Details: from Monheit 2011, the electronic case record included details of the "prebiopsy diagnoses (without dermoscopy and, if available, with dermoscopy) by the examining dermatologists", "if the dermatologic diagnosis was not melanoma, the reason for the biopsy was selected from the following: non‐melanoma skin cancer, patient’s concern, patient’s discomfort, cosmetic, or, if dermoscopic evaluation was used, clinical concern. A histologic specimen with the standard: hematoxylin‐eosin staining was provided for each lesion." "Histologic slides for each lesion ... were evaluated by 2 independent dermatopathologists. In cases of significant discordance, histologic slides were evaluated independently by a third dermatopathologist. When 1 dermatopathologist diagnosed melanoma and 2 others diagnosed a benign lesion, histologic slides were sent again to the dermatopathologist who diagnosed melanoma for a blind re‐review." Disease‐positive: 65; disease‐negative: 65 Target condition (final diagnoses) Melanoma invasive: 36; melanoma in situ: 29 'Benign' diagnoses: 65
Flow and timing	Participant exclusions: none reported Index test to reference standard interval: appears consecutive
Comparative
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Yes
Was a case‐control design avoided?	No
Did the study avoid inappropriate exclusions?	Yes
Are the included patients and chosen study setting appropriate?	No
Did the study avoid including participants with multiple lesions?	Unclear
		High	High
DOMAIN 2: Index Test Dermoscopy ‐ image‐based
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Unclear
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	Unclear
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	No
Was the test interpretation carried out by an experienced examiner?	Yes
		Unclear	High
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	Yes
Were the reference standard results interpreted without knowledge of the results of the index tests?	Unclear
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Yes
		Low	Low
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Yes
Did all patients receive the same reference standard?	Yes
Were all patients included in the analysis?	Yes
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?
		Low

Kittler 1998.

Study characteristics
Patient sampling	Study design: unclear Data collection: retrospective image selection/prospective interpretation Period of data collection: NR Country: Austria
Patient characteristics and setting	Inclusion criteria: PSL images 'selected' by PSL experts from pigmented lesion image database on the basis of quality of the photograph and the difficulty of diagnosis; all "melanomas selected provided only subtle ELM features as clues to the malignancy of the lesion and were difficult to differentiate from benign PSLs". Setting: secondary (not further specified) Prior testing: dermatoscopic suspicion in all cases Setting for prior testing: secondary (general dermatology); selected from PSL database Exclusion criteria: none reported Sample size (participants): NR Sample size (lesions): number included: 50 Participant characteristics: none reported Lesion characteristics: median Breslow thickness of the MMs: 0.7 mm (IQR 0.5‐0.95 mm)
Index tests	Dermoscopy: no algorithm Method of diagnosis: dermoscopic images; both photographic slides and compressed digital images assessed to determine whether compressed images are sufficiently informative for diagnosis; 2x2 based on digital images used for primary analysis Prior test data: no further information used. Images viewed in 2 sessions; in each session 25 slides and 25 digital images were viewed Diagnostic threshold: clinical diagnosis; rated as definitely or probably melanoma; unclear whether 2x2 based on 'definite' only as test positive or definite/probable combined Diagnosis based on: single observer; n = 8 readers, reported separately Observer qualifications: dermatologist Experience in practice: not described Experience with dermoscopy: not described; described as 'pre‐trained in ELM'
Target condition and reference standard(s)	Reference standard: histological diagnosis alone (no further details) Disease‐positive: 23; disease‐negative: 27 Target condition (final diagnoses) Melanoma (invasive or in situ): 23 Sebhorrheic keratosis: 1; atypical naevus 17; common naevus 9
Flow and timing	Participant exclusions: poor‐quality images excluded; "selected' from pigmented lesion image database on the basis of quality of the photograph" Index test to reference standard interval: NR
Comparative
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Unclear
Was a case‐control design avoided?	Unclear
Did the study avoid inappropriate exclusions?	Unclear
Are the included patients and chosen study setting appropriate?	No
Did the study avoid including participants with multiple lesions?	Unclear
		Unclear	High
DOMAIN 2: Index Test Dermoscopy ‐ image‐based
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Unclear
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	No
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	No
Was the test interpretation carried out by an experienced examiner?	Unclear
		Unclear	High
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	Yes
Were the reference standard results interpreted without knowledge of the results of the index tests?	Unclear
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Unclear
		Low	Unclear
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Unclear
Did all patients receive the same reference standard?	Yes
Were all patients included in the analysis?	Yes
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?
		Unclear

Kittler 1999.

Study characteristics
Patient sampling	Study design: case series Data collection: prospective Period of data collection: November 1996‐November 1997 Country: Austria (from authors' institution)
Patient characteristics and setting	Inclusion criteria: PSLs < 1 cm in diameter, consecutively excised Setting: secondary (general dermatology). From authors' institution Prior testing: selected for excision (no further detail) Setting for prior testing: NR Exclusion criteria: lesion size ≥ 1 cm Sample size (participants): number included: 352 Sample size (lesions): number included: 373 Participant characteristics: mean age 52 (SD 17 years); male: 49% Lesion characteristics: median thickness 0.65 mm (range, 0.2‐2 mm)
Index tests	Dermoscopy: ABCD; ABCDE (developed in this study) Method of diagnosis: in‐person diagnosis Prior test data: clinical examination Diagnostic threshold: range of numerical thresholds evaluated 'Standard' ABCD applied as previously described by Stolz 1994a and Nachbar 1994. Sensitivities reported for a range of specificities but cut‐offs NR (author communication suggested a threshold of > 4.75 was used but not clear which sensitivity/specificity pair this relates to); randomly selected dataset at 75% specificity for inclusion in primary analysis. 'Enhanced' ABCD‐E algorithm accounts for participant report of changes in the lesion within the previous year. The overall score was calculated by adding 1.2 to the standard ABCD score for changing lesions and subtracting 0.8 from the standard ABCD score for non‐changing lesions according to the results of a multivariate analysis. ABCDE results reported at cut‐offs ranging from 1.30‐7.35 Diagnosis based on: unclear; appears to be in clinic diagnoses (n = NR) Observer qualifications: NR; likely dermatologists Experience in practice: not described Experience with dermoscopy: not described
Target condition and reference standard(s)	Reference standard: histological diagnosis alone Details: "After excision all lesions were subjected to standard: histopathologic examination. The histologic diagnosis of an atypical naevus was based on the following criteria: cellular atypia, lentiginous hyperplasia of the epidermis, fibroplasia, bridging of rete ridges, suprabasal melanocytes, junctional nest disarray Disease‐positive: 73; disease‐negative: 283 Target condition (final diagnoses) Melanoma (invasive): 55 (51 superficial spreading, 4 nodular, 15 lentigo maligna, 3 otherwise non‐classified melanomas); melanoma (in situ): 18 Sebhorrheic keratosis: 4; 126 (35.4%) common naevi, 113 (31.7%) atypical (dysplastic) naevi, 3 (0.8%) congenital naevi, 13 (3.7%) pigmented SN, 7 (0%) blue naevi, 2 (0.6%) combined naevi, 14 (3.9%) solar lentigines, 1 DF
Flow and timing	Participant exclusions: non‐melanocytic lesions (n = 17; including angiomatous tumours, pigmented SK, dFs, and pigmented BCCs) easily distinguished by standard ELM criteria and pattern analysis Index test to reference standard interval: not described
Comparative
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Yes
Was a case‐control design avoided?	Yes
Did the study avoid inappropriate exclusions?	No
Are the included patients and chosen study setting appropriate?	No
Did the study avoid including participants with multiple lesions?	No
		High	High
DOMAIN 2: Index Test Dermoscopy ‐ in‐person
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	No
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	Unclear
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	Yes
Was the test interpretation carried out by an experienced examiner?	Unclear
		High	Unclear
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	Yes
Were the reference standard results interpreted without knowledge of the results of the index tests?	Unclear
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Unclear
		Low	Unclear
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Unclear
Did all patients receive the same reference standard?	Yes
Were all patients included in the analysis?	No
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?
		High

Kittler 2001.

Study characteristics
Patient sampling	Study design: case‐control Data collection: retrospective image selection/prospective interpretation Period of data collection: NR Country: NR
Patient characteristics and setting	Inclusion criteria: images of naevi from patients with multiple atypical naevi undergoing digital dermoscopy follow‐up. All melanomas were excised due to changes on follow‐up; benign melanocytic skin lesions included were taken at random from the participants with melanoma + other randomly selected patients with multiple atypical naevi. Setting: secondary (assumed); states "a database" Authors' Inst: Dept Dermatology, University of Vienna Prior testing: all undergoing follow‐up Setting for prior testing: NR Exclusion criteria: none reported Sample size (participants): number eligible: NR; number included: 20 Sample size (lesions): number eligible: NR; number included: 80 Participant characteristics: none reported Lesion characteristics: none reported
Index tests	Dermoscopy: no algorithm Method of diagnosis: dermoscopic images Prior test data: unclear Diagnostic threshold: data extracted for excise decision; data also presented for 3 option response of excise/follow‐up or no intervention Diagnosis based on: average (n = 24); 3 groups were recruited according to experience but 2x2 could be extracted only for overall average result, individual group results presented only graphically Observer qualifications: dermatologist Experience in practice: NR Experience with dermoscopy: mixed; group 1 (n = 9) had basic dermoscopy experience with no formal training, group 2 (n = 10) had dermoscopy training but only basic experience with digital dermoscopy, and group 3 included experienced dermatologists trained in dermoscopy and using digital dermoscopy routinely to follow‐up melanocytic lesions.
Target condition and reference standard(s)	Reference standard: histological diagnosis + follow‐up Details: all lesions were excised (n = 20; including all 10 melanomas) or had at least 2 years of follow‐up with no morphologic changes during multiple examinations (n = 60; all benign) Target condition (final diagnoses) Melanoma (invasive): 5, melanoma (in situ): 5 Benign melanocytic lesions: 70
Flow and timing	Excluded participants: none reported Time interval to reference test: NR for histology; clinical follow‐up lasted up to 2 years
Comparative
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Unclear
Was a case‐control design avoided?	No
Did the study avoid inappropriate exclusions?	Unclear
Are the included patients and chosen study setting appropriate?	No
Did the study avoid including participants with multiple lesions?	No
		High	High
DOMAIN 2: Index Test Dermoscopy ‐ image‐based
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Unclear
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	No
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	No
Was the test interpretation carried out by an experienced examiner?	Unclear
		Unclear	High
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	No
Were the reference standard results interpreted without knowledge of the results of the index tests?	Unclear
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Unclear
		High	Unclear
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Unclear
Did all patients receive the same reference standard?	No
Were all patients included in the analysis?	Yes
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?	Yes
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?
		High

Krahn 1998.

Study characteristics
Patient sampling	Study design: case series Data collection: prospective Period of data collection: NR Country: Germany
Patient characteristics and setting	Inclusion criteria: excised PSLs Setting: secondary (general dermatology) Prior testing: NR Setting for prior testing: NR Exclusion criteria: none reported Sample size (participants): number included: 80 Sample size (lesions): number included: 80 Participant characteristics: none reported Lesion characteristics range in thickness (melanomas) 0.18‐1.9 mm; 29/39 < 0.76 mm; 7/39 0.76‐1.5 mm; 3/39 > 1.5 mm
Index tests	VI: no algorithm reported Method of diagnosis: in‐person diagnosis Prior test data: unclear Diagnostic threshold: NR; no details Diagnosis based on: single observer (n = 1) Observer qualifications: NR; likely dermatologist Experience in practice: not described Experience with dermoscopy: not described Dermoscopy Method of diagnosis: in‐person diagnosis Prior test data: unclear Diagnostic threshold: NR; no details Test observers as described for VI (above)
Target condition and reference standard(s)	Reference standard: histological diagnosis alone including histometrics Disease‐positive: 39; disease‐negative: 41 Target condition (final diagnoses) Melanoma (invasive): 39 (SSM, lentigo MM, nodular M) Benign naevus: 37 common naevus; 3 dysplastic naevus, 1 SN
Flow and timing	Excluded participants: none reported Time interval to reference test: NR Time interval between index test(s): NR
Comparative	Blinding between tests: in‐person diagnosis without and then with dermoscopy Time interval between index test(s): same day; at time of face‐to‐face consultation
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Unclear
Was a case‐control design avoided?	Yes
Did the study avoid inappropriate exclusions?	Unclear
Are the included patients and chosen study setting appropriate?	No
Did the study avoid including participants with multiple lesions?	Yes
		Unclear	High
DOMAIN 2: Index Test Visual Inspection ‐ in‐person
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Unclear
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	Yes
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	No
Was the test interpretation carried out by an experienced examiner?	Unclear
		Unclear	High
DOMAIN 2: Index Test Dermoscopy ‐ in‐person
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Unclear
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	Yes
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	No
Was the test interpretation carried out by an experienced examiner?	Unclear
		Unclear	High
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	Yes
Were the reference standard results interpreted without knowledge of the results of the index tests?	Unclear
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Yes
		Low	Low
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Unclear
Did all patients receive the same reference standard?	Yes
Were all patients included in the analysis?	Yes
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?
		Unclear
DOMAIN 5: Comparative
Was each index test result interpreted without knowledge of the results of other index tests or testing strategies?	No
Was the interval between application of the index tests less than one month?	Yes
Were all tests applied and interpreted in a clinically applicable manner?	Yes
		Low	Low

Kreusch 1992.

Study characteristics
Patient sampling	Study design: case series Data collection: retrospective image selection/prospective interpretation Period of data collection: NR; 1.5‐year period Country: Germany
Patient characteristics and setting	Inclusion criteria: pigmented lesions suspected to be malignant melanoma with adequate photo‐ documentation and histology results Setting: secondary (dermatology) Prior testing: NR Setting for prior testing: NR Exclusion criteria: non‐melanocytic lesions Sample size (participants): total 856; NR for final sample Sample size (lesions): 265 melanocytic/1506 lesions included (317 excised and 52 non‐melanocytic lesions excluded) Participant characteristics: NR Lesion characteristics: NR
Index tests	Dermoscopy: algorithm from Kreusch 1991 Method of diagnosis: image‐based Prior test data: none; slides labelled only with patient code and lesion localisation Diagnostic threshold: ≥ 9; scored diameter > 5 mm; border irregularity; loss of surface's microstructure; scaling/erosion/ulcer; capillaries (each 1 point); multicomponent architecture; greyish colour (each 3 points) melanophages (6 points); pseudopods (10 points); regression (10 points) Diagnosis based on: single observer Observer qualifications: dermatologist (assumed) (n = 1; ‘experienced') (also presents results for inexperienced student – data not included) Experience in practice: ‘experienced' Experience with dermoscopy: ‘experienced'
Target condition and reference standard(s)	Reference standard: histology Target condition (final diagnoses) Invasive melanoma 96; benign naevi 169
Flow and timing	Excluded participants: 52 non‐melanocytic lesions excluded from second step evaluation Time interval to reference test: NR
Comparative
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Yes
Was a case‐control design avoided?	Yes
Did the study avoid inappropriate exclusions?	Unclear
Are the included patients and chosen study setting appropriate?	No
Did the study avoid including participants with multiple lesions?	Unclear
		Unclear	High
DOMAIN 2: Index Test Dermoscopy ‐ image‐based
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Yes
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	No
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	Yes
Was the test interpretation carried out by an experienced examiner?	Yes
		Low	High
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	Unclear
Were the reference standard results interpreted without knowledge of the results of the index tests?	Unclear
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Unclear
		Unclear	Unclear
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Unclear
Did all patients receive the same reference standard?	No
Were all patients included in the analysis?	Yes
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?
		High

Langley 2007.

Study characteristics
Patient sampling	Study design: case series Data collection: prospective Period of data collection: February 2002‐May 2005 Country: Canada
Patient characteristics and setting	Inclusion criteria: patients with suspicious pigmented lesions scheduled for biopsy due to clinical suspicion of malignancy determined by clinical appearance or a history of change in the lesion Setting: specialist unit (skin cancer/PLC); division of Dermatology Pigmented Lesion Clinic and the Plastic Surgery Clinics Prior testing: clinical suspicion of malignancy Setting for prior testing: specialist unit (skin cancer/PLC) Exclusion criteria: non‐pigmented; physically inaccessible lesion site; previous diagnostic biopsy of the lesion Sample size (participants): number eligible: 127; number included: 125 Sample size (lesions): number eligible: 127; number included: 125 Participant characteristics: mean age 44.2 years, range 16‐84 years Lesion characteristics: median thickness 0.62 mm, range 0.20 mm‐7.92 mm
Index tests	Dermoscopy: pattern analysis Method of diagnosis: in‐person diagnosis Prior test data: clinical examination and/or case notes Diagnostic threshold: pattern analysis; diagnosis of melanoma Diagnosis based on: single observer (n = 1) Observer qualifications: NR likely dermatologist; "Clinical, dermoscopic and confocal examinations were conducted sequentially by a single reviewer" and a diagnosis recorded after each. Experience in practice: not described Experience with dermoscopy: not described
Target condition and reference standard(s)	Reference standard: histological diagnosis alone Details: "When CSLM imaging was complete, the lesions were removed by excisional biopsy. A definitive diagnosis was made by a dermatopathologist with conventional hematoxylin‐eosin stained histopathological sections." Disease‐positive: 37; disease‐negative: 88 Target condition (final diagnoses) Melanoma (invasive): 22; melanoma (in situ): 15 Benign naevus: 88
Flow and timing	Participant exclusions: 2 participants were excluded from the database due to technical difficulties with the imaging. Index test to reference standard interval: when CSLM imaging was complete, the lesions were removed by excisional biopsy
Comparative
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Unclear
Was a case‐control design avoided?	Yes
Did the study avoid inappropriate exclusions?	Yes
Are the included patients and chosen study setting appropriate?	No
Did the study avoid including participants with multiple lesions?	Yes
		Low	High
DOMAIN 2: Index Test Dermoscopy ‐ in‐person
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Yes
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	Yes
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	No
Was the test interpretation carried out by an experienced examiner?	Unclear
		Low	High
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	Yes
Were the reference standard results interpreted without knowledge of the results of the index tests?	Unclear
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Yes
		Low	Low
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Yes
Did all patients receive the same reference standard?	Yes
Were all patients included in the analysis?	No
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?
		High

Lorentzen 1999a.

Study characteristics
Patient sampling	Study design: case series Data collection: prospective Period of data collection: 1994‐1997 Country: Denmark
Patient characteristics and setting	Inclusion criteria: patients with lesions suspicious for CMM referred to outpatients clinic; only excised included Setting: NR Prior testing: clinical suspicion of malignancy without dermatoscopic suspicion Setting for prior testing: NR Exclusion criteria: poor‐quality index test image (considered under flow/timing) Sample size (participants): number eligible: 242; number included: 232 Sample size (lesions): number eligible: 242; number included: 232* Participant characteristics: none reported Lesion characteristics: none reported *NB not all cases were assessed by all observers; 2x2 are based on presented sensitivity and specificity estimates for full dataset of lesions; "the dermatoscopy experts assessed almost all cases (98 ± 100%), whereas the non‐expert group completed fewer assessments, from 76%‐98%
Index tests	VI: no algorithm Method of diagnosis: clinical photographs Prior test data: no further information used; no option to change clinical diagnosis after viewing dermoscopic image Other test data: dermoscopic images presented to observer subsequent to diagnosis using clinical images alone; clinical images presented before dermoscopic images Diagnostic threshold: NR; clinical diagnosis Diagnosis based on: average; n = 9 Observer qualifications: dermatologist Experience in practice: high; moderate; mixed (average reported); 4 'experienced dermatologists' (4‐5 years' daily experience) & 5 'non‐expert dermatology residents' (1‐2 years' interest and formal training in dermatoscopy) Experience with index test: high; moderate; mixed Dermoscopy: no algorithm Method of diagnosis: dermoscopic images Prior test data: clinical image presented first Diagnostic threshold: clinical diagnosis; "observers were familiar with both the ABCD‐rule of dermatoscopy proposed by Stolz et al. (Stolz 1994b) and Kenet et al's risk‐stratifying algorithm of pigment network features of dermatoscopy (Kenet 1994). The observers were not constrained by either of the rules. The ABCD scores were not used to obtain the diagnoses. Rather a pattern recognition process was intended." Dermoscopy training: described as "formal training" Training format: non experts had undergone prior training in dermoscopy (not documented)
Target condition and reference standard(s)	Reference standard: histological diagnosis alone Details: a co‐author from Dept of Pathology "re‐evaluated all cases to confirm the pathology diagnosis, which was used as the gold standard in this study." Disease‐positive: 65; disease‐negative: 167 Target condition (final diagnoses) Melanoma (invasive): 49 'malignant melanoma' BCC: 16 Sebhorrheic keratosis: 12; benign naevus: 137 (pigmented naevi = 116; blue naevi = 16; atypical naevi = 5); other: 18 (SN, BD, sarcoid, naevus spilus, hemangioma, and others)
Flow and timing	Excluded participants: 10 cases were "considered unfit for evaluation" due to poor‐quality image Reference interval: "biopsy specimens...were obtained after the clinical and dermatoscopic photographs had been performed"
Comparative	Blinding between tests: Each observer first recorded the clinical diagnosis and then the dermatoscopic diagnosis on an entry form. Time interval between index test(s): same day; at time of face‐to‐face consultation
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Unclear
Was a case‐control design avoided?	Yes
Did the study avoid inappropriate exclusions?	Yes
Are the included patients and chosen study setting appropriate?	No
Did the study avoid including participants with multiple lesions?	Unclear
		Unclear	High
DOMAIN 2: Index Test Visual inspection ‐ image‐based
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Unclear
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	No
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	No
Was the test interpretation carried out by an experienced examiner?	Yes
		Unclear	High
DOMAIN 2: Index Test Dermoscopy ‐ image‐based
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Yes
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	No
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	Yes
Was the test interpretation carried out by an experienced examiner?	Yes
		Low	High
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	Yes
Were the reference standard results interpreted without knowledge of the results of the index tests?	Unclear
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Unclear
		Low	Unclear
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Yes
Did all patients receive the same reference standard?	Yes
Were all patients included in the analysis?	No
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?
		High
DOMAIN 5: Comparative
Was each index test result interpreted without knowledge of the results of other index tests or testing strategies?	No
Was the interval between application of the index tests less than one month?	Yes
Were all tests applied and interpreted in a clinically applicable manner?	No
		Low	High

Lorentzen 2000.

Study characteristics
Patient sampling	Study design: case series Data collection: retrospective image selection/prospective interpretation Period of data collection: 1995‐1999 Country: not clear; authors from Denmark and USA
Patient characteristics and setting	Inclusion criteria: PSLs from patients consecutively referred to the skin cancer outpatient clinic with available clinical photographs, dermatophotographs and a subsequent excision biopsy were included Setting: specialist unit (skin cancer/PLC) Prior testing: selected for excision (no further detail) Setting for prior testing: secondary (general dermatology) Exclusion criteria: none reported Sample size (participants): number included: 258 Sample size (lesions): number included: 258 Participant characteristics: none reported Lesion characteristics: none reported
Index tests	Dermoscopy ABCD; Kenet Risk Stratification Method of diagnosis: dermoscopic images; "Slides were projected to an 80 x 120 cm screen in a darkened room. Based on time studies in the outpatient clinic, each patient case was shown for approximately 3 min... additional time was allowed if any needed it." Prior test data: clinical photographs also projected Diagnostic threshold: ABCD ‐ 'possible' MM: > 4.75; 'probable' MM: > 5.45 Risk stratification method: 'possible' MM: stratum 1 or 2; 'probable' MM: stratum 1 only (1: probable CMM: pseudopods; radial streaming; heterogeneity of pigment network with thick dark extensions at the edge; blue‐grey areas, white scar‐like areas and presence of pigment network: possible CMM: marked irregular network with irregular pigment confluence) Diagnosis based on: single observer (n = 3; performed independently) Observer qualifications: dermatologist Experience in practice: high; senior dermatologists, "Three senior dermatologists with > 5 years daily experience in clinical use of dermatoscopy and familiar with (both) dermatoscopic (algorithms)" Experience with dermoscopy: high; > 5 years each
Target condition and reference standard(s)	Reference standard: histological diagnosis alone Details: lesions underwent haematoxylin and eosin staining, as well as immunohistochemical staining using HMB‐45 (human melanoma black) and S100 to identify melanocytic lesions. Breslow depth and Clark level were determined. All cases were assessed by an experienced dermatopathologist. Disease‐positive: 64; disease‐negative: 194 Target condition (final diagnoses) Melanoma (invasive): 64 CMM BCC: 25 Sebhorrheic keratosis: 14 Benign naevus: 135; dysplastic naevus 3; other: 11 blue naevi, 1 pigmented SN, + one each of were angioma, haemorraghia, papilloma and DF
Flow and timing	Participant exclusions: none reported Time interval to reference test: appears consecutive; "Only patients having taken clinical photographs, dermatophotographs and a subsequent excision biopsy were included"
Comparative
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Yes
Was a case‐control design avoided?	Yes
Did the study avoid inappropriate exclusions?	Unclear
Are the included patients and chosen study setting appropriate?	No
Did the study avoid including participants with multiple lesions?	Yes
		Unclear	High
DOMAIN 2: Index Test Dermoscopy ‐ image‐based
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Yes
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?	Unclear
Was the test applied and interpreted in a clinically applicable manner?	No
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	Yes
Was the test interpretation carried out by an experienced examiner?	Yes
		Low	High
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	Yes
Were the reference standard results interpreted without knowledge of the results of the index tests?	Unclear
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Yes
		Low	Low
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Yes
Did all patients receive the same reference standard?	Yes
Were all patients included in the analysis?	Yes
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?	Yes
		Low

Lorentzen 2008.

Study characteristics
Patient sampling	Study design: case series Data collection: unclear Period of data collection: NR Country: Denmark
Patient characteristics and setting	Inclusion criteria: patients referred to the specialist naevus clinic for lesion excision Setting: specialist unit (skin cancer/PLC) Prior testing: NR Setting for prior testing: NR Exclusion criteria: not specified Sample size (participants): number eligible: 120; number included: 119 Sample size (lesions): number included: 119 Participant characteristics: none reported Lesion characteristics: none reported
Index tests	Dermoscopy: mixed/no algorithm; describes using "the risk stratification and pattern analysis procedure" as described by Kenet 2001 and Lorentzen 2000. Method of diagnosis: dermoscopic images; compared accuracy using standard dermoscopy images (Dermaphot) and images obtained using a globe magnifier. Slides were randomised and evaluated on 2 different occasions with 3‐week intervals Prior test data: no further information used Diagnostic threshold: observer correct diagnosis of each lesion type Diagnosis based on: unclear (assumed average) (n = NR) Observer qualifications: dermatologist Experience in practice: high; "dermatologists who have performed dermatoscopy for 5–10 years, published scientific papers on dermatoscopy and carried out pre‐ and post‐specialist training in dermatoscopy" Experience with dermoscopy: high
Target condition and reference standard(s)	Reference standard: histological diagnosis alone Details: used haematoxylin‐eosin staining as well as immunohistochemistry using S‐100 protein and HMB‐45 (human melanoma black) on suspect melanoma lesions Disease‐positive: 24; disease‐negative: 95 Target condition (final diagnoses) Melanoma (invasive): 24 BCC: 13 Mild/moderate dysplasia: 2; SK: 9; hemangioma: 2; naevus pigmentosus: 69
Flow and timing	Excluded participants: 1 DF excluded Time interval to reference test: not described
Comparative
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Unclear
Was a case‐control design avoided?	Yes
Did the study avoid inappropriate exclusions?	Unclear
Are the included patients and chosen study setting appropriate?	No
Did the study avoid including participants with multiple lesions?	Yes
		Unclear	High
DOMAIN 2: Index Test Dermoscopy ‐ image‐based
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Yes
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	No
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	No
Was the test interpretation carried out by an experienced examiner?	Yes
		Low	High
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	Yes
Were the reference standard results interpreted without knowledge of the results of the index tests?	Unclear
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Unclear
		Low	Unclear
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Unclear
Did all patients receive the same reference standard?	Yes
Were all patients included in the analysis?	No
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?
		High

Malvehy 2014.

Study characteristics
Patient sampling	Study design: case series Data collection: prospective; dermoscopic images assessed remotely from the patient Period of data collection: March 2010‐November 2011 Country: conducted at 5 US and 17 European investigational sites (Austria, Germany, Hungary, Spain, Sweden, and UK)
Patient characteristics and setting	Inclusion criteria: all patients with skin lesions selected for total excision to rule out melanoma; dermatologists were encouraged to enrol a mix of lesions with an even distribution of low‐, medium‐ and high‐risk lesions Setting: secondary; authors' institutions primarily listed as Dept Dermatology with one, "Dermatology Clinical Research Center" Prior testing: selected for excision Exclusion criteria: lesions < 2 mm or > 20 mm and those located: on acral skin, e.g. sole or palm; areas of scars, crusts, psoriasis, eczema or similar skin conditions; hair‐covered areas, e.g. scalp, beards, moustaches or whiskers; genitalia; in an area that had been previously biopsied or subjected to any kind of surgical intervention or trauma; mucosal surfaces; with foreign matter, e.g. tattoo or splinter; acute sunburn; or skin surface not measurable, e.g. lesion on a stalk; surface not accessible, e.g. inside ears, under nails or not intact (measurement area) Sample size (participants): number eligible: 1951; number included: 1611 for Naevisense and NR for VI and dermoscopy Sample size (lesions): number eligible: 2416; number included: 1943 for Naevisense and 1701 for VI and dermoscopy Participant characteristics: for Naevisense sample: median age: 48 years (range 18‐91); male 47.5%; 97.5% of white ethnicity. Fitzpatrick skin types: I (7.3%); II (48.6%); III (37%); IV (9.8%); V (1.4%); VI (0.1%) Lesion characteristics: median Breslow thickness of 0.57 mm (153 invasive melanomas)
Index tests	Dermoscopy; ABCD; 7PCL; revised 7PCL; overall diagnosis (methods describe evaluation of the clinical ABCD rule but results not presented in Table) Method of diagnosis: image‐based; "A photograph and dermoscopic image of each included lesion was taken before and after Naevisense measurements" Prior test data available: clinical and dermoscopic images presented together; observers were blinded to Naevisense result Diagnostic threshold: ABCD > 4.75 and > 5.45; 7PCL and revised 7PCL NR, referenced to Argenziano 1998; overall diagnosis based on grading (0‐10) on a visual classification board with a fixed cut‐off at 4 Diagnosis based on: unclear; (n = 3) Observer qualifications: dermatologists; "images were reviewed by three dermatologists with 2–5 years of experience in dermoscopy assessment. The option to reach out to additional experienced dermoscopists in difficult cases was allowed" Experience in practice: not described Experience with dermoscopy: high; 2‐5 years
Target condition and reference standard(s)	Type of reference standard: histological diagnosis alone Details: lesions were excised and underwent usual histopathology at investigational site. A further histopathological evaluation was undertaken for study purposes by a panel of 3 experienced histopathologists who evaluated each lesion independently; blinded from the investigational site’s original histopathology diagnosis. If they agreed, the diagnosis was considered as the histopathological gold standard; if there was significant disagreement regarding malignancy the slides were submitted to 2 additional experts whose diagnosis was then chosen as the histopathological gold standard if they reached agreement. In case of disagreement by the 2 additional reviewers, the corresponding lesion was excluded from the efficacy analysis. Disease‐positive: 238 for VI/dermoscopy; disease‐negative: 1440 Target condition (final diagnoses) For VI/dermoscopy sample, 238 melanomas including 112 in situ Breakdon of non‐diseased not provided for VI/dermoscopy sample For Naevisense sample (includes additional 242 lesions: 153 invasive melanomas, 112 melanoma in situ, 48 BCC, 1 invasive SCC; 1 Merkel cell carcinoma 157 severely dysplastic, 988 mild‐moderate dysplasia, 352 benign naevi, 5 SN, 51 SK, 6 SCC in situ; 8 AK; 61 other
Flow and timing	Participant exclusions: 473 excluded from Naevisense analysis; all reasons listed; primary reason was investigator oversight or the inability to render a final histopathological diagnosis; 74 exclusions were device‐related (60 with inadequate reference measurement quality and 14 to device failure). A further 242 were excluded from VI/dermoscopy analysis due to image quality (12% of VI/dermoscopy sample) Index test to reference standard interval: appears consecutive; prospective recruitment with imaging and then "eligible and evaluable lesions were excised and subjected to the investigational site’s histopathology evaluation and managed accordingly." "A postprocedure follow‐up either by a telephone call or at a participant’s visit to the investigational site was conducted at 7 +/‐ 3 days after the Naevisense evaluation, at which time the patient was evaluated for any adverse events."
Comparative	Interval between index tests Consecutive; "A photograph and dermoscopic image of each included lesion was taken before and after Nevisense measurements to document evaluation according to the protocol.”
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Yes
Was a case‐control design avoided?	Yes
Did the study avoid inappropriate exclusions?	No
Are the included patients and chosen study setting appropriate?	No
Did the study avoid including participants with multiple lesions?	No
		High	High
DOMAIN 2: Index Test Dermoscopy ‐ image‐based
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Yes
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?	No
Was the test applied and interpreted in a clinically applicable manner?	No
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	Yes
Was the test interpretation carried out by an experienced examiner?	Yes
		Low	High
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	Yes
Were the reference standard results interpreted without knowledge of the results of the index tests?	Unclear
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Yes
		Low	Low
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Yes
Did all patients receive the same reference standard?	Yes
Were all patients included in the analysis?	No
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?	Yes
		High

Menzies 1996.

Study characteristics
Patient sampling	Study design: unclear. Abstract describes including a random sample of excised lesions from a larger database Data collection: retrospective image selection/prospective interpretation Period of data collection: NR Country: Australia Test set derived: NR; describes 'division' into a training set and a test set
Patient characteristics and setting	Inclusion criteria: PSLs from the Sydney Melanoma Unit with dermoscopic images and histological diagnoses; melanomas and randomly selected clinically atypical non‐melanoma lesions were included Setting: specialist unit (skin cancer/PLC) Prior testing: selected for excision Setting for prior testing: specialist unit (skin cancer/PLC) Exclusion criteria: unequivocal non‐melanoma excluded Sample size (participants): number included: NR Sample size (lesions): number included: 385 Participant characteristics: none reported Lesion characteristics: none reported
Index tests	Dermoscopy: Menzies criteria Method of diagnosis: dermoscopic images Prior test data: no further information used Diagnostic threshold: presence of 2 negative features and at least 1 positive feature. Negative features: point and axial symmetry of pigmentation or presence of only a single colour. Positive features of melanoma: multiple (5‐6) colours; blue‐white veil; multiple brown dots; multiple blue/grey; peripheral black dots or globules; a broadened network; pseudopods; radial streaming; scar‐like Diagnosis based on: unclear (n = NR) Observer qualifications: NR; likely dermatologists Experience in practice: not described Experience with dermoscopy: not described
Target condition and reference standard(s)	Reference standard: histological diagnosis alone (not further described) Disease‐positive: 107; disease‐negative: 278 Target condition (final diagnoses) Melanoma (invasive): 107; BCC: 18 Ephelis/lentigo 17; SK: 23; benign acquired naevi ‐ 58; dysplastic naevi ‐ 105; blue naevi 11; SN 6; spindle cell naevus 2; DF 2; hemangioma 13; solar keratosis 9; other 14
Flow and timing	Participant exclusions: none reported Index test to reference standard interval: not described
Comparative
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Yes
Was a case‐control design avoided?	Unclear
Did the study avoid inappropriate exclusions?	Yes
Are the included patients and chosen study setting appropriate?	No
Did the study avoid including participants with multiple lesions?	Unclear
		Unclear	High
DOMAIN 2: Index Test Dermoscopy ‐ image‐based
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Yes
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	No
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	Yes
Was the test interpretation carried out by an experienced examiner?	Unclear
		Low	High
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	Yes
Were the reference standard results interpreted without knowledge of the results of the index tests?	Unclear
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Unclear
		Low	Unclear
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Unclear
Did all patients receive the same reference standard?	Yes
Were all patients included in the analysis?	Yes
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?
		Unclear

Menzies 2005.

Study characteristics
Patient sampling	Study design: case series Data collection: retrospective image selection/prospective interpretation Period of data collection: June 1998‐September 2003 Country: multicentre (Australia, Germany, USA)
Patient characteristics and setting	Inclusion criteria: all melanocytic lesions from the independent test set taken at the Sydney Melanoma Unit that had clinical and dermoscopy photographic images; lesions imaged prior to excision due to clinical suspicion of malignancy or because of short‐term digital monitoring (study was part of a larger multicentre study of SolarScan) Setting: specialist unit Prior testing: clinical suspicion of malignancy or requirement for short‐term digital monitoring Setting for prior testing: specialist unit Exclusion criteria: awkwardly situated lesions (e.g. eyelids, some parts of the pinna, some genital sites, and perianal and mucosal surfaces); acral lesions; non‐pigmented pure amelanotic lesions (based on dermoscopy imaging); ulcerated lesions, or diagnosed as pigmented BCC, pigmented BD, or SCC Sample size (participants): number included: NR Sample size (lesions): number included for dermoscopy review: 78 (for full SolarScan study ‐ number eligible: 2430/number included: 1644 training; 786 test set) Participant characteristics: none reported Lesion characteristics: none reported
Index tests	Dermoscopy: no algorithm Method of diagnosis: dermoscopic images Prior test data: clinical photographs and patient histories (including details of age, sex, and lesion site; and a recorded history of whether the lesion had, within the past 2 years, bled without being scratched, changed in colour or pattern, or increased in size) Diagnostic threshold: data can be extracted at 2 thresholds: correct diagnosis of melanoma (in situ or invasive) and excise decision; no details on lesion characteristics used Diagnosis based on: average according to qualification level (n = 13) Observer qualifications: GP 3; dermatology registrar 3; dermatologists 4; + 3 international dermoscopy experts who headed pigmented lesion clinics Experience in practice: not described Experience with dermoscopy: expert/high/moderate/low
Target condition and reference standard(s)	Reference standard: histological diagnosis + other (full sample n = 2430) Histology: 71% of full SolarScan study sample including training and test set (n = 1725) Clinical follow‐up + histology of suspicious lesions; length of follow‐up: 3 months. 26% of full SolarScan study sample (n = 632) Expert opinion. 3% of full SolarScan study sample were non‐melanocytic pigmented lesions that were diagnosed clinically but not excised (n = 73) Target condition (final diagnoses) All numbers are for Sydney Melanoma Unit test sample lesions only (n = 78) Melanoma (invasive): 5; melanoma (in situ): 6; lentigo maligna: 2 Benign melanocytic lesions: 65
Flow and timing	Participant exclusions: poor‐quality index test image as exclusion criterion; lesions outside the field of view (24 x 18 mm), contamination of calibration surfaces, or excess artifacts (hair, air bubbles, or movement artifacts) Index test to reference standard interval: not described
Comparative
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Yes
Was a case‐control design avoided?	Yes
Did the study avoid inappropriate exclusions?	No
Are the included patients and chosen study setting appropriate?	No
Did the study avoid including participants with multiple lesions?	Unclear
		High	High
DOMAIN 2: Index Test Dermoscopy ‐ image‐based
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Unclear
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	No
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	No
Was the test interpretation carried out by an experienced examiner?	Yes
		Unclear	High
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	No
Were the reference standard results interpreted without knowledge of the results of the index tests?	Unclear
Expert opinion (with no histological confirmation) was not used as a reference standard	No
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Unclear
		High	High
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Unclear
Did all patients receive the same reference standard?	No
Were all patients included in the analysis?	No
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?
		High

Menzies 2008.

Study characteristics
Patient sampling	Study design: case series? Data collection: retrospective image selection/prospective interpretation Period of data collection: NR Country: multicentre
Patient characteristics and setting	Inclusion criteria: dermoscopic amelanotic (with no melanin pigmentation) or hypomelanotic (a melanin pigmentation area of < 25% of the total surface area or slightly pigmented but with no dark brown, deep blue, or black pigmentation) lesions. All melanomas included, and a random selection of melanocytic and non‐melanocytic lesions on a non‐melanoma to melanoma ratio of 3:1 Setting: multicentre Prior testing: NR Setting for prior testing: NR Exclusion criteria: lesions were excluded because of poor image quality or because they did not fit within any of the defined pigmentation categories Sample size (participants): NR Sample size (lesions): 497 Participant characteristics: NR Lesion characteristics: NR
Index tests	Dermoscopy: 7PCL; Menzies; 3PCL (new algorithm for distinguishing melanoma from non‐melanoma and any malignant from benign lesions was also developed on 80% of sample and tested on 20% but numbers disease‐positive and ‐negative for the test set were NR to allow 2x2 to be estimated.) Method of diagnosis: image‐based Prior test data: NR Diagnostic threshold: ≥ 3; Menzies standard threshold; ≥ 2 Diagnosis based on: single observer Observer qualifications: dermatologist (assumed) (n = 12); clinicians experienced in dermoscopic evaluation scored 99 individual morphological features in approximately equal sample sizes Experience in practice: NR Experience with dermoscopy: high
Target condition and reference standard(s)	Reference standard: histology and follow‐up (numbers NR; some naevi included that showed no changes following consecutive digital monitoring) Target condition (final diagnoses) Invasive melanoma 91; melanoma in situ 14; BCC 126; cSCC 4 Benign naevi 159; SN 11; SK 22; DF 17; BD 7; keratoacanthoma 1; AK 8; other 37
Flow and timing	Excluded participants: none reported Time interval to reference test: NR
Comparative
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Yes
Was a case‐control design avoided?	Yes
Did the study avoid inappropriate exclusions?	No
Are the included patients and chosen study setting appropriate?	No
Did the study avoid including participants with multiple lesions?	Unclear
		High	High
DOMAIN 2: Index Test Dermoscopy ‐ image‐based
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Yes
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?	No
Was the test applied and interpreted in a clinically applicable manner?	No
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	Yes
Was the test interpretation carried out by an experienced examiner?	Yes
		Low	High
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	Unclear
Were the reference standard results interpreted without knowledge of the results of the index tests?	Unclear
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Unclear
		Unclear	Unclear
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Unclear
Did all patients receive the same reference standard?	No
Were all patients included in the analysis?	Yes
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?
		High

Menzies 2009.

Study characteristics
Patient sampling	Study design: case series Data collection: prospective Period of data collection: December 2005‐August 2006 Country: Australia
Patient characteristics and setting	Inclusion criteria: pigmented lesions which, after routine naked‐eye examination by the GP, would have been biopsied or referred, i.e. a suspicious pigmented lesion. GPs were recruited from practices with at least 3 clinicians; excluded if they already used dermoscopy or sequential digital dermoscopy imaging (SDDI) in their routine practice. Setting: primary Prior testing: clinical suspicion of malignancy without dermatoscopic suspicion Setting for prior testing: primary Exclusion criteria: none reported Sample size (participants): NR Sample size (lesions): number included: 374 Participant characteristics: none reported Lesion characteristics: none reported
Index tests	VI: no algorithm Method of diagnosis: in‐person diagnosis Prior test data: N/A; in‐person diagnosis Diagnostic threshold: NR. Initial diagnosis recorded along with confidence of diagnosis (scale 1‐10; 1 not at all confident and 10 extremely confident), certainty of melanoma (scale 0%‐100%; 0 definitely not melanoma and 100 definitely melanoma) and management (biopsy, referral) Diagnosis based on: single observer (n = 63; 102 GPs initially recruited; 74 (75%) completed the educational intervention and online assessment; 63 GPs from 19 practices finally participated) Observer qualifications: GP Experience in practice: not described Experience with dermoscopy: not fully described; classified as 'trained'. GPs must have each excised or referred ≥ 10 PSL in previous 12‐month period; excluded if dermoscopy or SDDI already used in routine practice. Dermoscopy: no algorithm Method of diagnosis: in‐person diagnosis Prior test data: clinical examination and/or case notes Diagnostic threshold: NR. After clinical exam and dermoscopy GP recorded the site of the lesion and the initial diagnosis, confidence of diagnosis, certainty of melanoma and management (as for VI above). Approach to dermoscopy interpretation not further reported; 2x2 can be constructed for decision to excise or to excise or monitor. Triage management options included: biopsy due to clinician concern; biopsy due to patient concern; referral due to clinician concern; referral due to patient concern; short‐term SDDI; and patient to return if changes occur Test observers as described for VI (above) Dermoscopy training: online textbook in dermoscopic diagnosis and the use of SDDI, a CD‐ROM tutorial showing examples of changed and unchanged monitored lesions; a 2‐h workshop on the use of the diagnostic devices and recruitment procedures. Assessment through online pre‐ and post‐education intervention test of 245 lesions not seen in the textbook or the CD‐ROM. Before formal patient recruitment began, GPs assessed at least one pretrial lesion to determine the quality of imaging with the SDDI instrument and undertake completion of trial paperwork. GPs were allowed to practise using the dermoscopy device during this pretrial phase. The pretrial phase of education and run‐in period occurred from May 2005‐January 2006. Length of training: self‐learning + 2‐h workshop Post‐training experience: 6‐12 months Training format: online; CD‐ROM: workshop
Target condition and reference standard(s)	Reference standard: histological diagnosis + other (study author confirmed that all melanoma had histological diagnosis and > 50% of benign had histology or follow‐up) Histology: described as conducted to standard practice and not necessarily blinded to the GP’s diagnosis. Total excised or referred: 163. Immediate excision/referral: 110. Excision/referral after SDDI: 48. Excision/examination after patient self‐referral: 5 Disease‐positive: 37; disease‐negative: total of 126 benign or unknown were 'excised OR referred' Clinical follow‐up + histology of suspicious lesions: short‐term digital monitoring (SDDI) available as an option for lesions considered not to be melanoma but that were still considered suspicious; follow‐up imaging occurred initially at 3 months with any morphological changes to result in biopsy or referral; some lesions continued SDDI for a further 3 months; length of follow‐up: 3‐6 months. Number participants: initially recommended for SDDI 192; SDDI continued for further 3 months: 6; underwent SDDI only (no excision) 146 Disease‐positive: 15 (SDDI then histologically confirmed); disease‐negative: 176 benign (incl 1 missed in situ melanoma); 4 unknown Expert opinion: GPs could refer for specialist opinion or lesions could undergo dermoscopy telemedicine (images reviewed by an expert in dermoscopy and SDDI). Dermoscopy telemedicine was blinded to the GP’s diagnosis. Observe for change group, i.e. discharged after dermoscopy: 72 (plus a proportion of those in excise/refer group will have had expert diagnosis alone but details not given) Disease‐positive: 0; disease‐negative: 71 benign; 1 unknown Target condition (final diagnoses) Melanoma (invasive): 33; melanoma (in situ): 1 BCC: 6 2 BD; 323 benign; 9 unknown
Flow and timing	Excluded participants: 9 lesions with unknown diagnoses, + BCC and BD excluded from some analyses Time interval to reference test: NR; histopathological and specialist examination occurred according to standard practice
Comparative
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Yes
Was a case‐control design avoided?	Yes
Did the study avoid inappropriate exclusions?	Yes
Are the included patients and chosen study setting appropriate?	Yes
Did the study avoid including participants with multiple lesions?	Unclear
		Low	Unclear
DOMAIN 2: Index Test Visual Inspection ‐ in‐person
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Unclear
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	Yes
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	No
Was the test interpretation carried out by an experienced examiner?	Unclear
		Unclear	High
DOMAIN 2: Index Test Dermoscopy ‐ in‐person
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Yes
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	Yes
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	No
Was the test interpretation carried out by an experienced examiner?	No
		Low	High
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	No
Were the reference standard results interpreted without knowledge of the results of the index tests?	No
Expert opinion (with no histological confirmation) was not used as a reference standard	No
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Unclear
		High	High
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Unclear
Did all patients receive the same reference standard?	No
Were all patients included in the analysis?	No
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?	Yes
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?
		High

Menzies 2013.

Study characteristics
Patient sampling	Study design: case series Data collection: retrospective image selection/prospective interpretation Period of data collection: NR Country: multicentre (photographic libraries at various institutions; obtained from members of the IDS from 5 continents)
Patient characteristics and setting	Inclusion criteria: nodular malignant melanoma (an invasive melanoma without an in situ (junctional) component beyond 3 rete ridges of the dermal invasive component) and a random selection of non‐nodular invasive primary melanoma, benign nodular melanocytic lesions, and nodular non‐melanocytic lesions at a ratio of NM to other subgroups of 1:2. Nodular benign melanocytic lesions and nodular non‐melanocytic lesions were identified by the clinical appearance of a solitary nodule and confirmed using dermoscopic examination. Setting: mixed Prior testing: NR Setting for prior testing: NR Exclusion criteria: excluded if the image quality was poor Sample size (participants): NR Sample size (lesions): 467 Participant characteristics: excluded if the image quality was poor Lesion characteristics: pigmented 314/467; 67%
Index tests	Dermoscopy: ABCD; Menzies; CASH; 7PCL; 3PCL; Menzies algorithm for amelanotic lesions (Menzies 2008) Method of diagnosis: image‐based Prior test data: NR Diagnostic threshold: > 5.45; standard Menzies; ≥ 8; standard 7PCL; standard 3PCL; ≥ 1 and ≥ 0 Diagnosis based on: single Observer qualifications: dermatologist (n = 2; experience NR). 12 scorers blinded to the lesion diagnosis scored 99 individual features in each lesion of approximately equal sample sizes, as previously described Experience in practice: NR Experience with dermoscopy: high
Target condition and reference standard(s)	Reference standard: histology or follow‐up (‘some’ benign melanocytic naevi showed no change over time compared with baseline photographs). Target condition (final diagnoses) Invasive melanoma 217 (including 83 nodular) Benign naevi 115
Flow and timing	Excluded participants: none reported Time interval to reference test: NR
Comparative
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Yes
Was a case‐control design avoided?	Yes
Did the study avoid inappropriate exclusions?	No
Are the included patients and chosen study setting appropriate?	No
Did the study avoid including participants with multiple lesions?	Unclear
		High	High
DOMAIN 2: Index Test Dermoscopy ‐ image‐based
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Yes
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?	No
Was the test applied and interpreted in a clinically applicable manner?	No
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	Yes
Was the test interpretation carried out by an experienced examiner?	Yes
		Low	High
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	Unclear
Were the reference standard results interpreted without knowledge of the results of the index tests?	Unclear
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Unclear
		Unclear	Unclear
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Unclear
Did all patients receive the same reference standard?	No
Were all patients included in the analysis?	Yes
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?
		High

Morales Callaghan 2008.

Study characteristics
Patient sampling	Study design: case series Data collection: prospective Period of data collection: 1 January 2005‐31 December 2005 Country: Spain
Patient characteristics and setting	Inclusion criteria: randomly selected melanocytic lesions; melanocytic on both clinical and dermoscopic criteria Setting: secondary (general dermatology) Prior testing: dermatoscopic suspicion in all cases Setting for prior testing: NR Exclusion criteria: palms, soles, mucous membranes of face, under nails; non‐melanocytic appearance Sample size (participants): number included: 166 Sample size (lesions): number included: 200 Participant characteristics: mean age 33.7 years (SD 14.5), range 8‐84 years; male: 64 (38.6%); Fitzpatrick phototype II (44%); type III (41.5%) Lesion characteristics: macular component = 181 (90.5%), papular component = 125 (65%) both = 106 (53%), either one or other = 94 (47%). Asymmetrical 144 (72%). Irregular borders 154 (77%). 4 colours in 40 (20%), 3 colours in 96 (48%), 2 colours in 57 (28.5%), 1 colour in 1 (0.5%). History of bleeding 7 (3.5%). Changes reported by participant 154 (77%). Lesion site: trunk 155 (77.5%), including the back in 106 (53%). Lesion size: mean long axis diameter 7.9 mm (SD 8.6 mm), mean short axis diameter 5.1 mm (SD 5 mm)
Index tests	VI: no algorithm Method of diagnosis: in‐person diagnosis Prior test data: clinical examination and/or case notes Other test data: appears that dermoscopy was undertaken by same clinician(s) subsequent to clinical evaluation; clinical history was constructed following a standardised protocol and a presumptive clinical diagnosis recorded. Each lesion was then photographed and immediately afterwards examined using a manual dermatoscope Diagnostic threshold: NR; presumptive clinical diagnosis Diagnosis based on: consensus (n = 2) Observer qualifications: dermatologist Experience in practice: not clearly described; assumed to be high, “both dermatologists had experience in dermoscopy.” Experience with dermoscopy: not clearly described; assumed to be high, “both dermatologists had experience in dermoscopy.” Dermoscopy: pattern analysis Method of diagnosis: in‐person diagnosis Prior test data: clinical examination and/or case notes Diagnostic threshold: NR; diagnosed "on the basis of predominant dermoscopic pattern(s) using the pattern analysis algorithm" Test observers as described for VI (above)
Target condition and reference standard(s)	Reference standard: histological diagnosis alone Details: lesions described using terminology proposed by US National Institutes of Health Disease‐positive: 6/6 lesions; disease‐negative: 194/194 lesions (assuming the 9 'other' diagnosis lesions were not malignant), or 185/185 (removing the 9 'other' diagnosis lesions from dataset) Target condition (final diagnoses) Melanoma (in situ and invasive, or NR): 6 (3%) Other: atypical mole (104), common mole (70), congenital naevus (6), blue naevus (3), Spitz/Reed naevus (1), spilus naevus (1), others (unclear whether benign or malignant) (9)
Flow and timing	Exclusions: none reported Time interval to reference test: "Samples for histologic analysis were taken immediately after clinical and dermoscopic examination"
Comparative	Blinding between tests: in‐person, without and then with dermoscopy Time interval between index test(s): same day
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Yes
Was a case‐control design avoided?	Yes
Did the study avoid inappropriate exclusions?	Yes
Are the included patients and chosen study setting appropriate?	No
Did the study avoid including participants with multiple lesions?	No
		Low	High
DOMAIN 2: Index Test Visual Inspection ‐ in‐person
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Unclear
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	No
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	No
Was the test interpretation carried out by an experienced examiner?	Yes
		Unclear	High
DOMAIN 2: Index Test Dermoscopy ‐ in‐person
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Yes
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	No
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	No
Was the test interpretation carried out by an experienced examiner?	Yes
		Low	High
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	Yes
Were the reference standard results interpreted without knowledge of the results of the index tests?	Unclear
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Unclear
		Low	Unclear
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Yes
Did all patients receive the same reference standard?	Yes
Were all patients included in the analysis?	Yes
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?
		Low
DOMAIN 5: Comparative
Was each index test result interpreted without knowledge of the results of other index tests or testing strategies?	No
Was the interval between application of the index tests less than one month?	Yes
Were all tests applied and interpreted in a clinically applicable manner?	No
		Low	High

Nachbar 1994.

Study characteristics
Patient sampling	Study design: case series Data collection: prospective Period of data collection: November 1991‐July 1992 Country: NR (authors' institutions Germany and USA)
Patient characteristics and setting	Inclusion criteria: pigmented melanocytic skin lesions consecutively excised Setting: secondary (general dermatology) Prior testing: selected for excision (no further detail) Setting for prior testing: NR Exclusion criteria: unequivocal appearance/diagnosis criteria used to exclude non‐melanocytic described in detail in Table 1 Sample size (participants): NR Sample size (lesions): number included: 194 Participant characteristics: none reported Lesion characteristics: thickness, 35/69 MM ≤ 0.75 mm (50.7%)
Index tests	Dermoscopy: ABCD Method of diagnosis: in‐person diagnosis Prior test data: clinical examination and/or case notes Diagnostic threshold: > 5.45 (determined based on retrospective analysis of the data) For the calculation of ABCD score the criteria of asymmetry (A), abrupt cutoff of the pigment pattern at the border (B), different colours (C), and different structural components (D) were assessed to yield a semiquantitative score (all described in detail). "The results of the retrospective study showed that melanocytic pigmented skin lesions could be differentiated into two diagnostic groups as follows: melanocytic naevi (MN) if the final score was less than 5.45 and MM if the score was higher than 5.45. Retrospective analysis showed an early melanoma could not be completely excluded in all lesions with an ABCD score between 4.75 and 5.45.Therefore these lesions were excised. All lesions were examined by two independent dermatopathologists." Diagnosis based on: unclear (n = NR) Observer qualifications: NR; presumably dermatologists; "colleagues in our department" Experience in practice: high experience or ‘Expert’ Experience with dermoscopy: high experience /‘Expert’ users Study also presents 2x2 data for VI; excluded from review as clinicians 'mostly' also used dermoscope for diagnosis. From text: "In comparing the clinical with the dermatoscopic diagnosis with the ABCD rule it must be noted that all our colleagues in this department referring patients for the study were experienced and in most cases used the dermatoscope without applying the new ABCD rule. Thus clinical diagnosis in our study was expected to be already biased by the dermatoscopic feature and therefore to be more accurate than by the naked eye"
Target condition and reference standard(s)	Reference standard: histological diagnosis alone 194 (not further described) Disease‐positive: 69; disease‐negative: 125 Target condition (final diagnoses) Melanoma (in situ and invasive, or NR): 69 BCC: 3 Sebhorrheic keratosis: 19 'Benign' diagnoses: 103 melanocytic naevus
Flow and timing	Time interval to reference test: NR Time interval between index test(s): appears consecutive
Comparative
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Yes
Was a case‐control design avoided?	Yes
Did the study avoid inappropriate exclusions?	Yes
Are the included patients and chosen study setting appropriate?	No
Did the study avoid including participants with multiple lesions?	Unclear
		Low	High
DOMAIN 2: Index Test Dermoscopy ‐ in‐person
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	No
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	No
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	Yes
Was the test interpretation carried out by an experienced examiner?	Yes
		High	High
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	Yes
Were the reference standard results interpreted without knowledge of the results of the index tests?	Unclear
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Yes
		Low	Low
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Unclear
Did all patients receive the same reference standard?	Yes
Were all patients included in the analysis?	Yes
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?
		Unclear

Nilles 1994.

Study characteristics
Patient sampling	Study design: case series Data collection: retrospective image selection/prospective interpretation Period of data collection: 1989‐1991 Country: Germany Derivation of test set: images collected 1989‐1990 were used to develop a new algorithm; lesions investigated in 1991 were used for model validation (latter data included in review)
Patient characteristics and setting	Inclusion criteria: melanocytic skin lesions that underwent excision Setting: secondary (general dermatology) Prior testing: selected for excision (no further detail) Setting for prior testing: NR Exclusion criteria: non‐melanocytic appearance Sample size (participants): number included: 260 (1989‐1990 group); NR for 1991 group Sample size (lesions): number included: 260 (1989‐1990 group); 209 for 1991 group Participant characteristics: none reported Lesion characteristics none reported
Index tests	Dermoscopy: new algorithm Method of diagnosis: for training set dermoscopic images were projected onto a screen; method NR for test set (assumed same procedures followed) Prior test data: no further information used Diagnostic threshold: significance of '8 clues of malignancy' (Braun‐Falco 1990) were investigated in data collected 1989‐1990. A subset of relevant components were identified and evaluated on the test set of lesions (appears to be presence of any one considered test positive): asymmetrical pigment distribution, > 3 colours, asymmetrical depigmentation, black pigment, sharp pigment border and atypical radial streaming Diagnosis based on: single observer (n = 1) Observer qualifications: NR, likely dermatologist ("one of the authors") Experience in practice: not described Experience with dermoscopy: not described Derivation aspect: the 8 clues of malignancy were graded from 0 (absent) to 3 (distinct) on the test set of lesions (including asymmetrical pigment distribution, > 3 colours, black‐brown pigment, dark brown pigment, prominent pigment network, asymmetrical depigmentation, peripheral stripes, sharp pigment border and atypical radial streaming). Stepwise logistical regression used to select the variables that resulted in the best model for identification of melanoma
Target condition and reference standard(s)	Reference standard: histological diagnosis alone (not further described) Disease‐positive: 41 in test set; disease‐negative: 168 in test set Target condition (final diagnoses) Full breakdown reported only for training set; for test set: Melanoma (invasive): 41 Benign naevus: 168
Flow and timing	Participant exclusions: none reported Index test to reference standard interval: not described
Comparative
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Unclear
Was a case‐control design avoided?	Yes
Did the study avoid inappropriate exclusions?	Unclear
Are the included patients and chosen study setting appropriate?	No
Did the study avoid including participants with multiple lesions?	Unclear
		Unclear	High
DOMAIN 2: Index Test Dermoscopy ‐ image‐based
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Yes
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	No
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	Unclear
Was the test interpretation carried out by an experienced examiner?	Unclear
		Low	High
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	Yes
Were the reference standard results interpreted without knowledge of the results of the index tests?	Unclear
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Unclear
		Low	Unclear
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Unclear
Did all patients receive the same reference standard?	Yes
Were all patients included in the analysis?	Yes
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?
		Unclear

Pagnanelli 2003.

Study characteristics
Patient sampling	Study design: unclear; likely a case‐control type selection process Data collection: retrospective image selection/prospective interpretation (dermoscopy training study) Period of data collection: NR Country: Italy
Patient characteristics and setting	Inclusion criteria: images of PSLs from the training set of the Consensus Net Meeting on Dermoscopy (CNMD) (referenced to Soyer 2001), selected by 2 experts Setting: unclear Prior testing: NR Setting for prior testing: NR Exclusion criteria: none reported Sample size (participants): NR Sample size (lesions): number included: 20 Participant characteristics: none reported Lesion characteristics: none reported
Index tests	Dermoscopy: pattern analysis; 7PCL; ABCD; Menzies criteria Method of diagnosis: clinical photographs and dermoscopic images. Partipants were given a CD with lesion images and asked to evaluate the 20 cases independently over a 20‐day period. This was repeated approximately 5 weeks post‐dermoscopy training. Prior test data: case notes, "Each case contained the following clinical information: age, sex, skin phototype, total number of naevi, personal and/or family history of melanoma, location, diameter and duration of the lesion, as well as medical history concerning morphological changes within the year preceding excision of the lesion." It appears as though this information was given to participants along with lesion images. Diagnostic threshold: clinical diagnosis of melanoma; "For each case, the participants completed an electronic data sheet that listed criteria for diagnosing PSLs by pattern analysis and by the various algorithms. Participants offered a dermoscopic diagnosis for each case" Diagnosis based on: average (n = 16); authors' colleagues from Department of Dermatology were recruited to participate Observer qualifications: dermatology registrar 9; dermatologist 4; medical students 3 Experience in practice: not described Experience with dermoscopy: low; dermoscopic knowledge of this group consisted only of limited personal experience; none had formal training in this technique and ⁄ or used dermoscopy in daily professional practice Dermoscopy training: a 1‐h lecture introduced the principles of dermoscopy and the algorithms to be evaluated. A web‐based tutorial was then made available and participants were asked to spend 1 h/d for 2 weeks to learn and improve dermoscopy knowledge (dermoscopy.org) Training format: in person and online Post‐training experience: none reported
Target condition and reference standard(s)	Reference standard: histological diagnosis alone (not further described) Disease‐positive: 6 (30%); disease‐negative: 14 (70%) Target condition (final diagnoses) Melanoma (in situ and invasive, or NR): 6 (30%) BCC: 2 (10%) Sebhorrheic keratosis: 2 (10%); Clark naevi 8 (40%); Reed/SN 2 (10%)
Flow and timing	Participant exclusions: none reported Index test to reference standard interval: not described
Comparative
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Unclear
Was a case‐control design avoided?	Unclear
Did the study avoid inappropriate exclusions?	Unclear
Are the included patients and chosen study setting appropriate?	No
Did the study avoid including participants with multiple lesions?	Unclear
		Unclear	High
DOMAIN 2: Index Test Dermoscopy ‐ image‐based
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Yes
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?	No
Was the test applied and interpreted in a clinically applicable manner?	No
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	Yes
Was the test interpretation carried out by an experienced examiner?	No
		Low	High
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	Yes
Were the reference standard results interpreted without knowledge of the results of the index tests?	Unclear
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Unclear
		Low	Unclear
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Unclear
Did all patients receive the same reference standard?	Yes
Were all patients included in the analysis?	Yes
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?	Unclear
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?
		Unclear

Piccolo 2000.

Study characteristics
Patient sampling	Study design: case series Data collection: prospective Period of data collection: states 3 months but no specific dates given Country: Austria (Graz)
Patient characteristics and setting	Inclusion criteria: lesions included in the study were selected because of their diagnostic difficulty and were excised for a histopathological evaluation Setting: unspecified, described as a multicentre study Prior testing: lesions included in the study were selected because of their diagnostic difficulty does not specify what prior tests were done Setting for prior testing: unspecified Exclusion criteria: poor‐quality index test image (all images scoring 4 were excluded from the study) Sample size (participants): number included: 40 participants Sample size (lesions): number included: 43 Participant characteristics: median age 39.5 years, (range 3–91 years). Male: 21 (53%); female 19 (47%) Lesion characteristics: site ‐ face 2; head 1, neck 1, trunk 8, arms 3, legs 7, back 20, buttocks 1
Index tests	Dermoscopy: no algorithm Method of diagnosis: all lesions were examined with a dermatoscope during the face‐to‐face clinical diagnosis. Diagnosis was made by a expert dermatologist based on clinical features and dermoscopic findings. No specific algorithm (e.g. the Stolz index) was used for dermoscopic diagnosis. Prior test data: unclear Diagnostic threshold: NR Diagnosis based on: single (n = 1) Observer qualifications: dermatologist (an expert in the diagnosis of PSLs) Experience in practice: high Experience with index test: high (Also evaluated teledermatology assessment of transmitted images)
Target condition and reference standard(s)	Reference standard: histological diagnosis alone Details: all lesions were excised for a histopathological evaluation Target condition (final diagnoses) Melanoma (invasive): 11, BCC: 3 Sebhorrheic keratosis: 2, benign naevus: melanocytic naevus 23, 'Benign' diagnoses: angiokeratoma 1, lentigines 3
Flow and timing	Excluded participants: NR Time interval to reference test: NR Time interval between index test(s): NR
Comparative
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Unclear
Was a case‐control design avoided?	Yes
Did the study avoid inappropriate exclusions?	Unclear
Are the included patients and chosen study setting appropriate?	No
Did the study avoid including participants with multiple lesions?	Yes
		Unclear	High
DOMAIN 2: Index Test Dermoscopy ‐ in‐person
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Unclear
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	Yes
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	Unclear
Was the test interpretation carried out by an experienced examiner?	Yes
		Unclear	Unclear
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	Yes
Were the reference standard results interpreted without knowledge of the results of the index tests?	Unclear
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Unclear
		Low	Unclear
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Unclear
Did all patients receive the same reference standard?	Yes
Were all patients included in the analysis?	Yes
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?

Piccolo 2002a.

Study characteristics
Patient sampling	Study design: case series Data collection: retrospective image selection/prospective interpretation Period of data collection: NR; 6‐month period Country: Italy
Patient characteristics and setting	Inclusion criteria: pigmented lesions excised because of equivocal dermoscopic findings or at the patient’s request Setting: secondary (general dermatology); from authors' institution Prior testing: dermatoscopic suspicion; patient request for evaluation/excision Setting for prior testing: NR Exclusion criteria: none reported Sample size (participants): number included: 289 Sample size (lesions): number included: 341 Participant characteristics: mean age 33.6 years, range 3–83 years; male: 127 (43.9%); Fitzpatrick phototype I‐II (31.4%); type III (42%); type IV‐V (26.4%) Lesion characteristics: none reported
Index tests	Dermoscopy: no algorithm Method of diagnosis: clinical photographs and dermoscopic images. Cases were clinically and dermoscopically evaluated on a high‐resolution colour monitor, in a random sequence Prior test data: none; appears to be based on images only Diagnostic threshold: correct diagnosis of melanoma Diagnosis based on: single observer (n = 2) Observer qualifications: dermatologist; (dermatology?) resident Experience in practice: high, dermatologist had 5 years of experience; low, resident with minimal training in PSLs Experience with dermoscopy: high and low (resident had 6 months of experience, comprising 8 h of specialised training on 3 consecutive days and 2 h/week in routine dermoscopy)
Target condition and reference standard(s)	Reference standard: histological diagnosis alone Details: "All excised lesions were examined histopathologically by a dermatopathologist" Disease‐positive: 13; disease‐negative: 328 Target condition (final diagnoses) Melanoma (in situ and invasive, or NR): 13 Sebhorrheic keratosis: 3; benign naevus: 316; dFs 7; angiomas 2
Flow and timing	Time interval to reference test: NR
Comparative
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Unclear
Was a case‐control design avoided?	Yes
Did the study avoid inappropriate exclusions?	Unclear
Are the included patients and chosen study setting appropriate?	No
Did the study avoid including participants with multiple lesions?	No
		Unclear	High
DOMAIN 2: Index Test Dermoscopy ‐ image‐based
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Unclear
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	Unclear
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	No
Was the test interpretation carried out by an experienced examiner?	Yes
		Unclear	High
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	Yes
Were the reference standard results interpreted without knowledge of the results of the index tests?	Unclear
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Yes
		Low	Low
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Unclear
Did all patients receive the same reference standard?	Yes
Were all patients included in the analysis?	Yes
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?
		Unclear

Piccolo 2014.

Study characteristics
Patient sampling	Study design: case series Data collection: retrospective image selection/prospective interpretation Period of data collection: September 2010‐October 2013 Country: Italy
Patient characteristics and setting	Inclusion criteria: dermoscopically atypical PSLs selected from the archives of the Dermatology Department at the University of L’Aquila, Italy; described as "a panel of ... retrospectively selected PSLs" Setting: secondary (general dermatology) Prior testing: NR Setting for prior testing: NR Exclusion criteria: location/site of lesion ‐ acral sites and the face Sample size (participants): number included: 165 Sample size (lesions): number included: 165 Participant characteristics: mean age 43.5 years (range 12‐84 years); male: 59.4% Lesion characteristics: lesion site; upper extremities 18 (11%); lower extremities 53 (31%); 62 (37.5%) on the back; 32 (19.4%) on the chest. Melanoma thickness 87.9% (29/33) < 0.75 mm; 11% (4/33) > 1.5 mm
Index tests	Dermoscopy: ABCD Method of diagnosis: dermoscopic images Prior test data: no further information used Diagnostic threshold: total dermoscopic score > 4.75 and > 5.45 Diagnosis based on: single observer (n = 4) Observer qualifications: 3 dermatologists and 1 GP with different degrees of dermoscopic experience Experience in practice: mixed Experience with dermoscopy: high (observer 1 ‐ dermatologist); moderate (observers 2 and 3 ‐ dermatologists); low (observer 4 ‐ GP; underwent dermoscopic training by studying an interactive atlas of dermoscopy between time periods T0 and T1) Any other detail: experience was scored based on number of years specialising in dermoscopy; number of PSLs assessed by dermoscopy on a daily basis; number of relevant workshops/seminars attended; and the number of authored publications on dermoscopy.
Target condition and reference standard(s)	Reference standard: histological diagnosis alone (not further described) Target condition (final diagnoses) Melanoma (invasive): 23; melanoma (in situ): 10 Benign naevus: 105 Clark naevi; 19 Spitz/Reed naevi; 5 blue naevi; 3 dermal naevi
Flow and timing	Excluded participants: none reported Time interval to reference test: not described
Comparative
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Unclear
Was a case‐control design avoided?	Yes
Did the study avoid inappropriate exclusions?	No
Are the included patients and chosen study setting appropriate?	No
Did the study avoid including participants with multiple lesions?	Yes
		High	High
DOMAIN 2: Index Test Dermoscopy ‐ image‐based
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Yes
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	No
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	Yes
Was the test interpretation carried out by an experienced examiner?	Yes
		Low	High
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	Yes
Were the reference standard results interpreted without knowledge of the results of the index tests?	Unclear
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Unclear
		Low	Unclear
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Unclear
Did all patients receive the same reference standard?	Yes
Were all patients included in the analysis?	Yes
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?
		Unclear

Pizzichetta 2002.

Study characteristics
Patient sampling	Study design: case series Data collection: retrospective image selection/prospective interpretation Period of data collection: April 1996‐September 1998 Country: Italy
Patient characteristics and setting	Inclusion criteria: small (≤ 5 mm) melanocytic skin lesions with "dermoscopic appearance not excluding melanona" that were surgically excised at the Centro di Riferimento Oncologico (National Cancer Institute), Aviano Setting: specialist unit; National Cancer Institute Prior testing: NR Setting for prior testing: NR Exclusion criteria: size > 5 mm Sample size (participants): number included: 123 Sample size (lesions): number included: 129 Participant characteristics: median age 30 years, range 13‐65 years. Lesion site: trunk: 67 (52%); upper limbs/shoulder: 16 (14%); lower limbs/hip: 21 (16.3%); abdomen 21 (16.3%); foot 4 (3.1%) Lesion characteristics: median diameter 4 mm (range: 1.2‐5 mm)
Index tests	Dermoscopy: pattern analysis; ABCD Method of diagnosis: dermoscopic images Prior test data: no further information used; only images assessed for presence/absence of dermoscopic criteria and dermoscopic diagnosis Diagnostic threshold: ABCD > 5.45 and ≥ 4.75; pattern analysis: "Dermoscopic criteria used for evaluation were pigment network alterations, irregular extensions, branched streaks. grey·blue areas. pseudopods. brown globules. black dots, whitish blue veil, hypopigmentation, white scar‐like areas and linear and dotted vascular patterns." Diagnosis based on: single observer (n = 2) Observer qualifications: NR; likely oncologist/dermatologist (1 observer based in Dept of Oncology, other in Dermatology dept) Experience in practice: not described Experience with dermoscopy: not described
Target condition and reference standard(s)	Reference standard: histological diagnosis alone Details: histopathologic diagnosis of all specimens was performed by a single pathologist at the Department of Pathology of the Centro di Riferimento Oncologico Disease‐positive: 5 lesions; disease‐negative: 124 lesions Target condition (final diagnoses) Melanoma (in situ and invasive, or NR): 5 lesions Benign naevus: 124 lesions
Flow and timing	Participant exclusions: none reported Index test to reference standard interval: each lesion imaged "before surgery"
Comparative
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Yes
Was a case‐control design avoided?	Yes
Did the study avoid inappropriate exclusions?	No
Are the included patients and chosen study setting appropriate?	No
Did the study avoid including participants with multiple lesions?	Yes
		High	High
DOMAIN 2: Index Test Dermoscopy ‐ image‐based
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Yes
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?	Unclear
Was the test applied and interpreted in a clinically applicable manner?	No
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	Yes
Was the test interpretation carried out by an experienced examiner?	Unclear
		Low	High
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	Yes
Were the reference standard results interpreted without knowledge of the results of the index tests?	Unclear
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Unclear
		Low	Unclear
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Yes
Did all patients receive the same reference standard?	Yes
Were all patients included in the analysis?	Yes
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?
		Low

Pizzichetta 2004.

Study characteristics
Patient sampling	Study design: case series Data collection: retrospective image selection/prospective interpretation Period of data collection: January 1996 to December 2001 Country: participants recruited from 5 participating centres (4 in Italy and 1 in USA) study conducted in Italy
Patient characteristics and setting	Inclusion criteria: clinical and/or dermoscopic hypomelanotic (extent of pigmentation ≤ 30%) and amelanotic skin lesions seen and excised at the 5 participating centres Setting: secondary (general dermatology) Prior testing: clinical and/or dermatoscopic suspicion Setting for prior testing: NR Exclusion criteria: poor‐quality index test image (considered under Flow and timing) Sample size (participants): number included: 151 Sample size (lesions): number eligible: 174; number included: 151 Participant characteristics: mean age 47 years (± 17.5 SD); male: 73 (48%) Lesion characteristics: lesion site ‐ head/neck (5.3%); trunk (20.5%); upper limbs/shoulder (11.9%); lower limbs/hip (25.2%); back (21.2%); abdomen (11.3%); hand (3.3%); foot (1.3%). Melanoma thickness: ≤ 1 mm 85.3% (n = 29); > 1 mm 14.7% (n = 15)
Index tests	VI: no algorithm Method of diagnosis: clinical photographs Prior test data: only gender, age at diagnosis and the site of the skin lesion were known to the observer Other test data: file contained clinical and dermoscopic images; unclear whether both observed at the same time. Diagnostic threshold: investigated clinical features such as elevation, ulceration, shape, borders, colour Diagnosis based on: single observer (n = 1) Observer qualifications: NR; assumed dermatologist Experience in practice: not described Experience with index test: not described Dermoscopy: pattern analysis Method of diagnosis: dermoscopic images Prior test data: clinical image also available Diagnostic threshold: assessed the lesions using the following dermoscopic criteria associated with melanoma and non‐melanocytic skin lesions: pigment network, pigmentation, streaks, dots ⁄ globules, blue‐whitish veil, regression structures, hypopigmentation, leaf‐like areas, multiple grey‐bluish globules, central white patch and vascular pattern Test observers as described for VI (above) Other detail: 122 images were taken with a digital stereomicroscope and 52 were taken with a Dermaphot camera (Heine Optotechnik; Herrsching, Germany) (x10 magnification)and then digitalised with the Kodak PhotoCDsystem. Ultrasound gel was used on all the lesions (52) photographed with the Dermaphot in the Aviano centre. The other centres used the digital stereomicroscope consisting of a stereomicroscope and a Sony 3CCDDXC‐930P colour video camera. The digital images were taken at a magnification of x10–20
Target condition and reference standard(s)	Reference standard: histological diagnosis alone Target condition (final diagnoses) Melanoma (invasive): 34 (39 in full sample); melanoma (in situ): 5 Other diagnoses reported only for full sample of 151 (only 108 with clinical images for VI evaluation): 55 (40 with clinical images) "amelanotic ⁄ hypomelanotic non‐melanocytic lesions" (25 BCC, 4 SCC, 10 DF, 8 BD, 8 SK) 52 (29 with clinical images) "amelanotic ⁄ hypomelanotic benign melanocytic lesions" (24 compound naevi, 17 dermal naevi, 5 SN, 4 congenital naevi and 2 combined naevi)
Flow and timing	Excluded participants: 23 lesions excluded due to image quality; further 43 lesions were not available for evaluation by clinical images ("mainly benign melanocytic lesions"). Time interval to reference test: NR
Comparative	Blinding between tests: unclear whether both clinical and dermoscopic images observed at the same time Time interval between index test(s): NR
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Unclear
Was a case‐control design avoided?	Yes
Did the study avoid inappropriate exclusions?	No
Are the included patients and chosen study setting appropriate?	No
Did the study avoid including participants with multiple lesions?	Yes
		High	High
DOMAIN 2: Index Test Visual inspection ‐ image‐based
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Yes
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	No
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	No
Was the test interpretation carried out by an experienced examiner?	Unclear
		Low	High
DOMAIN 2: Index Test Dermoscopy ‐ image‐based
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Yes
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	Unclear
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	Yes
Was the test interpretation carried out by an experienced examiner?	Unclear
		Low	Unclear
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	Yes
Were the reference standard results interpreted without knowledge of the results of the index tests?	Yes
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Unclear
		Low	Unclear
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Unclear
Did all patients receive the same reference standard?	Yes
Were all patients included in the analysis?	No
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?
		High
DOMAIN 5: Comparative
Was each index test result interpreted without knowledge of the results of other index tests or testing strategies?	Unclear
Was the interval between application of the index tests less than one month?	Unclear
Were all tests applied and interpreted in a clinically applicable manner?	No
		Unclear	High

Pupelli 2013.

Study characteristics
Patient sampling	Study design: case‐control Data collection: retrospective Period of data collection: 2007‐2011 Country: Italy
Patient characteristics and setting	Inclusion criteria: consecutively excised melanomas < 5 mm diameter and 3 randomly sampled histologically proven small‐diameter naevi for each included melanoma Setting: specialist unit (skin cancer/PLC) (from Author institution) Prior testing: selected for excision (no further detail); all had undergone dermoscopy and RCM in order to be included Setting for prior testing: specialist unit (skin cancer/PLC) Exclusion criteria: lesion size > 5 mm excluded; disagreement between evaluators on tumour histological classification Sample size (participants): number included: 96 Sample size (lesions): number included: 96 Participant characteristics: mean age: melanoma group 48 years (IQR 17‐77 years); naevi 41 years (IQR 6‐82 years). Male: 54% of melanoma group; 58% of naevi group Lesion characteristics: lesion site ‐ trunk: 62% naevi; lower limbs/hip: 46% melanomas; melanoma thickness: mean 0.37 mm (SD 0.44 mm). Lesion size (invasive melanoma): 77% (n = 10) < 1 mm, 13% (n = 3) ≥ 1 mm
Index tests	Dermoscopy: 7PCL Method of diagnosis: dermoscopic images Prior test data: body site and age; it appears that RCM images also available at time of image interpretation. "For each lesion a complete set of dermoscopic and confocal images (including the whole lesion) was available"; "Dermoscopic and confocal microscopic images were evaluated – in blind from histological diagnosis, but not from the body site or the age of the patient". Diagnostic threshold: score ≥ 3 Diagnosis based on: unclear likely single (n = NR) Observer qualifications: NR; no description of observers was provided Experience in practice: not described Experience with dermoscopy: not described
Target condition and reference standard(s)	Reference standard: histological diagnosis alone Details: histopathology performed by 2 independent board‐certified pathologists; disagreements were reviewed by both pathologists to obtain a consensus diagnosis. Disease‐positive: 24; disease‐negative: 72 Target condition (final diagnoses) Melanoma (invasive): 13; melanoma (in situ): 11 Benign naevus: 72 ( 29 junctional, 19 compound, intra‐dermal, 8 blue, 4 LS and 7 Spitz)
Flow and timing	Participant exclusions: none reported Index test to reference standard interval: not described
Comparative
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Yes
Was a case‐control design avoided?	No
Did the study avoid inappropriate exclusions?	No
Are the included patients and chosen study setting appropriate?	No
Did the study avoid including participants with multiple lesions?	Yes
		High	High
DOMAIN 2: Index Test Dermoscopy ‐ image‐based
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Yes
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	Unclear
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	Yes
Was the test interpretation carried out by an experienced examiner?	Unclear
		Low	Unclear
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	Yes
Were the reference standard results interpreted without knowledge of the results of the index tests?	Unclear
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Yes
		Low	Low
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Unclear
Did all patients receive the same reference standard?	Yes
Were all patients included in the analysis?	Yes
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?
		Unclear

Rao 1997.

Study characteristics
Patient sampling	Study design: case series Data collection: retrospective image selection/prospective interpretation Period of data collection: NR Country: USA
Patient characteristics and setting	Inclusion criteria: patients with atypical melanocytic lesions or suspected early malignant melanoma Setting: private care Prior testing: selected for excision (no further detail) Setting for prior testing: private care Exclusion criteria: lesions > 13 mm in diameter were excluded as they could not fit entirely within the standardised photographs Sample size (participants): number included: 63 Sample size (lesions): number included: 72 Participant characteristics: none reported Lesion characteristics: melanoma thickness ≤ 1 mm: 100% of MM (n = 21)
Index tests	VI: ABCD Method of diagnosis: clinical photographs and dermoscopic images Prior test data: dermoscopic images also presented to observer but unclear whether both viewed at the same time or not, "Each color transparency was independently analyzed" by observers. The 1) clinical, 2) 'overall' dermoscopic, and 3) 'ABCD scored' dermoscopic diagnoses of either MM or atypical melanocytic naevi were recorded for each lesion by the same observers. No indication of blinding between images Diagnostic threshold: clinical variables were defined as follows: asymmetry (A): both silhouette and colour distribution were considered. Border irregularity (B): this was judged by the unevenness of the perimeter. Colour (C): colour variegation and number of colours were evaluated. Diameter (D): the largest in situ diameter in mm of each lesion was recorded Diagnosis based on: single observer (n = 4) Observer qualifications: 2 experienced dermatologists, and 2 melanoma fellows Experience in practice: mixed experience (low and high experience combined) Experience with dermoscopy: NR Dermoscopy: ABCD and no algorithm Method of diagnosis: clinical photographs and dermoscopic images Prior test data: clinical examination and/or case notes. The 1) clinical, 2) 'overall' dermoscopic, and 3) 'ABCD scored' dermoscopic diagnoses of either MM or atypical melanocytic naevi were recorded for each lesion by the same observers. No indication of blinding between images Diagnostic threshold: ABCD‐scored dermoscopic diagnosis (lesions with a score of ≤ 4.75 were classified as benign, those with scores 4.76‐5.45 as suspicious, and those with scores > 5.45 as melanomas. Each feature was given a score of ”1”. Thus, the score ranged from 1 to 5.) Overall dermoscopic diagnosis ‐ no threshold reported; the overall dermoscopic impression was recorded based on criteria in the recently published textbook (Stolz 1994b). Test observers: as described for VI (above) Any other detail: all photographs were taken with the Dermophot standard lens‐to‐lesion distance, aperture, and flash. Fujichrome 50 colour 35 mm‐transparency film was used and all exposed film was processed in the same laboratory (Colorite, New York, NY, USA)
Target condition and reference standard(s)	Reference standard: histological diagnosis alone Details: each of the 72 melanocytic neoplasms was histopathologically diagnosed as with atypical melanocytic naevi or an early MM by a dermapathologist with special expertise in melanocytic neoplasms. Each lesion was completed excised and step sectioned. Disease‐positive: 21 MMs; disease‐negative: 51 atypical melanocytic naevi Target condition (final diagnoses) MM (invasive): 21 51 atypical melanocytic naevus
Flow and timing	Excluded participants: none reported Time interval to reference test: NR
Comparative	Blinding between tests: unclear whether both images were viewed at the same time or not Time interval between index test(s): Image‐based; images likely acquired consecutively
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Unclear
Was a case‐control design avoided?	Yes
Did the study avoid inappropriate exclusions?	Unclear
Are the included patients and chosen study setting appropriate?	No
Did the study avoid including participants with multiple lesions?	No
		Unclear	High
DOMAIN 2: Index Test Visual inspection ‐ image‐based
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Yes
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	No
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	No
Was the test interpretation carried out by an experienced examiner?	Yes
		Low	High
DOMAIN 2: Index Test Dermoscopy ‐ image‐based
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Yes
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?	Unclear
Was the test applied and interpreted in a clinically applicable manner?	Unclear
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	Yes
Was the test interpretation carried out by an experienced examiner?	Unclear
		Low	Unclear
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	Yes
Were the reference standard results interpreted without knowledge of the results of the index tests?	Unclear
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Yes
		Low	Low
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Yes
Did all patients receive the same reference standard?	Yes
Were all patients included in the analysis?	Yes
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?
		Low
DOMAIN 5: Comparative
Was each index test result interpreted without knowledge of the results of other index tests or testing strategies?	Unclear
Was the interval between application of the index tests less than one month?	Yes
Were all tests applied and interpreted in a clinically applicable manner?	No
		Unclear	High

Rigel 2012.

Study characteristics
Patient sampling	Study design: unclear Data collection: retrospective image selection/prospective interpretation Period of data collection: NR Country: USA
Patient characteristics and setting	Inclusion criteria: PSLs that had been analysed as part of a prior study using a MSDSLA system (Monheit 2011); melanomas and other pigmented lesions presumably selected on a case‐control type basis Setting: unclear Prior testing: NR Setting for prior testing: NR Exclusion criteria: none reported Sample size (participants): number included: NR Sample size (lesions): number included: 24 Participant characteristics: none reported Lesion characteristics: none reported
Index tests	Dermoscopy: no algorithm Method of diagnosis: dermoscopic images; interactive melanoma session where dermatologists were first presented with clinical and dermoscopic images and asked to make a diagnosis; then presented with information from MelaFind Prior test data: patient history and clinical images were presented along with dermoscopic images Diagnostic threshold: clinical decision to excise or not Diagnosis based on: average (n = 179) Observer qualifications: dermatologist; practicing dermatologists attending an educational conference Experience in practice: assumed high (median duration of practice 11‐15 years) Experience with dermoscopy: NR
Target condition and reference standard(s)	Reference standard: histological diagnosis alone (not further described) Disease‐positive: 5; disease‐negative: 19 Target condition (final diagnoses) Melanoma (in situ and invasive, or NR): 5; 'Benign' diagnoses: 19
Flow and timing	Participant exclusions: none reported Index test to reference standard interval: not described
Comparative
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Unclear
Was a case‐control design avoided?	Unclear
Did the study avoid inappropriate exclusions?	Unclear
Are the included patients and chosen study setting appropriate?	No
Did the study avoid including participants with multiple lesions?	Unclear
		Unclear	High
DOMAIN 2: Index Test Dermoscopy ‐ image‐based
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Unclear
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	Unclear
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	No
Was the test interpretation carried out by an experienced examiner?	Unclear
		Unclear	High
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	Yes
Were the reference standard results interpreted without knowledge of the results of the index tests?	Unclear
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Unclear
		Low	Unclear
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Unclear
Did all patients receive the same reference standard?	Yes
Were all patients included in the analysis?	Yes
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?
		Unclear

Rosendahl 2011.

Study characteristics
Patient sampling	Study design: case series Data collection: retrospective image selection/prospective interpretation Period of data collection: 30‐month period; dates NR Country: Australia
Patient characteristics and setting	Inclusion criteria: consecutive series of pigmented lesions submitted for histology from the primary care skin cancer practice of 1 study author Setting: primary/private; skin cancer practice of 1 study author Prior testing: selected for excision (no further detail) Setting for prior testing: primary Exclusion criteria: poor image quality (considered under Flow and Timing) Sample size (participants): number included: 389 Sample size (lesions): number eligible: 466 pigmented lesions out of 1959 lesions excised or biopsied; number included: 463 Participant characteristics: mean age: 57 years (SD 17); male: 67.4% Lesion characteristics: (53.1%) melanocytic. Lesion site: 17.7% head or face; trunk: 52.1%; 27.6% extremities; 2.2% palms or soles. Melanoma thickness: ≤ 1 mm: 1/29 melanoma (3.4%)
Index tests	VI: no algorithm Method of diagnosis: clinical photographs; overview and close‐up image presented Prior test data: no further information used Other test data: dermoscopic images presented to observer subsequent to diagnosis using clinical images alone Diagnostic threshold: clinical diagnosis/subjective impression. Observers gave a diagnosis with level of confidence (from 0 for definitely benign to 100 for definitely malignant) after viewing the clinical images. (NB used study authors' threshold for detection of any skin cancer, which includes lesions clinically considered to be MM, BCC pigmented epithelial carcinoma including SCC, keratoacanthoma, AK and BD as test‐positive; review only considered histologically confirmed MM, BCC or invasive SCC to be disease‐positive) Diagnosis based on: single observer (n = NR) Observer qualifications: expert dermatologist (based on author communication). Experience in practice: expert Experience with dermoscopy: expert Dermoscopy: pattern analysis; new algorithm; Chaos and clues Method of diagnosis: clinical photographs (one overview and one close‐up), followed by one dermoscopic image presented to a blinded observer on a computer screen Prior test data: clinical image only; diagnosis made based on clinical image before presentation of dermoscopic image Diagnostic threshold: observers gave a diagnosis with level of confidence (from 0 for definitely benign to 100 for definitely malignant). Chaos and clues short algorithm; each assessed for evidence of ‘‘chaos’’ (asymmetry of colour or structure); if present then ‘‘clues’’ searched for. Chaos: asymmetry of structure and colour defined according to the basic principles of pattern analysis as revised by Kittler 2007. Clues included: eccentric structureless zone (any colour except skin colour), grey or blue structures, peripheral black dots or clods, segmental radial lines or pseudopods, polymorphous vessels, white lines, thick reticular or branched lines, and parallel lines on ridges (acral lesions). Observers as for VI
Target condition and reference standard(s)	Reference standard: histological diagnosis alone Details: excise or biopsy Disease‐positive: 138; disease‐negative: 325 Target condition (final diagnoses) Melanoma (invasive): 9; melanoma (in situ): 20; BCC: 72; cSCC: 5 (including 2 keratoacanthoma) 'Benign' diagnoses: 18 BD and 14 AK, 217 benign melanocytic + additional 140 benign non‐melanocytic Note: study authors considered BD, AK and keratoacanthoma as malignant; all considered benign for review analysis
Flow and timing	Excluded participants: lesions were excluded due to poor image quality (n = 3) Time interval to reference test: unclear; lesions 'routinely photographed' if scheduled for excision or biopsy but not further described
Comparative	Blinding between tests: clinical photographs (one overview and one close‐up), followed by one dermoscopic image presented to a blinded observer on a computer screen Time interval between index test(s): consecutive
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Yes
Was a case‐control design avoided?	Yes
Did the study avoid inappropriate exclusions?	No
Are the included patients and chosen study setting appropriate?	No
Did the study avoid including participants with multiple lesions?	No
		High	High
DOMAIN 2: Index Test Visual inspection ‐ image‐based
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Yes
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	No
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	No
Was the test interpretation carried out by an experienced examiner?	Yes
		Low	High
DOMAIN 2: Index Test Dermoscopy ‐ image‐based
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Yes
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?	No
Was the test applied and interpreted in a clinically applicable manner?	Unclear
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	Yes
Was the test interpretation carried out by an experienced examiner?	Yes
		Low	Unclear
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	Yes
Were the reference standard results interpreted without knowledge of the results of the index tests?	Unclear
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Unclear
		Low	Unclear
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Unclear
Did all patients receive the same reference standard?	Yes
Were all patients included in the analysis?	No
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?
		High
DOMAIN 5: Comparative
Was each index test result interpreted without knowledge of the results of other index tests or testing strategies?	No
Was the interval between application of the index tests less than one month?	Yes
Were all tests applied and interpreted in a clinically applicable manner?	No
		Low	High

Rubegni 2012.

Study characteristics
Patient sampling	Study design: case series Data collection: retrospective image selection/prospective interpretation Period of data collection: January2008‐December 2010 Country: Italy
Patient characteristics and setting	Inclusion criteria: all palmoplantar PSLs observed and removed because of the presence of clinical and/or dermoscopic suspicious features and in the absence of any clear benignity pattern (parallel furrow pattern, lattice‐like pattern or fibrillar pattern). Setting: secondary (general dermatology) Prior testing: clinical and/or dermatoscopic suspicion Setting for prior testing: secondary (general dermatology) Exclusion criteria: non‐acral lesions; site of lesion in volar skin of the folds near the toes; lesion size > 26 mm diameter; non‐melanocytic appearance; elevated or ulcerated appearance Sample size (participants): number included: 107 Sample size (lesions): number included: 107 Participant characteristics: mean age: 49.8 years (women); 44.9 years (men); range 19‐73 years; male: 58.9%; ethnicity white: 100% Lesion characteristics 78 on soles and 19 on palms; 9 (36%) melanomas ≤ 0.75 mm (incl 4 in situ); 11 (44%) 0.76‐1.5 mm in 11/25 lesions; 5 (20%) ≥ 1.50 mm
Index tests	Dermoscopy: pattern analysis; 3‐step algorithm for palmoplantar lesions (Koga 2011) Method of diagnosis: dermoscopic images Prior test data: no further information used Diagnostic threshold: clinical diagnosis (melanoma/no melanoma). For the 3‐step algorithm the conventional options are "removal, follow‐up or no follow‐up"; the latter 2 were combined under the term ‘no melanoma’ for study purposes Diagnosis based on: single observer (n = 2; one per algorithm) Observer qualifications: dermatologist Experience in practice: high Experience with dermoscopy: high; 2 dermatologists with 20 years’ experience in dermoscopy Any other detail: ELM images achieved with the DB‐Mips System; (magnification x 16)
Target condition and reference standard(s)	Reference standard: histological diagnosis alone Details: "Histopathological diagnosis was based on the criteria of the National Institute of Health Consensus Conference" Disease‐positive: 25; disease‐negative: 82 Target condition (final diagnoses) Melanoma (invasive): 21; melanoma (in situ): 4 'Benign' diagnoses: 82
Flow and timing	Participant exclusions: none reported Index test to reference standard interval: not described
Comparative
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Yes
Was a case‐control design avoided?	Yes
Did the study avoid inappropriate exclusions?	No
Are the included patients and chosen study setting appropriate?	No
Did the study avoid including participants with multiple lesions?	Yes
		High	High
DOMAIN 2: Index Test Dermoscopy ‐ image‐based
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Yes
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?	Yes
Was the test applied and interpreted in a clinically applicable manner?	No
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	Yes
Was the test interpretation carried out by an experienced examiner?	Yes
		Low	High
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	Yes
Were the reference standard results interpreted without knowledge of the results of the index tests?	Unclear
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Yes
		Low	Low
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Unclear
Did all patients receive the same reference standard?	Yes
Were all patients included in the analysis?	Yes
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?
		Unclear

Rubegni 2016.

Study characteristics
Patient sampling	Study design: case series Data collection: retrospective image selection/prospective interpretation Period of data collection: 2010–2014 Country: NR. Majority of study authors based in Italy, but source of lesion images not described
Patient characteristics and setting	Inclusion criteria: consecutive melanocytic skin lesions showing clear‐cut dermoscopic features of regression that were excised for suspected malignancy. Regression features included: blue‐grey veil, blue grey globules and white scar‐like areas, hypopigmented areas and atypical network (all of which may be present in benign and malignant lesions) Setting: secondary; not clearly reported but study authors all based in dermatology units or departments Prior testing: dermatoscopic suspicion in all cases Setting for prior testing: NR Exclusion criteria: none reported Sample size (participants): number included: 95 Sample size (lesions): number included: 95 Participant characteristics: median age: naevi group 36 years (14‐59 years); melanoma group 54.4 years (17‐89 years). Male: 43; 45.2% Lesion characteristics: lesion site: head/neck: 20 (40%) of naevi; trunk 23(46%) of naevi and 24 (55%) of melanoma group; extremities 7 (14%) of naevi group; other areas 20 (45%) of melanomas. Lesion size: mean 7.63 mm, range 4‐16 mm (naevi) and 10.33 mm 5‐19 mm (melanomas)
Index tests	Dermoscopy: pattern analysis using 12 dermoscopic features of regression (study also developed a new classifier but data excluded from review, due to use of 'leave one out' procedure for validation) Method of diagnosis: dermoscopic images; randomly presented to observers blinded to histopathological diagnosis Prior test data: unclear; data on morphology, site, age and gender were collected but not clear if presented along with image Diagnostic threshold: diagnosis of melanoma or naevus following assessment of 12 dermoscopic structures suggestive of regression selected according to the literature (Zalaudek 2004; Seidenari 2010) including blue‐grey areas, blue‐whitish veil, blue globules and blue‐grey peppering, white scar‐like areas, white shiny streaks, atypical network, hypopigmented areas, irregular dots and globules, irregular streaks, irregular pigmented blotches and pink areas Diagnosis based on: single observer and consensus of 2/3 (n = 3) Observer qualifications: dermatologist Experience in practice: high; expert dermatologists Experience with dermoscopy: high; "experienced dermoscopists"
Target condition and reference standard(s)	Reference standard: histological diagnosis alone Details: every histological diagnosis was confirmed by 2/3 expert dermopathologists Disease‐positive: 45; disease‐negative: 50 Target condition (final diagnoses) Melanoma (in situ and invasive, or NR): 45 Benign naevus: 50
Flow and timing	Participant exclusions: none reported Index test to reference standard interval: not described
Comparative
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Yes
Was a case‐control design avoided?	Yes
Did the study avoid inappropriate exclusions?	Yes
Are the included patients and chosen study setting appropriate?	No
Did the study avoid including participants with multiple lesions?	Yes
		Low	High
DOMAIN 2: Index Test Dermoscopy ‐ image‐based
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Yes
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	No
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	Yes
Was the test interpretation carried out by an experienced examiner?	Yes
		Low	High
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	Yes
Were the reference standard results interpreted without knowledge of the results of the index tests?	Unclear
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Yes
		Low	Low
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Unclear
Did all patients receive the same reference standard?	Yes
Were all patients included in the analysis?	Yes
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?
		Unclear

Sboner 2004.

Study characteristics
Patient sampling	Study design: case series Data collection: retrospective image selection/prospective interpretation Period of data collection: NR Country: Italy (based on authors' institution)
Patient characteristics and setting	Inclusion criteria: melanocytic lesion images acquired consecutively by d‐ELM at the Department of Dermatology of Santa Chiara Hospital, Trento Setting: secondary (general dermatology) Prior testing: selected for excision (no further detail) Setting for prior testing: NR Exclusion criteria: seems that dysplastic naevi were excluded; "In this experimental setting, there were no dysplastic naevi" Sample size (participants): NR Sample size (lesions): number included: 152 Participant characteristics: none reported Lesion characteristics: mean Breslow thickness for the invasive lesions is 1.0 +/‐ 0.7 mm; 81% ≤ 1.5 mm
Index tests	Dermoscopy: no algorithm Method of diagnosis: dermoscopic images; digitial‐ELM images presented on video device Prior test data: no further information used Diagnostic threshold: NR; appears to be correct diagnosis of melanoma Diagnosis based on: single observer and average (n = 8) Observer qualifications: dermatologist Experience in practice: not described Experience with dermoscopy: not described Any other detail: the d‐ELM Image Acquisition consists of a Leica WILD M‐650 stereomicroscope (Leica Microsystem, Heerbrugg, Switzerland), with a SONY 3CCD DXC‐930P colour camera (Sony Corporation,Tokyo, Japan). The software for image acquisition was DBDERMO MIPS (Dell’Eva/Burroni Studio, Florence/Siena, Italy). The digital image size has a spatial resolution of 768 x 576 pixels and a 24‐bit colour resolution
Target condition and reference standard(s)	Reference standard: histological diagnosis alone Disease‐positive: 42; disease‐negative: 110 Target condition (final diagnoses) Melanoma (invasive): 31; melanoma (in situ): 11 Benign naevus: 110
Flow and timing	Participant exclusions: none reported Index test to reference standard interval: not described
Comparative
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Yes
Was a case‐control design avoided?	Yes
Did the study avoid inappropriate exclusions?	No
Are the included patients and chosen study setting appropriate?	No
Did the study avoid including participants with multiple lesions?	Unclear
		High	High
DOMAIN 2: Index Test Dermoscopy ‐ image‐based
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Unclear
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	No
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	No
Was the test interpretation carried out by an experienced examiner?	Unclear
		Unclear	High
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	Yes
Were the reference standard results interpreted without knowledge of the results of the index tests?	Unclear
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Unclear
		Low	Unclear
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Unclear
Did all patients receive the same reference standard?	Yes
Were all patients included in the analysis?	Yes
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?
		Unclear

Seidenari 1998.

Study characteristics
Patient sampling	Study design: case‐control Data collection: retrospective image selection/prospective interpretation Period of data collection: NR; 4 year period Country: Italy
Patient characteristics and setting	Inclusion criteria: melanomas and benign PSLs from a larger series of PSLs used to develop a new automated classifier; all melanomas with x20 magnification images were included + a random sample of benign lesions with the same magnification. For the larger series, lesions were referred by dermatologists or general physicians because of ≥ 1 PSL that were difficult to interpret on clinical grounds alone, numerous PSLs, or because the patients were at increased risk for melanoma or had had a malignant PSL in the past. Setting: secondary Prior testing: clinical suspicion of malignancy Setting for prior testing: primary; secondary (general dermatology) Exclusion criteria: none reported Sample size (participants): NR Sample size (lesions): number eligible: 917; number included: 100 Participant characteristics: none reported Lesion characteristics: melanoma thickness: ≤ 1 mm : 70.8% (n = 46), < 1 mm 58.5% (n = 38). Mean thickness 0.73 ± 0.69 mm
Index tests	Dermoscopy: no algorithm Method of diagnosis: dermoscopic images; (obtained via videomicroscopy) Prior test data: no further information. "Images appeared in a random sequence on the computer screen, and no information about the patient (such as history, skin site, age of the patient, evolution of the lesion) was given to the evaluators" Diagnostic threshold: clinical diagnosis Diagnosis based on: single observer (n = 2) Observer qualifications: dermatologist Experience in practice: not described Experience with dermoscopy: low; 1 'untrained' dermatologist; high; 1 routinely used videomicroscopy Any other detail: for instrumental examination a 10‐ (39 cases), 20‐ (501 cases), or 50‐fold–magnification (377 cases) was chosen according to the size of the lesion, enabling the whole lesion to be seen on the monitor. For the study, the 31 MM with x20 magnification were selected + a random sample of 59 benign
Target condition and reference standard(s)	Reference standard: histological diagnosis alone Details: describes using "conventional histopathologic criteria" Disease‐positive: 31; disease‐negative: 59 Target condition (final diagnoses) Melanoma (in situ and invasive, or NR): 31 'Benign' diagnoses: 59 "nonmelanoma cases consisted of naevi including dysplastic naevi"
Flow and timing	Participant exclusions: none reported Index test to reference standard interval: not described
Comparative
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Yes
Was a case‐control design avoided?	No
Did the study avoid inappropriate exclusions?	Yes
Are the included patients and chosen study setting appropriate?	No
Did the study avoid including participants with multiple lesions?	Unclear
		High	High
DOMAIN 2: Index Test Dermoscopy ‐ image‐based
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Unclear
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	No
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	No
Was the test interpretation carried out by an experienced examiner?	Yes
		Unclear	High
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	Yes
Were the reference standard results interpreted without knowledge of the results of the index tests?	Unclear
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Unclear
		Low	Unclear
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Unclear
Did all patients receive the same reference standard?	Yes
Were all patients included in the analysis?	Yes
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?
		Unclear

Seidenari 2005.

Study characteristics
Patient sampling	Study design: case series Data collection: retrospective image selection/prospective interpretation Period of data collection: NR Country: Italy
Patient characteristics and setting	Inclusion criteria: patients with melanocytic lesions referred to a pigmented lesion clinic by a dermatologist for examination of a particular lesion or the whole skin; all lesions were excised for clinical, dermoscopic, or cosmetic reasons. Setting: specialist unit Prior testing: clinical and/or dermatoscopic suspicion; patient request for evaluation/excision Setting for prior testing: NR Exclusion criteria: none reported Sample size (participants): NR Sample size (lesions): number included: 603 Participant characteristics: none reported Lesion characteristics: none reported
Index tests	Dermoscopy: pattern analysis Method of diagnosis: dermoscopic images Prior test data: no further information used; images were retrospectively subdivided into 4 groups according to diagnoses performed exclusively by dermoscopy by 2 dermatologists trained in dermoscopy Diagnostic threshold: images grouped according to degree of atypia, with those grade 3 considered to be melanomas dermoscopically, and those at grade 2 as dermoscopically atypical, to be excised to rule out melanoma Diagnosis based on: not clear but appears to be consensus (2 observers) (n = 2); "diagnoses performed exclusively by dermoscopy by two dermatologists trained in dermoscopy and experienced in using polarised light videomicroscopes" Observer qualifications: dermatologist Experience in practice: NR Experience with dermoscopy: high Any other detail images were captured using a digital videomicroscope (VMS‐110A, Scalar Mitsubishi, Tama‐shi, Tokyo, Japan), with a 20‐fold magnification
Target condition and reference standard(s)	Reference standard: histological diagnosis alone (no further details) Disease‐positive: 112; disease‐negative: 491 Target condition (final diagnoses) Melanoma (in situ and invasive, or NR): 112 Benign naevus: 491
Flow and timing	Excluded participants: NR Time interval to reference test: NR Time interval between index test(s): NR
Comparative
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Yes
Was a case‐control design avoided?	Yes
Did the study avoid inappropriate exclusions?	Unclear
Are the included patients and chosen study setting appropriate?	No
Did the study avoid including participants with multiple lesions?	Unclear
		Unclear	High
DOMAIN 2: Index Test Dermoscopy ‐ image‐based
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Yes
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	No
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	No
Was the test interpretation carried out by an experienced examiner?	Yes
		Low	High
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	Yes
Were the reference standard results interpreted without knowledge of the results of the index tests?	Unclear
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Unclear
		Low	Unclear
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Unclear
Did all patients receive the same reference standard?	Yes
Were all patients included in the analysis?	Yes
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?
		Unclear

Seidenari 2007.

Study characteristics
Patient sampling	Study design: case series Data collection: retrospective image selection/prospective interpretation Period of data collection: NR Country: Italy Test set derived NR; the training set consisted of 369 melanocytic lesion images (including 43 MMs); test set comprised 243 images (including 43 MMs)
Patient characteristics and setting	Inclusion criteria: dermoscopic images of melanocytic lesion that had undergone excision Setting: unclear Prior testing: selected for excision (no further detail) Setting for prior testing: unspecified Exclusion criteria: none reported Sample size (participants): NR Sample size (lesions): number eligible: 612; number included: 243 in test set Participant characteristics: none reported Lesion characteristics: MMs of the test set included 8 in situ with mean thickness 0.77 mm
Index tests	Dermoscopy: no algorithm Method of diagnosis: dermoscopic images observed on a computer screen Prior test data: no further information used; clinicians had no access to the clinical image or to clinical data Diagnostic threshold: clinical diagnosis of melanoma Diagnosis based on: single observer (n = 4; results presented per observer, but not identifiable by experience level) Observer qualifications: dermatology registrar (n = 3); dermatologist (n = 1) Experience in practice: NR Experience with dermoscopy: mixed: trained (residents had undergone 6‐month daily training on dermoscopy); high (dermatologist employed dermoscopy on a regular basis)
Target condition and reference standard(s)	Reference standard: histological diagnosis alone (no further details) Disease‐positive: 43; disease‐negative: 200 Target condition (final diagnoses) Melanoma (invasive): 35; melanoma (in situ): 8 Benign naevus: 200
Flow and timing	Participant exclusions: none reported Index test to reference standard interval: not described
Comparative
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Unclear
Was a case‐control design avoided?	Yes
Did the study avoid inappropriate exclusions?	Unclear
Are the included patients and chosen study setting appropriate?	No
Did the study avoid including participants with multiple lesions?	Unclear
		Unclear	High
DOMAIN 2: Index Test Dermoscopy ‐ image‐based
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Unclear
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	No
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	No
Was the test interpretation carried out by an experienced examiner?	Unclear
		Unclear	High
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	Yes
Were the reference standard results interpreted without knowledge of the results of the index tests?	Unclear
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Unclear
		Low	Unclear
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Unclear
Did all patients receive the same reference standard?	Yes
Were all patients included in the analysis?	Yes
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?
		Unclear

Skvara 2005.

Study characteristics
Patient sampling	Study design: case‐control Data collection: retrospective image selection/prospective interpretation Period of data collection: July 1996‐September 1996 Country: Austria
Patient characteristics and setting	Inclusion criteria: consecutive lesions excised due to changes over time during digital dermoscopy follow‐up (appear to be from patients with multiple melanocytic naevi); all lesions were assessed for presence of dermoscopic characteristics and all melanomas + random sample of same number of benign were assessed by dermoscopic algorithms (included in review) Setting: secondary (general dermatology) Prior testing: NR Setting for prior testing: unspecified Exclusion criteria: location/site of lesion ‐ palmar, plantar, facial lesions; lesion size lesions that exceeded maximum field of view of the electronic camera Sample size (participants): NR Sample size (lesions): number included: 126 Participant characteristics: none reported Lesion characteristics none reported
Index tests	Dermoscopy 7PCL and ABCD Method of diagnosis: dermoscopic images presented on a computer screen Prior test data: no further information used Diagnostic threshold: ABCD score > 4.75; 7PCL score > 2 Diagnosis based on: single observer (n = 2) Observer qualifications: dermatologist Experience in practice: assumed high; paper describes assessment of baseline images for dermoscopic criteria by "2 experienced dermatologists"; "additionally, the baseline images of (a subgroup of lesions) were evaluated by 2 blinded investigators). These appear to be separate groups of observers but have assumed similar levels of experience. Experience with dermoscopy: assumed high (as above)
Target condition and reference standard(s)	Reference standard: histological diagnosis alone (but all lesions followed up) Details: "standard: histopathology" following lesion changes over time Disease‐positive: 63; disease‐negative: 63 Target condition (final diagnoses) Melanoma (in situ and invasive, or NR): 63 Benign naevus: 63
Flow and timing	Participant exclusions: none reported Index test to reference standard interval: lesions suggestive of melanoma at baseline were removed at the patient’s initial visit (immediately); the others were followed up for 3‐6 months until lesion changes initiated excision
Comparative
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Yes
Was a case‐control design avoided?	No
Did the study avoid inappropriate exclusions?	No
Are the included patients and chosen study setting appropriate?	No
Did the study avoid including participants with multiple lesions?	Unclear
		High	High
DOMAIN 2: Index Test Dermoscopy ‐ image‐based
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Yes
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?	Unclear
Was the test applied and interpreted in a clinically applicable manner?	No
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	Yes
Was the test interpretation carried out by an experienced examiner?	Unclear
		Low	High
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	Yes
Were the reference standard results interpreted without knowledge of the results of the index tests?	Unclear
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Unclear
		Low	Unclear
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Yes
Did all patients receive the same reference standard?	Yes
Were all patients included in the analysis?	Yes
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?	Yes
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?	Yes
		Low

Soyer 1995.

Study characteristics
Patient sampling	Study design: case series Data collection: unclear Period of data collection: NR Country: Austria
Patient characteristics and setting	Inclusion criteria: PSL, difficult to diagnose on clinical grounds alone Setting: specialist unit (skin cancer/PLC) Prior testing: clinical suspicion Setting for prior testing: secondary (general dermatology); referred by dermatologists or general physicians Exclusion criteria: none reported Sample size (participants): NR Sample size (lesions): number included: 159 Participant characteristics: none reported Lesion characteristics: "23 melanomas with a Breslow index of </= 0.75mm, 13 melanomas with a Breslow index >/=0.76mm and </= 1.5mm, 12 melanomas with a Breslow index >/=1.51mm and </=3.5mm, 2 melanomas with a Breslow index of >/=3.5mm."
Index tests	VI: no algorithm Method of diagnosis: in‐person diagnosis Prior test data: N/A in‐person diagnosis Other test data: dermoscopy undertaken by same clinician(s) subsequent to clinical evaluation Diagnostic threshold: NR Diagnosis based on: n = 2 (1 or 2 per lesion) Observer qualifications: dermatologist Experience in practice: not clearly described; assumed to be high; "the examination was performed by a dermatologist expert in dermoscopy" Experience with dermoscopy: high; "the examination was performed by a dermatologist expert in dermoscopy" Other detail: "Photographic documentation was performed using an incident light stereomicroscope (Wild M 650) equipped with a Minolta XG‐M camera" Dermoscopy: pattern analysis Method of diagnosis: in‐person diagnosis "After application of a drop of immersion oil, each lesion was examined with a hand‐held dermatoscope" Prior test data: clinical examination and/or case notes Diagnostic threshold: criteria included: pigment network, irregular extensions, radial streaming, brown globules, black dots, whitish veil, white scar‐like areas, grey‐blue areas, hypopigmented areas, reticular depigmentation, amongst others Any other detail "After application of a drop of immersion oil, each lesion was examined with a handheld dermatoscope (Heine, Optotechnik, Herrsching, Germany) at a magnification of x10 and with an incident light stereomicroscope (Wild M 650, Heerburg, Switzerland) with 6‐ to 40‐fold magnification."
Target condition and reference standard(s)	Reference standard: histological diagnosis alone Disease‐positive: 65 (41%); disease‐negative: 94 (59%) Target condition (final diagnoses) Melanoma (invasive): 50; melanoma (in situ): 15 BCC: pigmented BCC (3) Sebhorrheic keratosis: 18; Clark's naevus of dysplastic naevus (61 cases); lentigo actinica lentigo (2), pigmented AK (4), angioma (3), angiokeratoma (2)
Flow and timing	Excluded participants: none reported Time interval to reference test: NR
Comparative	Blinding between tests: in‐person diagnosis without and then with dermoscopy Time interval between index test(s): same day; at time of face‐to‐face consultation
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Unclear
Was a case‐control design avoided?	Yes
Did the study avoid inappropriate exclusions?	Unclear
Are the included patients and chosen study setting appropriate?	Unclear
Did the study avoid including participants with multiple lesions?	Unclear
		Unclear	Unclear
DOMAIN 2: Index Test Visual Inspection ‐ in‐person
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Yes
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	No
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	No
Was the test interpretation carried out by an experienced examiner?	Yes
		Low	High
DOMAIN 2: Index Test Dermoscopy ‐ in‐person
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Unclear
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	Yes
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	Unclear
Was the test interpretation carried out by an experienced examiner?	Yes
		Unclear	Unclear
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	Yes
Were the reference standard results interpreted without knowledge of the results of the index tests?	Unclear
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Unclear
		Low	Unclear
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Unclear
Did all patients receive the same reference standard?	Yes
Were all patients included in the analysis?	Yes
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?
		Unclear
DOMAIN 5: Comparative
Was each index test result interpreted without knowledge of the results of other index tests or testing strategies?	No
Was the interval between application of the index tests less than one month?	Yes
Were all tests applied and interpreted in a clinically applicable manner?	Yes
		Low	Low

Soyer 2004.

Study characteristics
Patient sampling	Study design: case series Data collection: retrospective (for expert observer data; previously acquired images prospectively interpreted by 6 inexperienced observers ‐ data excluded as 3/6 medical students) Period of data collection: January‐December 2000 Country: Italy
Patient characteristics and setting	Inclusion criteria: lesions at pigmented lesion clinic considered by experienced dermatologists to merit excision on clinical grounds Setting: specialist unit Prior testing: clinical and/or dermatoscopic suspicion Setting for prior testing: specialist unit Exclusion criteria: none reported Sample size (participants): number included: 225 Sample size (lesions): number included: 231 Participant characteristics: median age 34 years. Male: 110/225 (48.9%) Lesion characteristics: none reported
Index tests	Dermoscopy: no algorithm (study also develops 3PCL but data ineligible due to use of medical student observers) Method of diagnosis: in‐person Prior test data: clinical examination Diagnostic threshold: diagnosis of malignancy (melanoma or BCC) Diagnosis based on: single observer (n = 1) Observer qualifications: dermatologist Experience in practice: high; "experienced dermatologists" Experience with dermoscopy: high; "Each lesion was diagnosed dermoscopically by an experienced dermoscopist"
Target condition and reference standard(s)	Reference standard: histological diagnosis alone (not further described) Disease‐positive: 77; disease‐negative: 154 Target condition (final diagnoses) Melanoma (in situ and invasive, or NR): 68 BCC: 9 'Benign' diagnoses: 154
Flow and timing	Participant exclusions: none reported Index test to reference standard interval: appears consecutive; "before excision, each lesion was diagnosed dermoscopically"
Comparative
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Yes
Was a case‐control design avoided?	Yes
Did the study avoid inappropriate exclusions?	Yes
Are the included patients and chosen study setting appropriate?	No
Did the study avoid including participants with multiple lesions?	Yes
		Low	High
DOMAIN 2: Index Test Dermoscopy ‐ in‐person
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Unclear
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	Yes
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	Unclear
Was the test interpretation carried out by an experienced examiner?	Yes
		Unclear	Unclear
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	Yes
Were the reference standard results interpreted without knowledge of the results of the index tests?	Unclear
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Unclear
		Low	Unclear
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Yes
Did all patients receive the same reference standard?	Yes
Were all patients included in the analysis?	Yes
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?
		Low

Stanganelli 1998a.

Study characteristics
Patient sampling	Study design: case‐control Data collection: retrospective image selection/prospective interpretation Period of data collection: just states 1997 Country: Italy
Patient characteristics and setting	Inclusion criteria: images of PSLs selected from computerised files of the skin cancer clinic Setting: training study; images selected from skin cancer clinic Prior testing: NR Setting for prior testing: unspecified Exclusion criteria: none reported Sample size (participants): NR Sample size (lesions): number included: 30 PSLs Participant characteristics: none reported Lesion characteristics none reported
Index tests	VI: no algorithm Method of diagnosis: clinical photographs Prior test data: no further information used Other test data: dermoscopic images presented to observer subsequent to diagnosis using clinical images alone (images were randomised) Diagnostic threshold: NR Diagnosis based on: average; n = 20 Observer qualifications: dermatologist Experience in practice: not described; Experience with dermoscopy: 30 dermatologists with “experience in ELM but (with) no formal training” attended a seminar on clinical and ELM diagnosis of PSL; 20 then participated in a test of their diagnostic accuracy. A second session on ELM was then held. Dermoscopy: no algorithm Method of diagnosis: dermoscopic images Prior test data: post‐training, clinical image presented alongside dermoscopic image Diagnostic threshold: NR Test observers as described for VI (above) Dermoscopy training: participants undertook 75‐min seminar on the overview of the principles of ELM using digital ELM (D‐ELM) images from the files at the clinic. A second session 45 min long focused on the major aspects of the differential diagnosis of PSL as evaluated by D‐ELM Length of training 2 h Post‐training experience: < 6 months Training format: in‐person teaching
Target condition and reference standard(s)	Reference standard: histological diagnosis alone Target condition (final diagnoses) Melanoma (in situ and invasive, or NR): 10 BCC: 4 Mild/moderate dysplasia: 3; SK: 3; benign naevus: melanocytic naevi‐7 Other: 1 hemangioma, 1 subungunal haemorrhage, 1 plantar intraepidermal haemorrhage
Flow and timing	Excluded participants: none reported Time interval to reference test: NR
Comparative	Blinding between tests: dermoscopic images presented to observer subsequent to diagnosis using clinical images alone (images were randomised) then with dermoscopy Time interval between index test(s): NR
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Unclear
Was a case‐control design avoided?	No
Did the study avoid inappropriate exclusions?	Unclear
Are the included patients and chosen study setting appropriate?	No
Did the study avoid including participants with multiple lesions?	Unclear
		High	High
DOMAIN 2: Index Test Visual inspection ‐ image‐based
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Unclear
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	No
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	No
Was the test interpretation carried out by an experienced examiner?	Unclear
		Unclear	High
DOMAIN 2: Index Test Dermoscopy ‐ image‐based
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Unclear
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	No
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	Yes
Was the test interpretation carried out by an experienced examiner?	Unclear
		Unclear	High
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	Yes
Were the reference standard results interpreted without knowledge of the results of the index tests?	Unclear
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Unclear
		Low	Unclear
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Unclear
Did all patients receive the same reference standard?	Yes
Were all patients included in the analysis?	Yes
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?
		Unclear
DOMAIN 5: Comparative
Was each index test result interpreted without knowledge of the results of other index tests or testing strategies?	Yes
Was the interval between application of the index tests less than one month?	Unclear
Were all tests applied and interpreted in a clinically applicable manner?	No
		Unclear	High

Stanganelli 1999.

Study characteristics
Patient sampling	Study design: case‐control (dermoscopy training study) Data collection: retrospective image selection/prospective interpretation Period of data collection: 15 November 1997‐25 January 1998 Country: Italy
Patient characteristics and setting	Inclusion criteria: PSL images (of melanomas, melanocytic naevi and non‐melanocytic naevi) selected from the dermoscopy files of 2 skin cancer clinics Setting: specialist unit databases Prior testing: NR (all lesions excised) Setting for prior testing: NR Exclusion criteria: none reported Sample size (participants): NR Sample size (lesions): number included: 30 Participant characteristics: none reported Lesion characteristics: melanoma thickness median 0.61 mm, range 0.28‐20 mm
Index tests	Dermoscopy: no algorithm (training course covered principles of clinical and dermoscopic diagnosis of PSLs and referred to a number of diagnostic algorithms, however it did not teach any one particular method of diagnosis; same slides evaluated both pre‐ and post‐dermoscopy training) Method of diagnosis: clinical photographs and dermoscopic images Prior test data: no further information used; pairs of slides were projected onto a screen without access to patient information Diagnostic threshold: correct diagnosis of melanoma Diagnosis based on: average (n = 83 out of 465 professionals who participated in the meetings and workshops over the course of a year) Observer qualifications: dermatologists Experience in practice: mixed; "an average of 10y of general experience in dermatology (range 1‐22years)" Experience with dermoscopy: mixed; "A routine use of ELM was reported by 52 (63%) individuals". 35 (42%) see > 20 PSLs per week Dermoscopy training: attendees could choose from several classes: clinical classification and diagnosis of PSLs; management of patients with PSLs; basic principles of ELM; ELM criteria; ELM diagnosis; limitations of ELM Length of training: 4 + 2 h for each session attended Training format: in‐person teaching; delivered as 1‐day workshops and meetings
Target condition and reference standard(s)	Reference standard: histological diagnosis alone (not further described; original histological diagnosis used) Disease‐positive: 11; disease‐negative: 19 Target condition (final diagnoses) Melanoma (invasive): 10; melanoma (in situ): 1 14 melanocytic naevi; 5 non‐melanocytic lesions
Flow and timing	Participant exclusions: none reported Index test to reference standard interval: not described
Comparative
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Unclear
Was a case‐control design avoided?	No
Did the study avoid inappropriate exclusions?	Unclear
Are the included patients and chosen study setting appropriate?	No
Did the study avoid including participants with multiple lesions?	Unclear
		High	High
DOMAIN 2: Index Test Dermoscopy ‐ image‐based
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Unclear
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	No
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	Unclear
Was the test interpretation carried out by an experienced examiner?	Unclear
		Unclear	High
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	Yes
Were the reference standard results interpreted without knowledge of the results of the index tests?	Unclear
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Unclear
		Low	Unclear
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Unclear
Did all patients receive the same reference standard?	Yes
Were all patients included in the analysis?	Yes
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?
		Unclear

Stanganelli 2000.

Study characteristics
Patient sampling	Study design: case series Data collection: retrospective Period of data collection: 1994‐1996 Country: Italy
Patient characteristics and setting	Inclusion criteria: patients with PSLs referred by dermatologists and GPs either for pre‐surgical assessment or consultation Setting: specialist unit; "skin cancer clinic of Ravenna" Prior testing: patients referred for pre‐surgical assessment or consultation indicating they had had prior tests Setting for prior testing: primary; some patients referred for consultation only; dermoscopy findings are reported back and management decision remains with referring clinician; secondary (general dermatology) Exclusion criteria: none reported Sample size (participants): number eligible: 1556 Sample size (lesions): number eligible: 3372; number included: 3372 Participant characteristics: median age 30 years, range 10‐94; male: 522 (34%) Lesion characteristics: none reported
Index tests	VI: ABCD Method of diagnosis: in‐person diagnosis Prior test data: N/A in‐person diagnosis Other test data: dermoscopic and clinical images subsequently presented separately to observer subsequent to diagnosis using clinical images alone Diagnostic threshold: NR Diagnosis based on: single observer; n = 1 Observer qualifications: NR; described as one of the co‐authors and study based in skin cancer clinic ‐ likely dermatologist Experience in practice: not described Experience with dermoscopy: not described Other detail: a crude clinical image (magn x6 and x10) was recorded in the digital database Dermoscopy: pattern analysis Method of diagnosis: unclear; Patients seen in person but dermoscopic diagnosis made based on digital ELM image (by same clinician as in‐person clinical diagnosis) Prior test data: combined clinical/dermoscopy diagnosis Diagnostic threshold: diagnosis described as based on an integrated synopsis of the patterns most commonly described in the literature (Steiner 1993) and generally associated with known histologic counterparts. Features were assessed described in detail with multiple references, including: presence of pigment network, sharp margins, abrupt edge of pigment network, branched streaks, pseudopods, radial streaming, brown globules, pigment dots, whitish or whitish blue veil, grey‐blue areas, white or depigmented areas, maple leaf areas, milia‐cysts, horny plugs and vascular patterns Test observers as described for VI (above) Experience with dermoscopy: Any other detail: the equipment consisted of a Leica Wild M‐650 stereomiscroscope (Leica AG, Heerbrugg, Switzerland), a Sony 3ccd DXC‐930P colour video camera, an AT‐Vista videographics adapter, and IBM personal computer, a Sony Trinitron Analog PVM‐2043MD monitor, and the DBDERMO MIPS software
Target condition and reference standard(s)	Reference standard: histological diagnosis + follow‐up; histology report of known surgical excisions (n = 262) + a cancer‐registry‐based follow‐up of benign cases (n = 3110) Target condition (final diagnoses) Melanoma (in situ and invasive, or NR): 55; BCC: 43 'Benign' diagnoses: 3274
Flow and timing	Excluded participants: none reported Time interval to reference test: NR
Comparative	Blinding between tests: unclear how dermoscopic diagnosis was made Time interval between index test(s): not clearly reported just indicated that D‐ELM was performed soon after clinical examination
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Yes
Was a case‐control design avoided?	Yes
Did the study avoid inappropriate exclusions?	Yes
Are the included patients and chosen study setting appropriate?	Yes
Did the study avoid including participants with multiple lesions?	No
		Low	High
DOMAIN 2: Index Test Visual Inspection ‐ in‐person
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Yes
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	Yes
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	Yes
Was the test interpretation carried out by an experienced examiner?	Unclear
		Low	Unclear
DOMAIN 2: Index Test Dermoscopy ‐ in‐person
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Yes
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	Unclear
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	Yes
Was the test interpretation carried out by an experienced examiner?	Unclear
		Low	Unclear
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	No
Were the reference standard results interpreted without knowledge of the results of the index tests?	Unclear
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Unclear
		High	Unclear
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Unclear
Did all patients receive the same reference standard?	No
Were all patients included in the analysis?	Yes
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?	Yes
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?
		High
DOMAIN 5: Comparative
Was each index test result interpreted without knowledge of the results of other index tests or testing strategies?	Unclear
Was the interval between application of the index tests less than one month?	Unclear
Were all tests applied and interpreted in a clinically applicable manner?	Unclear
		Unclear	Unclear

Stanganelli 2005.

Study characteristics
Patient sampling	Study design: unclear (likely case series) Data collection: retrospective image selection/prospective interpretation Period of data collection: NR Country: Italy Test set derived: a training set of 22 melanomas and 218 melanocytic naevi was randomised from the dataset. The test set was formed by the complement (the remaining 20 melanomas and 217 naevi). A further subset of images from the original dataset, consisting of 31 melanomas and 103 naevi, was used for the comparison between observers and CAD; derivation of the subset NR
Patient characteristics and setting	Inclusion criteria: melanocytic lesions from patients referred to the Skin Cancer Unit and undergoing clinical and dermoscopic evaluation; images were 'selected' from a larger image database. Potential overlap with Stanganelli 2000 (not possible to determine) Setting: specialist unit; Skin Cancer Unit in Ravenna Prior testing: clinical and/or dermatoscopic suspicion Setting for prior testing: specialist unit (skin cancer/PLC) Exclusion criteria: none reported Sample size (participants): number eligible: 1556 referred/number included: NR Sample size (lesions): number eligible: 3274/number included: 477 melanocytic lesions; 237 in test set and 134 in comparison between CAD and human operators Participant characteristics: none reported Lesion characteristics: melanoma thickness 61.2%, < 0.75 mm
Index tests	VI: no algorithm Method of diagnosis: clinical photographs Prior test data: GPs evaluated only clinical images; dermatologists examined both clinical and dermoscopic images but unclear whether clinical diagnosis was made prior to presentation of dermoscopic images Diagnostic threshold: NR Diagnosis based on: average (n = 6) Observer qualifications: GP 3; dermatologist 3 Experience in practice: NR Experience with dermoscopy: assumed Low for GPs; high for dermatologists ‐ described as “dermatologists with experience in ELM (2 years)” Other detail: digital images included melanocytic lesions evaluated in ELM with a fixed x16 magnification Dermoscopy: no algorithm Method of diagnosis: dermoscopic images (dermatologists only) Prior test data: dermatologists examined both clinical and dermoscopic images Diagnostic threshold: NR Test observers as described for VI (above)
Target condition and reference standard(s)	Reference standard: histological diagnosis + cancer registry All included lesions underwent histology but some were identified using a cancer‐registry‐based follow‐up of benign diagnoses. Target condition (final diagnoses) Melanoma (in situ and invasive, or NR): 42 in full sample; 31 in CAD vs human observer interpretation and 20 in test set 'Benign' diagnoses: 435 melanocytic naevi; 103 in CAD‐observer complement and 217 in test set
Flow and timing	Excluded participants: none reported Time interval to reference test: NR
Comparative	Blinding between tests: dermatologists examined both clinical and dermoscopic images but unclear whether clinical diagnosis was made prior to presentation of dermoscopic images Time interval between index test(s): obtained from patients undergoing clinical and dermoscopic evaluation
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Unclear
Was a case‐control design avoided?	Unclear
Did the study avoid inappropriate exclusions?	Unclear
Are the included patients and chosen study setting appropriate?	No
Did the study avoid including participants with multiple lesions?	Unclear
		Unclear	High
DOMAIN 2: Index Test Visual inspection ‐ image‐based
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Unclear
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	No
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	No
Was the test interpretation carried out by an experienced examiner?	Yes
		Unclear	High
DOMAIN 2: Index Test Dermoscopy ‐ image‐based
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Unclear
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	No
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	No
Was the test interpretation carried out by an experienced examiner?	Yes
		Unclear	High
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	Yes
Were the reference standard results interpreted without knowledge of the results of the index tests?	Unclear
Expert opinion (with no histological confirmation) was not used as a reference standard	Unclear
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Unclear
		Low	Unclear
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Unclear
Did all patients receive the same reference standard?	No
Were all patients included in the analysis?	Yes
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?
		High
DOMAIN 5: Comparative
Was each index test result interpreted without knowledge of the results of other index tests or testing strategies?	Unclear
Was the interval between application of the index tests less than one month?	Yes
Were all tests applied and interpreted in a clinically applicable manner?	No
		Low	High

Stanganelli 2015.

Study characteristics
Patient sampling	Study design: case series Data collection: retrospective image selection/prospective interpretation Period of data collection: July 2010‐July 2012 Country: Italy
Patient characteristics and setting	Inclusion criteria: melanocytic lesions excised at the Skin Cancer Unit on the basis of clinical and/or dermoscopic changes at follow‐up suggesting a malignancy Setting: specialist unit; "conducted at the Skin Cancer Unit at the ‘Istituto Scientifico Romagnolo per lo Studio e la Cura dei Tumori’ (IRST IRCCS), in Ravenna/Forli and Meldola" Prior testing: changes on digital monitoring; lesions showing clinical or dermoscopic changes on follow‐up Setting for prior testing: specialist unit Exclusion criteria: lack of baseline and follow‐up dermoscopic images; lack of RCM images; lack of histology Sample size (participants): number included: 70 Sample size (lesions): number included: 70 Participant characteristics: mean age; women 39 years; men 40 years. Male: 54%. History of melanoma/skin cancer: 37%. Total naevus counts, 27 (39%) with > 50 melanocytic naevi, 33 (47%) with 10–50 naevi; and 10 (14%) with < 10 naevi. Fitzpatrick phototype I‐II = 19 (27%); type III to IV = 50 (73%). Median follow‐up was 25 months (range 3–134 months) Lesion characteristics: lesion site head/neck 7.1%; trunk: 80%; upper limbs/shoulder: 1.4%; lower limbs/hip: 11.4%. Melanoma thickness median 0.4 mm (0.2‐1 mm). Lesion size: mean at baseline 8 mm (range 2–22 mm); mean at follow‐up 9 mm (range 3–24 mm)
Index tests	Dermoscopy: revised 7PCL (for follow‐up purposes) Method of diagnosis: dermoscopic images; baseline images assessed using standard 7PCL and compared to follow‐up images to determine criteria indicating significant change Prior test data: baseline dermoscopic image Diagnostic threshold: presence of "Major change" (asymmetrical structural and chromatic changes, or the appearance of melanoma‐specific criteria, i.e. major or minor criteria on original 7‐point checklist as per Argenziano 1998). Revised approach referenced to Argenziano 2010. ("Minor change" assigned if there was only symmetrical change in structural or chromatic pattern; "moderate change" if either structural or chromatic changes were asymmetrical, but there were no melanoma‐specific criteria; and "no change" was assigned if all variables remained constant, with a tolerance of major axis change of 2 mm (Beer 2011; Terushkin 2012)) Diagnosis based on: unclear; n = NR for dermoscopy Observer qualifications: NR but likely dermatologists (RCM images in same study were evaluated jointly by 3 expert dermatologists who had no knowledge of the clinical, dermoscopic or histopathology information) Experience in practice: not described Experience with dermoscopy: not described
Target condition and reference standard(s)	Reference standard: histological diagnosis alone Details: "histopathological diagnosis was based on the consensus of at least two out of three board‐certified pathologists" Disease‐positive: 12; disease‐negative: 58 Target condition (final diagnoses) Melanoma (invasive): 11; melanoma (in situ): 1 'Benign' diagnoses: 55 melanocytic naevi (79%) and three non‐melanocytic lesions (4%)
Flow and timing	Participant exclusions: none reported Index test to reference standard interval: appears consecutive, "Lesions showing clinical and/or dermoscopic aspects suggesting a malignancy are excised. RCM imaging is performed before surgical excision."
Comparative
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Unclear
Was a case‐control design avoided?	Yes
Did the study avoid inappropriate exclusions?	Yes
Are the included patients and chosen study setting appropriate?	No
Did the study avoid including participants with multiple lesions?	Yes
		Unclear	High
DOMAIN 2: Index Test Dermoscopy ‐ image‐based
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Yes
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	No
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	Yes
Was the test interpretation carried out by an experienced examiner?	Unclear
		Low	High
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	Yes
Were the reference standard results interpreted without knowledge of the results of the index tests?	Unclear
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Yes
		Low	Low
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Yes
Did all patients receive the same reference standard?	Yes
Were all patients included in the analysis?	Yes
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?
		Low

Stolz 1994a.

Study characteristics
Patient sampling	Study design: case series Data collection: retrospective image selection/prospective interpretation Period of data collection: from 1989‐1991 Country: Germany Test set derived: 157 cases were randomly divided into a test and training set.
Patient characteristics and setting	Inclusion criteria: equivocal melanocytic skin lesions < 9 x13 mm, melanoma tumour thickness of ≤ 1 mm and melanoma Clark's ≤ level III Setting: secondary (general dermatology); Univerisity of Munich Department of Dermatology Prior testing: selected for excision (no further detail) Setting for prior testing: NR Exclusion criteria: none reported Sample size (participants): NR Sample size (lesions): number eligible: 650 cases/number included: 157 lesions Participant characteristics: none reported Lesion characteristics: melanoma thickness: 50 ≤ 0.4 mm; 30 ≤ 0.75 mm; 15 ≤ 1 mm
Index tests	Dermoscopy: ABCD Method of diagnosis: dermoscopic images; colour prints examined for 31 dermoscopic features, most listed in the guidelines of the Consensus Conference of Surface Microscopy held in Hamburg in 1989 (Bahmer 1990); described as a "blind study" Prior test data: no further information used Diagnostic threshold: > 5.45; multivariate analysis of training set data identified 8 features with the lowest P values; the total dermoscopic score (TDS) was then developed based on: asymmetry score x 1.3 + Border score x 0.1 + Colour score x 0.5 + Differential structure score x 0.5. New formula then evaluated on the test set of images. Diagnosis based on: single observer (n = 1) Observer qualifications: NR; co‐author, assumed to be dermatologist Experience in practice: not described Experience with dermoscopy: not described
Target condition and reference standard(s)	Reference standard: histological diagnosis alone Details: histology undertaken by 2 independent histopathologists Disease‐positive: test set = 48; disease‐negative: test set = 31 Target condition (final diagnoses) Melanoma (invasive): 85; melanoma (in situ): 10 'Benign' diagnoses: 62 melanocytic naevi; 17 junctional; 40 compound; 5 dermal
Flow and timing	Participant exclusions: none reported Index test to reference standard interval: not described
Comparative
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Yes
Was a case‐control design avoided?	Yes
Did the study avoid inappropriate exclusions?	No
Are the included patients and chosen study setting appropriate?	No
Did the study avoid including participants with multiple lesions?	Unclear
		High	High
DOMAIN 2: Index Test Dermoscopy ‐ image‐based
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Yes
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	No
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	Yes
Was the test interpretation carried out by an experienced examiner?	Unclear
		Low	High
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	Yes
Were the reference standard results interpreted without knowledge of the results of the index tests?	Unclear
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Unclear
		Low	Unclear
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Unclear
Did all patients receive the same reference standard?	Yes
Were all patients included in the analysis?	Yes
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?
		Unclear

Tan 2009.

Study characteristics
Patient sampling	Study design: case‐control (dermoscopy training study) Data collection: retrospective image selection/prospective interpretation Period of data collection: NR Country: UK
Patient characteristics and setting	Inclusion criteria: test series of images of melanomas and benign lesions; source of images NR Setting: not described; training images Prior testing: selected for excision (no further detail) Setting for prior testing: unspecified Exclusion criteria: none reported Sample size (participants): NR Sample size (lesions): number included: 30 Participant characteristics: none reported Lesion characteristics: none reported
Index tests	Dermoscopy: (modified) pattern analysis Method of diagnosis: clinical photographs and dermoscopic images Prior test data: participants presented with a test card printed on A4 laminated paper for each lesion, each consisting of 1 macroscopic and 1 dermatoscopic image Diagnostic threshold: excise or not (algorithm not further described) Diagnosis based on: average (n = 6; all based at same university hospital); the study authors presented 2x2 based on adding each 2x2 cell together for all observers; to avoid double counting of lesions for this review, all 2x2 cells were divided by 6 to get average result. Observer qualifications: dermatology specialist registrar 3; dermatologist 3 Experience in practice: mixed Experience with dermoscopy: low; before the study, none had routinely used a dermatoscope. Dermoscopy training: participants received an online tutorial (www.dermatoscopy.org) teaching the Modified Pattern Analysis Diagnostic Algorithm (Steiner 1987a; Carli 2003a) and was given a dual polarizing LED dermatoscope to use in clinical practice for 10 months. At the end of the study, the test‐card questionnaire was repeated. Length of training NR; online tutorial Post‐training experience: 10 months Training format: online
Target condition and reference standard(s)	Reference standard: histological diagnosis alone (not further described) Disease‐positive: 15; disease‐negative: 15 Target condition (final diagnoses) Melanoma (in situ and invasive, or NR): 15 Other: 15 (9 naevi, 1 blue naevus, 3 seborrhoeic keratoses, 1 lentigo and 1 vascular lesion)
Flow and timing	Participant exclusions: none reported Index test to reference standard interval: not described
Comparative
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Unclear
Was a case‐control design avoided?	No
Did the study avoid inappropriate exclusions?	Unclear
Are the included patients and chosen study setting appropriate?	No
Did the study avoid including participants with multiple lesions?	Unclear
		High	High
DOMAIN 2: Index Test Dermoscopy ‐ image‐based
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Yes
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	No
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	Yes
Was the test interpretation carried out by an experienced examiner?	No
		Low	High
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	Yes
Were the reference standard results interpreted without knowledge of the results of the index tests?	Unclear
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Unclear
		Low	Unclear
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Unclear
Did all patients receive the same reference standard?	Yes
Were all patients included in the analysis?	Yes
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?
		Unclear

Tenenhaus 2010.

Study characteristics
Patient sampling	Study design: case‐control Data collection: retrospective image selection/prospective interpretation Period of data collection: not described Country: France
Patient characteristics and setting	Inclusion criteria: dermoscopic images of all melanoma lesions recorded on two PSL databases + random sample of benign naevus Setting: secondary (general dermatology) Prior testing: NR Setting for prior testing: unspecified Exclusion criteria: none reported Sample size (participants): NR Sample size (lesions): number included: 227 Participant characteristics: none reported Lesion characteristics: none reported
Index tests	Dermoscopy: no algorithm Method of diagnosis: dermoscopic images Prior test data: clinical photographs Diagnostic threshold: clinical diagnosis of melanoma and excise decision; presence of ABCD and "malignancy‐predictive" dermoscopic features were assessed (dichotomic answer) and diagnosis (melanoma, dysplastic or benign lesion) and therapeutic decision (dichotomic answer, excision/non‐excision) given Diagnosis based on: single and average (n = 5); observers assessed lesion images independently; sensitivity and specificity also presented for "pooled" advice Observer qualifications: dermatologist Experience in practice: high; "senior dermatologists" Experience with dermoscopy: not described; assumed high
Target condition and reference standard(s)	Reference standard: histological diagnosis + other Histology: excision and histopathology of lesions considered to be melanomas (n = 32), dysplastic lesions (n = 118) and some of those considered benign (n = 15) Disease‐positive: 32; disease‐negative: 165 Other: "lesions considered benign were not surgically excised"; assume observer diagnosis was used Disease‐positive: 0; disease‐negative: 62 Target condition (final diagnoses) Melanoma (invasive): 28; lentigo maligna 4 Dysplastic naevus 118; blue benign naevus 2; congenital benign naevus 5; junctional and dermic benign naevus 7; palmar‐plantar benign naevus 1; 'benign naevus' not excised: 62
Flow and timing	Participant exclusions: none reported Index test to reference standard interval: not described
Comparative
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Yes
Was a case‐control design avoided?	No
Did the study avoid inappropriate exclusions?	Yes
Are the included patients and chosen study setting appropriate?	No
Did the study avoid including participants with multiple lesions?	Unclear
		High	High
DOMAIN 2: Index Test Dermoscopy ‐ image‐based
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Unclear
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	Unclear
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	Unclear
Was the test interpretation carried out by an experienced examiner?	Yes
		Unclear	Unclear
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	No
Were the reference standard results interpreted without knowledge of the results of the index tests?	Unclear
Expert opinion (with no histological confirmation) was not used as a reference standard	No
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Unclear
		High	High
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Unclear
Did all patients receive the same reference standard?	No
Were all patients included in the analysis?	Unclear
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?	Unclear
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?
		High

Troyanova 2003.

Study characteristics
Patient sampling	Study design: case‐control Data collection: retrospective image selection/prospective interpretation Period of data collection: NR Country: NR
Patient characteristics and setting	Inclusion criteria: images of PSLs selected for a dermoscopy training study Setting: training study Prior testing: NR Setting for prior testing: NR Exclusion criteria: lesions that were > 13 mm were not included Sample size (participants): NR Sample size (lesions): number included: 50 lesions Participant characteristics: none reported Lesion characteristics: melanoma thickness: ≤ 1 mm: 100%
Index tests	VI: no algorithm Method of diagnosis: clinical photographs and dermoscopic images Other test data: dermoscopic images presented to observer subsequent to diagnosis using clinical images alone. Prior test data: no further information used Diagnostic threshold: NR Diagnosis based on: average; n = 32 Observer qualifications: dermatologist Experience in practice: high experience or ‘Expert’ Experience with index test: low experience/novice users; experienced in PSL field but not ELM Dermoscopy: no algorithm; possibly pattern analysis Method of diagnosis: dermoscopic images Prior test data: no further information used. Previously made diagnosis based on clinical images only; dermoscopic images presented after all clinical diagnoses had been made Diagnostic threshold: NR Test observers as described for VI (above) Dermoscopy training: The group of 32 volunteer dermatologists had no formal training in the use of ELM, but had good theoretical knowledge and personal experience; participated in a teaching course comprised 6 h of teaching on 2 consecutive days. The training was based on the presentation of several hundred slides with oral explanation of the ELM criteria. Tests were performed at the beginning and end of the teaching course. Length of training 2 days (12 h in total) Post‐training experience: < 6 months Training format: in‐person teaching
Target condition and reference standard(s)	Reference standard: histological diagnosis alone Disease‐positive: 25; disease‐negative: 25 Target condition (final diagnoses) Melanoma (invasive): 25 'Benign' diagnoses: 50 "not melanoma"
Flow and timing	Excluded participants: none reported Time interval to reference test: NR
Comparative	Blinding between tests: dermoscopic images presented after all clinical diagnoses had been made Time interval between index test(s): NR
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Unclear
Was a case‐control design avoided?	No
Did the study avoid inappropriate exclusions?	Unclear
Are the included patients and chosen study setting appropriate?	No
Did the study avoid including participants with multiple lesions?	Unclear
		High	High
DOMAIN 2: Index Test Visual inspection ‐ image‐based
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Unclear
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	No
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	No
Was the test interpretation carried out by an experienced examiner?	Yes
		Unclear	High
DOMAIN 2: Index Test Dermoscopy ‐ image‐based
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Unclear
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	No
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	No
Was the test interpretation carried out by an experienced examiner?	Yes
		Unclear	High
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	Yes
Were the reference standard results interpreted without knowledge of the results of the index tests?	Unclear
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Unclear
		Low	Unclear
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Unclear
Did all patients receive the same reference standard?	Yes
Were all patients included in the analysis?	Yes
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?
		Unclear
DOMAIN 5: Comparative
Was each index test result interpreted without knowledge of the results of other index tests or testing strategies?	Unclear
Was the interval between application of the index tests less than one month?	Yes
Were all tests applied and interpreted in a clinically applicable manner?	No
		Unclear	High

Unlu 2014.

Study characteristics
Patient sampling	Study design: case series Data collection: retrospective image selection/prospective interpretation Period of data collection: January 2008‐January 2010 Country: Turkey
Patient characteristics and setting	Inclusion criteria: melanocytic lesions excised at Ankara University Department of Dermatology Pigmented Lesion Clinic Setting: specialist unit; Ankara University Department of Dermatology Pigmented Lesion Clinic Prior testing: selected for excision (no further detail) Setting for prior testing: specialist unit (skin cancer/PLC) Exclusion criteria: location/site of lesion facial, nail and volar acral lesions were excluded; non‐melanocytic appearance Sample size (participants): number included: 115 Sample size (lesions): number included: 115 Participant characteristics: mean age: 38.72 years (+/‐ 18.46 years). Male: 56 (49%) Lesion characteristics: lesion site: 100% trunk and limbs. Melanoma thickness: 10 (41.7%) < 0.75 mm; 14 (58.3%) ≥ 0.75 mm
Index tests	VI: no algorithm; appears to be original clinical diagnosis at time of lesion presentation Method of diagnosis: in‐person diagnosis. Appears to be diagnosis on presentation Prior test data: N/A; in‐person diagnosis Other test data: dermoscopic images presented to different observers Diagnostic threshold: NR Diagnosis based on: unclear; for VI appears to be single examiner at time of clinic diagnosis (n = NR); dermoscopic images "scored by three other experienced dermatoscopists" Observer qualifications: NR; assumed dermatologists; described as experienced dermatoscopists Experience in practice: unclear for clinic diagnosis; dermatoscopists described as "experienced" Experience with index test: described as "experienced" Dermoscopy 3‐point rule; 7PCL; ABCD; CASH algorithm Method of diagnosis: dermoscopic images Prior test data: no further information used; clinical image evaluation appears to be separate from dermoscopy interpretation Diagnostic threshold: ABCD score ≥ 5.45 highly suggestive for melanoma; 7‐point score ≥ 3; 3‐point score 2 or 3 criteria present; CASH algorithm ≥ 8 Observers: as described for Visual Inspection above
Target condition and reference standard(s)	Reference standard: histological diagnosis alone Disease‐positive: 24; disease‐negative: 91 Target condition (final diagnoses) Melanoma (in situ and invasive, or NR): 24 'Benign' diagnoses: 91 melanocytic benign lesions; 37 (32.2%) dermal naevi; 15 (13%) Clark's naevi; 14 (12.2%) compound naevi; 13 (11.3%) blue naevi; 6 (5.2%) SN; 4 (3.5%) congenital melanocytic naevi; 2 (1.7%) junctional naevi
Flow and timing	Excluded participants: none reported Time interval to reference test: NR
Comparative	Blinding between tests: used in‐person clinical diagnosis; clinical image evaluation appears to be separate from dermoscopy interpretation Time interval between index test(s): appear to be consecutively applied but not described
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Yes
Was a case‐control design avoided?	Yes
Did the study avoid inappropriate exclusions?	Yes
Are the included patients and chosen study setting appropriate?	No
Did the study avoid including participants with multiple lesions?	Yes
		Low	High
DOMAIN 2: Index Test Visual Inspection ‐ in‐person
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Unclear
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	Yes
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	No
Was the test interpretation carried out by an experienced examiner?	Unclear
		Unclear	High
DOMAIN 2: Index Test Dermoscopy ‐ image‐based
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Yes
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?	Unclear
Was the test applied and interpreted in a clinically applicable manner?	No
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	Yes
Was the test interpretation carried out by an experienced examiner?	Yes
		Low	High
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	Yes
Were the reference standard results interpreted without knowledge of the results of the index tests?	Unclear
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Yes
		Low	Low
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Unclear
Did all patients receive the same reference standard?	Yes
Were all patients included in the analysis?	Yes
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?
		Unclear
DOMAIN 5: Comparative
Was each index test result interpreted without knowledge of the results of other index tests or testing strategies?	Yes
Was the interval between application of the index tests less than one month?	Unclear
Were all tests applied and interpreted in a clinically applicable manner?	No
		Unclear	High

Viglizzo 2004.

Study characteristics
Patient sampling	Study design: case series Data collection: NR Period of data collection: NR Country: Italy
Patient characteristics and setting	Inclusion criteria: PSLs examined at the Dermoscopy Service and undergoing excision; a modified version of Kenet's risk stratification approach for dermoscopy (Ascierto 1998) was used to select high‐ and very high‐risk lesions for excision; medium‐ and low‐risk lesions were excised based on cosmetic or functional reasons. (2x2 data extracted only for melanocytic subgroup). Setting: specialist unit (skin cancer/PLC). Dermoscopy service at a university department (Department of Endocrinologic and Metabolic Disease) Prior testing: clinical suspicion of malignancy without dermatoscopic suspicion Setting for prior testing: specialist unit (skin cancer/PLC) Exclusion criteria: none reported Sample size (participants): number eligible: 349 participants; number included: NR Sample size (lesions): number eligible: 520 lesions; number included: 79 lesions excised included in the final analysis Participant characteristics: none reported Lesion characteristics: none reported
Index tests	VI: no algorithm Method of diagnosis: in‐person diagnosis Prior test data: unclear Diagnostic threshold: NR; correct diagnosis of melanoma Diagnosis based on: single observer (n = NR; "All dermoscopic evaluations were performed by the same operators") Observer qualifications: NR; "each lesion was... diagnosed clinically and dermoscopically" at the dermoscopy service Experience in practice: not described Experience with dermoscopy: not described; assumed high as diagnosis at 'Dermoscopy Service' Dermoscopy: no algorithm; appears to be based on pattern analysis Method of diagnosis: in‐person diagnosis Prior test data: clinical examination and/or case notes Diagnostic threshold: lesion classification based on typical dermoscopic features: lesions with a pigment network and any of the classical dermoscopic features specific for melanoma, i.e. pseudopods, radial streaming or blue‐grey veil, were classified as very high‐risk. Lesions with a pigment network and dermoscopic features that might suggest melanoma but often seen in atypical naevi were classified as high‐risk. Test observers: as described above
Target condition and reference standard(s)	Reference standard: histological diagnosis alone Target condition (final diagnoses) Melanoma (invasive): 12 Melanocytic lesion: 67
Flow and timing	Excluded participants: none reported Time interval to reference test: NR
Comparative	Blinding between tests: used in‐person diagnosis without and then with dermoscopy Time interval between index tests: not clearly reported but assumed consecutive as both recorded at Dermoscopy Service
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Yes
Was a case‐control design avoided?	Yes
Did the study avoid inappropriate exclusions?	Unclear
Are the included patients and chosen study setting appropriate?	No
Did the study avoid including participants with multiple lesions?	Unclear
		Unclear	High
DOMAIN 2: Index Test Visual Inspection ‐ in‐person
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Yes
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	Yes
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	No
Was the test interpretation carried out by an experienced examiner?	Unclear
		Low	High
DOMAIN 2: Index Test Dermoscopy ‐ in‐person
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Yes
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	Yes
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	Yes
Was the test interpretation carried out by an experienced examiner?	Yes
		Low	Low
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	Yes
Were the reference standard results interpreted without knowledge of the results of the index tests?	Unclear
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Unclear
		Low	Unclear
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Unclear
Did all patients receive the same reference standard?	Yes
Were all patients included in the analysis?	Yes
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?
		Unclear
DOMAIN 5: Comparative
Was each index test result interpreted without knowledge of the results of other index tests or testing strategies?	No
Was the interval between application of the index tests less than one month?	Yes
Were all tests applied and interpreted in a clinically applicable manner?	Yes
		Low	Low

Wells 2012.

Study characteristics
Patient sampling	Study design: case‐control Data collection: retrospective image selection/prospective interpretation Period of data collection: NR Country: USA
Patient characteristics and setting	Inclusion criteria: pigmented lesions (melanomas and benign pigmented lesions) selected from a repository of lesions amassed during an acquisition study conducted by MELA Sciences Inc for the US Food and Drug Administration Setting: company database (MELA Sciences Inc) of lesion images Prior testing: selected for excision (no further detail) Setting for prior testing: NR Exclusion criteria: none reported Sample size (participants): NR Sample size (lesions): number included: 47 Participant characteristics: none reported Lesion characteristics: none reported
Index tests	Dermoscopy: no algorithm Method of diagnosis: dermoscopic images Prior test data: clinical images and detailed clinical history; observers "viewed the images and a detailed case history for each lesion but were unaware of the MelaFind recommendations" Diagnostic threshold: clinical diagnosis of melanoma or not; decision to biopsy the lesion Diagnosis based on: average (n = 39) Observer qualifications: dermatologist Experience in practice: not described Experience with dermoscopy: not described
Target condition and reference standard(s)	Reference standard: histological diagnosis alone Details: "Lesions were biopsied in toto and evaluated by a panel of dermatopathologists who were unaware of the MelaFind recommendations" Disease‐positive: 23/disease‐negative: 24 Target condition (final diagnoses) Melanoma (in situ and invasive, or NR): 23 'Benign' diagnoses: 24
Flow and timing	Participant exclusions: none reported Index test to reference standard interval: consecutive; "prior to biopsy of the lesion, photographs of the lesion were taken"
Comparative
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Unclear
Was a case‐control design avoided?	No
Did the study avoid inappropriate exclusions?	Unclear
Are the included patients and chosen study setting appropriate?	No
Did the study avoid including participants with multiple lesions?	Unclear
		High	High
DOMAIN 2: Index Test Dermoscopy ‐ image‐based
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Unclear
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	No
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	No
Was the test interpretation carried out by an experienced examiner?	Unclear
		Unclear	High
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	Yes
Were the reference standard results interpreted without knowledge of the results of the index tests?	Unclear
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Yes
		Low	Low
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Yes
Did all patients receive the same reference standard?	Yes
Were all patients included in the analysis?	Yes
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?
		Low

Westerhoff 2000.

Study characteristics
Patient sampling	Study design: case‐control (for lesion selection; study was an RCT of dermoscopy training for PCPs) Data collection: retrospective Period of data collection: NR Country: Australia
Patient characteristics and setting	Inclusion criteria: clinically atypical PSLs; 50 invasive melanomas and 50 non‐melanomas randomly selected from the Sydney Melanoma Unit PSLs image database Setting: specialist unit (lesion selection) Prior testing: selected for excision or followed up Setting for prior testing: specialist unit (skin cancer/PLC) Exclusion criteria: none reported Sample size (participants): number included: NR Sample size (lesions): number included: 100 Participant characteristics: none reported Lesion characteristics: median Breslow thickness 0.6 mm
Index tests	VI: no algorithm Method of diagnosis: clinical photographs Prior test data: unclear; all participants "were instructed not to look at the surface microscopic image until they had scored the clinical image" Diagnostic threshold: NR Diagnosis based on: average (n = 37; 74 practising primary care practitioners randomised to dermoscopy education intervention or not). Diagnoses were recorded for both groups of GPs at baseline (pre‐test) and after the training intervention had been administered to the intervention group (post‐test), resulting in 8 sets of 2x2 data based on interpretation of the same set of 100 lesions; post‐test data for the intervention group of GPs was used for the VI analysis. Observer qualifications: GP Experience in practice: considered to be low; only practitioners who had had no formal training with surface microscopy and did not use a surface microscope in their clinical practice were included. Experience with dermoscopy: low experience/novice users (non‐training arm); "Trained" for the intervention arm Other detail: camera designed for close‐up clinical photography (Elicar Macrolens, Japan) Dermoscopy: no algorithm (non‐training arm); Menzies criteria (training/intervention arm) Method of diagnosis: dermoscopic images Prior test data: diagnosis was first based on the clinical image and then the dermoscopic image for each lesion. Diagnostic threshold: NR; intervention arm instructed in Menzies criteria Test observers: as above Any other detail: dermoscopy at x10 magnification with a Dermphot camera (Heine Ltd) using oil at the skin‐lens interface. Dermoscopy training: the education intervention included provision of the Menzies and colleagues pictorial atlas which reportedly describes the Menzies approach to dermoscopy diagnosis of melanoma (Menzies 1996); they also attended a 1‐h presentation on dermoscopy reviewing the Menzies approach and including a quiz based on images of 25 different PSLs Post‐training experience: < 6 months; the median interval between pretest and education intervention was 46 days (range 5‐155). Median interval from education intervention to post‐test was 23 days (range 2‐54). Training format: in‐person teaching; written materials
Target condition and reference standard(s)	Reference standard: histological diagnosis + follow‐up Histology: all the lesions except 2 had been excised after photography and subjected to histopathological examination. Disease‐positive: 50; disease‐negative: 48 Clinical follow‐up + histology of suspicious lesions: the two benign PSLs that had not been excised were monitored over a longer period of time and had shown no morphological change. Length of follow‐up: NR; disease‐positive: 0/disease‐negative: 2 Target condition (final diagnoses) Melanoma (invasive): 50/'Benign' diagnoses: 50
Flow and timing	Excluded participants: none reported Time interval to reference test: "All the lesions except two had been excised after photography"
Comparative	Blinding between tests: observers were instructed not to look at the dermoscopy image until they had scored the clinical image Time interval between index test(s): NR; lesions described as "excised after photography" therefore assumed consecutive
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Yes
Was a case‐control design avoided?	No
Did the study avoid inappropriate exclusions?	Unclear
Are the included patients and chosen study setting appropriate?	No
Did the study avoid including participants with multiple lesions?	Unclear
		High	High
DOMAIN 2: Index Test Visual inspection ‐ image‐based
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Unclear
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	No
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	No
Was the test interpretation carried out by an experienced examiner?	Unclear
		Unclear	High
DOMAIN 2: Index Test Dermoscopy ‐ image‐based
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Unclear
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?	Yes
Was the test applied and interpreted in a clinically applicable manner?	No
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	Yes
Was the test interpretation carried out by an experienced examiner?	No
		Unclear	High
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	Yes
Were the reference standard results interpreted without knowledge of the results of the index tests?	Unclear
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Unclear
		Low	Unclear
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Yes
Did all patients receive the same reference standard?	No
Were all patients included in the analysis?	Yes
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?	Unclear
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?
		High
DOMAIN 5: Comparative
Was each index test result interpreted without knowledge of the results of other index tests or testing strategies?	No
Was the interval between application of the index tests less than one month?	Yes
Were all tests applied and interpreted in a clinically applicable manner?	No
		Low	High

Winkelmann 2016.

Study characteristics
Patient sampling	Study design: case‐control Data collection: retrospective image selection/prospective interpretation Period of data collection: NR Country: NR
Patient characteristics and setting	Inclusion criteria: images of PSLs previously analysed by a digital classifier MSDSLA; method of selection of the 12 NR Setting: unclear; images selected for a dermoscopy conference Prior testing: NR Setting for prior testing: unspecified Exclusion criteria: none reported Sample size (participants): NR Sample size (lesions): number included: 12 Participant characteristics: none reported Lesion characteristics: none reported
Index tests	VI: no algorithm Method of diagnosis: clinical photographs Prior test data: unclear Other test data: dermoscopic images presented to observer subsequent to diagnosis using clinical images alone Diagnostic threshold: NR; biopsy decision Diagnosis based on: average (n = 70) Observer qualifications: dermatologist Experience in practice: not described; recruited “dermatologists at a dermoscopy conference”. No further details Other detail: study authors report that practitioners with a particular interest in skin cancer or technology may have chosen to attend this conference and/or self‐selected to take part in the study. Dermoscopy: no algorithm Method of diagnosis: dermoscopic images Prior test data: clinical images provided Diagnostic threshold: NR; biopsy decision Test observers as described for VI (above)
Target condition and reference standard(s)	Reference standard: histological diagnosis alone Disease‐positive: 5; disease‐negative: 7 Target condition (final diagnoses) Melanoma (invasive): 3; melanoma (in situ): 2 Mild/moderate dysplasia: 7 low‐grade dysplastic naevi
Flow and timing	Excluded participants: none reported Time interval to reference test: NR
Comparative	Blinding between tests: dermoscopic images presented to observer subsequent to diagnosis using clinical images alone Time interval between index test(s): same day; at time of face‐to‐face consultation
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Unclear
Was a case‐control design avoided?	No
Did the study avoid inappropriate exclusions?	Unclear
Are the included patients and chosen study setting appropriate?	No
Did the study avoid including participants with multiple lesions?	Unclear
		High	High
DOMAIN 2: Index Test Visual inspection ‐ image‐based
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Unclear
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	No
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	No
Was the test interpretation carried out by an experienced examiner?	Unclear
		Unclear	High
DOMAIN 2: Index Test Dermoscopy ‐ image‐based
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Unclear
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	No
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	No
Was the test interpretation carried out by an experienced examiner?	Unclear
		Unclear	High
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	Yes
Were the reference standard results interpreted without knowledge of the results of the index tests?	Unclear
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Unclear
		Low	Unclear
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Unclear
Did all patients receive the same reference standard?	Yes
Were all patients included in the analysis?	Yes
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?	Yes
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?
		Unclear
DOMAIN 5: Comparative
Was each index test result interpreted without knowledge of the results of other index tests or testing strategies?	No
Was the interval between application of the index tests less than one month?	Yes
Were all tests applied and interpreted in a clinically applicable manner?	No
		Low	High

Zalaudek 2006.

Study characteristics
Patient sampling	Study design: case series Data collection: retrospective image selection/prospective interpretation Period of data collection: February 2003‐January 2004 Country: Naples, Italy
Patient characteristics and setting	Inclusion criteria: excised, equivocal and nonequivocal, pigmented and non‐PSLs with good image quality and melanin or haemoglobin pigmentation in all or part of the lesion Setting: specialist unit; specialised PLC database Prior testing: selected for excision (no further detail) Setting for prior testing: specialist unit Exclusion criteria: none reported Sample size (participants): NR Sample size (lesions): eligible: 2621; included 150 (+ 15 lesions used for training purposes) Participant characteristics: none reported Lesion characteristics 37/165 (26%) considered equivocal on clinical and dermoscopic grounds Thickness/depth: mean Breslow 0.9 mm
Index tests	Dermoscopy: 3PCL Method of diagnosis: dermoscopic images, "optimized for colour, brightness and contrast by using Adobe photoshop standards" Prior test data: age, site, and gender provided Diagnostic threshold: ≥ 1 criterion present indicates malignancy (asymmetry ‐ in colour and/or structure, not in shape; atypical network; pigment network with thick lines and irregular holes; and blue white structures; presence of any blue and/or white colour within the lesion Diagnosis based on: average (n = 150 out of 170 participating observers, who finished all 15 training cases and performed at least one evaluation of the main set of images (test set). Participation was open to all individuals regardless of professional profile and experience in dermoscopy; study was advertised through personal communication, e‐mail correspondences, adverts during congresses and courses, as well as via the website dermoscopy.org. Observer qualifications: for full sample of 170: dermatologists (n = 125); GPs (n = 15); other professionals in the field of skin lesions (n = 12); medical students (n = 7); other medical specialty (n = 11) Experience in practice: not described Experience with dermoscopy: mixed; 146/170 (86%) reported some experience with dermoscopy; 24 with no dermoscopy experience, 45 (26%) with > 5 years' experience. Dermoscopy training: a web‐based tutorial was provided to describe the concept of the 3PCL of dermoscopy including complete definitions of criteria and example images. Following web‐based tutorial, observers initially scored a random sample of 15 images, receiving real‐time feedback for that case as judged by an expert observer. Training format: online
Target condition and reference standard(s)	Reference standard: histological diagnosis alone (no further details) Target condition (final diagnoses) Melanoma (invasive): 18; melanoma (in situ): 11 BCC: 18 79 melanocytic naevi; 26 seborrhoeic keratoses; 8 vascular tumours and 3 dFs
Flow and timing	Participant exclusions: poor‐quality index test image as exclusion criterion Index test to reference standard interval: not described
Comparative
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Yes
Was a case‐control design avoided?	Yes
Did the study avoid inappropriate exclusions?	Yes
Are the included patients and chosen study setting appropriate?	No
Did the study avoid including participants with multiple lesions?	Unclear
		Low	High
DOMAIN 2: Index Test Dermoscopy ‐ image‐based
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Yes
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	No
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	Yes
Was the test interpretation carried out by an experienced examiner?	Unclear
		Low	High
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	Yes
Were the reference standard results interpreted without knowledge of the results of the index tests?	Unclear
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Unclear
		Low	Unclear
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Unclear
Did all patients receive the same reference standard?	Yes
Were all patients included in the analysis?	No
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?
		High

3PCL: three‐point checklist; 7FFM: seven features for melanoma; 7PCL: seven‐point checklist; ABCD(E): asymmetry, border, colour, differential structures (enlargement); AJCC: American Joint Committee on Cancer; AK: actinic keratosis; AMN: acral melanocytic naevi; BCC: basal cell carcinoma; BD: Bowen’s disease; CD: compact disc; CAD: computer‐assisted diagnosis; CASH: colour, architecture, symmetry and homogeneity; CM: cutaneous melanoma; CMM: cutaneous malignant melanoma;DF: dermatofibroma; ELM: epiluminescence microscopy; GP: general practitioner; IDS: International Dermoscopy Society; IQR: interquartile range; LK: lichen sclerosis; LP: lichen planus; LS: lentigo simplex; MM: malignant melanoma; MN: melanocytic naevi; MSDSLA: multispectral digital skin lesion analysis device; N/A: not applicable; NR: not reported; PCP: primary care provider; PLC: pigmented lesion clinic; PSL: pigmented skin lesion; RCM: reflectance confocal microscopy; RCT: randomised controlled trial; SCC: squamous cell carcinoma; SD: standard deviation; SK: seborrhoeic keratosis; SN: Spitz naevi;SSM: superficial spreading melanoma; VI: visual inspection.

Characteristics of excluded studies [ordered by study ID]

Study	Reason for exclusion
Ahnlide 2013	Ineligible index test; 'clinical diagnosis' study
Akasu 1996	No 2x2 data only describing the dermoscopic features present in the lesions
Al Jalbout 2013	Small sample size; case study
Alendar 2009	Reference standard: only 7 reported verified histologically
Altamura 2006	Derivation study: Study was looking for characteristics associated with acral melanoma; does not give 2x2 for overall diagnosis
Altamura 2010	Wrong target condition
Amirnia 2016	Wrong target condition
Antonio 2013	Wrong target condition; atypical naevi does not fall within our definition of disease positive
Antoszewski 2015	Sample size too small; all excised lesions were benign No data for 2x2
Aoyagi 2010	Sample size too small
Argenziano 1997	Wrong study population; only melanoma included
Argenziano 1999	Wrong study population; only includes melanoma
Argenziano 2002	Not a primary study
Argenziano 2003	Table V gives se/sp data for 108 lesions but can't derive the number of melanoma for this subset of the original 128 Contacted study authors 10 May 2016 and 24 June 2016
Argenziano 2004a	Only lesions with vascular structures included; presence of 10 different characteristics assessed. 2x2 would be possible
Argenziano 2004b	Not a primary study; letter
Argenziano 2008	Ineligible index test; surveillance/monitoring study
Argenziano 2010	Ineligible index test and no 2x2 data. Test used for follow‐up looking at dermoscopic features of melanomas diagnosed 1 year after follow‐up
Argenziano 2011a	Wrong target condition and sample size too small; only 2 melanomas
Argenziano 2011b	Wrong target condition; 5 melanoma metastases included as disease positive
Argenziano 2012	Ineligible reference standard; no follow‐up of test‐negatives
Armstrong 2011	Ineligible reference standard; no reference standard results presented for the screened lesions; just compares naked‐eye judgements with dermoscopy
Ascierto 1998	The data presented do not contribute to the review ‐ no 2x2 Duplicate or related publication; data included in Ascierto 2003
Ascierto 2000	No 2x2 data. For excised lesions, study cross‐tabulates ELM high/very high risk classification against some histological classification (Table 2). Number disease positive = 580 (2x2: 504, 79, 76, 2072); 580 not mentioned anywhere else in paper Contacted authors 10 May 2016 and 24 June 2016
Ascierto 2003	Not a primary study
Bafounta 2001	Not a primary study; systematic review
Bajaj 2016	Unclear reference standard for benign diagnoses
Bauer 2005	Ineligible index test; follow‐up/monitoring study
Bauer 2006	Ineligible index test; dermoscopy used to improve histopathology diagnosis
Benati 2015	Assesses individual lesion characteristics only
Benelli 2000b	Insufficient data to populate 2x2 table; only inter‐rater reliability data given (n = 25); authors have published much larger evaluations of 7FFM and ABCD
Benvenuto‐Andrade 2006	Diagnostic confidence rather than accuracy; no 2x2 data
Benvenuto‐Andrade 2007	Agreement on lesion characterisation; not test accuracy; no 2x2 data
Binder 1997	Training study; only ROC curves/AUC presented pre and post‐training; no 2x2 data Contacted study authors 10 May 2016 and 24 June 2016
Blum 2003c	Not a primary study
Blum 2004c	Not a primary study; comment paper
Blum 2004d	Letter only; limited data presented ‐ evaluates '3‐colour' rule as developed By MacKie 2002 (excluded as assessment of individual lesion features only)
Blum 2004e	Not a primary study; letter
Blum 2006	Wrong target condition; differentiates melanocytic from non‐melanocytic lesions only
Blum 2011	Wrong study population; mucosal lesions only
Blum 2014	Sample size too small; case studies
Boespflug 2015	Wrong study population; study aim is estimate the efficacy of an online spaced educational training for dermoscopy
Bono 2001	Aim of the study is to determine what features are present in amelanotic cutaneous melanoma; no 2x2 data
Borsari 2010	Paper focuses on diagnostic prediction of dermoscopic island for early melanoma, however the Methods describe the calculation of the total dermoscopy score and the 7‐PCL score; mean scores on each checklist per lesion type are then presented Contacted study authors but no reply
Bowns 2006	Ineligible index test; teledermatology study
Braun 2000	Derivation study: this is a pilot study on the new 'wobble sign' in ELM no training/test sets used
Braun 2007	Assesses individual lesion characteristics only
Braun‐Falco 1990	Not a test accuracy study; no 2x2 data
Brown 2000	Not a primary study; systematic review
Buhl 2012	Ineligible index test; follow up/monitoring Duplicate or related publication; same participants as Haenssle 2010 #191
Bystryn 2003	Not a primary study; letter
Cabrijan 2008	Can't get 2x2; reports % correct diagnoses for each different lesion classification and not % misdiagnosed as melanoma or melanomas missed Study states, "Dermatoscopic diagnosis were conformable with pathohistological diagnosis in 75 cases (72.82%) out of 103. The highest conformation was in diagnosing melanoma, in 5 out of 6 cases (83.3%).", which would give us sensitivity. Asked study authors for data on numbers mis‐classified as melanoma, i.e. false positive. Study author replied 5 July 2016 with some data but not sufficient to allow 2x2.
Canpolat 2011	Derivation study: looks at dermoscopic characteristics of acral lesions; only 4 suspicious lesions excised
Cardenas 2009	Wrong study population; includes participants with palpable lesions; not all suspected of having skin cancer
Carli 1998	Sample size too small; se/sp data are based on sample with only 4 MM
Carli 2000	Wrong target condition; only lesions histologically classified as common naevi or naevi with architectural disorder with/without cytological atypia were considered for the study.
Carli 2003c	Sample size too small
Carli 2004a	Sample size too small; < 5 MM per arm. No 2x2 data
Carli 2004b	Ineligible index test; can only estimate 2x2 for the full time period 1997‐2001 across all observers, however dermoscopy was only introduced routinely in 1998 so some diagnoses prior to that will have been with VI alone, and observers were classed as dermoscopy ’users’ (those working in PLCs) and nonusers (general dermatology). Author passed away; unable to make contact with co‐authors
Carli 2005	Study presents % MM correctly classified by naked eye +/‐ dermoscopy but doesn't give any detail on FPs. No 2x2 data Tried to contact study authors to ask whether available anywhere and/or are these lesions included in any subsequent publications? Author passed away; unable to make contact with co‐authors
Carlos‐Ortega 2007	Gives se/sp for VI and dermoscopy in the English abstract. 68 participants/70 lesions were included but only 36 seem to have had VI results and all underwent dermoscopy. 2 observers performed each test blinded to each other. Table I gives 22 with BCC and 11 with melanoma overall (number disease positive not reported for those with VI results), but using either or both of these numbers with the se/sp provided does not give the same PPV and NPV as given by the authors Data not clearly presented for 2x2; translator suggested alternative but still does not work out to what is in paper; tried contacting authors twice, no reply as of 28 July 2016
Carroll 1998	Derivation study: proposes new dermoscopic criteria for diagnosis of BCC; no 2x2 data
Chen 2013	Wrong test observer
Ciudad‐Blanco 2014	Wrong study population; includes melanoma only. No 2x2 data
de Giorgi 2006	Sample size insufficient; < 5 cases of participants with a final melanoma diagnosis
De Giorgi 2011	Duplicate or related publication. Assesses same lesions as in Carli 2003b but different observers
de Troya‐Martin 2008	Wrong study population; only MM included
Delfino 1997	Derivation study: only reports association of each characteristics with disease positive/disease negative, not 2x2
Di Chiacchio 2010	Wrong target condition; excluding nail bed melanoma Insufficient data to extract for a 2x2 table
Di Stefani 2007	Sample size insufficient; < 5 malignant
Dummer 1995	Assesses individual lesion characteristics only
Elwan 2016	Derivation study: sample size insufficient for 2x2 table
Fabbrocini 2008	Insufficient data provided for each index test to populate 2x2 table We contacted study authors to ask for a cross tabulation of each clinician's diagnosis (e.g. at threshold of ≥ 3 on 7PCL) against the histological diagnosis and/or a cross tabulation of the remote diagnosis against the face‐to‐face diagnoses? Study author replied 30 June 2016: unable to access data needed
Ferrara 2002	Ineligible index test. This study looks at histopathological and dermoscopic disagreements not necessarily looking at how well dermoscopy differentiates between benign and malignant diagnosis.
Fidalgo 2003	Insufficient data for 2x2 table Duplicate or related publication; appears to be superseded by Serrao 2006 Paper provides % of MM and of dysplastic naevi with algorithm scores of ≥ 5.5 and > 7 Contacted study author 10 May 2016 and 24 June 2016 to request the same information for the remaining 127 lesions in the study and ask whether any of the 247 lesions included in this study, overlap with the 652 reported in Serrao 2006 (#1144)
Fruhauf 2012	Ineligible reference standard; 35/219 underwent histology; 13 followed‐up; 171 expert clinical diagnosis
Fueyo‐Casado 2009	Ineligible reference standard; < 50% of the study population received histology as a test. No information given on those who were followed up
Giacomel 2005	Wrong study population; only BCC included
Giacomel 2014	Sample size insufficient
Giannotti 2004	Not a primary study; a review
Gill 2015	Derivation study Inadequate sample size
Gilmore 2009	Derivation study: principle of lacunarity has been looked at before but not this particular application/approach to it Ineligible reference standard It is possible to get 2x2 for 'standard dermoscopy criteria' however dermoscopy‐negative were not excised and assumed benign; 201/312 underwent excision so theoretically eligible
Grichnik 2003	Sample size insufficient
Grichnik 2004	Not a primary study; editorial
Guillod 1996	Derivation study developing new algorithm
Gunduz 2003	Sample size insufficient; case study
Hacioglu 2013	Wrong target condition; does not provide sufficient data for detection of melanoma
Haenssle 2006	Ineligible index test; surveillance study estimating accuracy of different approaches to follow‐up
Haenssle 2010	Insufficient data to populate 2x2 table; does not report specificity Duplicate or related publication; same participants as Haenssle 2010 #191
Haspeslagh 2016	Assesses individual lesion characteristics only Insufficient data to populate 2x2 table
Henning 2007	Derivation study: first application of CASH algorithm
Henning 2008	Derivation study developing new algorithm
Herschorn 2012	Not a primary study; systematic review
Hirata 2011	Wrong target condition Ineligible index test
Hoffmann 2003	Derivation study: uses leave one out cross‐validation procedure Only giving ROC values not able to extract a 2x2 table
Hoorens 2016	Ineligible index test Ineligible reference standard; no info on numbers undergoing histology; and no follow‐up reported for benign‐appearing lesions Insufficient data to populate 2x2 table
Ishioka 2009	Ineligible index test
Iyatomi 2006	Derivation study: uses leave‐one‐out procedure and same lesions and tumour extraction method as Iyatomi 2006 Insufficient data to populate 2x2 table
Iyatomi 2008	Derivation study: the performance was evaluated by averaging both combinations (training and test sets) they did not present the data separately; uses leave‐one‐out procedure Insufficient data to populate 2x2 table Not test accuracy; compares automated with manual extraction of tumour area
Johr 2002	Not a primary study
Kawabata 1998	Derivation study: aim of the study was to correlate findings between dermoscopy and histology findings of acral melanoma Insufficient data to populate 2x2 table Not test accuracy
Kawabata 2001	Derivation study. Aim of the study is to correlate findings between dermoscopy and histology findings of acral melanoma. Wrong study population; MM of the nail bed
Kefel 2012	Derivation study: no test set, first use of polarised light dermoscopy, various neural networks tested Insufficient data to populate 2x2 table
Kenet 1994	Not a primary study Insufficient data to populate 2x2 table; not an accuracy study
Kittler 2002	Not a primary study; systematic review
Kittler 2006	Conference abstract
Koga 2011	Ineligible reference standard; ˜23% of participants have their final diagnosis reached by histopathology 43/191
Korotkov 2012	Not a primary study; narrative review
Lallas 2015	Derivation study: develops new algorithm and does not use separate training/test sets of lesions
Liebman 2011	Not a primary study; comment
Liebman 2012	Not a primary study; comment
Lipoff 2008	Wrong target condition; study does not differentiate MM from benign/other but looks to identify lesion characteristics that might help id those at risk for MM
Liu 2012	Derivation study: asymmetry detection; 10‐fold cross validation Insufficient data to populate 2x2 table
Lorentzen 1999b	Insufficient data to populate 2x2 table Contacted study authors 10 May 2016 and 24 June 2016) to request number of melanomas that were included in the study so that we could estimate the 2x2 contingency tables using the se/sp data provided? Also to ask if there was overlap in the lesions included here with those included in the Lorentzen 2000 study? (see also author Qs for the 2000 study)
Luttrell 2012	Ineligible test observer; accuracy data only given for lay‐persons not interested in this population of test observers
MacKie 1971	Insufficient data to populate 2x2 table; only gives % with correct diagnosis rather than numbers misclassified as malignant
MacKie 2002	Presence of ≥ 3 colours on dermoscopy
Markowitz 2015	Wrong target condition; does not report sufficient data for detection of melanoma
Massi 2001	Assesses individual lesion characteristics only
Mayer 1997	Not a primary study; systematic review
Menzies 1996a	Only given the se/sp of individual characteristics; lesions make up the training set for Menzies 1996 (#1971)
Menzies 1999	Not a primary study
Menzies 2000	Wrong target condition; BCC only
Menzies 2001	Ineligible index test; monitoring purposes
Mun 2016	Ineligible reference standard; only 37% of benign group underwent adequate reference standard
Nathansohn 2007	Insufficient data to populate 2x2 table Not test accuracy; follow‐up study
Navarrete‐Dechent 2016	Wrong target population; 2x2 for BCC only
Pan 2008	Derivation study: looking to ID characteristics associated with superficial BCC; 2x2 could be extracted for combination of 3 selected characteristics. Dermoscopic features selected based on prior studies but only patients with 3 diagnoses included: BCC, intra‐epidermal carcinoma, and psoriasis
Panasiti 2009	Assesses individual lesion characteristics only Ineligible reference standard. Of the 1543 lesions analysed only 321 received histopathology diagnosis. The accuracy data were based on this (only 20%) not sure what happened to the 80% of participants as no mention of follow‐up.
Pazzini 1996	Insufficient data to populate 2x2 table
Pehamberger 1987	Insufficient data to populate 2x2 table Not test accuracy. This is a descriptive paper defining dermoscopic criteria. It is not a study testing accuracy of dermoscopy. From the authors' final sign off it looks like part 2 of this paper may have details on accuracy (Steiner 1987a).
Pellacani 2002	Not a primary study
Pellacani 2006	Derivation study: looks at detection of asymmetry between clinicians and computer Insufficient data to populate 2x2 table; 2x2 could be derived for overall asymmetry or border cut‐off but not overall diagnosis
Pellacani 2007	Derivation study: looking at blue hue
Pellacani 2009	Wrong target condition; focus is on identifying SN from melanoma and "Clark" naevi and it is looking to derive useful RCM characteristics. Although some data are given in the text for an RCM score of > 3 it is difficult to work out which are FP and which FN.
Peris 2002a	Wrong study population; only patients with BCC diagnosis included
Peris 2002b	Not a primary study
Phan 2010	Insufficient data to populate 2x2 table Not test accuracy investigating dermoscopic features of acral melanoma including of the nail apparatus; no accuracy data given
Piccolo 2002b	Not a primary study Insufficient data to populate 2x2 table; No breakdown of index test results and ref standard
Piccolo 2004	Ineligible index test
Piccolo 2006	Sample size insufficient; 3 MMs, but also 1 lentigo and 14 dysplastic naevus; data not presented to allow se/sp estimation Derivation for hypoluminescence microscopy
Pizzichetta 2001a	Wrong study population; population in study only those with malignant disease
Pizzichetta 2001b	Insufficient data to populate 2x2 table; observer agreement only
Pizzichetta 2007	Wrong study population; only patients with melanoma included
Pizzichetta 2010	Sample size insufficient; case study
Pizzichetta 2013	Presence of negative pigmented network
Pralong 2012	Wrong study population; only melanoma patients included
Provost 1998	Insufficient data to populate 2x2 table Not test accuracy; only reports concordance
Rader 2014	Assesses individual lesion characteristics only Insufficient data to populate 2x2 table
Rajpara 2009	Not a primary study; systematic review
Reggiani 2015	Not a primary study; systematic review keratinocyte skin cancer
Rigel 1997	Not a primary study
Ronger 2002	Assesses individual lesion characteristics only
Rosendahl 2012a	Assesses individual lesion characteristics only
Rosendahl 2012b	Not a primary study
Rossi 2000	Ineligible reference standard; unclear reference standard in disease‐negative
Rubegni 2002	Not a primary study
Rubegni 2005a	Not a primary study; editorial
Rubegni 2005b	Not a primary study
Rubegni 2010	Derivation study: uses leave‐one‐out procedure Insufficient data to populate 2x2 table
Sahin 2004	Derivation study: Assesses individual lesion characteristics only Insufficient data to populate 2x2 table; no accuracy data given, study looking at dermoscopic features of lentigo maligna
Saida 2002	Descriptive study looking at presence (%) of certain features. Not looking at accuracy. Has paragraph on diagnostic value of this specific feature quoting se/sp but this is based upon unpublished observations and the data are not given in this paper.
Saida 2004	Assesses individual lesion characteristics only
Sakakibara 2010	Only looking at different vascular structures
Salerni 2011	Sample size insufficient; < 5 cases
Salerni 2012	Ineligible index test; surveillance study Insufficient data to populate 2x2 table
Salerni 2013	Not a primary study; systematic review of surveillance with digital dermoscopy
Salvio 2011	Not a primary study Sample size insufficient
Sanchez‐Martin 2012	Wrong study population; only BCC cases
Savk 2004	Not a primary study; letter
Sawada 2013	Not a primary study
Sboner 2003	Derivation study: describes 10‐fold cross‐validation process for training/testing classifier
Schulz 2001	Wrong target condition; melanoma metastases
Scope 2015	Not a primary study
Segura 2009	Ineligible index test; RCM evaluation
Seidenari 2004	No data to populate 2x2 table just ROC curve values given. TABLE 5 provides AUC values for each diagnosis for both formats and observers. We contacted study authors to request data in 2x2 format , e.g. for melanoma 'certain' against final diagnosis and for melanoma 'certain or fairly certain' against final diagnosis but received no reply.
Seidenari 2006a	Wrong study population; assessing best means of follow‐in up patients with previous melanoma ‐ total body exam versus only lesions > 2 cm. No melanoma identified
Seidenari 2006b	Looks like this study is only looking at asymmetry judgement
Seidenari 2012	Looks at individual lesion characteristics to distinguish Melanoma in situ, also gives mean ABCD and 7PCL scores Insufficient data to populate 2x2 table Table 3 provides mean ABCD and 7PCL scores. We contacted study authors to request cross tabulation of results with each checklist at 'standard' thresholds against final diagnosis? e.g. ABCD > 4.75 and > 5.45 for MIS and benign groups 7PCL: presence ≥ 2 characteristics and ≥ 3 characteristics but received no reply.
Seidenari 2013	Ineligible index test
Serrao 2006	Ineligible index test
Sgouros 2014	Ineligible index test
Shakya 2012	Wrong target condition; SCC in situ is included in target condition
Shitara 2014	Assesses individual lesion characteristics only
Shitara 2015	Wrong study population; includes only melanoma
Sondak 2015	Not a primary study; comment paper
Soyer 1987	Insufficient data to populate 2x2 table; not test accuracy
Soyer 2001	Not a primary study; editorial
Stanganelli 1998b	Insufficient data to populate 2x2 table; can't derive specificity; only gives exact diagnoses for MM and 2 benign categories and not number benign misdiagnosed as MM
Steiner 1987a	Insufficient data to populate 2x2 table; study only reports % correct diagnosis per lesion type for dermoscopy and does not list incorrect diagnoses
Steiner 1987b	Insufficient data to populate 2x2 table; only given the correct diagnosis for malignant
Steiner 1993	Derivation study: assesses individual lesion characteristics only
Stephens 2013	Sample size insufficient
Stoecker 2009a	Derivation study: translucency Insufficient data to populate 2x2 table; data presented only as ROC curve and AUC
Stoecker 2009b	Not a primary study
Stoecker 2011	Derivation study: uses leave one out Insufficient data to populate 2x2 table; data presented only as ROC curve and AUC
Stolz 2002	Not a primary study
Stratigos 2007	Ineligible reference standard Insufficient data to populate 2x2 table
Stricklin 2011	Assesses individual lesion characteristics only
Strumia 2003	Conference abstract; letter only
Tasli 2012	Not a primary study; systematic review looking at frequency of publications ion dermoscopy
Teban 2003	Wrong study population; classification of Clark naevi into 12 types Insufficient data to populate 2x2 table
Terstappen 2007	Wrong study population; includes only BCC ‐ looking for BCC characteristics on Siascope Derivation study; first application of Siascope to pigmented BCC; 21/25 lesions were BCCs
Terushkin 2010a	Sample size insufficient; only 2 invasive SCC Insufficient data to populate 2x2 table
Terushkin 2010b	Insufficient data to populate 2x2 table Not test accuracy ‐ reports final diagnoses of those excised over a number of time periods and benign‐malignant ratio
Tromme 2012	Inadequate reference test for disease‐negatives; expert diagnosis only
Tschandl 2012	Ineligible index test; differentiating melanocytic from non‐melanocytic lesions
Tschandl 2015a	Ineligible test observer; medical students
Tschandl 2015b	Assesses individual lesion characteristics only
Ulrich 2015	Wrong target condition; does not provide sufficient data for evaluation of melanoma
Van der Leest 2011	Inadequate reference test for test‐negatives; expert diagnosis only
Van der Rhee 2010	Ineligible reference standard; < 50% of disease‐negative have an adequate reference standard
Van der Rhee 2011	Sample size insufficient; < 5 cases
Vasili 2010	Conference abstract
Verduzco‐Martinez 2013	Wrong study population; only BCC
Vestergaard 2008	Not a primary study; systematic review; check reference list
Wang 2008	Insufficient data to populate 2x2 table; not test accuracy; no details of misdiagnoses of benign lesions as malignant
Warshaw 2009a	Insufficient data to populate 2x2 table Duplicate or related publication; subgroup of participants from Warshaw 2010 Study presents diagnostic accuracy of teledermatology and clinic diagnosis in comparison to histopathology; we need the underlying 2x2 contingency tables (see Warshaw 2010 for author response)
Warshaw 2009b	Insufficient data to populate 2x2 table Duplicate or related publication; subgroup of participants from Warshaw 2010 Study presents diagnostic accuracy of teledermatology and clinic diagnosis in comparison to histopathology; we need the underlying 2x2 contingency tables (see Warshaw 2010 for author response)
Warshaw 2010	Insufficient data to populate 2x2 table Study presents diagnostic accuracy of teledermatology and clinic diagnosis in comparison to histopathology. Study author only able to provide numbers test‐positive and ‐negative for melanoma and not for the final 2 cells of the 2x2; data provided showed higher sensitivity for melanoma as the primary diagnosis rather than as the ‘aggregate’ diagnosis and the 2x2 using the study authors' data and the accuracy figures from the paper showed more TP from the primary diagnosis as opposed to the aggregate
Weismann 2002	Not a primary study
Wilkes 2010	Not a primary study
Winkelmann 2015a	Duplicate or related publication
Winkelmann 2015b	Duplicate or related publication
Witkowski 2016	Wrong target population
Yadav 1993	Insufficient data to populate 2x2 table; not test accuracy
Yamaura 2005	Derivation study: gene amplification in acral lesions
Yelamos 2016	Not a primary study
Yoo 2015	Conference abstract
Youl 2007a	Ineligible index test; evaluates 'clinical diagnosis' Contacted study authors for more information. Replied that dermoscopy used in some but not all lesions
Youl 2007b	Ineligible index test; evaluates 'clinical diagnosis' Contacted study authors for more information. Replied that dermoscopy used in some but not all lesions
Zaballos 2013	Wrong study population; they do not have enough benign cases to include as full report
Zalaudek 2010	Not a primary study; editorial
Zell 2008	Sample size insufficient; case study
Zortea 2014	Derivation study: although data are divided into training and test sets, the test set data are used more than once over 20 realisations of each model, especially the melanomas, for which the same 10 are used in each realisation
Zou 2001	Not a primary study; study uses results from Stolz 1994a Insufficient data to populate 2x2 table; just showing ROC curves

7FFM: seven features for melanoma; 7PCL: seven‐point checklist; ABCD(E): asymmetry, border, colour, differential structures (enlargement); AUC: area under the curve; BCC: basal cell carcinoma; CASH: colour, architecture, symmetry and homogeneity; ELM: epiluminescence microscopy; FN: false negative; FP: false positive; MM: malignant melanoma; se/sp: sensitivity/specification; PLC: pigmented lesion clinic; PPV: positive predictive values; NPV: negative predictive value; RCM: reflectance confocal microscopy; ROC: receiver operating characteristic; SCC: squamous cell carcinoma; SN: Spitz naevi; TN: true negative; TP: true positive; VI: visual inspection

Differences between protocol and review

We set out to review visual inspection and dermoscopy for the detection of melanoma in a single review, however due to the volume of evidence identified, we prepared two separate reviews, one for visual inspection alone and one for dermoscopy. This review of dermoscopy includes data for the accuracy of visual inspection but only where both tests were evaluated in the same study (direct comparisons).

We changed the primary objectives and primary target condition from detection of cutaneous invasive melanoma alone, to the detection of cutaneous invasive melanoma and atypical intraepidermal melanocytic variants, as the latter is more clinically relevant to the practicing clinician. We included the detection of the target condition of invasive melanoma alone as a secondary objective instead. We also amended the primary objectives to conduct separate analyses by in‐person/image‐based diagnosis rather than to investigate the effect on accuracy as a secondary objective, as originally proposed in the generic protocol. We took this decision very early in the review process, based on the fact that a diagnosis based on a dermoscopic image or clinical photograph cannot approximate a face‐to‐face patient and clinician consultation.

We have tailored secondary objectives to the individual test, with three objectives added: to determine the diagnostic accuracy of individual algorithms for dermoscopy; to determine the effect of observer experience; and to determine the effect on accuracy of observer training in dermoscopy.

Sources of heterogeneity that could be investigated (as listed in the protocol), were restricted due to lack of data.

We amended the text to clarify that studies available only as conference abstracts would be excluded from the review unless full papers could be identified; studies available only as conference abstracts do not allow a comprehensive assessment of study methods or methodological quality.

We excluded rather than included studies using cross‐validation, such as ’leave‐one‐out’ cross‐validation, as these methods are not sufficiently robust and are likely to produce unrealistic estimates of test accuracy.

To improve clarity of methods, this text from the protocol, "We will include studies developing new algorithms or methods of diagnosis (i.e., derivation studies), if they use a separate independent ’test set’ of participants or images to evaluate the new approach. We will also include studies using other forms of cross validation, such as 'leave‐one‐out' cross‐validation (Efron 1983). We will note for future reference (but not extract), any data on the accuracy of lesion characteristics individually, e.g., the presence or absence of a pigment network or detection of asymmetry" has been replaced with the following: "We included studies developing new algorithms or methods of diagnosis (i.e., derivation studies), were included if they:

• used a separate independent 'test set' of participants or images to evaluate the new approach; or

• investigated lesion characteristics that had previously been suggested as associated with melanoma and the study reported accuracy based on the presence or absence of particular combinations of characteristics.

We excluded studies if they:

• used a statistical model to produce a data driven equation, or algorithm based on multiple diagnostic features, with no separate test set;

• used cross‐validation approaches such as 'leave‐one‐out' cross‐validation (Efron 1983);

• evaluated the accuracy of the presence or absence of individual lesion characteristics or morphological features, with no overall diagnosis of malignancy;

• reported accuracy data for ‘clinical diagnosis’ with no clear description as to whether the reported data related to visual inspection alone or included dermoscopy in all study participants;

• were based on the experience of a skin cancer‐specific clinic, where dermoscopy may or may not have been used on an individual participant basis."

We proposed to supplement the database searches by searching the annual meetings of appropriate organisations (e.g. British Association of Dermatologists' Annual Meeting, American Academy of Dermatology Annual Meeting, European Academy of Dermatology and Venereology Meeting, Society for Melanoma Research Congress, World Congress of Dermatology, European Association of Dermato Oncology), however due to volume of evidence retrieved from database searches and time restrictions, we were unable to do this.

For quality assessment, we further tailored the QUADAS‐2 tool according to the review topic. In terms of analysis, we did not restrict analysis to per‐patient data due to lack of data.

Contributions of authors

JD was the contact person with the editorial base.
 JD co‐ordinated contributions from the co‐authors and wrote the final draft of the review.
 SB conducted the literature searches.
 JD, NC, LFR, DT, KYW, RBA, RA, and MF screened papers against eligibility criteria.
 JD and NC obtained data on ongoing and unpublished studies.
 JD, NC, LFR, DT, KYW, RBA, RA, and MF appraised the quality of papers.
 JD, NC, LFR, DT, KYW, RBA, RA, and MF extracted data for the review and sought additional information about papers.
 JD entered data into Review Manager 5 (Review Manager 2014).
 JD, MJG and JJD analysed and interpreted data.
 JD, JJD, NC, LFR, YT and CD worked on the methods sections.
 JD, FW, DT, KYW, RBA, RA, MF, RNM and HCW drafted the clinical sections of the background and responded to the clinical comments of the referees.
 JD, JJD, CD and YT responded to the methodology and statistics comments of the referees.
 KG was the consumer co‐author and checked the review for readability and clarity, as well as ensuring outcomes were relevant to consumers.
 JD is the guarantor of the update.

Disclaimer

This project presents independent research supported by the National Institute for Health Research, via Cochrane Infrastructure funding to the Cochrane Skin Group and Cochrane Programme Grant funding, and the NIHR Birmingham Biomedical Research Centre at the University Hospitals Birmingham NHS Foundation Trust and the University of Birmingham. The views and opinions expressed therein are those of the authors and do not necessarily reflect those of the Systematic Reviews Programme, NIHR, NHS or the Department of Health and Social Care.

Sources of support

Internal sources

• No sources of support supplied

External sources

• NIHR Systematic Review Programme, UK.

  This project was funded by an NIHR Cochrane Systematic Reviews Programme Grant (13/89/15)

• The National Institute for Health Research (NIHR), UK.

  The NIHR, UK, is the largest single funder of Cochrane Skin

• NIHR clinical fellowship, UK.

• NIHR Birmingham Biomedical Research Centre, UK.

  JD, JJD and YT receive support from the NIHR Birmingham Biomedical Research Centre

Declarations of interest

Jacqueline Dinnes: nothing to declare.
 Jonathan J Deeks: nothing to declare.
 Naomi Chuchu: nothing to declare.
 Lavinia Ferrante di Ruffano: nothing to declare.
 Rubeta N Matin: my institution received a grant for a Barco NV commercially sponsored study to evaluate digital dermoscopy in the skin cancer clinic. My institution also received Oxfordshire Health Services Research Charitable Funds for carrying out a study of feasibility of using the Skin Cancer Quality of Life Impact Tool (SCQOLIT) in non melanoma skin cancer. I have received payment from Public Health England for the "Be Clear on Cancer Skin Cancer" report and royalties for the Oxford Handbook of Medical Dermatology (Oxford University Press). I have no conflicts of interest to declare that directly relate to the publication of this work.
 David R Thomson: nothing to declare.
 Kai Yuen Wong: nothing to declare.
 Roger Benjamin Aldridge: nothing to declare.
 Rachel Abbott: nothing to declare.
 Monica Fawzy: nothing to declare.
 Susan E Bayliss: nothing to declare.
 Matthew J Grainge: nothing to declare.
 Yemisi Takwoingi: nothing to declare.
 Clare Davenport: nothing to declare.
 Kathie Godfrey: nothing to declare.
 Fiona M Walter: nothing to declare.
 Hywel C Williams: I am director of the NIHR HTA Programme. HTA is part of the NIHR which also supports the NIHR systematic reviews programme from which this work is funded.

Edited (no change to conclusions)