Visual inspection and dermoscopy, alone or in combination, for diagnosing keratinocyte skin cancers in adults

Abstract

Background

Early accurate detection of all skin cancer types is important to guide appropriate management, to reduce morbidity and to improve survival. Basal cell carcinoma (BCC) is almost always a localised skin cancer with potential to infiltrate and damage surrounding tissue, whereas a minority of cutaneous squamous cell carcinomas (cSCCs) and invasive melanomas are higher‐risk skin cancers with the potential to metastasise and cause death. Dermoscopy has become an important tool to assist specialist clinicians in the diagnosis of melanoma, and is increasingly used in primary‐care settings. Dermoscopy is a precision‐built handheld illuminated magnifier that allows more detailed examination of the skin down to the level of the superficial dermis. Establishing the value of dermoscopy over and above visual inspection for the diagnosis of BCC or cSCC in primary‐ and secondary‐care settings is critical to understanding its potential contribution to appropriate skin cancer triage, including referral of higher‐risk cancers to secondary care, the identification of low‐risk skin cancers that might be treated in primary care and to provide reassurance to those with benign skin lesions who can be safely discharged.

Objectives

To determine the diagnostic accuracy of visual inspection and dermoscopy, alone or in combination, for the detection of (a) BCC and (b) cSCC, in adults. We separated studies according to whether the diagnosis was recorded face‐to‐face (in person) or based on remote (image‐based) assessment.

Search methods

We undertook a comprehensive search of the following databases from inception up to August 2016: Cochrane Central Register of Controlled Trials; MEDLINE; Embase; CINAHL; CPCI; Zetoc; Science Citation Index; US National Institutes of Health Ongoing Trials Register; NIHR Clinical Research Network Portfolio Database; and the World Health Organization International Clinical Trials Registry Platform. We studied reference lists and published systematic review articles.

Selection criteria

Studies of any design that evaluated visual inspection or dermoscopy or both in adults with lesions suspicious for skin cancer, compared with a reference standard of either histological confirmation or clinical follow‐up.

Data collection and analysis

Two review authors independently extracted all data using a standardised data extraction and quality assessment form (based on QUADAS‐2). We contacted authors of included studies where information related to the target condition or diagnostic thresholds were missing. We estimated accuracy using hierarchical summary ROC methods. We undertook analysis of studies allowing direct comparison between tests. To facilitate interpretation of results, we computed values of sensitivity at the point on the SROC curve with 80% fixed specificity and values of specificity with 80% fixed sensitivity. We investigated the impact of in‐person test interpretation; use of a purposely‐developed algorithm to assist diagnosis; and observer expertise.

Main results

We included 24 publications reporting on 24 study cohorts, providing 27 visual inspection datasets (8805 lesions; 2579 malignancies) and 33 dermoscopy datasets (6855 lesions; 1444 malignancies). The risk of bias was mainly low for the index test (for dermoscopy evaluations) and reference standard domains, particularly for in‐person evaluations, and high or unclear for participant selection, application of the index test for visual inspection and for participant flow and timing. We scored concerns about the applicability of study findings as of ‘high’ or 'unclear' concern for almost all studies across all domains assessed. Selective participant recruitment, lack of reproducibility of diagnostic thresholds and lack of detail on observer expertise were particularly problematic.

The detection of BCC was reported in 28 datasets; 15 on an in‐person basis and 13 image‐based. Analysis of studies by prior testing of participants and according to observer expertise was not possible due to lack of data. Studies were primarily conducted in participants referred for specialist assessment of lesions with available histological classification. We found no clear differences in accuracy between dermoscopy studies undertaken in person and those which evaluated images. The lack of effect observed may be due to other sources of heterogeneity, including variations in the types of skin lesion studied, in dermatoscopes used, or in the use of algorithms and varying thresholds for deciding on a positive test result.

Meta‐analysis found in‐person evaluations of dermoscopy (7 evaluations; 4683 lesions and 363 BCCs) to be more accurate than visual inspection alone for the detection of BCC (8 evaluations; 7017 lesions and 1586 BCCs), with a relative diagnostic odds ratio (RDOR) of 8.2 (95% confidence interval (CI) 3.5 to 19.3; P < 0.001). This corresponds to predicted differences in sensitivity of 14% (93% versus 79%) at a fixed specificity of 80% and predicted differences in specificity of 22% (99% versus 77%) at a fixed sensitivity of 80%. We observed very similar results for the image‐based evaluations.

When applied to a hypothetical population of 1000 lesions, of which 170 are BCC (based on median BCC prevalence across studies), an increased sensitivity of 14% from dermoscopy would lead to 24 fewer BCCs missed, assuming 166 false positive results from both tests. A 22% increase in specificity from dermoscopy with sensitivity fixed at 80% would result in 183 fewer unnecessary excisions, assuming 34 BCCs missed for both tests. There was not enough evidence to assess the use of algorithms or structured checklists for either visual inspection or dermoscopy.

Insufficient data were available to draw conclusions on the accuracy of either test for the detection of cSCCs.

Authors' conclusions

Dermoscopy may be a valuable tool for the diagnosis of BCC as an adjunct to visual inspection of a suspicious skin lesion following a thorough history‐taking including assessment of risk factors for keratinocyte cancer. The evidence primarily comes from secondary‐care (referred) populations and populations with pigmented lesions or mixed lesion types. There is no clear evidence supporting the use of currently‐available formal algorithms to assist dermoscopy diagnosis.

Plain language summary

Does dermoscopy improve the accuracy of diagnosing basal cell or squamous cell skin cancer (BCC or cSCC) compared to using the naked eye alone?

What is the aim of the review?

We wanted to find out whether using a handheld illuminated microscope (dermatoscope or ‘dermoscopy’) is any better at diagnosing basal cell carcinoma (BCC) or cutaneous squamous cell carcinoma (cSCC) compared to just looking at the skin with the naked eye. We included 24 studies to answer this question.

Why is improving diagnosis of BCC or cSCC important?

There are a number of different types of skin cancer. BCC and cSCC are less serious than melanoma skin cancer, because they usually grow more slowly and BCC does not spread to other organs in the body. Making the correct diagnosis of BCC or cSCC is still important, because their treatment may differ. A missed BCC (known as a false negative result) can result in disfigurement and the need for more major surgery. A missed cSCC can spread to other parts of the body. Diagnosing BCC or cSCC when they are not actually present (a false positive result) may mean unnecessary treatment, e.g. surgical removal which may result in a disfiguring scar, and worry to patients if the lesion (a mole or area of skin with an unusual appearance in comparison with the surrounding skin) is benign (not a cancer), or may result in wrong treatment, e.g. a non‐surgical therapy, being used if the lesion is misdiagnosed.

What was studied in the review?

A dermatoscope is a handheld magnifier that includes a light source. Dermoscopy is often used by skin specialists to help diagnose skin cancer. It is also being used more by community doctors.

As well as seeing whether dermoscopy added anything to visual inspection alone overall, we also wanted to find out whether dermoscopy accuracy was different when used in a face‐to‐face consultation or when used on images of skin lesions sent to specialists. We also tried to find out whether the accuracy of dermoscopy was improved by use of a checklist, or if it was better when used by a skin specialist compared to a non‐specialist.

What are the main results of the review?

The review included 24 studies reporting information for people with lesions suspected of skin cancer.

Diagnosis of BCC with the patient present

We found 11 relevant studies. Eight studies (including 7017 suspicious skin lesions) investigated the accuracy of visual inspection on its own and seven studies (with 4683 suspicious skin lesions) investigated the accuracy of dermoscopy added to visual inspection (four of which reported data for both visual inspection on its own and for dermoscopy added to visual inspection). The results suggest that dermoscopy is more accurate than visual inspection on its own, both for identifying BCC correctly and for excluding things that are not BCCs.

The results can be illustrated using a group of 1000 lesions, of which 170 (17%) are BCC. In order to see how much better dermoscopy is in identifying BCC correctly when compared to just looking at the skin, we have to assume that both lead to the same number of lesions being falsely diagnosed as BCC (we assumed that 166 of the 830 lesions without BCC would have an incorrect diagnosis of BCC). In this fixed situation, adding dermoscopy to visual inspection would correctly identify an extra 24 BCCs (158 compared with 134) that would have been missed by just looking at the skin alone. In other words, more BCC cancers would be correctly identified.

In order to see how much better dermoscopy is in deciding if a skin lesion is not a BCC when compared to just looking at the skin, we have to assume that both lead to the same number of BCCs being correctly diagnosed (in this case we assumed that 136 out of the 170 BCCs would be correctly diagnosed). In this situation, adding in dermoscopy to visual inspection would reduce the number of lesions being wrongly diagnosed as being BCC by 183 (a reduction from 191 in the visual inspection group to eight people in the dermoscopy group). In other words, more lesions that were not BCC would be correctly identified, and fewer people would end up being sent for surgery.

Image‐based diagnosis of BCC

Eleven studies concerning BCC diagnosis using either clinical photographs or magnified images from a dermatoscope were included. Four studies, (including 853 suspicious skin lesions) used visual inspection of photographs and nine studies (including 2271 suspicious lesions) used dermoscopic images (two studies reported data for diagnosis using both photographs and using dermoscopic images). Results were very similar to the in‐person studies.

Value of checklists and observer expertise

There was no evidence that use of a checklist to help visual inspection or dermoscopy interpretation improved diagnostic accuracy. There was not enough evidence to examine the effect of clinical expertise and training.

Diagnosis of cSCC

There was not enough evidence to reliably comment on the accuracy of either test for the detection of cSCCs.

How reliable are the results of the studies of this review?

Most of our studies made a reliable final diagnosis by lesion biopsy and by following people up over time to make sure the skin lesion remained negative for skin cancer. Some studies used expert diagnosis to confirm the absence of skin cancer, which is less reliable*. Poor reporting of what was done in the studies made it difficult for us to judge how reliable they were. Some studies excluded certain types of skin lesion and some did not describe how a positive test result to trigger referral to a specialist or treatment was defined.

Who do the results of this review apply to?

Eleven studies were done in Europe (46%), and the rest in North America (n = 3), Asia (n = 5), Oceania (n = 2), or multiple countries (n = 3). People included in the studies were on average between 30 and 74 years old. The percentage of people with BCC ranged between 1% and 61% for in‐person studies and between 2% and 63% in studies using images. Almost all studies were done with people referred from primary care to specialist skin clinics. Over half of studies considered the ability of dermoscopy and visual inspection to diagnose any skin cancer, including melanoma and BCC, while 10 (42%) focused on just BCC. Variation in the expertise of doctors doing the examinations and differences in the definitions used to decide when a test was positive make it unclear how dermoscopy should be carried out and what level of training is needed in order to achieve the accuracy observed in studies.

What are the implications of this review?

When used by specialists, dermoscopy may be a useful tool to help diagnose BCC correctly when compared with visual inspection alone. It is not clear whether dermoscopy should be used by general practitioners to correctly identify people with suspicious lesions who need to be seen by a specialist. Checklists to help interpret dermoscopy do not seem to help improve accuracy for BCC. Further research is needed, to see if dermoscopy is useful in primary care.

How up‐to‐date is this review?

The review authors searched for and used studies published up to August 2016.

*In these studies biopsy, clinical follow‐up or specialist clinician diagnosis were the reference standards (means of establishing the final diagnosis).

Summary of findings

Summary of findings'. 'Summary of findings table.

Question:	What is the diagnostic accuracy of dermoscopy, in comparison to visual inspection, for the detection of keratinocyte skin cancer in adults?
Population:	Adults with skin lesions: suspicious for keratinocyte skin cancers, basal cell carcinoma (BCC) or cutaneous squamous cell carcinoma (cSCC) (e.g. non‐pigmented lesions); suspicious for any skin cancer, including melanoma (e.g. those with pigmented lesions only or mixed populations of pigmented and non‐pigmented lesions); or those at high risk of developing keratinocyte skin cancer
Index test:	Dermoscopy with or without the use of any established algorithms or checklist to aid diagnosis, including: in‐person evaluations (face‐to‐face diagnosis), and image‐based evaluations (diagnosis based on assessment of a dermoscopic image)
Comparator test	Visual inspection including: in‐person evaluations, and image‐based evaluations (diagnosis based on assessment of a clinical image)
Primary Target condition:	BCC or cSCC
Reference standard:	Histology with or without long‐term follow‐up
Action:	If accurate, negative results will stop patients having unnecessary excision or biopsy of skin lesions; positive results could inform the use of nonsurgical management options
	Number of studies		Total lesions			Total malignancies
Quantity of evidence	24		Visual Inspection: 8805 Dermoscopy: 6855			Visual Inspection: 2579 Dermoscopy: 1444
Limitations
Risk of bias: (in‐person (14); image‐based (12))	Potential risk of bias for participant selection from use of case‐control type design (3 image‐based), inappropriate exclusion criteria (3; 2) or lack of detail (8; 4). All visual inspection and dermoscopy interpretation considered blinded to reference standard diagnosis. Visual Inspection risk of bias not clear due to thresholds not clearly prespecified (8; 4). Threshold prespecification better reported for dermoscopy (6; 6). Low risk for reference standard (13; 11); high risk from use of expert diagnosis or > 20% of benign lesions with no histology (1; 1). High risk for participant flow due to differential verification (1; 1), and exclusions following recruitment (5; 6); timing of tests was not mentioned in (7; 7)
Applicability of evidence to question: (in‐person (14); image‐based (12))	High concern for participants (14; 12) due to restriction to those with histopathology results (13; 11) and including multiple lesions per participant (9; 2). High concern for Visual Inspection (7; 4) from lack of description of diagnostic thresholds. High concern for dermoscopy (3; 9) from no description of diagnostic thresholds (2; 4) or reporting of average or consensus diagnoses (2; 7). Dermoscopic image interpretation blinded to clinical images (10 image‐based). Unclear applicability of reference standard due to insufficient information concerning the expertise of the histopathologist (13; 11)
FINDINGS:
We included 24 studies. 14 studies reported data for in‐person visual inspection (n = 11) or in‐person dermoscopy (n = 8); 12 studies reported data for image‐based visual inspection (n = 4) or image‐based dermoscopy (n = 10). Two studies report both in‐person and image‐based data. The findings presented are based on results for the 21 studies reporting data for BCC alone or for cSCC alone. Due to the observed heterogeneity between studies, the results presented are points estimated from summary ROC curves rather than average sensitivity and specificity operating points. These are presented for illustrative purposes and should not be quoted as the actual performance of visual inspection or dermoscopy. We did not undertake analyses of studies by degree of prior testing due to a lack of relevant information provided in the study publications, most studies apparently being conducted in referred populations, and small study subgroups. There was not enough evidence to assess the use of algorithms or structured checklists for dermoscopy (or visual inspection)
Test (for BCC):	In‐person visual inspection alone versus visual inspection plus dermoscopy for the detection of BCC – any algorithm or threshold
Data analysed	Visual inspection				8 datasets ‐ 7017 lesions; 1586 cases
	Dermoscopy				7 datasets ‐ 4683 lesions; 363 cases
Resultsa	Sensitivity		Fixed specificity		Fixed sensitivity		Specificity
Visual inspection	79%		80%		80%		77%
Dermoscopy	93%						99%
Numbers applied to a hypothetical cohort of 1000 lesionsb
	TP	FN	FP	TN	TP	FN	FP	TN
At a prevalence of 10%	VI: 79 D: 93 ↑ 14	VI: 21 D: 7 ↓ 14	180	720	80	20	VI: 207 D: 9 ↓198	VI: 693 D: 891 ↑198
At a prevalence of 17%	VI: 134 D: 158 ↑24	VI: 36 D: 12 ↓ 24	166	664	136	34	VI: 191 D: 8 ↓183	VI: 639 D: 822 ↑183
At a prevalence of 53%	VI: 419 D: 493 ↑ 74	VI: 111 D: 37 ↓ 74	94	376	424	106	VI: 108 D: 5 ↓103	VI: 362 D: 465 ↑103
Consistency:	Wide range in prevalence of BCC; includes pigmented and non‐pigmented lesion populations and participants suspected of BCC or suspected of any malignancy, including melanoma. Sensitivities highly heterogeneous, particularly for visual‐inspection evaluations. Specificity for BCC lower in studies of non‐pigmented lesions
Test (for BCC):	Image‐based visual inspection alone versus visual inspection plus dermoscopy for the detection of BCC – any algorithm or threshold
Data analysed	Visual inspection				4 datasets ‐ 853 lesions; 156 cases
	Dermoscopy				9 datasets ‐ 2271 lesions; 737 cases
Results	Sensitivity		Fixed specificity		Fixed sensitivity		Specificity
Visual inspection	85%		80%		80%		87%
Dermoscopy	93%						96%
Numbers applied to a hypothetical cohort of 1000 lesionsc
	TP	FN	FP	TN	TP	FN	FP	TN
At a prevalence of 11%	VI: 94 D: 102 ↑ 8	VI: 16 D: 8 ↓ 8	178	712	88	22	VI: 116 D: 36 ↓80	VI: 774 D: 854 ↑80
At a prevalence of 16%	VI: 136 D: 149 ↑13	VI: 24 D: 11 ↓ 13	168	672	128	32	VI: 109 D: 34 ↓75	VI: 731 D: 806 ↑75
At a prevalence of 47%	VI: 400 D: 437 ↑ 37	VI: 70 D: 33 ↓ 37	106	424	376	94	VI: 69 D: 21 ↓48	VI: 461 D: 509 ↑48
Consistency:	Wide range in prevalence of BCC; includes mixed populations, as for in‐person evaluations. Sensitivities highly heterogeneous for visual inspection evaluations.
Test (for cSCC):	Visual inspection or dermoscopy for the detection of cSCC
	Datasets		Lesions	Cases	Sensitivity	(95%CIs)	Specificity	(95%CI)
Visual inspection (in‐person)	2		2684	538	57%	(53%, 61%)	79%	(77%, 81%)
Dermoscopy (image‐based)	2		717	119	55%	(29%, 79%)	84%	(32%, 98%)

^aNumbers for a hypothetical cohort of 1000 lesions are presented for two illustrative examples of points on the SROC curves: firstly for the sensitivities of tests at fixed specificities of 80%; and secondly for the specificities of tests at fixed sensitivities of 80%.
 ^bNumbers estimated at 25th, 50th (median) and 75% percentiles of BCC prevalence observed across 11 studies reporting in‐person evaluations of visual inspection (reported in eight studies) or visual inspection plus dermoscopy (reported in seven studies).
 ^cNumbers estimated at 25th, 50th (median) and 75% percentiles of BCC prevalence observed across 11 studies reporting image‐based diagnosis using clinical photographs (reported in four studies) or dermoscopic images (reported in nine studies)

Background

This review is one of a series of Cochrane Diagnostic Test Accuracy (DTA) Reviews on the diagnosis and staging of melanoma and keratinocyte skin cancers as part of the National Institute for Health Research (NIHR) Cochrane Systematic Reviews Programme. Appendix 1 shows the content and structure of the programme.

Target condition being diagnosed

The commonest skin cancers in white populations are those arising from keratinocyte cells: basal cell carcinoma (BCC) and cutaneous squamous cell carcinoma (cSCC) (Gordon 2013; Madan 2010). BCC is the more common of the two keratinocyte carcinomas, and approximately one‐third of people with a BCC will subsequently develop a second (Flohill 2013). In 2003, the World Health Organization (WHO) estimated that between two and three million ‘non‐melanoma’ skin cancers (of which BCC and cSCC are estimated to account for around 80% and 16% of cases respectively) and 132,000 melanoma skin cancers occur globally each year (WHO 2003).

Rather than defining BCC and cSCC by what they are not (i.e. non‐melanoma skin cancer), we collectively refer to these conditions using the preferred and more accurate term of 'keratinocyte carcinoma' in this DTA review (Karimkhani 2015). We define (a) BCC and (b) cSCC as the primary target conditions for this review. We also examine accuracy for the target condition of (c) any skin cancer, including keratinocyte skin cancer, melanoma or intra‐epidermal melanocytic variants and any other skin cancer. We have examined the accuracy of visual inspection for the diagnosis of melanoma in a previous review (Dinnes 2018a) and in a further review, we examine the potential benefit of dermoscopy added to visual inspection for the diagnosis of melanoma (Dinnes 2018b). Appendix 2 provides a glossary of terms used.

Basal cell carcinoma

BCC can arise from multiple stem cell populations, including from the follicular bulge and interfollicular epidermis (Grachtchouk 2011). Growth is usually localised, but it can infiltrate and damage surrounding tissue, which if left untreated can cause considerable destruction and disfigurement, particularly when located on the face (Figure 1). The four main types of BCC are superficial, nodular, morphoeic (infiltrative), and pigmented. Lesions typically present as slow‐growing asymptomatic papules, plaques, or nodules, which may bleed or form ulcers that do not heal (Firnhaber 2012). People with a BCC often present themselves to healthcare professionals with a non‐healing lesion rather than specific symptoms such as pain. Many lesions are diagnosed incidentally (Gordon 2013).

1.

Sample photograph of superficial spreading melanoma(left), BCC (centre) and SCC (right). Copyright © 2012 Dr Rubeta Matin: reproduced with permission.

BCC most commonly occurs on sun‐exposed areas of the head and neck (McCormack 1997), and are more common in men and in people over the age of 40. A rising incidence of BCC in younger people has been attributed to increased recreational sun exposure (Bath‐Hextall 2007a; Gordon 2013; Musah 2013). Other risk factors include Fitzpatrick skin types I and II (Fitzpatrick 1975; Lear 1997; Maia 1995); previous skin cancer history; immunosuppression; arsenic exposure; and genetic predisposition, such as in basal cell naevus (Gorlin) syndrome (Gorlin 2004; Zak‐Prelich 2004). Annual incidence is increasing worldwide; Europe has experienced an average increase of 5.5% per year since the 1970s, the USA 2% per year, while estimates for the UK show incidence appears to be increasing more steeply at a rate of an additional 6/100,000 persons a year (Lomas 2012). The rising incidence has been attributed to an ageing population, changes in the distribution of known risk factors, particularly ultraviolet radiation, and improved detection due to the increased awareness amongst both practitioners and the general population (Verkouteren 2017). Hoorens 2016 points to evidence for a gradual increase in the size of BCCs over time, with delays in diagnosis ranging from 19 to 25 months.

According to National Institute for Health and Care Excellence (NICE) guidance (NICE 2010), low‐risk BCCs are nodular lesions occurring in people older than 24 years who are not immunosuppressed and do not have Gorlin syndrome. Furthermore, lesions should be located below the clavicle; should be small (less than 1 cm) with clinically well‐defined margins; not recurrent following incomplete excision or other treatment; and not in awkward or highly‐visible locations (NICE 2010). Superficial BCCs are also typically low risk and may be amenable to medical treatments such as cryotherapy, photodynamic therapy or topical immunomodulatory therapy, e.g. 5% Imiquimod cream (Kelleners‐Smeets 2017). Assigning BCCs as low or high risk influences the management options (Batra 2002; Randle 1996).

Advanced locally‐destructive BCC can be found on the H‐area of the face (Lear 2014), can arise from long‐standing untreated lesions, or from a recurrence of aggressive basal cell carcinoma after primary treatment (Lear 2012). Very rarely, BCC may metastasise to regional and distant sites resulting in death; this is particularly true for large neglected lesions in those who are immunosuppressed, or those with Gorlin syndrome (McCusker 2014). Rates of metastasis are reported at 0.0028% to 0.55% with very poor survival rates (Lo 1991). It is recognised that basosquamous carcinoma (more like a high‐risk SCC in behaviour and not considered a true BCC) is likely to have accounted for many cases of apparent metastases of BCC, hence, the spuriously high reported incidence in some studies of up to 0.55%, which is not seen in clinical practice (Garcia 2009).

Squamous cell carcinoma of the skin

Primary cSCC arises from the keratinising cells of the epidermis or its appendages. cSCC typically presents with an ulcer or firm (indurated) papule, plaque, or nodule (Griffin 2016), often with an adherent crust (Madan 2010) (Figure 1). cSCC can arise in the absence of a precursor lesion, or may develop from pre‐existing actinic keratosis or Bowen's disease (considered by some clinicians to be cSCC in situ); the estimated annual risk of progression is less than 1% to 20% for newly‐arising lesions (Alam 2001) and 5% for pre‐existing lesions (Kao 1986). It remains locally invasive for a variable length of time, but has the potential to spread to the regional lymph nodes or via the bloodstream to distant sites, especially in immunosuppressed individuals (Lansbury 2010). High‐risk lesions are those arising on the lip or ear; recurrent cSCC; lesions arising on non‐exposed sites; within scars or chronic ulcers; tumours more than 20 mm in diameter and those with a histological depth of invasion exceeding 4 mm; and poor differentiation status on pathological examination (Motley 2009). Perineural nerve invasion (PNI) of at least 0.1 mm in diameter is a further documented risk factor for high‐risk cSCC (Carter 2013).

Chronic ultraviolet light exposure through recreation or occupation is strongly linked to cSCC occurrence (Alam 2001). It is particularly common in people with fair skin and in less common genetic disorders of pigmentation, such as albinism, xeroderma pigmentosum, and recessive dystrophic epidermolysis bullosa (RDEB) (Alam 2001). Other recognised risk factors include immunosuppression; chronic wounds; arsenic or radiation exposure; certain drug treatments, such as voriconazole and BRAF mutation inhibitors; and previous skin cancer history (Baldursson 1993; Chowdri 1996; Dabski 1986; Fasching 1989; Lister 1997; Maloney 1996; O'Gorman 2014). In solid organ transplant recipients, cSCC is the most common form of skin cancer; the risk of developing cSCC has been estimated at 65 to 253 times that of the general population (Hartevelt 1990; Jensen 1999; Lansbury 2010). Overall, local and metastatic recurrence of cSCC at five years is estimated at 8% and 5% respectively. The five‐year survival rate of metastatic cSCC of the head and neck is around 60% (Moeckelmann 2018).

Treatment

Treatment options for BCC and cSCC include surgery, other destructive techniques such as cryotherapy or electrodesiccation and topical chemotherapy. A Cochrane Review of 27 randomised controlled trials (RCTs) of interventions for BCC found very little good‐quality evidence for any of the interventions used (Bath‐Hextall 2007b). Complete surgical excision of primary BCC has a reported five‐year recurrence rate of less than 2% (Griffiths 2005; Walker 2006), leading to significantly fewer recurrences than treatment with radiotherapy (Bath‐Hextall 2007b). After apparent clear histopathological margins (serial vertical sections) after standard excision biopsy with 4 mm surgical peripheral margins taken, there is a five‐year reported recurrence rate of around 4% (Drucker 2017). Mohs micrographic surgery, whereby horizontal sections of the excised specimen are microscopically examined perioperatively, and re‐excision is undertaken until the margins are tumour‐free, can be considered for high‐risk lesions where standard wider excision margins might lead to incomplete excision or considerable functional and/or cosmetic impairment (Bath‐Hextall 2007b; Motley 2009; Lansbury 2010; Stratigos 2015). Bath‐Hextall 2007b found a single trial comparing Mohs micrographic surgery with a 3 mm surgical margin excision in BCC (Smeets 2004), showing non‐significantly lower recurrence at 10 years with Mohs micrographic surgery (4.4% compared to 12.2% after surgical excision, P = 0.10) (Van Loo 2014).

The main treatments for high‐risk BCC are wide local excision, Mohs micrographic surgery and radiotherapy. For low‐risk or superficial subtypes of BCC, or for small and/or multiple BCCs at low‐risk sites (Marsden 2010), destructive techniques other than excisional surgery may be used (e.g. electrodesiccation and curettage or cryotherapy (Alam 2001; Bath‐Hextall 2007b)). Alternatively, non‐surgical (or non‐destructive) treatments may be considered (Bath‐Hextall 2007b; Drew 2017; Kim 2014), including topical chemotherapy such as imiquimod (Williams 2017), 5‐fluorouracil (5‐FU) (Arits 2013), ingenol mebutate (Nart 2015) and photodynamic therapy (PDT) (Roozeboom 2016). Non‐surgical treatments are most frequently used for superficial forms of BCC, with one head‐to‐head trial suggesting topical imiquimod is superior to PDT and 5‐FU (Jansen 2018). Although non‐surgical techniques are increasingly used, they do not allow histological confirmation of tumour clearance, and their efficacy is dependent on accurate characterisation of the histological subtype and depth of tumour, and so a baseline diagnostic biopsy can be helpful. The 2007 systematic review of BCC interventions found limited evidence from very small RCTs for these approaches (Bath‐Hextall 2007b), which have only partially been filled by subsequent studies (Bath‐Hextall 2014; Kim 2014; Roozeboom 2012). Most BCC trials have compared interventions within the same treatment class, and few have compared medical versus surgical treatments (Kim 2014).

Vismodegib, a first‐in‐class Hedgehog signalling pathway inhibitor, is now available for the treatment of metastatic or locally‐advanced BCC based on the pivotal study ERIVANCE BCC (Sekulic 2012). It is licensed for use in people with BCC where surgery or radiotherapy is inappropriate, e.g. for treating locally‐advanced periocular and orbital BCCs with orbital salvage of patients who otherwise would have required exenteration (Wong 2017). However, NICE has recently recommended against the use of vismodegib based on cost effectiveness and uncertainty of evidence (NICE 2017).

A systematic review of interventions for primary cSCC found only one RCT eligible for inclusion (Lansbury 2010). Current practice therefore relies on evidence from observational studies, as reviewed in Lansbury 2013, for example. Surgical excision with predetermined margins is usually the first‐line treatment (Motley 2009; Stratigos 2015). Estimates of recurrence after Mohs micrographic surgery, surgical excision, or radiotherapy, which are likely to have been evaluated in higher‐risk populations, have shown pooled recurrence rates of 3%, 5.4% and 6.4%, respectively, with overlapping confidence intervals; the review authors advise caution when comparing results across treatments (Lansbury 2013).

Index test(s)

For the purposes of our series of reviews, each component of the diagnostic process, including visual inspection during clinical examination, is considered a diagnostic or index ‘test', the accuracy of which can be established in comparison with a reference standard of diagnosis, either alone or in combination with other available technologies that may assist the diagnostic process. In this review, two index tests are under consideration: visual inspection and dermoscopy, both of which can be undertaken in person (in a face‐to‐face consultation) or image‐based (remote diagnosis using images). As dermoscopy is effectively added to visual inspection of a skin lesion when it is undertaken in person, we effectively have three index tests: visual inspection alone (in person or using images), visual inspection plus dermoscopy (in‐person dermoscopy), and dermoscopy alone (image‐based dermoscopy).

Visual inspection

Clinical history‐taking and visual inspection (and palpation) of the lesion, surrounding skin and comparison with other lesions identified on complete examination of the body, is fundamental to the diagnosis of skin cancer. In the UK, clinical examination is typically done at two decision points: first in primary care where a decision is made to refer, treat (if low‐risk BCC is suspected), or reassure, and then a second time by a dermatologist or other secondary‐care clinician where a treatment decision is made if appropriate.

Visual inspection of a lesion involves clinical reasoning based on both non‐analytical and analytical pattern recognition strategies (Elstein 2002; Norman 1989; Norman 2009). Non‐analytical pattern recognition uses subconscious intuitive processes, while analytical pattern recognition uses more explicit rules based on hypothetico‐deductive reasoning (Norman 2009). The balance between non‐analytical and analytical reasoning varies between clinicians, according to factors such as constitutional reasoning style preference, experience and familiarity with the diagnostic question.

Unlike for melanoma, where a number of diagnostic algorithms or checklists have been developed to help recognise melanomas (Friedman 1985; MacKie 1985; MacKie 1990; Nachbar 1994; Pehamberger 1993; Sober 1979; Steiner 1987; Stolz 1994), visual inspection for keratinocyte skin cancers relies primarily on pattern recognition. Accuracy has been shown to vary according to the expertise of the clinician. Primary‐care physicians have been reported to miss over half of BCCs (Offidani 2002) and to inappropriately diagnose one‐third of BCCs (Gerbert 2000). In contrast, an Australian study found that skin‐cancer specialists were able to detect 89% of BCCs compared to 79% for general practitioners (GPs), with corresponding specificities of 79% (specialists) and 83% (GPs) (Youl 2007b).

Visual inspection of a digital photograph or ‘macroscopic’ image of a suspicious skin lesion can also be undertaken as part of a teledermatology consultation, whereby clinical photographs, dermoscopic images, or both, are taken by non‐specialist clinicians and forwarded to a dermatologist, to obtain a specialist opinion (Chuchu 2018a). Images can also be encompassed in a store‐and‐forward smartphone application whereby a photograph of a concerning lesion is taken by the smartphone user and forwarded for an assessment of skin‐cancer risk by a specialist clinician (Chuchu 2018b). Images are often accompanied by a summary of the medical history and demographic information as part of a consultation package (Ndegwa 2010). According to UK guidelines, both clinical and dermoscopic images must be sent for ‘full dermatology’, i.e. as a replacement for a face‐to‐face consultation, whereas for ‘triage teledermatology’ dermoscopic images should be sent where facilities permit (BAD 2013).

Dermoscopy

Dermoscopy (also referred to as dermatoscopy or epiluminescence microscopy (ELM)) has become a widely‐used tool for the specialist clinician and is also increasingly being used in primary‐care settings. It uses a hand‐held microscope and incident light (with or without oil immersion) to reveal subsurface images of the skin at increased magnification of x10 to x100 (Kittler 2011) (Figure 2). It is particularly useful for the identification of melanoma when used by specialists (Dinnes 2018b), but its role in the diagnosis of keratinocyte skin cancers is less clearly established.

2.

Dermatoscope. Copyright © 2018 HEINE Optotechnik: reproduced with permission.

The visual nature of dermoscopic interpretation means that when used on an in‐person basis, dermoscopy is essentially added to visual inspection of a skin lesion and similar non‐analytical and analytical pattern recognition strategies are employed to reach a dermoscopic diagnosis. Dermoscopic histological correlations have been established for the diagnosis of melanoma, allowing a number of diagnostic algorithms to be developed based on lesion colour, aspect, pigmentation pattern, and skin vessels (Dinnes 2018b). However, the diagnosis of keratinocyte skin cancers using dermoscopy again relies predominantly on subjective pattern recognition. Features of BCC on dermoscopy include arborising (branching of) blood vessels, superficial fine telangiectasia (abnormally tortuous and dilated blood vessels), grey‐blue ovoid nests and globules, in‐focus dots, spoke wheels and maple‐leaf‐like areas, concentric structures, ulceration, multiple small erosions, shiny white‐red structureless areas, and short white streaks (Tzellos 2014). Features favouring cSCC on dermoscopy include the presence of keratin, white circles, radial telangiectasia and blood spots (Rosendahl 2012a; Zalaudek 2012).

In modern practice, dermoscopic images are frequently obtained for skin lesions that are recommended for excision and are also obtained for lesions that have not yet met the diagnostic threshold for excision but are to be monitored over time in case of any further suspicious changes. Dermoscopic images are also a key component of teledermatology consultations, usually accompanied by digital photographs and other pertinent information (Chuchu 2018a), as discussed above.

Clinical pathway

The diagnosis of skin lesions occurs in primary‐, secondary‐, and tertiary‐care settings by both generalist and specialist healthcare providers. In the UK, people with concerns about a new or changing lesion will present to their general practitioner rather than directly to a specialist in secondary care. If the general practitioner has concerns, then a referral is usually made to a specialist in secondary care – usually a dermatologist, but sometimes to a surgical specialist such as a plastic surgeon or an ophthalmic surgeon. Suspicious skin lesions may also be identified in a referral setting, for example by a general surgeon, and referred for a consultation with a skin cancer specialist (Figure 3). Skin cancers identified by other specialist surgeons (such as an ear, nose, and throat (ENT) specialist or maxillofacial surgeon) will usually be diagnosed and treated without further referral.

3.

Current clinical pathway for people with skin lesions.

Current UK guidelines recommend that all suspicious pigmented lesions presenting in primary care should be assessed by taking a clinical history and visual inspection using the seven‐point checklist (MacKie 1990); lesions suspected to be melanoma or cSCC should be referred for appropriate specialist assessment within two weeks (Chao 2013; Marsden 2010; NICE 2015). Evidence is emerging, however, to suggest that excision of melanoma by GPs is not associated with increased risk compared with outcomes in secondary care (Murchie 2017). In the UK, low‐risk BCCs are usually recommended for routine referral, with urgent referral for those in whom a delay could have a significant impact on outcomes, for example due to large lesion size or critical site (NICE 2015). Appropriately‐qualified generalist care providers increasingly undertake management of low‐risk BCCs in the UK, such as by excision of low‐risk lesions (NICE 2010). Similar guidance is in place in Australia (CCAAC Network 2008).

For referred lesions, the specialist clinician will use history‐taking, visual inspection of the lesion (in conjunction with other skin lesions), palpation of the lesion and associated regional nodal basins in conjunction with dermoscopic examination to inform a clinical decision. If melanoma is suspected, then urgent 2 mm excision biopsy is recommended (Lederman 1985; Lees 1991); for cSCC predetermined surgical margin excision or a diagnostic biopsy may be considered. BCCs and pre‐malignant lesions potentially eligible for nonsurgical treatment may undergo a diagnostic biopsy before initiation of therapy if there is diagnostic uncertainty. Equivocal melanocytic lesions for which a definitive clinical diagnosis cannot be reached may undergo surveillance to identify any lesion changes that would indicate excision biopsy or reassurance and discharge for those lesions that remain stable over a period of time.

Theoretically, teledermatology consultations may aid appropriate triage of lesions into urgent referral; non‐urgent secondary‐care referral (e.g. for suspected basal cell carcinoma); or where available, referral to an intermediate care setting, e.g. clinics run by GPs with a special interest in dermatology. The distinction between setting and examiner qualifications and experience is important, as specialist clinicians might work in primary‐care settings (for example, in the UK, GPs with a special interest in dermatology and skin surgery who have undergone appropriate training), and generalists might practice in secondary‐care settings (for example, plastic surgeons who do not specialise in skin cancer). The level of skill and experience in skin cancer diagnosis will vary for both generalist and specialist care providers and will also impact on test accuracy.

Prior test(s)

Although smartphone applications and community‐based teledermatology services can increasingly be directly accessed by people who have concerns about a skin lesion (Chuchu 2018b), visual inspection of a suspicious lesion by a clinician is usually the first in a series of tests to diagnose skin cancer. In the UK this usually takes place in primary care, but in many countries people with suspicious lesions can present directly to a specialist setting. Although dermoscopy is frequently combined with visual inspection of a lesion in secondary‐care settings, it is also increasingly used in primary care, particularly in countries such as Australia (Youl 2007a).

Consideration of the degree of prior testing that study participants have undergone is key to interpretation of test accuracy indices, as these are known to vary according to the disease spectrum (or case‐mix) of included participants (Lachs 1992; Leeflang 2013; Moons 1997; Usher‐Smith 2016). Spectrum effects are often observed when tests that are developed further down the referral pathway have lower sensitivity and higher specificity when applied in settings with participants with limited prior testing (Usher‐Smith 2016). Studies of individuals with suspicious lesions at the initial clinical presentation stage ('test‐naïve') are likely to have a wider range of differential diagnoses and include a higher proportion of people with benign diagnoses compared with studies of participants who have been referred for a specialist opinion on the basis of visual inspection (with or without dermoscopy) by a generalist practitioner. Furthermore, studies in more specialist settings may focus on equivocal or difficult‐to‐diagnose lesions rather than lesions with a more general level of clinical suspicion. However this direction of effect is not consistent across tests and diseases, the mechanisms in action often being more complex than prevalence alone, and can be difficult to identify (Leeflang 2013). A simple categorisation of studies according to primary, secondary or specialist setting may therefore not always adequately reflect these key differences in disease spectrum that can affect test performance.

Role of index test(s)

When diagnosing potentially life‐threatening conditions, the consequences of falsely reassuring a person that they do not have skin cancer can be serious and potentially fatal, as the resulting delay to diagnosis means that the window for successful early treatment may be missed. To minimise these false‐negative diagnoses, a good diagnostic test will demonstrate high sensitivity and a high negative predictive value (NPV), i.e. so that very few of those with a negative test result will actually have a malignant lesion. Giving falsely‐positive test results (meaning the test has poor specificity and a high false‐positive rate) resulting in the removal of lesions that turn out to be benign is arguably less of an error than missing a potentially fatal lesion, but is not cost‐free. False‐positive diagnoses not only cause unnecessary scarring from the biopsy or excision procedure, but also increase anxiety (particularly during the time that people wait for results) and increase healthcare costs as the number of lesions that need to be removed to yield one malignant diagnosis increases.

Delay in diagnosis of a BCC as a result of a false‐negative test is not as serious as for melanoma, because BCCs are usually slow‐growing and very unlikely to metastasise (Betti 2017). However, delayed diagnosis can result in a larger and more complex excision with consequent greater morbidity. Very sensitive diagnostic tests for BCC, however, may compromise on lower specificity leading to a higher false‐positive rate, and an enormous burden of skin surgery, such that a balance between sensitivity and specificity is needed. The situation for cSCC is more similar to melanoma in that the consequences of falsely reassuring a person that they do not have skin cancer can be serious and potentially fatal, given that removal of an early cSCC is usually curative. Thus, a good diagnostic test for cSCC should demonstrate high sensitivity and a corresponding high negative predictive value. A test that can also reduce false positive clinical diagnoses without missing true cases of cSCC has patient and resource benefits.

Alternative test(s)

A number of other tests have been reviewed as part of our series of Cochrane DTA Reviews on the diagnosis of keratinocyte skin cancers, including reflectance confocal microscopy (RCM) (Dinnes 2018c), computer‐assisted diagnosis (CAD) or artificial intelligence‐based techniques using dermoscopic or spectroscopic images (Ferrante di Ruffano 2018a), optical coherence tomography (OCT) (Ferrante di Ruffano 2018b), high‐frequency ultrasonography (Dinnes 2018d) and exfoliative cytology (Ferrante di Ruffano 2018c). Evidence permitting, we will compare the accuracy of available tests in an overview review, exploiting within‐study comparisons of tests and allowing the analysis and comparison of commonly‐used diagnostic strategies where tests may be used singly or in combination.

We also considered and excluded a number of tests from this review, such as tests used for monitoring people (e.g. total body photography of those with large numbers of pigmented lesions). We also did not assess histopathological confirmation following lesion excision, because it is the established reference standard for skin cancer diagnosis and will be one of the standards against which the index tests are evaluated in these reviews.

Rationale

This series of reviews of diagnostic tests used to assist the clinical diagnosis of BCC and cSCC in clinical practice or research settings, aims to identify the most accurate approaches to diagnosis, and to provide clinical and policy decision‐makers with the highest possible standard of evidence on which to base diagnostic and treatment decisions. With the increasing availability of a wider range of tests, there is a need to differentiate and appropriately triage keratinocyte skin cancers to avoid sending too many people with benign or low‐risk lesions for a specialist opinion whilst not missing those people who have lesions that require treatment.

There is a lack of systematic reviews in the field. A 2007 review of a range of tests for diagnosis of BCC did not report the use of systematic methods for study inclusion or extraction and did not appear to apply any quality assessment (Mogensen 2007). Critical questions of comparative test accuracy and the impact of examiner, prior testing, and underlying risk status remain unanswered for the NHS. With the increasing availability of digital imaging systems and computerised instruments, there is a further need for an up‐to‐date analysis of their accuracy in comparison with visual inspection or dermoscopy.

This review follows a generic protocol which covers the full series of Cochrane DTA Reviews for the diagnosis of keratinocyte skin cancer (Dinnes 2015a). The Background and Methods sections of this review therefore use some text that was originally published in the protocol (Dinnes 2015a) and text that overlaps some of our other reviews (Dinnes 2018a; Dinnes 2018b).

Objectives

To determine the diagnostic accuracy of visual inspection and dermoscopy, alone or in combination, for the detection of BCC in adults.

To determine the diagnostic accuracy of visual inspection and dermoscopy, alone or in combination, for the detection of cSCC in adults.

For both visual inspection and dermoscopy, we estimated accuracy separately according to whether the diagnosis was based on a face‐to‐face (in person) encounter or based on remote (image‐based) assessment. We therefore aimed to compare tests in the following way:

• To estimate incremental accuracy for the diagnosis of BCC in adults, (a) from dermoscopy added to in‐person visual inspection of a skin lesion, or (b) from dermoscopic image‐based assessment in comparison to visual inspection of a clinical photograph.

• To estimate incremental accuracy for the diagnosis of cSCC in adults, (a) from dermoscopy added to in‐person visual inspection of a skin lesion, or (b) from dermoscopic image‐based assessment in comparison to visual inspection of a clinical photograph.

We also proposed to analyse data according to the prior testing undergone by study participants (comparing those with limited prior testing with those referred for further evaluation of a suspicious skin lesion). However, this was not possible due to limited data.

Secondary objectives

For the identification of BCC or cSCC:

• To compare the accuracy of dermoscopy added to in‐person visual inspection versus visual inspection alone, where both tests have been evaluated in the same studies (direct test comparisons);

• To compare the accuracy of image‐based dermoscopy versus visual inspection of digital photographs, where both tests have been evaluated in the same studies (direct test comparisons);

• To determine the diagnostic accuracy of individual algorithms used to assist visual inspection;

• To determine the diagnostic accuracy of individual algorithms used to assist dermoscopy;

• To determine the effect of observer experience on diagnostic accuracy.

To assess an alternative target condition:

• To determine the diagnostic accuracy of visual inspection or dermoscopy, alone or in combination, for the detection of any skin cancer, and to compare the accuracy of dermoscopy with that of visual inspection alone.

Investigation of sources of heterogeneity

We set out to address a range of potential sources of heterogeneity for investigation across our series of reviews, as outlined in our generic protocol (Dinnes 2015a) and as described in Appendix 3; however, our ability to investigate these was necessarily limited by the available data on each individual test reviewed.

The sources of heterogeneity that we investigated for this review were:

• In‐person versus image‐based evaluations

• Use of a diagnostic algorithm: no algorithm reported versus any named algorithm used

• Disease prevalence: 0% to 25%; > 25%

• Observer expertise.

Methods

Criteria for considering studies for this review

Types of studies

We included test‐accuracy studies that allow comparison of the result of the index test with that of a reference standard, including the following:

• studies where all participants receive a single index test and a reference standard;

• studies where all participants receive more than one index test(s) and reference standard;

• studies where participants are allocated (by any method) to receive different index tests or combinations of index tests and all receive a reference standard (between‐person comparative studies (BPC));

• studies that recruit series of participants unselected by true disease status (referred to as case series for the purposes of this review);

• diagnostic case‐control studies that separately recruit diseased and non‐diseased groups (see Rutjes 2005); however, we did not include studies that compared results for malignant lesions to those for healthy skin (i.e. with no lesion present);

• both prospective and retrospective studies;

• studies where previously‐acquired clinical or dermoscopic images were retrieved and prospectively interpreted for study purposes.

We excluded studies from which we could not extract 2 x 2 contingency data or if they included fewer than five cases of basal cell carcinoma (BCC) or cutaneous squamous cell carcinoma (cSCC), or fewer than five benign lesions. The size threshold of five is arbitrary. However, such small studies are unlikely to add precision to estimates of accuracy.

Studies available only as conference abstracts were excluded; however, attempts were made to identify full papers for potentially relevant conference abstracts (Searching other resources).

Participants

We included studies in adults with lesions suspicious for skin cancer. These could include participants:

• with lesion characteristics suspicious for keratinocyte skin cancers, including BCC or cSCC

• with lesion characteristics suspicious for any skin cancer, including melanoma (e.g. restricted to those with pigmented lesions only, or including both pigmented and non‐pigmented lesion types);

• those at high risk of developing BCC or cSCC

We excluded studies that recruited only participants with malignant or benign final diagnoses.

We excluded studies conducted in children or which clearly reported inclusion of more than 50% of participants aged 16 and under.

Index tests

Studies reporting accuracy data for visual inspection or dermoscopy, or both, with diagnosis made either in person (face‐to‐face diagnosis) or image‐based (diagnosis based on photographs or dermoscopic images, remotely from the study participant) were eligible for inclusion. We included all established algorithms or checklists to assist diagnosis.

Studies developing new algorithms or methods of diagnosis (i.e. derivation studies) wereincluded if they:

• used a separate independent 'test set' of participants or images to evaluate the new approach; or

• investigated lesion characteristics that had previously been suggested as associated with BCC or cSCC, and the study reported accuracy based on the presence or absence of specific combinations of characteristics.

Studies were excluded if they:

• used a statistical model to produce a data‐driven equation, or algorithm based on multiple diagnostic features, with no separate test set

• used cross‐validation approaches such as 'leave‐one‐out' cross‐validation (Efron 1983)

• evaluated the accuracy of the presence or absence of individual lesion characteristics or morphological features, with no overall diagnosis of malignancy

• reported accuracy data for ‘clinical diagnosis’ with no clear description of whether the reported data related to visual inspection alone or included dermoscopy in all study participants

• were based on the experience of a skin cancer‐specific clinic, where dermoscopy may or may not have been used on an individual basis.

Although primary‐care clinicians can have a specialist interest in skin cancer, for the purposes of this review we considered primary‐care physicians as generalist practitioners and dermatologists as specialists. Within each group, we extracted any reporting of special interest or accreditation in skin cancer.

Target conditions

The primary target conditions were the detection of:

• BCC, including all subtypes;

• Invasive cSCC (we did not consider cutaneous SCC in situ, such as Bowen’s disease, as disease‐positive)

We considered an additional target condition in secondary analyses, namely the detection of:

• any skin cancer, including BCC, cSCC, melanoma or any rare skin cancer (e.g. Merkel cell cancer), as long as skin cancers other than melanoma made up more than 50% of the disease‐positive group. Data from studies in which melanoma accounted for more than 50% of skin cancers were included in our reviews of visual inspection and of dermoscopy compared to visual inspection for the diagnosis of melanoma (Dinnes 2018a; Dinnes 2018b).

Reference standards

The ideal reference standard was histopathological diagnosis in all eligible lesions. A qualified pathologist or dermatopathologist should perform histopathology. Ideally, reporting should be standardised, detailing a minimum dataset to include the type of skin cancer (BCC, cSCC) and subtype of BCC, and may also refer to the tumour, node, and metastasis (TNM) classification of staging for cSCC (Royal College of Pathologists 2014). We did not apply the reporting standard as a necessary inclusion criterion, but extracted any pertinent information.

Partial verification (applying the reference test only to a subset of those undergoing the index test) was of concern, given that lesion excision or biopsy are unlikely to be carried out for all clinically‐benign skin lesions within a representative population sample. We therefore accepted clinical follow‐up of benign lesions as an eligible reference standard, whilst recognising the risk of differential verification bias (as misclassification rates of histopathology and follow‐up will differ).

Additional eligible reference standards included cancer registry follow‐up and 'expert opinion' with no histology or clinical follow‐up. Cancer registry follow‐up is considered less desirable than active clinical follow‐up, as follow‐up is not carried out within the control of the study investigators. Furthermore, if participant‐based analyses are presented as opposed to lesion‐based analyses, it may be difficult to determine whether the detection of a malignant lesion during follow‐up is the same lesion that originally tested negative on the index test.

All of the above are eligible reference standards, with the following caveats:

• all study participants with a final diagnosis of the target disorder must have a histological diagnosis, either subsequent to the application of the index test or after a period of clinical follow‐up, and

• at least 50% of all participants with benign lesions must have either a histological diagnosis or clinical follow‐up to confirm benignity.

Search methods for identification of studies

Electronic searches

The Information Specialist (SB) carried out a comprehensive search for published and unpublished studies. A single large literature search was conducted to cover all topics in the programme grant (see Appendix 1 for a summary of reviews included in the programme grant). This allowed for the screening of search results for potentially relevant papers for all reviews at the same time. A search combining disease related terms with terms related to the test names, using both text words and subject headings was formulated. The search strategy was designed to capture studies evaluating tests for the diagnosis or staging of skin cancer. As the majority of records were related to the searches for tests for staging of disease, a filter using terms related to cancer staging and to accuracy indices was applied to the staging test search, to try to eliminate irrelevant studies, for example, those using imaging tests to assess treatment effectiveness. A sample of 300 records that would be missed by applying this filter was screened and the filter adjusted to include potentially relevant studies. When piloted on MEDLINE, inclusion of the filter for the staging tests reduced the overall numbers by around 6000. The final search strategy, incorporating the filter, was subsequently applied to all bibliographic databases as listed below (Appendix 4). The final search result was cross‐checked against the list of studies included in five systematic reviews; our search identified all but one of the studies, and this study was not indexed on MEDLINE. The Information Specialist devised the search strategy, with input from the Information Specialist from Cochrane Skin. No additional limits were used.

We searched the following bibliographic databases to 29 August 2016 for relevant published studies:

• MEDLINE via OVID (from 1946);

• MEDLINE In‐Process & Other Non‐Indexed Citations via OVID; and

• Embase via OVID (from 1980).

We searched the following bibliographic databases to 30 August 2016 for relevant published studies:

• Cochrane Central Register of Controlled Trials (CENTRAL) Issue 7, 2016, in the Cochrane Library;

• Cochrane Database of Systematic Reviews (CDSR) Issue 8, 2016 in the Cochrane Library;

• Cochrane Database of Abstracts of Reviews of Effects (DARE) Issue 2, 2015;

• CRD Health Technology Assessment (HTA) database Issue 3, 2016; and

• Cumulative Index to Nursing and Allied Health Literature (CINAHL) (via EBSCO from 1960).

We searched the following databases for relevant unpublished studies using a strategy based on the MEDLINE search:

• CPCI (Conference Proceedings Citation Index), via Web of Science™ (from 1990; searched 28 August 2016); and

• SCI Science Citation Index Expanded™ via Web of Science™ (from 1900, using the 'Proceedings and Meetings Abstracts' Limit function; searched 29 August 2016).

We searched the following trials registers using the search terms 'melanoma', 'squamous cell', 'basal cell' and 'skin cancer' combined with 'diagnosis':

• Zetoc (from 1993; searched 28 August 2016).

• The US National Institutes of Health Ongoing Trials Register (www.clinicaltrials.gov); searched 29 August 2016.

• NIHR Clinical Research Network Portfolio Database (www.nihr.ac.uk/research‐and‐impact/nihr‐clinical‐research‐network‐portfolio/); searched 29 August 2016.

• The World Health Organization International Clinical Trials Registry Platform (apps.who.int/trialsearch/); searched 29 August 2016.

We aimed to identify all relevant studies regardless of language or publication status (published, unpublished, in press, or in progress). We applied no date limits.

Searching other resources

We have screened any relevant systematic reviews identified by the searches for their included primary studies, and have included any missed by our searches. We have checked the reference lists of all included papers, and subject experts within the author team have reviewed the final list of included studies. We have conducted no electronic citation searching.

Data collection and analysis

Selection of studies

At least one review author (JDi or NC), screened all titles and abstracts, with any queries discussed and resolved by consensus. A pilot screen of 539 MEDLINE references showed good agreement (89% with a kappa of 0.77) between screeners. We included primary test accuracy studies and test accuracy reviews (for scanning of reference lists) of any test used to investigate suspected melanoma, BCC, or cSCC at initial screening. Inclusion criteria (Appendix 5) were applied independently by both a clinical reviewer (from one of a team of 12 clinician reviewers) and a methodologist reviewer (JDi or NC) to all full‐text articles, with disagreements resolved by consensus or by a third party (JDe, CD, HW, or RM). We contacted authors of eligible studies when insufficient data were presented, to allow for the construction of 2 x 2 contingency tables.

Data extraction and management

One clinical (as detailed above) and one methodologist reviewer (JDi, NC or LFR) independently extracted data for details of the study design, participants, index test(s) or test combinations and criteria for index test positivity, reference standards, and data required to populate a 2 x 2 diagnostic contingency table for each index test, using a piloted data extraction form. We extracted data at all available index test thresholds, resolving disagreements by consensus or by a third party (JDe, CD, HW, and RM).

We contacted authors of included studies where information relating to the diagnostic threshold was missing. We contacted authors of conference abstracts published from 2013 to 2015 to ask whether full data were available. If we could not identify a full paper, we marked conference abstracts as 'pending' and will revisit them in a future review update.

Dealing with multiple publications and companion papers

Where we found multiple reports of a primary study, we maximised yield of information by collating all available data. Where there were inconsistencies in reporting or overlapping study populations, we contacted study authors for clarification in the first instance. If this contact with authors was unsuccessful, we used the most complete and up‐to‐date data source where possible.

Assessment of methodological quality

We assessed risks of bias and applicability of included studies using the QUADAS‐2 checklist (Whiting 2011), tailored to the topic of skin cancer (see Appendix 6). We piloted the modified QUADAS‐2 tool on a small number of full‐text articles included across the full series of diagnostic test accuracy reviews. One clinical and one methodologist reviewer (JDi, NC or LFR) independently assessed quality for the remaining studies, resolving any disagreement by consensus or by a third party where necessary (JDe, CD, HW, and RM).

Statistical analysis and data synthesis

We planned separate analyses according to the point that study participants have reached in the clinical pathway, the clarity with which the pathway could be determined, and the evaluation of in‐person versus image‐based diagnosis.

Our unit of analysis was the lesion rather than the person. This is because (i) in skin cancer initial treatment is directed to the lesion rather than systemically (thus it is important to be able to correctly identify cancerous lesions for each person), and (ii) it is the most common way in which the primary studies reported data. Although there is a theoretical possibility of correlations of test errors when the same people contribute data for multiple lesions, most studies include very few people with multiple lesions and any potential impact on findings is likely to be very small, particularly in comparison with other concerns regarding risk of bias and applicability. For each analysis, we included only one dataset per study, to avoid multiple counting of lesions. We retrieved few studies comparing algorithms, but where we assessed multiple algorithms in an individual study, we selected datasets on the following preferential basis:

• ‘no algorithm’ reported; data presented for clinician’s overall diagnosis or management decision

• pattern analysis or pattern recognition

• ABCD algorithm (or derivatives of) or other established algorithm such as seven‐point checklist, Menzies algorithm or three‐point checklist

• New algorithm developed by study authors

For the diagnosis of BCC (or cSCC), we considered any melanomas or cSCCs (BCCs) that were positively identified in the ‘disease‐negative’ group (i.e. that were mistaken for BCCs) false‐positive results. The clinical management of a lesion considered to be a BCC might be quite different from that for a melanoma or cSCC, and could potentially lead to a negative outcome for the participants concerned; for example, if a treatment other than excision was initiated.

For each index test, algorithm or checklist under consideration, we plotted estimates of sensitivity and specificity on coupled forest plots and in receiver operating characteristic (ROC) space. For tests where commonly‐used thresholds were reported we estimated summary operating points (summary sensitivities and specificities) with 95% confidence and prediction regions using the bivariate hierarchical model (Chu 2006; Reitsma 2005). Where inadequate data were available for the model to converge, we simplified the model, first by assuming no correlation between estimates of sensitivity and specificity and secondly by setting estimates of near‐zero variance terms to zero (Takwoingi 2017). Where all studies reported 100% sensitivity (or 100% specificity) we summed the number with disease (or no disease) across studies and used them to compute a binomial exact 95% confidence interval.

We drew comparisons between visual inspection and dermoscopy results with:

a. all visual inspection and all dermoscopy data from all studies, and then

b. only using data from studies that reported both visual inspection data and dermoscopy data for the same lesions, to enable a robust direct comparison (Takwoingi 2013).

We made comparisons between tests by comparing summary ROC curves using the hierarchical summary receiver‐operator curves (HSROC) model (Rutter 2001) rather than by estimating average operating points, as this approach allows incorporation of data at different thresholds as could arise with different algorithms or checklists. We used an HSROC model that assumed a constant SROC shape between tests and subgroups, but allowed for differences in threshold and accuracy by the addition of covariates. We assessed the significance of the differences between tests by the likelihood ratio test (LR test) assessing differences in both accuracy and threshold, and by a Wald test on the parameter estimate testing for differences in accuracy alone. We provide the P values from both tests in the Tables with the results from the LR test cited in the text, on the basis that differences in threshold between tests is likely. We fitted simpler models when convergence was not achieved due to small numbers of studies, first assuming symmetric SROC curves (setting the shape term to zero), and then setting random‐effects variance estimates to zero.

We present estimates of accuracy from HSROC models as diagnostic odds ratios (DORs) (estimated where the SROC curve crosses the sensitivity = specificity line) with 95% confidence intervals. We present differences between tests and subgroups from HSROC analyses as relative diagnostic odds ratios (RDORs) with 95% confidence intervals. To facilitate interpretation in terms of rates of false‐positive and false‐negative diagnoses, we have computed values of sensitivity at the point on the SROC curve with 80% specificity and of specificity at the point on the SROC curve with 80% sensitivity. We chose these 80% values as they lie within the estimates for most of the analyses. These results should only be considered as illustrative examples of possible sensitivities (and specificities) and differences in sensitivities (and specificities) that could be expected.

Where data were insufficient to estimate HSROC curves (e.g. for the analysis of cSCC),we estimated summary operating points (summary sensitivities and specificities) with 95% confidence and prediction regions using the bivariate hierarchical model (Chu 2006; Reitsma 2005).

For computation of likely numbers of true‐positive, false‐positive, false‐negative and true‐negative findings in the 'Summary of findings' table, we applied these indicative values to the lower quartile, median and upper quartiles of the prevalence observed in the study groups.

We fitted bivariate models using the xtmelogit command in STATA 15, and HSROC models using the NLMIXED procedure in the SAS statistical software package (SAS 2012) and the metadas macro (Takwoingi 2010).

Investigations of heterogeneity

We investigated heterogeneity, comparisons between algorithms and according to observer experience by comparing summary ROC curves using the HSROC model (Rutter 2001), with additional covariates for differences in threshold and accuracy as used for comparing tests.

Sensitivity analyses

We did not conduct any sensitivity analyses.

Assessment of reporting bias

Because of uncertainty about the determinants of publication bias for diagnostic accuracy studies and the inadequacy of tests for detecting funnel plot asymmetry (Deeks 2005), we did not perform tests to detect publication bias.

Results

Results of the search

We identified and screened 34,517 unique references for inclusion. Of these, we reviewed 1051 full‐text papers for eligibility for any one of the suite of reviews of tests to assist in the diagnosis of melanoma or keratinocyte skin cancer. Of the 1051 full‐text papers assessed, we eliminated 848 from all reviews in our series (see Figure 4 PRISMA flow diagram of search and eligibility results).

4.

PRISMA flow diagram.

Of the 466 studies tagged as potentially eligible for any of our reviews of visual inspection or dermoscopy, we include 24 publications in this review. Exclusions were mainly due to the inability to construct a 2 x 2 contingency table based on the data presented (n = 74); the use of ineligible index tests (n = 35; for example: reporting of data for ‘clinical diagnosis’ or for serial use of the index test in a follow‐up context); assessment of individual lesion characteristics (n = 32); or derivation‐type studies developing new algorithms or checklists without a separate training and test set of lesions (n = 31). Other reasons for exclusion included not meeting our requirements for an eligible reference standard (n = 32), ineligible study populations (n = 37) (for example, recruiting only malignant or only benign lesions), inadequate sample size (n = 30), ineligible definition of the target condition (n = 86; including those eligible only for reviews of the detection of melanoma) or with test interpretation by medical students or laypersons (n = 8). A list of the 442 publications excluded from this review with reasons for exclusion is available in Characteristics of excluded studies, with a list of all studies excluded from the full series of reviews available as a separate pdf (please contact skin.cochrane.org for a copy of the pdf).

We contacted the authors of 17 publications concerned with the evaluation of visual inspection or dermoscopy for further data to allow study inclusion; we received responses from four authors with regard to seven publications. Two authors provided additional data but these were insufficient to allow inclusion of the studies (Cabrijan 2008; Warshaw 2009a; Warshaw 2009b; Warshaw 2010a), one replied indicating that dermoscopy was not necessarily used in all study participants (Youl 2007a; Youl 2007b) and one replied but was unable to access the data needed (Fabbrocini 2008). We contacted the authors of a further seven included studies for further details of study methods, and received a responses for four studies; three provided further information about the diagnostic thresholds used (Amirnia 2016; Durdu 2011; Stanganelli 2000) and one provided full anonymised study data (Rosendahl 2011).

The 24 included study publications report on a total of 24 cohorts of lesions and provide 27 visual inspection datasets (8805 lesions; 2579 malignancies) and 33 dermoscopy datasets (6855 lesions; 1444 malignancies). We provide a summary of the tests and target conditions evaluated in each study in Appendix 7. Six studies contributed data for in‐person visual inspection alone (Chang 2013; Cooper 2002; Ek 2005; Hacioglu 2013; Schwartzberg 2005; Steiner 1987); three for dermoscopy added to visual inspection (Amirnia 2016; Durdu 2011; Gokdemir 2011); and five for both in‐person visual inspection alone and combined with dermoscopy (Argenziano 2006; Carli 2002a; Markowitz 2015; Stanganelli 2000; Ulrich 2015). Two studies contributed data for image‐based visual inspection of clinical photographs alone (Lorentzen 1999; Nori 2004); eight for image‐based dermoscopy (Altamura 2010; Carli 2002a; Hacioglu 2013 ; Lorentzen 2008; Menzies 2000; Navarrete Dechent 2016; Witkowski 2016; Zalaudek 2006); and two for both image‐based visual inspection and image‐based dermoscopy (Carli 2002b; Rosendahl 2011). Five studies compared the accuracy of visual inspection with or without dermoscopy to other tests, including: exfoliative cytology (Durdu 2011); computer‐assisted diagnosis (CAD) (Hacioglu 2013); optical coherence tomography (OCT) (Markowitz 2015; Ulrich 2015); and radiographic contrast medium (RCM) (Witkowski 2016). Thirteen studies also contributed data to our reviews of visual inspection (n = 9) and/or dermoscopy (n = 9) for the detection of melanoma (Dinnes 2018a; Dinnes 2018b).

Methodological quality of included studies

We summarise the overall methodological quality of all included studies according to in‐person or image‐based approaches to dermoscopy or to visual inspection. We present 14 studies reporting data for in‐person visual inspection (n = 11) and/or in‐person dermoscopy (added to visual inspection) (n = 8) in Figure 5, with results by study presented in Figure 6. Twelve studies reporting data for image‐based visual inspection (n = 4) and/or image‐based dermoscopy (n = 10) are presented in Figure 7, with results by study presented in Figure 8. Two studies appear in both sets of figures: Carli 2002a evaluated the accuracy of image‐based dermoscopy as well as in‐person visual inspection and dermoscopy, while Hacioglu 2013 reported data for in‐person visual inspection and image‐based dermoscopy.

5.

Risk of bias and applicability concerns graph for in‐person studies: review authors' judgements about each domain presented as percentages across included studies

6.

Risk of bias and applicability concerns summary for in‐person evaluations: review authors' judgements about each domain for each included study

7.

Risk of bias and applicability concerns graph for image‐based evaluations: review authors' judgements about each domain presented as percentages across included studies

8.

Risk of bias and applicability concerns summary for image‐based evaluations: review authors' judgements about each domain for each included study

In‐person evaluations

We judged the risk of bias to be low for most of the studies in only two of five quality domains assessed (dermoscopy index test, reference standard); we judged risk of bias to be high or unclear for most of the studies for participant selection, visual inspection index test, and flow and timing (Figure 5). We rated applicability of study findings as of high or unclear concern in all four domains (participant selection, dermoscopy index tests, visual inspection index tests, reference standards) assessed for all studies apart from one.

For participant selection: we rated three of the 14 studies (21%) at low risk of bias, and three (21%) at high risk (Figure 5) due to exclusion of lesions by size (Hacioglu 2013), or because of missing (Ulrich 2015) or equivocal pathology (Ek 2005). Five studies (36%) did not report the method of participant selection and eight (57%) did not clearly describe exclusions from the study. We rated all studies at high concern for applicability of participants, primarily due to inclusion of lesions selected for biopsy or excision based on the clinical or dermoscopic diagnosis. We judged only one to have included a representative population (Stanganelli 2000). Nine cohorts (64%) also included multiple lesions per participant (Chang 2013; Cooper 2002; Durdu 2011; Ek 2005; Gokdemir 2011; Markowitz 2015; Schwartzberg 2005; Stanganelli 2000; Ulrich 2015) and three did not clearly report the number of included participants (Argenziano 2006; Carli 2002a; Steiner 1987).

For the index test domain: there are eight evaluations of in‐person dermoscopy and 11 evaluations of in‐person visual inspection (Figure 5). For dermoscopy, we rated six evaluations (75%) at low risk of bias, and two did not provide sufficient information to allow us to fully judge the risk of bias. We rated all studies to have made the diagnosis blinded to the reference standard result, given that this is always undertaken prior to histology; six (75%) also clearly reported prespecification of the diagnostic threshold (all using named algorithms or pattern). We judged that all 11 visual‐inspection evaluations had made the diagnosis blinded to the reference standard result. Only three clearly reported prespecification of the threshold used, with two reporting use of formal algorithms (Argenziano 2006; Stanganelli 2000) and one describing the process by which the diagnosis was reached (Ulrich 2015).

We recorded high concern for the applicability of the index tests for three in‐person evaluations of dermoscopy (37%) and for seven evaluations of visual inspection (64%) (Figure 5). For the dermoscopy evaluations this was due to the presentation of average (Argenziano 2006) or consensus diagnoses (Carli 2002a), as opposed to the diagnosis of a single observer, and a lack of description of the diagnostic threshold used (Gokdemir 2011). Only two studies provided sufficient information on which to judge the level of observer expertise in dermoscopy (Carli 2002a; Gokdemir 2011). For visual inspection, we noted high concerns due to the presentation of average (Argenziano 2006) or consensus (Carli 2002a; Steiner 1987) diagnoses, or lack of detail about the threshold for diagnosis (Carli 2002a; Chang 2013; Cooper 2002; Ek 2005; Hacioglu 2013; Steiner 1987). Most studies (7/11) did not provide sufficient information on which to judge the level of observer expertise in lesion diagnosis.

For the reference standard: We judged all studies except Stanganelli 2000 at low risk of bias due to the use of an acceptable reference standard (73%) (Figure 5). In Stanganelli 2000 only 8% of included lesions underwent excision, with the remaining 3110 ‘benign’ diagnosed assumed to be benign based on cancer registry follow‐up. Blinding of the reference standard to the index test was recorded but did not contribute to the overall risk of bias for this domain. Blinding of the reference standard was reported in only one study (Amirnia 2016). The applicability of the reference standard was of low concern in one evaluation reporting pathology review by an expert histopathologist (Argenziano 2006), and we rated the remaining 13 (93%) as unclear.

For participant flow and timing: We rated five studies at low risk of bias (36%), three as unclear (21%), and six at high risk of bias (43%) (Figure 5). Of those at high risk, one did not use the same reference standard for all participants (Stanganelli 2000), and five did not include all participants in the analysis. Seven studies were unclear on the interval between the application of the index test and excision for histology.

Image‐based evaluations

Across the 12 studies providing image‐based data, we rated risk of bias to be high or unclear for at least half of the studies in all domains, apart from the reference standard domain (Figure 7). We also scored applicability of study findings as of high concern in almost all studies, apart from for the reference standard domain.

For participant selection: We judged six of the 12 evaluations (50%) at high risk of bias, four did not provide sufficient information to judge this domain, and two were at low risk of bias (Figure 7). Three studies (25%) used a case‐control design with separate sampling of malignant and benign lesions (Altamura 2010; Menzies 2000; Nori 2004), and two (17%) excluded lesions on the basis of size (Hacioglu 2013) or type of lesion (Navarrete Dechent 2016, excluding seborrhoeic keratosis). Five evaluations (42%) did not report the method of participant selection and six (50%) did not clearly describe exclusions from the study. We rated all evaluation cohorts at high concern for applicability of participants, primarily due to the restricted inclusion of lesions selected for excision or biopsy. Two studies also reported including multiple lesions per participant (Navarrete Dechent 2016; Rosendahl 2011).

For the index test domain: There are 10 evaluations of image‐based dermoscopy and four evaluations of visual inspection of clinical images (Figure 7). Insufficient information was provided on which to judge the risk of bias for visual inspection, due to unclear prespecification of the threshold for diagnosis of skin cancer. For dermoscopy, we rated five evaluations (50%) at low risk of bias, four as unclear (36%) and one at high risk. The high‐risk study developed a new algorithm for dermoscopy using characteristics previously suggested to be associated with BCC, but did not use a separate training set to develop the algorithm (Navarrete Dechent 2016). Four studies did not clearly report prespecification of the diagnostic threshold used (Altamura 2010; Carli 2002b; Hacioglu 2013; Witkowski 2016).

We had high concern for the applicability of the index tests for all four visual‐inspection and nine of 10 dermoscopy evaluations, due to the use of image‐based interpretations. None of the visual‐inspection evaluations provided further information on the participants concerned, and two presented average (Lorentzen 1999) or consensus (Carli 2002b) diagnoses. None of the four provided sufficient detail about the diagnostic threshold used. For dermoscopy, nine studies reported blinded interpretation of dermoscopic images and six reported average (Lorentzen 2008; Zalaudek 2006) or consensus (Carli 2002a; Carli 2002b; Navarrete Dechent 2016) diagnoses, or were not clear on the data provided (Menzies 2000). One study reported presentation of the clinical photograph of the lesion alongside the dermoscopic image (Rosendahl 2011), and also presented data for a single observer. Four studies provide insufficient information on the diagnostic threshold (Carli 2002b; Hacioglu 2013; Lorentzen 2008; Witkowski 2016) and four did not provide details of the observer expertise (Hacioglu 2013; Menzies 2000; Witkowski 2016; Zalaudek 2006).

For the reference standard: We judged 11 (92%) of the 12 included image‐based studies at low risk of bias (Figure 7). We considered Nori 2004 to be at high risk, as it did not meet our criteria for an adequate reference standard (histology or clinical follow‐up in at least 80% of benign lesions). Blinding of the reference standard to the original clinical diagnosis was not reported in any study. We judged the applicability of the reference standard to be of unclear concern in 11 studies, due to a lack of detail about the expertise of the histopathologist or by a dermatopathologist. Nori 2004 was of high concern, due to the use of expert opinion for classifying the final diagnosis of some lesions.

For participant flow and timing: Six studies were at high risk of bias (50%), four at low risk (33%) and two (17%) did not provide enough information on which to judge this domain (Figure 7). Of those at high risk, one evaluations did not use the same reference standard for all participants (differential verification) (Nori 2004), and none of the six included all participants in the analysis. Seven studies (58%) were unclear on the interval between the application of the index test and lesion excision, with only five (42%) considered to report consecutive diagnosis and excision or biopsy (Carli 2002b; Hacioglu 2013; Lorentzen 1999; Menzies 2000; Witkowski 2016).

Findings

1. Target condition: BCC

Twenty‐one studies reported accuracy data for the detection of BCC. Twelve studies provided data for visual inspection alone; eight evaluations were conducted in person and four were image‐based. Fifteen studies reported accuracy data for the detection of BCC by using dermoscopy; seven evaluations were in person and nine were image‐based. One study reported dermoscopy data for both in‐person and image based dermoscopy (Carli 2002a).

We provide summary details of the in‐person and image‐based studies in Appendix 8. We present results for the primary analyses in Table 2, with heterogeneity investigations presented in Table 3 and Table 4. Forest plots of study data for each analysis are shown in Figure 9 and Figure 10; summary estimates for in‐person comparisons are depicted in Figure 11 and Figure 12, and for image‐based comparisons in Figure 13 and Figure 14.

1. Comparison of visual inspection and dermoscopy for detection of BCC.

Test	Datasets	Lesions (BCCs)	DOR (95% CI)	Specificity at 80% sensitivity	Sensitivity at 80% specificity	Relative DOR (95% CI)	P value (LR)a	P value (Wald)b
In‐person evaluations
Visual inspection	8	7017 (1586)	19.9 (7.8 to 51.2)	77%	79%	8.2 (3.5 to 19.3)	< 0.001	< 0.001
Visual inspection + Dermoscopy	7	4683 (363)	164 (56.8 to 475)	99%	93%
In‐person evaluations (direct studies)
Visual inspection	4	3974 (257)	12.8 (3.3 to 48.8)	36%	71%	7.5 (2.7 to 21.3)	< 0.001	< 0.001
Visual inspection + Dermoscopy	4	3974 (258)	96.2 (21.1 to 439)	97%	87%
Image‐based evaluations
Visual inspection (clinical images)	4	853 (156)	26.8 (11.9, 60.4)	87%	85%	3.9 (1.2, 5.0)	0.006	0.025
Dermoscopic images	9	2271 (737)	75.7 (21.3, 269)	96%	93%
Image‐based evaluations (direct studies)
Visual inspection (clinical images)	2	516 (82)	81.1 (39.1, 168)	95%c	95%c	Not estimable	Not estimable		Not estimable
Dermoscopic images	2	516 (79)	275.5 (112, 678)	99%c	99%c

BCC ‐ basal cell carcinoma; DOR ‐ diagnostic odds ratio; RDOR ‐ relative diagnostic odds ratio; CI ‐ confidence interval; LR ‐ likelihood ratio.

^aTests whether there is a difference in test performance between defined groups in terms of either DOR or threshold.
 ^bTests the significance of the difference in DOR between defined groups at a particular SROC curve intercept value.
 ^cComputed assuming symmetric SROC curve.

2. Investigations of sources of heterogeneity for studies of visual inspection for detection of BCC.

Test	Datasets	Lesions (BCCs)	DOR (95% CI)	Specificity at 80% sensitivity	Sensitivity at 80% specificity	Relative DOR (95% CI)	P value (LR)a	P value (Wald)b
Difference in‐person and image based
In‐person	8	7017 (1586)	11.9 (4.4 to 32.2)	64%	74%	0.45 (0.26 to 9.2)	0.88	0.62
Image	4	853 (156)	18.5 (4.3 to 80.6)	78%	79%
Prevalence
0% ‐ 25%	6	4643 (168)	50.5 (17.1 to 149)	94%	91%	9.7 (2.3 to 40.8)	0.002	0.002
> 25%	6	3227 (1574)	5.2 (2.3 to 11.7)	50%	60%

BCC ‐ basal cell carcinoma; DOR ‐ diagnostic odds ratio; RDOR ‐ relative diagnostic odds ratio; CI ‐ confidence interval; LR ‐ likelihood ratio

^aTests whether there is a difference in test performance between defined groups in terms of either DOR or threshold.
 ^bTests the significance of the difference in DOR between defined groups at a particular SROC curve intercept value.

3. Investigations of sources of heterogeneity for studies of dermoscopy for detection of BCC.

Test	Datasets	Lesions (cases)	DOR (95% CI)	Specificity at 80% sensitivity	Sensitivity at 80% specificity	Relative DOR (95% CI)	P value (LR)a	P value (Wald)b
Difference in person and image based
In person	7	4683 (363)	388 (68.6 to 2194)	100%	96%	4.0 (0.46 to 33.8)	0.39	0.21
Image	9	2271 (737)	98.2 (21.6 to 446)	98%	91%
Use of an algorithm
No algorithm	9	5427 (338)	371 (86.9 to 1587)	100%	98%	7.8 (0.90 to 68.2)	0.004	0.06
Any algorithm	7	1527 (762)	47.4 (10.2 to 219)	94%	90%
Prevalence (in‐person studies)
0% ‐ 25%	9	5524 (349)	309 (69.2 to 1380)	100%	97%	4.5 (0.49 to 41.8)	0.04	0.18
> 25%	7	1430 (751)	68.4 (13.2 to 356)	96%	91%

BCC ‐ basal cell carcinoma; DOR ‐ diagnostic odds ratio; RDOR ‐ relative diagnostic odds ratio; CI ‐ confidence interval; LR ‐ likelihood ratio

^aTests whether there is a difference in test performance between defined groups in terms of either DOR or threshold.
 ^bTests the significance of the difference in DOR between defined groups at a particular SROC curve intercept value.

9.

In‐person evaluations of the accuracy of visual inspection and visual inspection plus dermoscopy (VI+Dermoscopy) according to BCC prevalence and use of a formal algorithm

10.

Image‐based evaluations of the accuracy of visual inspection and dermoscopy alone according to BCC prevalence and use of a formal algorithm

11.

Comparison of the accuracy of visual inspection with visual inspection plus dermoscopy (VI+Dermoscopy) for detection of BCC from in‐person studies

12.

Paired comparisons of the accuracy of visual inspection with visual inspection plus dermoscopy for detection of BCC from in‐person studies

13.

Comparison of the accuracy of image‐based visual inspection with image‐based dermoscopy for detection of BCC

14.

Paired comparisons of the accuracy of visual inspection with visual inspection plus dermoscopy for detection of BCC from image‐based studies

Analyses by clinical pathway and in‐person versus image‐based design

Attempts to classify studies according to where on the clinical pathway they had been conducted were hindered by lack of information. We considered that only eight studies had provided a clear description of the prior testing of included participants and only three were conducted in a limited prior testing population, as opposed to studies in participants referred for specialist assessment (Appendix 8). We were therefore unable to analyse data by pathway for either visual inspection or for dermoscopy.

We found no clear differences in accuracy between studies undertaken in person and those which evaluated images (Table 3 and Table 4). The accuracy of visual inspection was non‐significantly lower for in‐person studies of visual inspection compared to image‐based (relative diagnostic odds ratio (RDOR) 0.45, 95% confidence interval (CI) 0.26 to 9.2, LR test P = 0.88) (Table 3; Figure 15), while the accuracy of in‐person dermoscopy was non‐significantly higher compared to diagnosis based on dermoscopic images (RDOR 4.0, 95% CI 0.46 to 33.8; LR test P = 0.39) (Table 4; Figure 16). The lack of effect observed is probably due to other sources of heterogeneity, particularly given the much bigger and highly‐significant effect observed for this analysis for the detection of melanoma (Dinnes 2018a). We elected to undertake our primary analyses separately for in‐person and image‐based analyses, to be consistent with the approach used in the melanoma review.

15.

Comparison of the accuracy of visual inspection for detection of BCC between in‐person and image‐based

16.

Comparison of the accuracy of dermoscopy for detection of BCC between in‐person (VI+Dermoscopy) and image‐based (Dermoscopy alone)

In‐person evaluations

The 11 studies reporting in‐person evaluations of visual inspection alone (n = 4; Cooper 2002; Ek 2005; Schwartzberg 2005; Steiner 1987), for visual inspection plus dermoscopy (n = 3; Amirnia 2016; Durdu 2011; Gokdemir 2011) or for both (n=4; Carli 2002a; Markowitz 2015; Stanganelli 2000; Ulrich 2015) were all conducted in referred populations undergoing biopsy or excision (Appendix 9). Three were considered to have been conducted in participants with equivocal lesions (Markowitz 2015; Steiner 1987; Ulrich 2015) and one in participants at high risk for developing skin cancer following renal transplantation (Cooper 2002). Seven evaluations were prospective case series, one was retrospective (Stanganelli 2000), and three did not clearly report the direction of the design (Amirnia 2016; Carli 2002a; Gokdemir 2011).

Five of the 11 studies primarily aimed to examine accuracy for the detection of BCC (Amirnia 2016; Markowitz 2015; Schwartzberg 2005; Ulrich 2015) or ‘non‐melanoma’ skin cancer (Cooper 2002), while the remaining six also provided data for our reviews of visual inspection or dermoscopy or both for the diagnosis of melanoma (Dinnes 2018a; Dinnes 2018b). Two evaluations included any lesion considered suspicious for skin cancer (Ek 2005; Cooper 2002); two included lesions suspicious for BCC (Amirnia 2016; Schwartzberg 2005), one of these restricted to lesions on the face (Amirnia 2016); five included only pigmented lesions (Carli 2002a; Durdu 2011; Gokdemir 2011; Stanganelli 2000; Steiner 1987) and two to non‐pigmented ‘pink’ lesions (Markowitz 2015; Ulrich 2015), one of these restricted to head and neck lesions only (Markowitz 2015). The prevalence of BCC ranged from 1% (Stanganelli 2000) to 61% (Markowitz 2015); median 17% (interquartile range (IQR) 10, 53%). The lowest prevalence was generally observed in the studies in pigmented lesions (1% to 10% in four studies) and the highest in non‐pigmented or lesions suspicious for BCC (58% to 61% in three studies). Six studies reported including invasive melanoma or melanoma in situ (Carli 2002a; Durdu 2011; Ek 2005; Gokdemir 2011; Stanganelli 2000; Steiner 1987) and two included cSCC (Cooper 2002; Ek 2005) in the disease‐negative group.

Diagnosis was recorded by dermatologists or clinicians presumed to be dermatologists (based on author’s institutions) in most of the studies (9/11; 82%), a mixed group of dermatology residents (trainees) and consultants (Cooper 2002) or plastic surgery residents, consultants and a clinical assistant (Ek 2005). Where reported (n = 7), the number of observers ranged from 1 to 17 (median 2).

Test accuracy was reported for a single observer in just over half of the evaluations (n = 6), for a consensus of two or three observers in two (Carli 2002a; Steiner 1987), and this information was not reported by the remaining three evaluations (Ek 2005; Gokdemir 2011; Markowitz 2015).

Visual inspection (in‐person)

Across the eight evaluations of visual inspection, no formal algorithm to assist diagnosis was reported in 87% (n = 7) and one reported using the ABCD approach (Stanganelli 2000). Sensitivity ranged from 20% to 90% and specificity from 29% to 100% (Figure 9). Examinations in six studies were undertaken by dermatologists, (or were assumed to be dermatologists, based on study institution) and in two studies by consultant or registrar dermatologists (Cooper 2002) or plastic surgeons (Ek 2005). The lowest sensitivities were reported in studies restricted to pigmented lesions, particularly Carli 2002a and Stanganelli 2000. We pooled results across algorithms and thresholds as a summary ROC curve (7017 lesions; 1586 BCCs; Figure 11). Estimates of accuracy obtained from the curve suggest that the specificity of visual inspection would be 77% at a fixed threshold of 80% sensitivity, and sensitivity would be 79% at a fixed threshold of 80% specificity (Table 2). We chose these 80% fixed values as they lie within the estimates for most of the analyses and should only be considered as illustrative examples of the values that might be achieved based on the observed data (Statistical analysis and data synthesis). Of the three datasets which included melanomas in the disease‐negative group (Carli 2002a; Stanganelli 2000; Steiner 1987), five of the 15 false positive results were melanoma mistaken for BCCs (Carli 2002a; Steiner 1987).

Dermoscopy added to visual inspection

For the seven evaluations of dermoscopy added to visual inspection, two did not report using any algorithm to assist diagnosis (Durdu 2011; Gokdemir 2011), two used pattern analysis (Carli 2002a; Stanganelli 2000), and three used formal algorithms to assist diagnosis, including the three‐point checklist for BCC (Amirnia 2016) and the Marghoob and colleagues (Marghoob 2010) two‐step approach for classifying skin lesions (Markowitz 2015; Ulrich 2015). Sensitivity ranged from 79% to 100% and specificity from 54% to 100% (Figure 9). The low specificities of 54% (Ulrich 2015) and 56% (Markowitz 2015) appeared as outliers (with non‐overlapping confidence intervals), all other studies having specificities of 96% or above. Both studies included particularly high percentages of BCC (60% to 61%) and included non‐pigmented lesions with a high clinical suspicion of being BCC.

We pooled results across algorithms and thresholds as a summary ROC curve (4683 lesions; 363 BCCs; Figure 11). Estimates of accuracy obtained from the curve suggest that the specificity of dermoscopy would be 99% at a fixed threshold of 80% sensitivity, and sensitivity would be 93% at a fixed threshold of 80% specificity (Table 2). Of the four datasets which included melanomas in the disease‐negative group (Carli 2002a; Durdu 2011; Gokdemir 2011; Stanganelli 2000), three of the 19 false‐positive results were melanoma mistaken for BCCs (Durdu 2011; Gokdemir 2011).

Comparison of in‐person dermoscopy added to visual inspection versus visual inspection alone

The accuracy of visual inspection was compared with the accuracy of dermoscopy estimated from (a) all eight in‐person visual inspection and all seven dermoscopy studies (Figure 11) and (b) estimated from direct comparisons in the subset of four studies that evaluated both visual inspection and dermoscopy on an in‐person basis (3974 lesions; 258 BCCs; Figure 12). In both comparisons the accuracy of dermoscopy in addition to visual inspection exceeded that of visual inspection alone (Table 2). In (a) the diagnostic odds ratio (DOR) for dermoscopy was 8.2 (95% CI 3.5 to 19.3; LR test P < 0.001) times that of visual inspection alone; in (b) it was 7.5 (95% CI 2.7 to 21.3; LR test P < 0.001) times that of visual inspection alone. These effects correspond to predicted differences in specificity of (a) 22% (99% versus 77%) and (b) 61% (97% versus 36%) at a fixed sensitivity of 80% (Table 2) and predicted differences in sensitivity of (a) 14% (93% versus 79%) and (b) 16% (87% versus 71%) at a fixed specificity of 80% (Table 2).

Image‐based evaluations

The 11 studies reporting image‐based diagnosis using clinical photographs (n = 2; Lorentzen 1999; Nori 2004), dermoscopic images (n = 7; Altamura 2010; Carli 2002a; Lorentzen 2008; Menzies 2000; Navarrete Dechent 2016; Witkowski 2016; Zalaudek 2006) or both (n = 2; Carli 2002b; Rosendahl 2011) were primarily conducted in referred populations undergoing biopsy or excision (Appendix 9). Two studies were conducted in a limited prior testing setting, recruiting participants from primary care (Rosendahl 2011) or from a private dermatology practice (Navarrete Dechent 2016). Of the remaining nine, one was conducted in participants with equivocal lesions (Witkowski 2016). Two evaluations used a case‐control design, separately recruiting diseased and non‐diseased participants (Altamura 2010; Menzies 2000), one was a prospective case series (Lorentzen 1999), five retrospectively selected series of images for prospective interpretation within the context of the study (Navarrete Dechent 2016; Nori 2004; Rosendahl 2011; Witkowski 2016; Zalaudek 2006), and three did not clearly report the direction of the design (Carli 2002a; Carli 2002b; Lorentzen 2008).

Five of the 11 studies primarily aimed to examine accuracy for the detection of BCC (Altamura 2010; Menzies 2000; Navarrete Dechent 2016; Nori 2004; Witkowski 2016), while the remaining six also provided data for our reviews of visual inspection or dermoscopy or both for the diagnosis of melanoma (Dinnes 2018a; Dinnes 2018b). Four evaluations included any lesion, pigmented or non‐pigmented (Altamura 2010; Lorentzen 1999; Lorentzen 2008; Zalaudek 2006); four included only pigmented lesions (Carli 2002a; Carli 2002b; Menzies 2000; Rosendahl 2011); two included non‐pigmented lesions only (Navarrete Dechent 2016; Witkowski 2016), and one included biopsy‐confirmed BCCs and lesions with a range of common diagnoses (Nori 2004). The prevalence of BCC ranged from 2% (Carli 2002a) to 63% (Navarrete Dechent 2016); median 16% (IQR 11, 47%). The highest prevalence was generally observed in the studies in non‐pigmented lesions or lesions suspicious for BCC (44% to 63% in four studies, one of which used a case‐control design; Altamura 2010). All studies apart from Nori 2004 reported including invasive melanoma or melanoma in situ, and five also included cSCC in the disease‐negative group (Altamura 2010; Navarrete Dechent 2016; Nori 2004; Rosendahl 2011; Witkowski 2016).

Diagnosis was recorded by dermatologists or clinicians presumed to be dermatologists (based on author’s institutions) in most of the studies (9/11; 73%), or by a mixed group of clinicians in two (Lorentzen 1999; Zalaudek 2006). Where reported (n = 9), the number of observers ranged from two (reported for five studies) to 150 (median 2).

Test accuracy was reported for a single observer in four studies, for a consensus of two observers in three (Carli 2002a; Carli 2002b; Navarrete Dechent 2016), the average across observers in three (Lorentzen 1999; Lorentzen 2008; Zalaudek 2006), and this information was not reported by one (Menzies 2000).

Visual inspection of clinical photographs

The four evaluations of image‐based visual inspection reported no formal algorithm to have been used to assist diagnosis. Sensitivity ranged from 48% to 89%, and specificity from 62% to 98% (Figure 10). We pooled results as a summary ROC curve (853 lesions; 156 BCCs; Figure 13). Estimates of accuracy obtained from the curve suggest that the specificity of image‐based visual inspection would be 87% at a fixed threshold of 80% sensitivity, and sensitivity would be 85% at a fixed threshold of 80% specificity (Table 2). Of the three datasets which included melanoma in the disease‐negative group (Carli 2002b; Lorentzen 1999; Rosendahl 2011), three of 39 false‐positive results were melanoma mistaken for BCCs (Rosendahl 2011).

Dermoscopic image‐based diagnosis

Of the nine evaluations of image‐based dermoscopy, two did not report using any algorithm to assist diagnosis (Carli 2002b; Witkowski 2016), three used pattern analysis (Carli 2002a; Lorentzen 2008; Rosendahl 2011), and four used formal algorithms to assist diagnosis, including the three‐point checklist (Zalaudek 2006), the Menzies algorithm for BCC (Menzies 2000) or a modification thereof (Altamura 2010), or a new algorithm ‘shiny white blotches and strands' (Navarrete Dechent 2016). Only one study provided the clinical photograph alongside the dermoscopic image (Rosendahl 2011), with the rest reporting blinded dermoscopy interpretations. Sensitivity ranged from 40% to 97% and specificity from 50% to 100% (Figure 10). We observed particularly low sensitivities in Carli 2002a and Navarrete Dechent 2016 (which respectively had the lowest (2%) and highest (63%) prevalence of BCC), the latter also reporting the lowest specificity (50%). All other studies reported sensitivities of 85% or above and specificities of 72% or more.

We pooled results across algorithms and thresholds as a summary ROC curve (2271 lesions; 737 BCCs; Figure 13). Estimates of accuracy obtained from the curve suggest that the specificity of dermoscopy would be 96% at a fixed threshold of 80% sensitivity, and sensitivity would be 93% at a fixed threshold of 80% specificity (Table 2). All nine evaluations included melanomas in the disease‐negative group; 23 of the 178 false‐positive results were melanomas mistaken for BCCs in five studies (Menzies 2000; Navarrete Dechent 2016; Rosendahl 2011; Witkowski 2016; Zalaudek 2006) and 45 were cSCCs mistaken for BCCs (Navarrete Dechent 2016; Witkowski 2016). Navarrete Dechent 2016 alone was responsible for 53 false positives (44 cSCC and nine melanomas).

Comparison of diagnosis based on dermoscopic images versus visual inspection of images

We compared the accuracy of image‐based visual inspection with the accuracy of dermoscopy estimated from (a) all four image‐based visual inspection and all nine dermoscopy studies (Figure 13), and (b) estimated from direct comparisons in the subset of two studies that evaluated both clinical photographs and dermoscopic images (516 lesions; 79 BCCs; Figure 14). In both comparisons the accuracy of dermoscopy in addition to visual inspection exceeded that of visual inspection alone (Table 2). In (a) the DOR for dermoscopy was 3.9 (95% CI 1.2 to 5.0, LR test P = 0.006) times that of visual inspection alone, and in (b) the RDOR was not estimable but the DOR of 275.5 (95% CI 112 to 678) for dermoscopy exceeded visual inspection alone (DOR 81.1, 95% CI 39.1 to 168). These effects correspond to predicted differences in specificity of (a) 9% (96% versus 87%) and (b) 4% (99% versus 95%) at a fixed sensitivity of 80% (Table 2), and predicted differences in sensitivity of (a) 8% (93% versus 85%) and (b) 4% (99% versus 95%) at a fixed specificity of 80% (Table 2).

Secondary analyses for the detection of BCC

Covariate investigations

Table 3 and Table 4 report the results of the heterogeneity investigations for visual inspection and for dermoscopy respectively. As discussed above, we found no clear differences in accuracy between studies undertaken in person and those which evaluated images for either test. Although our primary analyses are presented separately for in‐person and image‐based approaches, due to a paucity of data we have based all subsequent covariate investigations on the complete datasets for each test.

Visual inspection: Due to a lack of data, we could not investigate the use of a formal algorithm versus no formal algorithm for visual inspection. Observed accuracy was significantly higher, however, where disease prevalence of BCC was 25% or less (RDOR 9.7, 95% CI 2.3 to 40.8; LR test P = 0.002), compared to those where disease prevalence was greater than 25% (Table 3). This result appears to be driven by lower specificities with non‐overlapping confidence intervals in the studies in the higher‐prevalence group, most of which were conducted in populations with lesions suspicious for BCC (Schwartzberg 2005; Ulrich 2015; Markowitz 2015; Nori 2004). Sensitivities reported in these studies were largely within the range of those reported by studies in the lower prevalence group (Appendix 10).

Dermoscopy: Observed accuracy was somewhat higher in studies using no formal algorithm to assist diagnosis, as opposed to those reporting use of an algorithm (RDOR 7.8, 95% CI 0.90 to 68.2; LR test P = 0.004) Table 4. Accuracy was also non‐significantly higher where disease prevalence of BCC was 25% or less (RDOR 4.5, 95% CI 0.49 to 41.8; LR test P = 0.04), compared to those with disease prevalence greater than 25% (Table 4). There is considerable overlap in the studies included in the ‘named algorithm’ and higher‐prevalence groups (with six of the seven same studies appearing in each group: Altamura 2010; Amirnia 2016; Markowitz 2015; Menzies 2000; Navarrete Dechent 2016; Ulrich 2015). It seems likely that both factors play a role in the observed differences in accuracy (Appendix 10).

Analyses by algorithms used to assist diagnosis

We provide details of the algorithms used to assist diagnosis in Appendix 9. We report results by algorithm used (or not used) in Table 5 for each of the target conditions under consideration in this review.

4. Algorithm and threshold analysis for each definition of the target condition.

Target condition Test	No Datasets	Lesions (Cases)	Pooled Sensitivity (95% CI)	Pooled Specificity (95% CI)	No studies	Lesions (Cases)	Pooled Sensitivity (95% CI)	Pooled Specificity (95% CI)
a. BCC – Visual inspection	IN‐PERSON				IMAGE‐BASED
No algorithm at any threshold	7	3645 (1543)	0.68 (0.48 to 0.83)	0.82 (0.55 to 0.95)	4	853 (156)	0.71 (0.51 to 0.86)	0.92 (0.76 to 0.98)
No algorithm at BCC possible	1	141 (82)	0.89 (0.80 to 0.95)	0.37 (0.25 to 0.51)	1	105 (58)	0.78 (0.65 to 0.87)	0.38 (0.25 to 0.54)
ABCD threshold not reported	1	3372 (43)	0.49 (0.33 to 0.65)	1.00 (1.00 to 1.00)	‐	‐	‐	‐
Schwartzberg algorithm	1	141 (82)	0.89 (0.80 to 0.95)	0.37 (0.25 to 0.51)	‐	‐	‐	‐
b. BCC – Dermoscopy	IN‐PERSON				IMAGE‐BASED
Algorithm threshold not reported	2	648 (79)	0.92 (0.84 to 0.97)	0.97 (0.95 to 0.98)	2	313 (121)	0.85 (0.78 to 0.90)	0.93 (0.88 to 0.96)
Pattern analysis	2	3628 (48)	0.79 (0.65 to 0.88)	1.00 (1.00 to 1.00)	2	582 (85)	0.89 (0.81 to 0.94)	0.98 (0.96 to 0.99)
3 point at ≥ 2	1	61 (27)	1.00 (0.87 to 1.00)	0.97 (0.85 to 1.00)	1	150 (18)	0.89 (0.65 to 0.99)	0.72 (0.63 to 0.79)
2‐step algorithm	2	346 (209)	0.86 (0.76 to 0.92)	0.55 (0.46 to 0.63)	‐	‐	‐	‐
Menzies for BCC (new)	‐	‐	‐	‐	1	213 (71)	0.97 (0.90 to 1.00)	0.92 (0.87 to 0.96)
Menzies for BCC (revised)	‐	‐	‐	‐	1	300 (150)	0.95 (0.91 to 0.98)	0.87 (0.81 to 0.92)
New SWS at ≥ 1	‐	‐	‐	‐	1	457 (287)	0.54 (0.48 to 0.60)	0.50 (0.42 to 0.58)
Chaos/clues	‐	‐	‐	‐	1	463 (72)	0.99 (0.93 to 1.00)	0.55 (0.50 to 0.60)
c. cSCC – Visual inspection	IN‐PERSON				IMAGE‐BASED
No algorithm at threshold NR	2	2684 (538)	0.59 (0.42 to 0.82)	0.79 (0.77 to 0.81)	‐	‐	‐	‐
d. cSCC – Dermoscopy	IN‐PERSON				IMAGE‐BASED
No algorithm at threshold NR	‐	‐	‐	‐	1	260 (13)	0.77 (0.46 to 0.95)	0.97 (0.94 to 0.99)
SWS at > 1 char	‐	‐	‐	‐	1	457 (106)	0.42 (0.32 to 0.51)	0.49 (0.43 to 0.54)
e. Any – Visual inspection	IN‐PERSON				IMAGE‐BASED
No algorithm at threshold NR	4	3533 (1968)	0.91 (0.79 to 0.96)	0.61 (0.25 to 0.87)	2	517 (124)	0.77 (0.68 to 0.83)	0.84 (0.80 to 0.87)
ABCD at threshold NR	1	85 (53)	0.57 (0.42 to 0.70)	0.50 (0.32 to 0.68)	‐	‐	‐	‐
f. Any – Dermoscopy	IN‐PERSON				IMAGE‐BASED
No algorithm at threshold NR	1	200 (46)	0.98 (0.88 to 1.00)	0.98 (0.94 to 1.00)	3	393 (187)	0.89 (0.84 to 0.93)	0.79 (0.73 to 0.84)
No algorithm at excise	‐	‐	‐	‐	1	260 (140)	0.95 (0.90 to 0.98)	0.53 (0.44 to 0.62)
Pattern analysis	‐	‐	‐	‐	1	463 (104)	0.79 (0.70 to 0.86)	0.88 (0.85 to 0.91)
3 point at ≥ 2	1	77 (39)	0.85 (0.69 to 0.94)	0.26 (0.13 to 0.43)	‐	‐	‐	‐
Menzies for BCC (revised)	‐	‐	‐	‐	1	213 (142)	0.95 (0.90 to 0.98)	0.92 (0.83 to 0.97)
SWS	‐	‐	‐	‐	1	457 (414)	0.50 (0.45 to 0.55)	0.63 (0.47 to 0.77)
Chaos/Clues	‐	‐	‐	‐	1	463 (104)	0.92 (0.85 to 0.97)	0.58 (0.53 to 0.63)

BCC ‐ basal cell carcinoma; CI ‐ confidence interval; SWS ‐ shiny white streaks; NR ‐ not reported

For the diagnosis of BCC, Table 5 highlights the lack of available data for formal algorithms to diagnose BCC, particularly for visual inspection. Although a number of dermoscopic algorithms have been evaluated for the diagnosis of BCC, only the Menzies algorithm appears to show promise in terms of increasing sensitivity without sacrificing the specificity which can be achieved by observer diagnosis alone (with no algorithm). The data, however, come from the same study which developed the algorithm using dermoscopic images, and it remains to be seen whether results can be replicated on an in‐person basis (Menzies 2000).

Analyses by observer experience

Observer experience was generally poorly described in the study reports (Appendix 8), but we attempted broad classifications by reported expertise in visual inspection or dermoscopy, regardless of an in‐person or image‐based approach to diagnosis. The resulting study subgroups were small, and results highly heterogeneous, so we could undertake no further analyses by observer expertise. None of the included studies provided direct comparisons of observer accuracy according to expertise or qualifications.

2. Target condition: cSCC

Four studies reported accuracy data for the detection of cSCC. Two studies provided data for in‐person visual inspection (Cooper 2002; Ek 2005) and two for image‐based dermoscopy (Navarrete Dechent 2016; Witkowski 2016) (Appendix 8). We present results for the primary analyses in Table 6. Forest plots of study data are given in Figure 17.

5. Comparison of visual inspection and dermoscopy for the detection of cSCC.

Test	Datasets	Lesions (cSCC)	DOR (95% CI)	Summary sensitivity	Summary specificity
In‐person evaluations
Visual inspection	2	2684 (538)	5.0 (4.1 to 6.1)	0.57 (0.53 to 0.61)	0.79 (0.77 to 0.81)
Visual inspection + Dermoscopy	0	‐	‐	‐	‐
Image‐based evaluations
Visual inspection (clinical images)	0	‐	‐	‐	‐
Dermoscopic images	2	717 (119)	6.5 (0.45 to 93.2)	0.55 (0.29 to 0.79)	0.84 (0.32 to 0.98)

cSCC ‐ cutaneous squamous cell carcinoma; DOR ‐ diagnostic odds ratio; CI ‐ confidence interval

17.

Evaluations of the accuracy of visual inspection or dermoscopy for detecting invasive melanoma cSCC

Visual inspection (in‐person)

Both studies of visual inspection were conducted in secondary clinic specialist clinics, one of which was provided for renal transplant recipients (Cooper 2002). Both studies included participants with a range of different lesion types that might be observed in clinical practice. The prevalence of cSCC was 21% (Cooper 2002) and 20% (Ek 2005). Both studies reported data for observers’ correct diagnosis of cSCC using no formal algorithm.

Pooled sensitivity and specificity (2684 lesions; 538 cSCCs) were 57% (95% CI 53% to 61%) and 79% (95% CI 77% to 81%) respectively. In Cooper 2002 none of the 12 BCCs was mistaken for a cSCC, but in Ek 2005, 119 of 1214 included BCCs were diagnosed as cSCCs (accounting for 28% of the false positives in this study).

Dermoscopic image‐based diagnosis

The two studies evaluating dermoscopic images were both conducted in participants with non‐pigmented lesions: Navarrete Dechent 2016, using their own new algorithm for detection of BCC based on the presence of shiny white streaks and blotches (but also reporting accuracy data for detection of cSCC using the algorithm), and Witkowski 2016, using no algorithm. Navarrete Dechent 2016 primarily recruited participants with malignant lesions (90% of lesions), whereas Witkowski 2016 included participants with a wider range of different lesion types that might be observed in clinical practice. The prevalence of cSCC was 23% (Navarrete Dechent 2016) and 5% (Witkowski 2016).

Pooled sensitivity and specificity (717 lesions; 119 cSCCs) were 55% (95% CI 29% to 79%) and 84% (95% CI 32% to 98%) respectively. Both sensitivity and specificity were considerably higher in Witkowski 2016 compared to Navarrete Dechent 2016, and the resulting confidence intervals were therefore extremely wide.

Comparison of dermoscopy versus visual inspection

No formal comparison of visual inspection and dermoscopy is possible for the detection of cSCC, as visual inspection data are from in‐person studies and dermoscopy from image‐based studies.

3. Target condition: Any skin cancer

In this section we present the results for studies of visual inspection for the identification of any skin cancer, according to the approach taken for diagnosis: in‐person or image‐based evaluations. We present summary characteristics of studies in Appendix 8, forest plots of study data in Figure 18 and Figure 19, and results of meta‐analyses in Table 7, Figure 20 and Figure 21.

18.

Forest plot of tests: 27 Any ‐Visual inspection (in‐person), 29 Any ‐VI+Dermoscopy (in‐person).

19.

Forest plot of tests: 28 Any ‐Visual inspection (image‐based), 30 Any‐Dermoscopy alone (image‐based).

6. Comparison of visual inspection and dermoscopy for the detection of any skin cancer.

Test	Datasets	Lesions (cases)	DOR (95% CI)	Specificity at 80% sensitivity	Sensitivity at 80% specificity	Relative DOR (95% CI)	P value (LR)a	P value (Wald)b
In‐person evaluations
Visual inspection	5	3618 (2021)	28.7 (5.0 to 166)	88%	84%	NE	NE	NE
Visual inspection + Dermoscopy	2	277 (85)	126 (9.1 to 1751)	NE	NE
Image‐based evaluations
Visual inspection (clinical images)	2	517 (124)	16.3 (4.4 to 59.9)	79%	78%	1.5 (0.76 to 3.0)	0.50	0.24
Dermoscopic images	6	1526 (847)	24.5 (7.6 to 79.3)	84%	86%

DOR ‐ diagnostic odds ratio; RDOR ‐ relative diagnostic odds ratio; CI ‐ confidence interval; LR ‐ likelihood ratio; NE – not estimated; data not estimated due to extreme differences in results between the two studies of dermoscopy added to visual inspection

^aTests whether there is a difference in test performance between defined groups in terms of either DOR or threshold.
 ^bTests the significance of the difference in DOR between defined groups at a particular SROC curve intercept value.

20.

Comparison of the accuracy of visual inspection with visual inspection plus dermoscopy (VI+Dermoscopy) for detection of any skin cancer (Any). SROC curve estimated only for in‐person visual inspection.

21.

Comparison of the accuracy of image‐based visual inspection with image‐based dermoscopy (Dermoscopy alone) for detection of any skin cancer (Any)

In‐person evaluations

Five studies evaluated the accuracy of in‐person visual inspection for the detection of any skin cancer (Argenziano 2006; Chang 2013; Cooper 2002; Ek 2005; Hacioglu 2013) and two evaluated in‐person dermoscopy (Argenziano 2006; Durdu 2011). Three of these also reported accuracy data separately for BCC alone (Cooper 2002; Durdu 2011; Ek 2005) or for cSCC (Cooper 2002; Ek 2005).

All studies were based in secondary care or specialist referral clinics, apart from Argenziano 2006 which recruited participants from primary care (although only lesions selected for excision by an expert could be included). The prevalence of skin cancer ranged from 20% (Chang 2013) to 68% (Ek 2005). Studies included any lesion type, apart from Durdu 2011 which restricted inclusion to pigmented lesions only. Diagnoses were recorded by GPs (Argenziano 2006), dermatologists or assumed to be dermatologists based on study institution (Chang 2013; Durdu 2011; Hacioglu 2013) or by a clinician with mixed experience (Cooper 2002; Ek 2005). All studies used a histological reference standard.

Visual inspection

Studies either used no algorithm to aid diagnosis, or reported using the ABCD approach to diagnosis (Argenziano 2006). Sensitivities ranged from 57% to 98%; specificities ranged from 13% to 86% (Figure 18). In meta‐analysis the DOR was 28.7 (95% CI 5.0 to 166) (3618 lesions; 2021 skin cancer cases). Estimates of accuracy obtained from the curve suggest that the specificity of visual inspection would be 88% at a fixed threshold of 80% sensitivity, and sensitivity would be 84% at a fixed threshold of 80% specificity (Table 7).

Dermoscopy added to visual inspection

The two studies of in‐person dermoscopy reported data using the three‐point checklist (Argenziano 2006) and the ABCD approach (Durdu 2011) (Figure 18). In Argenziano 2006, GPs' diagnosis had a sensitivity of 85% (95% CI 69% to 94%) and specificity of 26% (95% CI 13% to 43%) for the subgroup of lesions selected for excision by an expert clinician. Of the six malignancies missed by GPs, four were BCCs, one cSCC and one melanoma. Durdu 2011 reported a sensitivity of 98% (95% CI 88% to 100%) and specificity 98% (95% CI 94% to 100%) for their sample of pigmented lesions which could not be diagnosed by a dermatologist with visual inspection alone.

In meta‐analysis the DOR was 126 (95% CI 9.1 to 1751) (277 lesions; 85 skin cancer cases) (Table 7). We could not obtain estimates of accuracy from the SROC curve due to extreme differences in results between the two studies (evidenced by the very wide range in confidence intervals around the DOR).

Comparison of in‐person dermoscopy versus visual inspection alone

No formal comparison of visual inspection and dermoscopy added to visual inspection was possible, due to the observed heterogeneity in results for the two dermoscopy studies (Figure 20).

Image‐based evaluations

Six studies reported data for image‐based diagnosis for the detection of any skin cancer. Two evaluated the accuracy of image‐based visual inspection (Carli 2002b; Rosendahl 2011) and all six evaluated diagnosis using dermoscopic images (Carli 2002b; Hacioglu 2013; Menzies 2000; Navarrete Dechent 2016; Rosendahl 2011; Witkowski 2016). Five of these also reported accuracy data separately for BCC alone (Carli 2002b; Menzies 2000; Navarrete Dechent 2016; Rosendahl 2011; Witkowski 2016) or for cSCC (Navarrete Dechent 2016; Witkowski 2016).

Two studies were conducted in a limited prior testing setting, recruiting participants from primary care (Rosendahl 2011) or from a private dermatology practice (Navarrete Dechent 2016). Of the remaining four, one was considered to have been conducted in participants with equivocal lesions (Witkowski 2016). Four of the six studies primarily aimed to examine accuracy for the detection of BCC (Menzies 2000; Navarrete Dechent 2016; Witkowski 2016) or ‘non‐melanoma’ skin cancer (Hacioglu 2013), with the remaining two also providing data for the diagnosis of melanoma (Carli 2002b; Rosendahl 2011). Three studies included only pigmented lesions (Carli 2002b; Menzies 2000; Rosendahl 2011); two included only non‐pigmented lesions (Navarrete Dechent 2016; Witkowski 2016) and one described lesions as ‘suspicious for malignancy’ (Hacioglu 2013). All studies apart from Hacioglu 2013 reported including invasive melanoma or melanoma in situ as disease‐negative and four also included cSCC (all apart from Carli 2002b and Menzies 2000) in the disease‐negative group. Diagnosis was recorded by dermatologists or by dermatology trainees (Navarrete Dechent 2016). All studies used a histological reference standard.

Visual inspection of images

The two included studies used no algorithm to aid diagnosis and both included pigmented lesions only (Carli 2002b; Rosendahl 2011). Sensitivities were 80% (95% CI 56% to 94%) and 76% (95% CI 67% to 84%) and specificities 74% (95% CI 56% to 87%) and 85% (95% CI 81% to 88%) in Carli 2002b and Rosendahl 2011, respectively (Figure 19).

In meta‐analysis the DOR was 16.3 (95%CI 4.4 to 59.9) (517 lesions; 124 skin cancer cases). Estimates of accuracy obtained from the curve suggest that the specificity of visual inspection would be 79% at a fixed threshold of 80% sensitivity, and sensitivity would be 78% at a fixed threshold of 80% specificity (Table 7).

Dermoscopic image‐based diagnosis

The six studies used no algorithm to assist diagnosis in three (Carli 2002b; Hacioglu 2013; Witkowski 2016), pattern analysis in one (Rosendahl 2011), and new algorithms for detection of BCC in two (Menzies 2000; Navarrete Dechent 2016).

Sensitivity ranged from 50% to 95% and specificity from 63% to 92% (Figure 19). We pooled results across algorithms and thresholds as a summary ROC curve (1526 lesions; 847 BCCs; Figure 21). Estimates of accuracy obtained from the curve suggest that the specificity of dermoscopy would be 84% at a fixed threshold of 80% sensitivity, and sensitivity would be 86% at a fixed threshold of 80% specificity (Table 7).

Comparison of diagnosis using dermoscopic images versus visual inspection of images

We compared accuracy using data from both visual inspection studies and all dermoscopy studies (Figure 21). The accuracy of diagnosis using dermoscopic images was non‐significantly higher than that based on clinical photographs (Table 7), with an RDOR of 1.5 (95% CI 0.76 to 3.0, LR test P = 0.50). Differences were marginal in sensitivity and specificity between tests in the two studies providing paired data.

Discussion

Summary of main results

We have evaluated visual inspection and the addition of dermoscopy for the detection of keratinocyte skin cancers in a range of study populations, on both an in‐person basis and using clinical photographs or dermoscopic images. Although a small number of published algorithms to assist diagnosis are available, most of the data relate to diagnosis without the use of an algorithm and relate to the detection of BCC rather than cSCC. Studies either did not recruit sufficient numbers of participants with cSCC to meet our inclusion criteria (i.e. five or more confirmed cSCCs) or did not present accuracy data for cSCC. For the detection of BCC, sensitivities and specificities were highly heterogeneous, especially for visual inspection. There was some suggestion that this heterogeneity was related to the case‐mix of included lesions, with studies in non‐pigmented lesions or those with a high index of suspicion of BCC having lower and more variable specificity, in comparison to those including pigmented lesions or lesions suspicious for any skin cancer. Studies were generally at high or unclear risk of bias across most domains assessed, particularly for image‐based interpretations, and of high or unclear concern about the applicability of the evidence, limiting the strength of conclusions that we can draw.

Table 1 presents key results for the primary target conditions of BCC and cSCC, and translates summary estimates to a hypothetical cohort of 1000 lesions. Due to the observed heterogeneity between studies, the results presented are points estimated from summary ROC curves rather than average sensitivity and specificity operating points. We present these for illustrative purposes, and they should not be quoted as the actual performance of visual inspection or dermoscopy. Due to the high risk of bias, concerns about applicability, the high level of unexplained heterogeneity and the necessity of the SROC curve analytical approach, we cannot confidently estimate the actual false‐negative and false‐positive rates for either test. Nevertheless, on average, the addition of dermoscopy to in‐person visual inspection of a lesion increases sensitivity and specificity for the diagnosis of BCC.

Sensitivity: At a fixed specificity of 80%, the use of dermoscopy increased the sensitivity of in‐person visual inspection by 14%, from 79% to 93%. Assuming BCC prevalence of 10%, 17% and 53% in a cohort of 1000 lesions, a test sensitivity of 93% would reduce the number of BCCs missed in comparison to using visual inspection alone by 14, 24 and 74 (resulting in 7, 12 and 37 BCCs missed). A test specificity of 80% (for both visual inspection and visual inspection plus dermoscopy) would result in 180, 166 and 94 false‐positive test results, i.e. lesions considered to be BCC which might then undergo unnecessary biopsy or treatment, in this case of benign lesions mistaken for BCCs, or inappropriate management, in the case of melanomas or cSCCs mistaken for BCCs.

Specificity: At a fixed sensitivity of 80%, the use of dermoscopy increased the specificity of in‐person visual inspection by 22%, from 77% to 99%. Applying these results to a cohort of 1000 lesions at the same three prevalences of disease, both tests would miss 20, 34 or 106 BCCs with the addition of dermoscopy reducing false positives by 198, 183 and 103 per 1000 from 207, 191 and 108 lesions mistaken as BCCs using visual inspection alone.

We found a similar pattern for image‐based comparisons of visual inspection and dermoscopy, although the differences in sensitivity and specificity were smaller (Table 1). It is notable that for the in‐person evaluations, up to a third of observed false‐positive results were melanomas mistaken for BCCs (33% (5/15) of false positives for visual inspection and 16% (3/19) for dermoscopy). This is of particular concern if non‐surgical treatment without biopsy is under consideration for lesions clinically presumed to be BCCs. In contrast to our review of dermoscopy versus visual inspection alone for the diagnosis of melanoma (Dinnes 2018b), there were no statistically significant differences between in‐person and image‐based evaluations for the diagnosis of BCC. Insufficient data were available to consider the effect of where in the clinical pathway the study was positioned, the use of formally‐developed algorithms to assist diagnosis of BCC, or the effect of observer experience on accuracy. In Dinnes 2018b, however, we were able to demonstrate that observer expertise and training in dermoscopy does improve accuracy for the diagnosis of melanoma.

Data for the detection of cSCC were limited, but suggest pooled sensitivity of 57% (95% CI 53% to 61%) and specificity of 79% (95% CI 77% to 81%) for visual inspection (in‐person), and sensitivity of 55% (95% CI 29% to 79%) and specificity of 84% (95% CI 32% to 98%) for dermoscopy (image‐based).

Strengths and weaknesses of the review

The strengths of this review include an in‐depth and comprehensive electronic literature search, systematic review methods including double extraction of papers by both clinicians and methodologists, and contact with authors to allow study inclusion or clarify data. We adopted a clear analysis structure focusing on estimating incremental gains in accuracy. We undertook a detailed and replicable analysis of methodologic quality.

The main concerns for the review are a result of relatively small numbers of studies, variation in the spectrum of included lesions and poor reporting of primary studies, hindering the assessment of study quality and limiting the conclusions that we can draw from the data. Our review of visual inspection for the diagnosis of melanoma identified a general trade‐off between sensitivity and specificity along the clinical pathway, with higher sensitivity and lower specificity in limited prior testing studies compared to those in referred populations (Dinnes 2018a). The lack of data from limited prior testing populations in this review and the lack of detailed information on the prior testing of participants included in referred populations meant that we could detect no clear patterns in sensitivity or specificity. We found some evidence of more variable accuracy, especially in terms of specificity, in studies with a higher prevalence of BCC or those conducted in populations of non‐pigmented lesions, or both. Many of these studies, however, also used new algorithms for detection of BCC rather than relying on the clinician’s diagnosis. The quality of dermatoscope and the resultant images may vary greatly, and there are further variations such as whether they are used with oil immersion or other light sources. None of our included studies provided enough detail to evaluate such effects on test performance. All of these factors together make it difficult to fully determine the cause of the observed heterogeneity.

Given these limitations, our results should be considered as exploratory rather than conclusive. We have, however, identified a clear suggestion of benefit from dermoscopy for the diagnosis of BCC, which requires further investigation. This is the first systematic review, to our knowledge, to have examined this critical question of dermoscopy use for the diagnosis of BCC, particularly given the increasing availability of newer imaging tests such as optical coherence tomography (OCT) or radiocontrast medium (RCM) which purport to assist in the diagnosis of BCC (Dinnes 2018c; Ferrante di Ruffano 2018b).

Applicability of findings to the review question

Our findings are particularly relevant to the use of visual inspection and dermoscopy for the diagnosis of BCC in referral settings. Limited data were available to consider accuracy in primary care or according to observer experience. We cannot be clear as to the likely error rates of visual inspection or dermoscopy in any particular lesion population, due to varying definitions and lack of clarity about the clinical pathway and any prior testing undergone.

Authors' conclusions

Implications for practice.

Dermoscopy may be a valuable tool to support visual inspection of a suspicious skin lesion for the diagnosis of BCC. The evidence primarily comes from secondary‐care (referred) populations and populations with pigmented lesions or mixed lesion types. There is no clear evidence supporting the use of formal algorithms to assist diagnosis.

Implications for research.

Surveys and qualitative research documenting dermoscopy use in a primary‐care setting in different countries and healthcare systems would help to better understand the purpose for which dermoscopy is being used. It may be that it is mainly used for triaging suspected melanoma (or high‐risk keratinocyte skin cancer) for urgent secondary referral; alternatively, dermoscopy may be used to differentiate between types of skin cancer (melanoma, BCC or cSCC) with a view to initial treatment of some lesions in primary care and referral of others to a secondary‐care setting. Prospective studies evaluating the use of dermoscopy in primary care for all forms of suspected skin cancer could better define where the gains might reside in terms of triage, and help to quantify diagnostic test accuracy. The need not to miss potentially lethal cancers such as melanomas must be balanced against the avoidance of unnecessary referral and biopsy resulting in raised morbidity and cost.

Further prospective evaluation of dermoscopy added to visual inspection in populations with a high clinical suspicion of BCC in both a primary‐care and secondary‐care setting by users with defined expertise is also likely to be warranted. Such evaluations should be conducted on an in‐person basis with prospective recruitment of consecutive series of participants and with systematic follow‐up of non‐excised lesions to avoid over‐reliance on a histological reference standard that can only provide information on excised cases. A clear identification of the level of training and experience required to achieve good results is required. It is unclear whether further research is warranted on the potential additional value of dermoscopy to visual inspection for lesions that are suspected to be cSCC in a primary‐ and secondary‐care setting, unless they are conducted in specific populations such as people with immunosuppression or who have received organ transplants in whom cSCC is a common problem.

Given the mixed results to date, it is unclear whether further research is warranted into the added value of dermoscopy algorithms to assist diagnosis above pattern recognition of characteristic morphological features. Any future research study needs to be clear about the diagnostic pathway followed by study participants prior to study enrolment, and should conform to the updated Standards for Reporting of Diagnostic Accuracy (STARD) guideline (Bossuyt 2015).

What's new

Date	Event	Description
19 December 2018	Amended	Affiliations, Disclaimer and Sources of support updated

Appendices

Appendix 1. Current content and structure of the Programme Grant

	LIST OF REVIEWS	Number of studies
	Diagnosis of melanoma
1	Visual inspection	49
2	Dermoscopy +/‐ visual inspection	104
3	Teledermatology	22
4	Smartphone applications	2
5a	Computer‐assisted diagnosis – dermoscopy‐based techniques	42
5b	Computer‐assisted diagnosis – spectroscopy‐based techniques	Review amalgamated into 5a
6	Reflectance confocal microscopy	18
7	High‐frequency ultrasound	5
	Diagnosis of keratinocyte skin cancer (BCC and cSCC)
8	Visual inspection +/‐ Dermoscopy	24
5c	Computer‐assisted diagnosis – dermoscopy‐based techniques	Review amalgamated into 5a
5d	Computer‐assisted diagnosis – spectroscopy‐based techniques	Review amalgamated into 5a
9	Optical coherence tomography	5
10	Reflectance confocal microscopy	10
11	Exfoliative cytology	9
	Staging of melanoma
12	Imaging tests (ultrasound, CT, MRI, PET‐CT)	38
13	Sentinel lymph node biopsy	160
	Staging of cSCC
	Imaging tests review	Review dropped; only one study identified
13	Sentinel lymph node biopsy	Review amalgamated into 13 above (n = 15 studies)

Appendix 2. Glossary of terms

Term	Definition
Atypical intraepidermal melanocytic variant	Unusual area of darker pigmentation contained within the epidermis that may progress to an invasive melanoma; includes melanoma in situ and lentigo maligna
Atypical naevi	Unusual looking but noncancerous mole or area of darker pigmentation of the skin
BRAF V600 mutation	BRAF is a human gene that makes a protein called B‐Raf which is involved in the control of cell growth. BRAF mutations (damaged DNA) occur in around 40% of melanomas, which can then be treated with particular drugs.
BRAF inhibitors	Therapeutic agents which inhibit the serine‐threonine protein kinase BRAF mutated metastatic melanoma.
Breslow thickness	A scale for measuring the thickness of melanomas by the pathologist using a microscope, measured in mm from the top layer of skin to the bottom of the tumour.
Congenital naevi	A type of mole found on infants at birth
Dermoscopy	Whereby a handheld microscope is used to allow more detailed, magnified, examination of the skin compared to examination by the naked eye alone
False negative	An individual who is truly positive for a disease, but whom a diagnostic test classifies them as disease‐free.
False positive	An individual who is truly disease‐free, but whom a diagnostic test classifies them as having the disease.
Histopathology/Histology	The study of tissue, usually obtained by biopsy or excision, for example under a microscope.
Incidence	The number of new cases of a disease in a given time period.
Index test	A diagnostic test under evaluation in a primary study
Lentigo maligna	Unusual area of darker pigmentation contained within the epidermis which includes malignant cells but with no invasive growth. May progress to an invasive melanoma
Lymph node	Lymph nodes filter the lymphatic fluid (clear fluid containing white blood cells) that travels around the body to help fight disease; they are located throughout the body often in clusters (nodal basins).
Melanocytic naevus	An area of skin with darker pigmentation (or melanocytes) also referred to as ‘moles’
Meta‐analysis	A form of statistical analysis used to synthesise results from a collection of individual studies.
Metastases/metastatic disease	Spread of cancer away from the primary site to somewhere else through the bloodstream or the lymphatic system.
Micrometastases	Micrometastases are metastases so small that they can only be seen under a microscope.
Mitotic rate	Microscopic evaluation of number of cells actively dividing in a tumour.
Morbidity	Detrimental effects on health.
Mortality	Either (1) the condition of being subject to death; or (2) the death rate, which reflects the number of deaths per unit of population in relation to any specific region, age group, disease, treatment or other classification, usually expressed as deaths per 100, 1000, 10,000 or 100,000 people.
Multidisciplinary team	A team with members from different healthcare professions and specialties (e.g. urology, oncology, pathology, radiology, and nursing). Cancer care in the National Health Service (NHS) uses this system to ensure that all relevant health professionals are engaged to discuss the best possible care for that patient.
Prevalence	The proportion of a population found to have a condition.
Prognostic factors/indicators	Specific characteristics of a cancer or the person who has it which might affect the patient’s prognosis.
Receiver operating characteristic (ROC) plot	A plot of the sensitivity and 1 minus the specificity of a test at the different possible thresholds for test positivity; represents the diagnostic capability of a test with a range of binary test results
Receiver operating characteristic (ROC) analysis	The analysis of a ROC plot of a test to select an optimal threshold for test positivity
Recurrence	Recurrence is when new cancer cells are detected following treatment. This can occur either at the site of the original tumour or at other sites in the body.
Reference Standard	A test or combination of tests used to establish the final or ‘true’ diagnosis of a patient in an evaluation of a diagnostic test
Reflectance confocal microscopy (RCM)	A microscopic technique using infrared light (either in a handheld device or a static unit) that can create images of the deeper layers of the skin
Sensitivity	In this context the term is used to mean the proportion of individuals with a disease who have that disease correctly identified by the study test
Specificity	The proportion of individuals without the disease of interest (in this case with benign skin lesions) who have that absence of disease correctly identified by the study test
Staging	Clinical description of the size and spread of a patient’s tumour, fitting into internationally agreed categories.
Subclinical (disease)	Disease that is usually asymptomatic and not easily observable, e.g. by clinical or physical examination.
Systemic treatment	Treatment, usually given by mouth or by injection, that reaches and affects cancer cells throughout the body rather than targeting one specific area.

Appendix 3. Proposed sources of heterogeneity

i. Population characteristics

• general versus higher‐risk populations

• patient population: Primary/secondary/specialist unit

• lesion suspicion: general suspicion/atypical/equivocal/NR

• lesion type: any pigmented; melanocytic

• inclusion of multiple lesions per participant

• ethnicity

ii. Index test characteristics

• the nature of and definition of criteria for test positivity

• observer experience with the index test

• approaches to lesion preparation (e.g. the use of oil or antiseptic gel for dermoscopy)

iii. Reference standard characteristics

• reference standard used

• whether histology‐reporting meets pathology‐reporting guidelines

• use of excisional versus diagnostic biopsy

• whether two independent dermatopathologists reviewed histological diagnosis

iv. Study quality

• consecutive or random sample of participants recruited

• index test interpreted blinded to the reference standard result

• index test interpreted blinded to the result of any other index test

• presence of partial or differential verification bias (whereby only a sample of those subject to the index test are verified by the reference test or by the same reference test with selection dependent on the index test result)

• use of an adequate reference standard

• overall risk of bias

Appendix 4. Final search strategies

Melanoma search strategies to August 2016

Database: Ovid MEDLINE(R) 1946 to August week 3 2016

Search strategy:

1 exp melanoma/

2 exp skin cancer/

3 exp basal cell carcinoma/

4 basalioma$1.ti,ab.

5 ((basal cell or skin) adj2 (cancer$1 or carcinoma$1 or mass or masses or tumour$1 or tumor$1 or neoplasm$1 or adenoma$1 or epithelioma$1 or lesion$1 or malignan$ or nodule$1)).ti,ab.

6 (pigmented adj2 (lesion$1 or mole$ or nevus or nevi or naevus or naevi or skin)).ti,ab.

7 (melanom$1 or nonmelanoma$1 or non‐melanoma$1 or melanocyt$ or non‐melanocyt$ or nonmelanocyt$ or keratinocyt$).ti,ab.

8 nmsc.ti,ab.

9 (squamous cell adj2 (cancer$1 or carcinoma$1 or mass or masses or tumor$1 or tumour$1 or neoplasm$1 or adenoma$1 or epithelioma$1 or epithelial or lesion$1 or malignan$ or nodule$1) adj2 (skin or epiderm$ or cutaneous)).ti,ab.

10 (BCC or CSCC or NMSC).ti,ab.

11 keratinocy$.ti,ab.

12 Keratinocytes/

13 or/1‐12

14 dermoscop$.ti,ab.

15 dermatoscop$.ti,ab.

16 photomicrograph$.ti,ab.

17 exp epiluminescence microscopy/

18 (epiluminescence adj2 microscop$).ti,ab.

19 (confocal adj2 microscop$).ti,ab.

20 (incident light adj2 microscop$).ti,ab.

21 (surface adj2 microscop$).ti,ab.

22 (visual adj (inspect$ or examin$)).ti,ab.

23 ((clinical or physical) adj examin$).ti,ab.

24 3 point.ti,ab.

25 three point.ti,ab.

26 pattern analys$.ti,ab.

27 ABCD$.ti,ab.

28 menzies.ti,ab.

29 7 point.ti,ab.

30 seven point.ti,ab.

31 (digital adj2 (dermoscop$ or dermatoscop$)).ti,ab.

32 artificial intelligence.ti,ab.

33 AI.ti,ab.

34 computer assisted.ti,ab.

35 computer aided.ti,ab.

36 neural network$.ti,ab.

37 exp diagnosis, computer‐assisted/

38 MoleMax.ti,ab.

39 image process$.ti,ab.

40 automatic classif$.ti,ab.

41 image analysis.ti,ab.

42 SIAscop$.ti,ab.

43 Aura.ti,ab.

44 (optical adj2 scan$).ti,ab.

45 MelaFind.ti,ab.

46 SIMSYS.ti,ab.

47 MoleMate.ti,ab.

48 SolarScan.ti,ab.

49 VivaScope.ti,ab.

50 (high adj3 ultraso$).ti,ab.

51 (canine adj2 detect$).ti,ab.

52 ((mobile or cell or cellular or smart) adj ((phone$1 adj2 app$1) or application$1)).ti,ab.

53 smartphone$.ti,ab.

54 (DermoScan or SkinVision or DermLink or SpotCheck).ti,ab.

55 Mole Detective.ti,ab.

56 Spot Check.ti,ab.

57 (mole$1 adj2 map$).ti,ab.

58 (total adj2 body).ti,ab.

59 exfoliative cytolog$.ti,ab.

60 digital analys$.ti,ab.

61 (image$1 adj3 software).ti,ab.

62 (teledermatolog$ or tele‐dermatolog$ or telederm or tele‐derm or teledermoscop$ or tele‐dermoscop$ or teledermatoscop$ or tele‐dermatoscop$).ti,ab.

63 (optical coherence adj (technolog$ or tomog$)).ti,ab.

64 (computer adj2 diagnos$).ti,ab.

65 exp sentinel lymph node biopsy/

66 (sentinel adj2 node).ti,ab.

67 nevisense.mp. or HFUS.ti,ab.

68 electrical impedance spectroscopy.ti,ab.

69 history taking.ti,ab.

70 patient history.ti,ab.

71 (naked eye adj (exam$ or assess$)).ti,ab.

72 (skin adj exam$).ti,ab.

73 physical examination/

74 ugly duckling.mp. or UD.ti,ab.

75 ((physician$ or clinical or physical) adj (exam$ or triage or recog$)).ti,ab.

76 ABCDE.mp. or VOC.ti,ab.

77 clinical accuracy.ti,ab.

78 Family Practice/ or Physicians, Family/ or clinical competence/

79 (confocal adj2 microscop$).ti,ab.

80 diagnostic algorithm$1.ti,ab.

81 checklist$.ti,ab.

82 virtual imag$1.ti,ab.

83 volatile organic compound$1.ti,ab.

84 dog$1.ti,ab.

85 gene expression analy$.ti,ab.

86 reflex transmission imag$.ti,ab.

87 thermal imaging.ti,ab.

88 elastography.ti,ab.

89 or/14‐88

90 (CT or PET).ti,ab.

91 PET‐CT.ti,ab.

92 (FDG or F18 or Fluorodeoxyglucose or radiopharmaceutical$).ti,ab.

93 exp Deoxyglucose/

94 deoxy‐glucose.ti,ab.

95 deoxyglucose.ti,ab.

96 CATSCAN.ti,ab.

97 exp Tomography, Emission‐Computed/

98 exp Tomography, X‐ray computed/

99 positron emission tomograph$.ti,ab.

100 exp magnetic resonance imaging/

101 (MRI or fMRI or NMRI or scintigraph$).ti,ab.

102 exp echography/

103 Doppler echography.ti,ab.

104 sonograph$.ti,ab.

105 ultraso$.ti,ab.

106 doppler.ti,ab.

107 magnetic resonance imag$.ti,ab.

108 or/90‐107

109 (stage$ or staging or metasta$ or recurrence or sensitivity or specificity or false negative$ or thickness$).ti,ab.

110 "Sensitivity and Specificity"/

111 exp cancer staging/

112 or/109‐111

113 108 and 112

114 89 or 113

115 13 and 114

Database: Ovid MEDLINE(R) In‐Process & Other Non‐Indexed Citations 29 August 2016

Search strategy:

1 basalioma$1.ti,ab.

2 ((basal cell or skin) adj2 (cancer$1 or carcinoma$1 or mass or masses or tumour$1 or tumor$1 or neoplasm$1 or adenoma$1 or epithelioma$1 or lesion$1 or malignan$ or nodule$1)).ti,ab.

3 (pigmented adj2 (lesion$1 or mole$ or nevus or nevi or naevus or naevi or skin)).ti,ab.

4 (melanom$1 or nonmelanoma$1 or non‐melanoma$1 or melanocyt$ or non‐melanocyt$ or nonmelanocyt$ or keratinocyt$).ti,ab.

5 nmsc.ti,ab.

6 (squamous cell adj2 (cancer$1 or carcinoma$1 or mass or masses or tumor$1 or tumour$1 or neoplasm$1 or adenoma$1 or epithelioma$1 or epithelial or lesion$1 or malignan$ or nodule$1) adj2 (skin or epiderm$ or cutaneous)).ti,ab.

7 (BCC or CSCC or NMSC).ti,ab.

8 keratinocy$.ti,ab.

9 or/1‐8

10 dermoscop$.ti,ab.

11 dermatoscop$.ti,ab.

12 photomicrograph$.ti,ab.

13 (epiluminescence adj2 microscop$).ti,ab.

14 (confocal adj2 microscop$).ti,ab.

15 (incident light adj2 microscop$).ti,ab.

16 (surface adj2 microscop$).ti,ab.

17 (visual adj (inspect$ or examin$)).ti,ab.

18 ((clinical or physical) adj examin$).ti,ab.

19 3 point.ti,ab.

20 three point.ti,ab.

21 pattern analys$.ti,ab.

22 ABCD$.ti,ab.

23 menzies.ti,ab.

24 7 point.ti,ab.

25 seven point.ti,ab.

26 (digital adj2 (dermoscop$ or dermatoscop$)).ti,ab.

27 artificial intelligence.ti,ab.

28 AI.ti,ab.

29 computer assisted.ti,ab.

30 computer aided.ti,ab.

31 neural network$.ti,ab.

32 MoleMax.ti,ab.

33 image process$.ti,ab.

34 automatic classif$.ti,ab.

35 image analysis.ti,ab.

36 SIAscop$.ti,ab.

37 Aura.ti,ab.

38 (optical adj2 scan$).ti,ab.

39 MelaFind.ti,ab.

40 SIMSYS.ti,ab.

41 MoleMate.ti,ab.

42 SolarScan.ti,ab.

43 VivaScope.ti,ab.

44 (high adj3 ultraso$).ti,ab.

45 (canine adj2 detect$).ti,ab.

46 ((mobile or cell or cellular or smart) adj ((phone$1 adj2 app$1) or application$1)).ti,ab.

47 smartphone$.ti,ab.

48 (DermoScan or SkinVision or DermLink or SpotCheck).ti,ab.

49 Mole Detective.ti,ab.

50 Spot Check.ti,ab.

51 (mole$1 adj2 map$).ti,ab.

52 (total adj2 body).ti,ab.

53 exfoliative cytolog$.ti,ab.

54 digital analys$.ti,ab.

55 (image$1 adj3 software).ti,ab.

56 (teledermatolog$ or tele‐dermatolog$ or telederm or tele‐derm or teledermoscop$ or tele‐dermoscop$ or teledermatoscop$ or tele‐dermatoscop$).ti,ab.

57 (optical coherence adj (technolog$ or tomog$)).ti,ab.

58 (computer adj2 diagnos$).ti,ab.

59 (sentinel adj2 node).ti,ab.

60 nevisense.mp. or HFUS.ti,ab.

61 electrical impedance spectroscopy.ti,ab.

62 history taking.ti,ab.

63 patient history.ti,ab.

64 (naked eye adj (exam$ or assess$)).ti,ab.

65 (skin adj exam$).ti,ab.

66 ugly duckling.mp. or UD.ti,ab.

67 ((physician$ or clinical or physical) adj (exam$ or triage or recog$)).ti,ab.

68 ABCDE.mp. or VOC.ti,ab.

69 clinical accuracy.ti,ab.

70 (Family adj (Practice or Physicians)).ti,ab.

71 (confocal adj2 microscop$).ti,ab.

72 clinical competence.ti,ab.

73 diagnostic algorithm$1.ti,ab.

74 checklist$.ti,ab.

75 virtual imag$1.ti,ab.

76 volatile organic compound$1.ti,ab.

77 dog$1.ti,ab.

78 gene expression analy$.ti,ab.

79 reflex transmission imag$.ti,ab.

80 thermal imaging.ti,ab.

81 elastography.ti,ab.

82 or/10‐81

83 (CT or PET).ti,ab.

84 PET‐CT.ti,ab.

85 (FDG or F18 or Fluorodeoxyglucose or radiopharmaceutical$).ti,ab.

86 deoxy‐glucose.ti,ab.

87 deoxyglucose.ti,ab.

88 CATSCAN.ti,ab.

89 positron emission tomograph$.ti,ab.

90 (MRI or fMRI or NMRI or scintigraph$).ti,ab.

91 Doppler echography.ti,ab.

92 sonograph$.ti,ab.

93 ultraso$.ti,ab.

94 doppler.ti,ab.

95 magnetic resonance imag$.ti,ab.

96 or/83‐95

97 (stage$ or staging or metasta$ or recurrence or sensitivity or specificity or false negative$ or thickness$).ti,ab.

98 96 and 97

99 82 or 98

100 9 and 99

Database: Embase 1974 to 29 August 2016

Search strategy:

1 *melanoma/

2 *skin cancer/

3 *basal cell carcinoma/

4 basalioma$.ti,ab.

5 ((basal cell or skin) adj2 (cancer$1 or carcinoma$1 or mass or masses or tumour$1 or tumor$1 or neoplasm$ or adenoma$ or epithelioma$ or lesion$ or malignan$ or nodule$)).ti,ab.

6 (pigmented adj2 (lesion$1 or mole$ or nevus or nevi or naevus or naevi or skin)).ti,ab.

7 (melanom$1 or nonmelanoma$1 or non‐melanoma$1 or melanocyt$ or non‐melanocyt$ or nonmelanocyt$ or keratinocyt$).ti,ab.

8 nmsc.ti,ab.

9 (squamous cell adj2 (cancer$1 or carcinoma$1 or mass or tumor$1 or tumour$1 or neoplasm$1 or adenoma$1 or epithelioma$1 or epithelial or lesion$1 or malignan$ or nodule$1) adj2 (skin or epiderm$ or cutaneous)).ti,ab.

10 (BCC or cscc).mp. or NMSC.ti,ab.

11 keratinocyte.ti,ab.

12 keratinocy$.ti,ab.

13 or/1‐12

14 dermoscop$.ti,ab.

15 dermatoscop$.ti,ab.

16 photomicrograph$.ti,ab.

17 *epiluminescence microscopy/

18 (epiluminescence adj2 microscop$).ti,ab.

19 (confocal adj2 microscop$).ti,ab.

20 (incident light adj2 microscop$).ti,ab.

21 (surface adj2 microscop$).ti,ab.

22 (visual adj (inspect$ or examin$)).ti,ab.

23 ((clinical or physical) adj examin$).ti,ab.

24 3 point.ti,ab.

25 three point.ti,ab.

26 pattern analys$.ti,ab.

27 ABCD$.ti,ab.

28 menzies.ti,ab.

29 7 point.ti,ab.

30 seven point.ti,ab.

31 (digital adj2 (dermoscop$ or dermatoscop$)).ti,ab.

32 artificial intelligence.ti,ab.

33 AI.ti,ab.

34 computer assisted.ti,ab.

35 computer aided.ti,ab.

36 neural network$.ti,ab.

37 MoleMax.ti,ab.

38 exp diagnosis, computer‐assisted/

39 image process$.ti,ab.

40 automatic classif$.ti,ab.

41 image analysis.ti,ab.

42 SIAscop$.ti,ab.

43 (optical adj2 scan$).ti,ab.

44 Aura.ti,ab.

45 MelaFind.ti,ab.

46 SIMSYS.ti,ab.

47 MoleMate.ti,ab.

48 SolarScan.ti,ab.

49 VivaScope.ti,ab.

50 confocal microscop$.ti,ab.

51 (high adj3 ultraso$).ti,ab.

52 (canine adj2 detect$).ti,ab.

53 ((mobile or cell$ or cellular or smart) adj ((phone$1 adj2 app$1) or application$1)).ti,ab.

54 smartphone$.ti,ab.

55 (DermoScan or SkinVision or DermLink or SpotCheck).ti,ab.

56 Spot Check.ti,ab.

57 Mole Detective.ti,ab.

58 (mole$1 adj2 map$).ti,ab.

59 (total adj2 body).ti,ab.

60 exfoliative cytolog$.ti,ab.

61 digital analys$.ti,ab.

62 (image$1 adj3 software).ti,ab.

63 (optical coherence adj (technolog$ or tomog$)).ti,ab.

64 (teledermatolog$ or tele‐dermatolog$ or telederm or tele‐derm or teledermoscop$ or tele‐dermoscop$ or teledermatoscop$).mp. or tele‐dermatoscop$.ti,ab.

65 (computer adj2 diagnos$).ti,ab.

66 *sentinel lymph node biopsy/

67 (sentinel adj2 node).ti,ab.

68 nevisense.ti,ab.

69 HFUS.ti,ab.

70 electrical impedance spectroscopy.ti,ab.

71 history taking.ti,ab.

72 patient history.ti,ab.

73 (naked eye adj (exam$ or assess$)).ti,ab.

74 (skin adj exam$).ti,ab.

75 *physical examination/

76 ugly duckling.ti,ab.

77 UD sign$.ti,ab.

78 ((physician$ or clinical or physical) adj (exam$ or recog$ or triage)).ti,ab.

79 ABCDE.ti,ab.

80 clinical accuracy.ti,ab.

81 *general practice/

82 (confocal adj2 microscop$).ti,ab.

83 clinical competence/

84 diagnostic algorithm$.ti,ab.

85 checklist$1.ti,ab.

86 virtual image$1.ti,ab.

87 volatile organic compound$1.ti,ab.

88 VOC.ti,ab.

89 dog$1.ti,ab.

90 gene expression analys$.ti,ab.

91 reflex transmission imaging.ti,ab.

92 thermal imaging.ti,ab.

93 elastography.ti,ab.

94 dog$1.ti,ab.

95 gene expression analys$.ti,ab.

96 reflex transmission imaging.ti,ab.

97 thermal imaging.ti,ab.

98 elastography.ti,ab.

99 or/14‐93

100 PET‐CT.ti,ab.

101 (CT or PET).ti,ab.

102 (FDG or F18 or Fluorodeoxyglucose or radiopharmaceutical$).ti,ab.

103 exp Deoxyglucose/

104 CATSCAN.ti,ab.

105 deoxyglucose.ti,ab.

106 deoxy‐glucose.ti,ab.

107 *positron emission tomography/

108 *computer assisted tomography/

109 positron emission tomograph$.ti,ab.

110 *nuclear magnetic resonance imaging/

111 (MRI or fMRI or NMRI or scintigraph$).ti,ab.

112 *echography/

113 Doppler.ti,ab.

114 sonograph$.ti,ab.

115 ultraso$.ti,ab.

116 magnetic resonance imag$.ti,ab.

117 or/100‐116

118 (stage$ or staging or metasta$ or recurrence or sensitivity or specificity or false negative$ or thickness$).ti,ab.

119 "Sensitivity and Specificity"/

120 *cancer staging/

121 or/118‐120

122 117 and 121

123 99 or 122

124 13 and 123

Database: Cochrane Library (Wiley) 2016 searched 30 August 2016 CDSR Issue 8 of 12 2016 CENTRAL Issue 7 of 12 2016 HTA Issue 3 of 4 July 2016 DARE Issue 3 of 4 2015

Search strategy:

#1 melanoma* or nonmelanoma* or non‐melanoma* or melanocyt* or non‐melanocyt* or nonmelanocyt* or keratinocyte*

#2 MeSH descriptor: [Melanoma] explode all trees

#3 "skin cancer*"

#4 MeSH descriptor: [Skin Neoplasms] explode all trees

#5 skin near/2 (cancer* or carcinoma* or mass or masses or tumour* or tumor* or neoplasm* or adenoma* or epithelioma* or lesion* or malignan* or nodule*)

#6 nmsc

#7 "squamous cell" near/2 (cancer* or carcinoma* or mass or masses or tumour* or tumor* or neoplasm* or adenoma* or epithelioma* or lesion* or malignan* or nodule*) near/2 (skin or epiderm* or cutaneous)

#8 "basal cell" near/2 (cancer* or carcinoma* or mass or masses or tumour* or tumor* or neoplasm* or adenoma* or epithelioma* or lesion* or malignan* or nodule*)

#9 pigmented near/2 (lesion* or nevus or mole* or naevi or naevus or nevi or skin)

#10 #1 or #2 or #3 or #4 or #5 or #6 or #7 or #8 or #9

#11 dermoscop*

#12 dermatoscop*

#13 Photomicrograph*

#14 MeSH descriptor: [Dermoscopy] explode all trees

#15 confocal near/2 microscop*

#16 epiluminescence near/2 microscop*

#17 incident next light near/2 microscop*

#18 surface near/2 microscop*

#19 "visual inspect*"

#20 "visual exam*"

#21 (clinical or physical) next (exam*)

#22 "3 point"

#23 "three point"

#24 "pattern analys*"

#25 ABDC

#26 menzies

#27 "7 point"

#28 "seven point"

#29 digital near/2 (dermoscop* or dermatoscop*)

#30 "artificial intelligence"

#31 "AI"

#32 "computer assisted"

#33 "computer aided"

#34 AI

#35 "neural network*"

#36 MoleMax

#37 "computer diagnosis"

#38 "image process*"

#39 "automatic classif*"

#40 SIAscope

#41 "image analysis"

#42 "optical near/2 scan*"

#43 Aura

#44 MelaFind

#45 SIMSYS

#46 MoleMate

#47 SolarScan

#48 Vivascope

#49 "confocal microscopy"

#50 high near/3 ultraso*

#51 canine near/2 detect*

#52 Mole* near/2 map*

#53 total near/2 body

#54 mobile* or smart near/2 phone*

#55 cell next phone*

#56 smartphone*

#57 "mitotic index"

#58 DermoScan or SkinVision or DermLink or SpotCheck

#59 "Mole Detective"

#60 "Spot Check"

#61 mole* near/2 map*

#62 total near/2 body

#63 "exfoliative cytolog*"

#64 "digital analys*"

#65 image near/3 software

#66 teledermatolog* or tele‐dermatolog* or telederm or tele‐derm or teledermoscop* or tele‐dermoscop* or teledermatoscop* or tele‐dermatolog*

#67 "optical coherence" next (technolog* or tomog*)

#68 computer near/2 diagnos*

#69 sentinel near/2 node*

#70 #11 or #12 or #13 or #14 or #15 or #16 or #17 or #18 or #19 or #20 or #21 or #22 or #23 or #24 or #25 or #26 or #27 or #28 or #29 or #30 or #31 or #32 or #33 or #34 or #35 or #36 or #37 or #38 or #39 or #40 or #41 or #42 or #43 or #44 or #45 or #46 or #47 or #48 or #49 or #50 or #51 or #52 or #53 or #54 or #55 or #56 or #57 or #58 or #59 or #60 or #61 or #62 or #63 or #64 or #65 or #66 or #67 or #68 or #69

#71 ultraso*

#72 sonograph*

#73 MeSH descriptor: [Ultrasonography] explode all trees

#74 Doppler

#75 CT or PET or PET‐CT

#76 "CAT SCAN" or "CATSCAN"

#77 MeSH descriptor: [Positron‐Emission Tomography] explode all trees

#78 MeSH descriptor: [Tomography, X‐Ray Computed] explode all trees

#79 MRI

#80 MeSH descriptor: [Magnetic Resonance Imaging] explode all trees

#81 MRI or fMRI or NMRI or scintigraph*

#82 "magnetic resonance imag*"

#83 MeSH descriptor: [Deoxyglucose] explode all trees

#84 deoxyglucose or deoxy‐glucose

#85 "positron emission tomograph*"

#86 #71 or #72 or #73 or #74 or #75 or #76 or #77 or #78 or #79 or #80 or #81 or #82 or #83 or #84 or #85

#87 stage* or staging or metasta* or recurrence or sensitivity or specificity or "false negative*" or thickness*

#88 MeSH descriptor: [Neoplasm Staging] explode all trees

#89 #87 or #88

#90 #89 and #86

#91 #70 or #90

#92 #10 and #91

#93 BCC or CSCC or NMCS

#94 keratinocy*

#95 #93 or #94

#96 #10 or #95

#97 nevisense

#98 HFUS

#99 "electrical impedance spectroscopy"

#100 "history taking"

#101 "patient history"

#102 naked next eye near/1 (exam* or assess*)

#103 skin next exam*

#104 "ugly duckling" or (UD sign*)

#105 MeSH descriptor: [Physical Examination] explode all trees

#106 (physician* or clinical or physical) near/1 (exam* or recog* or triage*)

#107 ABCDE

#108 "clinical accuracy"

#109 MeSH descriptor: [General Practice] explode all trees

#110 confocal near microscop*

#111 "diagnostic algorithm*"

#112 MeSH descriptor: [Clinical Competence] explode all trees

#113 checklist*

#114 "virtual image*"

#115 "volatile organic compound*"

#116 dog or dogs

#117 VOC

#118 "gene expression analys*"

#119 "reflex transmission imaging"

#120 "thermal imaging"

#121 elastography

#122 #97 or #98 or #99 or #100 or #101 or #102 or #103 or #104 or #105 or #106 or #107 or #108 or #109 or #110 or #111 or #112 or #113 or #114 or #115 or #116 or #117 or #118 or #119 or #120 or #121

#123 #70 or #122

#124 #96 and #123

#125 #96 and #90

#126 #125 or #124

#127 #10 and #126

Database: CINAHL Plus (EBSCO) 1937 to 30 August 2016

Search strategy:

S1 (MH "Melanoma") OR (MH "Nevi and Melanomas+")

S2 (MH "Skin Neoplasms+")

S3 (MH "Carcinoma, Basal Cell+")

S4 basalioma*

S5 (basal cell) N2 (cancer* or carcinoma* or mass or masses or tumor* or tumour* or neoplasm* or adenoma* or epithelioma* or lesion* or malignan* or nodule*)

S6 (pigmented) N2 (lesion* or mole* or nevus or nevi or naevus or naevi or skin)

S7 melanom* or nonmelanoma* or non‐melanoma* or melanocyt* or non‐melanocyt* or nonmelanocyt*

S8 nmsc

S9 TX BCC or cscc or NMSC

S10 (MH "Keratinocytes")

S11 keratinocyt*

S12 S1 OR S2 OR S3 OR S4 OR S5 OR S6 OR S7 OR S8 OR S9 OR S10 OR S11

S13 dermoscop* or dermatoscop* or photomicrograph* or (3 point) or (three point) or ABCD* or menzies or (7 point) or (seven point) or AI or Molemax or SIASCOP* or Aura or MelaFind or SIMSYS or MoleMate or SolarScan or smartphone* or DermoScan or SkinVision or DermLink or SpotCheck

S14 (epiluminescence or confocal or incident or surface) N2 (microscop*)

S15 visual N1 (inspect* or examin*)

S16 (clinical or physical) N1 (examin*)

S17 pattern analys*

S18 (digital) N2 (dermoscop* or dermatoscop*)

S19 (artificial intelligence)

S20 (computer) N2 (assisted or aided)

S21 (neural network*)

S22 (MH "Diagnosis, Computer Assisted+")

S23 (image process*)

S24 (automatic classif*)

S25 (image analysis)

S26 SIAScop*

S27 (optical) N2 (scan*)

S28 (high) N3 (ultraso*)

S29 elastography

S30 (mobile or cell or cellular or smart) N2 (phone*) N2 (app or application*)

S31 (mole*) N2 (map*)

S32 total N2 body

S33 exfoliative cytolog*

S34 digital analys*

S35 image N3 software

S36 teledermatolog* or tele‐dermatolog* or telederm or tele‐derm or teledermoscop* or tele‐dermoscop* or teledermatoscop* or tele‐dermatoscop* teledermatolog* or tele‐dermatolog* or telederm or tele‐derm or teledermoscop*

S37 (optical coherence) N1 (technolog* or tomog*)

S38 computer N2 diagnos*

S39 sentinel N2 node

S40 (MH "Sentinel Lymph Node Biopsy")

S41 nevisense or HFUS or checklist* or VOC or dog*

S42 electrical impedance spectroscopy

S43 history taking

S44 "Patient history"

S45 naked eye

S46 skin exam*

S47 physical exam*

S48 ugly duckling

S49 UD sign*

S50 (physician* or clinical or physical) N1 (exam*)

S51 clinical accuracy

S52 general practice

S53 (physician* or clinical or physical) N1 (recog* or triage)

S54 confocal microscop*

S55 clinical competence

S56 diagnostic algorithm*

S57 checklist*

S58 virtual image*

S59 volatile organic compound*

S60 gene expression analys*

S61 reflex transmission imag*

S62 thermal imaging

S63 S13 or S14 or S15 OR S16 OR S17 OR S18 OR S19 OR S20 OR S21 OR S22 OR S23 OR S24 OR S25 OR S26 OR S27 OR S28 OR S29 OR S30 OR S31 OR S32 OR S33 OR S34 OR S35 OR S36 OR S37 OR S38 OR S39 OR S40 OR S41 OR S42 OR S43 OR S44 OR S45 OR S46 OR S47 OR S48 OR S49 OR S50 OR S51 OR S52 OR S53 OR S54 OR S55 OR S56 OR S57 OR S58 OR S59 OR S60 OR S61 OR S62

S64 CT or PET

S65 PET‐CT

S66 FDG or F18 or Fluorodeoxyglucose or radiopharmaceutical*

S67 (MH "Deoxyglucose+")

S68 deoxy‐glucose or deoxyglucose

S69 CATSCAN

S70 CAT‐SCAN

S71 (MH "Deoxyglucose+")

S72 (MH "Tomography, Emission‐Computed+")

S73 (MH "Tomography, X‐Ray Computed")

S74 positron emission tomograph*

S75 (MH "Magnetic Resonance Imaging+")

S76 MRI or fMRI or NMRI or scintigraph*

S77 echography

S78 doppler

S79 sonograph*

S80 ultraso*

S81 magnetic resonance imag*

S82 S64 OR S65 OR S66 OR S67 OR S68 OR S69 OR S70 OR S71 OR S72 OR S73 OR S74 OR S75 OR S76 OR S77 OR S78 OR S79 OR S80 OR S81

S83 stage* or staging or metasta* or recurrence or sensitivity or specificity or (false negative*) or thickness

S84 (MH "Neoplasm Staging")

S85 S83 OR S84

S86 S82 AND S85

S87 S63 OR S86

S88 S12 AND S87

Database: Science Citation Index SCI Expanded (Web of Science) 1900 to 30 August 2016

Conference Proceedings Citation Index (Web of Science) 1900 to 1 September 2016

Search strategy:

#1 (melanom* or nonmelanom* or non‐melanoma* or melanocyt* or non‐melanocyt* or nonmelanocyt* or keratinocyt*)

#2 (basalioma*)

#3 ((skin) near/2 (cancer* or carcinoma or mass or masses or tumour* or tumor* or neoplasm* or adenoma* or epithelioma* or lesion* or malignan* or nodule*))

#4 ((basal) near/2 (cancer* or carcinoma* or mass or masses or tumour* or tumor* or neoplasm* or adenoma* or epithelioma* or lesion* or malignan* or nodule*))

#5 ((pigmented) near/2 (lesion* or mole* or nevus or nevi or naevus or naevi or skin))

#6 (nmsc or BCC or NMSC or keratinocy*)

#7 ((squamous cell (cancer* or carcinoma* or mass or masses or tumour* or tumor* or neoplasm* or adenoma* or epithelioma* or lesion* or malignan* or nodule*))

#8 (skin or epiderm* or cutaneous)

#9 #8 AND #7

#10 #9 OR #6 OR #5 OR #4 OR #3 OR #2 OR #1

#11 ((dermoscop* or dermatoscop* or photomicrograph* or epiluminescence or confocal or "incident light" or "surface microscop*" or "visual inspect*" or "physical exam*" or 3 point or three point or pattern analy* or ABCDE or menzies or 7 point or seven point or dermoscop* or dermatoscop* or AI or artificial or computer aided or computer assisted or neural network* or Molemax or image process* or automatic classif* or image analysis or siascope or optical scan* or Aura or melafind or simsys or molemate or solarscan or vivascope or confocal microscop* or high ultraso* or canine detect* or cellphone* or mobile* or phone* or smartphone or dermoscan or skinvision or dermlink or spotcheck or spot check or mole detective or mole map* or total body or exfoliative psychology or digital or image software or optical coherence or teledermatology or telederm* or teledermoscop* or teledermatoscop* or computer diagnos* or sentinel))

#12 ((nevisense or HFUS or impedance spectroscopy or history taking or patient history or naked eye or skin exam* or physical exam* or ugly duckling or UD sign* or physician* exam* or physical exam* or ABCDE or clinical accuracy or general practice or confocal microscop* or clinical competence or diagnostic algorithm* or checklist* or virtual image* or volatile organic or VOC or dog* or gene expression or reflex transmission or thermal imag* or elastography))

#13 #11 or #12

#14 ((PET or CT or FDG or deoxyglucose or deoxy‐glucose or fluorodeoxy* or radiopharma* or CATSCAN or positron emission or computer assisted or nuclear magnetic or MRI or FMRI or NMRI or scintigraph* or echograph* or Doppler or sonograph* or ultraso* or magnetic reson*))

#15 ((stage* or staging or metast* or recurrence or sensitivity or specificity or false negative* or thickness*))

#16 #14 AND #15

#17 #16 OR #13

#18 #10 AND #17

Refined by: DOCUMENT TYPES: (MEETING ABSTRACT OR PROCEEDINGS PAPER)

Appendix 5. Full‐text inclusion criteria

The title and abstract screening will lead to the retrieval of a large number of full text journal papers and conference abstracts from which to populate the four sets of test accuracy reviews and the intervention review. The systematic reviews will largely be carried out sequentially, beginning with the reviews of tests for melanoma diagnosis; however, the full‐text papers need to be screened at the beginning of the Programme Grant and papers meeting the inclusion criteria tagged accordingly by review.

The table below summarises the inclusion criteria to be applied; these will be transferred to an Excel spreadsheet or Google Forms so that pertinent information can be recorded about each eligible study and reasons for exclusion recorded about each ineligible study.

Criterion	Inclusion	Exclusion
Study design	For diagnostic and staging reviews Any study for which a 2 × 2 contingency table can be extracted, e.g. diagnostic case control studies 'cross‐sectional' test accuracy study with retrospective or prospective data collection studies where estimation of test accuracy was not the primary objective but test results for both index and reference standard were available RCTs of tests or testing strategies where participants were randomised between index tests and all undergo a reference standard (i.e. accuracy RCTs)	< 5 melanoma cases (diagnosis reviews) < 10 participants (staging reviews) Studies developing new criteria for diagnosis unless a separate 'test set' of images were used to evaluate the criteria (mainly digital dermoscopy) Studies using 'normal' skin as controls Letters, editorials, comment papers, narrative reviews Insufficient data to construct a 2 × 2 table
Target condition	Melanoma Keratinocyte skin cancer (or non‐melanoma skin cancer) BCC or epithelioma cSCC	Studies exclusively conducted in children Studies of non‐cutaneous melanoma or SCC
Population	For diagnostic reviews Adults with a skin lesion suspicious for melanoma, BCC, or cSCC (other terms include pigmented skin lesion/naevi, melanocytic, keratinocyte, etc.) Adults at high risk of developing melanoma skin cancer, BCC, or cSCC For staging reviews Adults with a diagnosis of melanoma or cSCC undergoing tests for staging of lymph nodes or distant metastases or both	People suspected of other forms of skin cancer Studies conducted exclusively in children
Index tests	For diagnosis Visual inspection/clinical examination Dermoscopy/dermatoscopy Teledermoscopy Smartphone/mobile phone applications Digital dermoscopy/artificial intelligence Confocal microscopy Ocular coherence tomography Exfoliative cytology High‐frequency ultrasound Canine odour detection DNA expression analysis/gene chip analysis Other For staging CT PET PET‐CT MRI Ultrasound +/fine needle aspiration cytology FNAC SLNB +/high‐frequency ultrasound Other Any test combination and in any order Any test positivity threshold Any variation in testing procedure (e.g. radioisotope used)	Sentinel lymph biopsy for therapeutic rather than staging purposes Tests to determine melanoma thickness Tests to determine surgical margins/lesion borders Tests to improve histopathology diagnose LND
Reference standard	For diagnostic studies Histopathology of the excised lesion Clinical follow‐up of non‐excised/benign‐appearing lesions with later histopathology if suspicious Expert diagnosis (studies should not be included if expert diagnosis is the sole reference standard) For studies of imaging tests for staging Histopathology (via LND or SLMB) Clinical/radiological follow‐up A combination of the above For studies of SLNB accuracy for staging LND of both SLN+ and SLn participants to identify all diseased nodes LND of SLN+ participants and follow‐up of SLN participants to identify a subsequent nodal recurrence in a previously investigated nodal basin	For diagnostic studies Exclude if any disease‐positive participants have diagnosis unconfirmed by histology Exclude if > 50% of disease‐negative participants have diagnosis confirmed by expert opinion with no histology or follow‐up Exclude studies of referral accuracy, i.e. comparing referral decision with expert diagnosis, unless evaluations of teledermatology or mobile phone applications
BCC: basal cell carcinoma; cSCC: cutaneous squamous cell carcinoma; CT: computed tomography; FNAC: fine needle aspiration cytology; LND: lymph node dissection; MRI: magnetic resonance imaging; PET: positron emission tomography; PET‐CT: positron emission tomography computed tomography; RCT: randomised controlled trial; SCC: squamous cell carcinoma; SLN+: positive sentinel lymph node; SLn: negative sentinel lymph node; SLNB: sentinel lymph node biopsy.

Appendix 6. Quality assessment (based on QUADAS‐2)

The following tables use text that was originally published in the QUADAS‐2 tool by Whiting and colleagues (Whiting 2011).

Item	Response (delete as required)
PARTICIPANT SELECTION (1) ‐ RISK OF BIAS
1) Was a consecutive or random sample of participants or images enrolled?	Yes – if paper states consecutive or random No – if paper describes other method of sampling Unclear – if participant sampling not described
2) Was a case‐control design avoided?	Yes – if consecutive or random or case‐control design clearly not used No – if study described as case‐control or describes sampling specific numbers of participants with particular diagnoses Unclear – if not described
3) Did the study avoid inappropriate exclusions, e.g., 'difficult to diagnose' lesions not excluded lesions not excluded on basis of disagreement between evaluators	Yes – if inappropriate exclusions were avoided No – if lesions were excluded that might affect test accuracy, e.g., 'difficult to diagnose' lesions, or where disagreement between evaluators was observed Unclear – if not clearly reported but there is suspicion that difficult to diagnose lesions may have been excluded
4) For between‐person comparative studies only (i.e., allocating different tests to different study participants): A) were the same participant selection criteria used for those allocated to each test? B) was the potential for biased allocation between tests avoided through adequate generation of a randomised sequence? C) was the potential for biased allocation between tests avoided through concealment of allocation prior to assignment?	For A) Yes – if same selection criteria were used for each index test, No – if different selection criteria were used for each index test, Unclear – if selection criteria per test were not described, N/A – if only 1 index test was evaluated or all participants received all tests For B) Yes – if adequate randomisation procedures are described, No – if inadequate randomisation procedures are described, Unclear – if the method of allocation to groups is not described (a description of 'random' or 'randomised' is insufficient), N/A – if only 1 index test was evaluated or all participants received all tests For C) Yes – if appropriate methods of allocation concealment are described, No – if appropriate methods of allocation concealment are not described, Unclear – if the method of allocation concealment is not described (sufficient detail to allow a definite judgement is required), N/A – if only 1 index test was evaluated
Could the selection of participants have introduced bias? For non‐comparative and within‐person comparative studies If answers to all of questions 1), 2), and 3) 'Yes': If answers to any 1 of questions 1), 2), or 3) 'No': If answers to any 1 of questions 1), 2), or 3) 'Unclear': For between‐person comparative studies If answers to all of questions 1), 2), 3), and 4) 'Yes': If answers to any 1 of questions 1), 2), 3), or 4) 'No': If answers to any 1 of questions 1), 2), 3), or 4) 'Unclear':	For non‐comparative and within‐person comparative studies Risk is low Risk is high Risk unclear For between‐person comparative studies Risk is low Risk is high Risk unclear
PARTICIPANT SELECTION (1) ‐ CONCERNS REGARDING APPLICABILITY
1) Are the included participants and chosen study setting appropriate to answer the review question, i.e., are the study results generalisable? This item is not asking whether exclusion of certain participant groups might bias the study's results (as in Risk of Bias above), but is asking whether the chosen study participants and setting are appropriate to answer our review question. Because we are looking to establish test accuracy in both primary presentation and referred participants, a study could be appropriate for 1 setting and not for the other, or it could be unclear as to whether the study can appropriately answer either question For each study assessed, please consider whether it is more relevant for A) participants with a primary presentation of a skin lesion or B) referred participants, and respond to the questions in either A) or B) accordingly. If the study gives insufficient details, please respond Unclear to both parts of the question	A) For studies that will contribute to the analysis of participants with a primary presentation of a skin lesion (i.e., test naive) Yes – if participants included in the study appear to be generally representative of those who might present in a usual practice setting No – if study participants appear to be unrepresentative of usual practice, e.g., in terms of severity of disease, demographic features, presence of differential diagnosis or co‐morbidity, setting of the study, and previous testing protocols Unclear – if insufficient details are provided to determine the generalisability of study participants B) For studies that will contribute to the analysis of referred participants (i.e., who have already undergone some form of testing) Yes – if study participants appear to be representative of those who might be referred for further investigation. If the study focuses only on those with equivocal lesions, for example, we would suggest that this is not representative of the wider referred population No – if study participants appear to be unrepresentative of usual practice, e.g., if a particularly high proportion of participants have been self‐referred or referred for cosmetic reasons. Other factors to consider include severity of disease, demographic features, presence of differential diagnosis or co‐morbidity, setting of the study, and previous testing protocols Unclear – if insufficient details are provided to determine the generalisability of study participants
2) Did the study avoid including participants with multiple lesions?	Yes – if the difference between the number of included lesions and number of included participants is less than 5% No – if the difference between the number of included lesions and number of included participants is greater than 5% Unclear – if it is not possible to assess
Is there concern that the included participants do not match the review question? If the answer to question 1) or 2) 'Yes': If the answer to question 1) or 2) 'No': If the answer to question 1) or 2) 'Unclear':	Concern is low Concern is high Concern is unclear
INDEX TEST (2) ‐ RISK OF BIAS (to be completed per test evaluated)
1) Was the index test or testing strategy result interpreted without knowledge of the results of the reference standard?	Yes – if index test described as interpreted without knowledge of reference standard result or, for prospective studies, if index test is always conducted and interpreted prior to the reference standard No – if index test described as interpreted in knowledge of reference standard result Unclear – if index test blinding is not described
2) Was the diagnostic threshold at which the test was considered positive (i.e., BCC or cSCC present) prespecified?	Yes – if threshold was prespecified (i.e., prior to analysing study results) No – if threshold was not prespecified Unclear – if not possible to tell whether or not diagnostic threshold was prespecified
3) For within‐person comparisons of index tests or testing strategies (i.e., > 1 index test applied per participant): was each index test result interpreted without knowledge of the results of other index tests or testing strategies?	Yes – if all index tests were described as interpreted without knowledge of the results of the others No – if the index tests were described as interpreted in the knowledge of the results of the others Unclear – if it is not possible to tell whether knowledge of other index tests could have influenced test interpretation N/A – if only 1 index test was evaluated
Could the conduct or interpretation of the index test have introduced bias? For non‐comparative and between‐person comparison studies If answers to questions 1) and 2) 'Yes': If answers to either questions 1) or 2) 'No': If answers to either questions 1) or 2) 'Unclear': For within‐person comparative studies If answers to all questions 1), 2), and 3) for any index test 'Yes': If answers to any 1 of questions 1), 2), or 3) for any index test 'No': If answers to any 1 of questions 1), 2), or 3) for any index test 'Unclear':	For non‐comparative and between‐person comparison studies Risk is low Risk is high Risk is unclear For within‐person comparative studies Risk is low Risk is high Risk is unclear
INDEX TEST (2) ‐ CONCERN ABOUT APPLICABILITY
1) Was the diagnostic threshold to determine presence or absence of disease established in a previously published study? E.g., previously evaluated/established algorithm/checklist used lesion characteristics indicative of BCC or cSCC used objective (usually numerical) threshold used	Yes – if a previously evaluated/established tool to aid diagnosis of BCC or cSCC was used or if the diagnostic threshold used was established in a previously published study No – if an unfamiliar/new tool to aid diagnosis of BCC or cSCC was used, if no particular algorithm was used, or if the objective threshold reported was chosen based on results in the current study Unclear – if insufficient information was reported
2) Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication? Study results can only be reproduced if the diagnostic threshold is described in sufficient detail. This item applies equally to studies using pattern recognition and those using checklists or algorithms to aid test interpretation	Yes – if the criteria for diagnosis of BCC or cSCC were reported in sufficient detail to allow replication No – if the criteria for diagnosis of BCC or cSCC were not reported in sufficient detail to allow replication Unclear – if some but not sufficient information on criteria for diagnosis to allow replication were provided
3) Was the test interpretation carried out by an experienced examiner?	Yes – if the test was interpreted by 1 or more speciality‐accredited dermatologists, or by examiners of any clinical background with special interest in dermatology and with any formal training in the use of the test No – if the test was not interpreted by an experienced examiner (see above) Unclear – if the experience of the examiner(s) was not reported in sufficient detail to judge or if examiners described as 'Expert' with no further detail given N/A – if system‐based diagnosis, i.e., no observer interpretation
Is there concern that the index test, its conduct, or interpretation differ from the review question? If answers to questions 1), 2), and 3) 'Yes': If answers to questions 1), 2), or 3) 'No': If answers to questions 1), 2), or 3) 'Unclear':	Concern is low Concern is high Concern is unclear
REFERENCE STANDARD (3) ‐ RISK OF BIAS
1) Is the reference standard likely to correctly classify the target condition? A) Disease‐positive ‐ 1 or more of the following: histological confirmation of BCC or cSCC following biopsy or lesion excision clinical follow‐up of benign‐appearing lesions for at least 6 (or 3 for cSCC) months following the application of the index test, leading to a histological diagnosis of BCC or cSCC B) Disease‐negative ‐ 1 or more of the following: histological confirmation of absence of BCC or cSCC following biopsy or lesion excision in at least 80% of disease‐negative participants clinical follow‐up of benign‐appearing lesions for a minimum of 6 months (or 3 for cSCC) following the index test in up to 20% of disease‐negative participants	A) Disease‐positive Yes – if all participants with a final diagnosis of BCC or cSCC underwent 1 of the listed reference standards No – if a final diagnosis of BCC or cSCC for any participant was reached without histopathology Unclear – if the method of final diagnosis was not reported for any participant with a final diagnosis of BCC or cSCC or if the length of clinical follow‐up used was not clear or if a clinical follow‐up reference standard was reported in combination with a participant‐based analysis and it was not possible to determine whether the detection of a malignant lesion during follow‐up is the same lesion that originally tested negative on the index test B) Disease‐negative Yes – if at least 80% of benign diagnoses were reached by histology and up to 20% were reached by clinical follow‐up for a minimum of 6 (or 3) months following the index test No – if more than 20% of benign diagnoses were reached by clinical follow‐up for a minimum of 6 (or 3) months following the index test or if clinical follow‐up period was less than 6 (or 3) months Unclear – if the method of final diagnosis was not reported for any participant with benign diagnosis
2) Were the reference standard results interpreted without knowledge of the results of the index test? Please score this item for all studies even though histopathology interpretation is usually conducted with knowledge of the clinical diagnosis (from visual inspection or dermoscopy or both). We will deal with this by not including the response to this item in the 'Risk of bias' assessment for these tests. For reviews of all other tests, this item will be retained	Yes – if the reference standard diagnosis was reached blinded to the index test result No – if the reference standard diagnosis was reached with knowledge of the index test result Unclear – if blinded reference test interpretation was not clearly reported
Could the reference standard, its conduct, or its interpretation have introduced bias? For visual inspection/dermoscopy evaluations If answer to question 1) 'Yes': If answer to question 1) 'No': If answer to question 1) 'Unclear': For all other tests If answers to questions 1) and 2) 'Yes': If answers to questions 1) or 2) 'No': If answers to questions 1) or 2) 'Unclear':	For visual inspection/dermoscopy evaluations Risk is low Risk is high Risk is unclear For all other tests Risk is low Risk is high Risk is unclear
REFERENCE STANDARD (3) ‐ CONCERN ABOUT APPLICABILITY
1) Are index test results presented separately for each component of the target condition (i.e., separate results presented for those with invasive melanoma, melanoma in situ, lentigo maligna, severe dysplasia, BCC, and cSCC)?	Yes – if index test results for each component of the target condition can be disaggregated No – if index test results for the different components of the target condition cannot be disaggregated Unclear – if not clearly reported
2) Expert opinion (with no histological confirmation) was not used as a reference standard 'Expert opinion' means diagnosis based on the standard clinical examination, with no histology or lesion follow‐up ***do not complete this item for teledermatology studies	Yes – if expert opinion was not used as a reference standard for any participant No – if expert opinion was used as a reference standard for any participant Unclear – if not clearly reported
3) Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Yes – if histology interpretation was reported to be carried out by an experienced histopathologist or dermatopathologist No – if histology interpretation was reported to be carried out by a less experienced histopathologist Unclear – if the experience/qualifications of the pathologist were not reported
Is there concern that the target condition as defined by the reference standard does not match the review question? If answers to all questions 1), 2), and 3) 'Yes': If answers to any 1 of questions 1), 2), or 3) 'No': If answers to any 1 of questions 1), 2), or 3) 'Unclear': ***For teledermatology studies only If answers to all questions 1) and 3) 'Yes': If answers to questions 1) or 3) 'No': If answers to questions 1) or 3) 'Unclear':	Concern is low Concern is high Concern is unclear ***For teledermatology studies only Concern is low Concern is high Concern is unclear
FLOW AND TIMING (4): RISK OF BIAS
1) Was there an appropriate interval between index test and reference standard? A) For histopathological reference standard, was the interval between index test and reference standard ≤ 1 month? B) If the reference standard includes clinical follow‐up of borderline/benign‐appearing lesions, was there at least 6 (or 3) months' follow‐up following application of index test(s) for studies of BCC (or cSCC)?	A) Yes – if study reports ≤ 1 month between index and reference standard No – if study reports > 1 month between index and reference standard Unclear – if study does not report interval between index and reference standard B) Yes – if study reports ≥ 6 (or 3 for cSCC) months' follow‐up No – if study reports < 6 (or 3 for cSCC) months' follow‐up Unclear – if study does not report length of clinical follow‐up
2) Did all participants receive the same reference standard?	Yes – if all participants underwent the same reference standard No – if more than 1 reference standard was used Unclear – if not clearly reported
3) Were all participants included in the analysis?	Yes – if all participants were included in the analysis No – if some participants were excluded from the analysis Unclear– if not clearly reported
4) For within‐person comparisons of index tests Was the interval between application of index tests ≤ 1 month?	Yes – if study reports ≤ 1 month between index tests No – if study reports > 1 month between index tests Unclear – if study does not report interval between index tests
Could the participant flow have introduced bias? For non‐comparative and between‐person comparison studies If answers to questions 1), 2), and 3) 'Yes': If answers to any 1 of questions 1), 2), or 3) 'No': If answers to any 1 of questions 1), 2), or 3) 'Unclear': For within‐person comparative studies If answers to all questions 1), 2), 3), and 4) 'Yes': If answers to any 1 of questions 1), 2), 3), or 4) 'No': If answers to any 1 of questions 1), 2), 3), or 4) is 'Unclear':	For non‐comparative and between‐person comparison studies Risk is low Risk is high Risk is unclear For within‐person comparative studies Risk is low Risk is high Risk is unclear
BCC = basal cell carcinoma; cSCC = cutaneous squamous cell carcinoma

Appendix 7. Summary of tests and target conditions evaluated per study

	In‐person		Image‐based		Other tests evaluated in study	Target conditions reported			Appears in melanoma review
	Visual inspection	Dermoscopy added to VI	Visual inspection	Dermoscopic images		BCC	SCC	KER
Altamura 2010	‐	‐	‐	X	‐	X	‐	‐	‐
Amirnia 2016	‐	X	‐	‐	‐	X	‐	‐	‐
Argenziano 2006	X	X	‐	‐	‐	‐	‐	X	X
Carli 2002a	X	X	‐	X	‐	X	‐	‐	X
Carli 2002b	‐	‐	X	X	‐	X	‐	X	X
Chang 2013	X	‐	‐	‐	‐	‐	‐	X	X
Cooper 2002	X	‐	‐	‐	‐	X	X	X
Durdu 2011	‐	X	‐	‐	Exfoliative cytology	X	‐	X	X
Ek 2005	X	‐	‐	‐	‐	X	X	X	X
Gokdemir 2011	‐	X	‐	‐	‐	X	‐	‐	X
Hacioglu 2013	X	‐	‐	X	CAD	‐	‐	X	‐
Lorentzen 1999	‐	‐	X	‐	‐	X	‐	‐	X
Lorentzen 2008	‐	‐	‐	X	‐	X	‐	‐	X
Markowitz 2015	X	X	‐	‐	OCT	X	‐	‐	‐
Menzies 2000	‐	‐	‐	X	‐	X	‐	X	‐
Navarrete Dechent 2016	‐	‐	‐	X	‐	X	X	X	‐
Nori 2004	‐	‐	X	‐	‐	X	‐	‐	‐
Rosendahl 2011	‐	‐	X	X	‐	X	‐	X	X
Schwartzberg 2005	X	‐	‐	‐	‐	X	‐	‐	‐
Stanganelli 2000	X	X	‐	‐	‐	X	‐	‐	X
Steiner 1987	X	‐	‐	‐	‐	X	‐	‐	X
Ulrich 2015	X	X	‐	‐	OCT	X	‐	‐	‐
Witkowski 2016	‐	‐	‐	X	RCM	X	X	X	‐
Zalaudek 2006	‐	‐	‐	X	‐	X	‐	‐	X
Footnotes: BCC – basal cell carcinoma; CAD – computer‐assisted diagnosis; cSCC – cutaneous squamous cell carcinoma; KER ‐ any skin cancer; OCT ‐ optical coherence tomography; RCM – reflectance confocal microscopy; VI ‐ visual inspection

Appendix 8. Summary study details

Study author Outcomes reported Pathway	Study type Country Setting	Inclusion criteria	Index tests (algorithm) Diagnostic approach	Threshold	Observer qualifcation (number) Experience	Reference standard Final diagnoses Prevalence (Any) Exclusions (if reported)
In‐person evaluations
Amirnia 2016 BCC Referred (selected on reference) (c)	NC NR‐CS Iran Secondary 61 / 61	Patients suspected of BCC or melanocytic naevi of the face who were referred to dermatology clinic	Dermoscopy (3‐point checklist plus dermatoscopic criteria of melanocytic naevi and BCC) In person	≥ 2 characteristics present; diagnosis of BCC	Dermatologist (assumed) (n = NR; experience NR) Single observer	Histology BCC 27 Benign 28 27/61; 44%
Argenziano 2006 Any Limited prior testing; selected on refererence standard (c)	BPC RCT Italy, Spain Primary NR / 85 (Full sample 1203 lesions*)	Patients asking for screening or exhibiting 1 or more skin tumours as seen during routine physical examination (patient‐finding screening). Participating PCPs randomised to either visual inspection alone or visual inspection plus dermoscopy; only excised lesions can be included for each arm.	VI (ABCD) Dermoscopy (3‐point checklist) In person	Subjective impression; dx of malignancy	GPs (n = 37) All trained in ABCD rule Single observer	Histology MEL 6 BCC 37; SCC 10 Benign 32 53/85; 62% NB: Only those patients who were considered to have lesions suggestive of skin cancer had histology and could be included; rest had expert diagnosis (making full dataset ineligible for this review)
Carli 2002a BCC (MEL) Referred (selected on reference) (u)	WPC NR‐CS Italy Secondary NR/256	Clinically equivocal or suspicious PSL subjected to excisional biopsy at the Institute of Dermatology	1. VI (no algorithm) 2. Dermoscopy (pattern) In‐person (Dermoscopy – image‐based)	Subjective impression	Dermatologist (n = 2; High experience – “extensive experience in both clinical and dermoscopic diagnosis”) Consensus of 2	Histology MM 40; MiS 14 BCC 5 BN 177; SN 16; SK 4 BCC: 5/256; 2% No exclusions reported NB: BCC (VI): 2 MMS were FP; BCC (Derm – pattern): all MM TN
Chang 2013 Any Referred (selected on reference) (u)	NC R‐CS Taiwan Secondary 676/769	Potentially malignant biopsied or excised skin lesions (nontumour specimens excluded)	VI (no algorithm) In person	Subjective impression; definitely malignant	Dermatologists; n = 25 Board‐certified Single observer	Histology MM 4; MiS 4 BCC: 110; cSCC: 20 'Benign' diagnoses: 595 Skin cancer: 152/769; 20% Exclusions: Poor‐quality index test image; mis‐registered or poor‐quality images (unfocused or containing a motion artifact)
Cooper 2002 BCC cSCC Any Follow‐up (c)	NC P‐CS UK Spec. clinic NR/102	Patients attending the open‐access dermatology renal transplant clinic with suspicious lesions	VI (No algorithm) In person	NR; correct diagnosis of malignancy	Mixed (n = 2; experience NR) Single observer	Histology BCC 12; cSCC 21 KA 2; BD 19; Solar 16; viral warts 7; other 25 BCC: 12/102; 12% SCC: 21/102; 21% Exclusions: BCC: 3 SCCs were FP
Durdu 2011 BCC Any (MEL) Referred (selected on reference) (u)	WPC P‐CS Secondary Turkey 176/200	PSL that could not be diagnosed with only dermatologic physical examination; 2 x 2 included for melanocytic subset	Dermoscopy (No algorithm (ABCD for diagnosis of melanoma only) Also evaluated exfoliative cytology In person	NR	Dermatologist (n = 1; experience NR) Single observer	Histology MEL 10; BCC: 34; Other malignant 2 SK 24; BN 100; DF 12; Warts 16; Dirt 1; Other 1 BCC: 34/200; 17% ‐
Ek 2005 BCC cSCC Any (MEL) Referred (selected on reference) (c)	NC P‐CS Aus. Specialist clinic 1223/2582	Lesions excised for which malignancy could not be excluded	VI (no algorithm) In person	Subjective impression	Plastic surgeon (n = 4 or 5; mixed experience; 3 consultants, 1 plastic surgery trainee (usually 1st year, on 6‐month rotation) and a clinical assistant) Unclear	Histology MEL 23 BCC 1214; SCC 517; BD 188; SK 63; 577 other benign (incl 330 solar keratosis) BCC: 1214/2582; 47% SCC: 517/2582; 20% Exclusions: Incomplete or incorrectly entered proformas were excluded – 79 patients with 96 lesions NB for BCC: 202 SCC and 6 MM were counted as FPs
Gokdemir 2011 BCC [MEL] Referred (selected on reference) (u)	NC NR‐CS Secondary Turkey 362/449	Patients with melanocytic and non‐melanocytic skin lesions with dermoscopic and histologic diagnoses	Dermoscopy (no algorithm) Unclear if in‐person or image‐based	Subjective assessment (dx of MM)	Dermatologist (n = NR; experience High “at least 2 years’ experience with Molemax II”) Unclear obs interp	Histology MEL 13; BCC: 45 Benign: 390 BCC: 45/448; 10% NB for BCC: 1 MM was counted as FP
Hacioglu 2013 Any Referred (selected on reference) (u)	WPC NR‐CS Turkey Secondary 76 / 80	Patients with skin lesions <12 mm diameter suspicious for malignancy; lesions that had a crusted or rough surface were excluded. NB aim is diagnose non melanoma skin cancers	VI (no algorithm) In‐person [Also evaluates image‐based dermoscopy and CAD]	Subjective impression; diagnosis of BCC/cSCC	Dermatologist (assumed) (n = 1; experience NR) Single observer	Histology MM 3; BCC 24; cSCC 3; basosquamous 2 SK 19; AK 8; intradermal nevus 4; DF 3; KA 2; Other 12 Skin cacner: 29/80; 36% Study reports 0 excluded from analysis after histopathology results NB: 3 MM considered disease negative by authors; cannot be disaggregated
Markowitz 2015 BCC Equivocal lesions (selected on reference) (u)	WPC P‐CS US Secondary 100 / 115	Adults with ≤ 3 suspicious lesions, if they had ≥ 1 clinically challenging pink lesions, on the head or neck, that was suspicious for BCC, and to be biopsied to rule BCC in or out, and if they were eligible for Mohs surgery	VI (no algorithm) Dermoscopy (2‐step algorithm Marghoob 2010) In‐person (Also evaluates OCT)	Possible BCC	Dermatologist (assumed) (n = NR; experience NR) Unclear	Histology BCC 70 Benign 45 BCC: 70/115; 61% No exclusions reported
Schwartzberg 2005 BCC Referred (selected on reference) (u)	WPC‐algs P‐CS US Secondary 141/141	Patients with suspected BCC undergoing biopsy	VI (no algorithm; own new algorithm) In‐person	BCC certain or likely (Confidence level 1 or 2)	Dermatologist (assumed) (n = 17; experience NR) Single	Histology BCC 82 Benign 59 BCC: 82/141; 58% ‐
Stanganelli 2000 BCC Any (MEL) Referred (unselected on reference) (u)	WPC R‐CS Italy Specialist clinic NR/3372	PSL referred by dermatologists and general practitioners either for pre‐surgical assessment or consultation	1. VI (ABCD) 2. Dermoscopy (pattern analysis) In person	NR Subjective impression	NR (assumed dermatologist ‐ described as one of the co‐authors; n = 1) Single observer	Histology / Registry FU MEL 55 BCC 43; Benign 3274 43/3372; 1% No exclusions reported NB for BCC: all MMs were TN for VI and for dermoscopy
Steiner 1987 BCC Any (MEL) Equivocal (selected on reference) (u)	WPC P‐CS Austria Spec. clinic NR / 318	Small (< 10 mm) diagnostically equivocal PSL; no absolute agreement on clinical diagnosis among investigating clinicians at a pigmented lesion clinic	1. VI (no algorithm) In person (also evaluated dermoscopy)	Subjective impression	Dermatologists (n = 3; High experience ‐ "experienced dermatologists") Consensus of 3 observers	Histology MM 49; MiS 24 BCC 20 BN 143; SK 20; lentigo simplex and naevoid lentigo 19; Other 15 BCC: 20/318; 9% No exclusions reported NB: Dermoscopy data excluded as no breakdown of incorrect diagnoses For BCC (VI): 3 MMs were counted as FP
Ulrich 2015 BCC Equivocal (selected on reference) (u)	WPC P‐CS Germany Secondary 155/231	Patients with non‐pigmented pink lesions with clinical suspicion of BCC requiring biopsy for diagnostic confirmation. Pink lesions defined as clinically unclear erythematous papule or plaque; either reddish macules, patches or small papules with or without scale	VI (no algorithm) Dermoscopy (2‐step algorithm Marghoob 2012) In person (Also evaluates OCT)	Clinical characteristics of BCC	Dermatologist (assumed) (n = NR; experience NR) Single observer	Histology *BCC 141 Benign 94 BCC:141/235; 60% Exclusions: Histology was missing for 21 lesions, and 1 case was found to have a combination of both BCC and SK or AK, leaving 235 lesions for analysis NB: 231 diagnoses available for VI (140 BCC) and 231 for dermoscopy (139 BCCs)
Image‐based evaluations
Altamura 2010 BCC Referred (selected on reference) (c)	NC RP‐CCS Secondary Italy; Aus; Austria NR/300	Skin lesions randomly selected from digital databases at dermatology departments and tertiary referral centre; all excised	Dermoscopy (Menzies for BCC (rev)) Image‐based (none)	Diagnosis of BCC	Dermatologist (assumed) (n = 3; experience High) observers experienced in dermatoscopic evaluation Single observer	Histology MM 40; MiS 10; BCC 150; cSCC 2 BN 50; SK 20; AK 12; DF 10; Other 6 BCC: 150/300; 50% NB: MM and cSCC results not disaggregated from Disease negative group
Carli 2002a BCC (MEL) Referred (selected on reference) (u)	WPC R‐CS Italy Secondary NR/256	Clinically equivocal or suspicious PSL subjected to excisional biopsy at the Institute of Dermatology	(Dermoscopy – image‐based) In person (Also evaluates in‐person VI and dermoscopy (see above))	Subjective impression	Dermatologist (n = 2; High experience – “extensive experience in both clinical and dermoscopic diagnosis”) Consensus of 2	Histology MM 40; MiS 14 BCC 5 BN 177; SN 16; SK 4 BCC: 5/256; 2% No exclusionsne reported NB for BCC: all MEL were test negative
Carli 2002b BCC Any (MEL) Referred (selected on reference) (u)	WPC R‐CS Italy Secondary NR / 57	Clinically suspicious or equivocal PSL undergoing excision for diagnostic purposes; all ≤ 14mm diameter	1. VI (NR) 2. Dermoscopy (NR) Image‐based (blinded)	NR	Dermatologists (n = 2) High experience ('with experience in the field of '); consensus of 2	Histology MM 6, MiS 5 BCC 10 BN 31, SK 1; Other 4 BCC; 10/57; 18% Exclusions: 4 ‘not evaluables’ excluded (NB these differ between clinical images and dermoscopic images (1 MM excluded from VI analysis)
Hacioglu 2013 Any Referred (selected on reference) (u)	WPC NR‐CS Turkey Secondary 76/80	Patients with skin lesions < 12 mm diameter suspicious for malignancy; lesions that had a crusted or rough surface were excluded. NB aim is diagnose non‐melanoma skin cancers	Dermoscopy (no algorithm) Image‐based (blinded) (Also evaluates in‐person VI and CAD)	Subjective impression; diagnosis of BCC/cSCC	Dermatologist (assumed) (n = 1; experience NR) Single observer	Histology MM 3; BCC 24; cSCC 3; basosquamous 2 SK 19; AK 8; intradermal naevus 4; DF 3; KA 2; Other 12 Skin cancer: 29/80; 36% Exclusions: Study reports 0 excluded from analysis after histopathology results B: 3 MM considered disease‐negative by study authors; cannot be disaggregated
Lorentzen 1999 BCC (MM) Referred (selected on reference) (c)	WPC P‐CS Specialist clinic Denmark 232/232	Patients with lesions suspicious for CMM referred to outpatients clinic	1. VI (no algorithm) 2. Dermoscopy (no algorithm) Image based (clinical image)	Subjective impression; correct dx of M	Mixed: Dermatologist (n = 4; experience High (4‐5 years daily experience) & 'non‐expert dermatology residents' (n = 5; 1 ‐ 2 years interest and formal training in dermatoscopy) Average	Histology MM 49; BCC 16 SK 12; BN 137 Other: 18 (SN, BD plus others) BCC: 16/232; 7% Exclusions Poor‐quality index test image 10 cases excluded NB for BCC: MM results not disaggregated
Lorentzen 2008 BCC MM Any Referred (selected on reference) (c)	WPC NR‐CS Specialist clinic Denmark 119/119	Patients referred to the specialist naevus clinic; compared classic dermoscopy to acrylic globe magnifer	Dermoscopy (Kenet risk stratification) Image‐based (blinded)	NR	Dermatologist (n = NR) Average	Histology MM 24; BCC 13 BN 69; Mild/moderate dysplasia 2; SK 9; Other 2 BCC: 13/119; 11% Exclusions: 1 dermatofibroma
Menzies 2000 BCC Any (MM‐excl) Referred (selected on reference) (u)	NC RP‐CCS Spec. clinic Aus; US Test set: NR/213 (Full sample 426)	PSL with dermoscopic images and histological diagnoses	Dermoscopy (Menzies for BCC (new)) Image‐based (none)	Absence of pigment network and ≥ 1 other char present; Dx	Dermatologist (assumed) (n = 2; experience NR) NR	Histology MM 71; BCC 71 BN 59; SK 5; Solar 3; DF 1; Other 3 BCC: 71/213; 33% NB: Included 142 BCCs, 142 invasive melanomas and 142 randomly‐sampled benign For BCC: 5 MM classed as FP
Navarrete Dechent 2016 BCC cSCC Any (MEL excl) Referred (selected on reference) (u)	NC RP‐CS Spec clinic US NR/457	Consecutively excised nonpigmented lesions; no discernible pigment on clinical or dermoscopic images.	Dermoscopy (Shiny white blotches and strands (new)) Image‐based (blinded)	≥ 1 char present	Dermatologist (assumed) and medical student (n = 2; experience NR) Consensus of 2	Histology MEL 21; BCC 287; cSCC 106 lichen planus–like keratosis 39; Naevus 4 BCC: 287/457; 63% cSCC: 106/457; 23% NB for BCC: 9 MM and 44 cSCC were counted as FP
Nori 2004 BCC Referred (selected on reference) (u)	WPC RP‐NR Secondary US; Spain 105 (VI) Full sample: 145/152	Biopsy confirmed BCC and convenience sample of non‐BCC with 'range of common diagnoses'; lesions with superior clinical image quality selected for VI	VI (no algorithm) Image based (blinded) (Also evaluates RCM)	Subjective impression: High/Med probability of BCC	Dermatologist (n = 2; experience NR) Single observer	Histology and Expert opinion* BCC 58 Benign 47 (Full sample includes 83 BCC; 4 SCC; 65 benign) BCC: 58/105; 55% NB: 15 lesions not biopsied because the clinical diagnosis was considered diagnostic (e.g.SK) cSCC results not disaggregated
Rosendahl 2011 BCC Any (MEL) Limited prior test (selected on reference) (u)	WPC‐algs R‐CS Aus. Primary 389/463	PSL submitted for histology from the primary‐care skin cancer practice of 1 author	1. VI (no algorithm) 2. Dermoscopy (pattern; chaos and clues)	1. Subjective impression 2. NR; both characteristics present	Dermatologist (n = 1) High experience (confirmed by author); Single observer	Histology MM 9; MiS 20 BCC 72; SCC 5 BN 217; BD 18; AK 14*; BNM 140 AK were considered malignant by study authors but not by review team BCC: 72/463; 16% Exclusions: 3 poor‐quality images excluded NB for BCC (VI): 3 MM were counted as FP; for BCC (Derm chaos/clues) 23 MM/MiS were counted as FPs; and for BCC (Pattern) 1 MM was counted as FP
Witkowski 2016 BCC cSCC Any (MEL excl) Equivocal (selected on reference) (u)	WPC RP‐CS Secondary Italy NR/260	Consecutive clinically equivocal ‘pink’ cutaneous lesions with absent pigmentation or containing < 10% pigment and absence of pigment network. All lesions were excised at first visit or follow‐up video dermoscopy control visit	Dermoscopy (No algorithm) Image based (blinded) (Also evaluates RCM)	NR	Dermatologist (assumed) (n = NR; experience NR) Single	Histology MEL 12; BCC 114; cSCC 13; Other malig 1 BN 47; SN 6; SL/SK/LPLK/AK 25; DF 18 Other 24 BCC: 114/260; 44% cSCC: 13/260; 5% NB for BCC: 1 MM and 1 cSCC were counted as FP
Zalaudek 2006 BCC Any (MEL) Referred (selected on reference) (u)	NC R‐CS Specialist clinic Italy NR/165	Random sample of excised, equivocal and nonequivocal, PSL and and non‐PSLs with melanin or haemoglobin pigmentation in all or part of the lesion.	Dermoscopy (3PCL) Image‐based (age, site, gender)	≥ 2 characteristics present	Mixed (n = 150; experience NR) Average result	Histology Full sample: MM 18; MiS 11 BCC: 18 79 BN; 26 SK; 8 vascular; 3 DF BCC: 18/150; 12% Exclusions: 15 used for training purposes NB for BCC: 7 MM were counted as FP
Footnotes: 3PCL ‐ three‐ point checklist; 7PCL ‐ seven‐point checklist; AK – actinic keratosis; BCC – basal cell carcinoma; BD – Bowen’s disease; BN – benign naevi; BPC – between person comparison (of tests); c ‐ clearly positioned on clinical pathway; CAD – computer‐assisted diagnosis; CCS – case control study; CS – case series; cSCC – cutaneous squamous cell carcinoma; DF – dermatofibroma; dx ‐ diagnosis; FP ‐ false positive; FU – follow‐up; KA ‐ keratoacanthoma; LPLK ‐ lichen planus‐like keratosis; LS – lentigo simplex; MEL: invasive melanoma or atypical intraepidermal melanocytic lesions; MiS – melanoma in situ (or lentigo maligna); MM – malignant (invasive) melanoma; NC – non comparative; NR – not reported; OCT ‐ optical coherence tomography; P – prospective; PLC – pigmented lesion clinic; PSL – pigmented skin lesion; R – retrospective; RCM – reflectance confocal microscopy; SK – seborrheic keratosis; SL ‐ solar lentigo; SN – Spitz naevi; TN ‐ true negative; u – unclear position on clinical pathway; WPC – within person comparison (of tests).

Appendix 9. Content of algorithms for BCC

Menzies algorithm for pigmented BCC Menzies 2000	Menzies (revised; pigmented and non‐pigmented BCC) Altamura 2010	Two‐step algorithm (Marghoob 2010); non‐pigmented BCC Markowitz 2015	3‐point checklist plus dermoscopic criteria (pigmented BCC) Amirnia 2016	Shiny White Structures (SWSs); non‐pigmented BCC Navarrete Dechent 2016
No pigment network (Negative feature absent) > 1 positive feature present 1. Spoke wheel areas (well‐circumscribed radial projections) 2. Large grey‐blue ovoid nests (well circumscribed, confluent or near confluent pigmented ovoid or elongated areas, larger than globules, not intimately connected to a pigmented tumor body 3. Arborizing telangiectasia (telangiectasia with distinct treelike branching) 4. Multiple grey‐blue globules (as opposed to multiple grey‐blue dots) 5. Maple leaflike areas (brown to grey‐blue discrete bulbous extensions forming leaflike pattern 6. Ulceration (absence of epidermis often associated with congealed blood; not due to recent trauma).	'Classic' BCC patterns for pigmented BCC (Menzies 2000) 1. ulceration, 2. multiple blue/grey globules, 3. leaflike areas, 4. large blue/grey ovoid nests, 5. spoke‐wheel areas, 6. arborizing telangiectasia Plus 'Non‐classic' patterns short fine superficial telangiectasia, multiple small erosions, concentric structures, multiple in‐focus blue/grey dots	Dermoscopic features consistent with BCC: arborized vessels, pink white shiny background, blue/grey ovoid nests, ash leaf pattern, dot‐globular‐like pattern, spoke wheel, and crystalline‐like structures	1. Asymmetry in colour or structure in one or two orthogonal axis asymmetric 2. Pigment network with irregular holes and thick lines atypical network 3. Any kind of blue or white colour Blue ‐ white structures Dermoscopic criteria of BCC tree‐like arteries blue‐grey points	SWSs were classified as 1. blotches (clods; discrete, small or large structure‐less areas); 2. strands (long thick or thin lines, randomly distributed or parallel, not orthogonally oriented); 3. rosettes (cluster of 4 white dots in a 4‐leaf clover–like arrangement); and 4. short white lines (crystalline structures and chrysalis; fine lines that intersect or are oriented orthogonally to each other) 5. non‐specified. All lesions also evaluated for Menzies 2000 criteria; ‘featureless’ lesions further evaluated for: short fine telangiectasias; multiple in‐focus, blue‐grey dots; multiple small erosions;and concentric structures
BCC ‐ basal cell carcinoma

Appendix 10. Forest plots for covariate investigations by prevalence and use of an algorithm

Figure 100; Figure 101

22.

Forest plot of tests: 1 BCC‐Visual Inspection (in‐person), 2 BCC‐Visual Inspection (image‐based).

23.

Forest plot of tests: 3 BCC‐VI+Dermoscopy (in‐person), 4 BCC‐Dermoscopy alone (image‐based).

Data

Presented below are all the data for all of the tests entered into the review.

Tests. Data tables by test.

Test	No. of studies	No. of participants
1 BCC‐Visual Inspection (in‐person)	8	7017
2 BCC‐Visual Inspection (image‐based)	4	853
3 BCC‐VI+Dermoscopy (in‐person)	7	4683
4 BCC‐Dermoscopy alone (image‐based)	9	2271
5 BCC‐VI ‐ no algorithm at any threshold (in‐person)	7	3645
6 BCC‐VI ‐ no algorithm at BCC possible (in‐person)	1	141
7 BCC‐VI ‐ ABCD at threshold NR (in‐person)	1	3372
8 BCC‐VI ‐ Schwartzberg algorithm (in‐person)	1	141
9 BCC‐VI ‐ no algorithm at any threshold (image‐based)	4	853
10 BCC‐VI ‐ no algorithm at BCC possible (image‐based)	1	105
11 BCC‐ VI+Dermoscopy no algorithm at NR (in‐person)	2	648
12 BCC‐VI+Dermoscopy pattern analysis_obs_dx (in‐person)	2	3628
13 BCC‐ VI+Dermoscopy 3 point at >= (in‐person)	1	61
14 BCC‐VI+Dermoscopy Two step_obs_dx (in‐person)	2	346
15 BCC‐Dermoscopy ‐ no algorithm at any threshold (image‐based)	2	313
16 BCC‐Dermoscopy ‐ pattern analysis at NR (image‐based)	2	582
17 BCC‐Dermoscopy ‐ Menzies for BCC(rev)_obsdx (image‐based)	1	300
18 BCC‐Dermoscopy ‐ Menzies for BCC(new) ‐ 1 char absent&>=1 other +ve (image‐based)	1	213
19 BCC‐Dermoscopy ‐ 3 point checklist at >= 2 (image‐based)	1	150
20 BCC‐Dermoscopy ‐ new SWS at >=1 (image‐based)	1	457
21 BCC‐Dermoscopy ‐ Chaos/clues (image‐based)	1	463
22 cSCC‐Visual inspection (in‐person)	2	2684
23 cSCC‐Dermoscopy alone (image‐based)	2	717
24 cSCC‐VI ‐ no algorithm at NR (in‐person)	2	2684
25 cSCC‐Dermoscopy ‐ no algorithm at NR (image‐based)	1	260
26 cSCC‐Dermoscopy ‐ SWS at >1 char (image‐based)	1	457
27 Any ‐Visual inspection (in‐person)	5	3618
28 Any ‐Visual inspection (image‐based)	2	517
29 Any ‐VI+Dermoscopy (in‐person)	2	277
30 Any‐Dermoscopy alone (image‐based)	6	1526
31 KER‐VI ‐ no algorithm at NR (in‐person)	4	3533
32 KER‐VI ‐ ABCD at NR (in‐person)	1	85
33 KER‐VI ‐ no algorithm at NR (image‐based)	2	517
34 KER‐ VI+Dermoscopy no algorithm at NR (in‐person)	1	200
35 KER‐VI+Dermoscopy ‐ 3 point at >=2 (in‐person)	1	77
36 KER‐Dermoscopy ‐ no algorithm at any threshold (image‐based)	3	393
37 KER‐Dermoscopy ‐ no algorithm at excise (image‐based)	1	260
38 KER‐ Dermoscopy ‐ pattern at NR (image‐based)	1	463
39 KER‐Dermoscopy‐ SWS (image‐based)	1	457
40 KER‐Dermoscopy ‐ Chaos/Clues (image‐based)	1	463
41 KER‐Dermoscopy ‐ Menzies for BCC(rev)_obsdx (image‐based)	1	213
42 BCC‐VI ‐ experience ‐ high (in‐person)	3	615
43 BCC‐VI ‐ experience ‐ mixed (in‐person)	2	2684
44 BCC‐VI ‐ experience ‐ NR (in‐person)	3	3718
45 BCC‐VI ‐ experience ‐ high (image‐based)	2	158
46 BCC‐VI ‐ experience ‐ mixed (image‐based)	1	232
47 BCC‐VI ‐ experience ‐ NR (image‐based)	1	463
48 BCC‐VI+Dermoscopy ‐ experience ‐ high (in‐person)	2	704
49 BCC‐VI+Dermsocopy ‐ experience ‐ NR (in‐person)	5	3979
50 BCC‐Dermoscopy ‐ experience ‐ high (image‐based)	3	428
51 BCC‐Dermoscopy ‐ experience ‐ mixed (image‐based)	1	150
52 BCC‐Dermoscopy ‐ experience ‐ trained (image‐based)	1	457
53 BCC‐Dermoscopy ‐ experience ‐ NR (image‐based)	4	1236
54 BCC‐VI ‐ qualification ‐ Consultant expert (in‐person)	4	668
55 BCC‐VI ‐ qualification ‐ Consultant (in‐person)	3	3719
56 BCC‐VI ‐ qualification ‐ Mixed (Secondary care) (in‐person)	2	2684
57 BCC‐VI ‐ qualification ‐ Consultant expert (image‐based)	1	463
58 BCC‐VI ‐ qualification ‐ Consultant (image‐based)	1	105
59 BCC‐VI+Dermoscopy ‐ qualification ‐ Consultant expert (in‐person)	3	1167
60 BCC‐VI+Dermoscopy ‐ qualification ‐ Consultant (in‐person)	4	3748
61 BCC‐Dermoscopy ‐ qualification ‐ Consultant expert (image‐based)	4	728
62 BCC‐Dermoscopy ‐ qualification ‐ Consultant (image‐based)	2	473
63 BCC‐Dermoscopy ‐ qualification ‐ Resident (image‐based)	1	457
64 BCC‐Dermoscopy ‐ qualification ‐ Mixed (dermoscopy trained) (image‐based)	1	150
65 cSCC‐VI ‐ experience ‐ mixed (in‐person)	1	2582
66 cSCC‐VI ‐ experience ‐ NR (in‐person)	1	102
67 cSCC‐Dermoscopy ‐ experience ‐ trained (image‐based)	1	457
68 cSCC‐Dermoscopy ‐ experience ‐ NR (image‐based)	1	260
73 KER‐VI ‐ experience ‐ high (in‐person)	1	769
74 KER‐VI ‐ experience ‐ mixed (in‐person)	1	2582
75 KER‐VI ‐ experience ‐ NR (in‐person)	3	267
76 KER‐VI ‐ experience ‐ high (image‐based)	1	54
77 KER‐VI ‐ experience ‐ NR (image‐based)	1	463
78 KER‐VI+Dermoscopy ‐ experience ‐ trained (in‐person)	1	77
80 KER‐VI+Dermoscopy ‐ experience ‐ NR (in‐person)	1	200
81 KER‐Dermoscopy ‐ experience ‐ high (image‐based)	1	53
82 KER‐Dermoscopy ‐ experience ‐ trained (image‐based)	1	457
83 KER‐Dermoscopy ‐ experience ‐ NR (image‐based)	4	1016

1. Test.

BCC‐Visual Inspection (in‐person).

2. Test.

BCC‐Visual Inspection (image‐based).

3. Test.

BCC‐VI+Dermoscopy (in‐person).

4. Test.

BCC‐Dermoscopy alone (image‐based).

5. Test.

BCC‐VI ‐ no algorithm at any threshold (in‐person).

6. Test.

BCC‐VI ‐ no algorithm at BCC possible (in‐person).

7. Test.

BCC‐VI ‐ ABCD at threshold NR (in‐person).

8. Test.

BCC‐VI ‐ Schwartzberg algorithm (in‐person).

9. Test.

BCC‐VI ‐ no algorithm at any threshold (image‐based).

10. Test.

BCC‐VI ‐ no algorithm at BCC possible (image‐based).

11. Test.

BCC‐ VI+Dermoscopy no algorithm at NR (in‐person).

12. Test.

BCC‐VI+Dermoscopy pattern analysis_obs_dx (in‐person).

13. Test.

BCC‐ VI+Dermoscopy 3 point at >= (in‐person).

14. Test.

BCC‐VI+Dermoscopy Two step_obs_dx (in‐person).

15. Test.

BCC‐Dermoscopy ‐ no algorithm at any threshold (image‐based).

16. Test.

BCC‐Dermoscopy ‐ pattern analysis at NR (image‐based).

17. Test.

BCC‐Dermoscopy ‐ Menzies for BCC(rev)_obsdx (image‐based).

18. Test.

BCC‐Dermoscopy ‐ Menzies for BCC(new) ‐ 1 char absent&>=1 other +ve (image‐based).

19. Test.

BCC‐Dermoscopy ‐ 3 point checklist at >= 2 (image‐based).

20. Test.

BCC‐Dermoscopy ‐ new SWS at >=1 (image‐based).

21. Test.

BCC‐Dermoscopy ‐ Chaos/clues (image‐based).

22. Test.

cSCC‐Visual inspection (in‐person).

23. Test.

cSCC‐Dermoscopy alone (image‐based).

24. Test.

cSCC‐VI ‐ no algorithm at NR (in‐person).

25. Test.

cSCC‐Dermoscopy ‐ no algorithm at NR (image‐based).

26. Test.

cSCC‐Dermoscopy ‐ SWS at >1 char (image‐based).

27. Test.

Any ‐Visual inspection (in‐person).

28. Test.

Any ‐Visual inspection (image‐based).

29. Test.

Any ‐VI+Dermoscopy (in‐person).

30. Test.

Any‐Dermoscopy alone (image‐based).

31. Test.

KER‐VI ‐ no algorithm at NR (in‐person).

32. Test.

KER‐VI ‐ ABCD at NR (in‐person).

33. Test.

KER‐VI ‐ no algorithm at NR (image‐based).

34. Test.

KER‐ VI+Dermoscopy no algorithm at NR (in‐person).

35. Test.

KER‐VI+Dermoscopy ‐ 3 point at >=2 (in‐person).

36. Test.

KER‐Dermoscopy ‐ no algorithm at any threshold (image‐based).

37. Test.

KER‐Dermoscopy ‐ no algorithm at excise (image‐based).

38. Test.

KER‐ Dermoscopy ‐ pattern at NR (image‐based).

39. Test.

KER‐Dermoscopy‐ SWS (image‐based).

40. Test.

KER‐Dermoscopy ‐ Chaos/Clues (image‐based).

41. Test.

KER‐Dermoscopy ‐ Menzies for BCC(rev)_obsdx (image‐based).

42. Test.

BCC‐VI ‐ experience ‐ high (in‐person).

43. Test.

BCC‐VI ‐ experience ‐ mixed (in‐person).

44. Test.

BCC‐VI ‐ experience ‐ NR (in‐person).

45. Test.

BCC‐VI ‐ experience ‐ high (image‐based).

46. Test.

BCC‐VI ‐ experience ‐ mixed (image‐based).

47. Test.

BCC‐VI ‐ experience ‐ NR (image‐based).

48. Test.

BCC‐VI+Dermoscopy ‐ experience ‐ high (in‐person).

49. Test.

BCC‐VI+Dermsocopy ‐ experience ‐ NR (in‐person).

50. Test.

BCC‐Dermoscopy ‐ experience ‐ high (image‐based).

51. Test.

BCC‐Dermoscopy ‐ experience ‐ mixed (image‐based).

52. Test.

BCC‐Dermoscopy ‐ experience ‐ trained (image‐based).

53. Test.

BCC‐Dermoscopy ‐ experience ‐ NR (image‐based).

54. Test.

BCC‐VI ‐ qualification ‐ Consultant expert (in‐person).

55. Test.

BCC‐VI ‐ qualification ‐ Consultant (in‐person).

56. Test.

BCC‐VI ‐ qualification ‐ Mixed (Secondary care) (in‐person).

57. Test.

BCC‐VI ‐ qualification ‐ Consultant expert (image‐based).

58. Test.

BCC‐VI ‐ qualification ‐ Consultant (image‐based).

59. Test.

BCC‐VI+Dermoscopy ‐ qualification ‐ Consultant expert (in‐person).

60. Test.

BCC‐VI+Dermoscopy ‐ qualification ‐ Consultant (in‐person).

61. Test.

BCC‐Dermoscopy ‐ qualification ‐ Consultant expert (image‐based).

62. Test.

BCC‐Dermoscopy ‐ qualification ‐ Consultant (image‐based).

63. Test.

BCC‐Dermoscopy ‐ qualification ‐ Resident (image‐based).

64. Test.

BCC‐Dermoscopy ‐ qualification ‐ Mixed (dermoscopy trained) (image‐based).

65. Test.

cSCC‐VI ‐ experience ‐ mixed (in‐person).

66. Test.

cSCC‐VI ‐ experience ‐ NR (in‐person).

67. Test.

cSCC‐Dermoscopy ‐ experience ‐ trained (image‐based).

68. Test.

cSCC‐Dermoscopy ‐ experience ‐ NR (image‐based).

73. Test.

KER‐VI ‐ experience ‐ high (in‐person).

74. Test.

KER‐VI ‐ experience ‐ mixed (in‐person).

75. Test.

KER‐VI ‐ experience ‐ NR (in‐person).

76. Test.

KER‐VI ‐ experience ‐ high (image‐based).

77. Test.

KER‐VI ‐ experience ‐ NR (image‐based).

78. Test.

KER‐VI+Dermoscopy ‐ experience ‐ trained (in‐person).

80. Test.

KER‐VI+Dermoscopy ‐ experience ‐ NR (in‐person).

81. Test.

KER‐Dermoscopy ‐ experience ‐ high (image‐based).

82. Test.

KER‐Dermoscopy ‐ experience ‐ trained (image‐based).

83. Test.

KER‐Dermoscopy ‐ experience ‐ NR (image‐based).

Characteristics of studies

Characteristics of included studies [ordered by study ID]

Altamura 2010.

Study characteristics
Patient sampling	Study design: Case control Data collection: Retrospective Period of data collection January 1991 ‐ May 2007 Country Italy, Australia and Austria Test set derived. BCC characteristics assessed on a random sample of BCC lesions; observer accuracy for diagnosis of BCC assessed on a separately‐derived random sample of 4 lesion types
Patient characteristics and setting	Inclusion criteria: Skin lesions randomly selected from digital image databases of all lesions excised; separately sampled BCCs, melanomas, 50 melanocytic naevi, and nonmelanocytic skin lesions Setting: Secondary; Departments of Dermatology of the University of L'Aquila. Specialist unit; tertiary referral centre of the Sydney Melanoma Diagnostic Center (Sydney, Australia) Prior testing: Unclear; all selected for excision Setting for prior testing: Unspecified Exclusion criteria: Poor‐quality images excluded (considered under Flow and Timing) Sample size (patients): Not reported Sample size (lesions): No. included: 300 Participant characteristics: Not reported for test set of images Lesion characteristics: Not reported in full for test set of images. BCC included 38 pigmented, 38 heavily pigmented, 37 nonpigmented, and 37 lightly pigmented); median Breslow thickness for melanomas 0.4 mm; range 0 ‐ 2.7 mm. Non‐BCC lesions reportedly had "a similar degree and distribution of pigmentation"
Index tests	Dermoscopy Modified version of Menzies algorithm for BCC (Menzies 2000) Method of diagnosis: Dermoscopic images Prior test data: No further information used; images were scored "without knowledge of any clinical data of the patients and lesions" Diagnostic threshold: Observer diagnosis of BCC. On diagnosis of a BCC, observer was asked to report the presence or absence of 'classic' and 'nonclassic' BCC dermatoscopic patterns as identified in the first phase of the study (assessment of 609 confirmed BCCs for global and local dermatoscopic features as described in Menzies 2000 and Menzies 1996a; 'classic' BCC patterns were defined as those associated with pigmented BCC (i.e. ulceration, multiple blue/grey globules, leaflike areas, large blue/grey ovoid nests, spoke‐wheel areas, and arborising telangiectasia), 'nonclassic' patterns were dermoscopic features "representing a possible variation on the theme of the (classic) patterns ... (i.e. short fine superficial telangiectasia, multiple small erosions, concentric structures, multiple in‐focus blue/gray dots)". Diagnosis based on: Single observer (n = 3) Observer qualifications: Likely dermatologists; described as "3 observers experienced in dermatoscopic evaluation". It is unclear whether the same observer participated in the first phase of the study Experience in practice: Assumed high "experienced in dermatoscopic evaluation" Experience with index test: Assumed high
Target condition and reference standard(s)	Reference standard: Histological diagnosis alone Details: None provided; states "blinded to the histopathologic diagnosis" Target condition (Final diagnoses): BCC: 150; melanoma (invasive): 40; melanoma (in situ): 10; cSCC: 2 Melanocytic naevi 50 (including 28 atypical, 9 Spitz/ Reed, 5 blue, 5 dermal, 3 compound); Nonmelanocytic naevi 50 (20 seborrhoeic keratosis, 12 AKs, 10 Dermatofibromas, 4 haemangiomas, 1 eccrine poroma, 1 viral wart)
Flow and timing	Participant exclusions: Poor‐quality index test image "large lesions present on the database but not completely comprised within the field of view were not included in the study" Index test to reference standard interval: Not described
Comparative
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Yes
Was a case‐control design avoided?	No
Did the study avoid inappropriate exclusions?	Unclear
Are the included patients and chosen study setting appropriate?	No
Did the study avoid including participants with multiple lesions?	Unclear
		High	High
DOMAIN 2: Index Test Dermoscopy (image based)
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Unclear
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	No
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	Unclear
Was the test interpretation carried out by an experienced examiner?	Yes
		Unclear	High
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	Yes
Were the reference standard results interpreted without knowledge of the results of the index tests?	Unclear
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Unclear
Were the reference standard results interpreted without knowledge of the referral diagnosis?	Unclear
		Low	Unclear
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Unclear
Did all patients receive the same reference standard?	Yes
Were all patients included in the analysis?	No
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?	Unclear
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?
		High

Amirnia 2016.

Study characteristics
Patient sampling	Study design: Case series Data collection: Unclear Period of data collection February 2012 ‐ February 2014 Country Iran
Patient characteristics and setting	Inclusion criteria: Randomly‐selected patients suspected of BCC or melanocytic naevi of the face, referred to dermatology clinic for excision or examination; all included lesions were excised Setting: Secondary (general dermatology) Prior testing: Selected for excision (no further detail) Setting for prior testing: NR Exclusion criteria: NR Sample size (patients): N eligible: 67; N included: 61 Sample size (lesions): N eligible: NR; N included: 61 Participant characteristics: Mean age: 49.5 (± 18.9; 24 ‐ 81). Male: 25 (41%) Lesion characteristics: Face (100%). mean lesion duration 6 years and 10 months (1 month to 20 years).
Index tests	Dermoscopy; 3‐point checklist Method of Diagnosis: In‐person diagnosis Prior test Clinical examination Diagnostic threshold: Presence of 2 or more criteria. Asymmetry in colour or structure in 1 or 2 orthogonal axis asymmetric; pigment network with irregular holes and thick lines atypical network; any kind of blue or white colour Diagnosis based on: Single observer (N NR) Observer qualifications: NR; assume dermatologist Experience in practice: NR Experience with index test: NR
Target condition and reference standard(s)	Reference standard: Histological diagnosis alone (biopsy) Target condition (Final diagnoses): BCC: 27; melanocytic naevi: 28; sebhorrheic keratosis:1; 1 reaction to foreign substance, 1 folliculitis associated with calcification, 1 abscess; 2 reported as "in situ carcinoma" but not further described
Flow and timing	Participant exclusions: NR Index test to reference standard interval: Not described
Comparative
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Yes
Was a case‐control design avoided?	Yes
Did the study avoid inappropriate exclusions?	Unclear
Are the included patients and chosen study setting appropriate?	No
Did the study avoid including participants with multiple lesions?	Yes
		Unclear	High
DOMAIN 2: Index Test Dermoscopy (in‐person)
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Yes
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	Yes
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	Yes
Was the test interpretation carried out by an experienced examiner?	Unclear
		Low	Unclear
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	Yes
Were the reference standard results interpreted without knowledge of the results of the index tests?	Yes
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Unclear
Were the reference standard results interpreted without knowledge of the referral diagnosis?	Unclear
		Low	Unclear
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Unclear
Did all patients receive the same reference standard?	Yes
Were all patients included in the analysis?	Yes
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?
		Unclear

Argenziano 2006.

Study characteristics
Patient sampling	Study design: Randomised controlled trial allocating primary‐care physicians to use either visual inspection alone or visual inspection plus dermoscopy (only excised lesions can be included for each arm) Data collection: Prospective Period of data collection May 2003 ‐ Sept 2004 Country Italy and Spain
Patient characteristics and setting	Inclusion criteria: Patients asking for screening or exhibiting 1 or more skin tumours as seen during routine physical examination (patient‐finding screening) were considered for inclusion; those undergoing excision were included in this review (i.e. those deemed sufficiently suspicious by the Expert evaluation). PCPs were invited to participate in the trial; only those who attended the training sessions and who then screened patients and referred them to the Pigmented Lesion Clinics were randomised Setting: Primary Prior testing: No prior testing Setting for prior testing: N/A Exclusion criteria: NR Sample size (patients): N eligible: 3271 patients screened; 1325 participants allocated to Naked Eye observation (VI) and 1197 participants allocated to dermoscopy observation; N included: 162 received histology after Expert evaluation at the PLC Sample size (lesions): 85 in VI arm and 77 in Dermoscopy arm underwent excision Participant characteristics: Based on full sample: mean age 40, range 2 ‐ 90 (VI group)/41, range 3 ‐ 94 (dermoscopy group). Male 498 (38%): VI group/451 (38%) dermoscopy Lesion characteristics NR
Index tests	Visual inspection (VI) ABCD (control arm of RCT comparing naked‐eye examination to naked eye plus dermoscopy) Method of diagnosis: In‐person diagnosis Prior test data: N/A in‐person diagnosis Diagnostic threshold: Qualitative NR; Described in Intro as: simple morphologic features summarised by the asymmetry, border irregularity, colour variegation, and diameter 5 mm (ABCD) Diagnosis based on: Average (N = 37) Observer qualifications: Primary care physicians Experience in practice: Not described Experience with index test: Not described Other detail: Pre‐randomisation all participating PCPs underwent training in ABCD rule for clinical diagnosis and 3‐point checklist for dermoscopy Dermoscopy 3‐point rule (intervention arm of RCT) Method of diagnosis: In person diagnosis Prior test data: N/A in‐person diagnosis Diagnostic threshold: ≥ 2 characteristics present (algorithm is based on the recognition of only 3 individual features: dermoscopic asymmetry (in colour or structure or both, not in shape), atypical network (pigmented network with thick lines and irregular distribution), and blue‐white structures (presence of any blue or white colour within the lesion). Each PCP in both groups examined the individual lesions and scored the patient outcome, as banal or suggestive of skin cancer Diagnosis based on: Average (N = 36) Observer qualifications: Primary care physicians Experience in practice: Not described Experience with dermoscopy: Not described Dermoscopy training: All PCPs received training (2‐hour session) on the clinical ABCD rule for diagnosis of melanoma, basic recognition of nonmelanoma skin cancers including BCC and SCC plus a 2‐hour session describing the dermoscopy 3‐point checklist
Target condition and reference standard(s)	Reference standard: Histological diagnosis alone All lesions considered suggestive of skin cancer at the PLC were excised and subsequently diagnosed histopathologically. Equivocal lesions by histopathologic examination were reviewed by a second independent pathologist and a final diagnosis made Target condition (Final diagnoses): Melanoma (in situ and invasive, or not reported): 12; BCC: 66; cSCC: 14 sebhorrheic keratosis: 13; melanocytic naevi 51; other: 6
Flow and timing	Excluded participants: Data can only be extracted for those with histology (i.e. patients considered to have lesions suggestive of skin cancer); remainder had expert diagnosis (not included in the final 2 x 2 data extracted) Time interval to reference test: NR Time interval between index test(s): N/A (RCT)
Comparative	RCT examining effect of making dermoscopy available to primary care practitioners
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Yes
Was a case‐control design avoided?	Yes
Did the study avoid inappropriate exclusions?	Unclear
Are the included patients and chosen study setting appropriate?	No
Did the study avoid including participants with multiple lesions?	Unclear
		Unclear	High
DOMAIN 2: Index Test Visual Inspection (in‐person)
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Yes
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	No
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	Yes
Was the test interpretation carried out by an experienced examiner?	Unclear
		Low	High
DOMAIN 2: Index Test Dermoscopy (in‐person)
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Yes
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	No
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	Yes
Was the test interpretation carried out by an experienced examiner?	Unclear
		Low	High
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	Yes
Were the reference standard results interpreted without knowledge of the results of the index tests?	Unclear
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Yes
Were the reference standard results interpreted without knowledge of the referral diagnosis?
		Low	Low
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Unclear
Did all patients receive the same reference standard?	Yes
Were all patients included in the analysis?	No
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?
		High

Carli 2002a.

Study characteristics
Patient sampling	Study design: Case series Data collection: Unclear. Visual inspection and in‐vivo dermoscopy diagnoses recorded at time of patient consultation; Ex vivo (image‐based) dermoscopy interpretation undertaken retrospectively Period of data collection June 1997 ‐ December 1998 Country Italy
Patient characteristics and setting	Inclusion criteria: Clinically equivocal or suspicious pigmented skin lesions subjected to excisional biopsy at the Institute of Dermatology Setting: Secondary (not further specified) Prior testing: Clinical or dermatoscopic suspicion, or both Setting for prior testing: Secondary Exclusion criteria: NR Sample size (patients): NR Sample size (lesions): 256 Participant characteristics: NR Lesion characteristics Of the cutaneous melanomas, 14 (25.9%) were in situ melanoma (Clark level I); 18 (33.3%) were invasive with < 0.75 mm thickness; 19 (35.3%) were of intermediate thickness (0.76 – 1.50 mm); and 3 (5.5%) were > 1.5 mm. The median thickness of invasive melanomas was 0.94 mm ± 0.5 (SD) (range 0.2 – 6)
Index tests	Visual inspection (VI) No algorithm Method of diagnosis: In‐person diagnosis Prior test data: Unclear Other test data: Clinical examination and in vivo dermoscopy were performed before excision by 2 trained dermatologists and diagnosis reached Diagnostic threshold: NR Diagnosis based on: Consensus (2 observers); final clinical diagnosis was based on agreement between the 2 observers. In case of disagreement, the opinion of a third observer (BG) was considered to be the judge for the diagnosis Observer qualifications: Dermatologist Experience in practice: High experience or ‘Expert’; described as “dermatologists with extensive experience in both clinical and dermoscopic diagnosis of pigmented skin lesions” Dermoscopy Pattern analysis Method of diagnosis: In‐person diagnosis and image‐based diagnosis. Clinical examination and in vivo dermoscopy were performed before excision by 2 trained dermatologists and diagnosis reached. Dermoscopic images were re‐analysed by the same 2 observers at the end of the inclusion period (December 1998), blind to the previous clinical and histological diagnoses Prior test data: N/A for in person; For image‐based: slides of dermoscopic images were evaluated using a viewer that made it impossible to analyse the clinical features of the lesion; both observers had access to clinical information, including the age of the participant, the site of the lesion, the history of change over time as reported by the participant at the time of in vivo examination Diagnostic threshold: Dermoscopic diagnosis was based on the ELM pattern analysis criteria, using the same diagnostic categories used for clinical diagnosis; characteristics investigated included pigment network, pigmentation, hypopigmentation, brown globules, black dots, pseudopods, radial streaming, grey‐blue veil, atypical vascular pattern Test observers as described for Visual Inspection (above)
Target condition and reference standard(s)	Reference standard: Histological diagnosis alone Target condition (final diagnoses): Melanoma (invasive): 40; Melanoma (in situ): 14; BCC: 5; Sebhorrheic keratosis: 4; Common melanocytic naevi: 90; Melanocytic naevi: 78; Blue naevi: 9; Spitz reed naevi: 16
Flow and timing	Excluded participants: NR Time interval to reference test: NR
Comparative	In person clinical examination and dermoscopy Time interval between index test(s): the interval between the time in‐vivo dermoscopy and re‐evaluation of dermoscopic images was reported as 1 year
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Yes
Was a case‐control design avoided?	Yes
Did the study avoid inappropriate exclusions?	Unclear
Are the included patients and chosen study setting appropriate?	No
Did the study avoid including participants with multiple lesions?	Unclear
		Unclear	High
DOMAIN 2: Index Test Visual Inspection (in‐person)
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Unclear
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	No
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	Yes
Was the test interpretation carried out by an experienced examiner?	Yes
		Unclear	High
DOMAIN 2: Index Test Dermoscopy (in‐person)
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Yes
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	No
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	Yes
Was the test interpretation carried out by an experienced examiner?	Yes
		Low	High
DOMAIN 2: Index Test Dermoscopy (image based)
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Yes
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	No
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	Yes
Was the test interpretation carried out by an experienced examiner?	Yes
		Low	High
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	Yes
Were the reference standard results interpreted without knowledge of the results of the index tests?	Unclear
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Unclear
Were the reference standard results interpreted without knowledge of the referral diagnosis?	Unclear
		Low	Unclear
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Unclear
Did all patients receive the same reference standard?	Yes
Were all patients included in the analysis?	Yes
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?
		Unclear

Carli 2002b.

Study characteristics
Patient sampling	Study design: Case series Data collection: NR Period of data collection NR Country Italy
Patient characteristics and setting	Inclusion criteria: Clinically‐suspicious or equivocal pigmented skin lesions undergoing excision for diagnostic purposes; only lesions with a diameter of 14 mm or less were included Setting: Secondary (general dermatology) Prior testing: Clinical suspicion of malignancy without dermatoscopic suspicion Setting for prior testing: Secondary (general dermatology) Exclusion criteria: NR Sample size (patients): N included: NR Sample size (lesions): N included: 57 Participant characteristics: NR Lesion characteristics: Thickness ≤ 1mm: 11 cases (5 in situ, 6 invasive); All ≤ 14 mm diameter
Index tests	Visual inspection (VI) No algorithm Method of diagnosis: Clinical photographs; Fixed‐focus distance of 10 cm; images observed using a viewer in 2 separate diagnostic sessions Prior test data: No further information used; contact (dermoscopic) images viewed first and then distant images (clinical), without knowing the classification of the contact image of the individual lesions Diagnostic threshold: NR Diagnosis based on: Consensus (2 observers); N = 2 Observer qualifications: Dermatologist Experience in practice: High experience or ‘Expert’; states "with experience in the field of PSL" Experience with dermoscopy: High experience/‘Expert’ users; "experienced in the field of PSLs" Other detail: Used an AF micro Nikkor 60 lens objective mounted on a Nikon f50 camera, with a fixed‐focus distance of 10 cm Dermoscopy No algorithm Method of diagnosis: Dermoscopic images Prior test data: No further information used; contact (dermoscopic) images viewed first and then distant images (clinical), without knowing the classification of the contact image of the individual lesions Diagnostic threshold: NR Test observers As described for Visual Inspection (above) Any other detail Dermaphot device placed directly on the lesion without previous application of oil; only lesions with a diameter of 14 mm or less were included in the study. The image has an automatic, original magnification of x 10
Target condition and reference standard(s)	Reference standard: Histological diagnosis alone (not further described) Target condition (Final diagnoses): Melanoma (invasive): 6; melanoma (in situ): 5; BCC: 10 'Benign' diagnoses: 36
Flow and timing	Excluded participants: No exclusions reported Time interval to reference test: Photographic procedures performed consecutively prior to surgery
Comparative	Photographic procedures performed consecutively prior to surgery
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Unclear
Was a case‐control design avoided?	Yes
Did the study avoid inappropriate exclusions?	Unclear
Are the included patients and chosen study setting appropriate?	No
Did the study avoid including participants with multiple lesions?	Unclear
		Unclear	High
DOMAIN 2: Index Test Visual inspection (image based)
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Unclear
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	No
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	No
Was the test interpretation carried out by an experienced examiner?	Yes
		Unclear	High
DOMAIN 2: Index Test Dermoscopy (image based)
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Unclear
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	No
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	No
Was the test interpretation carried out by an experienced examiner?	Yes
		Unclear	High
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	Yes
Were the reference standard results interpreted without knowledge of the results of the index tests?	Yes
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Unclear
Were the reference standard results interpreted without knowledge of the referral diagnosis?	Unclear
		Low	Unclear
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Yes
Did all patients receive the same reference standard?	Yes
Were all patients included in the analysis?	Yes
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?
		Low

Chang 2013.

Study characteristics
Patient sampling	Study design: Case series Data collection: Retrospective Period of data collection: Jan 2006 ‐ Jul 2009 Country: Taiwan
Patient characteristics and setting	Inclusion criteria: Potentially malignant biopsied or excised skin lesions (non‐tumour specimens excluded) Setting: Secondary (general dermatology) Prior testing: Selected for excision (no further detail) Setting for prior testing: Secondary (general dermatology) Exclusion criteria: Prior surgery; image mis‐registered or poor‐quality images (unfocused or containing a motion artefact) (considered under Flow and Timing) Sample size (patients): N eligible: 3964; N included: 676 Sample size (lesions): N eligible: 4192; N included: 769 Participant characteristics: Mean age: 47.6 (SD 21.0); Male: 296; 43.8% Lesion characteristics: NR
Index tests	Visual inspection (VI) No algorithm Method of diagnosis: In‐person diagnosis Prior test data: N/A in‐person diagnosis Diagnostic threshold: NR; clinicians’ impressions prior to biopsy were classified as ‘‘benign’’, ‘‘malignant’’, or ‘‘indeterminate’’. When the clinicians were not confident enough to make a definite benign or malignant diagnosis, the clinical impression was considered as ‘‘indeterminate’’ data extracted for malignant vs rest and malignant/indeterminate vs rest Diagnosis based on: Single observer; board‐certified staff dermatologists from institute; N = 25 Observer qualifications: Dermatologist Experience in practice: Board certified Experience with index test: High
Target condition and reference standard(s)	Reference standard: Histology (not further described) Target condition (Final diagnoses): Melanoma (invasive): 4; melanoma (in situ): 4; BCC: 110; cSCC: 20 'Benign' diagnoses: 595
Flow and timing	Excluded participants: Mis‐registered or poor‐quality images (unfocused or containing a motion artefact) as a study inclusion criterion Time interval to reference test: Not described
Comparative
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Yes
Was a case‐control design avoided?	Yes
Did the study avoid inappropriate exclusions?	Yes
Are the included patients and chosen study setting appropriate?	No
Did the study avoid including participants with multiple lesions?	No
		Low	High
DOMAIN 2: Index Test Visual Inspection (in‐person)
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Unclear
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	Yes
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	No
Was the test interpretation carried out by an experienced examiner?	Yes
		Unclear	High
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	Yes
Were the reference standard results interpreted without knowledge of the results of the index tests?	Unclear
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Unclear
Were the reference standard results interpreted without knowledge of the referral diagnosis?
		Low	Unclear
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Unclear
Did all patients receive the same reference standard?	Yes
Were all patients included in the analysis?	No
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?
		High

Cooper 2002.

Study characteristics
Patient sampling	Study design: Case series Data collection: Prospective Period of data collection May 2000 ‐ September 2000 Country UK
Patient characteristics and setting	Inclusion criteria: Patients attending the open‐access dermatology renal transplant clinic with lesions suspicious for malignancy or premalignancy and booked for biopsy Setting: Specialist unit; dermatology renal transplant clinic Prior testing: Clinical suspicion Setting for prior testing: Specialist unit Exclusion criteria: NR Sample size (patients): N eligible: 70; N included: NR Sample size (lesions): N eligible: 125; N included: 102 Participant characteristics: Mean age: 60; Male: 75% Lesion characteristics Head/neck: 43; 34.4%; Limbs: 21; 16.8%; 3 genitals; 2.4%
Index tests	Visual inspection (VI) No algorithm Method of diagnosis: In‐person diagnosis Prior test data: N/A in‐person diagnosis Diagnostic threshold: Observer provisional diagnosis Diagnosis based on: Single observer (N = 2) Observer qualifications: Consultant dermatologist and a registrar Experience in practice: Not described Experience with index test: Not described
Target condition and reference standard(s)	Reference standard: Histological diagnosis alone (biopsy, no further details) Target condition (Final diagnoses): BCC: 12; cSCC: 23 (incl 2 keratoacanthoma); Bowen's disease 19; viral warts 7; solar keratoses 16; other 25
Flow and timing	Participant exclusions: 23 lesions did not undergo biopsy; 11 resolved prior to biopsy, 6 patients died (10 lesions) and 2 patients failed to attend (2 lesions). No diagnosis was made in a further 3 samples Index test to reference standard interval: Not described
Comparative
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Yes
Was a case‐control design avoided?	Yes
Did the study avoid inappropriate exclusions?	Unclear
Are the included patients and chosen study setting appropriate?	Unclear
Did the study avoid including participants with multiple lesions?	No
		Unclear	High
DOMAIN 2: Index Test Visual Inspection (in‐person)
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Unclear
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	Yes
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	No
Was the test interpretation carried out by an experienced examiner?	Unclear
		Unclear	High
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	Yes
Were the reference standard results interpreted without knowledge of the results of the index tests?	Unclear
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Unclear
Were the reference standard results interpreted without knowledge of the referral diagnosis?
		Low	Unclear
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Unclear
Did all patients receive the same reference standard?	Yes
Were all patients included in the analysis?	No
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?
		High

Durdu 2011.

Study characteristics
Patient sampling	Study design: Case series Data collection: Prospective Period of data collection Jan 2006 ‐ January 2009 Country Turkey
Patient characteristics and setting	Inclusion criteria: Pigmented skin lesions that could not be diagnosed with only dermatologic physical examination Setting: Secondary (general dermatology) Prior testing: Clinical examination and dermoscopy Setting for prior testing: Secondary (general dermatology) Exclusion criteria: None reported Sample size (patients): N included: 176 Sample size (lesions): N included: 200 Participant characteristics: Mean age: 48 (4 ‐ 85). Male: 64; 36.4% Lesion characteristics: 9% nodulo‐ulcerative, 56% papular, 17% macular, 10% nodular, 8% plaque
Index tests	Dermoscopy: No algorithm Method of diagnosis: In‐person diagnosis Prior test data: Clinical examination Diagnostic threshold: 2‐step process: step 1 melanocytic and non‐melanocytic were differentiated (Braun 2005; Zalaudek 2008); step 2 ABCD applied to melanocytic lesions for diagnosis of melanoma only (threshold > 5.45). Previously reviewed dermoscopic characteristics used to diagnose non‐melanocytic lesions Diagnosis based on: Single observer; N = 2; 1 for dermoscopy diagnosis and 1 for Tzanck smear Observer qualifications: Dermatologist Experience in practice: Not described Experience with dermoscopy: Not described
Target condition and reference standard(s)	Reference standard: Histological diagnosis alone (excisional biopsies (N = 166) or punch biopsy (N = 34) Details: "Biopsy specimens were stained with hematoxylin and eosin. Immunohistochemical (anti‐S‐100 and human melanoma black [HMB]‐45) and histochemical (Fontana‐Masson) stains were also applied, if necessary"; interpretation by a 'pathologist' Target condition (Final diagnoses): Melanoma (in situ and invasive, or not reported): 10; BCC: 34; 1 pigmented mammary Paget disease; 1 pigmented metastatic mammary carcinoma Sebhorrheic keratosis: 24; Benign melanocytic naevus: 100; Dermatofibroma 12; Warts 16; Dirt 1; hereditary hemorrhagic telangiectasia 1
Flow and timing	Participant exclusions: NR Time interval to reference test: Appears consecutive. Following dermoscopic examination and cytology "either a punch or an excisional biopsy specimen was taken from the lesions and was examined histopathologically"
Comparative
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Unclear
Was a case‐control design avoided?	Yes
Did the study avoid inappropriate exclusions?	Unclear
Are the included patients and chosen study setting appropriate?	Yes
Did the study avoid including participants with multiple lesions?	No
		Unclear	High
DOMAIN 2: Index Test Dermoscopy (in‐person)
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Yes
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	Yes
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	Yes
Was the test interpretation carried out by an experienced examiner?	Unclear
		Low	Unclear
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	Yes
Were the reference standard results interpreted without knowledge of the results of the index tests?	Unclear
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Unclear
Were the reference standard results interpreted without knowledge of the referral diagnosis?	Unclear
		Low	Unclear
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Yes
Did all patients receive the same reference standard?	Yes
Were all patients included in the analysis?	Yes
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?
		Low

Ek 2005.

Study characteristics
Patient sampling	Study design: Case series Data collection: Prospective Period of data collection January 2001 ‐ December 2002 Country Australia
Patient characteristics and setting	Inclusion criteria: Lesions excised at tertiary referral centre for the management of cancers; only those lesions in which malignancy could not be excluded were included Setting: Specialist unit (skin cancer/pigmented lesions clinic) Prior testing: Selected for excision (no further detail) Setting for prior testing: Specialist unit (skin cancer/pigmented lesions clinic) Exclusion criteria: Punch, shave or incisional biopsies and palliative excisions. Equivocal pathology report (N = 56). Sample size (patients): N eligible: 1302; N included: 1223 Sample size (lesions): N eligible: 2678; N included: 2582 Participant characteristics: Mean age: 73.6 (16 – 102). Male: 784 (64.1%); History of melanoma/skin cancer (%) 224; 8.7% recurrent lesions Lesion characteristics: Head/neck: 61%; Trunk: 14.4%; Limbs: 24.6%
Index tests	Visual inspection (VI) No algorithm Method of diagnosis: In‐person diagnosis Prior test data: N/A in person diagnosis Diagnostic threshold: NR pre‐operative diagnosis Diagnosis based on: Unclear; likely single (N = 5) Observer qualifications: 3 consultants, a plastic surgery trainee and a clinical assistant Experience in practice: Mixed (low and high experience combined); Plastic surgery trainee usually 1st year, on 6‐month rotation; clinical assistant described as having “many years of experience” Other detail: Some results are presented for consultant, senior registrar and registrar but underlying participant numbers are not provided per observer to allow separate 2 x 2 estimation. The Discussion does describe the “six MM misdiagnosed as benign … as .. assessed by non‐consultants”
Target condition and reference standard(s)	Reference standard: Histological diagnosis alone Target condition (Final diagnoses): Melanoma (in situ and invasive, or not reported): 23; BCC: 1214; cSCC: 517 'Benign' diagnoses: 188 (7.3%) SCC in situ (Bowen’s disease), 330 (12.8%) solar keratoses, 63 (2.4%) seborrhoeic keratoses, 247 (9.6%) were other benign lesions
Flow and timing	Excluded participants: Lesions with incomplete or incorrectly entered pro formas were excluded (N = 40) Index to reference interval: Consecutive; used pre‐operative clinical diagnosis of lesions undergoing biopsy
Comparative
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Yes
Was a case‐control design avoided?	Yes
Did the study avoid inappropriate exclusions?	No
Are the included patients and chosen study setting appropriate?	Unclear
Did the study avoid including participants with multiple lesions?	No
		High	High
DOMAIN 2: Index Test Visual Inspection (in‐person)
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Unclear
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	Yes
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	No
Was the test interpretation carried out by an experienced examiner?	Unclear
		Unclear	High
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	Yes
Were the reference standard results interpreted without knowledge of the results of the index tests?	Unclear
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Unclear
Were the reference standard results interpreted without knowledge of the referral diagnosis?
		Low	Unclear
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Yes
Did all patients receive the same reference standard?	Yes
Were all patients included in the analysis?	No
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?
		High

Gokdemir 2011.

Study characteristics
Patient sampling	Study design: Case series Data collection: NR Period of data collection: 2005 ‐ 2009 Country: Turkey
Patient characteristics and setting	Inclusion criteria: Patients with melanocytic and non‐melanocytic skin lesions excised due to dermoscopic suspicion of malignancy or dysplasia Setting: Secondary (general dermatology) Prior testing: NR Setting for prior testing: Unspecified Exclusion criteria: NR Sample size (patients): N eligible: 1264; N included: 362 Sample size (lesions): N included: 449 Participant characteristics: Mean age 40.3 (± 1.08), range 1 ‐ 89; Male: 160; 44.2% Lesion characteristics: NR
Index tests	Dermoscopy No algorithm Method of diagnosis: Unclear; appears to be in‐person diagnosis Prior test data: Clinical examination Diagnostic threshold: Not reported; diagnosis of melanoma Diagnosis based on: Unclear (N NR) Observer qualifications: Dermatologist Experience in practice: Not described Experience with dermoscopy: High experience ‐ at least 2 years experience with Molemax II
Target condition and reference standard(s)	Reference standard: Histological diagnosis alone; not further described Target condition (Final diagnoses): Melanoma (in situ and invasive, or not reported): 13; BCC: 45 Benign: Not described
Flow and timing	Participant exclusions: None reported Index test to reference standard interval: Not reported
Comparative
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Unclear
Was a case‐control design avoided?	Yes
Did the study avoid inappropriate exclusions?	Unclear
Are the included patients and chosen study setting appropriate?	No
Did the study avoid including participants with multiple lesions?	No
		Unclear	High
DOMAIN 2: Index Test Dermoscopy (in‐person)
Were the index test results interpreted without knowledge of the results of the reference standard?	Unclear
If a threshold was used, was it pre‐specified?	Unclear
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	Unclear
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	No
Was the test interpretation carried out by an experienced examiner?	Yes
		Unclear	High
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	Yes
Were the reference standard results interpreted without knowledge of the results of the index tests?	Unclear
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Unclear
Were the reference standard results interpreted without knowledge of the referral diagnosis?	Unclear
		Low	Unclear
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Unclear
Did all patients receive the same reference standard?	Yes
Were all patients included in the analysis?	Yes
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?
		Unclear

Hacioglu 2013.

Study characteristics
Patient sampling	Study design: Case series Data collection: Unclear; diagnoses recorded at initial consultation but unclear whether the study was prospective in design. Also report prospective interpretation of previously‐acquired images (SIAscopy and dermoscopy) Period of data collection January 2009 ‐ January 2010 Country Turkey
Patient characteristics and setting	Inclusion criteria: Patients with skin lesions < 12 mm in diameter, suspicious for malignancy; only excised lesions included Setting: Secondary (general dermatology) Prior testing: Selected for excision Setting for prior testing: Unspecified Exclusion criteria: lesion size > 12 mm; lesions with a crusted or rough surface Sample size (patients): N included: 76 Sample size (lesions): N included: 80 Participant characteristics: Mean age: 57.6 (SD 15.48: range 23 ‐ 84). Male: 45 (52%) Lesion characteristics: NR
Index tests	Visual inspection (VI): No algorithm Method of diagnosis: In person; "clinical diagnosis based on the patient's history and dermatological findings." NB: unclear whether dermoscopy was used to inform initial diagnosis; dermoscopy use not described but dermoscopic images later evaluated Prior test data: N/A in‐person diagnosis Diagnostic threshold: Observer diagnosis Diagnosis based on: Single observer (N = 3) Observer qualifications: NR; likely dermatologist Experience in practice: Not described; 3 investigators ‐ 1 made preliminary clinic diagnosis and evaluated Siascope images 8 months later; second investigator evaluated all Siascope images; a third investigator evaluated dermoscopic images Experience with index test: Not described Dermoscopy: No algorithm Method of diagnosis: Dermoscopic images Prior test data: No further information used; "a third investigator (EBB), also blinded to the previous diagnoses, evaluated all the lesions using dermatoscopic images only." Diagnostic threshold: Observer diagnosis Observers: As described above.
Target condition and reference standard(s)	Reference standard: Histological diagnosis alone Details: Skin biopsies (3 or 4 mm in size) Target condition (Final diagnoses): BCC: 24; melanoma (in situ and invasive, or not reported): cSCC 3; Basosquamous cancer 2; sebhorrhoeic keratosis: 19; actinic keratosis 8; intradermal naevus 4; dermatofibroma 3; keratoacanthoma 2; Other 12 ‐ including: epidermal proliferation, pseudoepithelial hyperplasia, solar degeneration, lichen simplex chronicus, compound naevus, dysplastic naevus, prurigo nodularis, chronic inflammatory granulation, dysplastic junctional naevus
Flow and timing	Participant exclusions: NR Index test to reference standard interval: Appears consecutive; "Images ... were obtained ... and skin biopsies ... were taken"
Comparative	3. Time interval between index test(s): 8 months between visual and SIAscopetime between visual/SIAscope and dermatoscopy not reported
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Unclear
Was a case‐control design avoided?	Yes
Did the study avoid inappropriate exclusions?	No
Are the included patients and chosen study setting appropriate?	No
Did the study avoid including participants with multiple lesions?	Yes
		High	High
DOMAIN 2: Index Test Visual Inspection (in‐person)
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Unclear
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	Yes
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	No
Was the test interpretation carried out by an experienced examiner?	Unclear
		Unclear	High
DOMAIN 2: Index Test Dermoscopy (image based)
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Unclear
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	No
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	No
Was the test interpretation carried out by an experienced examiner?	Unclear
		Unclear	High
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	Yes
Were the reference standard results interpreted without knowledge of the results of the index tests?	Unclear
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Unclear
Were the reference standard results interpreted without knowledge of the referral diagnosis?	Unclear
		Low	Unclear
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Yes
Did all patients receive the same reference standard?	Yes
Were all patients included in the analysis?	Yes
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?
		Low

Lorentzen 1999.

Study characteristics
Patient sampling	Study design: Case series Data collection: Prospective Period of data collection 1994 ‐ 1997 Country Denmark
Patient characteristics and setting	Inclusion criteria: Patients with lesions suspicious for CMM referred to outpatients clinic; only excised included Setting: NR Prior testing: Clinical suspicion of malignancy without dermatoscopic suspicion Setting for prior testing: NR Exclusion criteria: Poor‐quality index test image (considered under flow/timing) Sample size (patients): N eligible: 242; N included: 232 Sample size (lesions): N eligible: 242; N included: 232 NB: Not all cases were assessed by all observers; 2 x 2 are based on presented sensitivity and specificity estimates for full dataset of lesions; "the dermatoscopy experts assessed almost all cases (98 ± 100%), whereas the non‐expert group completed fewer assessments, from 76 to 98%". Participant characteristics: NR Lesion characteristics: NR
Index tests	Visual inspection (VI) No algorithm Method of diagnosis: Clinical photographs Prior test data: No further information used; no option to change clinical diagnosis after viewing dermoscopic image Other test data: Dermoscopic images presented to observer subsequent to diagnosis using clinical images alone; clinical images presented before dermoscopic images Diagnostic threshold: NR; clinical diagnosis Diagnosis based on: Average; N = 9 Observer qualifications: Dermatologist Experience in practice: High; moderate; mixed (average reported); 4 'experienced dermatologists' (4 ‐ 5 years daily experience) & 5 'non‐expert dermatology residents' (1 ‐ 2 years interest and formal training in dermatoscopy Experience with index test: High; moderate; mixed
Target condition and reference standard(s)	Reference standard: Histological diagnosis alone Details: a co‐author from Dept of Pathology "re‐evaluated all cases to confirm the pathology diagnosis, which was used as the gold standard in this study" Target condition (Final diagnoses): Melanoma (invasive): 49 'malignant melanoma'; BCC: 16; sebhorrheic keratosis: 12; benign naevus: 137 (pigmented naevi = 116; blue naevi = 16; atypical naevi = 5); Other: 18 (Spitz naevi, Bowen's disease, sarcoid, naevus spilus, hemangioma, and others)
Flow and timing	Excluded participants: 10 cases were "considered unfit for evaluation" due to poor‐quality image Reference interval: "biopsy specimens...were obtained after the clinical and dermatoscopic photographs had been performed"
Comparative	tbc
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Unclear
Was a case‐control design avoided?	Yes
Did the study avoid inappropriate exclusions?	Yes
Are the included patients and chosen study setting appropriate?	No
Did the study avoid including participants with multiple lesions?	Unclear
		Unclear	High
DOMAIN 2: Index Test Visual inspection (image based)
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Unclear
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	No
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	No
Was the test interpretation carried out by an experienced examiner?	Yes
		Unclear	High
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	Yes
Were the reference standard results interpreted without knowledge of the results of the index tests?	Unclear
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Unclear
Were the reference standard results interpreted without knowledge of the referral diagnosis?	Unclear
		Low	Unclear
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Yes
Did all patients receive the same reference standard?	Yes
Were all patients included in the analysis?	No
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?
		High

Lorentzen 2008.

Study characteristics
Patient sampling	Study design: Case series Data collection: NR Period of data collection: NR Country: Denmark
Patient characteristics and setting	Inclusion criteria: Patients referred to the specialist naevus clinic for lesion excision Setting: Specialist unit (skin cancer/pigmented lesions clinic) Prior testing: NR Setting for prior testing: NR Exclusion criteria: Not specified Sample size (patients): N eligible: 120; N included: 119 Sample size (lesions): N included: 119 Participant characteristics: NR Lesion characteristics: NR
Index tests	Dermoscopy: Mixed/no algorithm; describes using "the risk stratification and pattern analysis procedure as described by Kenet 2001 and Lorentzen 2000". Method of diagnosis: Dermoscopic images; compared accuracy using standard dermoscopy images (Dermaphot) and images obtained using a globe magnifier. Slides were randomised and evaluated on 2 different occasions with 3‐week intervals Prior test data: No further information used Diagnostic threshold: Observer correct diagnosis of each lesion type Diagnosis based on: Unclear (assumed average) (N NR) Observer qualifications: Dermatologist Experience in practice: High; "dermatologists who have performed dermatoscopy for 5–10 years, published scientific papers on dermatoscopy and carried out pre‐ and post specialist training in dermatoscopy" Experience with dermoscopy: High
Target condition and reference standard(s)	Reference standard: Histological diagnosis alone Details: used haematoxylin‐eosin staining as well as histochemistry performed using S‐100 and HMB‐45 on suspect melanoma lesions Target condition (Final diagnoses): Melanoma (invasive): 24; BCC: 13; mild/moderate dysplasia: 2; sebhorrheic keratosis: 9; haemangioma: 2; naevus pigmentosus: 69
Flow and timing	Excluded participants: 1 dermatofibroma excluded Time interval to reference test: Not described
Comparative
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Unclear
Was a case‐control design avoided?	Yes
Did the study avoid inappropriate exclusions?	Unclear
Are the included patients and chosen study setting appropriate?	No
Did the study avoid including participants with multiple lesions?	Yes
		Unclear	High
DOMAIN 2: Index Test Dermoscopy (image based)
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Yes
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	No
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	No
Was the test interpretation carried out by an experienced examiner?	Yes
		Low	High
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	Yes
Were the reference standard results interpreted without knowledge of the results of the index tests?	Unclear
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Unclear
Were the reference standard results interpreted without knowledge of the referral diagnosis?	Unclear
		Low	Unclear
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Unclear
Did all patients receive the same reference standard?	Yes
Were all patients included in the analysis?	No
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?
		High

Markowitz 2015.

Study characteristics
Patient sampling	Study design: Case series Data collection: Prospective Period of data collection: NR Country: USA
Patient characteristics and setting	Inclusion criteria: Consecutive patients with at least 1 clinically‐challenging pink lesion on the head or neck that was suspicious for BCC and was therefore to be biopsied to rule BCC in or out; all eligible for Mohs surgery. 'Clinically‐challenging' defined as lesions that did not have the usual characteristics of BCC, such as ulceration, bleeding, crusting, isolated pink scaly patches, or pearly papules Setting: Secondary (general dermatology) Prior testing: Clinical suspicion of malignancy without dermatoscopic suspicion Setting for prior testing: Secondary (general dermatology) Exclusion criteria: Previous history of skin cancer/prior treatment at site; > 3 lesions per participant Sample size (patients): N included: 100 Sample size (lesions): N included: 115 Participant characteristics: NR Lesion characteristics: NR
Index tests	Visual inspection (VI) No algorithm Method of diagnosis: In‐person diagnosis Prior test data: N/A in‐person diagnosis Diagnostic threshold: Observer diagnosis of possible BCC; "lesions were diagnosed based on the patient’s clinical history of a nonhealing area of concern or the clinician’s inability to rule out BCC" Diagnosis based on: Unclear; appears that diagnoses made in clinic after acquisition of each type of image Number of examiners Not specified Observer qualifications: Not described; likely dermatologist Experience in practice: Not described Experience with index test: Not described Dermoscopy: 2‐step algorithm Method of diagnosis: In‐person diagnosis; images also taken but diagnosis made in person Prior test data: Clinical examination; diagnoses made after each step in the clinical process Diagnostic threshold: Observer diagnosis of possible BCC; 2‐step algorithm described as similar to Marghoob 2010 and Malvehy 2002. Lesions inspected for dermoscopic features consistent with BCC ... "including arborized vessels, pink white shiny background, blue/grey ovoid nests, ash leaf pattern, dot‐globular‐like pattern, spoke wheel, and crystalline‐like structures" Test observers: As described for Visual Inspection (above)
Target condition and reference standard(s)	Reference standard: Histological diagnosis alone Details: A biopsy was taken and the final diagnosis and lesion depth based on histopathology Target condition (Final diagnoses): BCC: 70; 'Benign' diagnoses: 45
Flow and timing	Participant exclusions: NR Index test to reference standard interval: Consecutive; After "the patient was returned for standard‐of‐care treatment. A biopsy was taken"
Comparative	Time interval between index test(s): consecutive
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Yes
Was a case‐control design avoided?	Yes
Did the study avoid inappropriate exclusions?	Yes
Are the included patients and chosen study setting appropriate?	No
Did the study avoid including participants with multiple lesions?	No
		Low	High
DOMAIN 2: Index Test Visual Inspection (in‐person)
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Unclear
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	Yes
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	No
Was the test interpretation carried out by an experienced examiner?	Unclear
		Unclear	Unclear
DOMAIN 2: Index Test Dermoscopy (in‐person)
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Unclear
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	Yes
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	Yes
Was the test interpretation carried out by an experienced examiner?	Unclear
		Unclear	Unclear
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	Yes
Were the reference standard results interpreted without knowledge of the results of the index tests?	Unclear
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Unclear
Were the reference standard results interpreted without knowledge of the referral diagnosis?	Unclear
		Low	Unclear
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Yes
Did all patients receive the same reference standard?	Yes
Were all patients included in the analysis?	Yes
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?
		Low

Menzies 2000.

Study characteristics
Patient sampling	Study design Case control Data collection Retrospective image selection/Prospective interpretation Period of data collection: NR Country: Australia and USA Test set derived: Sample randomly divided into training and test sets
Patient characteristics and setting	Inclusion criteria: Pigmented skin lesions with dermoscopic images and histological diagnoses; BCCs, invasive melanomas and clinically atypical 'nonmelanoma' lesions separately sampled Study setting: Specialist unit; Sydney Melanoma Unit and Florida Skin and Cancer Unit databases Prior testing: Selected for excision (no further detail) Exclusion criteria: NR Sample size (patients): NR Sample size (lesions) N included: 213 Participants Characteristics: NR Lesion characteristics: Median Breslow thickness for invasive melanoma (71/213) was 0.67 mm for the test set
Index tests	Dermoscopy: Own new algorithm (Menzies) for diagnosis of pigmented BCC Method of diagnosis: Dermoscopic images; images studies on a viewer Prior test: No further information used Diagnostic threshold: Pigment network absent with at least 1 positive feature present: ulceration, large blue‐grey ovoid nests, multiple blue‐grey globules, maple leaflike areas, spoke wheel areas, arborising (treelike) telangiectasia (all defined in detail) Diagnosis based on: Unclear; training set images assessed by 2 observers; unclear if consensus or average and whether same observers also assessed the test set images; N = 2 Observer qualification: NR: likely dermatologists Observer experience in practice: NR Observer experience with index test: NR Derivation aspect: Training set was assessed for the presence/absence of 45 dermoscopic features and a simple model constructed using negative features with low sensitivity and high specificity for invasive melanoma and benign nonmelanoma lesions. The optimal model was then evaluated on the test set of images
Target condition and reference standard(s)	Reference standard: Histological diagnosis alone (not further described) Target condition (Final diagnoses): Test set: BCC: 71; melanoma (invasive): 71; sebhorrheic keratosis: 5; ephelis 1; solar lentigo 3; common naevus 19; dysplastic naevus 38; blue naevus 2; dermatofibroma 1; haemangioma 1; Other 1
Flow and timing	Participant exclusions: NR Index test to reference standard interval: PSLs photographed prior to excision
Comparative
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Unclear
Was a case‐control design avoided?	No
Did the study avoid inappropriate exclusions?	Unclear
Are the included patients and chosen study setting appropriate?	No
Did the study avoid including participants with multiple lesions?	Unclear
		High	High
DOMAIN 2: Index Test Dermoscopy (image based)
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Yes
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	No
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	Yes
Was the test interpretation carried out by an experienced examiner?	Unclear
		Low	High
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	Yes
Were the reference standard results interpreted without knowledge of the results of the index tests?	Unclear
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Unclear
Were the reference standard results interpreted without knowledge of the referral diagnosis?	Unclear
		Low	Unclear
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Yes
Did all patients receive the same reference standard?	Yes
Were all patients included in the analysis?	Yes
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?
		Low

Navarrete Dechent 2016.

Study characteristics
Patient sampling	Study design: Case series Data collection: Retrospective image selection/Prospective interpretation Period of data collection: 2009 ‐ 2012 Country: USA
Patient characteristics and setting	Inclusion criteria: Consecutively‐excised nonpigmented lesions with no discernible pigment on clinical or dermoscopic images Setting: Specialist unit; Memorial Sloane Kettering Cancer Centre Prior testing: Selected for excision (no further detail) Setting for prior testing: Specialist unit Exclusion criteria: Collision tumours, dermatofibromas and seborrhoeic keratoses were excluded Sample size (patients): N eligible: 2375; N included: NR Sample size (lesions): N eligible: 2891; N included: 457 Participant characteristics: Mean age: 64.3 (SD 14.1); Male: 282; 61.7% Lesion characteristics: Head/neck: 134; 29.3%; trunk: 124; 27.1%; upper extremity 84; 18.4%; lower extremity 113; 24.7%; genitalia 1; 0.2%; missing 1; 0.2%
Index tests	Dermoscopy: Own new algorithm (shiny white streaks (SWSs)) Method of diagnosis: Dermoscopic images; Each individual lesion’s close‐up clinical (cropped images without patient identifiers) and dermoscopic images were reviewed for inclusion by a single author Prior test data: No further information used Diagnostic threshold: Presence of any SWSs; these were classified as (1) blotches (also known as clods; discrete, small or large structureless areas); (2) strands (long thick or thin lines, randomly distributed or parallel, and not orthogonally oriented); (3) rosettes (cluster of 4 white dots in a 4‐leaf clover–like arrangement); and (4) short white lines (also known as crystalline structures and chrysalis; fine lines that intersect or are oriented orthogonally to each other) (Liebman 2012; Liebman 2011). Shiny white structures that could not be classified into one of these specific morphologies were categorised as nonspecified. (All lesions were also evaluated for Menzies criteria (Menzies 2000); those without Menzies criteria were considered featureless and were further evaluated for presence of: SFT; multiple in‐focus, blue‐grey dots; multiple small erosions;and concentric structures). Diagnosis based on: Consensus (2 observers); N = 2 Observer qualifications: 1 observer appears to be a dermatologist and the other was a medical student (based on authors' institutions); both trained by a third observer (expert dermoscopist) who also acted as arbitrator in case of any disagreement Experience in practice: Not described Experience with index test: Trained; Described as "trained in dermoscopic analysis by an expert dermoscopist" Any other detail: Images were captured with a Nikon 1 camera (Nikon USA, Inc) using Dermlite DL2 pro HR for polarized images and Dermlitefluid for nonpolarised images at 10‐fold magnification(3Gen, LLC)
Target condition and reference standard(s)	Reference standard: Histological diagnosis alone Target condition (Final diagnoses): BCC: 287; cSCC: 106; melanoma (in situ and invasive, or not reported): 21; lichen planus–like keratosis 39; naevus 4
Flow and timing	Participant exclusions: NR Index test to reference standard interval: Appears consecutive; "Standard procedures in this practice included capturing clinical and dermoscopic images of all lesions selected for biopsy"
Comparative
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Yes
Was a case‐control design avoided?	Yes
Did the study avoid inappropriate exclusions?	No
Are the included patients and chosen study setting appropriate?	No
Did the study avoid including participants with multiple lesions?	No
		High	High
DOMAIN 2: Index Test Dermoscopy (image based)
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	No
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	No
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	Yes
Was the test interpretation carried out by an experienced examiner?	No
		High	High
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	Yes
Were the reference standard results interpreted without knowledge of the results of the index tests?	Unclear
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Unclear
Were the reference standard results interpreted without knowledge of the referral diagnosis?	Unclear
		Low	Unclear
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Yes
Did all patients receive the same reference standard?	Yes
Were all patients included in the analysis?	Yes
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?
		Unclear

Nori 2004.

Study characteristics
Patient sampling	Study design: Case control Data collection: Retrospective image selection/Prospective interpretation Period of data collection 2 years ‐ date range not specified Country USA and Spain
Patient characteristics and setting	Inclusion criteria: Biopsy‐confirmed BCC and convenience sample of non‐BCC with "'range of common diagnoses"; of these images with superior clinical quality were selected for clinical assessment Setting: Secondary (general dermatology); Private care Prior testing: Most underwent biopsy but no detail of selection process Setting for prior testing: Unspecified Exclusion criteria: NR Sample size (patients): N included: 145 Sample size (lesions): N included: 152; 105 in VI analysis Participant characteristics: Male: 98; 64% Lesion characteristics: Face/ears: 35%; trunk: 13%; limbs: extremities 45%; back 7%; only 7 of 69 non‐BCC lesions "had BCC on the list of possible differential diagnoses"
Index tests	Visual inspection (VI): No algorithm Method of diagnosis: Clinical photographs; "set of randomised clinical images was ... analysed in a blinded fashion by two dermatologists" Prior test data: No further information used Diagnostic threshold: High and high/medium probability of BCC. Lesions assigned to: high probability (BCC until proven otherwise), medium probability (would biopsy to rule out BCC), and low probability (no biopsy needed) Diagnosis based on: Single observer (N = 2) Observer qualifications: Dermatologist Experience in practice: Not described Experience with index test: Not described
Target condition and reference standard(s)	Reference standard Histological diagnosis plus other. Histology not further described Expert opinion: 15 lesions were not biopsied (e.g. lesions like seborrhoeic keratosis) because the clinical diagnosis was considered diagnostic Target condition (Final diagnoses): BCC: 83; 58 in VI analysis; cSCC: 4 'Benign' diagnoses: 65
Flow and timing	Participant exclusions: 47 lesions were not included because of poor clinical image quality Index test to reference standard interval: Not described
Comparative
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	No
Was a case‐control design avoided?	No
Did the study avoid inappropriate exclusions?	Unclear
Are the included patients and chosen study setting appropriate?	No
Did the study avoid including participants with multiple lesions?	Yes
		High	High
DOMAIN 2: Index Test Visual inspection (image based)
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Unclear
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	No
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	No
Was the test interpretation carried out by an experienced examiner?	Unclear
		Unclear	High
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	No
Were the reference standard results interpreted without knowledge of the results of the index tests?	Unclear
Expert opinion (with no histological confirmation) was not used as a reference standard	No
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Unclear
Were the reference standard results interpreted without knowledge of the referral diagnosis?	Unclear
		High	High
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Unclear
Did all patients receive the same reference standard?	No
Were all patients included in the analysis?	No
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?
		High

Rosendahl 2011.

Study characteristics
Patient sampling	Study design: Case series Data collection: Retrospective image selection/Prospective interpretation Period of data collection 30‐month period; dates NR Country Australia
Patient characteristics and setting	Inclusion criteria: Consecutive series of pigmented lesions submitted for histology from the primary‐care skin cancer practice of 1 author Setting: Primary‐care skin cancer practice Prior testing: Selected for excision (no further detail) Setting for prior testing: Primary Exclusion criteria: Poor image quality (considered under Flow and Timing) Sample size (patients): N included: 389 Sample size (lesions): N eligible: 466 pigmented lesions out of 1959 lesions excised or biopsied; N included: 463 Participant characteristics: Mean age: 57 (SD 17). Male: 67.4% Lesion characteristics: (53.1%) melanocytic. Lesion site: 17.7% head or face; trunk: 52.1%; 27.6% extremities; 2.2% palms or soles. melanoma thickness: ≤ 1 mm: 1/29 melanoma (3.4%)
Index tests	Visual inspection (VI) No algorithm Method of diagnosis: Clinical photographs overview and close‐up image presented Prior test data: No further information used Other test data: Dermoscopic images presented to observer subsequent to diagnosis using clinical images alone Diagnostic threshold: Clinical diagnosis/subjective impression. Observers gave a diagnosis with level of confidence (from 0 for definitely benign to 100 for definitely malignant) after viewing the clinical images. (NB used authors' threshold for detection of any skin cancer which includes lesions clinically considered to be MM, BCC pigmented epithelial carcinoma including SCC, keratoacanthoma, actinic keratosis and Bowen's disease as test positive; review only considered histologically‐confirmed MM, BCC or invasive SCC to be disease‐positive) Diagnosis based on: Single observer (N NR) Observer qualifications: Expert dermatologist (based on author communication). Experience in practice: Expert Experience with dermoscopy: Expert Dermoscopy Pattern analysis; new algorithm ‐ Chaos and clues Method of diagnosis: Clinical photographs (1 overview and 1 close‐up), followed by 1 dermoscopic image presented to a blinded observer on a computer screen Prior test data: Clinical image only; Diagnosis made based on clinical image before presentation of dermoscopic image Diagnostic threshold: Observers gave a diagnosis with level of confidence (from 0 for definitely benign to 100 for definitely malignant) Chaos and clues short algorithm ‐ each assessed for evidence of ‘‘chaos’’ (asymmetry of colour or structure); if present then ‘‘clues’’ searched for. Chaos ‐ asymmetry of structure and colour defined according to the basic principles of pattern analysis as revised by Kittler 2007. Clues included: eccentric structure‐less zone (any colour except skin colour), grey or blue structures, peripheral black dots or clods, segmental radial lines or pseudopods, polymorphous vessels, white lines, thick reticular or branched lines, and parallel lines on ridges (acral lesions) Observers as for visual inspection
Target condition and reference standard(s)	Reference standard: Histological diagnosis alone Details: Excise or biopsy Target condition (Final diagnoses): Melanoma (invasive): 9; melanoma (in situ): 20; BCC: 72; cSCC: 5 (including 2 keratoacanthoma); 'Benign' diagnoses: 18 Bowen's disease and 14 actinic keratosis, 217 benign melanocytic plus additional 140 benign non‐melanocytic *authors considered Bowen's disease, actinic keratosis and keratoacanthoma as malignant"; all considered benign for review analysis
Flow and timing	Excluded participants: Lesions were excluded due to poor image quality (N = 3) Time interval to reference test: Unclear; lesions 'routinely photographed' if scheduled for excision or biopsy but not further described
Comparative	Time interval between index test(s): consecutive
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Yes
Was a case‐control design avoided?	Yes
Did the study avoid inappropriate exclusions?	No
Are the included patients and chosen study setting appropriate?	Yes
Did the study avoid including participants with multiple lesions?	No
		High	High
DOMAIN 2: Index Test Visual inspection (image based)
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Unclear
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	No
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	No
Was the test interpretation carried out by an experienced examiner?	Yes
		Unclear	High
DOMAIN 2: Index Test Dermoscopy (image based)
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Yes
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?	No
Was the test applied and interpreted in a clinically applicable manner?	Unclear
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	Yes
Was the test interpretation carried out by an experienced examiner?	Yes
		Low	Unclear
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	Yes
Were the reference standard results interpreted without knowledge of the results of the index tests?	Unclear
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Unclear
Were the reference standard results interpreted without knowledge of the referral diagnosis?	Unclear
		Low	Unclear
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Unclear
Did all patients receive the same reference standard?	Yes
Were all patients included in the analysis?	No
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?
		High

Schwartzberg 2005.

Study characteristics
Patient sampling	Study design: Case series Data collection: Prospective Period of data collection October 2002 ‐ December 2003 Country USA
Patient characteristics and setting	Inclusion criteria: Patients with suspected BCC undergoing biopsy; dermatology faculty performing biopsies on patients in whom BCC was a consideration were asked to complete a study questionnaire Setting: Secondary; refers to 'Dermatology faculty' Prior testing: Clinical suspicion Setting for prior testing: Unspecified Exclusion criteria: NR Sample size (patients): N eligible: 161; N included: 141. If multiple biopsies were performed on the same participant, only the first biopsy performed was included in the study Sample size (lesions): N eligible: 161; N included: 141 Participant characteristics: Mean age: 64 (28 ‐ 92); Male: 65%; Immunosuppresion (%) 5.7% Lesion characteristics: Pigmented: 19%; non‐pigmented: 81%; ulcerated (%): 25%; erythematous 49%, telangiectasis 60%, pearly border 75%, crusty 33%, scaly 41%. Head/neck: 61%; mean lesion area was 31 mm2 (range 1 mm2 – 1.8 cm2)
Index tests	Visual inspection (VI) No algorithm Method of diagnosis: In‐person diagnosis Prior test data: No further information used Diagnostic threshold: Clinical diagnosis (certainty of diagnosis of BCC); plus combinations of characteristics predictive of BCC Diagnosis based on: Single observer Number of examiners 17 (11 full‐time faculty members and 6 part‐time faculty) Observer qualifications: Likely all dermatologists; (1 full‐time faculty member and 1 part‐time faculty member perform Mohs surgery and the others perform dermatologic surgery within the context of their general dermatology practice) Experience in practice: Assumed high Experience with index test: Not described Other detail: Information about the lesions being biopsied was collected, including: length of time the lesion was present, the location, and the presence of telangiectasias, ulceration, crusting, surrounding erythema, scale, pigmentation, or a pearly border, or both. Multivariate logistic regression analysis using backward selection used to id best predictors of BCC diagnosis
Target condition and reference standard(s)	Reference standard: Histological diagnosis alone Details: Dermatology faculty performed biopsies. No further detail Target condition (Final diagnoses): BCC: 82; Other diagnoses not reported apart from FPs for those with clinical certainty level 1 (6 were actinic keratoses, 2 were dermal naevi, and 1 each were scar, dermal elastosis, and trichoepithelioma)
Flow and timing	Participant exclusions: NR Index test to reference standard interval: Consecutive; diagnoses recorded prior to dermatology faculty performing biopsies
Comparative
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Unclear
Was a case‐control design avoided?	Yes
Did the study avoid inappropriate exclusions?	Unclear
Are the included patients and chosen study setting appropriate?	No
Did the study avoid including participants with multiple lesions?	No
		Unclear	High
DOMAIN 2: Index Test Visual Inspection (in‐person)
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Unclear
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	Yes
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	Unclear
Was the test interpretation carried out by an experienced examiner?	Yes
		Unclear	Unclear
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	Yes
Were the reference standard results interpreted without knowledge of the results of the index tests?	Unclear
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Unclear
Were the reference standard results interpreted without knowledge of the referral diagnosis?	Unclear
		Low	Unclear
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Yes
Did all patients receive the same reference standard?	Yes
Were all patients included in the analysis?	Yes
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?
		Low

Stanganelli 2000.

Study characteristics
Patient sampling	Study design: Case series Data collection: Retrospective Period of data collection 1994 ‐ 1996 Country Italy
Patient characteristics and setting	Inclusion criteria: Patients with pigmented skin lesions referred by dermatologists and general practitioners either for pre‐surgical assessment or consultation Setting: Specialist unit (skin cancer/pigmented lesions clinic) Prior testing: Patients referred for pre‐surgical assessment or consultation indicating they have had prior tests Setting for prior testing: Primary: some patients referred for consultation only; dermoscopy findings are reported back and management decision remains with referring clinician; Secondary (general dermatology) Exclusion criteria: NR Sample size (patients): N eligible: 1556 Sample size (lesions): N eligible: 3372; N included: 3372 Participant characteristics: Median age 30 years, range 10 to 94; Male: 522 (34%) Lesion characteristics: NR
Index tests	Visual inspection (VI) ABCD Method of diagnosis: In‐person diagnosis Prior test data: N/A in‐person diagnosis Other test data: Dermoscopic and clinical images subsequently presented separately to observer subsequent to diagnosis using clinical images alone Diagnostic threshold: NR Diagnosis based on: Single observer; N = 1 Observer qualifications: NR; described as 1 of the co‐authors and study based in skin cancer clinic ‐ likely dermatologist Experience in practice: Not described Experience with dermoscopy: Not described Other detail: A crude clinical image (magn x 6 and x 10) was recorded in the digital database Dermoscopy: Pattern analysis Method of diagnosis: Unclear; participants seen in person but dermoscopic diagnosis made based on digital ELM image (by same clinician as in‐person clinical dx) Prior test data: Combined clinical/dermoscopy diagnosis Diagnostic threshold: Diagnosis described as based on an integrated synopsis of the patterns most commonly described in the literature (Steiner 1993) and generally associated with known histologic counterparts. Features were assessed described in detail with multiple references, including: presence of pigment network, sharp margins, abrupt edge of pigment network, branched streaks, pseudopods, radial streaming, brown globules, pigment dots, whitish or whitish‐blue veil, grey‐blue areas, white or depigmented areas, maple leaf areas, milia‐cysts, horny plugs and vascular patterns. Test observers: As described for Visual Inspection (above) Experience with dermoscopy: Any other detail. The equipment consisted of a Leica Wild M‐650 stereomicroscope (Leica AG, Heerbrugg, Switzerland), a Sony 3ccd DXC‐930P colour video camera, an AT‐Vista videographics adapter, and IBM personal computer, a Sony Trinitron Analog PVM‐2043MD monitor, and the DBDERMO MIPS software
Target condition and reference standard(s)	Reference standard: Histological diagnosis plus follow‐up; histology report of known surgical excisions (n = 262) plus a cancer registry‐based follow‐up of benign cases (N = 3110) Target condition (Final diagnoses): Melanoma (in situ and invasive, or not reported): 55; BCC: 43; 'Benign' diagnoses: 3274
Flow and timing	Excluded participants: None reported Time interval to reference test: NR
Comparative	Time interval between index test(s): not clearly reported just indicated that D‐ELM was performed soon after clinical examination
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Yes
Was a case‐control design avoided?	Yes
Did the study avoid inappropriate exclusions?	Yes
Are the included patients and chosen study setting appropriate?	Yes
Did the study avoid including participants with multiple lesions?	No
		Low	High
DOMAIN 2: Index Test Visual Inspection (in‐person)
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Yes
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	Yes
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	Yes
Was the test interpretation carried out by an experienced examiner?	Unclear
		Low	Unclear
DOMAIN 2: Index Test Dermoscopy (in‐person)
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Yes
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	Unclear
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	Yes
Was the test interpretation carried out by an experienced examiner?	Unclear
		Low	Unclear
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	No
Were the reference standard results interpreted without knowledge of the results of the index tests?	Unclear
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Unclear
Were the reference standard results interpreted without knowledge of the referral diagnosis?	Unclear
		High	Unclear
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Unclear
Did all patients receive the same reference standard?	No
Were all patients included in the analysis?	Yes
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?	Yes
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?
		High

Steiner 1987.

Study characteristics
Patient sampling	Study design: Case series Data collection: Prospective Period of data collection: Not specified Country: Austria
Patient characteristics and setting	Inclusion criteria: Small (< 10 mm) pigmented skin lesions considered diagnostically equivocal in that there was no absolute agreement on the clinical diagnosis among investigating clinicians at a pigmented lesions clinic Setting: Specialist unit (skin cancer/pigmented lesions clinic) Prior testing: Clinical suspicion of malignancy without dermatoscopic suspicion Setting for prior testing: Specialist unit (skin cancer/pigmented lesions clinic) Exclusion criteria: > 10 mm diameter Sample size (patients): NR Sample size (lesions): 318 Participant characteristics: NR Lesion characteristics: NR
Index tests	Visual inspection (VI): No algorithm Method of diagnosis: In‐person diagnosis Prior test data: N/A Other test data: Dermoscopy undertaken by same clinician(s) subsequent to clinical evaluation Diagnostic threshold: NR Diagnosis based on: Consensus (3 observers) "All lesions were independently seen and diagnosed by the three investigators, and the diagnosis that appeared most probable to at least two of the three investigators was recorded as the clinical"; N = 3 Observer qualifications: Dermatologist Experience in practice: High experience or ‘Expert’; "experienced dermatologists" Experience with dermoscopy: Unclear; not explicitly described. Discussion describes ELM as standard procedure in clinic Study reported data for dermoscopy, but a breakdown of incorrect diagnoses by final diagnosis was not provided to allow a 2 x 2 to be estimated
Target condition and reference standard(s)	Reference standard: Histological diagnosis alone Target condition (Final diagnoses): Melanoma (invasive): 49; melanoma (in situ): 15; BCC: 20; lentigo maligna 9 (also includes lentigo maligna melanoma); Sebhorrheic keratosis: 20; junctional naevi 39; blue naevus 29; dysplastic naevus 75; lentigo simplex and naevoid lentigo 19; angioma/ angiokeratoma 15
Flow and timing	Excluded participants: None reported Time interval to reference test: Assumed consecutive; following diagnosis, lesions subsequently excised
Comparative	Time interval between index test(s): consecutive
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Unclear
Was a case‐control design avoided?	Yes
Did the study avoid inappropriate exclusions?	Unclear
Are the included patients and chosen study setting appropriate?	No
Did the study avoid including participants with multiple lesions?	Unclear
		Unclear	High
DOMAIN 2: Index Test Visual Inspection (in‐person)
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Unclear
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	No
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	No
Was the test interpretation carried out by an experienced examiner?	Yes
		Unclear	High
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	Yes
Were the reference standard results interpreted without knowledge of the results of the index tests?	Unclear
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Unclear
Were the reference standard results interpreted without knowledge of the referral diagnosis?	Unclear
		Low	Unclear
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Yes
Did all patients receive the same reference standard?	Yes
Were all patients included in the analysis?	Yes
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?
		Low

Ulrich 2015.

Study characteristics
Patient sampling	Study design: Case series Data collection: Prospective Period of data collection: April 2013 ‐ March 2014 Country: Germany
Patient characteristics and setting	Inclusion criteria: Patients with non‐pigmented pink lesions with clinical suspicion of BCC requiring biopsy for diagnostic confirmation. Pink lesions defined as clinically‐unclear erythematous papule or plaque; either reddish macules, patches or small papules with or without scale Setting: Multicentre study; authors' institutions included Dermatology departments (N = 4) and private dermatology offices (N = 3) Prior testing: Clinical suspicion of malignancy Setting for prior testing: Unspecified Exclusion criteria: Lesions with the typical clinical appearance of BCC on clinical examination (such as the presence of a pearly border, central ulceration and obvious telangiectasias), as well as pigmented lesions, were excluded from the protocol. Patients with unstable or uncontrolled clinically‐significant medical conditions were excluded. Lesions with missing histology also excluded (N = 21) Sample size (patients): N eligible: 164; N included: 155 Sample size (lesions): N eligible: 256; N included: 235 (different sets of 231 lesions were available for each test) Participant characteristics: Median age: 70 (33 ‐ 90) Lesion characteristics Head/neck: 41%; upper body 48.8%
Index tests	Visual inspection (VI): No algorithm Method of diagnosis: In‐person diagnosis; "All assessments were documented before the histological results were available" Prior test data: N/A in‐person diagnosis Diagnostic threshold: Clinical diagnosis of BCC; describes diagnostic criteria as "pink or red lesions that could be either macules, patches or small papules with or without scale", but these also form part of inclusion criteria Diagnosis based on: Single observer; in‐clinic diagnosis (N NR) Observer qualifications: Not described; probably dermatologists, given authors' institutions Experience in practice: Not described Experience with index test: Not described Dermoscopy; No algorithm (referenced Marghoob 2012) Method of diagnosis: In‐person diagnosis Prior test data: Clinical examination Diagnostic threshold: Observer diagnosis of BCC: scattered vascular global pattern with loose haphazard distribution; shiny white to red structures with or without chrysalis‐like structures; small fine telangiectasias appearing as fine, kinked vessels of small calibre, with length < 1 mm in superficial BCC and larger arborising vessels in more invasive BCC (nodular/infiltrative) Observers: As above Any other detail After clinical examination dermoscopy was carried out using a Dermlite ProHr (3Gen Inc., San Juan Capistrano, CA, USA), attached to a Sony Cybershot DSC‐W710 camera (Sony, Tokyo, Japan) (supplied by MDL). As polarised light was used, no preparation of the area under examination was necessary
Target condition and reference standard(s)	Reference standard: Histological diagnosis alone Details: a biopsy or excision of the lesion was taken and sent for histological analysis Target condition (Final diagnoses): BCC: 141 (as different sets of 231 lesions were available for each test, the number diseased per 2 x 2 varies); 'Benign' diagnoses: 94
Flow and timing	Participant exclusions: Histology was missing for 21 lesions, and 1 case was found to have a combination of both BCC and SK or AK, leaving 235 lesions for analysis in the ITT group Index test to reference standard interval: Consecutively done after index test "All diagnostic steps had to be completed before histological confirmation was made"
Comparative	Time interval between index test(s): consecutive
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Yes
Was a case‐control design avoided?	Yes
Did the study avoid inappropriate exclusions?	No
Are the included patients and chosen study setting appropriate?	No
Did the study avoid including participants with multiple lesions?	No
		High	High
DOMAIN 2: Index Test Visual Inspection (in‐person)
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Yes
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	Yes
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	Unclear
Was the test interpretation carried out by an experienced examiner?	Unclear
		Low	Unclear
DOMAIN 2: Index Test Dermoscopy (in‐person)
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Yes
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	Yes
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	Yes
Was the test interpretation carried out by an experienced examiner?	Unclear
		Low	Unclear
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	Yes
Were the reference standard results interpreted without knowledge of the results of the index tests?	Unclear
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Unclear
Were the reference standard results interpreted without knowledge of the referral diagnosis?	Unclear
		Low	Unclear
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Yes
Did all patients receive the same reference standard?	Yes
Were all patients included in the analysis?	No
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?
		High

Witkowski 2016.

Study characteristics
Patient sampling	Study design: Case series Data collection: Retrospective image selection/Prospective interpretation Period of data collection: January 2009 – 2011 Country: Italy
Patient characteristics and setting	Inclusion criteria: Consecutive clinically‐equivocal ‘pink’ cutaneous lesions with absent pigmentation or containing < 10% pigment and absence of pigment network. All lesions were excised at first visit or follow‐up video dermoscopy control visit and had available digital dermoscopy images and a complete standard set of RCM images, with histopathology reports Setting: Secondary (general dermatology) Prior testing: Clinical suspicion of malignancy without dermatoscopic suspicion Setting for prior testing: Secondary (general dermatology) Exclusion criteria: Benign diagnosis made with high confidence; lack of histological report as a result of the lesion not being excised Sample size (patients): NR Sample size (lesions): N eligible: 3869 consecutive cases were reviewed; N included: 260 Participant characteristics: NR Lesion characteristics: NR
Index tests	Dermoscopy No algorithm Method of diagnosis: Dermoscopic images Prior test data: No further information used Diagnostic threshold: Correct diagnosis (of BCC, MM and SCC) and correct management decision (excise or not) Diagnosis based on: Single observer (N = 2; 1 reader evaluated only dermoscopic images while the second reader evaluated RCM images) Observer qualifications: Not clear; only given initials of the reader, likely dermatologist Experience in practice: Not described Experience with index test: Not described Any other detail: Digital dermoscopy images were obtained with DermLite FOTO System (DermLite Photo 3Gen, San Juan Capistrano, CA, USA)
Target condition and reference standard(s)	Reference standard: Histological diagnosis alone Target condition (Final diagnoses): BCC: 114; cSCC: 13; melanoma (in situ and invasive, or not reported): 12; Other malignant: 1 syringoid eccrine carcinoma; sebhorrheic keratosis: 25 grouped solar lentigo/seborrhoeic keratosis/lichen planus‐like keratosis/actinic keratosis (SL/SK/LPLK/AK); benign naevus: 47 naevi; 6 Spitz naevi; 18 dermatofibromas (DF), 4 vascular lesions, and 20 other type benign lesions. Other types of benign lesions included 1 clear cell acanthoma, 1 discoid lupus, 10 inflammatory lesions, 1 perivascular hyperplasia, 4 granulomatous hyperacanathosis reactions, 1 papulous fibrosis, 1 eccrine poroma, and 1 eczematous lesion
Flow and timing	Excluded participants: Around 357 cases were excluded due to the lack of a histopathology report, as a result of the lesion not being excised, or a benign diagnosis was made with high confidence Time interval to reference test: lesions excised at first visit or follow‐up video dermoscopy control visit
Comparative
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Yes
Was a case‐control design avoided?	Yes
Did the study avoid inappropriate exclusions?	Yes
Are the included patients and chosen study setting appropriate?	No
Did the study avoid including participants with multiple lesions?	Unclear
		Low	High
DOMAIN 2: Index Test Dermoscopy (image based)
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Unclear
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	No
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	No
Was the test interpretation carried out by an experienced examiner?	Unclear
		Unclear	High
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	Yes
Were the reference standard results interpreted without knowledge of the results of the index tests?	Unclear
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Unclear
Were the reference standard results interpreted without knowledge of the referral diagnosis?	Unclear
		Low	Unclear
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Yes
Did all patients receive the same reference standard?	Yes
Were all patients included in the analysis?	Yes
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?
		Low

Zalaudek 2006.

Study characteristics
Patient sampling	Study design: Case series Data collection: Retrospective image selection/Prospective interpretation Period of data collection February 2003 ‐ January 2004 Country Naples, Italy
Patient characteristics and setting	Inclusion criteria: Excised, equivocal and nonequivocal, pigmented and nonpigmented skin lesions with good image quality and melanin or haemoglobin pigmentation in all or part of the lesion Setting: Specialist unit; specialized Pigmented Lesion Clinic database Prior testing: Selected for excision (no further detail) Setting for prior testing: Specialist unit Exclusion criteria: NR Sample size (patients): NR Sample size (lesions): Eligible: 2621; Included: 150 (plus 15 lesions used for training purposes) Participant characteristics: NR Lesion characteristics 37/165 (26%) considered equivocal on clinical and dermoscopic grounds Thickness/depth: Mean Breslow 0.9 mm
Index tests	Dermoscopy: 3‐point checklist Method of diagnosis: Dermoscopic images, "optimized for colour, brightness and contrast by using Adobe photoshop standards" Prior test data: Age, site, and gender provided Diagnostic threshold: 1+ criteria present indicates malignancy (asymmetry ‐ in colour and/or structure, not in shape; atypical network ‐ pigment network with thick lines and irregular holes; and blue‐white structures ‐ presence of any blue and/or white colour within the lesion) Diagnosis based on: Average (N = 150 out of 170 participating observers, who finished all 15 training cases and performed at least 1 evaluation of the main set of images (test set). Participation was open to all individuals regardless of professional profile and experience in dermoscopy; study was advertised through personal communication, e‐mail correspondences, adverts during congresses and courses, as well as via the website (www.dermoscopy.org)) Observer qualifications: For full sample of 170: dermatologists (N = 125); GPs (N = 15); other professionals in the field of skin lesions (N = 12); medical students (N = 7); other medical specialty (N = 11) Experience in practice: Not described Experience with dermoscopy: Mixed; 146/170 (86%) reported some experience with dermoscopy; 24 with no dermoscopy experience, 45 (26%) with > 5 years experience Dermoscopy training: A web‐based tutorial was provided to describe the concept of the 3‐point checklist of dermoscopy including complete definitions of criteria and example images. Following web‐based tutorial, observers initially scored a random sample of 15 images, receiving real‐time feedback for that case as judged by an expert observer Training format: Online
Target condition and reference standard(s)	Reference standard: Histological diagnosis alone (no further details) Target condition (Final diagnoses): Melanoma (invasive): 18; melanoma (in situ): 11; BCC: 18; 79 melanocytic naevi; 26 seborrhoeic keratoses; 8 vascular tumours and 3 dermatofibromas
Flow and timing	Participant exclusions: Poor‐quality index test image as exclusion criterion Index test to reference standard interval: Not described
Comparative
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Yes
Was a case‐control design avoided?	Yes
Did the study avoid inappropriate exclusions?	Yes
Are the included patients and chosen study setting appropriate?	No
Did the study avoid including participants with multiple lesions?	Unclear
		Low	High
DOMAIN 2: Index Test Dermoscopy (image based)
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Yes
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?
Was the test applied and interpreted in a clinically applicable manner?	No
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	Yes
Was the test interpretation carried out by an experienced examiner?	Unclear
		Low	High
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	Yes
Were the reference standard results interpreted without knowledge of the results of the index tests?	Unclear
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Unclear
Were the reference standard results interpreted without knowledge of the referral diagnosis?	Unclear
		Low	Unclear
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Unclear
Did all patients receive the same reference standard?	Yes
Were all patients included in the analysis?	No
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?
		High

AK – actinic keratosis; BCC – basal cell carcinoma; BD – Bowen’s disease; BN – benign naevi; BPC – between‐person comparison (of tests); CAD – computer‐assisted diagnosis; CCS – case control study; CS – case series; cSCC – cutaneous squamous cell carcinoma; DF – dermatofibroma; ELM ‐ epiluminescence microscopy (dermoscopy); FU – follow‐up; LS – lentigo simplex; MiS – melanoma in situ (or lentigo maligna); MM – malignant melanoma; N ‐ number; N/A ‐ not applicable; NC – non‐comparative; NR – not reported; P – prospective; PCP ‐ primary‐care physician; PLC – pigmented lesion clinic; PSL – pigmented skin lesion; R – retrospective; RCM – reflectance confocal microscopy; SK – seborrhoeic keratosis; SN – Spitz naevi; WPC – within‐person comparison (of tests).

Characteristics of excluded studies [ordered by study ID]

Study	Reason for exclusion
Abbasi 2004	Not a primary study Systematic review
Ahnlide 2013	Ineligible index test 'clinical diagnosis' study
Ahnlide 2016	Ineligible target condition; does not present data for detection of BCC or cSCC
Akasu 1996	Insufficient data for 2 x 2 table No 2 x 2 data only describing the dermoscopic features present in the lesions
Al Jalbout 2013	Inadequate sample size Case study
Alarcon 2014	Ineligible target condition; does not present data for detection of BCC or cSCC
Aldridge 2011a	Ineligible test observer Medical students and lay persons
Aldridge 2011b	Ineligible test observer
Aldridge 2013	Insufficient data for 2 x 2 table Not test accuracy study
Alendar 2009	Ineligible reference standard Only 7 reported verified histologically
Altamura 2006	Assesses individual lesion characteristics only Insufficient data for 2 x 2 table Looking for characteristics associated with acral melanoma; does not give 2 x 2 for overall diagnosis
Annessi 2007	Ineligible target condition; does not report data for BCC or cSCC
Antonio 2013	Ineligible target condition Atypical naevi does not fall within our definition of D+
Antoszewski 2015	Inadequate sample size All excised lesions were benign. Insufficient data for 2 x 2 table
Aoyagi 2010	Inadequate sample size
Arevalo 2008	Ineligible target condition; does not present data for detection of BCC or cSCC
Argenziano 1997	Wrong study population Only melanoma included
Argenziano 1998	Ineligible target condition; does not present data for detection of BCC or cSCC
Argenziano 1999	Wrong study population Only includes melanoma
Argenziano 2002	Not a primary study
Argenziano 2003	Insufficient data for 2 x 2 table Table V gives se/sp data for 108 lesions but cannot derive the number of melanoma for this subset of the original 128 Contact authors; contacted 10 May 2016 and 24 June 2016
Argenziano 2004a	Assesses individual lesion characteristics only Only lesions with vascular structures included; presence of 10 different characteristics assessed. 2 x 2 would be possible
Argenziano 2004b	Not a primary study Letter
Argenziano 2008	Ineligible index test Surveillance/monitoring study
Argenziano 2010	Ineligible index test Test used for follow‐up looking at dermoscopic features of melanomas diagnosed 1 yr after follow‐up Insufficient data for 2 x 2 table
Argenziano 2011	Ineligible target condition Inadequate sample size Only 2 melanomas
Argenziano 2011a	Ineligible target condition 5 melanoma metastases included as D+
Argenziano 2011b	Ineligible target condition; does not present data for detection of BCC or cSCC
Argenziano 2012	Ineligible reference standard no follow‐up of test negatives
Argenziano 2014	Insufficient data for 2 x 2 table
Armstrong 2011	Ineligible reference standard No reference standard results presented for the screened lesions; just compares naked eye judgements with dermoscopy
Ascierto 1998	Insufficient data for 2 x 2 table The data presented do not contribute to the review Duplicate or related publication. Data included in Ascierto 2003
Ascierto 2000	Insufficient data for 2 x 2 table Contact authors For excised lesions, study cross‐tabulates ELM high/very high‐risk classification against some histological classification (Table 2). Number D+ = 580 (2 x 2: 504, 79, 76, 2072); 580 not mentioned anywhere else in paper (contacted 10 May 2016 and 24 June 2016)
Ascierto 2003	Not a primary study
Ascierto 2010	Ineligible target condition; does not present data for detection of BCC or cSCC
Badertscher 2015	Insufficient data for 2 x 2 table
Bafounta 2001	Not a primary study Systematic review
Bajaj 2016	Ineligible reference standard Unclear ref standard for benign diagnoses
Banky 2005	Ineligible target condition Ineligible index test
Barzegari 2005	Ineligible target condition; does not present data for detection of BCC or cSCC
Basarab 1996	Wrong study population Not all suspected of skin cancer Insufficient data for 2 x 2 table
Bauer 2000	Ineligible index test Does not provide 2 x 2 data for visual inspection alone
Bauer 2005	Ineligible index test Follow‐up/monitoring study
Bauer 2006	Ineligible index test Dermoscopy used to improve histopathology diagnosis
Becker 1954	Not a primary study
Benati 2015	Assesses individual lesion characteristics only
Benelli 1999	Ineligible target condition; does not present data for detection of BCC or cSCC
Benelli 2000a	Ineligible target condition; does not present data for detection of BCC or cSCC
Benelli 2000b	Insufficient data for 2 x 2 table Only inter‐rater reliability data given (n = 25); authors have published much larger evaluations of 7FFM and ABCD
Benelli 2001	Ineligible target condition; does not present data for detection of BCC or cSCC
Benvenuto‐Andrade 2006	Insufficient data for 2 x 2 table Diagnostic confidence rather than accuracy
Benvenuto‐Andrade 2007	Insufficient data for 2 x 2 table Agreement on lesion characterisation; not test accuracy
Binder 1994	Ineligible target condition; does not present data for detection of BCC or cSCC
Binder 1995	Ineligible target condition; does not present data for detection of BCC or cSCC
Binder 1997	Insufficient data for 2 x 2 table Training study; only ROC curves/AUC presented pre‐ and post‐training Contact authors (contacted 10 May 2016 and 24 June 2016)
Binder 1999	Ineligible target condition; does not present data for detection of BCC or cSCC
Blum 2003a	Not a primary study
Blum 2003b	Ineligible target condition; does not present data for detection of BCC or cSCC
Blum 2003c	Ineligible target condition; does not present data for detection of BCC or cSCC
Blum 2004a	Ineligible target condition; does not present data for detection of BCC or cSCC
Blum 2004b	Not a primary study Comment paper
Blum 2004c	Not a primary study Letter Letter only; limited data presented ‐ evaluates '3‐colour' rule as developed By MacKie 2002 (excluded as assessment of individual lesion features only)
Blum 2004d	Ineligible target condition; does not present data for detection of BCC or cSCC
Blum 2004e	Not a primary study Letter
Blum 2006	Ineligible target condition Differentiates melanocytic from non‐melanocytic lesions only
Blum 2011	Wrong study population Mucosal lesions only
Blum 2014	Inadequate sample size case studies
Boespflug 2015	Wrong study population Study aim is estimate the efficacy of an online spaced educational training for dermoscopy
Bolognia 1990	Ineligible reference standard No ref standard diagnosis for index test negatives
Bono 1996	Ineligible target condition; does not present data for detection of BCC or cSCC
Bono 2001	Insufficient data for 2 x 2 table Aim of the study is to determine what features are present in amelanotic cutaneous melanoma
Bono 2002a	Ineligible target condition; does not present data for detection of BCC or cSCC
Bono 2002b	Ineligible target condition; does not present data for detection of BCC or cSCC
Bono 2006	Ineligible target condition; does not present data for detection of BCC or cSCC
Borsari 2010	Assesses individual lesion characteristics only Contact authors Paper focuses on diagnostic prediction of dermoscopic island for early melanoma, however the Methods describe the calculation of the total dermoscopy score and the 7‐point checklist score; mean scores on each checklist per lesion type are then presented (no reply from authors)
Borsari 2015	Assesses individual lesion characteristics only
Borve 2012	Wrong study population Includes participants without skin lesions Inadequate sample size < 5 BCC
Bourne 2012	Ineligible target condition; does not present data for detection of BCC or cSCC
Bowns 2006	Ineligible index test; teledermatology study
Braun 2000	Derivation study This is a pilot study on the new "wobble sign" in ELM no training/test sets used
Braun 2007	Assesses individual lesion characteristics only
Braun‐Falco 1990	Insufficient data for 2 x 2 table Not a test accuracy study
Broganelli 2005	Ineligible target condition; does not present data for detection of BCC or cSCC
Brown 2000	Not a primary study Systematic review
Brown 2009	Ineligible test observer lay persons
Buhl 2012	Ineligible index test Follow‐up/monitoring Duplicate or related publication. Same participants as Haenssle 2010a #191
Burki 2015	Not a primary study
Burr 2015	Not a primary study
Burton 1998	Ineligible reference standard Can only get 2 x 2 data for referral accuracy Insufficient data for 2 x 2 table
Bystryn 2003	Not a primary study Letter
Cabrijan 2008	Insufficient data for 2 x 2 table Cannot get 2 x 2; reports % correct diagnoses for each different lesion classification and not % misdiagnosed as melanoma or melanomas missed Contact authors Study states "Dermatoscopic diagnosis were conformable with pathohistological diagnosis in 75 cases (72.82%) out of 103. The highest conformation was in diagnosing melanoma, in 5 out of 6 cases (83.3%)." which would give us sensitivity; do you have data on numbers mis classified as melanoma, i.e false positives? (author replied 5 July 2016 with some data but not sufficient to allow 2 x 2)
Canpolat 2011	Derivation study Looks at dermoscopic characteristics of acral lesions; only 4 suspicious lesions excised
Cardenas 2009	Wrong study population Includes participants with palpable lesions; not all suspected of having skin cancer
Carli 1994	Ineligible target condition; does not present data for detection of BCC or cSCC
Carli 1998	Inadequate sample size se/sp data are based on sample with only 4 MM
Carli 2000	Ineligible target condition Only lesions histologically classified as common naevi or naevi with architectural disorder with/without cytological atypia were considered for the study
Carli 2003a	Ineligible reference standard Only 39/1042 with ref test
Carli 2003b	Inadequate sample size
Carli 2003c	Ineligible target condition; does not present data for detection of BCC or cSCC
Carli 2003d	Ineligible target condition; does not present data for detection of BCC or cSCC
Carli 2004a	Inadequate sample size < 5 MM per arm Insufficient data for 2 x 2 table
Carli 2004b	Ineligible index test; can only estimate 2 x 2 for the full time period 1997 to 2001 across all observers, but dermoscopy was only introduced routinely in 1998, so some diagnoses prior to that will have been with visual inspection alone, and observers were classed as dermoscopy ’users’ (those working in pigmented lesion clinics) and nonusers (general dermatology). Contact authors Author passed away; unable to make contact with co‐authors
Carli 2004c	Ineligible index test 'Clinical diagnosis' ‐ Dataset covers 1997‐2001, but dermoscopy routinely introduced 1998; authors contacted but no response
Carli 2005	Insufficient data for 2 x 2 table Contact authors Study presents % MM correctly classified by naked eye ± dermoscopy but does not give any detail on FPs, is this available anywhere and/or are these lesions included in any subsequent publications? Author passed away; unable to make contact with co‐authors
Carlos‐Ortega 2007	Insufficient data for 2 x 2 table Gives se/sp for visual inspection and dermoscopy in the English abstract. 68 participants/70 lesions were included but only 36 seem to have had visual inspection results and all underwent dermoscopy. Two observers performed each test blinded to each other. Table I gives 22 with BCC and 11 with melanoma overall (N D+ not reported for those with VI results), but using either or both of these numbers with the se/sp provided does not give the same PPV and NPV as given by the authors Contact authors Data not clearly presented for 2 x 2; translator suggested alternative but still does not work out to what is in paper; tried contacting authors twice, no reply as of 28 July 2016
Carrera 2016	Ineligible target condition; does not present data for detection of BCC or cSCC
Carroll 1998	Derivation study Derivation study; proposes new dermoscopic criteria for dx of BCC Insufficient data for 2 x 2 table
Chen 2001	Not a primary study Systematic review comparing PCP accuracy with dermatologist accuracy.
Chen 2006	Insufficient data for 2 x 2 table Only given AUC
Chen 2013	Ineligible test observer
Chiaravalloti 2014	Wrong study population Includes melanoma only
Ciudad‐Blanco 2014	Wrong study population Includes melanoma only Assesses individual lesion characteristics only Insufficient data for 2 x 2 table
Collas 1999	Ineligible target condition; does not present data for detection of BCC or cSCC
Coras 2003	Ineligible target condition; does not present data for detection of BCC or cSCC
Cornell 2015	Ineligible test observer
Cox 2008	Ineligible reference standard Se and sp estimates for diagnosis of melanoma for both the seven‐point checklist and the revised (10‐point) checklist; reference standard not reported for any of the 381 TWR referrals for melanoma Contact authors Author contacted 10 May 2016; co‐author contacted 24 June 2016
Cristofolini 1994	Ineligible target condition; does not present data for detection of BCC or cSCC
Cristofolini 1997	Ineligible target condition; does not present data for detection of BCC or cSCC
Dal Pozzo 1999	Ineligible target condition; does not present data for detection of BCC or cSCC
De Giorgi 2006	Inadequate sample size < 5 cases of participants with a final melanoma diagnosis
De Giorgi 2011	Duplicate or related publication. Assesses same lesions as in Carli 2003c but different observers
De Giorgi 2012	Ineligible target condition; does not present data for detection of BCC or cSCC
De Troya‐Martin 2008	Wrong study population Only MM included
DeCoste 1993	Insufficient data for 2 x 2 table Not given the total number of D+/D‐ or total number of lesions included. Just given the sens/spec values
Delfino 1997	Assesses individual lesion characteristics only Derivation study Insufficient data for 2 x 2 table Only reports association of each characteristics with D+/D‐, not 2 x 2
Di Carlo 2014	Ineligible index test. Videothermography not relevant for the review and there are no 2x2 data for dermoscopy Derivation study. Only includes AK and BCC; no 2x2 for dermoscopy
Di Chiacchio 2010	Ineligible target condition Excluding nail bed melanoma Insufficient data for 2 x 2 table There are insufficient data to extract for a 2 x 2 table
Di Meo 2016	Ineligible target condition; does not present data for detection of BCC or cSCC
Di Stefani 2007	Inadequate sample size < 5 malignant
Dolianitis 2005	Ineligible target condition; does not present data for detection of BCC or cSCC
Dreiseitl 2009	Ineligible target condition; does not present data for detection of BCC or cSCC
Duff 2001	Ineligible index test Does not evaluate visual inspection alone
Dummer 1993	Ineligible target condition; does not present data for detection of BCC or cSCC
Dummer 1995	Assesses individual lesion characteristics only
Edmondson 1999	Ineligible reference standard It seems that the reference standard here is expert diagnosis. This is not a teledermatology paper
Elwan 2016	Inadequate sample size Derivation study Insufficient data for 2 x 2 table
Emmons 2011	Insufficient data for 2 x 2 table Not test accuracy study; promoting primary prevention
Engelberg 1999	Inadequate sample size Only 1 confirmed melanoma and 3 BCC
English 2003	Insufficient data for 2 x 2 table No accuracy data given
English 2004	Insufficient data for 2 x 2 table No accuracy data
Fabbrocini 2008	Insufficient data for 2 x 2 table There is insufficient data provided for each index test to populate 2 x 2 table Contact authors As we can only include DTA studies ‐ Do you have a cross tabulation of each clinician's diagnosis (e.g. at threshold of 3 or more on 7‐point checklist) against the histological diagnosis and/or a cross‐tabulation of the remote diagnosis against the face‐to‐face diagnoses? (author reply; 30 June 2016 cannot access data needed)
Feci 2015	Ineligible target condition; does not present data for detection of BCC or cSCC
Federman 1995	Insufficient data for 2 x 2 table Not test accuracy
Feldmann 1998	Ineligible target condition; does not present data for detection of BCC or cSCC
Ferrara 2002	Ineligible index test This study looks at histopathological and dermoscopic disagreements not necessarily looking at how well dermoscopy differentiates between benign and malignant diagnosis
Ferrari 2015	Ineligible target condition; does not present data for detection of BCC or cSCC
Ferris 2015	Ineligible target condition; does not present data for detection of BCC or cSCC
Fidalgo 2003	Insufficient data for 2 x 2 table Duplicate or related publication. Appears to be superseded by Serrao 2006 Contact authors Paper provides % of MM and of DN with DNAOS scores of >=5.5 and >7, is it possible for you to provide the same information for the remaining 127 lesions in the study? Also can you advise as to whether any of the 247 lesions included in this study, overlap with the 652 reported in Serrao 2006 (#1144)? (author contacted 10 May 2016; 24 June 2016)
Fikrle 2013	Ineligible reference standard Follow‐up study < 50% of study participants have their final diagnosis reached by histopathology
Freeman 1963	Insufficient data for 2 x 2 table Only gives % correct for each lesion type Contact authors Tables 2 and 3 appear to give % correct diagnoses per lesion type, but do not give data on numbers misclassified as melanoma, or other malignancy, i.e. FPs. Author responded; paper too old, cannot provide data
Friedman 1985	Not a primary study
Friedman 2008	Ineligible target condition; does not present data for detection of BCC or cSCC
Fruhauf 2012	Ineligible reference standard 35/219 underwent histology; 13 followed up; 171 expert clinical Dx
Fueyo‐Casado 2009	Ineligible reference standard < 50% of the study population received histology as a test. No information given on those who were followed up
Funt 1963	Ineligible index test Insufficient data for 2 x 2 table No 2 x 2 data
Gachon 2005	Ineligible target condition; does not present data for detection of BCC or cSCC
Gerbert 1996	Ineligible target condition No breakdown of final diagnoses for included lesions Insufficient data for 2 x 2 table Only gives % correct for each lesion type; not sens/spec
Gerbert 1998	Insufficient data for 2 x 2 table
Gereli 2010	Ineligible target condition; does not present data for detection of BCC or cSCC
Giacomel 2005	Wrong study population Only BCC included
Giacomel 2014	Inadequate sample size
Giannotti 2004	Not a primary study A review
Gill 2015	Inadequate sample size Derivation study
Gilmore 2009	Derivation study Principle of lacunarity has been looked at before but not this particular application/approach to it Ineligible reference standard It is possible to get 2 x 2 for 'standard dermoscopy criteria' but dermoscopy‐negative were not excised and assumed benign; 201/312 underwent excision so theoretically eligible
Gilmore 2010	Ineligible target condition; does not present data for detection of BCC or cSCC
Glud 2009	Ineligible target condition; does not present data for detection of BCC or cSCC
Grana 2003	Ineligible index test Assesses individual lesion characteristics only Only looking at lesion border
Green 1991	Ineligible target condition; does not present data for detection of BCC or cSCC
Green 1994	Ineligible target condition; does not present data for detection of BCC or cSCC
Grichnik 2003	Inadequate sample size
Grichnik 2004	Not a primary study Editorial
Grimaldi 2009	Ineligible target condition; does not present data for detection of BCC or cSCC
Grob 1998	Not a primary study
Guibert 2000	Ineligible reference standard Not designed as an accuracy study, only observational. Cannot get 2 x 2 data > 50% of study participants did not receive histology as reference standard
Guillod 1996	Derivation study
Gunduz 2003	Inadequate sample size Case study
Gutierrez 2013	Ineligible index test Test to improve histopathology diagnosis
Haenssle 2006	Ineligible index test Surveillance study estimating accuracy of different approaches to follow‐up
Haenssle 2010a	Ineligible target condition; does not present data for detection of BCC or cSCC
Haenssle 2010b	Insufficient data for 2 x 2 table Does not report specificity Duplicate or related publication. Same participants as Haenssle 2010a #191
Hallock 1998	Ineligible index test 'clinical diagnosis'; dermoscopy used for 3 of 4 years
Haniffa 2007	Ineligible reference standard Looks like approximately 20% of participants received a final diagnosis by histology. 179 biopsies were performed. Total sample was 881 lesions
Har‐Shai 2001	Ineligible index test 'clinical diagnosis'
Haspeslagh 2016	Assesses individual lesion characteristics only Insufficient data for 2 x 2 table
Hauschild 2014	Ineligible target condition; does not present data for detection of BCC or cSCC
Heal 2008	Insufficient data for 2 x 2 table Sensitivities and PPVs are given so theoretically a 2 x 2 could be worked out but the numbers do not appear to work out Author response; the 2 x 2 table the Cochrane researchers want to create is not possible for our results, because sensitivity and PPV are based on different sample sizes
Healsmith 1994	Ineligible reference standard Benign lesions described as 'clinically diagnosed' rather than histology/follow‐up
Henning 2007	Derivation study First application of CASH algorithm
Henning 2008	Exclude as a derivation study
Herschorn 2012	Not a primary study Systematic review
Higgins 1992	Wrong study population Includes only benign lesions Inadequate sample size No melanomas Insufficient data for 2 x 2 table No malignant cases
Hirata 2011	Ineligible target condition Ineligible index test
Hoffmann 2003	Derivation study Uses 'leave one out' cross validation procedure Insufficient data for 2 x 2 table Only giving ROC values not able to extract a 2 x 2 table
Hoorens 2016	Ineligible index test Ineligible reference standard No info on numbers undergoing histology; and no follow‐up reported for benign appearing lesions Insufficient data for 2 x 2 table
Huang 1996	Assesses individual lesion characteristics only Border irregularity not overall dx Insufficient data for 2 x 2 table
Hubener 1956	Insufficient data for 2 x 2 table
Ishioka 2009	Ineligible index test ‐ include for teledermatology only
Iyatomi 2006	Derivation study Uses 'leave one out' procedure and same lesions and tumour extraction method as Iyatomi 2008 Insufficient data for 2 x 2 table
Iyatomi 2008	Derivation study The performance was evaluated by averaging both combinations (training and test sets) they did not present the data separately; uses 'leave one out' procedure Insufficient data for 2 x 2 table Not test accuracy; compares automated with manual extraction of tumour area
Jamora 2003	Ineligible reference standard No referene standard for index test negatives
Janda 2014	Inadequate sample size Only 1 case of melanoma, 1 case of BCC and 1 of SCC
Jensen 2015	Not a primary study Comment paper
Johr 2002	Not a primary study
Jolliffe 2001	Ineligible index test Provides data for clinical diagnosis (including dermoscopy for some cases)
Jonna 1998	Insufficient data for 2 x 2 table Only included index test positives to get PPV, not worth author contact on this one
Kaddu 1997	Inadequate sample size Sample size < 5; not test accuracy
Kawabata 1998	Derivation study Aim of the study is to correlate findings between dermoscopy and histology findings of acral melanoma Insufficient data for 2 x 2 table Not test accuracy
Kawabata 2001	Wrong study population MM of the nail bed
Keefe 1990	Ineligible reference standard Only 28% (60/214) of non‐melanoma group had excision
Kefel 2012	Derivation study No test set, first use of polarised light dermoscopy, various neural networks tested Insufficient data for 2 x 2 table
Kelly 1986	Ineligible target condition Cannot disaggregate the severely dysplastic/in situ MM Inadequate sample size Unclear whether > 5 in situ melanoma
Kenet 1994	Not a primary study Insufficient data for 2 x 2 table Not an accuracy study
Kittler 1998	Ineligible target condition; does not present data for detection of BCC or cSCC
Kittler 1999	Ineligible target condition; does not present data for detection of BCC or cSCC
Kittler 2001	Ineligible target condition; does not present data for detection of BCC or cSCC
Kittler 2002	Not a primary study Systematic review
Kittler 2006	Conference abstract
Koga 2011	Ineligible reference standard ˜ 23% of participants have their final diagnosis reached by histopathology 43/191
Koh 1990	Ineligible reference standard Screening study; no adequate reference standard
Kopf 1975	Ineligible target condition; does not present data for detection of BCC or cSCC
Korotkov 2012	Not a primary study Narrative review
Krahn 1998	Ineligible target condition; does not present data for detection of BCC or cSCC
Kreusch 1992	Ineligible target condition; does not present data for detection of BCC or cSCC
Kroemer 2011	Ineligible index test Provides data for clinical diagnosis (including dermoscopy for some cases)
Krol 1991	Ineligible reference standard No follow‐up reported for those who were test‐negative
Kurvers 2015	Ineligible index test Collective intelligence ‐ majority rule and quorum rule applied to large number of test interpreter decisions Duplicate or related publication. Re‐analyses data from 2 previously published studies to determine whether collective intelligence (i.e majority rules or quorum rules across a large number of observers) imporves test accuracy. We have excluded 1 of these studies as the number of melanomas is not provided (Argenziano 2003) and included the other in dermoscopy review (Zalaudek 2006)
Kvedar 1997	Wrong study population Not all suspected of skin cancer
Lallas 2015	Derivation study Develops new algorithm and does not use separate training/test sets of lesions
Langley 2001	Ineligible target condition; does not present data for detection of BCC or cSCC
Langley 2007	Ineligible target condition; does not present data for detection of BCC or cSCC
Lechner 2015	Not a primary study Erratum
Lewis 1999	Insufficient data for 2 x 2 table Study appears to meet all eligibility criteria but disease prevalence not given alongside se/sp Contact authors Authors contacted 10 May 2016; email returned
Liebman 2011	Not a primary study Comment
Liebman 2012	Not a primary study Comment
Lindelöf 1994	Wrong study population Only malignant melanoma Insufficient data for 2 x 2 table Not enough information given to derive a 2 x 2 table. Only given for a sample of 50 participants who had a strong suspicion of melanoma clinically. Do not know what happened to those with no suspicion clinically
Lipoff 2008	Ineligible target condition Study does not differentiate MM from benign/other but looks to identify lesion characteristics that might help id those at risk for MM
Liu 2012	Derivation study Asymmetry detection; 10‐fold cross‐validation Insufficient data for 2 x 2 table
Lorentzen 2000	Ineligible target condition; does not present data for detection of BCC or cSCC
Luttrell 2012	Ineligible test observer Accuracy data only given for lay persons; this population of test observers is not eligible
Machet 2005	Wrong study population This is a staging study
MacKenzie‐Wood 1998	Wrong study population Only malignant diagnosis
MacKie 1971	Insufficient data for 2 x 2 table Only gives % with correct diagnosis rather than numbers misclassified as malignant
MacKie 1990	Not a primary study
MacKie 1991	Not a primary study Letter
MacKie 2002	Assesses individual lesion characteristics only Presence of 3 or more colours on dermoscopy
Mahendran 2005	Ineligible index test Face‐to‐face is 'clinical diagnosis', i.e. visual inspection ± use of dermoscopy
Mahon 1997	Not a primary study A summary of a comparison of two screening checklists
Malvehy 2014	Ineligible target condition; does not present data for detection of BCC or cSCC
Marghoob 1995	Not a primary study Letter
Marghoob 2007	Not a primary study
Marghoob 2010	Not a primary study
Massi 2001	Assesses individual lesion characteristics only
Mayer 1997	Not a primary study Systematic review
McCarthy 1995	Not a primary study Leaflet
McGovern 1992	Ineligible target condition; does not present data for detection of BCC or cSCC
Menzies 1996a	Ineligible target condition; does not present data for detection of BCC or cSCC
Menzies 1996b	Assesses individual lesion characteristics only Only given the SE/SP of individual characteristics; lesions make up the training set for Menzies 1996a (#1971)
Menzies 1999	Not a primary study
Menzies 2001	Ineligible index test Monitoring purposes
Menzies 2005	Ineligible target condition; does not present data for detection of BCC or cSCC
Menzies 2008	Ineligible target condition; does not present data for detection of BCC or cSCC
Menzies 2009	Ineligible target condition; does not present data for detection of BCC or cSCC
Menzies 2011	Ineligible index test Surveillance study; data used to id factors predictive of lesion changes
Menzies 2013	Ineligible target condition; does not present data for detection of BCC or cSCC
Moffatt 2006	Ineligible index test 'clinical diagnosis'
Mohammad 2015	Wrong study population Only includes BCC
Morales Callaghan 2008	Ineligible target condition; does not present data for detection of BCC or cSCC
Morrison 2001	Insufficient data for 2 x 2 table Study gives % correct diagnosis within each histology group and then gives the % ‘correct’ diagnosis of skin cancer as 22% for FP and 87% for dermatologist. But these statistics appear to have been reached by taking the mean of the % correct diagnoses across the malignant groups and do not equate to sensitivity, i.e. If you take the mean of the FP correct (%) for the 4 malignant groups you get: (40 + 22 + 25 + 0)/4 = 21.75% and then the same for the 'dermatologist correct' (%) column: (95 + 77 + 75 + 100)/4 = 86.75%
Morton 1998	Ineligible target condition; does not present data for detection of BCC or cSCC
Mun 2016	Ineligible reference standard Only 37% of benign group underwent adequate reference standard
Nachbar 1994	Ineligible target condition; does not present data for detection of BCC or cSCC
Nathansohn 2007	Insufficient data for 2 x 2 table Not test accuracy; follow‐up study
Nilles 1994	Ineligible target condition; does not present data for detection of BCC or cSCC
Osborne 1998	Ineligible reference standard Not clear what the ref standard is Insufficient data for 2 x 2 table
Osborne 1999	Wrong study population Only participants with melanoma included
Pagnanelli 2003	Ineligible target condition; does not present data for detection of BCC or cSCC
Pan 2008	Derivation study Looking to id characteristics assoc with superficial BCC; 2 x 2 could be extracted for combination of 3 selected characteristics. Dermoscopic features selected based on prior studies but only participants with 3 diagnoses included: BCC, intra‐ep carcinoma and psoriasis
Panasiti 2009	Assesses individual lesion characteristics only Ineligible reference standard Of the 1543 lesions analysed on 321 received histopathology diagnosis. The accuracy data is based on this (only 20%); unclear what happened to the 80% of participants as no mention of follow‐up
Parslew 1997	Wrong study population Not all suspected of skin cancer
Pazzini 1996	Insufficient data for 2 x 2 table
Pehamberger 1987	Insufficient data for 2 x 2 table Not test accuracy. This is a descriptive paper defining dermoscopic criteria. It is not a study testing accuracy of dermoscopy. From the authors final sign‐off it looks like part 2 of this paper may have details on accuracy(Steiner 1987).
Pellacani 2002	Not a primary study
Pellacani 2006	Derivation study Looks at detection of asymmetry between clinicians and computer Insufficient data for 2 x 2 table 2 x 2 could be derived for overall asymmetry or border cut‐off but not overall diagnosis
Pellacani 2007	Assesses individual lesion characteristics only Derivation study Looking at blue hue
Pellacani 2009	Ineligible target condition Focus is on identifying Spitz naevi from melanoma and ‘clark’ naevi and is looking to derive useful RCM characteristics. Although some data are given in the text for an RCM score > 3 it is difficult to work out which are FP and which FN
Perednia 1992	Insufficient data for 2 x 2 table Not test accuracy
Peris 2002	Wrong study population Only participants with BCC diagnosis included
Perrinaud 2007	Ineligible index test Does not provide data for visual inspection alone
Phan 2010	Insufficient data for 2 x 2 table Not test accuracy investigating dermoscopic features of acral melanoma including of the nail apparatus; no accuracy data given
Piccolo 2000	Ineligible target condition; does not present data for detection of BCC or cSCC
Piccolo 2002	Not a primary study Insufficient data for 2 x 2 table Not enough data to populate 2 x 2 table. No breakdown of index test results and ref standard
Piccolo 2002a	Ineligible target condition; does not present data for detection of BCC or cSCC
Piccolo 2004	Ineligible index test; include for teledermatology anyway
Piccolo 2006	Inadequate sample size 3 MMs, but also 1 lentigo and 14 dysplastic nevus; data not presented to allow se/sp estimation Assesses individual lesion characteristics only Derivation study Derivation for hypoluminescence microscopy
Piccolo 2014	Ineligible target condition; does not present data for detection of BCC or cSCC
Pizzichetta 2001a	Wrong study population Population in study only those with malignant disease
Pizzichetta 2001b	Insufficient data for 2 x 2 table Observer agreement only
Pizzichetta 2002	Ineligible target condition; does not present data for detection of BCC or cSCC
Pizzichetta 2004	Ineligible target condition; does not present data for detection of BCC or cSCC
Pizzichetta 2007	Wrong study population Only participants with melanoma included
Pizzichetta 2010	Inadequate sample size Case study
Pizzichetta 2013	Assesses individual lesion characteristics only Presence of negative pigmented network
Pralong 2012	Wrong study population Only melanoma participants included
Provost 1998	Insufficient data for 2 x 2 table Not test accuracy; only reports concordance
Pupelli 2013	Ineligible target condition; does not present data for detection of BCC or cSCC
Quéreux 2011	Ineligible index test Self‐administered questions to patients attending a GP surgery before their appointment to determine whether they are at high risk of melanoma, which is meant to highlight to the GP which patient to examine during their consultation
Rader 2014	Assesses individual lesion characteristics only Insufficient data for 2 x 2 table
Rajpara 2009	Not a primary study Systematic review
Rallan 2006	Ineligible index test No data can be extracted for visual inspection alone
Rampen 1988	Wrong study population Only melanoma included
Rao 1997	Ineligible target condition; does not present data for detection of BCC or cSCC
Reeck 1999	Wrong study population Only includes index test negatives, i.e. those considered benign by referring clinician Ineligible target condition
Reggiani 2015	Not a primary study Systematic review of kerationcyte skin cancer
Riddell 1961	Wrong study population All malignant
Rigel 1993	Not a primary study
Rigel 1997	Not a primary study
Rigel 2012	Ineligible target condition; does not present data for detection of BCC or cSCC
Robati 2014	Ineligible reference standard No follow‐up of patients not referred to dermatology clinics, who did not receive histopathology
Robinson 2010	Ineligible index test Self‐examination
Ronger 2002	Assesses individual lesion characteristics only
Rosado 2003	Not a primary study Systematic review
Rosendahl 2012a	Assesses individual lesion characteristics only
Rosendahl 2012b	Not a primary study
Rossi 2000	Ineligible reference standard Unclear reference standard in disease‐negative
Roush 1986	Ineligible target condition Only dysplastic naevus
Rubegni 2002	Not a primary study
Rubegni 2005	Not a primary study Editorial
Rubegni 2010	Derivation study Uses 'leave one out' procedure Insufficient data for 2 x 2 table
Rubegni 2012	Ineligible target condition; does not present data for detection of BCC or cSCC
Rubegni 2016	Ineligible target condition; does not present data for detection of BCC or cSCC
Sahin 2004	Assesses individual lesion characteristics only Insufficient data for 2 x 2 table No accuracy data given, study looking at dermoscopic features of LM
Saida 2002	Assesses individual lesion characteristics only Descriptive study looking at presence (%) of certain features. Not looking at accuracy. Has paragraph on diagnostic value of this specific feature quoting sens & spec but this is based upon unpublished observations and the data are not given in this paper
Saida 2004	Assesses individual lesion characteristics only
Sakakibara 2010	Assesses individual lesion characteristics only Only looking at different vascular structures
Salerni 2011	Inadequate sample size < 5 cases
Salerni 2012	Ineligible index test Surveillance study Insufficient data for 2 x 2 table
Salerni 2013	Not a primary study Systematic review of surveillance with digital dermoscopy
Salvio 2011	Not a primary study Inadequate sample size
Sanchez‐Martin 2012	Wrong study population Only BCC cases
Savk 2004	Not a primary study Letter
Sawada 2013	Not a primary study
Sboner 2003	Derivation study Describes 10‐fold cross‐validation process for training/testing classifier
Sboner 2004	Ineligible target condition; does not present data for detection of BCC or cSCC
Schindewolf 1994	Ineligible index test Evaluates CAD not VI
Schmoeckel 1987	Not a primary study
Schulz 2001	Ineligible target condition Melanoma metastases
Scope 2008	Ineligible target condition; does not present data for detection of BCC or cSCC
Scope 2015	Not a primary study
Segura 2009	Ineligible index test; RCM evaluation
Seidenari 1998	Ineligible target condition; does not present data for detection of BCC or cSCC
Seidenari 2004	Insufficient data for 2 x 2 table No data to populate 2 x 2 table, just ROC curve values given Contact authors TABLE 5 provides AUC values for each diagnosis for both formats and observers; we are particularly interested in accuracy for the diagnosis of melanoma, are you able to provide data in 2 x 2 format, e.g. for melanoma 'certain' against final diagnosis and for melanoma 'certain or fairly certain' against final diagnosis? (no reply from authors)
Seidenari 2005	Ineligible target condition; does not present data for detection of BCC or cSCC
Seidenari 2006a	Wrong study population Assessing best means of follow‐up in patients with previous melanoma ‐ total body exam versus only lesions > 2 cm. No melanoma identified
Seidenari 2006b	Assesses individual lesion characteristics only Looks like this study is only looking at asymmetry judgement
Seidenari 2007	Ineligible target condition; does not present data for detection of BCC or cSCC
Seidenari 2012	Assesses individual lesion characteristics only Looks at individual lesion characteristics to distinguish melanoma in situ, also gives mean ABCD and 7‐point scores Insufficient data for 2 x 2 table Contact authors Table 3 provides mean ABCD and 7‐point checklist scores, are you able to provide us with a cross‐tabulation of results with each checklist at 'standard' thresholds against final diagnosis? e.g. ABCD > 4.75 and > 5.45 for MIS and benign groups 7‐point checklist: presence of 2or more characteristics and 3 or more characteristics? (no reply)
Seidenari 2013	Ineligible index test
Serrao 2006	Ineligible index test; include for CAD review only
Sgouros 2014	Ineligible index test; include for CAD review only
Shakya 2012	Ineligible target condition SCC in situ is not included in target condition
Shariff 2010	Ineligible reference standard
Shitara 2014	Assesses individual lesion characteristics only
Shitara 2015	Wrong study population Includes only melanoma
Skvara 2005	Ineligible target condition; does not present data for detection of BCC or cSCC
Sondak 2015	Not a primary study Comment paper
Soyer 1987	Insufficient data for 2 x 2 table Not test accuracy
Soyer 1995	Ineligible target condition; does not present data for detection of BCC or cSCC
Soyer 2001	Not a primary study Editorial
Soyer 2004	Ineligible target condition; does not present data for detection of BCC or cSCC
Stanganelli 1998a	Ineligible target condition; does not present data for detection of BCC or cSCC
Stanganelli 1998b	Insufficient data for 2 x 2 table Cannot derive specificity; only gives 'exact diagnoses' for MM and 2 benign categories and not number benign misdiagnosed as MM
Stanganelli 1999	Ineligible target condition; does not present data for detection of BCC or cSCC
Stanganelli 2005	Ineligible target condition; does not present data for detection of BCC or cSCC
Stanganelli 2015	Ineligible target condition; does not present data for detection of BCC or cSCC
Stanley 2003	Assesses individual lesion characteristics only Fuzzy histogram is based on the lesion's colour, which is an individual lesion characteristic
Stathopoulos 2015	Insufficient data for 2 x 2 table Only includes index test‐positive participants, i.e. no FN or TN results
Steiner 1993	Assesses individual lesion characteristics only Derivation study
Stephens 2013	Inadequate sample size
Stoecker 2009	Derivation study Translucency Insufficient data for 2 x 2 table Data presented only as ROC curve and AUC
Stoecker 2011	Assesses individual lesion characteristics only Derivation study Uses 'leave one out' procedure Insufficient data for 2 x 2 table Data presented only as ROC curve and AUC
Stolz 1994	Ineligible target condition; does not present data for detection of BCC or cSCC
Stolz 2002	Not a primary study
Stratigos 2007	Ineligible reference standard Insufficient data for 2 x 2 table
Stricklin 2011	Assesses individual lesion characteristics only
Strumia 2003	Conference abstract; letter only
Tan 2009	Ineligible target condition; does not present data for detection of BCC or cSCC
Tandjung 2015	Ineligible target condition 'Malignant' includes: AK, Bowen's, dysplastic naevus, lentigo maligna, SCC, BCC, MM, keratoacanthoma Ineligible index test GPs sent images for telederm opinion; then free to send for biopsy or not; results shown are only for those that wer biopsied, according to TD advice
Tasli 2012	Not a primary study Systematic review looking at frequency of publications ion dermoscopy
Teban 2003	Wrong study population Classification of Clark naevi into 12 types Insufficient data for 2 x 2 table No 2 x 2 data; classification of Clark naevi into 12 types
Tenenhaus 2010	Ineligible target condition; does not present data for detection of BCC or cSCC
Terrill 2009	Ineligible index test Whole‐body skin examination after participants referred on for further assessment by a specialist Insufficient data for 2 x 2 table
Terstappen 2007	Wrong study population Includes only BCC ‐ looking for BCC characteristics on Siascope Derivation study Derivation study; first application of Siascope to pigmented BCC; 21/25 lesions were BCCs
Terushkin 2010a	Inadequate sample size Only 2 invasive SCCs Insufficient data for 2 x 2 table
Terushkin 2010b	Insufficient data for 2 x 2 table Not test accuracy ‐ reports final diagnoses of those excised over a number of time periods and benign‐malignant ratio
Thomas 1998	Ineligible target condition; does not present data for detection of BCC or cSCC
Thomson 2005	Not a primary study Letter
Torrey 1941	Ineligible target condition Includes non‐cutaneous lesions
Tromme 2012	Ineligible reference standard Inadequate reference test for disease‐negatives; expert dx only
Troyanova 2003	Ineligible target condition; does not present data for detection of BCC or cSCC
Tschandl 2012	Ineligible index test Differentiating melanocytic from non‐melanocytic lesions
Tschandl 2015	Ineligible test observer Medical students
Unlu 2014	Ineligible target condition; does not present data for detection of BCC or cSCC
Van der Leest 2011	Ineligible reference standard Inadequate reference test for test‐negatives; expert dx only
Van der Rhee 2010	Ineligible reference standard < 50% of disease‐negative have an adequate reference standard
Van der Rhee 2011	Inadequate sample size < 5 cases
Vasili 2010	Conference abstract
Verduzco‐Martinez 2013	Wrong study population Only BCC
Vestergaard 2008	Not a primary study Systematic review; check reference list
Viglizzo 2004	Ineligible target condition; does not present data for detection of BCC or cSCC
Wagner 1985	Insufficient data for 2 x 2 table
Walter 2010	Not a primary study Clinical trial protocol
Walter 2012	Ineligible target condition; does not present data for detection of BCC or cSCC
Walter 2013	Ineligible reference standard Final diagnosis reached by histology or expert opinion; no follow‐up of non‐excised lesions reported in this paper. Walter 2012 does report follow‐up for enough benign lesions for control arm (weighted 7PCL) data to be included. Authors contacted and confirmed calculations (02 March 2016)
Wang 2008	Insufficient data for 2 x 2 table Not test accuracy; no details of misdiagnoses of benign lesions as malignant
Warshaw 2009a	Insufficient data for 2 x 2 table Duplicate or related publication. Subgroup of participants from Warshaw 2010a Contact authors Study presents diagnostic accuracy of teledermatology and clinic diagnosis in comparison to histopathology; we need the underlying 2 x 2 contingency tables (see Warshaw 2010a for author response)
Warshaw 2009b	Insufficient data for 2 x 2 table Duplicate or related publication. Subgroup of participants from Warshaw 2010a Contact authors Study presents diagnostic accuracy of teledermatology and clinic diagnosis in comparison to histopathology; we need the underlying 2 x 2 contingency tables (see Warshaw 2010afor author response)]
Warshaw 2010a	Insufficient data for 2 x 2 table Contact authors Study presents diagnostic accuracy of teledermatology and clinic diagnosis in comparison to histopathology. Author only able to provide numbers test‐positive and ‐negative for melanoma and not for the final 2 cells of the 2 x 2; data provided showed higher sensitivity for melanoma as the primary diagnosis rather than as the ‘aggregate’ diagnosis and the 2 x 2 using the authors' data and the accuracy figures from the paper showed more T+ from the primary diagnosis as opposed to the aggregate
Warshaw 2010b	Insufficient data for 2 x 2 table As per Warshaw 2009a; this 2010 paper presents combined data for pigmented and nonpigmented lesions
Weismann 2002	Not a primary study
Wells 2012	Ineligible target condition; does not present data for detection of BCC or cSCC
Westbrook 2006	Insufficient data for 2 x 2 table
Westerhoff 2000	Ineligible target condition; does not present data for detection of BCC or cSCC
Whitaker‐Worth 1998	Wrong study population Ineligible test observer Mixed medical student/clinicians Insufficient data for 2 x 2 table Not test accuracy study
Whited 1998	Inadequate sample size
Wilkes 2010	Not a primary study
Williams 1991	Insufficient data for 2 x 2 table
Winkelmann 2015a	Duplicate or related publication.
Winkelmann 2015b	Duplicate or related publication.
Winkelmann 2016	Ineligible target condition; does not present data for detection of BCC or cSCC
Wolf 1998	Ineligible index test Clinical diagnosis study; test clearly described ‐ "concerning the clinical diagnosis, we were not able to ascertain from the clinical data sheet whether the referring physicians used additional diagnostics techniques such as dermoscopy"
Yadav 1993	Insufficient data for 2 x 2 table Not test accuracy
Yamaura 2005	Derivation study Gene amplification in acral lesions
Yelamos 2016	Not a primary study. Commentary on Guitera 2016
Yoo 2015	Conference abstract
Youl 2007a	Ineligible index test; evaluates 'clinical diagnosis' Contact authors; author replied ‐ dermoscopy used in some but not all lesions
Youl 2007b	Ineligible index test; evaluates 'clinical diagnosis' Contact authors; author replied ‐ dermoscopy used in some but not all lesions
Zaballos 2013	Wrong study population They do not have enough benign cases to include as full report
Zalaudek 2010	Not a primary study Editorial
Zaumseil 1983	Ineligible target condition; does not present data for detection of BCC or cSCC
Zell 2008	Inadequate sample size Case study
Zortea 2014	Derivation study Although data are divided into training and test sets, the test set data are used more than once over 20 realisations of each model, especially the melanomas, for which the same 10 are used in each realisation
Zou 2001	Not a primary study Study uses results from Stolz 1994 Insufficient data for 2 x 2 table Just showing ROC curves

7PCL ‐ 7‐point checklist; AK ‐ actinic keratosis; BCC ‐ basal cell carcinoma; CAD ‐ computer‐assisted diagnosis; D+ ‐ disease positive; Dx ‐ diagnosis; FN ‐ false negative; FP ‐ false positive; LM ‐ lentigo meligna; MM ‐ malignant melanoma; NPV ‐ negative predictive value; PCP ‐ primary‐care physician; PPV ‐ positive predictive value; ROC: receiver operating characteristic; se ‐ sensitivity; SCC ‐ squamous cell carcinoma; sp ‐ specificity; VI ‐ visual inspection.

Differences between protocol and review

The proposed primary objective to analyse studies according to the prior testing undergone by study participants (comparing those with limited prior testing with those referred for further evaluation of a suspicious skin lesion) was not possible due to limited data.

The primary objectives were also amended to conduct separate analyses by in‐person/image‐based diagnosis rather than to investigate the effect on accuracy as a secondary objective, as originally proposed in the generic protocol. We took this decision very early in the review process and based it on the fact that a diagnosis based on a dermoscopic image or clinical photograph cannot approximate the context of a face‐to‐face patient/clinician consultation, and was not based on observed results.

We expanded the secondary objectives for the detection of BCC or cSCC to include: test comparisons restricted to studies where both tests were evaluated in the same studies (direct test comparisons); and investigations of the accuracy of individual algorithms used to assist visual inspection or dermoscopy, and any effect from observer experience on diagnostic accuracy.

The secondary objective has been changed from "for the detection of any skin cancer" to "for the detection of any skin cancer in adults, where keratinocyte skin cancers make up at least 50% of included skin cancers" in order to keep the focus on keratinocyte skin cancers for this review and in order not to replicate analyses conducted for the review of RCM for melanoma. These changes also affect the definition of the secondary target condition in the Methods section.

Sources of heterogeneity that could be investigated were restricted due to lack of data.

We amended the text to clarify that studies available only as conference abstracts would be excluded from the review unless full papers could be identified; studies available only as conference abstracts do not allow a comprehensive assessment of study methods or methodological quality.

We clarified the participant inclusion criteria to make it clear that studies of only malignant or benign lesions would be excluded.

To improve clarity of methods, this text from the protocol “We will include studies developing new algorithms or methods of diagnosis (i.e. derivation studies) if they use a separate independent ’test set’ of participants or images to evaluate the new approach.We will also include studies using other forms of cross validation, such as ’leave‐one‐out’ cross‐validation (Efron 1983). We will note for future reference (but not extract) any data on the accuracy of lesion characteristics individually, e.g. the presence or absence of a pigment network or detection of asymmetry.”

has been replaced with “Studies developing new algorithms or methods of diagnosis (i.e. derivation studies) were included if they:

• used a separate independent 'test set' of participants or images to evaluate the new approach, or

• investigated lesion characteristics that had previously been suggested as associated with melanoma and the study reported accuracy based on the presence or absence of particular combinations of characteristics.

Studies were excluded if they:

• used a statistical model to produce a data driven equation, or algorithm based on multiple diagnostic features, with no separate test set

• used cross‐validation approaches such as 'leave‐one‐out' cross‐validation (Efron 1983)

• evaluated the accuracy of the presence or absence of individual lesion characteristics or morphological features, with no overall diagnosis of malignancy

• reported accuracy data for ‘clinical diagnosis’ with no clear description as to whether the reported data related to visual inspection alone or included dermoscopy in all study participants

• were based on the experience of a skin cancer‐specific clinic, where dermoscopy may or may not have been used on an individual patient basis."

We proposed to supplement the database searches by searching the annual meetings of appropriate organisations (e.g. British Association of Dermatologists Annual Meeting, American Academy of Dermatology Annual Meeting, European Academy of Dermatology and Venereology Meeting, Society for Melanoma Research Congress, World Congress of Dermatology, European Association of Dermato Oncology), but due to the volume of evidence retrieved from database searches and time restrictions we were unable to do this.

As per the change to secondary objectives, this text from the protocol "For our secondary objective, the target condition will include any skin lesion requiring excision. We will include studies reporting data for keratinocyte skin cancer combined, and not differentiated according to BCC or cSCC, in this analysis, along with any melanoma or rare skin cancer (e.g. Merkel or amelanotic melanoma) that may be detected. We will not consider in situ cancers or actinic keratosis as disease‐positive" has been changed to:

"An additional definition of the target condition was considered in secondary analysis, the detection of:

• any skin cancer, including BCC, cSCC, melanoma, or any rare skin cancer (e.g. Merkel cell cancer), as long as skin cancers other than melanoma made up more than 50% of the disease positive group. Data from studies in which melanoma accounted for more than 50% of skin cancers were included in the reviews of visual inspection and dermoscopy with and without visual inspection for the diagnosis of melanoma (Dinnes 2018a; Dinnes 2018b)."

For quality assessment, we further tailored the QUADAS‐2 tool according to the review topic.

In terms of analysis, we did not restrict analysis of per‐patient data, due to lack of data. We did not perform heterogeneity investigations or sensitivity analyses as planned, due to lack of data.

Contributions of authors

JD was the contact person with the editorial base.
 JD co‐ordinated contributions from the co‐authors and wrote the final draft of the review.
 SB conducted the literature searches.
 JD, NC, LJ, KYW, RBA, AD, AG and SAC screened papers against eligibility criteria.
 JD and NC obtained data on ongoing and unpublished studies.
 JD, NC, LJ, KYW, RBA, AD, AG and SC appraised the quality of papers.
 JD, NC, LJ, KYW, RBA, AD, AG and SC extracted data for the review and sought additional information about papers.
 JD and NC entered data into RevMan.
 JD, JLB and JJD analysed and interpreted data.
 JD, JJD, NC, JJB, YT and CD worked on the methods sections.
 JD, AJ, FW, LJ, KYW, RBA, AD, AG, SC, RNM, HT, and HCW drafted the clinical sections of the background and responded to the clinical comments of the referees.
 JD, JJD, CD, and YT responded to the methodology and statistics comments of the referees.
 CO was the consumer co‐author and checked the review for readability and clarity, as well as ensuring outcomes are relevant to consumers.
 JD is the guarantor of the update.

Disclaimer

This project presents independent research supported by the National Institute for Health Research, via Cochrane Infrastructure funding to the Cochrane Skin Group and Cochrane Programme Grant funding, and the NIHR Birmingham Biomedical Research Centre at the University Hospitals Birmingham NHS Foundation Trust and the University of Birmingham. The views and opinions expressed therein are those of the authors and do not necessarily reflect those of the Systematic Reviews Programme, NIHR, NHS or the Department of Health and Social Care.

Sources of support

Internal sources

• No sources of support supplied

External sources

• The National Institute for Health Research (NIHR), UK.

  The NIHR, UK, is the largest single funder of the Cochrane Skin Group

• NIHR Systematic Review Programme, UK.

  This project was funded by an NIHR Cochrane Systematic Reviews Programme Grant (13/89/15)

• NIHR clinical fellowship, UK.

• NIHR Birmingham Biomedical Research Centre, UK.

  JD, JJD and YT receive support from the NIHR Birmingham Biomedical Research Centre

Declarations of interest

Jacqueline Dinnes: nothing to declare.
 Jonathan J Deeks: nothing to declare.
 Naomi Chuchu: nothing to declare.
 Rubeta N Matin: "my institution received a grant for a Barco NV commercially sponsored study to evaluate digital dermoscopy in the skin cancer clinic. My institution also received Oxfordshire Health Services Research Charitable Funds for carrying out a study of feasibility of using the Skin Cancer Quality of Life Impact Tool (SCQOLIT) in non melanoma skin cancer. I have received royalties for the Oxford Handbook of Medical Dermatology (Oxford University Press). I have received payment from Public Health England for the "Be Clear on Cancer" skin cancer report. I have no conflicts of interest to declare that directly relate to the publication of this work."
 Kai Yuen Wong: nothing to declare.
 Roger Benjamin Aldridge: nothing to declare.
 Alana Durack: nothing to declare.
 Abha Gulati: nothing to declare.
 Sue Ann Chan: nothing to declare.
 Louise Johnston: nothing to declare.
 Susan E Bayliss: nothing to declare.
 Jo Leonardi‐Bee: nothing to declare.
 Yemisi Takwoingi: nothing to declare.
 Clare Davenport: nothing to declare.
 Colette O'Sullivan: nothing to declare.
 Hamid Tehrani: nothing to declare.
 Hywel C Williams: I am director of the NIHR HTA Programme. HTA is part of the NIHR which also supports the NIHR systematic reviews programme from which this work is funded.

Edited (no change to conclusions)