Carrera 2016.

Study characteristics
Patient sampling	Study design: case‐control Data collection: retrospective image selection/prospective interpretation. Each PLC provided up to 50 lesions with a 1:3 ratio of melanomas to naevi. Each contributor randomly selected either polarised or non‐polarised images based on 1:1 randomisation. Following exclusions, lesions were randomised into 12 image sets containing 39 (n = 8) or 40 (n = 7) unique lesions and 5 non‐unique lesion images (2 melanoma, 3 benign) that were repeated in all sets. Period of data collection: NR Country: multicentre (images contributed from PLCs in Australia, Austria, Germany, Italy, Spain, Switzerland, and the USA)
Patient characteristics and setting	Inclusion criteria: images of melanocytic lesions including melanomas with an unequivocal histopathologic diagnosis, and histopathologically verified naevi or naevi demonstrating stability under sequential dermoscopic imaging over time. Setting: specialist unit (skin cancer/PLC) 12 PLCs Prior testing: selected for excision (no further detail) Setting for prior testing: specialist unit (skin cancer/PLC) Exclusion criteria: acral, mucosal, or facial sites excluded; non‐melanocytic appearance; lesions with equivocal (final) diagnosis after review of the pathology report or sequential imaging Sample size (participants): number eligible: NR; number included: NR Sample size (lesions): number eligible: 580 lesion images were contributed; number included: 477 (103 excluded on review by Memorial Sloan Kettering Cancer Center investigators) Participant characteristics: none reported Lesion characteristics: none reported
Index tests	Dermoscopy 3‐point rule; 7PCL; ABCD; Menzies criteria; chaos and clues Method of diagnosis: dermoscopic images Prior test data: clinical image, evaluators examined the close‐up clinical image of each lesion before viewing the dermoscopic image; image contributors also asked to provide information on anatomical location, patient age and sex, and imaging modality (polarised vs non‐polarised) but unclear whether this information was provided to observers or not. Diagnostic threshold: observers asked to evaluate "a comprehensive list" of dermoscopic structures abstracted from various algorithms; overlapping criteria were merged into 1 criterion. Criteria were grouped into (1) global pattern, (2) pattern organization, (3) symmetry of contour, (4) symmetry of pattern, (5) architectural disorder, (6) abruptness of lesion border, (7) colours, and (8) melanocytic structures, including network and vascular structures. Algorithm performance was retrospectively assessed based on the following thresholds: 7PCL ≥ 3; CASH ≥ 6; Menzies NR; ABCD > 4.75; 3PCL NR; chaos and clues NR. For NR thresholds, author communications state "We assessed all of the algorithms. There isn't a threshold for these algorithms, there are just published rules of usage that determine the benign/malignant classification of the lesion" Diagnosis based on: consensus (≥ 50%); when ≥ 50% of the observers identified a dermoscopic feature for a given study lesion, the attribute was considered present; n = 130 (240 participants registered via the IDS website for the study; 103 completed all available images in their data sets and 130 evaluated ≥ 20 lesions) Observer qualifications: GP 24; dermatology registrar 25; dermatologist 73; 1 medical student and 7 'other' Experience in practice: mixed; mean 12 (SD 8.7) years of dermatology experience Experience with dermoscopy: mixed; 122 (93.8%) reported being comfortable using dermoscopy, and 121 (93.1%) were regular users of dermoscopy Dermoscopy training: algorithm tutorials were created and posted by dermoscopic experts through the IDS website; review of these was encouraged but not mandatory.
Target condition and reference standard(s)	Reference standard: histological diagnosis + follow‐up Histology: all melanomas (n = 119) and a proportion of benign lesions (n = NR) Clinical follow‐up + histology of suspicious lesions: sequential dermoscopic imaging over time; not further detailed; length of follow‐up NR; naevi required to be either histopathologically verified or to have demonstrated stability under sequential dermoscopic imaging over time. Target condition (final diagnoses) Melanoma (in situ and invasive, or NR): 119 Benign naevus: melanocytic naevus: 358
Flow and timing	Excluded participants: poor quality index test image as exclusion criterion Time interval to reference test: NR Time interval between index test(s): in‐person; sequential
Comparative
Notes	‐
Methodological quality
Item	Authors' judgement	Risk of bias	Applicability concerns
DOMAIN 1: Patient Selection
Was a consecutive or random sample of patients enrolled?	Yes
Was a case‐control design avoided?	No
Did the study avoid inappropriate exclusions?	No
Are the included patients and chosen study setting appropriate?	No
Did the study avoid including participants with multiple lesions?	Unclear
		High	High
DOMAIN 2: Index Test Dermoscopy ‐ image‐based
Were the index test results interpreted without knowledge of the results of the reference standard?	Yes
If a threshold was used, was it pre‐specified?	Yes
For studies reporting the accuracy of multiple diagnostic thresholds, was each threshold or algorithm interpreted without knowledge of the results of the others?	Yes
Was the test applied and interpreted in a clinically applicable manner?	No
Were thresholds or criteria for diagnosis reported in sufficient detail to allow replication?	Yes
Was the test interpretation carried out by an experienced examiner?	Unclear
		Low	High
DOMAIN 3: Reference Standard
Is the reference standards likely to correctly classify the target condition?	Unclear
Were the reference standard results interpreted without knowledge of the results of the index tests?	Unclear
Expert opinion (with no histological confirmation) was not used as a reference standard	Yes
Was histology interpretation carried out by an experienced histopathologist or by a dermatopathologist?	Unclear
		Unclear	Unclear
DOMAIN 4: Flow and Timing
Was there an appropriate interval between index test and reference standard?	Unclear
Did all patients receive the same reference standard?	No
Were all patients included in the analysis?	No
If the reference standard includes clinical follow‐up of borderline/benign appearing lesions, was there a minimum follow‐up following application of index test(s) of at least: 3 months for melanoma or cSCC or 6 months for BCC?	Unclear
If more than one algorithm was evaluated for the same test, was the interval between application of the different algorithms 1 month or less?	Yes
		High