| Study | Reason for exclusion |
|---|---|
| Armstrong 2007 | Exclude not a primary study |
| Baba 2005 | Exclude on study population |
| Badertscher 2015 | Exclude on 2×2 data; for VI/Derm – only gave number of correct diagnoses (not broken down by TP/TN) and gives GP 'score' between T0 and T1 Exclude on index test – not a teledermatology study |
| Barnard 2000 | Exclude on target condition; the primary target condition was not relevant to our reviews. |
| Bashshur 2015 | Exclude not a primary study; narrative review |
| Bataille 2011 | Exclude conference abstract |
| Bergmo 2000 | Exclude not a primary study |
| Borve 2013 | Exclude on target condition; included 1 melanoma metastases and 1 in situ SCC as D+ (these made up 5% of malignant group); author contacted ("Table 2 provided estimates of the diagnostic accuracy of the face‐to‐face dermatologist and the two teledermoscopists, however in order to include the results in our review we would need the underlying 2×2 contingency tables for these statistics. Is it at all possible for you to provide us with them for each observer, particularly in regard to the 'Primary diagnosis' and the 'Benign vs malignant'") |
| Boyce 2011 | Exclude on study population; no breakdown given, just 7 suspicious lesions. Exclude on reference standard; no data given but only 7 referred on for formal assessment. Definitely < 50% histology rate |
| Braun 2000 | Exclude on 2×2 data Exclude but contacted authors. Sensitivty was reported in Table II but specificity unclear. Study reports % of correct identification of each lesion type rather than FPs and does not provide numbers misclassified as melanoma, or other malignancy. Also possibility of overlap with later publications, e.g. Coras 2003 |
| Brown 2000 | Exclude not a primary study |
| Burgiss 1997 | Exclude on reference standard reports actions on telediagnosis (primarily for cost purposes) and did not give final diagnoses (histo) or FTF diagnosis for all lesions |
| Chen 2002 | Exclude on sample size Study was based on individual participant images (< 5 cases). Only 4 lesions in total used in study (1 non‐BCC, 1 SK, 1 KA, 1 AK) |
| D'Elia 2007 | Exclude on study population; not focused on suspected skin cancer Exclude on 2×2 data; looked at agreement between diagnoses only |
| Di Stefani 2007 | Exclude on sample size; < 5 malignant: of 7 excised: 1 melanoma (MiS associated with a nevus), 1 pigmented BCC and 5 melanocytic nevi. Both tele‐dx recommended the 2 malignancies for excision but no further breakdown of agreement/disagreement given Exclude on 2×2 data 7 'D+' according to FTF decision to excise but only reports kappa values for agreement with tele‐dx |
| Du Moulin 2003 | Exclude on study population Exclude on target condition |
| Edison 2008 | Exclude on study population |
| Eminovic 2009 | Exclude, not a primary study Exclude on study population Exclude on target condition |
| Fabbrocini 2008 | Exclude on 2×2 data; there was insufficient data provided for each index test to populate 2×2 table Exclude but contacted authors: "As we can only include DTA studies – do you have a cross tabulation of each clinician's diagnosis (e.g. at threshold of >=3 on 7 point checklist) against the histological diagnosis and/or a cross tabulation of the remote diagnosis against the Face to Face diagnoses?" |
| Ferrandiz 2007 | Exclude on study population; all had a strong clinical diagnosis of NMSC or fast‐growing tumour, the purpose of the test was primarily to inform treatment plans. Exclude on 2×2 data; 4 of the original 134 appeared to have missing histology and did not know what the additional clinical cases of each lesion were classed as on histology, e.g. the 14 extra BCCs and 5 KAs. |
| Ferrandiz 2012 | Exclude on target condition; on included CM |
| Gilmour 1998 | Exclude on study population Exclude on target condition |
| Granlund 2003 | Exclude on study population |
| Griffiths 2010 | Exclude on reference standard |
| Harrison 1998 | Exclude on target condition, no breakdown of final diagnoses (histo) or of recommendations from FTF consultation Exclude on sample size; could not determine how many D+ for either reference standard Exclude on 2×2 data; no underlying data provided; diagnostic 'accuracy' of teleconsultation reported as 71% for 210 participants compared to 49% for the referring GPs (49%). Also stated that "telemedicine was able to detect malignancies in 94% of cases compared with only 70% detected by general practitioners." |
| Heffner 2009 | Exclude on target condition |
| Hicks 2003 | Exclude not a primary study |
| High 2000 | Exclude on study population; not all suspected of skin cancer 'dermatological conditions' including dermatitis, acne, verruca, etc. Exclude on 2×2 data; reported agreement only; no breakdown of diagnoses/management decisions Exclude on reference standard; table of final diagnoses appeared to be based on FTF diagnoses with histology obtained for 69/106 (65%), so was ineligible for a tele‐dx vs histo comparison (diagnostic accuracy) and is not tele‐dx vs FTF (referral accuracy) either. |
| Hue 2016 | Exclude on reference standard; final diagnoses/FTF decisions given only for 17 recommended for rapid referral on tele‐dx; no data for remaining 395 with non‐urgent derm referral/annual FU recommendation/discharge. Exclude on sample size; for tele‐dx vs histo, only 5 excised included 1 melanoma Exclude on 2×2; no data for tele‐dx vs FTF |
| Hwang 2014 | Exclude on study population |
| Ishioka 2009 | Exclude on target condition No breakdown of disease positive 'malignant' provided |
| Kahn 2013 | Exclude on target condition |
| Knol 2006 | Exclude on study population |
| Krupinski 1999 | Exclude on 2×2 data; reported only diagnostic concordance/agreement |
| Lamel 2012 | Exclude on 2×2 data; reported only summary concordance in diagnosis and management decisions between decisions based on mobile phone image and in person; insufficient detail to work out referral accuracy Exclude but contacted authors,"Study presents data on diagnostic and management concordance between in person and remote (via mobile phone app) diagnoses, are any diagnostic accuracy data available, e.g. observers diagnosis of malignant lesion when assessed remotely versus FtF diagnosis of malignancy?" |
| Lamminen 2000 | Exclude not a primary study |
| Lesher 1998 | Exclude on study population; included people with 'skin problems' (included wide range of diagnoses) Exclude on sample size; could extract referral accuracy (tele‐dx vs FTF) but only 4 lesions with malignant diagnosis on FTF (4 BCCs) |
| Lewis 1999 | Exclude on 2×2 data; study appeared to meet all eligibility criteria but disease prevalence not given alongside sensitivity/specificity Exclude but contacted authors: "(Sensitivity and specificity of remote diagnosis in comparison to FtF diagnosis are provided but we would need number D+ in order to complete 2×2 table)." |
| Loane 1998a | Exclude on study population |
| Loane 1998b | Exclude not a primary study |
| Loane 2000 | Exclude on study population; not focused on potentially malignant skin lesions; could not derive any comparative data for detection of 'tumours' |
| Loane 2001a | Exclude not a primary study |
| Loane 2001b | Exclude on study population |
| Lowitt 1998 | Exclude on study population |
| Lyon 1997 | Exclude on 2×2 data: for operative referrals, only histo diagnosis given with overall number of disagreements by FTF and tele‐dx; no clear breakdown of index test results to derive 2×2 Exclude on study population; not all suspected of skin cancer |
| Martinez‐Garcia 2007 | Exclude on 2×2 data; not test accuracy Exclude on reference standard; no reference standard reported; describes only tele‐dx |
| Massone 2007 | Exclude on reference standard; did not meet our criteria for diagnostic accuracy reference standard (only 12/25 (48%) of benign group with histo, including 1 AK as benign instead of malignant); and data not presented to allow extraction of referral accuracy (tele‐dx vs FTF) (included 955 lesions and reported tele‐dx recommendations for all 955; 121 were recommended for excision or for FTF consult but FU data only available for 32 of these (19 with histo dx and 13 with FTF diagnosis)) Exclude on 2×2 data; data not clearly presented to allow extraction of referral accuracy (tele‐dx vs FTF alone); could only extract tele‐dx vs FTF diagnosis of malignancy if we assume that all 7 malignant lesions were diagnosed as such by FTF; from table 2 we only know that all 7 were excised presumably following FTF consult. |
| May 2008 | Exclude on 2×2 data: no data for a 2×2 table Exclude on index test; effect on consultant priority of GP referral with photograph vs referral with no accompanying photograph |
| McGraw 2009 | Exclude on study population |
| McManus 2008 | Exclude on study population |
| Moreno‐Ramirez 2006 | Exclude on sample size; comparison is tele‐dx vs final diagnosis (58/61 were biopsied); only 1 melanoma and 2 BCC |
| Moreno‐Ramirez 2007 | Exclude on reference standard: did not meet either criteria for eligible reference standard. Table 1 cross‐tabulates the telediagnosis (refer/not refer) against a gold standard which appeared to be a combination of the FTF diagnosis (for those with clinically and dermoscopically benign and non‐suspicious lesions, and with a diagnostic confidence level of 3 after the FTF diagnosis) and histology (those with higher concern at FTF evaluation were excised). |
| Moreno‐Ramirez 2009 | Exclude on study population Exclude on reference standard |
| Ndegwa 2010 | Exclude not a primary study; technology report |
| Nordal 2001 | Exclude on study population Exclude on target condition |
| Oakley 1997 | Exclude on study population Exclude on target condition |
| Oakley 2006 | Exclude on 2×2 data; insufficient data presented Exclude but contacted authors, "We are looking to compare telederm dx with FtF diagnosis within a diagnostic accuracy framework (i.e. in a 2×2 contingency table) but in order to include your paper we would need information on the misdiagnoses. Using the FtF diagnosis as the gold standard, we can use the data in Table 2 to derive the 'sensitivity' of the tele‐Dx agreement for diagnosis of melanoma, BCC and SCC (%agreement), but we would need to know the % of tele‐dx reports that 'misdiagnosed' the other lesion types as malignant in order to derive 'specificity'. Would you be at all able to supply this data? We could use the data in Table 6 to cross‐tabulate the management decisions between the two approaches, if we collapse the tele‐Dx cat3 and cat4 groups together. However the % agreement for the teledermatology classification adds to greater than 100." |
| Oztas 2004 | Exclude on study population Exclude on target condition |
| Pak 1999 | Exclude not a primary study; review of service |
| Pak 2002 | Exclude conference abstract; no full text paper found |
| Pak 2003a | Exclude on study population |
| Pak 2003b | Exclude on study population |
| Pak 2007 | Exclude on reference standard; reference standard not clear |
| Pak 2009 | Exclude not a primary study |
| Patro 2015 | Exclude on target condition |
| Perednia 1998 | Exclude on study population; not all suspected of skin cancer Exclude on index test; not an evaluation of accuracy of teledermatology but of effect of access to telemedicine on GP confidence and referral decisions |
| Phillips 1997 | Exclude on sample size; 68 lesions; FTF diagnosis of 4 skin cancers (tele‐ and FTF diagnoses concordant for 1 melanoma and 2 BCC; FTF dx of 1 additional SCC in 1 further participant); no final diagnoses (histo) recorded Exclude on 2×2 data; presents agreement between observers only Exclude on study population; not all suspected of skin cancer |
| Piccolo 1999 | Exclude on 2×2 data; reports data for tele‐dx, FTF and histo; breakdown of discordant results between tele‐dx vs FTF and tele‐dx vs histo is given but only gives number (%) concordant; did not give number TP and TN to allow 2×2 to be estimated for either comparison |
| Piccolo 2002 | Exclude not a primary study; review article |
| Rashid 2003 | Exclude on study population |
| Ribas 2010 | Exclude on study population |
| Romero 2010 | Exclude on study population |
| Romero 2014 | Exclude on study population |
| Seidenari 2004 | Exclude on 2×2 data; no data to populate 2×2 table just ROC curve values given. Exclude but contacted authors, "TABLE 5 provides AUC values for each diagnosis for both formats and observers; requested data in 2×2 format, e.g. for melanoma 'certain' against final diagnosis and for melanoma 'certain or fairly certain' against final diagnosis?" |
| Senel 2013 | Exclude on 2×2 data; no accuracy data available. For teledermatology it looks like they only gave the % of correct diagnoses (Table 6) per dermatologist but give no breakdown by disease positive/negative so could not work out 2×2 data. |
| Shin 2014 | Exclude on target condition |
| Tait 1999 | Exclude on study population; not all suspected of skin cancer 'people with visible skin lesion or lesions' (wide range in diagnoses recorded) |
| Tan 2010a | Exclude on reference standard; no reference standard described Exclude on 2×2 data; not test accuracy; reported agreement between 5 dermatologists' telediagnosis, not against histology or FTF diagnosis |
| Tan 2010b | Exclude on 2×2 data; study appeared to meet eligibility criteria; however, although sensitivity and specificity values were provided in Table 4 per dermatologist, it was not possible to work back to the underlying 2×2 (final diagnoses by histopath not given and FTF diagnoses for the same 491 lesions differ in Table 1 according to dermatologist A and dermatologist B). |
| Tandjung 2015 | Exclude on target condition; 'malignant' included: AK, BD, dysplastic nevus, lentigo maligna, SCC, BCC, MM and KA Exclude on index test; GPs sent images for teledermatology opinion; then free to send for biopsy or not; results shown were only for those that were biopsied, according to teledermatology advice. |
| Taylor 2001 | Exclude on study population; not all suspected of skin cancer; wide variety of conditions included Exclude on 2×2 data; reported % agreement only |
| Tucker 2005 | Exclude on target condition; no breakdown of either final diagnoses or tele‐dx Exclude on 2×2 data; reports agreement only Exclude on study population; not all suspected of skin cancer |
| van der Heijden 2013 | Exclude on 2×2 data; only reported Kappa values for histology vs FTF and histology vs tele‐dx (for each of 4 teledermatologists) but no underlying data given |
| Vano‐Galvan 2011 | Exclude on study population: not specific to skin cancer, population included infectious disease and inflammatory disease Exclude on 2×2 data; only gives % agreement between tele‐dx and FTF (gold standard) |
| Warshaw 2009a | Exclude on 2×2 data; only reports accuracy Exclude duplicate or related publication; author contacted in regard to 2010 paper Exclude but contact author. Study presented diagnostic accuracy of teledermatology and clinic diagnosis in comparison to histopathology; in order to include in our review, data would need to be presented as a 2×2 contingency table, either per type of malignancy e.g. tele‐dx classification of melanoma vs not melanoma against histological diagnosis of melanoma/not melanoma, or with malignant diagnoses grouped together, i.e. tele‐dx of malignancy vs not malignant against same histological breakdown. Requested these data for the clinic diagnosis and for each method of telediagnosis for this study and for Warshaw 2009b. **Author provided some data for detection of melanoma only and for use of macro images only for 2010 paper (pigmented and non‐pigmented lesion combined) |
| Warshaw 2009b | Exclude on 2×2 data; only reports accuracy Exclude duplicate or related publication Exclude but contact author. See Warshaw 2009a |
| Warshaw 2010a | Exclude on 2×2 data; not test accuracy; interobserver agreement for subsample of Warshaw 2009a/Warshaw 2010b trial |
| Warshaw 2015 | Exclude on 2×2 data; only gives agreement between teledermatology diagnosis and FTF diagnosis |
| Watson 2010 | Exclude on target condition |
| Weingast 2013 | Exclude on study population |
| Weinstock 2002 | Exclude not a primary study |
| Weinstock 2009 | Exclude not a primary study |
| Whited 1999 | Exclude on 2×2 data; only gave % agreement between tele‐dx and FTF and % correct diagnoses of tele‐dx vs histo |
| Whited 2002 | Exclude not a primary study |
| Whited 2003 | Exclude not a primary study |
| Whited 2004 | Exclude not a primary study |
| Whited 2006 | Exclude not a primary study; review article |
| Whited 2010 | Exclude not a primary study |
| Whited 2016 | Exclude not a primary study |
| Williams 2001 | Exclude not a primary study |
| Williams 2007 | Exclude not a primary study |
| Wootton 2000 | Exclude on study population Exclude on target condition |
| Zelickson 1997 | Exclude on study population |
AK: actinic keratosis; AUC: area under curve; BCC: basal cell carcinoma; BD: Bowen’s disease; CM: cutaneous melanoma; D+: disease positive; Derm: dermoscopy; DTA: Diagnostic Test Accuracy; Dx: diagnosis; FP: false positives; FTF: face‐to‐face; FU: follow‐up; GP: general practitioner; histo: histology; KA: keratoacanthoma; MM: malignant melanoma; MiS: melanoma in situ (or lentigo maligna); NMSC: non‐melanoma skin cancer; ROC: receiver operating characteristic; SCC: squamous cell carcinoma; SK: seborrhoeic keratosis; tele‐Dx: teledermatology diagnosis; TP: true positive; TN: true negative; VI: visual inspection.