Contemporary Diagnostic Imaging Modalities for the Staging and Surveillance of Melanoma Patients: a Meta-analysis

Abstract

Background

Meta-analyses were performed to examine the utility of ultrasonography, computed tomography (CT), positron emission tomography (PET), and a combination of both (PET-CT) for the staging and surveillance of melanoma patients.

Method

Patient-level data from 74 studies containing 10 528 patients (between January 1, 1990, and June, 30, 2009) were used to derive characteristics of the diagnostic tests used. Meta-analyses were conducted by use of Bayesian bivariate binomial models to estimate sensitivity and specificity. Diagnostic odds ratios [ie, true-positive results/false-negative results)/(false-positive results/true-negative results)] and their 95% credible intervals (CrIs) and positive predictive values were used as indicators of test performance.

Results

Among the four imaging methods examined for the staging of regional lymph nodes, ultrasonography had the highest sensitivity (60%, 95% CrI = 33% to 83%), specificity (97%, 95% CrI = 88% to 99%), and diagnostic odds ratio (42, 95% CrI = 8.08 to 249.8). For staging of distant metastases, PET-CT had the highest sensitivity (80%, 95% CrI = 53% to 93%), specificity (87%, 95% CrI = 54% to 97%), and diagnostic odds ratio (25, 95% CrI = 3.58 to 198.7). Similar trends were observed for melanoma surveillance of lymph node involvement, with ultrasonography having the highest sensitivity (96%, 95% CrI = 85% to 99%), specificity (99%, 95% CrI = 95% to 100%), and diagnostic odds ratio (1675, 95% CrI = 226.6 to 15,920). For distant metastases, PET-CT had the highest sensitivity (86%, 95% CrI = 76% to 93%), specificity (91%, 95% CrI = 79% to 97%), and diagnostic odds ratio (67, 95% CrI = 20.42 to 229.7). Positive predictive values were likewise highest for ultrasonography in lymph node staging and for PET-CT in detecting distant metastases.

Conclusion

Among the compared modalities, ultrasonography was superior for detecting lymph node metastases, and PET-CT was superior for the detection of distant metastases in both the staging and surveillance of melanoma patients.

CONTEXT AND CAVEATS

Prior knowledge

Melanoma may recur in up to 50% of melanoma survivors, especially during the first years after diagnosis. Positron emission tomography (PET) and a combination of PET and computed tomography (CT) (PET-CT) have gained acceptance as the imaging modalities to identify recurrence in survivors and to stage lymph nodes and metastatic melanoma but evidence-based data on their risks and benefits are scarce.

Study design

Meta-analysis of patient-level data from published studies was used to derive characteristics (sensitivity, specificity, diagnostic odds ratio, and positive predictive value) of the four diagnostic imaging modalities.

Contribution

Among the four imaging methods examined, for regional lymph node staging, ultrasonography had the best performance. For staging of distant metastases, PET-CT had the best performance. Similar patterns were observed for surveillance of melanoma survivors for lymph node involvement and for distant metastases.

Implications

The superior modality for lymph node staging and detecting lymph node involvement was ultrasonography. The superior modality for staging and detecting distant metastases was PET-CT.

Limitations

Diagnostic criteria and the quality of the imaging equipment for each modality varied during the period studied. Most studies included in this meta-analysis had a retrospective design.

From the Editors

Staging guidelines for distant metastatic melanoma published by the National Comprehensive Cancer Network indicate that melanoma patients with regional lymph node involvement (ie, American Joint Committee on Cancer stage III) (1) should undergo diagnostic imaging at the time of diagnosis (2), although the detection rate is low particularly in subsets of asymptomatic patients with microscopically detected disease in the lymph nodes (stage IIIA) (3). The guidelines further state that those with less advanced disease (ie, stage IA or II) with clinical indications (ie, suspected or palpable lymph nodes) should be considered for imaging. Although sentinel lymph node biopsy is the acknowledged gold standard for pathological staging of clinically lymph node–negative patients (4), in some clinical settings, ultrasound has also been used for preoperative lymph node assessment and postoperative surveillance (5,6). Positron emission tomography (PET) and a combination of PET and computed tomography (CT) (PET-CT) have rapidly gained acceptance as the imaging modalities of choice for identifying metastatic melanoma but have often been applied without regard to tumor-specific risk strata or known benefits (7–10). Given the limited nature of health-care resources, it is critical to examine these and other new technologies as they emerge.

In 2006, the number of melanoma survivors in the United States was estimated to be more than four million (11), largely as a result of the successful treatment of most patients with newly diagnosed early-stage melanoma (12). In up to 50% of these patients, however, the tumor may recur (13–15), with the risk of first recurrence being greatest in the initial years after diagnosis (16–18). It has been estimated that 20% of all first recurrences occur locally, 50% occur in the regional lymph nodes, and 30% arise at distant sites (19–22). Although surgical resection continues to be the standard treatment for local and regional recurrences, reports of surgical resection or metastasectomy for distant recurrence in select patients have also been associated with improved survival (23–27). These optimistic reports of survival after salvage surgical resection of melanoma recurrences offer a rationale for defining optimal follow-up strategies. Despite the benefit of early detection of locoregional (19,28,29) or distant (23–27) recurrences in these patients, there are no evidence-based guidelines for their surveillance, and clinical practice patterns vary widely.

Currently, the most commonly used imaging modalities for melanoma patients include ultrasonography, CT, PET, and PET-CT. The proposed advantage of PET-CT is that differences in metabolism and function can be detected that complement anatomical imaging techniques (30). Several studies (7–10,31) have reported characteristics of the individual diagnostic imaging tests for the evaluation of melanoma recurrences. However, the utility of each modality as applied to various clinical scenarios has not been examined, and the modalities have not been directly compared. The objective of this meta-analysis was to analyze the contemporary literature related to diagnostic imaging in melanoma patients and to compare the test characteristics of various imaging modalities, such as ultrasonography, CT, PET, and PET-CT, for the staging and surveillance of patients with melanoma.

Patients, Studies, and Methods

Studies and Patients Included in the Meta-Analysis

A comprehensive literature search of MEDLINE (from January 1, 1990, through June 30, 2009), EMBASE (from January 1, 2001, through June 30, 2009), Cancerlit (from January 1, 1990, through October 31, 2002), and the Controlled Trials Register from the Cochrane Library (from January 1, 1990, through June 30, 2009) was performed with the following keywords: “melanoma”; “lymph node metastasis”; “ultrasound”; “computed tomography”; “positron-emission tomography”; and “positron emission tomography with computerized tomography.” Articles identified from the search were reviewed in detail and included in the analysis if they met following criteria: 1) included more than 10 patients with melanoma and 2) included comparisons of single or multiple imaging modalities (ie, ultrasonography, CT, PET, and/or PET-CT) to a gold standard. For primary staging of regional lymph nodes, sentinel lymph node biopsy with pathological confirmation is the gold standard for clinically lymph node–negative patients (2,5,32). For surveillance studies, a minimum of 6 months of follow-up was required for clinical confirmation. No language restrictions were applied, and additional references in identified articles were also reviewed for inclusion.

The literature search yielded 1096 unique citations. In total, 1020 (93%) citations were excluded. The two most common reasons for exclusion were inadequate reporting of patient-level data that were required to calculate test characteristics and/or lack of a reported gold standard. Two reports that met inclusion criteria were excluded because of overlapping study populations (33,34). Thus, 74 studies containing 10 528 patients were included in this meta-analysis.

Patient-level data were extracted and used to construct two-by-two tables. Each melanoma patient who was included in this study had undergone ultrasonography, CT, PET, or PET/CT. Their test results had been classified as true positive, true negative, false negative, or false positive by the histological analysis of lymph node specimens or distant metastasis specimens or by the outcome after long-term follow-up (ie, >6 months) as the gold standards. Diagnostic test characteristics were analyzed according to standard definitions for each individual study: sensitivity [TP/(TP + FN)], specificity [TN/(FP + TN)], false-negative rate [FN/(FN + TP), or 1 − sensitivity], and the positive predictive value [TP/(TP + FP)] (where TP is the number of patients with a true-positive result, TN is the number of patients with a true-negative result, FP is the number of patients with a false-positive result, and FN is the number of patients with a false-negative result). Accuracy was calculated as [(TP + TN)/number of patients in the study. Several studies reported true-positive, false-positive, false-negative, and true-negative results at the patient level, whereas others reported these values at the lesion or assessment level. In these instances, independence between lesions on the same image and different assessments for the same patient were assumed.

After data abstraction, two raters (Y. Xing and J. N. Cormier) independently assessed the quality of the included studies by use of the Quality Assessment of Diagnostic Accuracy Studies (QUADAS) scale (35). Discrepancies were resolved by consensus or third-party review (by R. L. Askew). This scale contains 14 items that examine potential sources of bias in diagnostic studies, with one point assigned for each criterion satisfied. The questions related to the representativeness of the sample, selection criteria, and the appropriateness of the reference standard test. Higher scores reflect higher quality, and no articles were excluded because of the assigned quality score.

Statistical Analysis

Bayesian Bivariate Binomial Model

To provide overall summary estimates for sensitivity and specificity for each imaging modality, Bayesian bivariate binomial models were applied that were similar to those proposed by Chu and Cole (36). This model assumed a binomial distribution for the number of patients with true-positive and true-negative results and allowed the inclusion of covariates and random effects. The inherent association between sensitivity and specificity was modeled in the bivariate normal distribution by assuming random effects. The full model can be expressed as:

where i represent the individual diagnostic studies n_l,i = TP_i + FN_i (ie, the number of subjects with the disease), n_0,i = TN_i + FP_i (ie, the number of subjects without the disease), X_i and Z_i were vectors of covariates related to specificity and sensitivity, respectively, α and β were the corresponding regression coefficients, and μ and ν are the variables representing random effects, N is normal distribution, σ² is the between-study variance, and ρ is correlation coefficient.

To fully specify the model, the following vague previous probability distribution were assigned to the parameters:

Separate analyses were conducted for detecting lymph node and distant metastases, but ultrasound imaging was not included in the model for detecting distant metastasis. Both models included the specific surveillance tests used to study melanoma patients (eg, ultrasonography, CT, PET, and PET/CT) as covariates along with other clinically important covariates to account for between-study heterogeneity, including study design (eg, prospective vs retrospective), reason for diagnostic imaging (eg, primary staging only, restaging, or both), and the level of tumor assessment (eg, patient or lesion). Sensitivity, specificity, and diagnostic odds ratios were calculated as conditional probabilities that were identified as specific values for the covariates in the model. The diagnostic odds ratio, defined as [(TP/FN)/(FP/TN)], was used as an indicator of test performance because it combines information from all four statistical cells. Its value ranges from zero to infinity, with a higher value indicating better discriminatory power. A value of 1.0 is expected for tests with no difference detected between disease and nondisease groups (37). The 95% credible intervals (CrIs), or Bayesian confidence intervals, which are equivalent to the frequentist’s confidence interval, were calculated for sensitivity, specificity, and diagnostic odds ratios. Positive predictive values were also calculated for each of the diagnostic modalities by use of the following formula: (sensitivity × prevalence)/{(sensitivity × prevalence) + [(1 − specificity) × (1 − prevalence)]}, with 5-year recurrence serving as prevalence estimates (38). The three prevalence risk categories for lymph node and distant metastasis were defined as follows for the calculation of the positive predictive value: low = 5%, intermediate = 15%; and high = 30%.

Model Implementation

Bayesian bivariate binomial models were constructed with Markov chain Monte Carlo methods by use of WinBUGS, version 1.4.2 (39). Each covariate was centered about its mean to ensure approximate previous independence between the regression coefficients and good convergence of the three Markov chains (39). The first 10 000 draws were discarded, and only the second 10 000 draws were used to obtain posterior estimates that were based on three separate chains with overdispersed starting values. The Brooks, Gelman, and Rubin convergence statistics (40) were used to assess model convergence, and only properly converged models were further considered. The results were based on 30 000 draws, and R-hat (40) for all parameters was equal to 1.0 in two models, indicating model convergence.

Funnel plots were created to assess publication bias by examining the relationship between the effect measure (log diagnostic odds ratio) and its standard error. Standard error serves as a good proxy for sample size because of their inversely proportional relationship; a small value reflects high precision for an effect size estimate. Conversely, smaller studies are more likely to exhibit a larger spread around the summary estimate of effect size (41). Egger tests were used to assess asymmetry of the funnel plot and to quantitatively assess bias (42). For this analysis, P values were two-sided and statistical significance was defined as a P value of less than or equal to .05. These analyses were performed with Stata, version 10.0 (Stata Corporation, College Station, TX).

Results

Quality assessment scores for the diagnostic studies were calculated as the number of individual criteria satisfied from the 14-point QUADAS scale (35). Figure 1 shows the distribution of the quality rankings for the 74 studies included in this analysis, with an overall mean score of 5.8 (standard deviation = 2.5). Approximately 90% of the studies had a total quality score of less than 9.0, and the majority of articles satisfied items pertaining to details of the reference standard and index test. The most commonly unmet quality criteria related to insufficient detail when reporting patient withdrawals, intermediate results, and the selection and training of raters.

Figure 1.

Histogram of the Distribution of Quality Assessment of Diagnostic Accuracy Studies (QUADAS) scores for diagnostic studies (n = 74).

Characteristics of the 74 studies as stratified by diagnostic imaging modality for the detection of lymph node or distant metastases are shown in Tables 1 and 2. Twenty-one studies met the inclusion criteria for ultrasonography (28,43–62), 13 for CT (63–75), 45 for PET (51,66–109), and 13 for PET-CT (68,69,71,72,74,75, 94,110–115). The median patient age for these studies was 55 years (range = 14–93 years); the percentage of men was 55 (range = 37%–79%), and the mean number of participants per study was 140 (range = 10–2008 participants per study). Patients were enrolled exclusively for the purposes of primary staging in 30 studies (48,51,52,54–56,58–62,70,83,85–87,91,93,98–106,108,109,114) or surveillance in 34 studies (28,44–47,49,53,57,63–65,67–69,71,72,73,75–77,80–82,87–89,92,94–96,110,111,113,115).

Table 1.

Published reports on diagnostic test characteristics of ultrasonography (US) and computed-tomography (CT) for staging and surveillance testing in patients with melanoma^*

Imaging modality, first author, and year (reference)	Study design	No. of patients	Sensitivity, % (95% CI)	Specificity, % (95% CI)	Accuracy, % (95% CI)	Prevalence, %
US of regional lymph nodes
Prayer, 1990 (43)	Retrospective	217	100 (86 to 100)	97 (93 to 99)	97 (94 to 99)	13
Tregnaghi, 1997 (44)	Retrospective	87	87 (61 to 97)	95 (86 to 98)	93 (86 to 97)	NA
Binder, 1997 (45)	Retrospective	264	90 (78 to 96)	99 (98 to 99)	99 (98 to 99)	19
Uren, 1999 (46)†	Prospective	52	94 (NA)	87 (NA)	89 (NA)	NA
Voit, 2001 (47)	Prospective	829	99 (97 to 100)	98 (98 to 99)	98 (98 to 99)	NA
Rossi, 2000 (48)	Retrospective	69	33 (15 to 59)	100 (92 to 100)	87 (77 to 93)	11
Garbe, 2003 (28)	Prospective	2008	86 (79 to 90)	99 (98 to 99)	98 (98 to 98)	6
Brountzos, 2003 (49)	Prospective	148	98 (87 to 100)	98 (93 to 100)	98 (94 to 100)	30
Schmid-Wendtner, 2003 (50)	Prospective	1395	92 (86 to 96)	98 (98 to 99)	98 (97 to 98)	8
Hafner, 2004 (51)	Prospective	100	8 (1 to 25)	88 (78 to 94)	67 (57 to 75)	26
Hocevar, 2004 (52)	Retrospective	57	71 (45 to 88)	84 (70 to 92)	81 (68 to 89)	25
Machet, 2005 (53)	Retrospective	361	93 (80 to 98)	98 (95 to 99)	97 (95 to 99)	12
Starritt, 2005 (54)	Prospective	304	23 (11 to 40)	100 (98 to 100)	92 (88 to 95)	11
Testori, 2005 (55)	Prospective	88	94 (71 to 100)	90 (82 to 95)	91 (83 to 95)	NA
Voit, 2006 (56)	Prospective	127	79 (63 to 90)	72 (62 to 81)	74 (66 to 81)	28
Dalle, 2006 (57)	Prospective	67	98 (90 to 100)	96 (86 to 100)	97 (92 to 99)	48
Sibon, 2007 (58)	Prospective	131	21 (10 to 37)	90 (82 to 94)	26 (64 to 79)	27
Schmid-Wendtner, 2004 (59)‡	Prospective	22	80 (48 to 95)	79 (52 to 93)	26 (59 to 91)	NA
Rossi, 2003 (60)§	Prospective	125	33 (15 to 59)	100 (92 to 100)	87 (77 to 93)	22
Van Rijk, 2006 (61)§	Prospective	107	5 (1 to 19)	99 (91 to 100)	66 (57 to 75)	35
Schafer-Hesterberg, 2007 (62)§	Prospective	400	65 (54 to 74)	99 (97 to 100)	92 (89 to 94)	20
Voit, 2006 (56)§	Prospective	127	59 (42 to 74)	100 (94 to 100)	88 (80 to 93)	28
CT
All lesions
Buzaid, 1993 (63)	Retrospective	151	100 (29 to 100)	82 (75 to 87)	82 (75 to 87)	1
Buzaid, 1995 (64)	Retrospective	89	100 (55 to 100)	76 (66 to 84)	77 (68 to 85)	7
Kuvshinoff, 1997 (65)†	Retrospective	136	57 (NA)	NA	NA	NA
Holder, 1998 (66)	Prospective	76	55 (41 to 69)	84 (71 to 92)	70 (59 to 78)	NA
Swetter, 2002 (67)	Retrospective	104	58 (49 to 66)	70 (52 to 83)	60 (53 to 67)	39
Finkelstein, 2004 (68)	Prospective	18	75 (62 to 85)	87 (73 to 94)	81 (72 to 88)	NA
Romer, 2006 (69)	Retrospective	34	88 (79 to 93)	95 (93 to 96)	94 (92 to 95)	NA
Brady, 2006 (70)	Prospective	103	48 (34 to 62)	95 (85 to 99)	75 (66 to 82)	43
Pfannenberg, 2007 (71)	Prospective	64	77 (72 to 82)	70 (61 to 77)	75 (71 to 79)	61
Iagaru, 2007 (72)	Retrospective	106	69 (55 to 79)	94 (83 to 98)	80 (71 to 87)	53
Lymph node metastases
Fuster, 2004(73)║	Retrospective	115	56 (39 to 71)	81 (69 to 89)	72 (62 to 80)	NA
Reinhardt, 2006 (74)║	Retrospective	250	85 (75 to 91)	87 (81 to 91)	86 (82 to 90)	31
Veit-Haibach 2009 (75)║	Prospective	74	23 (8 to 51)	100 (90 to 100)	82 (70 to 90)	23
Distant metastases
Fuster, 2004 (73)║	Retrospective	115	74 (61 to 83)	83 (79 to 87)	82 (78 to 85)	NA
Reinhardt, 2006 (74)║	Retrospective	250	74 (63 to 82)	88 (82 to 92)	83 (78 to 87)	34
Veit-Haibach 2009 (75)║	Prospective	74	25 (9 to 54)	93 (81 to 98)	79 (66 to 87)	21

^*Prevalence is the number of patients with disease divided by the total number of patients in the 2 × 2 table. The number of patients excludes the patients who were not used to calculate sensitivity and specificity. CI = confidence interval; NA = not available.

^†This study was not included in the statistical models because of missing data on true-positive, false-positive, false-negative, and true-negative results.

^‡Color Doppler sonography was used in this study.

^§Ultrasound-guided fine needle aspiration cytology was used in this study.

^║These studies provided detailed results on different levels of lesions.

Table 2.

Published reports on diagnostic test characteristics of positron emission tomography (PET) and PET–computed tomography (CT) for staging and surveillance testing of patients with melanoma^*

Imaging modality, first author, year (reference)	Study design	No. of patients	Sensitivity, % (95% CI)	Specificity, % (95% CI)	Accuracy, % (95% CI)	Prevalence, %
PET
All lesions
Gritters, 1993 (76)†	Prospective	12	100 (76 to 100)	78 (62 to 89)	85 (72 to 92)	NA
Boni, 1995 (77)†	Prospective	15	91 (75 to 98)	67 (30 to 90)	89 (75 to 96)	NA
Blessing, 1995 (78)	Retrospective	20	76 (61 to 87)	93 (81 to 98)	86 (76 to 92)	NA
Steinert, 1995 (79)	Prospective	33	93 (79 to 98)	100 (73 to 100)	94 (84 to 99)	NA
Damian, 1996 (80)‡	Retrospective	100	93 (NA)	NA	NA	NA
Holder, 1998 (66)	Prospective	76	94 (84 to 99)	83 (70 to 91)	89 (81 to 94)	NA
Nguyen, 1999 (81)	Retrospective	45	81 (65 to 90)	80 (54 to 93)	80 (67 to 89)	NA
Jadvar, 2000 (82)	Retrospective	38	81 (56 to 94)	77 (56 to 90)	79 (63 to 89)	42
Eigtved, 2000 (83)	Prospective	38	97 (81 to 100)	56 (27 to 81)	87 (72 to 95)	76
Paquet, 2000 (84)	Prospective	24	73 (43 to 91)	88 (64 to 98)	82 (64 to 92)	NA
Acland, 2000 (85)	Retrospective	54	78 (58 to 91)	87 (73 to 95)	84 (73 to 91)	NA
Tyler, 2000 (86)	Prospective	95	87 (81 to 92)	44 (32 to 55)	74 (68 to 80)	NA
Klein, 2000 (87)†	Prospective	17	86 (46 to 99)	53 (30 to 75)	64 (43 to 80)	41
Stas, 2002 (88)†	Retrospective	84	85 (80 to 89)	90 (86 to 93)	88 (85 to 90)	NA
Swetter, 2002 (67)	Retrospective	104	84 (78 to 88)	97 (91 to 99)	88 (84 to 91)	39
Cobben, 2003 (89)	Prospective	10	88 (69 to 97)	60 (23 to 88)	83 (66 to 93)	20
Gulec, 2003 (90)	Retrospective	46	70 (56 to 82)	57 (25 to 84)	69 (55 to 80)	86
Reinhardt, 2002 (91)	Retrospective	67	92 (73 to 99)	98 (87 to 100)	95 (87 to 99)	36
Finkelstein, 2004 (68)	Prospective	18	79 (65 to 88)	87 (74 to 94)	83 (74 to 89)	NA
Harris, 2005 (92)	Retrospective	92	92 (87 to 95)	88 (64 to 98)	92 (87 to 94)	NA
Brady, 2006 (70)	Prospective	103	68 (53 to 80)	92 (81 to 97)	82 (73 to 88)	0.43
Horn, 2006 (93)	Retrospective	33	83 (42 to 98)	85 (67 to 95)	85 (68 to 94)	0.18
Romer, 2006 (69)	Retrospective	34	85 (76 to 92)	99 (98 to 99)	98 (97 to 99)	NA
Pfannenberg, 2007 (71)	Prospective	64	70 (65 to 75)	84 (76 to 89)	74 (70 to 78)	61
Mottaghy, 2007 (94)‡	Retrospective	92	86 (NA)	94 (NA)	NA	NA
Koskivuo, 2007 (95)	Prospective	30	86 (46 to 99)	96 (77 to 100)	93 (77 to 99)	23
Iagaru, 2007 (72)	Retrospective	106	89 (78 to 95)	82 (69 to 90)	86 (78 to 91)	53
Lymph node metastases
Gritters, 1993 (76)†	Prospective	12	100 (59 to 100)	100 (55 to 100)	100 (73 to 100)	NA
Boni, 1995 (77)†	Prospective	15	88 (50 to 100)	100 (29 to 100)	90 (57 to 100)	NA
Rinne, 1998 (96)	Prospective	100	98 (88 to 100)	100 (85 to 100)	99 (92 to 100)	NA
Macfarlane, 1998 (97)	Prospective	23	85 (56 to 97)	91 (60 to 100)	88 (68 to 96)	NA
Crippa, 2000 (98)	Prospective	38	95 (81 to 99)	84 (61 to 95)	91 (80 to 96)	NA
Klein, 2000 (87)†	Prospective	17	67 (20 to 94)	100 (77 to 100)	95 (73 to 100)	12
Acland, 2001 (99)	Prospective	50	0 (0 to 26)	89 (74 to 96)	64 (50 to 76)	28
Kokoska, 2001 (100)	Prospective	18	40 (12 to 77)	100 (71 to 100)	82 (58 to 94)	29
Stas, 2002 (88)†	Retrospective	84	89 (82 to 94)	90 (79 to 96)	90 (84 to 93)	NA
Belhocine, 2002 (101)	Prospective	21	14 (1 to 54)	93 (66 to 100)	67 (45 to 83)	33
Longo, 2003 (102)	Prospective	25	22 (6 to 56)	100 (69 to 100)	52 (34 to 70)	36
Havenga, 2003 (103)	Prospective	45	15 (3 to 44)	84 (68 to 93)	64 (50 to 77)	29
Fink, 2004 (104)	Prospective	48	13 (0 to 50)	100 (89 to 100)	85 (72 to 93)	17
Hafner, 2004 (51)	Prospective	100	8 (1 to 25)	100 (94 to 100)	76 (67 to 83)	26
Fuster, 2004 (73)†	Retrospective	115	88 (73 to 96)	95 (86 to 99)	93 (86 to 97)	35
Wagner, 2005 (105)	Prospective	144	21 (11 to 36)	97 (93 to 99)	79 (73 to 85)	28
Vereecken, 2005 (106)	Prospective	43	40 (17 to 69)	13 (6 to 27)	18 (10 to 31)	23
Bastiaannet, 2006 (107)	Retrospective	220	68 (55 to 79)	95 (90 to 98)	88 (83 to 92)	26
Reinhardt, 2006 (74)†	Retrospective	250	92 (84 to 97)	98 (94 to 99)	96 (93 to 98)	31
Clark, 2006 (108)	Retrospective	64	10 (2 to 33)	96 (84 to 100)	70 (58 to 80)	30
Maubec, 2007 (109)	Prospective	25	0 (0 to 41)	92 (64 to 100)	60 (39 to 78)	37
Veit-Haibach, 2009 (75)†	Prospective	74	38 (18 to 65)	100 (90 to 100)	86 (74 to 93)	23
Distant metastases
Boni, 1995 (77)†	Prospective	15	92 (74 to 99)	50 (15 to 85)	86 (69 to 95)	NA
Fuster, 2004(73)†	Retrospective	115	72 (59 to 82)	98 (95 to 99)	94 (91 to 96)	NA
Reinhardt, 2006 (74)†	Retrospective	250	89 (81 to 94)	95 (91 to 98)	93 (89 to 96)	34
Veit-Haibach, 2009 (75)†	Prospective	74	33 (14 to 61)	91 (78 to 97)	79 (66 to 87)	22
PET-CT
All lesions
Finkelstein 2004 (68)§	Prospective	18	88 (75 to 94)	91 (79 to 97)	89 (81 to 94)	NA
Romer, 2006 (69)§	Retrospective	34	94 (86 to 98)	100 (99 to 100)	100 (99 to 100)	NA
Akcali, 2007 (110)	Prospective	38	92 (62 to 100)	92 (75 to 99)	92 (78 to 98)	32
Strobel, 2007 (111)	Prospective	47	97 (85 to 100)	100 (62 to 100)	98 (88 to 100)	83
Pfannenberg, 2007 (71)	Prospective	64	91 (87 to 93)	77 (69 to 84)	87 (83 to 90)	61
Mottaghy, 2007 (94)‡	Retrospective	92	91 (NA)	94 (NA)	NA	NA
Iagaru, 2007 (72)§	Retrospective	106	90 (81 to 95)	85 (72 to 92)	88 (81 to 92)	53
Falk, 2007 (112)	Retrospective	60	67 (49 to 80)	84 (70 to 92)	76 (66 to 84)	55
Lymph node metastases
Kell, 2007 (113)	Retrospective	37	22 (6 to 56)	89 (72 to 97)	73 (57 to 85)	24
Yancovitz, 2007 (114)	Retrospective	158	100 (17 to 100)	94 (82 to 98)	94 (83 to 98)	NA
Singh, 2008 (115)	Prospective	52	14 (3 to 41)	95 (82 to 99)	73 (60 to 83)	41
Reinhardt, 2006 (74)†	Retrospective	250	95 (87 to 98)	100 (97 to 100)	98 (96 to 100)	31
Veit-Haibach, 2009 (75)†	Prospective	74	38 (18 to 65)	100 (90 to 100)	86 (74 to 93)	23
Distant metastases
Reinhardt, 2006 (74)†	Retrospective	250	99 (93 to 100)	98 (94 to 99)	98 (95 to 99)	34
Veit-Haibach, 2009 (75)†	Prospective	74	42 (19 to 68)	93 (81 to 98)	82 (70 to 90)	21

^*Prevalence is the number of patients with disease divided by the total number of patients in the 2 × 2 table. CI = confidence interval; NA = not available.

^†These studies provided detailed results on different levels of lesions.

^‡This study was not included in the statistical models because of missing data on true-positive, false-positive, false-negative, and true-negative results.

^§For PET + CT, mean PET and CT were conducted independently but the final test results were determined using both tests.

Results of these 74 studies were also assessed according to the clinical intent of the tests (primary staging or follow-up surveillance) and the anatomical site of evaluation (lymph node or distant metastases) (Table 3 and Figure 2). For the staging of regional lymph nodes, ultrasonography had the highest sensitivity (60%, 95% CrI = 33% to 83%), specificity (97%, 95% CrI = 88% to 99%), and diagnostic odds ratio (42, 95% CrI = 8.08 to 249.8) compared with the respective values for CT (9%, 95% CrI = 1% to 52%; 92%, 95% CrI = 50% to 99%; and 1.13, 95% CrI = 0.04 to 33.25), PET (30%, 95% CrI = 12% to 55%; 96%, 95% CrI = 87% to 99%; and 9.45, 95% CrI = 1.89 to 48.12), and PET-CT (11%, 95% CrI = 1% to 50%; 97%, 95% CrI = 78% to 100%; and 4.39, 95% CrI = 0.21 to 94.2). For staging of distant metastases, PET-CT had the highest sensitivity (80%, 95% CrI = 53% to 93%), specificity (87%, 95% CrI = 54% to 97%), and diagnostic odds ratio (25, 95% CrI = 3.58 to 198.7) compared with the respective values for CT (51%, 95% CrI = 24% to 76%; 69%, 95% CrI = 30% to 92%; and 2.29, 95% CrI = 0 .34 to 14.98) and PET (74%, 95% CrI = 51% to 88%; 75%, 95% CrI = 45% to 91%; and 8.14, 95% CrI = 1.76 to 38.45).

Table 3.

Estimates of sensitivity, specificity, and diagnostic odds ratio for the staging and surveillance of metastatic sites for ultrasonography (US), computed-tomography (CT), positron emission tomography (PET), and PET-CT^*

Clinical scenario and imaging modality	Median sensitivity, % (95% CrI)	Median specificity, % (95% CrI)	Median dOR (95% CrI)
Lymph node
Primary staging
US	60 (33 to 83)	97 (88 to 99)	42.37 (8.08 to 249.80)
CT	9 (1 to 52)	92 (50 to 99)	1.13 (0.04 to 33.25)
PET	30 (12 to 55)	96 (87 to 99)	9.45 (1.89 to 48.12)
PET-CT	11 (1 to 50)	97 (78 to 100)	4.39 (0.21 to 94.20)
Surveillance
US	96 (85 to 99)	99 (95 to 100)	1675.00 (226.6 to 15920.00)
CT	61 (15 to 93)	97 (70 to 100)	46.25 (2.27 to 1354.02)
PET	87 (67 to 96)	98 (93 to 100)	391.05 (68.06 to 2737.00)
PET-CT	65 (20 to 93)	99 (92 to 100)	195.85 (10.77 to 4675.05)
Distant
Primary staging
CT	51 (24 to 76)	69 (30 to 92)	2.29 (0.34 to 14.98)
PET	74 (51 to 88)	75 (45 to 91)	8.14 (1.76 to 38.45)
PET-CT	80 (53 to 93)	87 (54 to 97)	25.23 (3.58 to 198.70)
Surveillance
CT	63 (46 to 77)	78 (58 to 90)	6.05 (2.04 to 17.83)
PET	82 (72 to 88)	83 (70 to 91)	21.54 (9.31 to 50.93)
PET-CT	86 (76 to 93)	91 (79 to 97)	67.00 (20.42 to 229.70)

^*All estimates are at the lesion level. The 95% credible intervals (CrIs), or Bayesian confidence intervals, are equivalent to the frequentist’s confidence interval.

Figure 2.

Forest plots of the test characteristics of four diagnostic modalities. A) Forest plots for sensitivity and specificity for ultrasonography (US), computed tomography (CT), positron emission tomography (PET), and PET-CT for the clinical scenarios indicated. B) Forest plots for diagnostic odds ratio for ultrasonography (US), computed tomography (CT), positron emission tomography (PET), and PET-CT for the clinical scenarios indicated. The squares represent the median and the whiskers represent the 95% credible interval.

Similar trends were observed for melanoma surveillance of lymph node involvement. Ultrasonography had the highest sensitivity (96%, 95% CrI = 85% to 99%), specificity (99%, 95% CrI = 95% to 100%), and diagnostic odds ratio (1675, 95% CrI = 226.6 to 15920) compared with the respective values for CT (61%, 95% CrI = 15% to 93%; 97%, 95% CrI = 70% to 100%; and 46.25, 95% CrI = 2 .27 to 1354), PET (87%, 95% CrI = 67% to 96%; 98%, 95% CrI = 93% to 100%; and 391, 95% CrI = 68 to 2737), and PET-CT (65%, 95% CrI = 20% to 93%; 99%, 95% CrI = 92% to 100%; and 196, 95% CrI = 10.77 to 4675) (Table 3). Likewise, positive predictive values for the surveillance of lymph nodes were consistently higher for ultrasonography among low-risk patients (positive predictive value = 83%, 95% confidence interval [CI] = 36% to 100%), intermediate-risk patients (positive predictive value = 94%, 95% CI = 68% to 100%), and high-risk patients (positive predictive value = 98%, 95% CI = 83% to 100%) (Table 4). For the surveillance of distant metastases, PET-CT had the highest sensitivity (86%, 95% CrI = 76% to 93%), specificity (91%, 95% CrI = 79% to 97%), and diagnostic odds ratio (67, 95% CrI = 20.42 to 229.7) compared with the respective values for CT (63%, 95% CrI = 46% to 77%; 78%, 95% CrI = 58% to 90%; and 6, 95% CrI = 2 to 17.83) and PET alone (82%, 95% CrI = 72% to 88%; 83%, 95% CrI = 70% to 91%; and 22, 95% CrI = 9.31 to 51) (Table 3). Positive predictive values for the surveillance of distant metastasis were consistently higher for PET-CT among patients at low risk (positive predictive value = 33%, 95% CI = 9% to 61%), intermediate risk (positive predictive value = 63%, 95% CI = 38% to 82%), and high risk (positive predictive value = 80%, 95% CI = 64% to 93%) (Table 4). Because of the higher number of false-positive results (ie, the lower specificity), the positive predictive values were lower for distant metastasis than for regional lymph nodes.

Table 4.

Positive predictive value (PPV) estimates for surveillance for ultrasonography (US), computed tomography (CT), positron emission tomography (PET), and PET-CT, stratified by level of risk and metastatic site^*

Site and risk level	5-y recurrence probability, %	PPV, % (95% CI)
		US	CT	PET	PET-CT
Lymph node metastasis
Low risk	5	83 (36 to 100)	52 (12 to 88)	70 (22 to 96)	77 (19 to 99)
Intermediate risk	15	94 (68 to 100)	78 (43 to 95)	88 (60 to 98)	92 (59 to 100)
High risk	30	98 (83 to 100)	90 (68 to 99)	95 (81 to 100)	97 (76 to 100)
Distant metastasis
Low risk	5	—	13 (27 to 32)	20 (6 to 44)	33 (9 to 61)
Intermediate risk	15	—	34 (16 to 52)	46 (27 to 67)	63 (38 to 82)
High risk	30	—	55 (38 to 73)	67 (50 to 82)	80 (64 to 93)

^*The 95% confidence intervals (CIs) were calculated by assuming the total number of patients in the study was 100.

Funnel plots that demonstrate the effects of small study size for each diagnostic imaging modality are presented in Figure 3. Using diagnostic odds ratio as the effect measure, potential publication bias was identified for the studies examining ultrasonography because estimates from 11 of the 16 studies fell outside of the funnel. Results of the Egger test for small study effects were not statistically significant (P = .44), indicating that no trend toward higher levels of test accuracy was observed among studies with smaller sample sizes.

Figure 3.

Funnel plot for diagnostic odds ratio (dOR) with standard error. A) Ultrasonography (US). B) Computed tomography (CT). C) Positron emission tomography (PET). D) PET-CT. Data from each modality are plotted against their standard error (SE). Solid line = summary estimate of the dOR; dashed line = 95% confidence limits around the dOR.

Discussion

The results of this meta-analysis indicate that when selecting among the four diagnostic imaging modalities examined, the anatomical site to be evaluated was more important than the clinical scenario (ie, staging or surveillance). Among the four diagnostic imaging modalities for the assessments of lymph node metastasis, ultrasonography was superior to CT, PET, and PET-CT. PET-CT had the highest positive predictive value for the surveillance of distant metastasis; however, the higher number of false-positive results (ie, lower specificity) from PET-CT lead to the loss of precision. Furthermore, for patients at low risk of metastasis, the positive predictive value of PET-CT (ie, 33%, 95% CI = 9% to 61%) indicated that use of PET-CT is not warranted without additional clinical indications.

Practice guidelines are becoming an increasingly important element in disseminating treatment algorithms to physicians who treat patients in a community setting (116–123), and investigators have suggested that these guidelines can be used as a means of measuring the quality of care delivered (124). However, evidence-based surveillance strategies for survivors of most cancers including melanoma do not exist. A recent report (125) that was based on Surveillance, Epidemiology, and End Results–Medicare data acknowledges geographic and patient variation in the receipt of surveillance after treatment of primary melanoma. With the increasing number of melanoma survivors and rapid advances in health-care technology, the costs of caring for these survivors are rising (126, 127). In 1997, Mooney et al. (128) reported that screening for melanoma recurrence (in this report for asymptomatic pulmonary metastasis) accounted for approximately 80% of program costs, totaling between $27 and $32 million for a 20-year program. As technological advances permit us to more precisely determine metastatic tumor spread, physicians and patients alike are faced with making clinical decisions on the basis of contemporary risk assessment. Nevertheless, controversy continues to surround the optimal imaging modality and interval of patient surveillance.

Sentinel lymph node biopsy is the acknowledged gold standard for the pathological staging of clinically lymph node–negative melanoma (3,5). A recent study by Sanki et al. (5) comparing ultrasonography with sentinel lymph node biopsy found that the sensitivity of targeted high-resolution ultrasound was only 24.3% (95% CI = 19.5% to 28.7%) compared with that of the sentinel lymph node biopsy. The combination of preoperative ultrasound and fine needle biopsy in select high-risk patients can, however, eliminate the need for sentinel lymph node biopsy by preoperatively identifying lymph node metastases, which indicate the need for therapeutic lymph node dissection (52,129,130). The primary utility of ultrasonography for the assessment of metastases in regional lymph nodes is for lymph node surveillance (31,46,131). PET-CT was superior for detection of distant metastases. Given the low positive-predictive value of CT, PET, and PET-CT in the surveillance of patients at low risk of lymph node metastasis, ultrasonography is the only justifiable imaging choice for lymph node surveillance.

The overall point estimates for the diagnostic test characteristics in this study are lower than those reported in two recently published prospective studies (6,132) that evaluated the utility of ultrasonography, CT, and PET in primary staging. Voit et al. (6) reported that ultrasonography combined with fine needle aspiration cytology had a sensitivity of 65% and a specificity of 99% in a cohort of 400 consecutive melanoma patients. Another study (132) reported the sensitivities of PET and CT for 251 patients with clinically palpable (stage III) lymph nodes as 86% and 78%, respectively, with a specificity of 94% for both tests. These discrepancies likely relate to heterogeneity among patient populations in these studies.

The purpose of staging and surveillance is to detect treatable tumors, monitor success of therapy, and provide reassurance and support to patients (133,134). However, these benefits must be balanced with the risks of testing to patients and their associated costs. Costs for CT, PET, and PET-CT can often be more than twice that of ultrasonography, with differences in charges of up to four times more. Although sufficient evidence regarding clinical effectiveness is not yet available to justify the use of new technologies, such as PET-CT, instead of the best existing alternatives, they are already widely used in oncology. Imaging is one of the fastest growing health-care services (135) and is a prime example of technology that must be examined in the context of comparative effectiveness to “improve the quality and affordability of US health care” (136). Quality medical care has been summarized by Earle et al. (137) as the “delivery of optimal health services” (138), with “technical proficiency” (139); “avoiding overuse, underuse, or misuse of technologies”(140); and “incorporating patient centered preferences in shared decision making” (141).

Inappropriate imaging, which adds to health-care costs without improving the quality of care, has been attributed to both physician and patient factors (142). Lack of knowledge (143) and fear of liability for missed diagnoses (144) attributed to physicians have commonly resulted in the inappropriate use of imaging. In addition, patients with a newly diagnosed cancer often expect certain examinations (145), particularly whole-body imaging. A negative imaging result, even when unnecessary, is often reassuring for the patient and physician and is often perceived to come with few if any negative consequences. However, levels of radiation exposure are known to vary widely even with the same imaging modality, potentially leading to health consequences, including increased lifetime risk of cancer (146,147). Furthermore, incidental abnormalities that can be identified on by imaging that do not affect health but require additional evaluation (eg, further imaging or interventional procedures) can result in additional associated costs, complications, and patient anxiety (142).

Compared with previous meta-analyses (7–10,31) that examined test characteristics of diagnostic imaging modalities in patients with melanoma, this analysis has a number of strengths. First, all eligible studies from January 1, 1990 through June 30, 2009 with sufficient data on four widely used contemporary imaging modalities (ie, ultrasonography, CT, PET, and PET-CT) were examined. Patient-level data from these studies were extracted and analyzed according to specific clinical scenarios (eg, initial staging vs surveillance); these data have been reported by few studies, despite the large potential impact of diagnostic imaging on both the quality and cost of medical care (148). Second, Bayesian bivariate binomial models were used for the meta-analysis of diagnostic test characteristics to capture the variability in both sensitivity and specificity simultaneously, as well as their intercorrelation. Such models are applicable to both large and small studies without ad hoc correction (36). Because of the methodological advantages of bivariate models, Harbord et al. (149) have recommended that such models be considered standard methods for meta-analysis of diagnostic accuracy.

This study has several limitations that must also be considered. First, technology has advanced over the last two decades, and the diagnostic criteria for each modality have varied during the period studied. Second, selection bias and work-up bias inherent to each individual study could be considerable in this pooled analysis because most of the studies of patients undergoing the index test were retrospective in design. Third, partial verification bias may exist when only those patients undergoing a reference test are included in a sample, and no data were reported on the remaining patients who only underwent the index test. Another well-described drawback of meta-analyses is publication bias because studies with favorable results have a higher likelihood of being published than those with unfavorable results. The studies examining the diagnostic accuracy of ultrasonography reported widely varying estimates of sensitivity and specificity ranging from 5% to 100% with similar variations observed for PET imaging. There are several potential explanations for such variation including small sample sizes in some studies, differing study designs, varying quality of imaging equipment, and differing imaging criteria for diagnosis. An inherent strength of a meta-analysis in evaluating a large body of literature is that it can overcome limitations of small sample sizes and heterogeneous designs of individual trials by pooling the data and obtaining summary sensitivities and specificities.

With the ever-increasing number of melanoma survivors and limited health-care resources, the need to tailor current consensus-based National Comprehensive Cancer Network guidelines toward an evidence-based cost-effective surveillance program is becoming increasingly critical. Test characteristics and performance are considered the first two levels of the evidence hierarchy for all diagnostic technologies (150). The objective of this analysis was to use contemporary techniques of meta-analysis to summarize the existing evidence for four common diagnostic imaging modalities that are used in the staging and surveillance of regional and distant metastasis for patients with melanoma. Future comparative effectiveness analyses should use decision-analytic modeling to simulate the effectiveness and cost-effectiveness of various surveillance strategies with respect to imaging modality and frequency on stage-specific patient outcomes.

In summary, when diagnostic imaging is indicated for staging or surveillance, we found that ultrasonography was the best diagnostic imaging test to detect lymph node metastases and that PET-CT was more suitable for the detection of distant metastases in patients at intermediate or high risk or when distant metastases are clinically indicated. Results of this meta-analysis should provide information for clinical decisions on the staging and surveillance of patients with melanoma.

Funding

National Cancer Institute, National Institutes of Health (1 R01 CA127328-01 to J.N.C., PI).