Utility of a Model for Predicting the Risk of Sentinel Lymph Node Metastasis in Patients With Cutaneous Melanoma

Key Points

Question

What is the clinical benefit of the i31-GEP-SLNB model for sentinel lymph node (SLN) biopsy selection using a 5% risk threshold?

Findings

This decision-analytic study found that, compared with SLN biopsy for all or none, use of i31-GEP-SLNB had greater net benefit for patients with T1b, T2a, and T2b melanoma, but not for those with high-risk T1a melanoma. Compared with SLN biopsy all, use of i31-GEP-SLNB would be equivalent to a strategy that reduced unnecessary biopsies by 23% in patients with T1b and 3% to 4% in those with T2 melanoma without missing any patients with SLN metastasis.

Meaning

The potential for benefit of i31-GEP-SLNB appears greatest among patients with T1b melanoma.

This decision-analytic study examines the association of clinical benefit with use of the i31-GEP-SLNB model for sentinel lymph node biopsy selection using a 5% risk threshold.

Abstract

Importance

A neural network-based model (i31-GEP-SLNB) that uses clinicopathologic factors (thickness, mitoses, ulceration, patient age) plus molecular analysis (31-gene expression profiling) has become commercially available to guide selection for sentinel lymph node (SLN) biopsy in cutaneous melanoma, but its clinical utility is not well characterized.

Objective

To determine if use of the i31-GEP-SLNB model is associated with clinical benefit when used to select patients for SLN biopsy.

Design, Setting, and Participants

This decision-analytic study used data derived from a published external validation study of the i31-GEP-SLNB prediction model. Participants included patients with primary cutaneous melanoma.

Main Outcomes and Measures

The primary outcome was the net benefit associated with using the i31-GEP-SLNB model for SLN biopsy selection compared with other selection strategies (SLN biopsy for all patients and SLN biopsy for no patients) at a 5% risk threshold. Analyses were stratified by American Joint Committee on Cancer (AJCC) T category. The reduction in the number of avoidable SLN biopsies and relative utility were also calculated.

Results

Compared with other SLN biopsy selection strategies, use of the i31-GEP-SLNB model had greater net benefit for patients with T1b (+0.012), T2a (+0.002), and T2b melanoma (+0.002) but not for those with high-risk T1a (−0.003) disease. The improvement in relative utility was +22% in patients with T1b, +1% in T2a, and +2% in T2b melanoma. Compared with SLN biopsy for all patients, use of the model would equate to a 23% decrease in SLN biopsies among patients with T1b disease without an SLN metastasis with no increase in the number of patients with an SLN metastasis left untreated; among patients with T2a and T2b melanoma, the net decrease in avoidable biopsies compared with SLN biopsy for all was 3% and 4%, respectively.

Conclusions and Relevance

The findings of this decision-analytic study suggest that i31-GEP SLNB has significant potential for risk-stratifying patients with T1b melanoma if using a 5% risk threshold; its role among patients with T1a and T2 melanoma or using other risk thresholds requires further study. A prospective validation study confirming the added clinical benefit and cost-effectiveness of i31-GEP-SLNB compared with free clinicopathologic-based prediction models is needed in patients with T1b melanoma.

Introduction

National Comprehensive Care Network guidelines recommend that a sentinel lymph node (SLN) biopsy be considered in patients diagnosed with cutaneous melanoma with a 5% or higher risk of positivity and offered to those with a greater than 10% risk of positivity.¹ To improve selection for this procedure, a neural network-based prediction model using clinicopathologic factors (thickness, mitoses, ulceration, age) plus molecular analysis (31-gene expression profiling) has become commercially available and received Medicare approval (i31-GEP-SLNB).² Although this model provides absolute risk estimates, its clinical utility is not well characterized. Our objective was to identify whether use of the i31-GEP-SLNB model would lead to clinical benefit in selecting patients for SLN biopsy.^3,4,5,6,7

Methods

The study was exempt from institutional review board review because data were publicly available. Raw data and classification measures of the i31-GEP-SLNB model by American Joint Committee on Cancer (AJCC) T category were extracted from Whitman et al² for patients with T1a-HR, T1b, and T2 disease; data were not reported for patients with T3/T4 disease and could not be analyzed. Patients with T1a melanoma and mitotic index of 2 mm² or larger, lymphovascular invasion, absence of tumor infiltrating lymphocytes, age 40 years or younger, microsatellites, regression, or transected base were considered high-risk T1a (T1a-HR). Classification measures were reported by Whitman et al² using a 5% risk threshold (ie, ≥5% risk was a positive test and a <5% risk was a negative test). The net benefit^4,8,9 of the model and competing strategies (SLN biopsy for all patients [100% sensitivity, 0% specificity] and SLN biopsy for no patients [0% sensitivity, 100% specificity]) were computed. Net benefit is a decision analytic measure that puts the benefits of identifying an SLN metastasis and the harms of unnecessary SLN biopsies on the same scale through the use of an exchange rate (net benefit = [true positives ÷ total sample size] – [(false positives ÷ total sample size) × (exchange rate)]). The unit of net benefit is true positives.⁴ A 5% risk threshold implies that the harm of delaying the detection of an SLN metastasis is 19 times greater than an unnecessary SLN biopsy. Thus, the exchange rate is 1 ÷ 19 (exchange rate = [p_t ÷ (1-p_t)]), where p_t is the threshold probability that defines a positive result). The reduction in the number of avoidable SLN biopsies (avoidable biopsies = [true negatives ÷ total sample size] − [(false negatives ÷ total sample size) ÷ (exchange rate)]) and relative utility (relative utility = net benefit ÷ maximum achievable utility) and relative utility (net benefit divided by maximum achievable utility) were also calculated.³

Results

With higher T categories, sensitivity of i31-GEP-SLNB increased and specificity decreased (Table 1). The positive predictive value (PPV) using 5% risk as the cutoff for a positive result ranged from 4% among patients with T1a-HR melanoma to 13% among those with T2b disease. The negative predictive value was greater than 95% for patients with T1a-HR, T1b, T2a, and T2b disease.

Table 1. Classification Measures of the i31-GEP-SLNB Prediction Model.

T Category	SLN cases, No.		SLN positivity rate, %	i31-GEP-SLNB, % (95% CI)
	Positive	Negative		Sensitivity	Specificity	Overall SLN biopsy reduction rate	Predictive value
							Positive	Negative
T1a-HR	7	228a	3	43 (10-82)	69 (62-75)	69 (62-75)	4 (1-11)	98 (94-99)
T1b	18	310b	5	83 (59-96)	42 (37-48)	41 (36-46)	8 (4-12)	98 (94-100)
T2a	48	368c	12	96 (86-99)	14 (11-18)	13 (9-16)	13 (9-17)	96 (87-100)
T2b	15	103d	13	100 (78-100)	5 (2-11)	4 (1-8)	13 (7-21)	100 (48-100)

Abbreviations: SLN, sentinel lymph node; T1a-HR, T1a high-risk patients (mitotic index ≥2 mm²; lymphovascular invasion, absence of tumor infiltrating lymphocytes, age <40 years, microsatellites, regression, or transected base).

^a Negative SLN status determined via SLN biopsy in 86 cases and clinical examination in 142 cases.

^b Negative SLN status determined via SLN biopsy in 261 cases and clinical examination in 49 cases.

^c Negative SLN status determined via SLN biopsy in 330 cases and clinical examination in 38 cases.

^d Negative SLN status determined via SLN biopsy in 91 cases and clinical examination in 12 cases.

Compared with SLN biopsy all, use of i31-GEP-SLNB was associated with greater net benefit for patients with T1b, T2a, and T2b melanoma (Table 2 and the Figure). Among patients with T1a-HR disease, use of i31-GEP-SLNB was associated with lower net benefit than SLN biopsy none (−0.003) but higher net benefit than SLN biopsy all (+0.018).

Table 2. Net Benefit and Relative Utility of the i31-GEP-SLNB Prediction Model Using a 5% Risk Threshold.

T Category	SLN biopsy
	Strategy 1: nonea	Strategy 2: allb	Strategy 3: using i31-GEP SLNB model
Net benefit (95% CI)
T1a-HR	0	−0.021 (−0.044 to 0.001)	−0.003 (−0.018 to 0.119)
T1b	0	0.005 (−0.019 to 0.030)	0.017 (−0.006 to 0.040)
T2a	0	0.069 (0.037 to 0.100)	0.070 (0.039 to 0.101)
T2b	0	0.081 (0.026 to 0.136)	0.083 (0.025 to 0.142)
Relative utility, % (95% CI) c
T1a-HR	0	NA	NA
T1b	0	9 (0-64)	31 (0-69)
T2a	0	60 (46-73)	61 (48-74)
T2b	0	64 (38-89)	66 (46-85)

Abbreviations: SLN, sentinel lymph node; T1a-HR, T1a high-risk patients (mitotic index ≥2 mm²; lymphovascular invasion, absence of tumor infiltrating lymphocytes, age <40 years, microsatellites, regression, or transected base).

^a The SLN biopsy for none is equivalent to a strategy with 0% sensitivity and 100% specificity.

^b The SLN biopsy for all is equivalent to a strategy with 100% sensitivity and 0% specificity.

^c Relative utility is calculated by dividing the net benefit by the maximum achievable utility (prevalence) and ranges from 0% to 100%. In other words, relative utility is the maximum fraction of expected utility achieved by risk prediction compared with perfect prediction. Relative utility allows an assessment of the potential for improved performance with better prediction models.

Figure. Association of Net Benefit Using a 5% Risk Threshold for i31-GEP-SLNB Model, SLN Biopsy for All, and SLN Biopsy for None, Stratified by T Category.

Net benefit is a decision-analytic measure that informs clinical choices involving trade-offs. In classical decision theory, the strategy with the highest expected benefit to patients should generally be chosen, irrespective of size or statistical significance. That said, if 1 strategy required data from an invasive, harmful, or expensive procedure, the added net benefit may not be justifiable. The unit of net benefit is true-positive results. It signifies the strategy that leads to the greatest net true-positive patients treated, after appropriately subtracting the harms of unnecessary treatment (false-positive results) from the benefits of appropriate treatment (true-positive results) using an exchange rate. A risk threshold of 5% for SLN biopsy decisions implies that the harms of not biopsying an SLN metastasis are 19 times greater than the harms of an unnecessary SLN biopsy. The maximum achievable net benefit (perfect prediction) is the disease prevalence (ie, proportions of 0.03 T1a-HR, 0.05 T1b, 0.12 T2a, and 0.13 T2b) in this data set. In clinical terms, use of i31-GEP-SLNB for SLN biopsy decisions vs SLN biopsy for all would lead to 1 more net true-positive result identified for every 83 patients with T1b, 500 with T2a, and 500 with T2b disease, assuming a risk threshold of 5%. SLN Indicates sentinel lymph node; T1a-HR, T1a high-risk patients (mitotic index ≥2 mm²; lymphovascular invasion, absence of tumor infiltrating lymphocytes, age <40 years, microsatellites, regression, or transected base).

The greatest improvement in net benefit (+0.012) and relative utility (+22%) occurred in patients with T1b melanoma. This improvement remained for most conditions in a sensitivity analysis (eTable in the Supplement). In clinical terms, use of i31-GEP-SLNB for SLN biopsy decisions vs biopsy all would lead to 1 more net true positive identified for every 83 patients with T1b disease; this is the equivalent of a 23% decrease in unnecessary SLN biopsies with no increase in the number of patients with an SLN metastasis left untreated. Per 100 patients, this could avoid $81 052 to $350 129 in direct SLN biopsy-related charges^10,11 (potentially higher if adjusted to 2022 dollars) but at a cost of $719 300 for i31-GEP-SLNB testing at the CMS Medicare rate.^11,12 In absolute terms, for this data set, use of the model would result in failing to biopsy 3 of 18 patients with T1b melanoma with an SLN metastasis but would correctly avoid biopsy in 131 of 310 node-negative patients. It should be emphasized that a selection technique can miss SLN metastases but achieve benefit via avoidance of more than 19 negative SLN biopsies for each missed SLN metastasis (at a 5% risk threshold).

Among patients with T2a and T2b melanoma, the model also was associated with improvements in net benefit (+0.002 for both) and relative utility (+1% and +2%, respectively) compared with biopsy all. In clinical terms, use of the model could lead to 1 more net true positive result identified for approximately every 500 patients with each of T2a and T2b disease, respectively. Compared with biopsy all, this is equivalent to a 3% to 4% decrease in unnecessary SLN biopsies with no increase in the number of patients with an SLN metastasis left untreated.

Discussion

These analyses suggest that use of i31-GEP-SLNB to risk stratify patients for SLN biopsy appears most promising for patients diagnosed with T1b melanoma. Prior to adoption, however, additional prospective studies confirming the accuracy of the model are needed. Of note, Whitman et al² determined SLN status via clinical examination alone (ie, no SLNB) for 25% of the validation cohort (approximately 15% T1b cases), potentially affecting the reported classification estimates. Second, few patients with T1b melanoma with SLN metastasis were enrolled (n = 18), resulting in uncertainty in the sensitivity estimate and a lower bound of the net benefit 95% CI that crossed zero. Lastly, it remains untested whether i31-GEP-SLNB has greater utility compared with free clinicopathologic-based prediction models; such a comparison should be conducted on an independent external validation data set.^13,14,15

The i31-GEP-SLNB model was found to be associated with a negative net benefit for patients with T1a-HR melanoma, suggesting potential for harm. These data should be interpreted cautiously because they are affected by the low PPV of the model (ie, among those in which the risk of SLN metastasis was predicted to be ≥5%, the observed risk was 4%). The SLN status was determined via clinical examination alone for 60% of patients with T1a-HR disease, so it is possible that the true PPV is greater than 5% if only 1 or more patients had been incorrectly labeled as false positive results. In addition, use of i31-GEP-SLNB could lead to clinical benefit if a clinician were to perform SLN biopsy on all high-risk patients with T1a melanoma (ie, a 34% net decrease in avoidable biopsies).

Although i31-GEP-SLNB was associated with a net benefit among patients with T2a and T2b melanoma, the absolute magnitude of the difference compared with SLN biopsy for all patients was small. For every 100 patients with T2 disease in which the i31-GEP-SLNB is performed, a net of 3 to 4 patients would be spared an unnecessary SLN biopsy (after adjusting for missed positive SLN biopsies). Further analyses across T categories are needed to determine if i31-GEP-SLNB is cost-effective.

Limitations

A significant limitation was that we could not compute net benefit at other risk thresholds. Although a 5% or greater threshold is commonly used for SLN biopsy decisions, some physicians and patients may choose a different threshold. Future studies should compare net benefit across a range of relevant thresholds (5%-10%), which may show a greater difference in net benefit between i31-GEP-SLNB and SLN biopsy for all. Second, Whitman et al² ascertained SLN status via clinical examination alone for many T1 cases, potentially affecting the accuracy of our estimates. Finally, net benefit analyses assume that a risk threshold has been chosen rationally, and that the expected benefits and harms are the same for all patients, independent of predicted risk.

Conclusions

At a risk threshold of 5%, the clinical benefit associated with i31-GEP-SLNB appeared most promising in risk-stratifying patients with T1b melanoma. Further prospective study of these patients is needed to define if i31-GEP-SLNB adds benefit beyond free clinicopathologic-based prediction models, its cost-effectiveness, and its utility at other risk thresholds.

Supplement.

eTable. One-way deterministic sensitivity analysis of i31-GEP-SLNB net benefit among patients with T1b melanoma using a 5% risk threshold