Assessing the clinical benefit of NMP22 in the surveillance of patients with non–muscle-invasive bladder cancer and negative cytology: a decision-curve analysis

Abstract

Background

Several studies have shown that abnormal levels of nuclear matrix protein 22 (NMP22) are associated with bladder cancer, leading to NMP22 being approved as a urinary biomarker by the FDA. Nonetheless, the clinical significance of NMP22 remains unclear.

Objective

To use decision analysis to determine whether NMP22 improves medical decision-making.

Design, Setting, and Participants

The study included 2,222 patients with a history of non–muscle-invasive bladder cancer and current negative cytology. We developed models to predict cancer recurrence or progression to muscle-invasive disease using NMP22 levels, age, and gender.

Intervention(s)

Voided NMP22 and cystoscopy.

Measurements

Clinical net benefit was calculated by summing the benefits (true positives) and subtracting the harms (false positives) and weighting these by the threshold probability at which a patient or clinician would opt for cytoscopy.

Results and limitations

After cystoscopy, 581 (26%) patients were found to have cancer. NMP22 level was significantly associated with bladder cancer recurrence and progression (p<0.001 for both). Using NMP22 in a model with age and gender was associated with better patient outcomes than performing cystoscopy on everyone for threshold probabilities above 8% for recurrence and above 3% for progression. Only offering cystoscopy to those with a 15% or greater risk would reduce the number of cystoscopies by 229, while missing only 25 cancer recurrences per 1000 men with a negative cytology. The study was limited by its multicenter design.

Conclusions

For clinicians who would perform a cystoscopy at a threshold of 5% for recurrence or 1% for progression, NMP22 will not aid clinical decision-making. For less risk-averse clinicians who would only perform a cystoscopy at a threshold probability >8% for recurrence or >3% for progression, NMP22 can help determine which patients require cystoscopy and which can be spared this procedure.

INTRODUCTION

Bladder cancer represents an important cause of morbidity and mortality. In 2010, it is again the second most common genitourinary cancer in the United States, with 70,530 new cases and 14,680 deaths.¹ Currently it is estimated that there are more than 500,000 US men and women alive who have a history of bladder cancer. At initial diagnosis, about 70% of patients have cancers confined to the epithelium or subepithelial connective tissue. These cancers are usually managed with endoscopic resection and selective use of intravesical therapy. The recurrence rate for these tumors ranges from 50% to 70%, and 10% to 15% progress to muscle invasion over a 5-year period.^{2, 3} Disease recurrence may occur locally or in the upper urinary tract even after several years, necessitating life-long surveillance.

The expense of cystoscopy and inadequate sensitivity of cytology have led to a search for noninvasive urinary markers. An ideal bladder tumor marker would allow accurate monitoring of patients with a history of bladder cancer, early identification of recurrence, and prevention of disease progression. An accurate marker would also have the potential to replace, delay, or complement cystoscopy and/or cytology in the monitoring of patients with bladder cancer. An ideal bladder cancer monitoring test would be noninvasive, objective, easy to perform and interpret, with high sensitivity and specificity, and would provide an immediate or rapid result. While most of the urine-based tests remain investigational, a few have been approved for clinical use based on clinical trials. As of January 2010, six tests including for nuclear matrix protein 22 (NMP22) have been approved by the FDA for the monitoring of patients with non–muscle-invasive bladder cancer. However, there is a significant heterogeneity in performance characteristics of NMP22 that may affect its predictive accuracy when evaluating cancer recurrence and progression in different populations.⁴ For a molecular marker to be clinically useful, it first needs to provide information that is more useful than clinical variables alone. We have shown that urinary levels of NMP22 improve the ability of a nomogram that includes urine cytology, age, and gender to predict urothelial carcinoma of the bladder (UCB) recurrence and progression by a statistically and clinically significant margin.⁵ However, it is unclear whether this increase in accuracy and risk assessment translates into improved patient outcomes. Indeed, an improvement in the predictive accuracy, although necessary, is not sufficient to assess whether using the marker in practice would actually benefit patients. The substantial human and financial resources necessary for randomized trials make such an evaluation impractical for most biomarkers. Decision analytic techniques can often be used instead of trials. The key point of decision analysis is that the consequences of clinical decisions, such as an increased number of unnecessary cystoscopies, are incorporated in analyses. Decision analysis allows one to weight the relative value of the benefits (true positives) to the harms (false positives) and thus incorporate the consequences of a clinical decision.^{6, 7} The aim of the current study was to use decision analysis to determine whether NMP22 improves medical decision-making for patients on surveillance for non–muscle-invasive urothelial carcinoma of the urinary bladder.

PATIENTS AND METHODS

Patients

All studies were performed after approval by a local Human Investigations Committee and in accord with an assurance filed with and approved by the Department of Health and Human Services, where appropriate. Informed consent was obtained from each subject. This study included 2,739 patients who presented with histologically confirmed bladder urothelial carcinoma without evidence of muscle invasion (stages Ta, T1, and/or CIS).⁵ Because a positive urinary cytology result would almost always be followed by cytoscopy, we restricted our cohort to the 2,222 patients with negative cytology results.

All subjects underwent office cystoscopy and urinary cytology evaluation and had voided urine levels of NMP22 measured. Subjects were included in the study at various stages of surveillance. Patients with suspicious cystoscopy findings were evaluated with transurethral biopsies and/or resection of suspicious lesion(s). All histology slides were reviewed by a genitourinary pathologist blinded to patients' clinical data. The 1997 TNM stage classification and the 1998 WHO grade classification were used. Urinary NMP22 levels were quantified according to the manufacturer's recommendations (Matritech Inc, Newton, MA).

Statistical methods

We created two logistic regression models to predict bladder cancer recurrence. The first included only the patient's age and gender; the second also included NMP22 levels. A separate logistic regression model was created to predict progression to invasive disease, defined as stage T1 or higher, using NMP22, age, and gender (age and gender alone were not predictive of disease progression). In all models, NMP22 was entered as a continuous variable with restricted cubic splines at the tertiles to allow for a nonlinear relationship with risk.

We used decision curve analysis to determine whether incorporating NMP22 levels into the statistical model improved patient outcomes. Decision-curve analysis is a method to evaluate the clinical net benefit of prediction models; one sums the benefits (true positives) and subtracts the harms (false positives). The weights assigned to true positives and false positives are derived from the threshold probability of the outcome. In this study, the threshold probability is the minimum probability of bladder cancer recurrence or progression at which a patient (or clinician) would opt for cytoscopy. For example, if a patient would opt for cytoscopy with a 10% risk of bladder cancer relapse, but would forgo the cytoscopy with only a 9% risk, then the threshold probability is 10%. As the threshold probabilities can vary from patient to patient, the net benefit is calculated across a range of probabilities. For the outcome of bladder cancer relapse, we chose 5% to 20% as the range of plausible threshold probabilities; for the outcome of bladder cancer progression, the range of threshold probabilities chosen was lower (1%–10%). All analyses were conducted using Stata 10 (StataCorp, College Station, TX).

Results

Table 1 shows the patient characteristics of the 2,222 patients undergoing cystoscopy following a negative cytology result. Of these patients, 581 (26%) were found to have disease recurrence, and among these, 234 (40%) experienced disease progression (T1 and higher). Patients who experienced disease recurrence tended to be older (median age 70 vs 66 years) and have higher NMP22 levels (median 8.5 vs 3.8 ng/mL) than those who did not experience disease recurrence.

Table 1.

Patient and tumor characteristics of 2,222 patients with previous non–muscle-invasive bladder urothelial carcinoma and negative cytology

	No Recurrence (n=1,641)	Disease Recurrence (n=581)
Age (years; median interquartile range)	66 (55, 73)	70 (60, 76)
NMP22 (ng/mL; median interquartile range)	3.8 (1.7, 7.5)	8.5 (3.3, 29)
Male gender	1229 (75%)	448 (77%)
Stage at cystoscopy
Ta	-	347 (60%)
T1/CIS	-	137 (24%)
T2	-	62 (11%)
T3 and T4	-	35 (6%)
Grade at cystoscopy	-
I	-	216 (37%)
II	-	237 (41%)
III	-	128 (22%)

On multivariable analysis, NMP22 was significantly associated with disease recurrence and progression (p<0.001 for both; Table 2). Fig. 1 shows a risk curve for NMP22 levels, with the levels of NMP22 along the x axis and the corresponding probability of either disease recurrence or progression along the y axis, adjusted for age and gender. The probability of disease recurrence and progression increases with increasing NMP22 levels. NMP22 levels substantially improved the predictive accuracy of a model with only age and gender for both disease recurrence (area under the curve [AUC] increased from 0.596 to 0.720) and progression (AUC increased from 0.570 to 0.815).

Table 2.

Multivariable logistic regression models predicting disease recurrence and progression in 2,222 patients with previous non–muscle-invasive bladder urothelial carcinoma and negative cytology

	Recurrence		Progression
	OR (95% CI)	p value	OR (95% CI)	p value
Age	1.03 (1.02, 1.04)	<0.001	1.02 (1.01, 1.03)	0.002
Male	1.02 (0.80, 1.29)	0.9	1.37 (0.93, 2.01)	0.11
NMP22 level	*	<0.001	*	<0.001

^*Entered with nonlinear terms.

Fig. 1.

Risk curves for NMP22 levels in 2,222 patients with previous non–muscle-invasive bladder urothelial carcinoma and negative cytology. NMP22 levels are along the x axis, and the corresponding predicted probability of disease recurrence (solid line) or progression (dashed line) is along the y axis. Probabilities shown are for a 68-year-old male. The distribution of NMP22 levels is shown by the histogram, with quartiles indicated by the shading.

Fig. 2 shows the decision-curve analysis of several treatment strategies for disease recurrence. The grey line represents the “cystoscopy all patients” strategy, which is essentially the current clinical approach. The horizontal black line indicates the “cystoscopy none” strategy. The thin black line represents a strategy based on the prediction model using age and gender alone, and the dashed line is the prediction model that also includes NMP22 levels. As expected, both prediction models are similar to the “cystoscopy all patients” strategy at low threshold probabilities. The prediction model using age, gender, and NMP22 has a highest net benefit at threshold probabilities above 8%. At a threshold probability of 8%, the net benefit of 0.001 can be interpreted as finding one additional patient with disease recurrence per 1,000 patients without an increase in the false-positive rate (unnecessary cystoscopy), compared to a strategy treating all patients.

Fig. 2.

Decision-curve analysis of bladder cancer recurrence using NMP22, age and gender compared to just age and gender in 2,222 patients with previous non–muscle-invasive bladder urothelial carcinoma and negative cytology. Dashed line: prediction model with NMP22, age and gender. Black line: prediction model with only age and gender. Horizontal line along x axis: assumes all patients will experience disease recurrence (= all patients need to undergo cystoscopy). Solid grey line: assumes no patient will experience disease recurrence (= none of the patients needs to undergo cystoscopy).

Table 3 illustrates the clinical implications of performing a cystoscopy based on the predicted risk of recurrence. Only offering cystoscopy to those with a 15% or greater risk from a model that includes NMP22, age and gender would reduce the number of cystoscopies by 23% (229 per 1000 men with negative cystoscopy), while missing 25 cancer recurrences; 3 of these recurrences would be classified as stage T2 or higher. Using the same risk threshold from a model without NMP22 would only avoid 8% of cystoscopies (75 per 1000 men) and miss 5 recurrences of which 1 would be stage T2 or higher.

Table 3.

Reduction in cystoscopies and number of recurrences that would be detected using various thresholds from a model using only age and gender, or using age, gender and NMP22 level. Numbers are given per 1000 men with negative cytology.

	No. cystoscopies		No. recurrences
	Performed	Avoided (%)	Found	Missed
Cytoscopy all	1000 (100%)	-	261 (100%)	-
Cytoscopy those with > 10% risk of recurrence
Model using age and gender alone	989	11 (1%)	260	1 (0%)
Model using age, gender and NMP22	923	77 (8%)	257	4 (2%)
Cytoscopy those with >15% risk of recurrence
Model using age and gender alone	925	75 (8%)	256	5 (2%)
Model using age, gender and NMP22	771	229 (23%)	236	25 (10%)
Cytoscopy those with >20% risk of recurrence
Model using age and gender alone	811	189 (19%)	229	32 (12%)
Model using age, gender and NMP22	582	418 (42%)	202	59 (23%)

Despite a large improvement in predictive accuracy, the full prediction model only offers a small clinical net benefit over the “cystoscopy all patients” strategy, and only at threshold probabilities greater than 8%. This result is likely due to the high prevalence of bladder cancer recurrence among patients who had a negative cytology (26%).

When the outcome was bladder cancer progression, the range of threshold probabilities was lower, as few patients would forgo a cytoscopy given a moderate risk of disease progression. Fig. 3 shows that for threshold probabilities above 3% there is a clinical net benefit to using the prediction model with NMP22 levels, age, and gender over a “cystoscopy all” strategy. Table 4 shows the clinical implications of basing the decision to perform cystoscopy on various risk thresholds. Only offering cystoscopy to those with a 2% or greater risk from a model that includes NMP22, age and gender would reduce the number of cystoscopies by 12%, but miss 3 cancer progressions. Using a higher threshold of 5% would avoid a greater number of cystoscopies (362 per 1000 men with a negative cystoscopy) while only missing 9 cancer progressions, of which 2 would be stage T2 or higher.

Fig. 3.

Decision-curve analysis of bladder cancer progression using NMP22, age and gender in 2,222 patients with previous non–muscle-invasive bladder urothelial carcinoma and negative cytology. Age and gender alone were not different from performing a cystoscopy on all strategy. Dashed line: prediction model with NMP22, age and gender. Horizontal line along x axis: assumes all patients will experience disease progression (= all patients need to undergo cystoscopy). Solid grey line: assumes no patient will experience disease progression (= none of the patients needs to undergo cystoscopy).

Table 4.

Reduction in cystoscopies and number of progressions that would be detected using various thresholds from a model using only age and gender, or using age, gender and NMP22 level. Numbers are given per 1000 men with negative cytology.

	No. cystoscopies		No. progression
	Performed	Avoided (%)	Found	Missed
Cytoscopy all	1000	-	105	-
Cytoscopy those with >2% risk of progression from full model	880	120 (12%)	102	3 (3%)
Cytoscopy those with >5% risk of progression from full model	638	362 (36%)	96	9 (9%)

Discussion

We and others have shown that NMP22 is elevated in patients with bladder urothelial carcinoma.^{4, 5, 8–14} A biomarker is only of clinical value if its use leads to improved patient outcomes. It is not sufficient to show that it is significantly related to the outcome or statistically significant in a multivariable model including the standard clinical and pathologic factors. Nor is it sufficient to show that the biomarker improves the predictive accuracy of over that of clinical and pathologic factors. In this study we use decision-curve analysis to evaluate the implications of using NMP22 in clinical practice.

In agreement with our previous findings,⁵ we found that addition of NMP22 improves the accuracy of the base model that includes age and gender to predict bladder cancer recurrence and progression by a statistically and clinically significant margin. Addition of NMP22 increased the predictive accuracy of the base model by 20.8% for disease recurrence and 43% for disease progression.

While demonstrating that NMP22 can increase the predictive accuracy of the base model, our previous study did not address whether this increase in accuracy translated into improved medical decision-making or patient outcomes. Indeed, an improvement in the predictive accuracy, although necessary, is not sufficient to assess whether using a biomarker in practice would actually benefit patients. Establishing clinical relevance of a biomarker test for guiding therapy decisions requires demonstrating that it can classify patients into distinct subgroups with different recommended management. Predictive accuracy (ie, AUC) does not account for the prevalence of the disease in the specific population. In addition, sensitivity and specificity are given equal weight, even though missing bladder cancer is more harmful than an unnecessary cystoscopy.

We found that NMP22 helps in making decisions about immediate versus delayed cystoscopy if the threshold for conducting a cystoscopy is at least an 8% risk of disease recurrence or at least a 3% risk of disease progression. For a very risk-averse clinician, who would perform a cystoscopy even if, for instance, a patient had a risk of recurrence of 5% or a risk of progression of 1%, NMP22 would not add any clinical benefit, and the optimal strategy would be to offer cystoscopy to every patient at risk.

Conclusions

NMP22 measured in urine can help predict the result of cystoscopy in patients on surveillance for a history of non–muscle-invasive bladder cancer. A multivariable model that includes NMP22 can determine which patients should be advised to undergo cystoscopy and which might be advised to defer cystoscopy until there was stronger evidence of malignancy. Decisions regarding immediate versus deferred cystoscopy on the basis of NMP22 are likely to lead to better patient outcomes than performing cystoscopy in everyone or using age and gender alone.

Abbreviations and Acronyms

AUC

area under the curve

FDA

United States Food and Drug Administration

NMP22

nuclear matrix protein 22

UCB

urothelial carcinoma of the urinary bladder

WHO

World Health Organization