Accuracy capabilities comparisons between Karakiewicz, Kattan and Cindolo nomograms in predicting outcomes for renal cancer carcinoma: A systematic review and meta-analysis

Abstract

Introduction:

Several prognostic models have been proposed to predict outcomes of patients affected by renal cell carcinoma. We analyze the discriminative capabilities of Karakiewicz, Kattan and Cindolo nomograms and perform a meta-analysis to yield pooled area under the receiver operator curves (AUCs) for model comparison. The end points of interest were disease-recurrence free survival (DFS) and cancer-specific survival (CSS).

Methods:

An electronic search of the Medline and Embase was undertaken until July 2014. The AUC value, total number of patients, number of disease recurrence, and cancer-related deaths were extracted from the included references. AUCs of the models were converted to odds ratios (ORs). For the meta-analysis, ln(OR) was used for data pooling. For each nomogram, the combined OR was transformed back to a converted AUC (cAUC).

Results:

A total of 16 studies were identified including 26 710 patients. The derived comparison of cAUC values revealed better predictive capability of DFS for the postoperative Karakiewicz nomogram versus Kattan nomogram (p < 0.01), but not versus Cindolo (p = 0.432) and between Cindolo versus Kattan (p = 0.03). The Mantel-Haenszel derived comparison of cAUC values revealed better predictive capability for the preoperative Karakiewicz nomogram versus the Kattan nomogram (p < 0.01) and versus the Cindolo model (p < 0.01), but also between the postoperative Karakiewicz model versus the Kattan model (p < 0.01) and the Cindolo model (p < 0.01). The Kattan model showed better discriminative capability versus the Cindolo model (p < 0.01).

Conclusions:

The predictive abilities of the pre- and postoperative Karakiewicz models are higher than Kattan or Cindolo in predicting DFS and CSS.

Introduction

Over the past years, the management options for patients with renal cell carcinoma (RCC) at all stages have increased.¹ Partial or total nephrectomy is the standard treatment for locally resectable tumours with curative intention.² However, 20% to 40% of surgically treated tumours will develop recurrence during follow-up, which underlines the importance of tailored follow-up regimens and the evaluation of effectiveness of adjuvant therapies.³

In this context, the use of several prognostic factors and models has gained popularity to predict outcomes of patients affected by RCC. In general, all these prognostic tools are more accurate than the standard TNM classification or Fuhrman grade in predicting survival outcomes.⁴ A substantial advantage of prognostic tools is the ability to measure the predictive accuracy, which allows an objective evaluation of the performance itself.⁵ Several predictive models have been proposed; however, some doubts still persist about their discriminative capabilities in predicting oncological outcomes for RCC.

To this regard, preoperative Karakiewicz, postoperative Karakiewicz, Kattan and Cindolo models have been internally and externally validated in different populations.^6–9 Limitations of nomograms include the racial difference among populations, the variability in accuracy, and their characteristics to outperform risk groups.

We review the discriminative capabilities of these four predictive models (preoperative Karakiewicz, postoperative Karakiewicz, Kattan and Cindolo models) and perform a meta-analysis to yield pooled area under the receiver operator curves (AUCs) for model comparison.

Methods

This analysis was conducted according to the Preferred Reporting Items for Systematic Reviews and Meta-analysis guidelines.¹⁰ An electronic search of Medline and Embase was undertaken until July 2014. The search was limited to English articles. The search terms included RCC and related terms, nomogram, integrated staging systems, cancer-specific survival, disease recurrence, predictors, and outcomes. Citation lists of retrieved articles were screened manually to ensure sensitivity of the search strategy. References of the included papers were also manually searched to identify other potential relevant studies. This meta-analysis did not include patient-level studies, but only included studies with statistically combined accuracies reporting the use of nomograms. Studies were reviewed by two independent reviewers (GIR, AD). Differences in opinion were discussed in consultation with the last author (GM).

The end points of interest were DFS and CSS. The AUC value, total number of patients, and the number of cancer-related deaths were extracted from the included references. A meta-analysis of the ROC curves was performed based on methods reported by Walter and colleagues.¹¹ Basically, the AUCs were converted to odds ratios (ORs) using the following equation (equation 1):

$$\mathit{AUC}=\frac{\mathit{OR}}{{\left(\mathit{OR}-1\right)}^2}[(\mathit{OR}-1)-\text{ln}(\mathit{OR})]$$ (1)

The standard error of the AUC and OR was calculated as follows:

$$\mathit{SE}(\mathit{AUC})=\sqrt{\frac{Q1}{n}+\frac{Q2}{m}-{\mathit{AUC}}^2(m+n)/\mathit{mn}}$$ (2)

In this equation, Q1 = AUC/(2-AUC), Q2=2AUC²/(1+AUC), and

$$\mathit{SE}(\mathit{OR})=\frac{\mathit{SE}(\mathit{AUC}){(\mathit{OR}-1)}^3}{(\mathit{OR}+1)\text{ln}(\mathit{OR})-2(\mathit{OR}-1)}$$ (3)

For the meta-analysis, ln(OR) was used for data pooling. SE[ln(OR)] was calculated through a first-order Taylor series conversion, where SE[ln(OR)] = (1/OR) × SE[OR]. Begg’s and Egger’s methods were used to assess publication bias.^12,13 Begg’s test was based on the rank correlation between the observed effect sizes and observed standard errors, while Egger’s regression intercept is similar to Begg’s but used actual values instead of ranks.

Statistical heterogeneity was assessed using the CochranQ and I² statistics. Specifically, statistical heterogeneity was tested using the chi-square test. A value of p < 0.10 was used to indicate heterogeneity. In the case of a lack of heterogeneity, fixed-effects model was used to assess the overall combined OR. For each nomogram, the combined OR was transformed back to a converted AUC (cAUC) using equation 1. All of the tests were two-tailed, and a p < 0.05 was regarded as significant. The analyses were performed using RevMan software v.5.1 (Cochrane Collaboration, Oxford, UK).

Results

After excluding redundant literature, a total of 16 studies were identified, which included 26 710 patients (Table 1, Fig. 1).^{1,4,6,7,9,14–24} In total, the preoperative Karakiewicz nomogram, postoperative Karakiewicz nomogram, Kattan nomogram, and the Cindolo nomogram were validated in 12 065, 12 868, 6036 and 4045 patients, respectively. In all of the included models, we did not observe any publication bias as assessed by the Begg’s and Egger’s methods (Fig. 2). The weighted median follow-up for all patients was 60 months (range: 33.6–82.0). In studies on DFS, the weighted median follow-up was 60 months (range: 37.0–81.0), while the weighted median follow-up for CSS was 55.2 months (range: 33.6–82.0). The pooled DFS for the preoperative Karakiewicz nomogram, postoperative Karakiewicz nomogram, the Kattan nomogram, and the Cindolo nomogram were 84.98%, 88.27%, and 87.07%, respectively.

Table 1.

Characteristics of the included studies

Reference	Data source		Model	No. patients	No. recurrences	Follow-up (median)	Outcomes	AUC
Kattan et al., 20019	Single institution		Kattan	601	66	40	DFS	0.740
Liu et al., 200912	Single institution		Cindolo Kattan Postoperative Karakiewicz	653	156	65	OS– CSS– DFS OS–CSS -DFS OS–CSS–DFS	0.700–0.715–0.752 0.752–0.793–0.841 0.716–0.754– 0.785
Cindolo et al., 20038	Single institution		Cindolo	660	110	42	DFS	N/A
Cindolo et al., 20054	Multi institution		Cindolo Kattan	2404	541	42	OS–CSS–DFS OS–CSS–DFS	0.615–0.648–0.672 0.706–0.771– 0.807
Cindolo et al., 20131	Multi institution		Preoperative Karakiewicz Postoperative Karakiewicz	3230	N/A	49	CSS CSS	0.784 0.842
Karakiewicz et al., 20097	Multi institution		Preoperative Karakiewicz	1972	N/A	42	CSS	0.842
Karakiewicz et al., 200913	Multi institution		Postoperative Karakiewicz	3560	N/A	32	CSS	0.867
Karakiewicz et al., 20076	Multi institution		Postoperative Karakiewicz	2530	N/A	39	CSS	0.865
Kutikov et al., 201014	Multi institution		Preoperative Karakiewicz	3560	N/A	45.6	CSS	0.867
Gontero et al., 201315	Multi institution		Preoperative Karakiewicz	3364	N/A	48	CSS	0.860
Tan et al., 201118	Single institution		Kattan Postoperative Karakiewicz	390	98	65	OS–CSS– DFS OS–CSS–DFS	0.670–0.730–0.730 0.770–0.840–0.810
Hupertan et al., 200616	Single institution		Kattan	565	101	60	CSS–DFS	0.847– 0.607
Utsumi et al., 201119	Multi institution	CUH Centre	Kattan Cindolo	152	36	60	DFS DFS	0.795 0.700
		CCC Centre	Kattan Cindolo	65	6	60	DFS DFS	0.745 0.634
Suzuki et al., 201117	Multi institution		Kattan	211	41	81	DFS	0.735
Klatte et al., 200821	Multi institution		Kattan Postoperative Karakiewicz	995	52	37	CSS–DFS CSS–DFS	0.778–0.755 0.724– 0.704
Brookman-Amissah, 200920	Single institution		Cindolo	771	173	67	DFS	0.690
Zastrow et al., 201322	Single institution		Postoperative Karakiewicz	1480	N/A	82	CSS	0.905

AUC: area under the curve; OS: overall survival; CSS: cancer-specific survival; DFS: disease-free survival; CUH: Chiba University Hospital; CCC: Chiba Cancer Center; N/A: not applicable.

Fig. 1.

Flow diagram of included studies.

Fig. 2.

Analysis of risk of publication bias. Funnel plot of studies included in meta-analysis on disease recurrence free survival (A) and cancer-specific survival (B). The effect of each study is marked by a circle. Uneven distributions of the studies around 95% confidence interval line should suggest the presence of publication bias, which is not the case in this funnel plot. SE: standard error; OR: odds ratio.

The pooled CSS for the preoperative Karakiewicz nomogram, the postoperative Karakiewicz nomogram, the Kattan nomogram, and the Cindolo nomogram were 82.68%, 86.03%, 86.33%, and 84.20%, respectively.

Disease-recurrence survival

The postoperative Karakiewicz model was validated in 3 studies. Non-significant heterogeneity was found in this nomogram (x² = 0.19, I² = 0%, p = 0.91). The weighted median follow-up for all patients was 53.5 months (range: 37.0–65.0). The pooled ORs (95% confidence interval [CI]) and the corresponding cAUC value were 4.32 (1.13–16.47) and 0.728, respectively.

The Kattan model was validated in 8 studies. Non-significant heterogeneity was found in this nomogram (x² = 4.02, I² = 0%, p = 0.86). The weighted median follow-up for all patients was 60 months (range: 33.6–82.0). The pooled ORs (95% CI) and the corresponding cAUC value were 2.97 (1.66–5.34) and 0.675, respectively.

The Cindolo model was in validated in 4 studies. Non-significant heterogeneity was found in this nomogram (x² = 3.53, I²=0%, p = 0.94).The weighted median follow-up for all patients was 60 months (range: 42.0–67.0). The pooled ORs (95% CI) and the corresponding cAUC value were 3.89 (2.06–7.34) and 0.713, respectively. The test of overall effect was statistical significant (Z = 5.92, p < 0.00001) (Fig. 3). The Mantel-Haenszel derived comparison of cAUC values revealed better predictive capability for the postoperative Karakiewicz nomogram versus the Kattan nomogram (p < 0.01), but not versus the Cindolo model (p = 0.432) and between the Cindolo versus Kattan models (p = 0.03) (Table 2).

Fig. 3.

Forest plot for postoperative Karakiewicz, Kattan and Cindolo nomograms in predicting disease recurrence-free survival.

Table 2.

Summary of the pooled ORs and corresponding AUCs of each models for predictive capability of disease recurrence free survival

	Postoperative Karakiewicz	Kattan	Cindolo
No. studies	3	8	4
Heterogeneity test
x2	0.19	4.012	3.53
df	2	8	4
p value	0.91	0.86	0.94
Combined ORs
OR	4.32	2.97	3.89
95% CI	1.13–16.47	1.66–5.34	2.06–7.34
Converted AUC (SE)	0.728 (0.01)	0.675 (0.01)	0.713 (0.01)
Gain in predictive accuracy % (p value)	0.053 (<0.01)a	−0.038 (0.03)c
	0.015 (0.432)b

OR: odds ratio; AUC: area under the curve; df: degree of freedom; CI: confidential interval; SE: standard error.

^aPostoperative Karakiewicz vs. Kattan;

^bPostoperative Karakiewicz vs. Cindolo;

^cKattan vs. Cindolo.

Cancer-specific survival

The preoperative Karakiewicz model was validated in 4 studies. Non-significant heterogeneity was found in this nomogram (x² = 0.40, I² = 0%, p = 0.94). The weighted median follow-up was 48.50 months (range: 48.0–50.4). The pooled ORs (95% CI) and the corresponding cAUC value were 8.47 (range: 2.79–25.70) and 0.81, respectively.

The postoperative Karakiewicz model was validated in 7 studies. Non-significant heterogeneity was found in this nomogram (x² = 0.46, I² = 0%, p = 1.00).The weighted median follow-up was 57.0 months (range: 36.6–82.0). The pooled ORs (95% CI) and the corresponding cAUC value were 8.82 (range: 2.08–37.40) and 0.814, respectively.

The Kattan model was validated in 4 studies. Non-significant heterogeneity was found in this nomogram (x² = 0.02, I² = 0%, p = 1.00).The weighted median follow-up was 62.5 months (range: 37.2–65.0). The pooled ORs (95% CI) and the corresponding cAUC value were 6.52 (range: 1.80–23.57) and 0.780, respectively.

The Cindolo model was in validated in 2 studies. Non-significant heterogeneity was found in this nomogram (x² = 0.30, I² = 0%, p = 0.59). The weighted median follow-up was 62.5 months (range: 60.0–65.0). The pooled ORs (95% CI) and the corresponding cAUC value were 2.61 (1.58–4.30) and 0.655, respectively.

The overall weighted follow-up was 55.2 (range: 33.6–82.0). The test of overall effect was statistical significant (Z = 6.26, p < 0.00001) (Fig. 2). The Mantel-Haenszel derived comparison of cAUC values revealed better predictive capability for the preoperative Karakiewicz nomogram versus the Kattan nomogram (p < 0.01) and versus the Cindolo model (p < 0.01), but also between the postoperative Karakiewicz model versus the Kattan model (p < 0.01) and the Cindolo model (p < 0.01). The Kattan model showed better discriminative capability versus the Cindolo model (p < 0.01). No statistical difference was observed between both Karakiewicz models (p = 0.730) (Table 3).

Table 3.

Summary of the pooled ORs and corresponding AUCs of each models for predictive capability of CSS

	Preoperative Karakiewicz	Postoperative Karakiewicz	Kattan	Cindolo
No. studies	4	7	4	2
Heterogeneity test
x2	0.40	0.40	0.02	0.20
df	3	6	3	1
p value	0.94	1.00	1.00	0.59
Combined Odds Ratio
OR	8.47	8.82	6.52	2.61
95%CI	2.79–25.70	2.08–37.40	1.80–23.57	1.58–4.30
Converted AUC (SE)	0.810 (0.01)	0.814 (0.01)	0.780 (0.01)	0.655 (0.01)
Gain in predictive accuracy % (p value)		0.004 (0.730)b	0.125 (<0.01)f	-
	0.030 (0.020)a	0.034 (<0.01)d
	0.155 (<0.01)c	0.159 (<0.01)e

OR: odds ratio; AUC: area under the curve; CSS: cancer-specific survival; df: degree of freedom; CI: confidential interval; SE: standard error.

^aPreoperative Karakiewicz vs. Kattan;

^bPreoperative Karakiewicz vs. Postoperative Karakiewicz;

^cPreoperative Karakiewicz vs. Cindolo;

^dPostoperative Karakiewicz vs. Kattan;

^ePostoperative Karakiewicz vs. Cindolo;

^fKattan vs. Cindolo.

Discussion

Renal cancer nomograms have been established to counsel patients before treatment. In this context, the Karakiewicz, Kattan and Cindolo models have been widely validated in different populations from different countries.²⁵ However, the best-performing model remains unknown.

Kattan and colleagues from the Memorial Sloan-Kettering Cancer Center developed a nomogram to predict the 5-year progression-free survival of patients undergoing radical nephrectomy for non-metastatic RCC of various histological subtypes. The four factors included in this nomogram were the presence of symptoms, histological subtype, tumour size, and standard TNM stage according to the 1997 version.⁹ When applied to external populations in Europe, the original Kattan nomogram has shown variable prognostic accuracy ranging from 61% to 81%.^4,18–21,23

In 2007, Karakiewicz and colleagues attempted to improve on the accuracy of the aforementioned models by including more variables that have traditionally been shown to predict survival among patients with RCC. The cohort on which the model was developed included over 2500 patients with various stages of RCC treated at 5 different centres. Their final model ultimately incorporated TNM stage, tumour size, histological subtype, age, sex, and symptoms at presentation to predict 1-, 2-, 5- and 10-year cancer-specific mortality. The internally validated accuracy of the nomogram was 86%,⁶ but the externally accuracy reached 90.5%.^1,6,15,24

Karakiewicz and colleagues examined the ability of T and M stages to predict freedom from cancer-specific mortality (CSM) (n = 2474).⁷ In addition to T and M stages, other variables, such as age, gender, tumour size, and symptoms, resulted in an integrated staging system that provided predictions of CSM-free survival at 1, 2, 5, and 10 years after nephrectomy. Discrimination of that model ranged from 84% to 88% within an external validation cohorts.^1,7,16,17

A second preoperative model focusing on RCC recurrence after nephrectomy was developed by Cindolo and colleagues (n = 660).⁸ This staging system relied on symptoms at presentation and on preoperative tumour size. The Cindolo and colleagues nomogram’s discriminatory ability ranged from 67% in European patients to 75% in Chinese patients.^4,8,22

The diffusion of several nomograms to discriminate between similar end points is problematic. It seems obvious that the choice of one or several of these models should be based on their predictive ability and accuracy.²⁵

One should also take into account that not all of these end points can be defined with certainty. For example, the recurrence-free rate could be limited by the heterogeneity of follow-up or the characteristics of the imaging techniques used. Moreover, it seems obvious that predicting mortality improves the gain in accuracy of the model itself. Based on our results, the converted cAUC values of the pooled ORs for predicting CSS were higher than those for predicting DFS. Therefore, common limitations of the models, such as racial difference among population and sample size, should be considered.

For these reasons, we performed a systematic review and meta-analysis to obtain the derived AUC from pooled ORs for each model and to compare models. We transformed the converted AUC values into ORs using methods reported by Walter and colleagues.¹¹

To the best of our knowledge, this is the first meta-analysis investigating the discriminative capabilities of current nomogram for RCC and including 26 710 patients.

Our results confirmed that the preoperative Karakiewicz, postoperative Karakiewicz, and the Kattan models had a combined AUC value more than 0.70 for predicting CSS, while only the postoperative Karakiewicz model to predict DFS suggested stable discriminative capabilities in different populations.

In particular, the postoperative Karakiewicz (p < 0.01) and Cindolo (p = 0.32) models better exhibited cAUC values than the Kattan nomogram for DFS (Table 2). Regarding the discriminative capability for CSS, both Karakiewicz models showed the best predictive ability over the Kattan (all p < 0.01) and the Cindolo (all p < 0.01) models (Table 3).

Based on accuracy and pooled ORs derived from the current meta-analysis, the preoperative and postoperative Karakiewicz models have given the better predictive capability for predicting CSS (both cAUC = 0.81), while the postoperative Karakiewicz (cAUC = 0.728) was better than Cindolo and Kattan for predicting DFS (cAUC = 0.728). On the contrary, the Kattan and Cindolo models showed intermediate predictive capability in predicting CSS and DFS, respectively.

The differences in pooled OR observed between nomograms could be explained by the heterogeneity of variables included in the models itself. In fact, this may be considered a use for these nomograms. We attempted to counteract these limitations by calculating the pooled AUC of all published data.

Our study has its limitations. Firstly, the median follow-up was different among studies. Secondly, we used a new method proposed by Walter and colleagues to convert the reported AUCs to ORs for the meta-analysis. However, the precision of this conversion will be affected by the reported AUC values with varying decimal places (we used three decimal places). Moreover, the conversion formula (equation 1) (from OR to AUC) cannot be inverted analytically (from AUC to OR). Therefore, we obtained the OR through by graphing using Derive v.6 (Texas Instruments, Inc.). Furthermore, the formula is a monotonically increasing function, guaranteeing the feasibility of getting OR through this method. Moreover we did not conduct this meta-analysis at a patient level, but only statistically combined accuracies of studies using previous nomograms. It may be expected that the same patients were included in more models. However, it is impossible to discriminate this at this manuscript level.

Thirdly, although there is a low risk of publication bias, the choice of nomograms was made based on previous publications and available local data. Finally we did not evaluate possible confounding factors that could have influenced that AUC. However, this was out of the scope of the study.

We would also underline that, although these nomograms have been originally created for specific outcomes, they have also been applied for different end points. We included forest plots to evaluate the same outcome and this can be translated in the clinical practice.

Conclusion

The predictive abilities of the pre- and post-operative Karakiewicz models are higher than Kattan or Cindolo in predicting DFS and CSS. The Cindolo and the Kattan nomogram showed relatively intermediate capability for DFS and CSS, respectively, if compared to other models. These results allow us to evaluate the risk of RCC-specific recurrence and mortality before suggesting nephrectomy, partial nephrectomy or adjuvant chemotherapy after surgery.

Fig. 4.

Forest plot for preoperative Karakiewicz, postoperative Karakiewicz, Kattan and Cindolo nomograms in predicting cancer-specific survival.