Interobserver Variability of R.E.N.A.L., Padua, and Centrality Index Nephrometry Score Systems

Abstract

Purpose

To assess interobserver variability of R.E.N.A.L., PADUA, and C-Index systems among observers with varying degrees of clinical experience and each system's subscale correlation with surgical outcome metrics.

Methods

Computed tomography images of 90 patients who underwent open, laparoscopic, or robot-assisted laparoscopic partial nephrectomy were scored by 1 radiology fellow, 2 urology fellows, 1 radiology resident, and 1 secondary school student. Agreement among readers was determined calculating intraclass correlation coefficients. Associations between radiology fellow scores (reference standard as reader with greatest clinical experience), ischemia time, and percent change in postoperative eGFR were evaluated using Spearman correlation.

Results

Agreement using C-Index method (ICC = 0.773) was higher than with PADUA (ICC = 0.677) or R.E.N.A.L (ICC = 0.660). Agreement between reference and secondary school student was lower than with other physicians, although the differences were not statistically significant. The reference's scores were significantly (p <0.05) associated with ischemia time on all three scoring systems and with percent change in eGFR at 6 weeks using C-index (p = 0.016). Tumor size, nearness to sinus, location relative to polar lines (R.E.N.A.L.) and tumor size, renal sinus involvement and collecting system involvement (PADUA) correlated with ischemia time (all p ≤0.001). No R.E.N.A.L. or PADUA subscales significantly correlated with percent change in postoperative eGFR.

Conclusions

Clinical experience reduces interobserver variability of existing nephrometry systems though not significantly and less so when using directly measureable anatomic variables. Consistently, only measures of tumor size and distance to intrarenal structures were useful in predicting clinically relevant outcomes.

Introduction

Comparisons of technical aspects and oncologic outcomes among management options for clinically localized renal masses [1] are often confounded by factors difficult to control for, particularly in characterizing the tumor with biometric data from imaging studies. Such information is crucial for evaluating disease management and disease-related outcomes.

Description of anatomic characteristics of renal masses using one of the recently introduced nephrometry scoring systems [2-4] allows for comparisons between reported experiences and has become almost a standard requirement for clinical studies on renal tumors. Nephrometry has been also increasingly incorporated into clinical practice [5] due to its potential usefulness in understanding the complexity of partial nephrectomy (PN) and the severity and aggressiveness of kidney cancer [6-8]. Though many systems have been devised, there is no current standard and the systems are not interchangeable or translatable creating obvious difficulties with reporting.

Conflicting studies suggest unresolved aspects of renal nephrometry [9-13], including: 1) determining the level of a reader's clinical expertise and training required for adequate score assignment; 2) choosing the most reliable and clinically relevant system among those available; and 3) identifying truly relevant biometric data to be reported. In an attempt to address these problems, we applied the R.E.N.A.L., the Preoperative Aspects and Dimensions Used for an Anatomical Classification System (PADUA), and the centrality index (C-Index) systems to patients undergoing either open, laparoscopic, or robot-assisted laparoscopic PN at Memorial Sloan Kettering Cancer Center (MSKCC); and evaluated the interobserver variability (IOV) among observers with varying degrees of clinical experience; each system's ability to predict surgical complexity and clinical outcomes; and the clinical relevance of nephrometry subscales.

Materials and Methods

Following Institutional Review Board approval, we retrospectively reviewed the records of 90 patients with a renal mass who underwent computed tomography (CT) and PN at our institution. The 30 most recent open, laparoscopic, and robot-assisted laparoscopic PN patients were selected backwards from January 2012. PN patients, whose CT were not performed at MSKCC using our institutional renal mass imaging protocol were excluded from the study.

All CT exams, performed using a 64-detector row CT scanner (General Electric Medical Systems, Pewaukee, WI, USA), consisted of non-contrast enhanced data acquisition from the top of the diaphragm to the lesser femoral trochanter, followed by contrast-enhanced data acquisitions during the nephrographic and excretory phases. Iodinated contrast agent (150 mL) was given at a constant flow rate of 3.5 mL/s via an antecubital vein. Nephrographic and excretory phase images were acquired after a 90-second and 3-minute delay, respectively. Transverse and coronal images were reconstructed at a slice thickness of 2.5 mm and reconstruction increment of 2 mm from the nephrographic and excretory phase data.

CT images of renal masses were independently scored by 1 radiology fellow (RFEL); 3 physicians (PHYS) including 2 urology fellows and 1 radiology resident; and 1 non-medical secondary school student (NMS) using the R.E.N.A.L. [2], PADUA [3], and C-Index systems [4]. Scores from the reader with the greatest clinical experience (RFEL) were used as reference standard.

Readers were blinded to other readers' results, patient demographics, surgical approach, and clinical outcomes. Intraclass correlation coefficient (ICC) was calculated to determine levels of agreement within each system among 1) all readers; 2) RFEL vs. PHYS; and 3) RFEL vs. NMS. Perfect agreement between the readers would result in ICC = 1, whereas ICC between 0 and 1 would show disagreement. Confidence intervals for differences in ICC were estimated using bootstrap re-sampling. We calculated the proportion of instances in which the scores of the PHYS and the scores of the NMS differed from RFEL's scores by less than 1, 2, 3, or 4 points. Since the three nephrometry scoring systems are not on the same scale, the proportion of matching scores was not used to compare the systems but instead to describe the readers' score consistency within each system.

To assess each system's ability to predict surgical complexity and postoperative outcomes, Spearman correlations were used to evaluate the relationship between RFEL scores and ischemia time and percent change in estimated glomerular filtration rate (eGFR) from baseline to postoperative 6 weeks (± 4 weeks) and 6 months (± 1 months). Clinically relevant components of the R.E.N.A.L. and PADUA systems were identified using Spearman correlations to evaluate the ability of RFEL scores on each of the R.E.N.A.L. and PADUA subscales to predict ischemia time and percent change in eGFR at the same intervals.

Correlations between maximum tumor diameter and tumor volume as continuous variables and outcomes of interest were also evaluated.

Results

Baseline characteristics are presented in Table 1. On ICC analysis (Table 2A), the C-Index method always had the highest level of agreement amongst the readers (RFEL and PHYS and NMS), regardless of clinical experience. This level of agreement was significantly higher than with R.E.N.A.L. though not with PADUA systems (difference 0.113, 95% confidence interval [CI] 0.011, 0.171; and difference 0.096, 95% CI -0.003, 0.158, respectively). When looking at RFEL and NMS scores only, the level of agreement was also highest with the C-Index and lower with PADUA and R.E.N.A.L. systems. Similarly, when looking at RFEL and PHYS scores only, the level of agreement continued to be highest with the C-Index, although the level of agreement when using the R.E.N.A.L. system was slightly higher than with the PADUA system. Irrespective of the scoring system used, the agreement amongst the more clinically experienced PHYS and the RFEL was always higher than the agreement between the NMS and the RFEL. However, these increases in agreement were not significant on any of the scoring systems (95% CI for R.E.N.A.L: -0.076, 0.159; for PADUA: -0.116, 0.115; for C-Index: -0.147, 0.133).

Table 1.

Patient characteristics. (Data presented as medians with interquartile ranges in parentheses or number of instances with percentages in parentheses. R.E.N.A.L., PADUA, and C-Index scores as assigned by the radiology fellow).

Characteristic
Age at surgery, yr, median (IQR)	58 (50, 66)
Gender, male, no. (%)	50 (56%)
Body mass index, median (IQR)	30 (26, 39)
Preoperative eGFR (mL/min/1.73 m2), median (IQR)	81 (66, 94)
ASA score, no. (%)
Class 1	3 (3%)
Class 2	34 (38%)
Class 3	51 (57%)
Class 4	2 (2%)
Maximum tumor size, cm, median (IQR)	3 (2, 5)
Tumor volume, mL, median (IQR)	38 (12, 151)
Clinical Stage
T1a	57 (63.5%)
T1b	30 (33.5%)
T2a	3 (3%)
R.E.N.A.L. score, points, median (IQR)	8 (7, 9)
PADUA score, points, median (IQR)	10 (8, 11)
C-Index, median (IQR)	3 (2, 4)

Abbreviations: R.E.N.A.L. = Radius, Exophytic/endophytic, Nearness, Anterior/posterior, Location; PADUA = preoperative aspects and dimensions used for an anatomical classification system; C-Index = Centrality Index; ASA = American Society of Anesthesiologists; eGFR = estimated glomerular filtration rate (CKD-EPI); UTI = urinary tract infection.

Table 2.

(A): Interclass correlation coefficients amongst readers; (B): Proportion of scores within different ranges of the reference standard (radiology fellow); and (C): Proportion of scores with perfect match between all readers and radiology fellow by subscale.

Interclass Correlation Coefficients amongst Readers
A		R.E.N.A.L.		PADUA		C-Index
		ICC	95% CI	ICC	95% CI	ICC	95% CI
	All readers	0.660	0.579, 0.741	0.677	0.598, 0.755	0.773	0.712, 0.834
	RFEL vs. NMS	0.534	0.386, 0.682	0.599	0.467, 0.732	0.696	0.589, 0.803
	RFEL vs. PHYS	0.732	0.659, 0.804	0.715	0.639, 0.791	0.784	0.722, 0.845
Proportion of Scores within Different Ranges of the Reference Standard (RFEL)
B		R.E.N.A.L.		PADUA		C-Index
		NMS	PHYS	NMS	PHYS	NMS	PHYS
	± 2	80%	93%	83%	93%	91%	91%
	± 3	91%	99%	97%	98%	96%	96%
	± 4	99%	100%	98%	100%	98%	98%
Perfect Match between All Readers and RFEL Scores, by Subscale.
C	R.E.N.A.L. Subscale				Percent Match
	(R)adius				94%
	(E)xophitic/Endophytic Properties				62%
	(N)earness to Collecting System or Sinus				66%
	(A)nterior/Posterior				81%
	(L)ocation relative to Polar Lines				47%
	PADUA Subscale				Percent Match
	Tumor Size				93%
	Renal Rim				83%
	Renal Sinus				81%
	Urinary Collecting System				78%
	Longitudinal (Polar) location				74%
	[Anterior/Posterior]				79%
	Exophytic Rate				63%

Abbreviations: R.E.N.A.L. = Radius, Exophytic/endophytic, Nearness, Anterior/posterior, Location; PADUA = preoperative aspects and dimensions used for an anatomical classification system; C-Index = Centrality Index; ICC = intraclass correlation coefficients; CI = confidence intervals; RFEL = Radiology Fellow; NMS = non-medical secondary school student; PHYS = physicians (Urology Fellows and Radiology Resident).

The proportion of scores within different ranges of the RFEL was also analyzed (Table 2B). When using the R.E.N.A.L. and the PADUA systems, the NMS always agreed with the RFEL less often than the PHYS for all point ranges. However, when using the C-Index, the rates of agreement between the NMS and the RFEL were similar to those between the PHYS and the RFEL for all point ranges. The proportions of scores with perfect match between all readers and the RFEL by subscale are also shown in Table 2C.

On Spearman correlation, RFEL scores on all scoring systems were significantly associated with ischemia time (Table 3). Only RFEL scores using the C-Index were associated with percent change in postoperative eGFR at 6 weeks (p = 0.016). None of the RFEL scores on any of the scoring systems were associated with percent change in postoperative eGFR at 6 months (all p > 0.05).

Table 3.

Spearman correlations for radiology fellow's scores on R.E.N.A.L., PADUA, and C-Index systems with ischemia time and percent change in postoperative estimated glomerular filtration rates at 6 week and 6 months with p-values in parentheses.

	Ischemia Time	Postoperative eGFR % Change at 6 Weeks (N = 71)	Postoperative Egfr % Change at 6 Months (N = 75)
R.E.N.A.L.	0.481 (<0.0001)	-0.152 (0.2)	-0.16 (0.2)
Radius	0.398 (0.0001)	-0.154 (0.2)	-0.129 (0.3)
Exophytic/Endophytic	-0.0326 (0.8)	0.123 (0.3)	-0.125 (0.3)
Nearness to Sinus	0.461 (<0.0001)	-0.214 (0.072)	-0.145 (0.2)
Anterior/Posterior	-0.0514 (0.6)	-0.122 (0.3)	0.129 (0.3)
Location Relative to Polar Line	0.351 (0.001)	-0.115 (0.3)	-0.0559 (0.6)
PADUA	0.431 (<0.0001)	-0.168 (0.2)	-0.146 (0.2)
Tumor Size	0.383 (0.0002)	-0.154 (0.2)	-0.132 (0.3)
Renal Rim	0.146 (0.2)	-0.129 (0.3)	0.168 (0.15)
Renal Sinus	0.437 (<0.0001)	-0.148 (0.2)	-0.0955 (0.4)
Collecting System	0.467 (<0.0001)	-0.172 (0.2)	-0.119 (0.3)
Polar Location	0.0834 (0.4)	-0.0875 (0.5)	-0.136 (0.2)
Anterior/Posterior	-0.0638 (0.6)	-0.0772 (0.5)	0.172 (0.14)
Exophytic Rate	-0.101 (0.3)	0.101 (0.4)	-0.0693 (0.6)
C-Index	-0.388 (0.0002)	0.284 (0.016)	0.182 (0.12)

Abbreviations: R.E.N.A.L. = Radius, Exophytic/endophytic, Nearness, Anterior/posterior, Location; PADUA = preoperative aspects and dimensions used for an anatomical classification system; C-Index = Centrality Index, eGFR = estimated glomerular filtration rate.

Spearman correlation between surgical outcomes and R.E.N.AL. and PADUA subscales was investigated to discern the most relevant aspects of these systems (Table 3). In the R.E.N.A.L. scoring system, showed a significant association between the subscales of tumor (R)adius, (N)earness to sinus, and (L)ocation relative to the polar lines and ischemia time. Using the PADUA system, tumor size, renal sinus involvement, and urinary collecting system involvement significantly (all p ≤0.0002) correlated with ischemia time.

On Spearman correlation, both maximum tumor size and tumor volume as continuous variables were significantly associated with ischemia time (Corr. = 0.45, p <0.0001 and Corr. = 0.44, p <0.0001, respectively), although no such association was found with percent change in postoperative eGFR at 6 weeks or 6 months (all p >0.05).

Discussion

Though some commonalities exist among the R.E.N.A.L. nephrometry scoring system [2], the PADUA classification system [3], and the C-Index method [4], each provides a unique score, using variables that cannot be transformed for cross compatibility. The assignment of an appropriate score using any of these three systems relies on systematic and consistent reading and interpretation of CT findings. Okhunov et al. showed the highest ICC (0.92) using the R.E.N.A.L. system, followed by the C-Index (0.84) and the PADUA (0.81) systems. Correlation with length of warm ischemia time and percent change in creatinine level (all systems), and inverse correlation with length of stay (LOS) (C-Index) was also found [9]. Good concordance and kappa values amongst readers were also found in studies on IOV of the R.EN.A.L. system used for open or laparoscopic PN patients, respectively [14,15]. In these studies, however, scores were assigned by small groups of urologists at different stages of professional development but with similar urological training.

Our study shows that accuracy, interpretability, and reproducibility of different nephrometry scoring systems become problematic when scores are assigned by readers with different areas of expertise and levels of training, or no medical training whatsoever. Although differences in levels of agreement amongst readers with varying degrees of experience were not statistically significant, we found that more extensive clinical and radiologic training resulted in higher ICC and smaller differences from the reference standard (RFEL). However, despite agreement between RFEL and PHYS was greater than between RFEL and NMS, PHYS' scores differed from RFEL's scores by up to 4 points on R.E.N.A.L. and PADUA. Such variability in systems with an 8-point range like R.E.N.A.L. and PADUA may lead to inconsistent classification of renal masses. For instance, when R.E.N.A.L nephrometry sums (NS) are categorized into low- (NS: 4-6), moderate- (NS: 7-9), and high- (NS: 10-12) complexity lesions for risk stratification [6], the same lesion might be scored 6 (low complexity) and 10 (high complexity) by different readers. For these reasons, based on the clear indication that reader experience - in addition to scoring system complexity - is a key factor in the fidelity of scoring outcomes, we strongly recommend standardized reporting for publications using nephrometry scores detailing reader experience level.

Before arbitrarily adopting one of the existing systems to score an entire database of PN cases both retrospectively and prospectively, we sought to determine whether any of the three nephrometry scoring systems studied should be preferentially utilized for practical and idealized characteristics (learning and ease of applicability; score consistency and reproducibility among readers; correlation of scores with surgical complexity and clinical outcomes). In our cohort of patients the mathematical model C-Index, which is a measure of tumor centrality, showed the highest level of agreement amongst readers and appeared to be the least prone to subjectivity of the three systems. Consistent with this finding and those of prior studies [9,14,15], the analysis of the IOV of each system's individual components showed that the (R)adius and the (A)nterior/posterior subscales of the R.E.N.A.L. system and tumor size in the PADUA system had the highest concordance rates amongst readers while the (L)ocation relative to polar lines (R.E.N.A.L.) and the estimated exophytic component (PADUA) had the lowest concordance rates. Increased variability and reduced reproducibility amongst readers was observed particularly in instances where tumor size and location, and the degree of rotation of the kidney in respect to its ideal coronal view reduced the readers' ability to identify anatomical landmarks needed for the determination of the tumor location relative to the polar and sinus lines in the R.E.N.A.L. and PADUA systems, respectively. Furthermore, the impact of all these factors on the point-based R.E.N.A.L. and PADUA scoring systems resulted in greater discordance amongst readers than that observed using the C-Index, which is based on continuous measures. Nonetheless, a certain degree of approximation and variability was observed with the C-Index as well, in particular when an imaginary ellipse around the kidney periphery has to be drawn to define the mid-polar reference point, and the distance (x) between the central axial reference point and the tumor center. In our study, however, the least discordance among readers was associated with the C-Index.

We were also interested in assessing each nephrometry system's ability to predict clinical outcomes. Despite having the lowest IOV, C-Index ability to predict ischemia time and percent change in eGFR was not much higher than that of R.E.N.A.L. and PADUA. In our cohort, the low rate of adverse events such as postoperative hematoma and urinary fistula formation prevented us from performing an analysis to find correlations between any of the nephrometry systems and these complications. Of note, operative time, estimated blood loss, and LOS were not included in our analysis because inherently different in different PN approaches. All three nephrometry systems showed the ability to help predicting ischemia time. However, we found no evidence that any of the systems was associated with percent change in eGFR from baseline with the exception of C-index at 6 postoperative weeks. Among subscales, only tumor size, nearness to sinus, and location relative to polar lines (R.E.N.A.L.) and tumor size, renal sinus involvement and collecting system involvement (PADUA) were correlated with ischemia time. No significant correlations were found between any of the subscales and percent change in eGFR from baseline. Our results, corroborated by the heterogeneity of outcomes from other studies [9.10,11,12], suggest that some of the arbitrary components of nephrometry systems may be clinically insignificant, supporting the need for further refinement to identify reliable predictors of post-surgical outcomes such as renal function and complication risk.

Limitations of this study include the relatively small size of the cohort which may have been insufficient in detecting small differences between groups.

Conclusions

IOV in existing nephrometry systems is reduced by increasing radiological and clinical expertise of the reader. Consistently, measures of tumor size and distance to intrarenal structures were useful in predicting surgical complexity and clinically relevant outcomes. Other morphologic descriptors of renal lesions appeared irrelevant for such predictions and may potentially be eliminated.

C-index scoring had the best performance of the three nephrometry systems studied and could predict short-term postoperative renal function.

Abbreviations

ASA

American Society of Anesthesiologists

C-Index

Centrality Index

CI

Confidence Interval

CT

Computed Tomography

CKD-EPI

Chronic Kidney Disease Epidemiology Collaboration

CT

Computed Tomography

eGFR

Estimated Glomerular Filtration Rate

ICC

Intraclass Correlation Coefficient

IOV

Interobserver Variability

IQR

Interquartile Range

MSKCC

Memorial Sloan Kettering Cancer Center

NMS

Non-Medical Secondary School Student

NS

Nephrometry Sum

PADUA

Preoperative Aspects and Dimensions Used for an Anatomical Classification System

PHYS

Physicians (Urology Fellows + Radiology Resident)

PN

Partial Nephrectomy

R.E.N.A.L

Radius, Exophytic/endophytic, Nearness, Anterior/posterior, Location

RFEL

Radiology Fellow