Variability of Inter-observer Agreement on Feasibility of Partial Nephrectomy Before and After Neoadjuvant Axitinib for Locally Advanced RCC: Independent Analysis from a Phase II Trial

Abstract

Objective

To evaluate how many patients could have undergone PN instead of RN before and after neoadjuvant axitinib therapy, as assessed by 5 independent urologic oncologists, and to study the variability of inter-observer agreement.

Patients and Methods

Pre- and post systemic treatment CT scans from 22 patients with ccRCC in a phase II neoadjuvant axitinib trial were reviewed by 5 independent urologic oncologists. RENAL score and Kappa statistics were calculated.

Results

Median RENAL score changed from 11 pre-treatment to 10 post-treatment, p=0.0017. Five tumors with moderate-complexity pre-treatment remained moderate-complexity post-treatment. Of 17 tumors with high-complexity pre-treatment, 3 became moderate-complexity post-treatment. Overall kappa statistic was 0.611. Moderate-complexity kappa was 0.611 vs. high-complexity kappa of 0.428. Pre-treatment kappa was 0.550 vs. post-treatment of 0.609. After treatment with axitinib, all 5 reviewers agreed that only 5 patients required RN (instead of 8 pre-treatment) and that 10 patients could now undergo PN (instead of 3 pre-treatment). The odds of PN feasibility were 22.8-times higher after treatment with axitinib.

Conclusions

There is considerable variability in inter-observer agreement on the feasibility of PN in patients treated with neoadjuvant targeted therapy. Although more patients were candidates for PN after neoadjuvant therapy, it remains difficult to identify these patients a priori.

INTRODUCTION

Patients with renal masses in the setting of advanced chronic kidney disease, bilateral tumors, or a solitary kidney have an imperative indication for partial nephrectomy (PN), but that may not be possible in large complex tumors.

Several small series, mostly retrospective, have studied the role of targeted therapy in downsizing renal tumors, and potentially allowing change of surgical technique[1–12]. For example, some tumors that were considered unresectable (based on imaging) were rendered resectable after a certain period of targeted therapy; and other tumors that were destined for radical nephrectomy (RN) were changed to PN after treatment with targeted therapy. However, these assessments remain highly subjective.

We recently reported the results of a phase II trial with neoadjuvant axitinib in patients with locally advanced clear cell renal cell carcinoma (ccRCC), with an impressive partial response rate of 45.8%, and a median tumor diameter decrease of 28.3%. While it may be possible to use axitinib to shrink renal tumors and allow for a PN after treatment, such judgment is still subjective.

The primary objective of this study was to evaluate how many patients could have potentially undergone PN instead of RN before and after neoadjuvant axitinib therapy, as assessed by 5 independent urologic oncologists, and to study the variability of inter-observer agreement of such an assessment. In this study, we used a novel approach by having blinded (to patient identity, and pre-/post-treatment status) independent urologic oncologists evaluate complete high-quality CT scans from a prospective clinical trial.

PATIENTS AND METHODS

Eligibility Criteria

The Institutional Review Board at The University of Texas MD Anderson Cancer Center approved this study. The current study is a post hoc investigation based on the imaging and clinical materials obtained from a previously published phase II study[13] of neoadjuvant axitinib in patients with locally advanced (clinical T2–T3b) non-metastatic ccRCC (NCT01263769). Original clinical trial details, such as inclusion/exclusion criteria, drug dosage/schedules, adverse events, intraoperative/postoperative/pathology findings have already been reported[13].

For the current study, patients were eligible if they had renal protocol CT scans available at baseline (pre-treatment) and at 11–12 weeks after treatment (post-treatment) for review. Twenty-four patients were included in the original study[13], of which 23 were included in the current study (as 1 patient was taken off study early). Additionally, one pretreatment CT scan could not be transferred to DVD format and was not available for review. The final total number of CT scans available for review was therefore 45 (but only 22 had both pre-treatment and posttreatment CT scans available for review).

Radiologic Evaluation

The images from CT scans were de-identified and copied on DVDs by the Radiology Department at The University of Texas MD Anderson Cancer Center (1 CT scan per DVD), and each complete CT scan was given a random unique identification number, known only to the study principal investigator. These 45 DVDs were then securely mailed to each of 5 independent reviewers with urologic oncology expertise. These independent reviewers were not involved in the original clinical trial. These reviewers were aware that the current study patients were treated with neoadjuvant axitinib at the University of Texas MD Anderson Cancer Center, but were blinded to the status of the CT scan (whether it was pre-treatment or post-treatment), patient demographics, response status after axitinib treatment, patient outcomes, type of surgery performed, as well as the identity and results from the CT readings of the other reviewers.

For each of the 45 CT scans, each independent reviewer answered the questions “Can you do a PN with >90% confidence of obtaining negative margins and leaving enough viable renal tissue?”; “If you answered No, what is the reason for not doing a PN?”; and “If you do a partial, what % risk of grade III or higher Clavien complication would you quote the patient?”. Results were sent back to the study principal investigator and analyzed by the study statistical team.

One of the study principal investigators calculated the RENAL score for each of the 45 CT scans[14]. RENAL scores of 7–9 were considered moderate-complexity and scores of 10–12 as high-complexity.

Study Objective

The primary objective of this study was to document how many patients, before, and after axitinib therapy, could have potentially undergone PN instead of RN, as assessed by 5 independent urologic oncologists, and study the variability of inter-observer agreement of such an assessment. In addition, we evaluated the inter-observer agreement stratified by the RENAL complexity and the RENAL nephrometry (score and complexity) changes after axitinib therapy.

Statistical methods

Kappa statistics and 95% bias-corrected confidence intervals were calculated for all 45 scans and for subsets of the scans based on the time point(pre- or post-treatment with axitinib) and RENAL complexity(moderate or high) of the renal tumor anatomy, as well as time by complexity. One-thousand bootstrap replicates were used to calculate the confidence intervals. Kappa statistics of 0.01–0.2, 0.21–0.40, 0.41–0.60, 0.61–0.80, 0.81–1.00 were considered to indicate slight, fair, moderate, substantial, and almost perfect agreement, respectively[15].

Additionally, the number of outside reviewers who agreed a PN could be completed was calculated for each scan, and its distribution was summarized separately for pre- and post-treatment scans. A transition table was created to depict how patients changed before and after treatment with regard to reviewers’ recommendations for surgery. Finally, a generalized linear mixed model was created to examine impact of treatment on surgery approach. The endpoint was PN (0=No, 1=Yes); a logit link function was used to model the data. Assessment time (pre- or post-treatment) was the predictor variable, reviewer was included in the model as a fixed effect, and intercept was treated as a random effect.

RESULTS

Twenty-four patients were treated with axitinib from 2011 to 2013. Twenty-two patients had paired pre and post-treatment CT scans available for review. Median age was 61 years (range 42–83), and 17 patients were male. Median RENAL score significantly changed from 11 pre-treatment (range 7–12) to 10 post-treatment (range 7–11), p=0.0017. The 5 tumors with moderate-complexity pre-treatment remained moderate-complexity post-treatment. Of the 17 tumors with high-complexity pre-treatment, 3 were rendered moderate-complexity post-treatment. Further details on individual patient characteristics and reviewer choices are shown in Table 1.

Table 1.

Individual patient characteristics and radiologic parameters

			Tumor Size (cm)		RENAL score		Number of Reviewers Agreeing PN Feasible
Age (years)	Clinical Stage	Pathologic Stage	Pre	Post	Pre	Post	Pre	Post	Partial Nephrectomy Actually Done?
57	3a	2b	14	11.6	11	11	0	0	No
60	3a	3a	10.2	6.9	12	11	0	0	No
66	3a	3a	12.5	10.2	11	11	0	0	No
50	3a	4	16.6	11.6	11	11	0	0	No
66	3a	3a	12.3	9.6	10	10	0	1	No
67	3a	3a	11.5	9.1	11	11	1	1	No
45	3a	3a	11	8.4	11	11	0	2	No
68	3a	3a	8.3	5.8	11	10	0	2	No
64	3a	3a	9.4	8.9	11	11	1	2	No
43	3a	3a	9.7	7.8	11	11	1	2	No
42	3a	3a	9.1	5.3	12	10	0	3	No
48	3a	3a	9.6	6.4	11	10	1	3	No
56	3a	3a	5.2	3.3	8	7	1	5	No
60	3a	3a	10.6	6.7	10	10	1	5	No
52	3a	3a	9.8	6.7	11	9	1	5	No
66	3a	3a	11.4	7.7	9	8	1	5	No
64	3a	3a	10.7	7.9	10	10	3	5	No
69	3a	1a	5.6	3.3	9	8	4	5	Yes
83	3a	1b	6.7	5.4	9	9	4	5	Yes
51	3a	1a	4.2	2.4	10	8	5	5	Yes
69	3a	1a	4.6	3.3	10	8	5	5	Yes
62	3a	3a	6.8	5.8	7	7	5	5	Yes

Table 2 shows the kappa statistics. The overall kappa statistic was 0.611 (95% CI: 0.452–0.772) indicating substantial agreement. Moderate-complexity kappa was 0.611 (95% CI: 0.000–0.734) vs. high-complexity kappa of 0.428 (95% CI: 0.180–0.734). Pre-treatment kappa was 0.550 (95% CI: 0.235–0.761) vs. post-treatment of 0.609 (95% CI: 0.378–0.814).

Table 2.

Kappa values and 95% bias-corrected confidence intervals (CI)

Cohort	N	Kappa	95% CI
Overall	45	0.611	0.452	0.772
Moderate complexity	13	0.611	0.000	0.734
High complexity	32	0.428	0.180	0.655
Pre-Treatment	22	0.550	0.235	0.761
Moderate complexity	5	0.461	0.000	0.697
High complexity	17	0.492	0.037	0.821
Post-Treatment	23	0.609	0.378	0.814
Moderate complexity	8	complete agreement*
High complexity	15	0.352	0.053	0.682

^*All 5 reviewers reported Yes (PN is feasible) for all 8 scans

During the actual clinical trial, prior to treatment with axitinib, all patients were deemed suitable candidates for RN; however, after treatment, 5 patients underwent PN (a sixth patient was offered PN after treatment but preferred to undergo RN). Prior to treatment with axitinib, all 5 independent reviewers agreed on the necessity to perform a radical nephrectomy on 8 of the 22 patients, and all 5 reviewers agreed on the feasibility of partial nephrectomies in only 3 patients. However, after treatment, all 5 reviewers agreed that only 5 patients required radical nephrectomies (instead of 8) and 10 patients could now undergo partial nephrectomies (instead of 3) (Table 3). Comparing pre- and post-treatment agreement of the 5 reviewers in these 10 potential PN cases to the actual partial nephrectomies being performed, 4–5 of the 5 reviewers agreed that PN was feasible on pre-treatment scans in the 5 cases who actually underwent a PN. However, in 5 other patients who were considered by the reviewers as candidates for PN on post-treatment scans but who were actually treated by RN, only 1–3 of the reviewers considered PN feasible on pretreatment scans (Table 1). Examples of CT scans of patients pre- and post-treatment, along with reviewer agreement, are shown in Figure 1.

Table 3.

Number of patients for whom PN was deemed feasible, stratified by assessment time and agreement among independent reviewers

	Number of Independent Reviewers Agreeing PN Is Feasible
Assessment	0	1	2	3	4	5
Pre-Treatment	8 (36.36%)	8 (36.36%)	0 (0.00%)	1 (4.55%)	2 (9.09%)	3 (13.64%)
Post-Treatment	5 (21.74%)	2 (8.70%)	4 (17.39%)	2 (8.70%)	0 (0.00%)	10 (43.48%)

Figure 1.

Number of reviewers reporting that partial nephrectomy is feasible. Note that although figures below are displayed in a paired fashion, the reviewers were blinded to patient identity as well as pre-/post-axitinib status.

The transition table (Table 4) shows how agreement regarding PN feasibility changed after axitinib treatment for individual patients. Although table 3 shows that all reviewers agreed that 8 of the patients required radical nephrectomies prior to treatment and that 5 patients could have PN after treatment, table 4 shows that of the original 8 patients that were deemed to require a RN, all 5 reviewers agreed that 4 of these 8 patient still required a RN after treatment, while one reviewer felt a PN was feasible in one patient after treatment, two felt a PN was feasible in two patients after treatment, and three felt a PN was feasible in a fourth patient after treatment.

Table 4.

Transition Table: Change in feasibility of PN by number of independent reviewers agreeing upon PN

Pre-Treatment		Post-Treatment
		Number of Independent Reviewers Agreeing PN Is Feasible
		0	1	2	3	4	5
Number of Independent Reviewers Agreeing PN Is Feasible	0	4 (50)	1 (12.5%)	2 (25)	1 (12.5%)	0	0
	1	0	1 (12.5%)	2 (25)	1 (12.5%)	0	4 (50%)
	2	0	0	0	0	0	0
	3	0	0	0	0	0	1 (100%)
	4	0	0	0	0	0	2 (100%)
	5	0	0	0	0	0	3 (100%)

For the reviewers who reported that PN is not feasible, the reasons for lack of feasibility were “Cannot achieve negative margins” in 14.6%(N=18), “Not enough viable kidney left in place” in 22%(N=27), “High risk of renal vascular injury” in 17.9%(N=22), a combination of 2 of the 3 reasons in 21.1%(N=26), all three reasons simultaneously in 16.3%(N=20), and other reasons in 8.1%(N=10).

For the reviewers who reported that PN is feasible, the % risk of grade III or higher Clavien complication that they would quote to the patient was 0–10% in 62.8%(N=59), 15–30% in 27.7%(N=26), and >30% in 9.6%(N=9).

Based on the independent reviewer input, the generalized linear mixed model indicated that the odds of PN being feasible were 22.8 times higher after treatment with axitinib than the odds before treatment with axitinib(95% CI: 7.4 –71.4).

DISCUSSION

In RCC patients with poor renal function and/or solitary kidneys, PN may significantly improve quality of life. But unfortunately, PN may not be possible in patients with large and complex tumors. Although neoadjuvant and pre-surgical targeted therapy may shrink primary tumors[1–12], few studies have focused on the ability of targeted therapy to facilitate PN[16]. A notable weakness for designing clinical trials to facilitate PN is the subjective and multifactorial determination of whether PN is feasible. Therefore, the current study was designed to evaluate the inter-observer agreement on feasibility to perform a PN using five blinded reviewers to determine their chosen approach from complete high-quality CT images obtained from a recent phase 2 clinical trial. Of the 22 patients with paired pre and post-treatment CT scans, all 5 reviewers agreed that treatment with axitinib potentially enabled 7 patients (32%) not amenable to PN pre-treatment, to undergo PN post-treatment.

Not surprisingly, we found that the agreement on whether a PN is feasible was in general higher for moderate-complexity tumors than those with high complexity. When looking only at pre-treatment scans, agreement was only moderate for both moderate-complexity and high-complexity tumors. However, when evaluating post-treatment scans, there was almost perfect agreement on moderate-complexity tumors, and only fair agreement on high-complexity tumors.

When looking at scenarios where all 5 reviewers agreed on feasibility of PN, our results show that there are 10 patients where PN was deemed feasible (by all 5 reviewers) after axitinib treatment, compared to only 3 patients prior to treatment. Of equal importance is that all patients who were deemed to be candidates for PN pre-treatment were also considered by the reviewers for PN post-treatment, adding to the internal consistency of the study, as the reviewers were completely blinded to pre- versus post-treatment status. However, due to the lack of predictive clinical and biomarkers it remains difficult to identify possible candidates for PN after downsizing a priori.

Hellenthal and colleagues[17] prospectively studied preoperative sunitinib in 20 patients with ccRCC, of whom 8 underwent laparoscopic PN (One of whom had positive surgical margin). Pretreatment, 7 of these patients were cT1b and one was cT2. Post-treatment, 5 were pT1a, 2 pT1b, and 1 pT3a. The mean decrease in tumor size for the whole study cohort was 11.8%, however the mean change in size in the PN cohort was not reported separately. The authors did not annotate RENAL score nor specify conversion of surgery from radical to PN as an endpoint prospectively, so it is not clear if these 8 patients could have undergone PN without the use of sunitinib. Silberstein and colleagues [9] studied 12 patients with 14 renal tumors in a combined retrospective/prospective-pilot study. All 12 patients had biopsy-proven ccRCC and an imperative indication for PN (chronic kidney disease, solitary kidney or bilateral renal tumors). PN was done on 14 tumors in 12 patients, with negative surgical margins. Clinical staging was 1 cT1a, 6 cT1b, 6 cT2 and 1 cT3). Primary renal tumors decreased in size by a mean of 1.5cm (range 0.2–3.2cm), or 21.1% (mean 7.1cm pretreatment to 5.6cm posttreatment). Similarly, the authors did not annotate RENAL score nor specify conversion of surgery from radical to PN as an endpoint prospectively, so it is not clear if these 12 patients could have undergone PN as well without the use of sunitinib. Lane and colleagues [12] compiled data from 4 centers (including patients from studies in references 5 and 9) where patients were treated with preoperative sunitinib. Seventy-two patients were evaluated and only 12% were considered cT3 at baseline. PN was performed on 49 kidneys, including 91%, 88%, and 43% for localized cT1a, cT1b, and cT2–3 tumors. PN was performed in 11 of 27 tumors (41%) with post-treatment high-complexity RENAL score. Surgical margin status in patients who underwent PN was not reported in this study. More recently, Rini and colleagues [16] reported a phase II trial of neoadjuvant pazopanib in 25 patients with ccRCC (on biopsy). Median tumor diameter decreased from 7.3cm to 5.5cm after pazopanib, and RENAL score complexity decreased in 36% of tumors. Of the 13 patients that were deemed to require RN prior to pazopanib, 6 were able to undergo PN after treatment with pazopanib. In this study, 18 patients underwent PN, 2 of which had positive surgical margins.

In the neoadjuvant axitinib prospective study[13], 5 of 24 patients underwent PN, and all 5 were considered cT3a prior to therapy. All 5 had negative surgical margins. Pathologic evaluation revealed pT1a in 3, pT1b in 1 and pT3a in 1 patient. Two of these tumors had partial response while 3 had stable disease by RECIST. However, since we did not prospectively specify conversion of radical to PN as an endpoint in our study, we cannot necessarily consider the ability to perform PN in these 5 patients to be the result of axitinib treatment. In the current study, we found that while the median RENAL score itself decreased from 12 to 11 (p=0.0017), 19 of the 22 patients had stable complexity measurements and only 3 patients had a decrease in RENAL complexity category (from high to moderate complexity).

Using change from RN to PN as a primary endpoint for neoadjuvant targeted therapy clinical trials is still a very subjective measure. In addition, using high-complexity RENAL scores for inclusion in such trials is not necessarily accurate, as several studies have shown PN to be feasible (open, laparoscopic and robotic) in patients with high-complexity RENAL scores. For example, Simhan and colleagues[18] reported on the feasibility of open or robotic PN (2007–2010) in a cohort of 390 patients that included 217 (55.6%) patients with moderate-complexity tumors (136 open, 81 robotic), and 64 (16.4%) high-complexity tumors (54 open,10 robotic). Similarly, Tomaszewski and colleagues[19] from the same institution (2007–2012) reported on 254 patients who underwent PN, of whom 187 (73.6%) had moderate-complexity tumors and 67 (26.4%) had high-complexity tumors. Khalifeh and colleagues[20] reviewed a single surgeon series of 500 cases of laparoscopic and robotic PN (2002–2012), where 195 (39%) were moderate-complexity (63 laparoscopic, 132 robotic) and 46 (9.2%) were high-complexity (13 laparoscopic, 33 robotic), respectively. Similarly, Gorin and colleagues[21] reported on a multi-institutional study with 978 evaluable patients who underwent robotic PN, where 483 patients (49.4%) had moderate complexity tumors, and 83 (8.5%) had high complexity tumors. In addition, Lane and colleagues[22] evaluated 1,433 patients treated with surgery by 19 surgeons. In this study, 461 patients (32.1%) with intermediate-complexity tumors and 115 patients (8%) with high-complexity tumors underwent PN. Finally, the variability in performing PN has been shown, not surprisingly, to be dependent on both the surgeon experience, as well as the setting where the surgeon practices[23].

The limitations of our study are the small sample size, and the single-institutional setting of the original trial[13]. In addition, although the reviewers were blinded to the clinical trial results, they were aware of the fact that they were asked to review CT scans from a neoadjuvant axitinib phase II trial performed at a tertiary referral center. This may have introduced a Hawthorne-effect[24] (or observer effect), potentially biasing the results by giving positive answers on the ability to perform a PN. In addition, change in attenuation and tumor morphology on CT scans may have suggested which tumors were after neoadjuvant treatment. The reviewers were not asked to determine the RENAL scores independently, as this has been done in prior studies, and is not an aim of the current study. The choice of answers regarding what reasons the reviewers believe PN is not feasible are subjective and the reasons are likely multifactorial and are difficult to quantitate/capture in a purely objective manner. Similarly, the rate of grade III or higher complications is also subjective, as the published studies with predictive factors for urine leaks and other complications have not been externally validated.

In conclusion, our study shows that inter-observer agreement on the feasibility of PN in patients treated with neoadjuvant targeted therapy is quite variable. Although it seems that more patients were deemed candidates for PN after neoadjuvant therapy, it is still difficult to identify these patients a priori. In addition, using RENAL score is not necessarily sufficient as an inclusion criterion for clinical trial entry (and indeed it was not designed for this purpose), as even patients with high-complexity scores can sometimes undergo PN. Patients with imperative need for PN should still be carefully and selectively considered for neoadjuvant targeted therapy when appropriate. As a community interested in this very relevant clinical endpoint, we should use these data to devise a “resectability score” that is reliable and reproducible enough to be translated into clinical practice, and to be used across centers by different urologic surgeons, for both daily practice and clinical trial purposes.