Lessons Learned from EVOLVE for Planning of Future Randomized Trials in Patients on Dialysis

Abstract

The effect of the calcimimetic cinacalcet on cardiovascular disease in patients undergoing hemodialysis with secondary hyperparathyroidism was assessed in the Evaluation of Cinacalcet Hydrochloride Therapy to Lower Cardiovascular Events trial. This was the largest (in size) and longest (in duration) randomized controlled clinical trial undertaken in this population. During planning, execution, analysis, and reporting of the trial, many lessons were learned, including those related to the use of a composite cardiovascular primary endpoint, definition of endpoints (particularly heart failure and severe unremitting hyperparathyroidism), importance of age for optimal stratification at randomization, use of unadjusted and adjusted intention-to-treat analysis for the primary outcome, how to respond to a lower-than-predicted event rate during the trial, development of a prespecified analytic plan that accounted for nonadherence and for cointerventions that diminished the power of the trial to observe a treatment effect, determination of the credibility of a subgroup effect, use of adverse effects database to investigate rare diseases, collection of blood for biomarker measurement not designated before trial initiation, and interpretation of the benefits-to-harms ratio for individual patients. It is likely that many of these issues will arise in the planning of future trials in CKD.

Introduction

The Evaluation of Cinacalcet Hydrochloride Therapy to Lower Cardiovascular Events (EVOLVE) trial was the largest (n=3883) and longest (median follow-up, 50.5 months) randomized controlled trial (RCT) undertaken in patients receiving hemodialysis (1). The hypothesis tested was that the calcimimetic cinacalcet would, compared with placebo, reduce the risk of death or nonfatal cardiovascular (CV) events among patients with secondary hyperparathyroidism (sHPT; intact parathyroid hormone [iPTH] ≥300 pg/ml) who were undergoing hemodialysis. In the primary unadjusted intention-to-treat (ITT) analysis, cinacalcet did not significantly reduce the risk of the composite CV primary outcome, but there were many design and analysis issues that should be considered when the results are interpreted. In this review, we discuss study challenges faced, decisions made, and lessons learned from EVOLVE and how they may be relevant in the design of future trials in dialysis and CKD.

Choice, Definition, and Analysis of Primary Events

Composite Primary Outcome

The primary endpoint was a composite composed of all-cause mortality or nonfatal CV events (myocardial infarction, hospitalization for unstable angina, heart failure, or peripheral vascular disease). These CV events may occur via different pathophysiologic pathways: myocardial infarction and unstable angina are principally caused by atherosclerotic vascular disease, while heart failure results from cardiomyopathy (with systolic or diastolic dysfunction), for which arteriosclerosis and uremia are predisposing factors (2). Peripheral vascular disease is often atherosclerotic, but distal occlusive disease, particularly of the lower extremities, is often accompanied by dense calcific arteriosclerosis. The CV events were chosen because we hypothesized that cinacalcet would (1) reduce heart failure and death by reducing medial calcification of conduit arteries (thus improving vascular compliance and decreasing left ventricular hypertrophy) and by decreasing the potential cardiotoxic effect of parathyroid hormone (PTH), and (2) reduce atherosclerotic events by decreasing intimal calcification of atherosclerotic stenoses (3). If the endpoint had been mortality alone, a much larger sample size would have been necessary.

Inclusion of overall mortality in the composite outcome is an issue because CV mortality would likely be modified by cinacalcet, in contrast to non-CV mortality. We included overall mortality in the primary outcome because we thought it necessary to demonstrate that the intervention did not cause unanticipated serious adverse effects on non-CV outcomes (such as cancer or death after fracture).

Post hoc evaluation of the treatment effects on CV events using multivariable-adjusted ITT analysis showed relative hazards of 0.88 (95% confidence interval [95% CI], 0.76 to 1.01) for time to first atherosclerotic event, 0.79 (95% CI, 0.66 to 0.96) for heart failure, and 0.79 (95% CI, 0.64 to 0.98) for sudden death; the latter two events are considered outcomes of nonatherosclerotic disease (4). The 95% CIs for the atherosclerotic and nonatherosclerotic outcomes overlap, but the magnitude of the relative hazard for heart failure and sudden death is consistent with the hypothesis that cinacalcet may act through the nonatherosclerotic pathway. Because medial calcification and intimal calcification may well be different entities in CKD (5), this suggests that the dominant effect of cinacalcet may be through the inhibition of medial calcification.

Definition of Outcomes

Clinically relevant, precisely defined, reproducible end points with central adjudication are necessary in RCTs that aim to change clinical practice (6). Death and major atherosclerotic CV events fulfill these criteria, but the definition of heart failure in CKD has been problematic because of the difficulty in differentiating salt and water overload from impaired left ventricular function. This issue was recently crystalized by the proposal for a functional classification system of heart failure in patients with ESRD (7). This system may be limited by the cost of echocardiography and lack of specificity of dyspnea relief by dialysis for diagnosis of heart failure.

In patients receiving dialysis, heart failure has been defined as dyspnea with at least two of the following four manifestations: bilateral basilar rales on physical examination, raised jugular venous pressure, interstitial edema on chest radiography, and increased upper pulmonary vessel diameter on chest radiography (8). The presence or history of heart failure was associated with an almost two-fold increase in mortality (8). In EVOLVE, we used a definition similar to that used by Harnett et al. (8) but added a clause to improve specificity: the patient also was required to have received mechanical ultrafiltration or hemodialysis. Nonetheless, the event adjudication committee confirmed the diagnosis of heart failure in only about half of heart failure events submitted by the local investigators, a result lower than that observed in high-risk patients with hypertension (9). This lack of certainty in the diagnosis of heart failure may limit study power because the event rate may be lower than anticipated, and misclassification may limit the capacity to identify a treatment effect. A similar misreporting of myocardial end points occurred in the Second Platelet IIb/IIIa Antagonist for the Reduction of Acute Coronary Syndrome Events in a Global Organization Network Trial and Platelet Glycoprotein IIb/IIIa in Unstable Angina: Receptor Suppression Using Integrilin (Eptifibatide) Therapy Study, in which investigators misreported myocardial infarction end points, but most investigators later agreed with assessments of these events by the clinical events committee (10,11).

End Points with Selection Bias

Certain end points for mineral and bone disease in CKD are also problematic. For example, the end point for severe hyperparathyroidism (HPT) could be parathyroidectomy (PTX), but severe HPT may or may not be treated with PTX. PTX was a secondary end point in EVOLVE; however, before the start of the study it was evident that the criteria by which physicians selected PTX for the treatment of sHPT varied widely across the world, and the trial protocol provided no criteria for PTX. Factors associated with PTX during the trial indicated selection effects and included younger age, female sex, higher body mass index, markers of comorbidity (no history of valvular disease, angina, or peripheral vascular disease), higher serum calcium, and higher PTH; the use of PTX also varied widely by country of origin (lowest rates were in the United States) (12). In addition, PTX was performed at an advanced phase of sHPT: before PTX the mean PTH level was 1872 pg/ml and the mean serum calcium level was 10.3 mg/dl (2.58 mmol/L) (12).

Because there were likely to be additional patients not selected for PTX who might have undergone PTX but did not because of various reasons unrelated to disease severity (e.g., surgeon or patient reluctance or no trained surgeon available), we defined an outcome before the trial was completed. We termed this outcome severe unremitting HPT, which included PTX or the presence of severe HPT with hypercalcemia (iPTH>1000 pg/ml with serum calcium >10.5 mg/dl [2.61 mmol/L] on two consecutive occasions or iPTH>1000 pg/ml with serum calcium >10.5 mg/dl [2.61 mmol/d] on one occasion and subsequent use of commercial [off-protocol] cinacalcet within 2 months of laboratory assessment). Although hypercalcemia is a biochemical endpoint, its occurrence implies an advanced phase of HPT, and clinical practice supports PTX in this scenario. In the placebo group, severe unremitting HPT occurred in 24.3% (n=470) of patients, of whom 59% (n=278) had PTX. The relative hazard (cinacalcet versus placebo) for severe unremitting HPT was 0.43 (95% CI, 0.37 to 0.50). This relative hazard was similar whether baseline iPTH was mildly or markedly elevated, but the number of events prevented increased as iPTH increased (12).

Lessons from EVOLVE concerning choice and definitions of end points are presented in Table 1.

Table 1.

Lessons concerning choice and definition of composite end points

(1) Primary composite endpoints should include events resulting from pathophysiologic pathways highly likely to be influenced by the intervention.

(2A) Heart failure in patients receiving dialysis occurs frequently but is often misdiagnosed by nephrologists even when the definition includes manifestations of pulmonary edema. Sample size should be increased to account for misclassification.

(2B) Randomized clinical trials in CKD or ESRD that include an end point of heart failure should include focused education to ensure accurate identification of heart failure events.

(2C) A classification system of heart failure based on patient-reported dyspnea assessed pre- and post-ultrafiltration in conjunction with echocardiography may be useful (6) but requires validation.

(3) Selection bias associated with parathyroidectomy necessitated a more holistic approach to the definition of severe unremitting hyperparathyroidism.

Imbalance in Baseline Clinical Characteristics

Despite enrolling 3883 patients, there was a 0.8-year difference in mean age at baseline and a 1-year difference in median age (55 versus 54 years), an occurrence that confounded the primary outcome (1). This chance imbalance in a major prognostic factor for CV events necessitates covariate adjustment (13). In the prespecified primary analysis using an unadjusted ITT analysis, the relative hazard was 0.93 (95% CI, 0.85 to 1.02) and the prespecified multivariable-adjusted ITT relative hazard was 0.88 (95% CI, 0.80 to 0.98). The observed age imbalance probably occurred by chance because the likelihood of imbalance is dictated by the population SD for age and sample size. The probability of an age difference of >0.8 years occurring in EVOLVE was 0.08 as the SD was 14 and sample size was 3883. Because of more restrictive inclusion criteria and the older patients enrolled, the SD for age was narrower in the Study of Heart and Renal Protection (12 years) (14) and the Trial to Reduce Cardiovascular Events with Aranesp Therapy (10 years) (15), wherein the probability of observing an age difference (by chance) of >0.8 years between groups was 0.04 and 0.01, respectively.

The Consolidated Standards of Reporting Trials recommends reporting both unadjusted and adjusted analyses and stating whether the adjusted analysis was planned (16). Although handling of baseline covariates in RCTs varies substantially (17), covariate adjustment improves the accuracy of treatment effect estimation and statistical power and hence should be performed when strong prognostic factors are observed or anticipated (13,18).

Lessons concerning imbalance in baseline covariates are presented in Table 2.

Table 2.

Lessons concerning imbalance in baseline covariates

(1) Randomization is not guaranteed to prevent significant imbalance in baseline clinical characteristics (even for a trial with nearly 4000 participants).

(2) Randomized controlled trials in CKD/ESRD should use both multivariable-adjusted ITT and unadjusted ITT analyses of the primary outcome.

(3) If there is risk of imbalance of baseline covariates, multivariable-adjusted ITT analysis should be used as the primary analysis on which the trial is judged (18).

(4) If participants across a broad age range are enrolled, stratification by age may be advisable because age is an important determinant of many outcomes. A similar argument could be made for other key covariates (e.g., baseline BP or proteinuria) depending on the outcome(s) of interest.

ITT, intention-to-treat.

Threats to Statistical Power

The major causes of reduction in statistical power in EVOLVE were a lower-than-predicted primary composite event rate, discontinuation of study drug (particularly due to adverse effects in the cinacalcet group and failure to control sHPT in the placebo group), and PTH-lowering cointerventions (i.e., PTX, kidney transplantation, and use of commercial cinacalcet) (Figure 1), with PTX and commercial cinacalcet disproportionately used in the placebo group.

Figure 1.

Four main causes of reduction in statistical power that occurred in the Evaluation of Cinacalcet Hydrochloride Therapy to Lower Cardiovascular Events trial. KT, kidney transplant; PTX, parathyroidectomy.

Event Rate

We calculated the sample size on the basis of the following assumptions: an annual rate of the primary composite endpoint of 23.2% in the placebo group, a 20% treatment effect, a 1.5-year enrollment period, a 4-year total study duration, an annual rate of loss to follow-up of 1%, an annual rate of dropout (withdrawal from active treatment before a primary event) of 10% in the cinacalcet group, and a rate of drop-in (use of commercially available cinacalcet before a primary event) of 10% in the placebo group. On the basis of a two-sided log-rank test for equality of survival functions, accounting for planned interim analyses with an overall α level of 0.05, we determined that a primary event would need to occur in 1882 patients in order to ensure a power of approximately 90% (1).

We estimated event rates by integrating data from phase 3 short- and medium-term RCTs comparing cinacalcet to placebo, as well as the observational data linking expected changes in PTH, calcium, and phosphorus within components of the primary composite end point. After it became apparent that the overall (blinded) event rate was <20.8%, we extended the trial by 16 months to allow for accrual of the requisite number of events. The actual annual event rate in the placebo group was 15.5% (19).

The event rate in the placebo group of RCTs is frequently lower than predicted (20). When it became apparent that the primary composite event rate in EVOLVE was below that anticipated, the choice of intervention was to increase the number of enrolled participants or to extend the duration of trial follow-up. The first choice would have entailed re-engaging enrollment teams in dialysis units already extended by their contribution to EVOLVE and would have been more costly. The decision was made to extend follow-up. While additional events were expected to accrue over time, adherence to the intervention waned and more PTH-lowering interventions (i.e., kidney transplantation, PTX, and use of commercial cinacalcet) were instituted, particularly in the placebo group. High attrition of participants from the intervention group and drop-in to the intervention group eroded the original planned power of EVOLVE in such a manner that extension of the trial was of little benefit in identifying a treatment effect.

The lessons learned concerning event rates included:

• (1) Event rate estimates in the sample size calculation should be conservative.

• (2) If the event rate is lower than predicted and nonadherence during the trial (especially over time) is likely, then enrollment of more patients is generally preferable to extending follow-up.

Nonadherence

In the cinacalcet group, median time in the trial was 50.6 months and time on drug was 21.2 months. Corresponding times for the placebo group were 50.4 and 17.5 months, respectively. Time to first discontinuation of study drug for protocol-specified reasons was similar in both cinacalcet and placebo groups, but for nonprotocol-specified reasons it was significantly higher in the placebo group (driven by the fear of severe unremitting HPT) (1). In the placebo group, the drug was discontinued for adverse events in 11.8% of patients compared with 15.8% of patients in the cinacalcet group. Some of this discontinuation was anticipated. The drop-in rate was expected to be 10% per year and the observed rate was 7.4% per year. The dropout rate was derived from discontinuation rates observed in prior cinacalcet RCTs and was higher than anticipated: The expected rate was 10% per year and the observed rate was 27.3% per year. This may have occurred because of the long-term nature of the trial, which amplified the effect of kidney transplantation, PTX, and use of commercial cinacalcet, and because of comorbidity, which predisposed to study drug discontinuation due to adverse events and fear of severe HPT.

Drug discontinuation was a more important factor in diminishing the power of the study than the lower-than-predicted event rates. The increase in sample size necessary in the presence of x% discontinuation can be estimated as the reciprocal of (1−x)². Hence, with 40% discontinuation at the trial midpoint, the sample size necessary to maintain study power at the level planned was about 56% greater than planned.

An analytic plan to take into account nonadherence in the estimates of the treatment effect was prespecified. It included lag censoring, iterative parameter estimate, and inverse probability of censoring weights (IPCW). Lag censoring analysis uses data censored at a prespecified time point. In EVOLVE, we had prespecified the lag time to be 6 months after cinacalcet was discontinued in order to account for the possibility that the drug had persistent effects. Although lag censoring preserves randomization, there may be informative bias if nonadherent patients (compared with adherent patients) have different prognostic characteristics associated with the primary end point (21). This methodologic weakness also occurs with the iterative parameter estimate method. However, the IPCW is not prone to informative bias (22).

The IPCW approach censors data when nonadherence occurs. For patients who were adherent and had characteristics similar to those who were not, IPCW assigns larger weights to these patients to re-create the population that would have been observed (Figure 2) (22). Weights are calculated on the basis of the inverse of the probability that patients remain adherent using a logistic regression model. The final relative hazard is derived from a weighted regression model. In EVOLVE, age, sex, race, country, diabetes history, randomized treatment group, time-dependent covariates of PTH, and the adverse events of hypocalcemia and nausea/vomiting were used in the logistic regression model to estimate the probability of adherence (22). The IPCW method is sensitive to the number of nonadherent patients, assumes there are no unknown confounders, and is computationally difficult. Nonetheless, it is accepted by many health agencies. The relative hazard obtained with IPCW was 0.81 (95% CI, 0.70 to 0.92) (Table 3).

Figure 2.

Inverse probability of censoring weight (IPCW). IPCW method creates a scenario of missing follow-up data by censoring the follow-up of each participant at the time of stopping investigational product (IP) (i.e., weight =0 for time periods afterward). For participants with similar characteristics who did not stop IP, the IPCW method assigns bigger weights to participants with similar characteristics who were observed without stopping IP.

Table 3.

Treatment effect on the primary composite cardiovascular end point: prespecified sensitivity analyses

Type of Analysis	Relative Hazard (95% Confidence Interval)	P Value
Unadjusted ITT analysis	0.93 (0.85 to 1.02)	0.11
Multivariable-adjusted ITT analysis	0.88 (0.79 to 0.97)	0.01
Inverse probability of censoring weights	0.81 (0.70 to 0.92)	0.03
Censor at PTX	0.90 (0.82 to 0.99)	0.03
Censor at KT	0.90 (0.82 to 0.99)	0.03
Censor at commercial cinacalcet use	0.90 (0.82 to 0.99)	<0.001
Censor at PTX, commercial cinacalcet, or KT	0.84 (0.76 to 0.93)	<0.001
Age <65 yr	0.99 (0.88 to 1.11)	Interaction: 0.01
Age ≥65 yr	0.74 (0.63 to 0.86)

Data obtained from references 1 and 20. ITT, intention-to-treat; PTX, parathyroidectomy; KT, kidney transplant.

Effect of Cointerventions

Before the trial started, it was clear that three cointerventions that lower PTH (kidney transplantation, PTX, and use of commercial cinacalcet) could diminish the treatment effect of cinacalcet, particularly if they were delivered more frequently in the placebo group, as was expected for the latter two. We considered including PTX in a composite end point but were concerned about the lack of pathophysiologic concordance for PTX and CV events (see above). We considered censoring follow-up time after kidney transplantation or PTX but did not want to introduce bias: Patients who were too ill would be unlikely to receive either of these two interventions. Instead, we prespecified estimates of the treatment effect once follow-up time after these three PTH-lowering interventions was censored. Table 3 shows that the relative hazards were 0.90 (95% CI, 0.82 to 0.99) when censoring occurred at one of these three cointerventions and 0.84 (95% CI, 0.76 to 0.93) with censoring of time after any one of these cointerventions (1).

Table 4 outlines lessons concerning the analytic plan for nonadherence and cointerventions.

Table 4.

Lessons concerning the analytic plan for nonadherence and cointerventions

(1) Commercial availability of study drug can make drop-in a serious problem because the use of commercial drug off protocol limits the capacity of the trial to observe a treatment effect. During trial execution, major efforts are required to reduce use of commercial drug, and an analytic plan needs to take account of drop-in.

(2A) Discontinuation of study drug is likely in a long-term trial in patients on dialysis, and generous estimates should be incorporated into power calculations.

(2B) Longer follow-up time does not necessarily increase study power.

(3) An analytic plan to take into account nonadherence must be prespecified. IPCW may be the best method to account for nonadherence, particularly if determinants of nonadherence are well established.

(4) It is necessary to take account of the effect of cointerventions that limit assessment of the trial’s main treatment effect. Depending on the nature of the intervention, it may be wise to censor follow-up time after the intervention or to incorporate the intervention into a composite end point (e.g., initiation of dialysis in a trial aiming to slow progression of CKD).

IPCW, Inverse probability of censoring weight.

Subgroup Effects

We prespecified seven subgroup analyses, including age ≥65 and <65 years. The relative hazard for the primary end point using ITT was 0.74 (95% CI, 0.63 to 0.86) in the older group and 0.99 (95% CI, 0.88 to 1.11) in the younger group (19). There was a 27% reduction in mortality in patients age >65 years (P<0.001). The test of treatment×age interaction was significant (P=0.03 with age used as a continuous variable; P=0.01 with age dichotomized at 65 years). The age modification of the treatment effect was partly related to (1) three-fold higher rates of kidney transplantation and PTX in younger patients, cointerventions that limited detection of a treatment effect; (2) lower rate of CV disease at baseline in younger patients, which limited the potential of cinacalcet to decrease CV events rates; and (3) lower CV event rates that decreased the power to observe an effect (19).

Criteria by which to evaluate the credibility of subgroup effects have been published (23). Table 5 outlines the lessons from EVOLVE concerning the age subgroup effect.

Table 5.

Lessons concerning the credibility of subgroup effects

A subgroup effect is credible (23) when:  It is relatively large and highly statistically significant.

It is one of a small group of prespecified hypotheses tested.

The test of treatment×subgroup interaction is significant.

It is consistent with other reports (33).

It is consistent across related outcomes (19).

The biologic rationale is plausible.

Capture of Rare Diseases

RCTs collect accurate data on predetermined end points reviewed by an event adjudication committee, but local investigators also collect substantial amounts of data during drug exposure to assess adverse effects. This may facilitate the study of rare diseases, particularly if the disorder has been identified in advance and has a plausible biologic rationale. However, care must be taken with post hoc analyses of databases with multiple events because statistically significant associations may occur by chance. In EVOLVE, calcific uremic arteriolopathy (CUA) occurred infrequently, but cinacalcet reduced its incidence (relative hazard, 0.31; 95% CI, 0.13 to 0.79) (24). Predictors of CUA included allocation to placebo, female sex, higher body mass index, hypertension, prior PTX, and prior tobacco use. In addition, use of vitamin K antagonists at the time of CUA was significantly higher than use in patients without CUA.

The lesson learned was that RCTs can provide important information on rare diseases, particularly when the trial follows a large cohort of patients who are at relatively high risk for developing the rare disease (25), and for which a plausible biologic rationale exists.

Biomarkers

Collection and storage of blood for subsequent biomarker measurement may contribute to the understanding of disease pathogenesis (26), and in a large RCT the incremental cost of collecting patient samples is relatively low. Because of the long duration of trials (EVOLVE started in August 2006 and finished in January 2012, with primary publication of trial results in November 2012) and the potential for biomarker research to be published during the execution of the trial, we did not designate which biomarkers would be measured until after the trial was completed. From 2007 to 2011, accumulating data in patients with advanced CKD and ESRD showed that serum fibroblast growth factor-23 (FGF-23) concentrations were elevated and were associated with mortality, left ventricular hypertrophy, and CV events independent of serum phosphate, PTH, and a variety of demographic and other clinical factors (27,28). Sixty-four percent of patients assigned to cinacalcet had ≥30% reduction in serum FGF-23 concentrations from baseline to week 20 compared with 28% of the placebo group (29). Among patients randomly assigned to cinacalcet, a ≥30% reduction in FGF-23 was associated with a significant reduction in the composite CV end point (relative hazard, 0.82; 95% CI, 0.69 to 0.98) (29). This nonrandomized analysis generates the hypothesis that FGF-23 plays a role in the pathogenesis of CV events in CKD and the deleterious effects of FGF-23 might be ameliorated by cinacalcet.

The lesson learned was that collection of serum for biomarker investigation in a large RCT should be undertaken because it facilitates rapid testing of hypotheses generated after the RCT started and creation of new hypotheses.

Interpretation of Clinical Benefits

Trials should not be judged by the result of a single analysis and a single P value, but inferences should be based on the totality of the data. The decision to prescribe cinacalcet should be informed by consideration of the benefit-to-risk ratio in individual patients. Cinacalcet is approved for the treatment of secondary HPT, and in EVOLVE it was effective in preventing severe unremitting HPT. Treatment effects of cinacalcet on fracture rates were similar to effects on CV events in unadjusted ITT analysis, multivariable-adjusted ITT analysis, and censoring at cointerventions that reduce PTH; age was also an effect modifier (30). A recent review has examined the clinical and practical use of calcimimetics in dialysis patients (31). An economic evaluation of cinacalcet in the United States has been reported (32).

Conclusions

The EVOLVE trial provides multiple lessons in the planning, execution, analysis, and interpretation of RCTs in CKD/ESRD. These include deciding upon, defining, and analyzing outcomes; responding to lower-than-predicted event rates; taking into account nonadherence and cointerventions; subgroup effects; capture of rare diseases; biomarkers; and interpretation of benefit-to-harm ratio.

Disclosures

All authors have completed and submitted the International Committee of Medical Journal Editors form for disclosure of potential conflicts of interest. Please see Supplemental Material for completed forms, which appear in alphabetical order by author.