Weighting Composite Endpoints in Clinical Trials: Essential Evidence for the Heart Team

Abstract

Background

Coronary revascularization trials often use a composite endpoint of major adverse cardiac and cerebrovascular events (MACCE). The usual practice in analyzing data with a composite endpoint is to assign equal weights to each of the individual MACCE elements. Non-inferiority margins are used to offset effects of presumably less important components, but their magnitudes are subject to bias. This study describes the relative importance of MACCE elements from a patient perspective.

Methods

A discrete choice experiment was conducted. Survey respondents were presented with a scenario that would make them eligible for the SYNTAX 3-Vessel Disease cohort. Respondents chose among pairs of procedures that differed on the 3-year probability of MACCE, potential for increased longevity, and procedure/recovery time. Conjoint analysis derived relative weights for these attributes.

Results

In all, 224 respondents completed the survey. The attributes did not have equal weight. Risk of death was most important (relative weight 0.23), followed by stroke (.18), potential increased longevity and recovery time (each 0.17), MI (0.14) and risk of repeat revascularization (0.11). Applying these weights to the SYNTAX 3-year endpoints resulted in a persistent, but decreased margin of difference in MACCE favoring CABG compared to PCI. When labeled only as “Procedure A” and “B,” 87% of respondents chose CABG over PCI. When procedures were labeled as “Coronary Stent” and “Coronary Bypass Surgery,” only 73% chose CABG. Procedural preference varied with demographics, gender and familiarity with the procedures.

Conclusions

MACCE elements do not carry equal weight in a composite endpoint, from a patient perspective. Using a weighted composite endpoint increases the validity of statistical analyses and trial conclusions. Patients are subject to bias by labels when considering coronary revascularization.

Introduction

Studies comparing percutaneous coronary intervention (PCI) and coronary artery bypass grafting (CABG) for coronary revascularization have traditionally defined adverse outcomes as major adverse cardiac and cerebrovascular events (MACCE): death, stroke, non-fatal myocardial infarction and need for repeat revascularization. The use of a composite endpoint such as MACCE, in which an event for any component results in an endpoint event, can increase the efficiency and shorten the duration of a clinical trial [1]. A key practice in using a composite endpoint, however, is that the individual components have equal importance and therefore equal weight. If this practice misses important differences in the relative weights among MACCE components then the trial statistics, and possibly conclusions, may be misleading. Because trial sponsors and readers all have inherent bias, study conclusions based upon MACCE may be distorted by selective emphasis of one MACCE component or another when actual evidence for the relationships is lacking.

Patients choose among medical treatments based on their personal value systems and their providers’ recommendations. To date, no studies have been conducted to determine the relative weights and importance of the components of MACCE. One method to elicit patient preferences is conjoint analysis, a statistical technique using discrete choice experiments traditionally used in marketing studies. In this study, participants chose between procedures that differed with respect to the MACCE endpoints. The conjoint analysis estimates the relative weights of attributes; in this case, the MACCE events associated with coronary revascularization procedures. These weights are then applied to determine if non-equal weights produce different conclusions.

Patients and Methods

A discrete choice experiment was conducted in order to define the relevance and relative weights of four MACCE elements and two other considerations in coronary revascularization. These results were applied to the SYNTAX trial in order to illustrate the importance of considering individual element weights in a real-world setting. In addition, the experiment tested whether choices for coronary revascularization procedures are influenced by labels and prior experiences. The National Heart, Lung and Blood Institute (NHLBI) Cardiothoracic Surgical Trials Network sponsored this study, and the Duke University Medical Center Institutional Review Board approved the survey and research protocols (Pro00027452).

Survey respondents were members of an online panel recruited by Knowledge Networks Inc., and reflective of a location-based representative sample covering 97% of United States households [2]. Inclusion criteria for the study were age > 18 years and self-reported history at least one of the following: hypertension, hypercholesterolemia, coronary artery disease, congestive heart failure, cardiac arrhythmia, angina, previous myocardial infarction (MI), and peripheral vascular disease. Following introductory questions regarding respondents’ overall health state, respondents reported their use of cardiac-related medications such as antihypertensive and cholesterol-lowering drugs. Respondents were then asked to imagine, in lay terms, that they had a “serious heart condition” with symptoms of angina and a subsequent diagnosis of 3-vessel coronary disease, and that there was a risk of sudden death without undergoing a “procedure.” Then they indicated their familiarity with concepts such as procedure-related hospitalization, stroke, myocardial infarction and risk of revascularization. Descriptions of stroke and “heart attack” were provided to the respondents in the questions gauging their familiarity with these concepts.

The choice experiment presented respondents with 13 choices between of pairs of unlabeled procedures defined by 6 attributes. The attributes encompassed all of the clinical considerations in coronary revascularization: risk of death, stroke, myocardial infarction and repeat revascularization within 3 years; extent of procedure and recovery; and potential for increased longevity. The extent of procedure and recovery were described as duration of the procedure (hours), number of nights in the hospital, and number of weeks needed to recover at home. For the MACCE elements, attribute levels were approximated using 3-year results from the SYNTAX left main and/or 3-vessel disease cohort [3]. Table 1 lists the attributes and levels included in the conjoint task.

Table 1.

Attributes and levels of the conjoint task

	PCI	Intermediate Level	CABG
Extent of procedure (length of procedure, hospital stay, recovery time)	2 hours 2 nights 1 week	3 hours 3 nights 4 weeks	4 hours 4 nights 6 weeks
Risk of death*	6%	2%	3%
Risk of myocardial infarction*	7%		3%
Risk of stroke*	3%	1%	2%
Risk of revascularization*	20%	15%	10%
Expected change in life expectancy over 7 years	none	6 months longer	1 year longer

^*within 3 years

The last two choice tasks asked respondents to choose between PCI and CABG, first unlabeled then labeled, with the actual attribute levels for each procedure. Survey results were analyzed using Sawtooth Software, using hierarchical Bayes modeling, to generate the individual-level conjoint weights that best reproduce each respondent’s choices [4]. The results across questions were generated using IBM SPSS Statistical Software.

The relative weights of the attributes were then applied to the SYNTAX 3-year outcome data for the 3-vessel disease cohort. For the elements of the SYNTAX composite endpoint, the relative weight was normalized to the average weight of the 4 MACCE elements to create an adjustment factor. This factor was then multiplied by the measured difference between procedural outcomes to adjust that difference for the measured patient preference.

Results

Of 225 people screened for the survey, 224 (84 women and 140 men) met the eligibility criteria and completed the online survey. Table 2 shows the baseline demographic and self-reported medical history of the survey respondents. The mean age of survey respondents was 65 years (range 34–90 years). The median time to complete the survey was 22 minutes (range 5–5536 minutes) for 28 questions and 13 choices.

Table 2.

Baseline characteristics of survey respondents

Gender
Male	140 (62.5%)
Female	84 (37.5%)
Mean age (years)	65 (range 34 – 90)
Race
Caucasian	187 (83.5%)
African-American	13 (5.8%)
Other, Non-Hispanic	6 (2.7%)
Hispanic	9 (4.0%)
2+ Races, Non-Hispanic	9 (4.0%)
Self-Reported Medical History
Hypertension	169 (75.4%)
Hypercholesterolemia	174 (77.7%)
Coronary artery disease	103 (46%)
Congestive heart failure	45 (20%)
History of arrhythmia	74 (33.0%)
Angina	73 (32.6%)
History of myocardial infarction	103 (46%)
Peripheral vascular disease	13 (5.8%)
Prior Procedures
Stress test	200 (89.3%)
EKG	192 (85.7%)
Cardiac catheterization	134 (60.0%)
Prior PCI with stent placement	98 (43.8%)
Prior CABG	63 (28.1%)
Prior stent and CABG	25 (11.2%)
Prior valve replacement	11 (28.1%)

A majority of the survey respondents (136 of 224, 60.7%) had prior coronary revascularization. Of these, 98 (43.8%) reported previous PCI with stent placement; 63 (28.1%) had previously undergone CABG. Twenty-five (11.2%) of the survey respondents had undergone both PCI and CABG in the past.

Overall, the respondents were familiar with the concepts of hospitalization and recovery time, stroke, and myocardial infarction. While 136 (60.7%) had undergone a procedure over 2 hours in duration with a night in the hospital and one weeks’ recovery time, only 69 (30.8%) reported having experienced a procedure of 4 hours’ duration, requiring 4 nights in the hospital and 6 weeks’ recovery time. Respondents were more familiar with myocardial infarction than stroke, with 182 of the 224 respondents (81.2%) reported that either they or someone close to them had ever experienced a myocardial infarction, and 69 (51.8%) reporting the same for stroke.

The attributes were found to have unequal weights (Figure 1). Together, the MACCE elements accounted for 66% of the total weight. Risk of death was most important (relative weight 0.23; standard error 0.010), followed by stroke (0.18; SE 0.008), MI (0.14; SE 0.007) and risk of repeat revascularization (0.11; SE 0.004). The weights of the non-MACCE elements, potential increased longevity and recovery time, accounted for 34% of the total. The weights for these two attributes were equal, each 0.17 (SE 0.008 and 0.009, respectively). Respondents with greater aversion to death tended to be women, non-white, living alone, poorer and reporting problems with self care.

Figure 1.

Relative weights of attributes and total contribution of MACCE and non-MACCE elements.

Table 3 contrasts the effect of using equal weights for the attributes versus applying the empirical weights to the SYNTAX outcome data for the left main and 3-vessel disease cohort. The normalized relative weights were used to calculate an adjusted difference between the PCI and CABG cohorts. The equally weighted difference sum was 13.0% in favor of CABG, whereas the adjusted sum was 10.0% in favor of CABG.

Table 3.

Application of weighted attributes in SYNTAX cohort

	SYNTAX 3VD CABG 36 month	SYNTAX 3VD PCI 36 month	Difference	Relative Weight	Normalized Relative Weight	Adjusted Difference
Death	6.7%	8.6%	1.9%	0.23	1.38	2.6%
Stroke	3.4%	2.0%	−1.4%	0.18	1.08	−1.5%
MI	3.6%	7.1%	3.5%	0.14	0.84	2.9%
Revasc	10.7%	19.7%	9.0%	0.11	0.66	5.9%
MACCE*	24.4%	37.4%	13.0%			10.0%

^*Calculated

The last two choices involved procedures whose attributes reflect current information on PCI and CABG. The first of these mimicked the 11 previous choices, labeling them simply as alternatives A and B, while the second choice labeled the options as a “Coronary Stent Procedure” and “Coronary Bypass Procedure.” This labeling influenced respondents’ choices. When asked to choose between procedures labeled as “Procedure A” and “Procedure B,” 13% of respondents chose the option corresponding to PCI and 87% chose the option corresponding to CABG. In this abstract task, respondents more likely to choose CABG over PCI included those who valued longevity, reported less concern with hospital stays, were familiar with CABG and unfamiliar with PCI. In addition, they were more likely to have a lower income and rent versus own their home.

When the identical choices were later presented with the labels “Coronary Stent Procedure” and “Coronary Bypass Procedure,” the proportion of respondents that chose PCI over CABG increased from 13% to 27%. Those who switched from CABG to PCI tended to be younger, male, non-White and more likely to say they were depressed. Further, these respondents also were more likely to have reported a personal history of coronary artery disease and had undergone PCI but not CABG.

These previous analyses involved a forced choice, where respondents had to choose between CABG and PCI. When those who chose CABG were asked to assess the likelihood that they would actually proceed with the operation, 61% of respondents reported that they would be “very likely to accept” the procedure. Virtually the same percent, 63% of those who chose PCI indicated they would very likely comply with the recommendation to undergo the procedure. Overall, only 4.9% of the respondents stated that were “somewhat likely” or “very likely” to decline either procedure. Respondents who had previously undergone CABG or PCI, and were therefore familiar with revascularization, were more likely than others to follow through with a revascularization procedure. Other factors that generally increased the likelihood of compliance included placing a high value on longevity, reporting problems with self care, and reporting a diagnosis of coronary artery disease but not CHF.

Comment

Coronary revascularization is one of the most common procedures performed in the United States, with over 1 million performed each year [5]. Several multi-center, randomized trials, most recently ARTS II and SYNTAX, have demonstrated superior short- and long-term clinical outcomes for CABG over PCI in patients with multi-vessel coronary artery disease [3, 6]. Despite this, the proportion of patients undergoing revascularization by CABG has decreased significantly over the past 10 years [7]. A recent study analyzed the performance of PCI in patients with Class I indications for CABG, and showed that only 53% of these patients were recommended for CABG by their catheterization laboratory cardiologist [8].

There are several possible reasons for the decline in use of CABG. There may be a labeling issue similar to what we found. Additionally, negative patient and provider perceptions regarding the risks and invasiveness of surgery, as well as misperception of long-term benefits of PCI compared to CABG, likely influence decision-making. Patients are subject to provider bias. A recent study of hospitalized patients reported that the vast majority expressed a desire to have physicians present choices regarding their medical care [9]. Of these, nearly two-thirds reported that they would prefer to leave the decision-making to the physician. In other studies, provider specialty has been shown to influence provider preferences for therapy such as operative treatment for breast cancer and lumbar spine surgery [10, 11]. Given this, it is conceivable that patients are subject to bias by their treating physicians, very often cardiologists who perform both diagnostic and therapeutic interventions, when being presented with options for coronary revascularization.

It is also possible that physicians believe that patients desire PCI over CABG. In studies of physicians’ perceptions of patient preferences in screening for colorectal cancer and surgical management of esophageal cancer, the relative importance of attributes differed between physicians and patients [12, 13]. Of the 6 attributes in this study, the risk of death was most important. While it is likely that physicians would also rank death as most important, further study is warranted to determine whether physician perceptions of patient preferences for coronary revascularization are accurate.

In the absence of known relative weights for components of composite endpoints, two trends have become evident in trials comparing cardiac surgical procedures to their transcatheter alternative. First, non-inferiority has replaced superiority as the primary statistical test of the composite endpoint, as exemplified by the SYNTAX and EVEREST trials [14, 15]. A non-inferiority margin is pre-specified in part to supplement the composite endpoint for the unmeasured but presumed inherent preference for the percutaneous alternative. However, determining the magnitude of the non-inferiority margin in clinical trials is an inexact science at best [16, 17]. Second, it has become common practice to editorialize on the relative importance of the elements of the composite endpoint in order to re-interpret and redefine the overall study results. For SYNTAX, this occurred within the manuscript and in a simultaneously published editorial where the SYNTAX outcome favoring coronary bypass grafting was contested [15, 18].

If the composite outcome endpoint elements could incorporate all the important clinical elements and be accurately weighted, trial outcomes would be less subject to post-hoc reinterpretation. Accurate weighting, in turn, will necessitate an understanding of the relative importance of the composite elements to the interested parties. While the interested parties include study investigators, study sponsors, payers and the providers of the therapeutic alternatives, we respectfully suggest that the patient’s preferences are paramount and should form the foundation of a properly weighted composite endpoint. This perspective is completely consistent with recent calls to incorporate patient preferences into clinical guideline development [19] and to establish a Heart Team to determine recommendations for coronary revascularization [20].

This is the first study to explore patient preferences in coronary revascularization procedures. Together, the 6 attributes examined comprehensively describe considerations in coronary revascularization. The only exception is cost effectiveness, which is fairly neutral in comparing CABG and PCI for patients with complex disease [21].

The study indicates that the conventional wisdom considering death to be more important than stroke, myocardial infarction and repeat coronary revascularization is correct. Importantly, however, the study demonstrates that myocardial infarction is almost as important as stroke as an adverse outcome and that target vessel revascularization is not a trivial, unimportant event. When these results are applied to the SYNTAX composite outcome, it is clear that CABG remains superior to PCI, in terms of MACCE, for patients with 3-vessel coronary artery disease. These results are not vulnerable to interpretation with regard to the relative importance of stroke, myocardial infarction or target vessel revascularization because empirically determined relative importance is now incorporated into the MACCE value.

This study also confirms that patient perceptions of, and prior experiences with, coronary revascularization procedures influence their choices. In addition, patients may be subject to bias by labels when considering coronary revascularization. By simply labeling the procedures as “Coronary Stent Procedure” and “Coronary Bypass Procedure,” the proportion of respondents that chose PCI over CABG increased from 13% to 23%. Prior experiences with coronary revascularization influenced respondents’ choices in the survey, indicating that familiarity with PCI and CABG is positive among former patients.

This study has important limitations. The proportion of women in the study is comparable to the relative proportion of women versus men undergoing CABG, as well as those with coronary artery disease in general [22, 23]. However, the proportion of non-White respondents in our survey was slightly lower than those in the general population [24]. Since non-White respondents were more likely to change their final choice from PCI to CABG when labeled as such, the proportion of those choosing PCI over CABG may be slightly underestimated.

In exploring the SYNTAX outcomes as an example, we used the equally weighted individual MACCE components to compare to the weight-adjusted MACCE calculated from the weights determined by respondents’ choices. The small discrepancy between our calculated MACCE and MACCE difference, compared to the published SYNTAX results, is due to the fact that SYNTAX allocated a MACCE event only once even if more than one event occurred in an individual patient.

Finally, our conclusions can be criticized in that we did not attempt to directly include attributes such as procedural pain and procedural “invasiveness” that in many ways form the foundation for the employment of relatively large non-inferiority margins favoring transcatheter therapies. We chose to describe the extent of each procedure and the duration of hospitalization and recovery as an explicit attribute, reasoning that survey respondents would recognize when an open surgical procedure was being considered. The fact that the survey respondents valued longevity and the extent of the procedure equally supports the MACCE superiority of CABG, obviating the need for a “less invasive” statistical offset. The evidence presented that patients value longevity benefit with the same positive magnitude as the negative magnitude of procedural invasiveness suggests that a properly weighted MACCE result in favor of CABG should be definitive.

Conclusions

Assigning equal weights to the components of clinical trial composite endpoints may result in inaccurate statistical analysis and potentially alters study conclusions. In this study, when comparing procedures for coronary revascularization, patients at risk for coronary artery disease did not view the MACCE elements as having equal weight. Based on these results, future studies comparing PCI and CABG should consider using a weighted composite endpoint of death, stroke, MI and risk of revascularization for more accurate comparison. These results also indicate that existing clinical evidence be carefully and independently interpreted bearing the patient’s perspective in mind as recommendations for treatment selection are being formulated.