Agreement is Poor Amongst Current Criteria Used to Define Response to Cardiac Resynchronization Therapy

Abstract

Background

Numerous criteria believed to define a positive response to cardiac resynchronization therapy (CRT) have been utilized in the literature. No study has investigated agreement amongst these response criteria. We hypothesized that the agreement between the various response criteria would be poor.

Methods and Results

A literature search was conducted with the keywords “cardiac resynchronization” and “response”. The 50 publications with the most citations were reviewed. After excluding editorials and reviews, seventeen different primary response criteria were identified from 26 relevant articles. The agreement amongst fifteen of these seventeen response criteria was assessed in 426 patients from the PROSPECT study using Cohen's κ coefficient (two response criteria were not calculable from PROSPECT data). The overall response rate ranged from 32-91% for the fifteen response criteria. Ninety-nine percent of patients showed a positive response by at least one of the fifteen criteria while 94% were classified as a non-responder by at least one criterion. Kappa values were calculated for all 105 possible comparisons amongst the fifteen response criteria and classified into standard ranges: poor agreement (κ≤0.4), moderate agreement (0.4<κ<0.75) and strong agreement (κ≥0.75). Seventy-five percent of the comparisons showed poor agreement, 21% showed moderate agreement and only 4% showed strong agreement.

Conclusions

The 26 most cited publications on predicting response to CRT define response using 17 different criteria. Agreement between different methods to define response to CRT is poor 75% of the time and strong only 4% of the time, which severely limits the ability to generalize results over multiple studies.

Introduction

Predicting whether a patient will benefit, or “respond”, to cardiac resynchronization therapy (CRT) has been the focus of over 500 publications during the last 5 years. However, the definition of response to CRT varies widely between studies, and numerous criteria to define a positive response to CRT exist in the literature. “Echocardiographic” response is typically assessed by quantifying the change in left ventricular ejection fraction (LVEF) ^1-4 or left ventricular end-systolic volume (LVESV) ^{2, 5-10} three to six months after CRT implantation. “Clinical” response is assessed with the increase in the distance walked in six minutes (6MWD) ¹¹ or improvement in New York Heart Association (NYHA) functional class ^{2, 12-14} three to six months after CRT implantation. Some studies have defined response to CRT as a combination of several clinical measures ^15-17 or as a combination of both clinical and echocardiographic measures ¹⁸.

The heterogeneous approach to defining response to CRT is a potential barrier to progress in this field. No study has addressed this issue by investigating the agreement amongst the numerous published CRT response criteria. If these different response criteria show poor agreement, then the ability to generalize results from multiple studies is severely impaired, and a standard needs to be developed. We hypothesized that the agreement between the various published response criteria would be poor. We tested this hypothesis by identifying response criteria from a literature search and then assessing the statistical agreement amongst the different criteria in the 426 patients enrolled in the Predictors of Response to CRT (PROSPECT) study.

Methods

Literature Review

To identify commonly used criteria to define response to CRT, a literature search was conducted with the Web of Science “Science Citation Index Expanded” database ¹⁹ using the topics “cardiac resynchronization” and “response”. The 50 publications with the most citations were reviewed for relevance (Fig. 1). Four review articles and twenty publications that did not report individual response criteria were excluded.

Figure 1.

Flow chart showing the process by which response criteria were identified from the literature.

Seventeen different primary response criteria were identified from the 26 remaining publications ^{1-18, 20-27} (Table 1). Eight of these 17 response criteria were based on echocardiography, eight were based on clinical measures, and 1 criterion was based on a combination of both echocardiographic and clinical measures. Six of the 17 response criteria included either all-cause or heart failure mortality as a criterion to define a non-response, while the other 11 did not.

TABLE 1.

Seventeen different response criteria were identified from the 26 relevant publications

	Response Criterion
Echocardiographic	1. ↑LVEF ≥5% (absolute)1, 2
	2. ↑LVEF ≥15%3, 4
	3. ↓LVESV ≥10% and did not die of progressive HF within 6m20, 27
	4. ↓LVESV >15%2, 5-10
	5. LVESV <115% of baseline26
	6. ↓LVESVI >15%25
	7. ↓LVEDV >15%2
	8. ↑Stroke Volume ≥15%4, 21, 22
Clinical	9. ↓NYHA ≥12, 12-14
	10. ↓NYHA ≥1 and did not die of progressive HF within 6m23
	11. ↓NYHA ≥1 and ↑6MWD ≥25%15
	12. ↓NYHA ≥1 and ↑6MWD ≥25% and did not die of progressive HF within 6m16, 17
	13. ↑6MWD >10%, no heart transplant, did not die of progressive HF within 6m11
	14. (↓NYHA ≥1 or ↑VO2max >10% or ↑6MWD >10%) and alive, no hospitalization for decompensated HF24
	15. 2 of 3: 5
	↓NYHA ≥1
	↑6MWD ≥50m
	↓QOL ≥15
	16. Clinical composite score10
Combined	17. (↑LVEF ≥5% (absolute) or ↑6MWD ≥30m) and (↓NYHA ≥1 or ↓QOL ≥10)18

Note: If the authors did not specify whether death was considered a non-response, then it was assumed that deaths were excluded.

LVEF indicates left ventricular ejection fraction; LVESV, left ventricular end-systolic volume; HF, heart failure; LVESVI, left ventricular end-systolic volume indexed by body surface area; LVEDV, left ventricular end-diastolic volume; NYHA, New York Heart Association functional class; 6MWD, six-minute walk distance; VO2max, oxygen consumption at peak exercise; and QOL, quality of life score.

Patient Population

Agreement between response criteria was assessed using the baseline and 6-month follow-up visits from the 426 patients in the PROSPECT study ¹⁰. Briefly, PROSPECT was a prospective, multi-center study that was designed to test the ability of 12 different echocardiographic dyssynchrony parameters to predict response to CRT. Four hundred and fifty-seven patients with standard CRT indications (NYHA class III/IV heart failure, LVEF ≤35%, QRS ≥130ms, stable medical regimen) were enrolled in PROSPECT at 53 centers world-wide. After excluding 31 patients who exited the study early and did not receive an implant, 426 patients remained and were followed for 6 months after CRT implantation.

Statistics and Quantification of Agreement

Cohen's κ coefficient was used to assess agreement between the different response criteria. Kappa is an accepted statistical coefficient that is used to assess agreement between methodologies^{28, 29}. Kappa ranges from −1 (perfect disagreement) to +1 (perfect agreement), and a kappa of 0 indicates that the amount of agreement was exactly that expected by chance²⁸. Kappa ≥0.75 was defined as strong agreement, 0.4 < kappa <0.75 was defined as moderate agreement and kappa ≤0.4 was defined as poor agreement (Table 2)²⁹.

TABLE 2.

Kappa values utilized to define “strong”, “moderate” and “poor” agreement²⁹.

Level of Agreement	Kappa
Strong (clinically acceptable)	κ ≥ 0.75
Moderate (questionable value)	0.4 < κ < 0.75
Poor (not clinically acceptable)	κ ≤ 0.40

Kappa was calculated with the following equation:

$$\kappa =\frac{\text{observed agreement}(\%)-\text{expected agreement}(\%)}{100\%-\text{expected agreement}(\%)}$$ (1)

Expected agreement is the percentage of cases where two response criteria would agree based on chance alone and can be calculated as:

$$\begin{array}{l}\text{expected agreement}(\%)={\text{responders}}_{\text{criterion1}}•{\text{responders}}_{\text{criterion2}}\\ +{\text{non}-\text{responders}}_{\text{criterion1}}•{\text{non}-\text{responders}}_{\text{criterion2}}\end{array}$$ (2)

An example calculation of kappa is shown in Table 3 and presented in the results section.

TABLE 3.

An example comparing response criterion #3 (↓LVESV ≥ 10%, no HF death) to response criterion #13 (↑6MWD ≥10%, no HF death, no transplant) shows that kappa = 0.02 based on equations 1 and 2. This kappa value of 0.02 demonstrates poor agreement between the two criteria despite the fact that each criteria had nearly identical response rates of approximately 62%.

		Criterion #2		Sum
		Responders	Non-responders
Criterion #1	Responders	101	61	162
	Non-responders	60	39	99
Sum		161	100	261

Abbreviations as in Table 1.

Two of the 17 response criteria^{24, 25} could not be calculated from PROSPECT data due to the fact that 1) oxygen consumption at peak exercise was not measured in PROSPECT and 2) height and weight at 6 months were not measured in PROSPECT, which precludes the ability to calculate LVESV index. Agreement amongst the 15 remaining response criteria was assessed by calculating a kappa value for all possible pairs of criteria. Thus, 105 kappa values were calculated. Group kappa values were then quantified (mean, median and range) to summarize the agreement amongst criteria from four different groups: 1) all 105 comparisons, 2) comparisons between two echocardiographic criteria 3) comparisons between two clinical criteria and 4) comparisons between an echocardiographic criterion and a clinical criterion. Mean group kappa values were compared with a permutation test, and a Bonferroni correction was applied to account for multiple comparisons. To justify that the sample size to estimate kappa values was large enough, a bootstrap resampling procedure was utilized with a Kolmogorov-Smirnov test to assess the normality of the resulting distribution. A p value <0.05 was defined as statistically significant.

Subgroup Analysis Excluding Response Criteria Quantified Acutely and at 3 Months

One response criterion (↑Stroke Volume ≥15%^{4, 21, 22}) was used in the literature as an acute response measure that was quantified within 2 days of CRT implantation. In addition, two response criteria from the literature were only utilized in studies which assessed response at 3 months (criteria numbers 15 and 17 in Table 1). Acute (<2days) and 3m data for calculating all 15 response measures were not collected in PROSPECT, so all criteria were assessed with data from the 6m visit. To ensure that our results were not confounded by calculating these acute and 3m measures from 6m follow-up data, we performed a subgroup analysis in which we excluded them and re-calculated the group kappa values.

Results

Response Rates

The percentage of patients defined as having a positive “response” to CRT ranged from 32% to 91% for the 15 response criteria (Table 4). All 15 criteria could be calculated in 250 of the 426 patients in PROSPECT. Of these 250 patients, 99% showed a positive response by at least one of the fifteen criteria while 94% were classified as a non-responder by at least one criterion. Similarly, 95% of patients showed a positive response by at least two of the fifteen criteria while 87% were classified as a non-responder by at least two criteria.

TABLE 4.

Response rates for the different criteria were highly varied

	Response Criterion	Response Rate	Number Evaluable (% of total)
Echocardiographic	↑LVEF > 5 units	51%	286 (67%)
	↑LVEF>15% (relative)	54%	286 (67%)
	↓LVESV ≥ 10%, no HF death	62%	291 (68%)
	↓LVESV > 15%	56%	286 (67%)
	LVESV < 115% of baseline	91%	286 (67%)
	↓LVEDV > 15%	49%	286 (67%)
	↑Stroke Volume ≥ 15%	34%	286 (67%)
Clinical	↓NYHA ≥1	71%	385 (90%)
	↓NYHA ≥ 1, no HF death	70%	390 (92%)
	↓NYHA ≥1 and ↑6MWD ≥25%	33%	348 (82%)
	↓NYHA ≥1 and ↑6MWD ≥25%, no HF death	32%	353 (83%)
	↑6MWD ≥10%, no HF death, no transplant	61%	353 (83%)
	2/3 of: ↓NYHA ≥1, ↑6MWD ≥50m, ↓QOL ≥15	63%	339 (80%)
	Clinical composite score improved	69%	426 (100%)
Combined	↑LVEF >5 units or ↑6MWD ≥50m and ↓NYHA ≥1 or ↓QOL ≥10	71%	250 (59%)

Abbreviations as in Table 1.

Example Calculation of Kappa

An example of calculating kappa between response criterion #3 (↓LVESV ≥ 10%, no HF death) and response criterion #13 (↑6MWD ≥10%, no HF death, no transplant) is given in Table 3. Response criterion #3 identifies 62% of patients who receive CRT as responders (and, thus, 38% of the patients as non-responders), and criterion #13 also identifies 62% of patients as responders. The “expected agreement” due to chance alone is therefore 0.62 • 0.62 + 0.38 • 0.38 = 53%. The “observed agreement” is 0.39 + 0.15 = 54%. Equation 1 then shows that kappa = 0.02 for Table 3, which suggests poor agreement after accounting for the level of agreement expected due to chance.

Agreement Amongst the 15 Response Criteria

The fifteen response criteria showed poor agreement as a group (Figs. 2 and 3, mean κ = 0.22 ± 0.24, median = 0.14, range = −0.2 to 0.97). Seventy-nine of the 105 kappa values (75%) were classified as “poor” agreement while 22 kappa values (21%) were classified as “moderate” agreement (Fig. 3). Only 4 pairs of response criteria out of 105 total pairs (4%) showed “strong” agreement, and three of these 4 pairs were comparisons between a response criterion which excluded mortality and the same exact criterion which defined death as a non-response. The seven echocardiographic response criteria also showed poor agreement amongst each other (Figure 4, mean κ = 0.35 ± 0.28, median = 0.29, range = −0.2 to 0.88). The seven clinical response criteria showed moderate agreement (Figure 4, mean κ = 0.44 ± 0.23, median = 0.43, range = 0.14 to 0.97). Agreement between echocardiographic and clinical criteria was poor (Figure 4, mean κ = 0.05 ± 0.05, median = 0.04, range = −0.03 to 0.17), with all 49 kappa values showing poor agreement. The agreement amongst the echocardiographic parameters was not significantly different from the agreement amongst the clinical parameters (uncorrected p = 0.35). The response criterion based on a combination of both echocardiographic and clinical measures showed significantly better agreement (p=0.003) with clinical response criteria (0.44 ± 0.10) than with echocardiographic response criteria (0.21 ± 0.12). Bootstrap resampling of the kappa statistic comparing clinical composite response and a 15% reduction in LVESV justified that the sample size was large enough to estimate kappa (Kolmogorov-Smirnov test for normality p>0.15).

Figure 2.

Agreement amongst the 15 response criteria was poor. The Kappa axis is color-coded according to the following ranges: green = strong agreement (kappa ≥0.75), yellow = moderate agreement (0.4 < kappa <0.75), red = poor agreement (kappa ≤0.4). The worst agreement was between echocardiographic (Echo) and clinical (Clin) parameters. *p<0.001 vs “Echo vs Echo” and “Clin vs Clin”.

Figure 3.

Agreement amongst the 15 response criteria was classified as poor for 75% of the 105 possible comparisons. Kappa values are color-coded according to the following ranges: green = strong agreement (kappa ≥0.75), yellow = moderate agreement (0.4 < kappa <0.75), red = poor agreement (kappa ≤0.4). Echo indicates echocardiographic response criteria; and Clin, clinical response criteria.

Figure 4.

Agreement amongst the response criteria was poor 75% of the time and strong only 4% of the time. Kappa values are color-coded according to the following ranges: green = strong agreement (kappa ≥0.75), yellow = moderate agreement (0.4 < kappa <0.75), red = poor agreement (kappa ≤0.4). Abbreviations as in Table 1.

Subgroup Analysis Excluding Response Criteria Quantified Acutely and at 3 Months

Exclusion of the acute and 3-month response criteria did not significantly affect the results. After excluding one acute (criterion #8 in Table 1) and two 3-month (criteria #15 and 17 in Table 1) response measures, the agreement amongst the 12 remaining response criteria was poor as a group (mean κ = 0.22 ± 0.26, median = 0.14, range = −0.03 to 0.97). Agreement amongst the six remaining echocardiographic response criteria was moderate (mean κ = 0.44 ± 0.24, median = 0.47, range = 0.16 to 0.88). Agreement amongst the six remaining clinical response criteria was also moderate (mean κ = 0.42 ± 0.27, median = 0.32, range = 0.14 to 0.97). Finally, agreement between echocardiographic and clinical criteria remained poor (mean κ = 0.05 ± 0.05, median = 0.04, range = −0.03 to 0.17).

Discussion

The major findings of this study are: 1) the 26 most cited publications on predicting response to CRT utilize 17 different primary response criteria and the level of agreement not due to chance amongst 15 of these response criteria was poor on average in the 426 patients enrolled in the PROSPECT study; 2) agreement between echocardiographic and clinical response criteria was poor and nearly equal to the level of agreement expected by chance; 3) the percentage of patients defined as having a positive “response” to CRT ranged from 32% to 91% for the fifteen response criteria; 4) 99% of patients were classified as a “responder” by at least one of the fifteen criteria while 94% were classified as a non-responder by at least one criterion.

Comparison to the Literature

To our knowledge, no study has quantified agreement amongst response criteria with a kappa coefficient. However, a recent study by Bleeker et al aimed to quantify the agreement between echocardiographic and clinical measures of response to CRT ². The authors compared a decline in NYHA class (clinical response) to a 15% decrease in LVESV (echocardiographic response) in 144 consecutive patients undergoing CRT. The authors concluded that “the agreement between [clinical response and echocardiographic response] was good” based on the observed agreement of 76%. However, their data show that clinical and echocardiographic response would be expected to agree 52% of the time based on chance alone. The study did not calculate kappa values to account for this expected level of agreement due to chance. We estimated kappa to be 0.50 from their data, which is higher than the value of 0.17 observed in our study. However, the main conclusion which should be drawn from both studies is similar: the agreement between echocardiographic and clinical criteria for defining a positive response to CRT is only slightly better than that expected by chance alone.

In the MIRACLE trial, correlation between the change in left ventricular end-diastolic volume and change in NYHA class after 6 months of CRT was weak (r = 0.13) ³⁰. In addition, the correlation between the change in 6MWD and change in LVEF was weak (r = 0.15)³⁰. These data are consistent with our results which show poor agreement between clinical and echocardiographic response criteria.

Previous studies have reported different rates of response to CRT when using different definitions of response within the same population. For example, the PROSPECT study reported that 56% of patients were echocardiographic responders (defined by a reduction in LVESV of at least 15%) while 69% of patients were clinical responders (defined by an improvement in the clinical composite score)¹⁰. Thus, one would expect these measures to show poor agreement because of the different response rates. However, the actual response rate does not tell the entire story: table 3 shows two different response criteria with identical response rates of 62%, and, despite the identical response rates, the criteria show very poor agreement (kappa = 0.02). Thus, assessment of agreement with the kappa statistic provides valuable information in addition to the overall response rate of the population.

Other Inconsistencies in Defining Response to CRT

Length of Follow-up

Another area of inconsistency in defining response to CRT is the length of the follow-up period after which a patient is deemed either a responder or a non-responder. Some studies focused on acute (1-2 days) response^{1, 4, 21, 22}, while most focused on 3m^{5-9, 18} or 6m^{2-4, 10-17, 20, 23-27} response. CRT has been shown to have persistent, increasing benefits with a longer mean follow-up period of 29.5 months ³¹. We defined response at 6m because this was the pre-specified follow-up period for the PROSPECT study. We also performed a subgroup analysis after excluding criteria which were assessed in the literature at acute and 3m follow-up only, and this did not change our results. Future studies will be needed to address agreement amongst the different lengths of follow-up.

Mortality

Another area of inconsistency in defining response to CRT is whether death should be considered a non-response to CRT. There are at least 3 different methods which authors have used to incorporate death into their response criteria: 1) death from worsening heart failure is included in the non-responder group^{11, 16, 17, 20, 23, 27}, 2) death from any cause is included in the non-responder group²⁴ and 3) deaths are excluded from analysis^{3, 5, 6, 8, 9, 26}. Moreover, numerous publications fail to specify how death was incorporated into response criteria despite enrolling consecutive patients and following them for a 3-6 month period^{2, 12-14, 18}. While inclusion of all-cause mortality as a criterion for non-response may not be appropriate, a patient who dies of progressive heart failure should, objectively, be classified as a non-responder. Regardless, there is no consistent method for incorporating mortality into the definition of response to CRT, and this needs to be standardized.

A Consensus Definition of “Response to CRT”

Since heart failure is a debilitating, life-threatening disease, an effective heart failure therapy should treat both symptoms and quality and duration of life ³². Thus, measures of “response” to cardiac resynchronization therapy should either directly measure outcomes or have a surrogate relationship with benefits in heart failure symptoms, quality of life and duration of life. The clinical composite score³³ is a measure of response that accounts for all of these factors and, therefore, may be the best overall choice for defining response in future CRT trials.

Study Limitations

Our results are limited by the fact that we utilized data from a single study (PROSPECT). However, PROSPECT was a multi-center study enrolling 457 wide QRS patients from 53 different centers across Europe, Hong Kong and the United States. We would expect similar results from other large, multi-center databases.

Our results show that many different methods to define a positive “response to CRT” are being utilized in the literature and show poor agreement amongst each other. This begs the question: which method should we utilize in the future to determine whether a patient benefited from CRT? This study did not attempt to address this question, and future studies will need to explore this important issue.

The definition of clinically acceptable agreement based on the kappa coefficient is not standardized. Fleiss proposed a threshold of ≥0.75 to define strong evidence for agreement that is not due to chance ²⁹, which is what we utilized. However, this threshold is somewhat arbitrary. Landis and Koch proposed that any value of kappa above 0.60 suggests substantial agreement, and kappa >0.80 implies “almost perfect” agreement ³⁴. However, using a different threshold such as 0.6 to define strong agreement would not significantly alter our results: four out of the 105 kappa statistics that we calculated were greater than 0.8 and only eight were ≥0.6. Thus, regardless of the threshold utilized, we observed mostly poor agreement amongst the 15 different response criteria.

Conclusions

The 26 most cited publications on predicting response to cardiac resynchronization therapy define response using 17 different criteria. Agreement between these different published methods to define response to CRT is poor 75% of the time and strong only 4% of the time. This inconsistency in the definition of response to CRT severely limits the ability to generalize results over multiple studies and hinders progress in the field.

Clinical Summary.

A literature search revealed that the 26 most cited publications on predicting response to cardiac resynchronization therapy (CRT) defined response using seventeen different criteria. No study has investigated agreement amongst these various response criteria, and we hypothesized that this agreement would be poor. The agreement amongst fifteen of the seventeen response criteria was assessed in 426 patients from the PROSPECT study using Cohen's κ coefficient (two of the seventeen response criteria were not calculable from PROSPECT data). Response rates for the entire population were highly varied and ranged from 32-91% for the fifteen criteria. Ninety-nine percent of patients showed a positive response by at least one of the fifteen criteria while 94% were classified as a non-responder by at least one criterion. Kappa values were calculated for all 105 possible comparisons amongst the fifteen response criteria and classified into standard ranges: poor agreement (κ≤0.4), moderate agreement (0.4<κ<0.75) and strong agreement (κ≥0.75). Seventy-five percent of the comparisons showed poor agreement, 21% showed moderate agreement and only 4% showed strong agreement. Thus, agreement between different methods to define response to CRT is poor 75% of the time and strong only 4% of the time, which severely impairs the ability to generalize results over multiple studies. This lack of standardization hinders progress in CRT research and needs to be resolved.

Abbreviations

CRT

cardiac resynchronization therapy

LVEF

left ventricular ejection fraction

LVESV

left ventricular end-systolic volume

6MWD

six minute walk distance

NYHA

New York Heart Association

PROSPECT

Predictors of Response to Cardiac Resynchronization Therapy