The contribution of observational studies to the knowledge of drug effectiveness in heart failure

Abstract

Aims

Randomized controlled trials (RCTs) are the golden standard for the assessment of drug efficacy. Little is known about the add-on value of observational studies in heart failure (HF). We aimed to assess the contribution of observational studies to actual knowledge regarding the effectiveness of angiotensin-converting enzyme inhibitors (ACEI), and β-blockers (BB) in HF.

Methods

Observational studies that assessed the effectiveness of ACEI and BB in HF were identified by searching Medline, Embase, Cochrane Database (1990–2005) and the bibliographies of published articles. Cohort, case–control and time-series analysis studies were considered for inclusion. Studies with <100 patients and those who did not perform a multivariate analysis were excluded.

Results

A total of 23 cohort studies met the inclusion criteria. Studies of ACEI and BB showed a decrease in mortality with drug use in elderly patients with a broad range of ejection fraction (EF), and in those with depressed EF. Additionally, they showed a decrease in mortality in patients with renal insufficiency. The effect of ACEI and BB in HF with preserved EF was not clear, although last evidence suggests a potential benefit. Low-dose ACEI and BB may have beneficial effects. Target doses of ACEI seemed superior to low doses, but there was no clear dose–response relationship.

Conclusions

Observational studies in HF validate the effectiveness of ACEI and BB in populations underrepresented or excluded from RCTs. Observational studies of drug effectiveness provide relevant additional information for clinical practice.

Introduction

At present, the randomized controlled trial (RCT) design is the gold standard for providing evidence on drug efficacy [1, 2]. However, for ethical and scientific reasons, RCTs have strict patient inclusion criteria, which in turn limit the extrapolation of their findings to daily practice populations (external validation) [3]. In heart failure (HF), most RCTs have excluded the elderly, patients with severe comorbidities, and those with preserved ejection fraction (EF). Therefore, only a minority (13%) of patients treated in clinical practice would fulfil the entry criteria of at least one landmark trial [4]. The dissimilarity between clinical practice and trial setting has put forward as a possible explanation for the low use of evidence-based therapies [5].

The observational study design has emerged as a research tool to complement information provided by RCTs [3]. Observational studies allow the assessment of the drug benefit in the real-life healthcare setting (effectiveness) [6]. Further, they may be valuable in assessing drug effectiveness in subgroups not studied in RCTs, or assessing the long-term beneficial effects of drugs already proven effective in short-term RCTs. Also, they may generate hypotheses that can later be tested in a RCT. Though largely inclusive, observational studies are more exposed to biases, which can partly be addressed through rigorous study design or statistical analysis [7–10] (Table 1). At present, a number of observational studies have been conducted in HF, but no study has yet summarized their added value on the knowledge of drug effectiveness.

Table 1.

Comparison of cohort studies and randomized controlled trials

Item	Cohort studies	Randomized controlled trials
Populations studied	Diverse populations of patients who are treated in a range of settings	Highly selected populations who are treated at selected sites
Allocation to the intervention	Based on decisions made by providers or patients	Based on chance (randomization)
Outcomes	Can be defined after the intervention and can include rare events	Primary outcomes are determined before patients are entered into study and are based on predicted benefits and risks
Follow-up	Prospective or retrospective studies; provide an opportunity for long follow-up	Prospective studies; often have short follow-up because of costs and pressure to produce timely evidence
Analysis	Sophisticated statistical analysis	Analysis is straightforward
Threaths to validity	Internal validation (risk of confounding)	External validation (exclusions)

Adapted after Rochon et al. [10].

In this study, we aimed to assess the contribution of observational studies to actual knowledge regarding the effectiveness of angiotensin-converting enzyme inhibitors (ACEI) and β-blockers (BB) in patients with HF.

Methods

Study selection

Observational studies were selected that assessed the effectiveness of ACEIs and BBs. These classes of medication are the mainstay of current treatment in HF [11]. Cohort, case–control and time-series analysis studies were included, but not posthoc analysis of RCTs, as patient population in such studies complies with RCT selection criteria. Effectiveness was assessed by the impact on overall mortality and readmission. A qualified librarian performed the computerized search. First, we aimed to identify systematic reviews or meta-analyses of observational studies by searching Medline, Embase and Cochrane Database (1990–2005) with the keywords of each drug, e.g. ACE-inhibitors, beta-blockers (beta adrenergic antagonists), and systematic review, meta-analysis, observational, cohort, nonrandomized, retrospective study and heart failure. If a systematic review or meta-analysis was not found, observational studies were identified by using the same keywords, excluding systematic review and meta-analysis. After the first computerized search with the above-mentioned keywords, a secondary search was performed through the link ‘related articles’ from Medline. The references of identified articles were also hand-searched for additional studies. Studies with <100 patients and those which did not perform a multivariate analysis to adjust for potential confounders were excluded.

A single reviewer used the above-mentioned strategy to identify potentially relevant articles by examining titles and abstracts. Two reviewers independently assessed suitability for inclusion by applying the criteria for type of studies, drug exposure and outcome measure. Differences were resolved by consensus.

Methodological quality

An adapted version of the Newcastle-Ottawa Scale [12] was used to assess the quality of identified studies. This scale assesses three broad areas: selection, comparability and outcome. Selection refers mostly to the way drug exposure was defined in the study. Comparability refers to methods applied either in study design or statistical analysis to match the exposed and non-exposed individuals. Outcome refers to the way outcome was assessed in the study, as well as duration and adequacy of follow-up. To complete the information provided by this scale, some additional data were extracted, meant to assess the way the most important types of bias were addressed within the studies. Finally, the following information was collected:

1. The ascertainment of drug exposure (selection of exposed/non-exposed cohorts)

2. Duration of follow-up, and ascertainment of drug exposure during follow-up

3. Comparability of the exposed/non-exposed cohorts

4. Methods selected to address confounding factors (in study design or analysis)

5. Assessment of outcome during follow-up

6. Report of measure of effect size (hazard or odds ratio, confidence intervals)

7. Checking for effect modification (interactions).

Results

Study inclusion

We found no review or meta-analysis of observational studies. The Medline search for observational studies identified 544 articles, 20 relevant to our study. Embase database was searched (582 articles) and one additional study was identified. Four more studies were identified through the references of identified articles. The Cochrane database did not provide any additional study.

In total, 25 observational studies (cohort studies) were identified. Two studies were excluded from the analysis because they included <100 patients [13, 14]. In total, 23 studies remained eligible for inclusion. Most studies addressed specifically the effectiveness of ACEI (n = 16), followed by studies of BB (n = 4). Three cohort studies assessed the effectiveness of both ACEI and BB.

Study characteristics

Observational studies of ACEI

Table 2 shows the main characteristics of ACEI studies [15–33]. All studies but one [17] were retrospective and were conducted in clinical registries, quality performance measurement databases, or institutional clinical databases. Studies included elderly patients, mostly >65 years old. All studies but one [17] included inpatients and assessed the association between ACEI prescription at discharge and mortality during follow-up. Consecutive patients were included in the studies. Assessment of drug prescription at discharge was made through clinical records. Duration of follow-up varied from 30 days to 4 years, but most studies had at least 1 year's follow-up. All studies with one exception [32] considered medication constant during follow-up. This study presented the results with medication both as a time-dependent variable and constant during follow-up, with similar results. Outcomes (overall mortality or readmission) were mostly assessed through specific database record linkage. All studies performed a multivariate analysis (logistic or Cox regression analysis), but there were slight differences with regard to the variables considered for adjustment. For example, whereas most studies adjusted for a range of clinical variables, and concomitant medication, some others did not adjust for the latter class [24]. Only a limited number of studies used propensity score analysis to reduce residual confounding by indication [19, 20, 28, 32]. All studies but one [15] presented the adjusted outcome (hazard or odds ratio), with confidence intervals (CI). However, only two studies reported the use of interaction tests [28, 29].

Table 2.

Effectiveness of angiotensin-converting enzyme inibitors (ACEI) in observational studies of heart failure

Study	Age, years (mean)	No. of patients	Follow-up	Outcome	Relative risk ACEI/No ACEI (odds ratio or hazard ratio)
I. Effectiveness of ACEI in elderly patients (depressed and preserved LVEF)
Havranek et al. (1998) [15]	≥65	328	1 year	Mortality	0.64 (univariate OR) Multivariate analysis – SN
Sueta et al. (2000) [16]	≥65	1 195	30 days	Mortality or readmission	0.70 (0.53–0.93)
Pulignano et al. (2002) [17]	(63) all >70	3 327 1 033	1 year	Mortality overall, and in elderly (>70)	0.73 (0.59–0.89)* 0.75 (0.55–1.01)
Ahmed et al. (2003) [18]	≥65 (79)	1 090	3 years	Mortality	0.81 (0.69–0.96)
Johnson et al. (2003) [19]	≥65 (79)	10 638	1 year	Mortality	0.75†
II. Effectiveness of ACEI in elderly patients with depressed LVEF
Shlipak et al. (2001) [20]	≥65	20 902	1 year	Mortality	0.80 (0.73–0.87)‡
Masoudi et al. (2004) [21]	≥65 (78)	17 456	1 year	Mortality	0.86 (0.82–0.90)
III. Effectiveness of ACEI in patients with preserved LVEF
Philibin et al. (1997) [22] (EF ≥ 40%)	(74)	350	6 months	Mortality Overall readmission	0.63–NS NS
Philibin et al. (2000) [23] (EF ≥ 40%)	(75)	238 312	6 months	Mortality	EF (0.40–0.49); 0.37 (0.17–0.81) EF = 50; 0.61 (0.30–1.25)
				Overall readmission	NS
Dauterman et al. (2001) [24] (EF ≥ 45%)	≥65	430	1 year	Mortality HF readmission	0.87 (0.60–1.27)* NS
Ahmed et al. (2002) [25] (EF ≥ 40%)	≥65	200	4 years	Mortality	0.96 (0.65–1.42)
Sueta et al. (2003) [26] (EF ≥ 50%)	≥65	760	1 year	Mortality	0.60 (0.42–0.84)*
IV. Effectiveness of ACEI per different levels of renal function
Philibin et al. (1999) [27]	≥65	1 076	6 months	Mortality Mortality or readmission	Cr ≥ 2 mg dl−1; 0.90 (0.43–1.82) Cr < 2 mg dl−1; 0.75 (0.50–1.13) Cr < 2 mg dl−1—SN
Frances et al. (2000) [28]	≥65	20 902	1 year	Mortality	Cr > 3 mg dl−1; 0.63 (0.48–0.84) Cr ≥ 3 mg dl−1; 0.84 (0.77–0.92)
McAlister et al. (2004) [29]	Median 69	754	Median 2.5 years	Mortality	GFR < 60 ml m−1; 0.46 (0.26–0.82) GFR ≤ 60 ml m−1; 0.28 (0.23–0.70)
V. Effectiveness of ACEI according to dose used
Chen et al. (2001) [30]	≥65	554	1 year	Mortality Overall readmission	Target/low; 0.49 (0.29–0.84) Subtarget/low; 0.65 (0.44–0.96) NS
Luthi et al. (2002) [31]	≥65 (77)	621	1 year	Mortality	Target/no ACEI; 0.61 (0.38–0.98)* Target/subtarget; 0.77 (0.51–1.16)
Rochon et al. (2004) [32]	≥65 (78)	16 539	1 year	Mortality Mortality or readmission	Target/low; 0.76 (0.68–0.85) Subtarget/low—similar Low/no ACEI; 0.89 (0.82–0.98)*Target/low dose—SN
Luthi et al. (2004) [33]	(75)	1 634	30 days	Readmission	Target/no ACEI—NS Target/less target—NS

^*Mirrored effect of ACEI, as OR/HR of no ACEI vs. ACEI was reported in the review; SN, significant; NS, nonsignificant; Cr, creatinine; GFR, glomerular filtration rate.

^†Relative risk reduction of ACEI vs. no ACEI or β-blocker (BB) was calculated by dividing adjusted mortality rates in each group.

^‡Relative risk reduction of ACEI vs. no ACEI or BB.

With regard to the outcome, studies can be split into five main categories (Table 2). First, studies that assessed the effectiveness of ACEI in elderly patients with HF, irrespective of EF. Second, studies that assessed specifically the effect of ACEI in patients with HF and depressed EF. In these two categories, all studies reported a relative reduction (RR) in mortality with ACEI use of about 20–25%. Third, studies that assessed the effectiveness of ACEI in elderly patients with HF and preserved EF, in which only two out of five studies showed a beneficial effect. Philibin et al. [22] found a significant effect on mortality with ACEI in patients with an EF 40–50%, but not in those with EF >50%. In contrast, Sueta et al. [26] reported a significant effect on mortality in patients with EF >50%. The latter study included the largest sample size and followed the patients for 1 year, which may justify the positive effects in comparison with the other studies. Fourth, studies that assessed the effectiveness of ACEI for different levels of renal function. Frances et al. and McAlister et al. [28, 29] showed a significant effect of ACEI on mortality in patients with renal insufficiency, whereas Philibin et al. [27] found no such effect. The nonsignificant effect in the latter study may be explained by its short follow-up (6 months), as this study also did not find an effect of ACEI on mortality in patients with normal renal function. The different definitions of renal insufficiency, e.g. a cut-off of creatinine 2 or 3 mg dl⁻¹, or by glomerular filtration rate may also explain the differences. Frances et al. reported also a significant interaction between ACEI and aspirin, which was not confirmed by McAlister et al. Last, studies that assessed the effectiveness of ACEI according to dose use. The interpretation of findings in this class is difficult due to selection of different reference classes. Chen and Rochon et al. [30, 32] selected low-dose ACEI as a reference class. Both studies found a higher benefit of target vs. low-dose, but whereas the former study found also a higher benefit of subtarget vs. low dose, the latter found no difference. Rochon et al. reported also a benefit of low dose vs. no ACEI. Luthi et al. [31] selected target dose as a reference category. They found a higher benefit of target dose vs. no ACEI, but no difference between subtarget and target dose. In conclusion, it appears that target dose is superior to low dose or no medication, but there is no clear dose–response relationship.

Observational studies of BB

Table 3 shows the main characteristics of BB observational studies [19, 20, 29, 34–37]. As opposed to ACEI studies, three out of seven cohort studies were prospective [35–37], and most studies included incident cases of HF. All studies except one [35] included only inpatients and assessed the association between BB prescription at discharge and mortality during follow-up. Assessment of drug exposure at discharge was made through clinical records, and in three studies [19, 34, 35] ascertainment of BB status during follow-up was time dependent (pharmacy linkage or annual in-study medication inventory). In these three studies the propensity score method was used for additional adjustment, and in two of them sensitivity analysis was also performed [34, 35]. Outcomes were assessed through specific database record linkage. All studies presented the adjusted outcome (hazard or odds ratio) with CI, but only two studies reported the use of interaction tests [35, 37].

Table 3.

Effectiveness of β-blockers (BB) in observational studies of heart failure

Study	Age, years (mean)	No. of patients	Follow-up	Outcome	Relative risk BB/no BB (odds ratio or hazard ratio)
I. Effectiveness of BB in elderly patients (depressed and preserved LVEF)
Sin et al. (2002) [34]	≥65 (79)	11 942	Median 21 months	Mortality, overall Mortality, per dose Readmission	0.72 (0.65–0.80) Similar effect low/high dose 0.82 (0.74–0.92)
Johnson et al. (2003) [19]	≥65 (79)	10 638	1 year	Mortality	0.61* 0.56†
Chan et al. (2004) [35]	≥65 (80)	950	Median 27 months	Mortality	0.74 (0.56–0.98)
Tandon et al. (2004) [36]	Median 69	1 041	Median 32 months	Mortality	0.63 (0.50–0.81)
Lenzen et al. (2004) [37]	(69)	9 716	Three months	Mortality	0.61 (0.48–0.77)
II. Effectiveness of BB in elderly patients with depressed LVEF
Shlipak et al. (2001) [20]	≥65	20 902	1 year	Mortality	0.76 (0.64–0.90)‡
III. Effectiveness of BB per different levels of renal function
McAlister et al. (2004) [29]	Median 69	754	Median 2.5 years	Mortality	GFR < 60 mmHg; 0.40 (0.23–0.70) GFR = 60 mmHg; 0.41 (0.19–0.85)

^*Relative risk reduction of BB vs. no ACEI or BB was calculated by dividing the adjusted mortality rates in each group.

^†Relative risk reduction of BB and ACEI vs. none was calculated by dividing the adjusted mortality rates in each group.

^‡Relative risk reduction of BB vs. no ACEI or BB.

With regard to outcome, studies can be split into three main categories (Table 3). The first category, which included most of the studies, assessed the effectiveness of BB in elderly patients with HF, irrespective of EF. All studies showed a RR in mortality with BB use of about 30–40%. Sin et al. [34] have also shown a significant reduction in hospitalizations with BB use, and no difference between low- and high-dose BB therapy on the risk of death. Lenzen et al. [37] reported a similar effect of BB in patients with depressed and preserved EF (interaction test). The second category assessed the effectiveness of BB in patients with depressed EF and found a RR in mortality of about 25–30%. The third category assessed the effectiveness of BB in patients with renal insufficiency and reported in this class an effect similar to that obtained in patients with normal renal function.

Discussion

This study indicates that observational studies provide relevant information regarding drug effectiveness in patient populations underrepresented or excluded from RCTs. Observational studies of ACEI and BB show a significant decrease in mortality with drug use in elderly patients with a broad range of EF, elderly with depressed EF, and patients with renal insufficiency. The effect of ACEI in HF with preserved EF is not clear from published studies, although recent evidence suggests a potential benefit. Even a low dose of ACEI and BB may have beneficial effects. Target dose of ACEI seems superior to low dose, but there is no clear dose–response relationship.

Most identified observational studies have been conducted after RCTs, having as a clear goal the validation of drug effectiveness in real-life HF populations. All studies included consecutive patients, resulting in patient populations with a strong resemblance to day-to-day clinical practice. Nevertheless, studies assessing the effectiveness of ACEI in patients with preserved EF are also hypothesis generating for further RCTs, as none has yet been completed.

Although several studies assessed the effectiveness of BB in elderly HF patients, it is of note that no study has yet explored the effectiveness of BB in HF with preserved EF. At present, the EuroHeart Failure Survey [37] is the only observational study to have suggested a similar benefit of BB in patients with depressed and preserved EF, but this evidence is provided at short follow-up (3 months), and through an interaction test. The SENIORS was the only RCT that assessed the effectiveness of BB in elderly patients (age ≥70 years) with HF, one-third of whom had a preserved EF [38]. The study showed that nebivolol reduces the risk of all-cause mortality or cardiovascular admission in elderly patients with HF, but was not designed to assess the specific effects of BB use in patients with preserved EF.

Most observational studies applied several methods to reduce potential bias. First, they adjusted for the most important confounders (age, gender, disease severity, comorbidities and concomitent medication). Second, most were conducted in big databases and had followed-up periods of at least 1 year (sufficient power and follow-up to capture the outcome). Third, all studies assessed drug prescription at discharge through clinical records and did not rely on patient self-reporting, which can determine a high degree of misclassification of (drug) exposure [39]. Similarly, outcome (mortality) was accurately assessed through a computerized database linkage. Although most studies of ACEI considered medication constant during follow-up, Rochon et al. [32] have shown that results are in fact similar when medication is used as a time-to-event variable or considered constant during follow-up. This may be characteristic of HF, a severe disorder, in which ACEI therapy is unlikely to be prescribed in an outpatient setting if it is not initiated during hospital admission, and a small percentage of patients (up to 10%) is likely to discontinue therapy during follow-up [40]. All these factors may explain why, despite some methodological differences (e.g. additional use of propensity score), studies show an approximately similar effect.

Both RCTs and observational studies have limitations that result in potential threats to validity, either external or internal. The main advantage of RCTs is randomization, which allows similarity in measured, unmeasured and unknown risk factors between treatment and control groups [2]. However, randomization, along with strict patient selection criteria, transform RCTs in careful crafted experiments, which do not reflect routine practice [41]. Therefore, after the successful conclusion of treatment benefit under study conditions (efficacy), its applicability to real life (effectiveness) needs to be tested [42]. A major issue is then to what extent observational studies can provide reliable estimates of the drug benefit, especially when RCTs are not available for validation [43]. The main threat to validity in observational studies is ‘confounding by indication’, which relates to the idea that patient characteristics influence drug prescription, and at the same time also relate to outcome (survival), therefore acting as confounders. Several methods are available to deal with confounding, in both statistical analysis and study design. In the former class, multivariate analysis and propensity score techniques are the most frequently used. Propensity score is the probability that an individual would receive a certain treatment based on their pretreatment characteristics (e.g. age, disease severity). This score can be used in different ways to adjust for the uncontrolled assignment of treatment [8]. However, most of the statistical methods cannot adjust for unmeasured or unknown confounders (e.g. genetical markers). Therefore, study design may be even more important to decrease the risk of confounding. For example, assessment of drug effectiveness in specific cohorts of interest (e.g. patients with renal insufficiency) may be superior to global assessments over a broad population; the more homogeneous the population, the lower the risk of confounding. Another potential bias in observational studies is ‘survivor bias’; patients who live longer have more opportunities to select treatment, whereas those who die earlier may be untreated by default [44]. To address this bias, at a minimum, initiation of therapy should be treated as a time-dependent covariate in a Cox proportional hazard model.

Despite methodological differences, recent evidence suggests that treatment effects obtained from randomized and observational studies may differ, but one method does not give a consistently different effect size than another [45, 46]. This is particularly the case when potential prognostic factors are well understood and controlled for in observational studies [3]. Therefore, the estimates of observational studies may guide the reasoning for further RCTs [47]. When a treatment shows a large harmful effect in an observational study, a RCT may be discouraged. Similarly, for interventions with very small effects in observational studies (risk ratio 0.90–1.00) a RCT may be difficult to perform given the sample size requirements. Therefore, interventions with average effects (risk ratio 0.40–0.90) are likely to be targeted by RCTs for precise effect measurement.

In line with previous reports, it was found that estimates of treatment benefit in observational studies are close to those reported in RCTs. For example, observational studies of ACEI show an overall RR in mortality of about 20–30%, similar to the benefit reported in two previous meta-analyses of RCTs [48, 49]. Similarly, the relative risk reduction with BB observed in observational studies is close to that reported in the meta-analysis of RCTs (approximately 30%) [50, 51].

Although in this case the effect of medication in observational studies was similar to that reported in RCTs, several divergent results have been cited in the literature. For example, the case of hormonal replacement therapy in postmenopausal women, which has indicated a protective effect on cardiovascular risk in observational studies whereas in RCTs it has shown an actual increase in the risk, remains the most debated case [52, 53]. As mentioned previously, the risk of confounding may be higher in a broad population, such as that of postmenopausal women. Also, prognostic factors in this population may not be well understood and therefore not possible to adjust for. This may be different from assessment of drug effectiveness in HF, or other specific disorder. However, it does not imply that the divergences between RCTs and observational studies are always explained by the methodological limitations of observational studies. Such divergences might be explained by inclusion of different study populations, different durations of follow-up, different healthcare settings, the involvement of patients and professionals in treatment decisions [3], or simply by inappropriate study design of RCTs [54].

Given the current evidence, we subscribe to the view that RCTs and observational studies do not exclude but complement each other, and both studies are necessary to provide a comprehensive picture of drug benefit. RCTs should be particularly encouraged (where possible) when an observational study suggests a different effect in a subset of patients not covered in prior RCTs.

This review has a number of limitations. First, it is limited to studies published in large electronic databases and does not include conference abstracts or other sources of publication. Given that publication bias is higher for observational studies than for RCTs [3], we might have missed several studies that showed no effect or a harmful effect. Second, it includes studies from 1990 to 2005, and some relevant studies might have been published outside this time interval.

In conclusion, observational studies in HF validate the effectiveness of ACEI and BB in patient populations underrepresented or excluded from RCTs, such as elderly patients with a broad range of EF, elderly with depressed EF, and patients with renal insufficiency. Observational studies therefore provide relevant additional information for clinical practice.