Observational comparative effectiveness studies of drug therapies: high-quality answers or important clinical questions?

Beta-blockers are among the most widely used therapies for the prevention of cardiovascular disease. Multiple clinical trials have established their efficacy in preventing death after myocardial infarction (MI) and in treating congestive heart failure (CHF) due to systolic dysfunction.^1,2 Beta-blockers have also long been used to treat hypertension. Although low-dose diuretics are the recommended first-line agent for pharmacologic therapy for uncomplicated high blood pressure,³ several large trials funded by industry have used beta-blockers as the active-comparison control treatment,⁴ and the results of these trials suggest that other therapies are more effective than atenolol in preventing cardiovascular events, particularly stroke.^5,6 Because no hypertension primary prevention trial has compared atenolol head-to-head with other beta-blockers, their comparative effectiveness in this setting remains unknown.

To address this question, Parker and colleagues conducted an observational study that compared the new use of atenolol and metoprolol, two widely used beta-blockers in the United States, for the prevention of MI, stroke and CHF in patients with treated hypertension.⁷ This study was nested within the hypertension registry of the Cardiovascular Research Network (CVRN), which includes all adult patients with hypertension enrolled in 3 large integrated healthcare plans from 2000 to 2009. Most beta-blocker use was in combination with other therapies, and half of the study population used diuretics within 6 months prior to starting a beta-blocker. For all outcomes, the relative risk estimates were null, and the confidence intervals excluded a greater than 2% increased risk associated with metoprolol use compared with atenolol.

This study has several strengths. The validation of entry criteria in the hypertension registry and the use of electronic prescriptions records allowed for a new user study design, which compares users of different treatments at a similar point in the natural history of hypertensive disease and avoids some sources of bias that are common in studies that include prevalent users of medications.⁸ Because of the careful use of restriction to exclude persons with known prevalent cardiovascular disease and even persons referred to a cardiologist, who may be more likely than non-referred patients to have undocumented or suspected but undiagnosed cardiovascular disease, the observed cardiovascular events likely reflect incident disease.

The authors used several analytic methods to minimize confounding bias. In one set of analyses, factors associated with both the choice of beta-blocker and the risk of outcomes were adjusted for. In another, propensity scores were used to make comparisons among a subset of the study population with similar probabilities of treatment based on known risk factors. Furthermore, atenolol and metoprolol, which are both cardioselective beta-1 adrenergic-receptor blockers, have similar pharmacologic properties and similar indications.⁹ The relative risk estimates from the two analytic approaches were similar, and due to large sample sizes, the 95% confidence intervals were narrow.

The study by Parker and colleagues also shares the traditional and persistent weaknesses of observational studies, particularly those that rely on administrative data. Some potential confounding variables are not well captured by administrative codes, and information on others -- smoking and lipid levels -- is simply not available. Beta-blocker use at baseline was the primary exposure, an approach that ignores discontinuation during follow-up or the use of additional antihypertensive therapies. Under the assumption that after adjustment, the allocation of therapies in clinical practice is equivalent to randomization, such a comparison resembles an “intention to treat” analysis of a clinical trial. However, it provides an unbiased estimate of relative risk under certain conditions: high rates of non-adherence, crossover, loss to follow-up and use of other effective therapies can all bias relative risk estimates towards the null.¹⁰

A common threat to the validity of studies of the intended effects of medications is the potential for confounding by indication, which occurs when use of a particular therapy is itself a marker for the duration, severity, or even likelihood of the disease in question.¹¹ For example, metoprolol has been studied extensively for the secondary prevention of cardiovascular disease, and it is often used in that setting.^1,12 If users of metoprolol were more likely to have had angina or suspected cardiovascular disease than users of atenolol, and if this information were not captured by administrative codes or cardiology referrals, the resulting bias may have masked a beneficial effect of metoprolol if one exists. The cumulative effect of these potential weaknesses is residual uncertainty.

Despite the limitations, the study by Parker and colleagues is an example of a well-designed comparative effectiveness study. The epidemiological effort to create a high-quality study nonetheless comes at a price. One of the frequently cited advantages of comparative effectiveness research is the ability to study “real-world populations” and thus avoid the often extensive exclusions that sometimes limit the generalizability of randomized clinical trials.¹³ The exclusions that formed part of the design of this study eliminated 170,771 (88%) of the 193,123 new users of beta-blockers. If the investigators cannot provide assurance of internal validity, the question of generalizability simply does not arise. The effort to preserve internal validity rather than include a population of broad generalizability is the optimal approach, one that is in this instance only partly “real world.”

Another major consequence of attention to a high-quality design is the adverse effect that these choices may have on the questions that are answerable. Because atenolol and metoprolol are both from the same class and subclass of drugs, the investigators managed to minimize the possibility of confounding by indication and thus preserve internal validity. However, on the basis of meta-analyses of randomized trials,^14,15 beta-blockers have lost favor as first-line agents for the treatment of hypertension, and the comparative effectiveness of metoprolol and atenolol -- both generic drugs -- is not a prominent clinical question in hypertension. Indeed, initiation of therapy with both drugs decreased markedly after 2006 in the CVRN hypertension registry.

In the area of pharmacological drug treatment for high blood pressure, the current question of primary interest is whether health outcomes associated with the use of hydrochlorothiazide and chlorthalidone may differ. Low-dose thiazide-type diuretics are the first-line therapy for the treatment of uncomplicated hypertension,³ and recent evidence suggests the possibility that chlorthalidone may provide better control of blood pressure and a lower risk of cardiovascular events than the thiazides.¹⁶ But in the US chlorthalidone is not widely used, and such a comparison may not have been possible in the CVRN. Reliable and valid comparisons between hydrochlorothiazide and chlorthalidone will require a large long-term clinical trial.

While careful attention to study design is necessary for the results of observational comparative effectiveness studies to be credible, the high-quality answers may not involve the most important clinical questions, and the key clinical questions may not be susceptible to high-quality answers.