The challenges of comparing results between placebo controlled randomized trials and non-experimental new user, active comparator cohort studies: the example of Olmesartan

One of the challenges of pharmacoepidemiology that is both exciting and frustrating is the replication of findings. It is often easy to frame the question as “does drug X cause side effect Y”, but this ignores subtle differences in the actual contrasts that are being made. This issue is well highlighted by three recent studies of the cardiovascular safety of the medication Olmesartan published in this issue of the Journal ^1–3. These non-experimental (observational) studies were motivated by safety signals from two randomized, placebo controlled trials that suggested that Olmesartan increased all-cause mortality in patients with diabetes^4–5. Although the absolute number of deaths observed in the randomized trials was small, the possibility of such an effect is a legitimate cause for concern ^1–3. The three non-experimental studies are all well designed and placed in independent populations in two different countries (the United States of America and the United Kingdom).

However, right from the beginning, there are intrinsic challenges with replicating a randomized, placebo controlled trial using a non-experimental study design. Drugs are given for specific indications and one of the primary indications of angiotensin receptor blockers (ARBs), including Olmestartan, is the treatment of hypertension. Unfortunately, hypertension itself is a risk factor for major cardiovascular events. So any analysis of the cardiovascular safety of Olmesartan has to deal with confounding by indication⁶, which can be tricky to overcome.

This concern about confounding by indication makes it inadvisable to do a non-experimental absolute safety comparison in this particular context. Instead, the non-experimental studies ask a closely related question: is initiating patients on Olmesartan risky relative to initiating patients on other ARBs and angiotensin converting enzyme inhibitors (ACEI)? This new question restricts the potential for confounding by indication insofar as the indication for these drugs and drug classes is similar. It is also sensible, since medications in this class are known to have positive effects on all-cause and cardiovascular mortality⁷ when used to treat hypertension (a major reason for prescribing these medications). So it would be unlikely that participants who are prescribed this drug in the general population would ever be left untreated, but concerns about cardiovascular risk would likely lead to the substitution with another anti-hypertensive medication. Designing these non-experimental studies as comparative safety studies refocuses the inference from the specific comparison in the trials (Olmesartan versus placebo) to what is clearly the most concerning possibility - that there is a previously undetected excess risk of mortality and/or cardiovascular events with Olmesartan compared with readily available clinical alternatives.

When comparing results between these studies, we need to also understand the differences between the study populations and choice of subgroup analyses. In the non-experimental studies the study populations are more representative of the general population of Olmesartan initiators with subgroup analyses conducted on diabetics. In all patients initiating ARBs/ACEIs, none of the studies found an association between initiation of Olmesartan versus other ARBs/ACEIs and all-cause or cardiovascular disease mortality. In both trials, the study populations were diabetics either with risk factors for cardiovascular disease (ROADMAP) or overt nephropathy (ORIENT). For all-cause mortality, while ROADMAP reported an elevated risk (HR 1.70), in ORIENT the estimate was close to the null (HR 0.99). In diabetics, Zhou et al. also found a slightly elevated risk which was even higher among those treated with high versus low dose of Olmesartan (HR 1.35). Walker et al. found no relative increase in risk of all-cause mortality among diabetics. Regarding cardiovascular death, both trials reported an elevated risk in users of Olmesartan (HR 4.94 ROADMAP; 2.81 ORIENT). Only the study by Walker et al reported on cardiovascular related death among diabetics (HR 2.10), consistent with the trials. The exception was in diabetics with prior use of ACEIs in whom no association was found (HR 0.80). Graham et al. used an outcome definition enriched for cardiovascular deaths and conducted analysis among diabetics treated with high dose, comparing short versus long term use (<6 vs ≥6 months). They found an elevated risk among those who had longer use (HR 2.03).

Apart from the active comparator design decision, there are more general issues with the comparison of non-experimental and randomized studies, given the concern about generalizability, namely sub-group effects. In the presence of treatment effect heterogeneity it becomes problematic to generalize from the trials. In diabetics, prior cardiovascular disease is a potential modifier of the effect of Olmesartan on mortality. Only ROADMAP conducted a subgroup analysis stratifying by coronary heart disease history. In this trial participants with prior coronary heart disease had an elevated risk for all-cause mortality (HR 4.16) and cardiovascular death (HR 10.6) compared with participants without coronary heart disease (HR 1.08, and HR 2.00, respectively). It is known that there are also strong associations between age, sex and both all-cause and cardiovascular mortality. However differences by sex or age were not explored in the trials or the observational studies. Another important consideration is the scale in which results are reported. From a clinical standpoint, reporting both the relative and absolute difference in risk of death across subgroups could be useful to assess the change in individual risk associated with Olmesartan treatment.

Another key difference between the randomized controlled trials and the non-experimental studies is the shorter duration of treatment in the non-experimental cohorts. Both of the randomized trials had a mean duration of treatment greater than 3 years ^4–5, whereas the cohorts had mean duration of treatment of 1.2 years or less^1–3. As an illustration, in the Graham et al. study, there were only 10% of participants who were still taking the same drug at one year. This paucity of long term use mostly reflects the reality of how the drugs are taken in real clinical practice. It could have been alleviated to some extent by allowing patients to switch to another drug of the same class (at the cost of some exposure misclassification). While the lack of data on long term use may be seen as a limitation of any non-experimental new user study, it also speaks to the remoteness of RCTs with respect to actual treatment patterns. It also clearly limits the scale of the public health impact of any potential increase in risk with long term treatment, although knowing about such risks remains important.

Given the legitimate worry about confounding in non-experimental studies, all of the studies used some form of propensity score based approach as a part of their inference which enabled them to broadly adjust for the information on potential confounders available in large automated healthcare databases. The use of a large number of covariates in the propensity score is becoming increasingly common in pharmacoepidemiology as evidenced by the use of high dimensional propensity score⁸. One advantage of propensity score approaches is that patients always treated with one of the drugs compared based on their covariates can be excluded. After exclusion of such patients, in whom the treatment effect cannot be estimated, estimates apply to participants who could have received either treatment, i.e., resembling the target population of patients in whom a treatment decision needs to be made. In addition, trimming the tails of the overlapping propensity score distribution can reduce the potential for bias caused by poorly captured confounders¹⁰ that may be present in those patients treated contrary to prediction. The small loss of interpretability (estimate does not apply to all treated) is almost certainly worth it for the increase in internal validity, even if it does complicate direct comparisons. Additional considerations could also be made on the use of the overall versus a subgroup specific propensity score especially in when suspecting the presence of a strong confounder such as prior history of coronary heart disease¹⁴.

So what is the current evidence on Olmesartan and mortality? Combined, the three non-experimental studies provide current best evidence for the comparative safety of Olmesartan as currently used in the US and UK. None of these studies sheds a lot of light on the specific question asked by the trials, as the data that would be required to precisely replicate the precise causal question asked in the trials was simply not available. Furthermore, even attempting a precise replication would have been highly inadvisable, due to concerns about confounding by indication⁶ and underlying differences in participant propensity to adhere to medication assignment (as seen by the low persistence on therapy in the non-experimental studies). The results of these studies do, however, all appear to agree that there is little evidence that the excess cardiovascular mortality detected in the ROADMAP and ORIENT trials⁴ applies to the majority of users of Olmesartan. Among diabetics, however, the association between Olmesartan and cardiovascular mortality is less clear and may require a closer look. Small trials and subgroup analyses are vulnerable to chance findings, and regression to the mean suggests that the initial size of large effects may well shrink upon replication¹⁵. In the absence of additional data on Olmesartan in patients with diabetes, physicians and patients should weigh the potential benefits and harms versus available clinical alternatives before initiating Olmesartan in patients with diabetes.

Figure 1.

Forest Plot summarizing the results from the randomized controlled trials and non-experimental studies discussed in the commentary. Cardiovascular Disease (CVD) mortality was defined as sudden cardiac death in the study by Walker AM et al. Results from Graham et al. are not reported in some of the subgroup analysis due to lack of comparability.