Adverse Outcome Analyses of Observational Data: Assessing Cardiovascular Risk in HIV Disease

Clinical decision making often relies on observational data analyses. Attention to 7 readily understood statistical factors will assist in understanding observational studies. Responsible public access to de-identified source data, if feasible, is recommended to assess methodological limitations.

Abstract

Clinical decisions are ideally based on randomized trials but must often rely on observational data analyses, which are less straightforward and more influenced by methodology. The authors, from a series of expert roundtables convened by the Forum for Collaborative HIV Research on the use of observational studies to assess cardiovascular disease risk in human immunodeficiency virus infection, recommend that clinicians who review or interpret epidemiological publications consider 7 key statistical issues: (1) clear explanation of confounding and adjustment; (2) handling and impact of missing data; (3) consistency and clinical relevance of outcome measurements and covariate risk factors; (4) multivariate modeling techniques including time-dependent variables; (5) how multiple testing is addressed; (6) distinction between statistical and clinical significance; and (7) need for confirmation from independent databases. Recommendations to permit better understanding of potential methodological limitations include both responsible public access to de-identified source data, where permitted, and exploration of novel statistical methods.

Clinical treatment guidelines are based on data from various sources including randomized controlled trials (RCTs) and observational studies [1]. Observational data are frequently useful for clinical decision making, especially for potential drug-related adverse events, yet pose a unique set of challenges [2]. Knowledge of the strengths and limitations of observational data is essential for interpretation and application to clinical practice. While guidelines exist on standards for reporting of epidemiological analyses [3–8], similar standards for clinical decision making do not. The imperative to make clinical decisions, advise patients, and craft clinical guidelines justifies discussion of important factors to be considered by clinicians in reviewing observational data analyses. This article provides a summary of such factors, using adverse cardiovascular outcomes in patients with human immunodeficiency virus (HIV) as exemplary.

RCTs serve as the foundation for development of clinical guidelines. They are only rarely susceptible to bias given that, on average, the random assignment of treatment ensures equivalence of characteristics between study groups. RCTs are also the gold standard for inference of causation, or support for a conclusion that the investigated exposure or variable is causative of the disease outcome being assessed (in contrast to an association in which the exposure may be related in a noncausal manner). However, RCTs have certain limitations: for example, their generalizability can be limited if study patients with characteristics different from those of the target population are selected by restrictive enrollment criteria (eg, less complicated disease states, increased adherence, or predisposition for morbidity). In addition, RCTs often address questions that differ from those targeted by observational data analyses.

Observational studies have unique strengths and are widely used when RCTs are not feasible. An observational study design can be used to investigate adverse events that RCTs may lack power to assess if the events are rare or occur after a prolonged period of time. Furthermore, an observational study design may be used to analyze factors that cannot be randomized, such as smoking behavior. Observational data may establish associations between drug exposure and risk of complications that may serve as triggers for pathophysiological investigations or for confirmatory RCTs. Consistency across separate studies in direction of association and strength of findings, dose-exposure relationships, or a well-understood biological mechanism strengthen the support of observational studies for causal inference. However, the ultimate validity of estimates of causal effects based on observational data rests on an untestable assumption that one can remove or adjust for all bias.

Observational studies vary in design and include cross-sectional studies, case-control studies, case-cohort studies, and longitudinal studies. Temporality, a necessary criterion for causality, can be established in longitudinal observational studies that observe participants over time. Longitudinal studies can be both retrospective, when data are drawn from previously compiled databases such as medical records, and prospective, when data are collected progressively over time. In a prospective clinical cohort study, patient data can be culled from medical records going forward in time. In a prospective interval cohort study, specially designed data collection mechanisms are regularly administered prospectively, such as with surveys, case report forms, biological specimen collection, or physical examinations [9]. An ideally performed observational study entails (1) a primary hypothesis and a rigorous prespecified protocol for data collection (electronic posting of the study protocol could assist readers in the interpretation of the findings); (2) explicit definition and method of ascertainment for exposure(s) and outcome(s) of interest; (3) preferably a requirement for prospective data capture; (4) specification and validation of baseline and time-dependent confounders; and (5) full details of the primary statistical and sensitivity analyses.

Observational data have been particularly relevant to understanding the association between HIV infection and cardiovascular disease (CVD). Multiple studies have found higher CVD rates among HIV-infected persons, including children and adolescents [1, 10–12], compared with reference control groups [13–16]. Increased CVD rates seem to be independent of traditional cardiovascular risk factors (eg, smoking and hyperlipidemia), which are also elevated in this population [17]. Several studies have reported that use of specific antiretroviral medications or classes is associated with an increased risk of acute myocardial infarction (AMI) [18–21]. Some of these associations were expected (eg, dyslipidemia after administration of certain antiretroviral drugs), and some unexpected (eg, association between the nucleoside reverse transcriptase inhibitor abacavir and AMI) [22]. This latter result has prompted multiple subsequent investigations with inconsistent findings [23–32].

Several factors have precluded the ability of RCTs to assess the association between CVD and antiretroviral medications, with consequent necessity to rely on observational data. A study that requires patients to be randomized to specific HIV medications or removed from HIV treatment entirely might not be ethically feasible if the patient meets criteria for treatment or has a contraindication to certain drugs, or if the potential for drug resistance exists. Rapidly changing treatment paradigms for HIV may not permit lengthy RCTs, which would be required for development of CVD outcomes.

The inconsistent findings from observational data analyses of long-term cardiovascular risk in HIV-infected patients led the Forum for Collaborative HIV Research to convene a series of roundtable discussions addressing the development of clinical guidance on CVD prevention and management based on current observational data. The Forum process involves stepwise identification of public health issues, background research, formation of roundtable discussions composed of scientific experts, and publication of consensus. Statistical experts, basic scientists, and clinicians from cardiology and HIV medicine met to formulate a potential consensus. A concluding Forum report was presented as a panel discussion at the 2010 International AIDS Conference (Vienna) that summarized clinical implications of the current published data [33]. Subsequently, the authors identified a number of key factors that should influence interpretation and clinical applicability of observational study findings, which form the basis for this article.

CONFOUNDING AS A SOURCE OF BIAS

In RCTs, the patients are randomly assigned to an intervention or placebo to minimize differences between the groups. Differences in groups contribute to bias in estimates of causal effects. Confounding is a common source of bias in observational studies and is induced by the presence of factors that systematically influence the outcome being examined and are distributed differently between the exposed and unexposed groups. Confounding by indication occurs when the decision to administer a drug is not randomized. Those who receive the drug may disproportionately have characteristics other than drug exposure that could convey risk of an outcome, such as AMI. As an illustration, abacavir may have been preferentially prescribed to patients with increased risk factors for CVD, so increased CVD rates in the abacavir group may be the result of a lack of homogeneity in cardiovascular risk factors between the compared groups. Likewise, protease inhibitors (as a class of antiretroviral drugs) have been linked to dyslipidemia, so an alternative nonnucleoside reverse transcriptase inhibitor–based regimen may have been preferentially prescribed to patients with higher CVD risk and might then appear to be associated with increased CVD events.

In theory, adjustment for bias from confounding is possible if potential risk factors that may have governed prescribers’ decision making and preferences are known and correctly measured. Adjustment can help to correct the possible confounding effects of such inhomogeneously distributed variables, but adjustment cannot be assumed to fully eliminate bias. An unknown level of residual bias arises due to unknown or unmeasured confounders or insufficient adjustment, such that one exposure group has residual risk related to factors not present to the same degree in the other. That said, a substantial change in relative risk after adjustment for measured confounders suggests that the original groups were indeed heterogeneous. The key point to consider is that adjustment for confounding by indication requires that covariates that predict initiation of treatment must be measured and available before treatment initiation in order for any statistical approach to appropriately adjust for the potential bias.

Furthermore, care should always be taken when adjusting for potential confounding factors that may also lie on the causal pathway between the proposed risk factor and the outcome of interest. For example, dyslipidemia is associated with both the choice of antiretroviral drugs and AMI, and it is on the causal pathway between some antiretroviral drugs and AMI; it is both a confounder and an intermediate variable. Handling such time-varying confounding is problematic using standard regression methods. Causal modeling methods may be used but are complex to apply in anything but the most straightforward setting [34].

MISSING DATA

Incomplete or missing data represent a significant challenge in the use and interpretation of observational data. Data may be missing for a confounding covariate, such as smoking; for the exposure of interest, such as drug usage; or for the outcome variable, such as AMI. Reasons for missing data include incomplete or inefficient capture in medical records, errors in extraction of data, loss to follow-up, difficulties in capturing time-dependent variables such as exposure time, or differences in methodology for capture of events by centers, practitioners, or changes over time. The impact of missing data is most worrisome when data on confounders or the outcome are not missing at random. If the reason the data are missing is based to some degree on unmeasured factors that predict the outcome, then no statistical estimation method can recover the desired causal effect without making untestable assumptions. If, on the other hand, this “missingness” is a function of measured factors and can be well estimated from the data, then sophisticated methods can be used to deal with the challenge of estimating the effect, notwithstanding missing data.

Individual missing data can be handled by several methods: (1) by excluding patients with missing data from the analysis; (2) by maximum likelihood estimation that explicitly assigns or imputes a value to that missing information and providing proper statistical inference that acknowledges the uncertainty due to imputation; or (3) by using other advanced or novel statistical methods [35, 36]. Imputation generally results in less bias than other methods, or excluding those with missing covariate information, but careful understanding of the underlying assumptions of the method and its practical performance is required [37, 38]. Disclosure of the extent of missing data, how missing data were handled, underlying assumptions of the chosen method, and the potential impact of absent data will help to facilitate understanding of study findings, analytical limitations, and robustness.

Another form of bias involves incomplete data on the outcome of interest, such as AMI. Such ascertainment bias could occur when patients at low CVD risk, who may be concentrated in one group, have more missed outcome events. Efforts to ensure universal ascertainment are an important and recommended part of the ideally designed observational study.

DEFINITION OF CLINICAL TERMS

Comparison of observational studies is often complicated by inconsistency of definitions of clinical covariates, such as confounders, exposures, or outcomes. Recognition of different end point definitions is important in comparing study results. Many cardiovascular studies, including RCTs, define CVD with combined end points of myocardial infarction, revascularization, stroke, and/or cardiovascular death. In HIV studies, the Strategies for Management of Antiretroviral Therapy trial defined CVD as AMI (including death due to clinical AMI and silent AMI), stroke, or coronary artery disease requiring surgery or invasive procedures [39], whereas the D:A:D observational study defines AMI by World Health Organization MONICA project criteria [40] adjudicated centrally [18]. Some studies have used different International Classification of Diseases codes to indicate a cardiovascular outcome, with some using codes for coronary heart disease [13] and others using codes for myocardial infarction [16]. Historical changes in the definition of AMI limit direct comparison of CVD studies over time. As with RCTs, consideration and comparison of multiple observational studies on related risks are optimized when definitions of clinical terms are similar, although such standardization is not always feasible.

MULTIVARIATE MODELING

Standard statistical adjustment often assumes a linear or categorical relationship of confounders and exposures to outcome. Although this assumption is simple, it is not necessarily correct when it is applied to clinical data. For example, increases in total cholesterol have a nonlinear impact on CVD risk, with small increases in CVD risk for cholesterol levels from 100 to 200 mg/dL and escalating risk thereafter. In contrast, absolute changes in CD4 cell counts are more significant at the lower end. In addition, time dependency of exposure may not be considered in basic analyses. Categorical classification as “hypertensive” based on the use of antihypertensive agents or the most recent blood pressure does not take into account variation in blood pressure over time. To more accurately assess its impact on outcome, a confounder or drug exposure may be time-updated so that an exposure is calculated from baseline through the last available measurement. The analysis of the relationship between drug exposure and outcome becomes substantially more complicated. More sophisticated analytic methods may be required. Thus, linear regression models and categorical adjustments oversimplify a complex reality. More recent developments in statistics—for example, marginal structural models, machine learning, and data adaptive methods—and increased computing power may permit more accurate estimates of the desired causal effect of the exposure on the clinical outcome and should be evaluated for practical use. In particular, marginal structural models are appropriate and efficient when decisions to initiate treatment are based on past covariates and treatment initiation itself affects subsequent measures of these same covariates [34]. Simple adjustment for these time-updated covariates in this scenario may “adjust away” the effects of the treatment being investigated, whereas marginal structural models take these pathways into account.

In smaller databases or in situations where the outcome is rare, analysis of and adjustment for multiple covariates, or risk factors, may also be impeded by too few events. At least 10 events per covariate are needed in event-based regression models [41]. If the number of outcome events is small in comparison to the number of confounding factors to be adjusted for, interpretation of multivariate analysis findings may be less reliable.

MULTIPLE TESTING

If multiple testing is not adjusted for, the probability of falsely rejecting the null hypothesis—claiming an association is present when it is not—increases with the number of evaluations performed. Common use of 2-sided 95% confidence intervals assumes that 1 in 20 represents an acceptable degree of false-positive findings. Multiple tests applied to truly random associations must eventually result in at least 1 factor erroneously showing a significant association with the outcome. There are currently >20 antiretroviral drugs spanning 5 classes. If performing 29 statistical tests looking at association of antiretroviral drugs to CVD, the odds of incorrectly identifying a positive relationship is 80%.

Multiple testing may be accounted for through formal statistical techniques. In the absence of adjustment for multiple testing, the primary study objective should be taken as the main focus of the analysis. Secondary and investigative analyses should be treated as hypothesis generating, not hypothesis testing. Replication of findings in discrete data sources can help to address multiple testing. Discussion of the risk of false-positive findings, disclosure of the prespecified testing plan, and correction strategies for multiplicity or justification for not addressing multiple testing would assist readers in interpreting study findings.

STATISTICAL VERSUS CLINICAL SIGNIFICANCE

Analyses using large databases can achieve statistical significance with differences that lack clinical meaning. In a large database, small residual confounding effects may result in statistically significant exposure-outcome relationships, even when no underlying causal relationship exists. The absolute as well as relative increases in risk should be assessed, and interpretation of the statistical output in terms of clinical meaning should be provided. In general, determining clinical significance relies on the consensus opinion of clinical experts and the patient community.

CONFIRMATION OF FINDINGS

Confirmation of findings in discrete data sources can strengthen conclusions from observational studies. Findings can be confirmed internally by testing a portion of the original database but holding a second portion aside to confirm the finding—“test and hold.” However, the preferable approach is to confirm findings using a discrete data source. Because datasets differ with regard to specific sources of bias, data availability, and analytic techniques, repeating a study in an alternate dataset may address many of the potential limitations of an individual dataset, including confounding, missing data, and false-positive results. Furthermore, alternate statistical techniques may be used, such as those that use time-varying covariates. However, some limitations may be common to many datasets, such as confounding by indication based on common provider practices. Obtaining similar results from independent datasets with different intrinsic limitations increases confidence.

RECOMMENDATIONS

Two steps might be useful to facilitate responsible use of findings from observational study analyses. First, increasingly, public access to data from RCTs has been required by editorial policy [42], US law [43], the US Food and Drug Administration (FDA) through Summary Bases of Approval, adverse event reporting, or meta-analysis (see FDA analysis of abacavir exposure and risk of AMI) [27]. Similar responsible public access to de-identified observational databases may be feasible in some cases and offers the potential benefit of a deeper understanding and explanation of methodological effects. Availability of data for public health agency review may facilitate better and more informed decision making by regulators. Second, novel statistical methods for adjustment (including more sophisticated nonparametric methods) should be investigated to verify whether more reliable assessment of the causal effect of interest could be provided.

CONCLUSIONS

Observational data are critical to clinical decision making and to informing patients but must be understood in light of methodological limitations. Standards for rigor in observational analyses have not evolved concomitantly with those for RCTs. Seven key factors are important when interpreting observational data: (1) explanation of likely implications of confounding and approaches to adjustment; (2) the extent, nature, handling, and impact of missing data; (3) consistency of definitions for outcome and covariate terms when cross-comparing studies; (4) methods for adjusting for time-dependency; (5) the risk of false-positive findings and handling of multiplicity; (6) the difference between statistical and clinical significance; and (7) the importance of confirmation from independent databases. Many participants recommended that responsible public access to databases, if feasible, could permit investigation of methodological concerns and validation by novel analytical approaches. As observational data sources continue to increase, balanced understanding and responsible use of such data will be critical to inform clinical care and public policy.