Accounting for Biases when Linking Empirical Studies and Models

Comparative effectiveness and cost-effectiveness analyses have become increasingly common and important [1–3]. Such analyses aim to assess multiple interventions by comparing their long-term outcomes and costs for real-world patient populations and care delivery settings. To inform current decisions, empirical studies of sufficient size and duration to assess long-term outcomes are often infeasible. Computer-based mathematical models can play a valuable role by providing the needed estimates. Such models generate outcomes based on inputs derived from shorter-term empirical studies. Therefore, appropriate methods for linking empirical studies to mathematical models underpin rigorous analyses. An important concern is that estimates from empirical studies contain a variety of potential biases which if not corrected can influence modeled outcomes and resulting policy recommendations.

With recent developments such as the establishment of the Patient Centered Outcomes Research Institute [1,2], the methods literature on combining the strengths of empirical studies and mathematical models to support comparative effectiveness and cost-effectiveness analyses has grown rapidly. The article by Gastineau Campos and colleagues in this issue of Medical Decision Making contributes to this area [4]. The authors describe an approach for assessing and correcting potential biases in estimates from a randomized controlled trial (RCT) of cervical cancer screening technologies. They then provide an illustrative example of the importance of this approach by showing how such bias corrections might alter cervical cancer screening policy conclusions reached with biased estimates. Their approach is designed for situations where individual-level data from a single RCT are available. It represents a valuable tool for other researchers facing similar situations.

Their approach contributes to a consistent set of best practices for selecting, adjusting for bias, and incorporating empirical evidence into decision-analytic mathematical models -- a need that has been previously acknowledged [5,6]. Such best practices can aid researchers who confront a number of typical situations when selecting empirical study estimates to use in their model-based analyses. These situations can be characterized by: 1) number of available studies (one vs. more than one); 2) study types (randomized controlled trials (RCT) vs. observational studies); 3) available information (summary estimates from published articles vs. individual-level study data).

Ideally, when multiple studies are available, decision-analytic mathematical models should incorporate their combined information. Systematic review and meta-analytic methods, including Bayesian synthesis and meta-regression techniques are well developed [7,8]. One important source of potential bias when combining study estimates is failure to account for study differences that can influence outcomes, an area where meta-regression and other similar techniques can be particular useful. Another common challenge in combining all available, high-quality studies without introducing bias is accounting for that fact that studies report different and difficult-to-compare measures. For example, it may be important to assess the impact of diabetes control on health outcomes, but separate studies have measured diabetes control in terms of fasting plasma glucose, oral glucose tolerance tests, or HbA1c. Mapping the effect of a unit change in one measure to that of another requires careful analysis [9].

In attempting to avoid bias, analysts often prefer randomized trials to observational studies, though this choice is not always obvious. The rationale is that randomization balances unobserved covariates producing an unbiased estimate of the effect of the intervention. However, RCT estimates are only unbiased for the often highly selected patient population in which they are conducted. Incorporating them into a model to consider outcomes for a patient population that differs from the trial population risks introducing serious bias. Observational studies often are conducted in larger, less selected patient populations over longer time periods and may provide estimates that are more applicable and generalizable – potentially more appropriate for inputs into a model. However, observational studies may suffer from non-random selection issues that introduce confounding biases. Attempts to correct such biases in observational studies often employ propensity score methods [10]. Propensity score methods estimate effects after balancing observed covariates, but they only provide unbiased causal inference and effect estimates if the strong and non-verifiable assumption of also balancing unobserved covariates is met [11]. In the end, the choice may rest upon a subjective assessment of the relative magnitude and import of non-generalizability versus confounding.

In situations where RCTs are preferred or the only available study is an RCT, researchers may still be worried about biases in study estimates, as Gastineau Campos and colleagues illustrate. For example, when randomization does not result in balance for all observed covariates, it is possible that differences in treatment effects may be due to this imbalance. Even when randomization is successful, if study blinding is incomplete, selection issues can arise in which the de facto treatment assignments are not balanced with respect to observed or unobserved covariates, again influencing the observed treatment effects. Gastineau Campos and colleagues provide one practical approach to these challenges and also cite methods developed by Eddy [12]. A particularly challenging case is when biased selection leads to differential follow-up and hence differential missingness of outcomes across arms. While imputation methods are often employed in such situations [13,14], incorrect assumptions about the process generating the missingness can introduce bias. Manski describes bounding methods that require fewer strong assumptions about the generating process of missingness but often lead to large uncertainty intervals [15].

When individual-level data from multiple studies are available, one could imagine employing bias correction methods for each individual study and then using standard meta-analytic methods to combine their estimates. It is unclear whether the bias correction approaches could be directly incorporated into individual-level meta-analytic techniques [16] to maximize the available information while simultaneously reducing biases. This is a potentially important area of future research.

When multiple studies are available but their individual-level data are not, direct adjustments for bias may not be possible. Consistency across multiple studies may then be the best feasible criteria for inclusion in model-based analyses. Direct comparison of study outcomes to assess consistency is often possible. When it is not, the model itself may play a role in the assessment – examining whether inputs from one study lead model-generated outcomes that are consistent with other studies. It is still possible that both the input study and the comparator study are similarly biased, in which case consistency assessment is not helpful. However, when a larger number of studies are available, the possibility of correlated biases seems less likely. This approach applies techniques developed for model evaluation and external validation [17,18].

The rigor and applicability of comparative effectiveness and cost-effectiveness analyses can benefit from methods that appropriately link unbiased study estimates to decision-analytic mathematical models. Gastineau Campos and colleagues provide a cautionary tale to decision analysts about assessing empirical studies prior to including their unadjusted estimates in mathematical models. Their bias correction approach contributes to the growing body of methods literature in this area.