Propensity Score Methods and Unobserved Covariate Imbalance: Comments on “Squeezing the Balloon”

Abstract

In their recent Health Services Research article titled “Squeezing the Balloon: Propensity Scores and Unmeasured Covariate Balance,” Brooks and Ohsfeldt (2013) addressed an important topic on the balancing property of the propensity score (PS) with respect to unmeasured covariates. They concluded that PS methods that balance measured covariates between treated and untreated subjects exacerbate imbalance in unmeasured covariates that are unrelated to measured covariates. Furthermore, they emphasized that for PS algorithms, an imbalance on unmeasured covariates between treatment and untreated subjects is a necessary condition to achieve balance on measured covariates between the groups. We argue that these conclusions are the results of their assumptions on the mechanism of treatment allocation. In addition, we discuss the underlying assumptions of PS methods, their advantages compared with multivariate regression methods, as well as the interpretation of the effect estimates from PS methods.

The use of propensity score (PS) methods in observational studies of medical treatments to adjust for measured confounding has increased substantially during the last decade (Shah et al. 2005). The PS is defined as a subject#x0027;s probability of treatment given his or her characteristics. For groups of subjects with the same PS, measured covariates that were used to construct the score tend to be balanced across treatment groups (Rosenbaum and Rubin 1983). However, unlike random assignment of treatments in a randomized trial, covariates that were not measured (and thus not included in the PS model) will not necessarily be balanced when conditioning on the PS. Hence, imbalances in unmeasured covariates are not addressed by PS methods.

In a recent study, Brooks and Ohsfeldt assessed the balancing property of PSs with respect to unmeasured covariates (Brooks and Ohsfeldt 2013) and wondered why subjects with the same PS may receive different treatments. Essentially, they stated that for two subjects with the same PS, for example, a PS of 0.65 (i.e., a probability of 0.65 for receiving treatment), of whom one received treatment and the other did not, there must be a reason why one received treatment and the other did not. They argued that, apparently, this reason was not included in the PS model (thus being unmeasured covariates, potentially confounders). Thereafter, they concluded that PS methods balance measured confounders at a cost of exacerbating any imbalance in unmeasured covariates that are independent of the measured covariates. Further extended, if these unmeasured covariates are confounders (related to both treatment and outcome), PS methods can exacerbate the bias in treatment effect estimates.

We do not agree with their main conclusions for reasons outlined below. In the following paragraphs, we focus on three topics: (1) the assumptions underlying PS methods; (2) the conceptual advantage of PS methods in contrast to classical regression techniques; and (3) the estimand (treatment effect estimate) obtained using multivariable regression and the different PS approaches.

The Assumption Underlying PS Methods

Two main assumptions underlying PS methods are relevant to interpret the findings by Brooks and Ohsfeldt: exchangeability and positivity. As described in the original paper by Rosenbaum and Rubin (1983), PS methods rely on the assumption of strong ignorability or exchangeability, which is a stronger version of “ignorable” mechanisms coined by Rubin (1976, 1977). It can be stated formally as: {Y (0), Y (1)} ⊥ X|Z, where Z denotes a vector of (measured) covariates, X denotes treatment assignment (Yes/No), and Y (0) and Y (1) are the potential outcomes under control and treatment conditions, respectively. It means that conditional on (measured) covariates (Z), treatment assignment (X) is independent of potential outcomes (Y (0), Y(1)). Intuitively, this assumption is equivalent to the colloquial notion of all confounders having been measured (Hill 2008). This assumption of no systematic, unmeasured, pretreatment differences between treated and untreated subjects that are related to the outcome under study is needed not only for PS methods but also for ordinary methods to adjust for confounding (e.g., multivariate regression methods) to get an unbiased treatment effect estimate.

In a large randomized clinical trial (RCT) where subjects are assigned to treatment or control by flip of a fair coin, the probability of being assigned to treatment (i.e., the PS) is 0.5. This means that approximately half of the subjects will be treated while the other half will not. The randomization implies that, on average, measured as well as unmeasured covariates will be balanced between the two treatment groups. In contrast, in observational studies, treatment assignment is a non-random process. Nevertheless, PS methods help researchers mimic randomization by creating a sample of subjects receiving the treatment that is comparable on all measured covariates to a sample of subjects not receiving the treatment (Austin 2011, 2007; Jo and Stuart 2009). Hence, randomization in a RCT and mimicking random assignment of treatment in PS analysis (e.g., by PS matching) are sufficient to generate group of subjects with the same PS yet receiving different treatment modalities. This does not require differences in unmeasured covariates. However, randomization implies that unmeasured covariates are balanced as well, but PS methods will not guarantee balance of unmeasured covariates.

Brooks and Ohsfeldt considered a deterministic treatment assignment model rather than a random treatment assignment model given measured and unmeasured covariates. Particularly, in their simulation set-up, treatment status depends on a number of covariates. Given that the values of these covariates are known, the treatment status is fixed, that is, it no longer depends on any random process. Thus, if treatment status of a subject as well as the values of certain covariates (e.g., three out of four covariates) is known, this will restrict the values of the other covariate(s). In fact, the set-up of the data generation implies that among the treated subjects, those who have low values for certain covariates will have high values for the other covariates (and vice versa). Consequently, by design, balancing part of the covariates between the treatment groups (by PS matching) will result in an imbalance in the other covariates. Brooks and Ohsfeldt demonstrated an “exacerbation” in the imbalance of unmeasured covariates when using PS methods to balance measured covariates compared with the full unweighted sample. This finding could in part be explained by the fact that they not only assumed unmeasured covariate variation that is unrelated to measured covariates as a requirement for PS methods but also imposed this thought in generating their data (in particular, the simulation of treatment status). Moreover, in their simulations, Brooks and Ohsfeldt demonstrated that the “exacerbation” in imbalance of unmeasured confounders was only detected when the measured and unmeasured covariates are uncorrelated. Hence, if one attempts to measure as much covariates as possible, it seems more likely that risk factors correlated to unmeasured covariates or proxies for unmeasured covariates included in the PS model, for example, using high dimensional propensity score (hdPS) method (Schneeweiss et al. 2009; Rassen and Schneeweiss 2012), could lessen the exacerbation or improve balance of unmeasured covariates.

Apart from the assumption of no unmeasured covariates (confounders), another requirement for identifying a causal effect is that both treated and untreated subjects exist at all levels of confounders in the population under study, commonly known as the positivity assumption (Cole and Hernán 2008). In terms of the PS, this would mean that there is sufficient overlap of the PS distributions between treated and untreated subjects. For example, in case of PS matching (where treated and untreated subjects with the same PS are matched), this requirement is definitely met. The absence of sufficient overlap of PS between treatment groups (i.e., violation of the positivity assumption) can increase both the variance and bias of causal effect estimates (Petersen et al. 2010).Therefore, processing data, for example, using PS matching to generate two group of subjects with the same PS receiving different treatment modalities assures validity of positivity, without requiring systematic differences in an unmeasured covariate that is unrelated to measured covariates.

Advantage of Propensity Score Methods

Brooks and Ohsfeldt further claimed that “the conceptual advantage of PS-based methods relative to standard regression appears to hinge on the assumption that balancing measured covariates between treated and untreated subjects leads to unmeasured covariate balance between treated and untreated subjects.” This thought is shared by several researchers applying PS methods (Shah et al. 2005). However, the primary goal of PS methods is to create a balance of measured covariates between treatment groups, and PS methods may not seem to be superior to multivariable regression methods with respect to adjustment for unmeasured covariates. Nevertheless, Rosenbaum argued that “PS matching can lead to a reduction in both sample variability and the estimated treatment effect#x0027;s sensitivity to potential omitted variables” (Rosenbaum 2005). Furthermore, confounding by variables unmeasured in the main study can be addressed using variables measured in another validation study via PS calibration (Stürmer et al. 2005).

It is worth mentioning the potential advantages of PS methods compared with conventional regression methods, although both methods give similar effect estimates, as demonstrated in most published empirical and simulation studies (Shah et al. 2005; Stürmer et al. 2005). PS methods provide an effective way of controlling for covariates in case of a limited number of events, thereby overcoming the “dimensionality problem,” where the introduction of a new balancing covariate in the regression model increases the minimum necessary number of observations (outcome events) in the sample (Cepeda et al. 2003; Glynn, Schneeweiss, and Stürmer 2006). This is particularly common in pharmacoepidemiology, where outcomes are often rare compared to a large number of covariates available for estimation of (adverse) drug effects (Glynn, Schneeweiss, and Stürmer 2006). Cepeda et al. (2003) proposed a helpful guideline on when to use PS methods to effectively improve estimation (fewer than eight outcomes per included covariate). PS methods in general and PS matching in particular could also help to reduce “the dependence of causal inference on hard-to-justify but commonly made statistical modeling assumptions” and allow for a simple, transparent analysis (Glynn, Schneeweiss, and Stürmer 2006; Ho et al. 2007). In addition, unlike most multivariable regression models, assumptions regarding model specifications, variable functional form, normality, and linear projections beyond the observed data are not required particularly when matching on PS (Glynn, Schneeweiss, and Stürmer 2006; Zanutto 2006).

The Treatment Effect Estimate (The Estimand)

When addressing the bias in the treatment effect estimate, Brooks and Ohsfeldt compared effect estimates from different approaches, including regression and several PS methods. Furthermore, they concluded that PS methods can exacerbate the bias in treatment effect estimates when there are unmeasured confounders related to measured confounders. We agree that PS methods may not reduce bias from unmeasured confounders, but we would like to point out a potential drawback of direct comparison of estimates from PS methods and regression analysis where covariates are included in the adjustment model.

In studies in which non-linear models (such as logistic regression or Cox proportional hazard model) are applied, effect estimates may differ between PS methods and regression adjustment methods not only due to differential adjustment for confounding but also due to noncollapsibility (Greenland, Robins, and Pearl 1999). Noncollapsibility is the phenomenon that when estimating the treatment-outcome association using an odds ratio (OR) or hazard ratio (HR), the conditional OR or HR does not equal the marginal OR or HR in the presence of non-null treatment effect, even in the absence of confounding and effect modification (Greenland, Robins, and Pearl 1999; Austin 2007; Martens et al. 2008). Adjusting for covariates that are predictors of the outcome will change the treatment effect estimate, even when there is no confounding present. Obviously, the number of covariates included in the adjustment model can be very different when comparing PS matching with regression analysis where covariates are included in the adjustment model and thus a direct comparison may be flawed.

In addition, while both PS and regression methods provide different effect estimates, the inferential goal of the research question determines which estimand is appropriate. For example, in the absence of noncollapsibility (e.g., with linear models), marginal structural modeling using inverse probability of treatment weighing, multivariable regression, covariate adjustment using PS, and PS stratification estimate the average treatment effect in the population, ATE, the treatment effect estimate obtained from RCTs (i.e., the average causal treatment effect if everyone in the population is treated vs if everyone in the population is untreated) (Robins, Hernán, and Brumback 2000; Hernán, Brumback, and Robins 2000; Fang, Brooks, and Chrischilles 2012). On the other hand, PS matching typically focuses on either the average treatment effect in the treated or the average treatment effect in the untreated, not on the ATE; hence, the target of the causal contrast being the population that is going to receive the treatment (Hill 2008; Rassen et al. 2012). This is particularly important when there is treatment-effect modification regardless of the presence of confounding (Fang, Brooks, and Chrischilles 2012; Stürmer et al. 2006). In the study by Brooks and Ohsfeldt, exacerbation in bias was not clearly evident when using PS to balance measured covariates compared with regression estimates even in the presence of independent variation in the unmeasured covariate (example, Scenarios 1 and 2 of Table 3) despite inappropriate comparisons of treatment effect estimates from different models.

Concluding Remarks

The authors should be commended for raising an important topic in PS methodology, namely the implications of balancing measured covariates using PS on the (im)balance of unmeasured covariates. However, we disagree with Brooks and Ohsfeldt#x0027;s statement that systematic differences in the unmeasured covariates is required for PS algorithms to balance measured covariates between treated and untreated subjects. In addition, the impact of unmeasured covariate imbalance on the bias of estimated treatment effects, if any, cannot be inferred from their study. This is due to the fact that the estimands are different and direct comparison may not be appropriate. Therefore, the findings of Brooks and Ohsfeldt should be interpreted with caution and further research is needed to evaluate the balancing properties of PS with respect to unmeasured confounding using properly designed simulation settings.

Supporting Information

Additional supporting information may be found in the online version of this article:

Appendix SA1: Author Matrix.