ABSTRACT
Propensity score methods are frequently used to estimate treatment effects in observational studies. The propensity score is usually estimated using a logistic regression model with additive and linear effects. The limited literature on variable selection for the propensity score model is consistent in its suggestion that the propensity score model should include those variables that are prognostically important for the study outcome. However, the existing literature has focused only on main effects and assumed linear relations between continuous variables and outcomes. We used Monte Carlo simulations to study the impact of omitting quadratic terms and interaction terms from the propensity score model when these terms were prognostically important. We compared two formulations of the propensity score model: a simple formulation that included only main effects and omitted all interactions and quadratic terms and a complex formulation that included main effects in addition to prognostically important interactions and quadratic terms. We examined 168 scenarios defined by the nature of the true treatment‐selection model, the nature of the true outcome regression model, and by sample size. In 95% of scenarios, bias in estimated risk differences, along with coverage rates of estimated 95% confidence intervals, did not differ between the two specifications of the propensity score model when using inverse probability of treatment weighting. In only 5% of scenarios was bias substantially higher and coverage rates substantially lower when using the simple specification of the propensity score model compared to when using the complex specification of the propensity score model.
Keywords: inverse probability of treatment weighting, Monte Carlo simulations, propensity score, variable selection
1. Introduction
Observational studies are increasingly being used to estimate the effects of treatments, interventions, and exposures. Confounding occurs when the distribution of covariates that are associated with the study outcome differs between treated and control subjects. Statistical methods are crucial in any observational study to reduce or minimize the effect of confounding due to measured baseline covariates. Methods based on the propensity score are increasingly being used to account for confounding due to measured baseline variables. The propensity score is the probability of being treated (vs. control) conditional on an individual's measured baseline covariates [1, 2, 3]. The primary property of the propensity score is that it is a balancing score: conditional on the propensity score, the distribution of measured baseline variables will be the same between treated and control subjects. There are four ways of using the propensity score: matching on the propensity score, stratification on the propensity score, covariate adjustment using the propensity score, and weighting using propensity score‐based weights.
In practice, the true propensity score is not known and must be estimated using statistical methods. The most common approach to estimate the propensity score is to use a logistic regression model to regress the binary treatment variable (treated vs. control) on a set of measured baseline covariates. Alternative approaches include the use of regression trees, bagged regression trees, random forests, boosted regression trees, neural networks, support vector machines [4, 5, 6, 7, 8], which allow for non‐linear and non‐additive effects in the propensity score model.
Rubin highlighted the importance of separating the design of an observational study from the analysis of an observational study [9]. A critical component in designing observational studies is identifying those variables for which one needs to account to control for confounding. A related issue is identifying those variables that need to be included in the propensity score model. However, there is limited research on variable selection for the propensity score model. Rubin and Thomas suggested that variables that are unrelated to the study outcome be excluded from the propensity score model [10]. Using simulations, Brookhart and colleagues considered different variable selection strategies for the propensity score model [11]. They considered settings with three baseline variables: one variable related only to treatment‐selection, one variable related only to the outcome, and one variable related to both treatment‐selection and the outcome. They found that when using covariate adjustment using the propensity score and the outcome model was a Poisson regression model, including either the last two variables or only the last variable resulted in estimated treatment effects with the lowest bias. However, of these two variable selection strategies, including the last two variables resulted in estimated treatment effects that displayed lower variance and mean squared error (MSE). Their recommendation was that the optimal variable selection strategy was to include all variables that are related to the outcome, regardless of their association with treatment‐selection. This recommendation is consistent with that of Rubin and Thomas described above. Austin and colleagues examined variable selection for the propensity score model [12]. In the context of matching on the propensity score, they recommended that the propensity score model either include all variables that are associated with the outcome or include those variables that are associated with both treatment‐selection and the outcome. Similarly, Stuart suggested that it is important to account for all variables that are related to both treatment selection and the outcome, and that including variables that are only related to treatment selection can result in increased variance of the estimated treatment effect [13]. VanderWeele highlights the importance of only including pre‐treatment variables and excluding those variables that only affect treatment selection [14].
Questions about variable selection for the propensity score model have typically been framed in terms of main effects. For example, if the study outcome is known to be related to an individual's age, then the above studies suggest that age should be included in the propensity score model. However, if the study outcome is quadratically related to an individual's age, then both age and its square are variables that are related to the outcome. Thus, both age and its square should be included in the propensity score model. Similarly, if the effect of age on the outcome varies between males and females, then the outcome model contains a variable denoting the interaction between age and sex. The effect of the omission of prognostically important non‐linear and interaction terms from a logistic regression propensity score model has not been thoroughly investigated.
The paper is structured as follows: In Section 2, we describe a series of Monte Carlo simulations that were designed to address specification of the propensity score model. In particular, we examined the impact of omitting quadratic terms and interaction terms from the propensity score model when these terms are present in the outcome regression model. In Section 3, we report the results of these simulations. Finally, in Section 4, we summarize our findings and place them in the context of the existing literature.
2. Monte Carlo Simulation Methods
We conducted a series of Monte Carlo simulations to examine the impact of omitting interactions between prognostically important covariates from the propensity score model and of omitting quadratic terms of prognostically important covariates from the propensity score model in settings in which these interactions and quadratic terms were associated with the outcome of interest. Our simulations were based on a design used by Setoguchi and colleagues in a study to assess the performance of regression trees and neural networks for estimating propensity scores [4]. A similar design was also used by Lee and colleagues to examine the performance of propensity score weighting when using machine learning methods to estimate the propensity score [5] and by Wyss and colleagues when comparing the covariate‐balancing propensity score (CBPS) with logistic regression and boosted regression trees for estimating the propensity score [15].
2.1. Factors in the Monte Carlo Simulations
We allowed three factors to vary in the Monte Carlo simulations: (i) the specification of the true propensity score model (i.e., of the treatment‐selection model); (ii) the specification of the true outcome model; (iii) the sample size. The first factor had seven levels (see below for description), the second had six levels (see below for description), while the third factor took on four values (N = 1000, 5000, 10,000, and 20,000). In each of the 168 scenarios we simulated 1000 datasets.
2.2. Simulating Baseline Variables for a Super‐Population
We used a data‐generating process identical to that of Setoguchi and colleagues to generate 10 baseline covariates for each individual in a super‐population of size 1000,000. Ten continuous covariates were generated from a multivariate normal distribution with mean zero and a variance–covariance matrix . As in Setoguchi, was a matrix whose diagonal components were set to one, and specific non‐diagonal components were set as follows: (this variance–covariance matrix is identical to that used by Setoguchi and colleagues).
Four of the baseline variables (W2, W4, W7, and W10) were retained as continuous. Six of the variables (W1, W3, W5, W6, W8, and W9) were transformed into binary variables by dichotomizing them at the population mean value (i.e., values that were less than zero were set equal to zero, while values that were equal to or greater than zero were set equal to one [in the original simulations by Setoguchi and colleagues, they did not explicitly state at what percentile the six variables were dichotomized]).
Of these 10 baseline covariates, four (W1, W2, W3, and W4) were associated with both treatment‐selection and the outcome, three (W5, W6, and W7) were associated with only treatment‐selection, and three (W8, W9, and W10) were associated with only the outcome. The directed acyclic graph (DAG) describing the relationship between these 10 covariates, treatment, and the outcome was provided by Wyss and colleagues (Figure 1 in the cited work) [15]. The treatment‐selection and outcome models will be defined in the following two sub‐sections.
FIGURE 1.

Distribution of the propensity score in treated and control subjects under each of the 7 propensity score models.
2.3. Generating Treatment Status Using Six Propensity Score Models
Let Z denote a binary treatment variable (Z = 0 for control vs. Z = 1 for treated). Setoguchi and colleagues considered seven different treatment‐selection models (i.e., propensity score models), which we reproduce here:
In the seven regression models above, the regression coefficients were set to 0, 0.8, −0.25, 0.6, −0.4, −0.8, −0.5, and 0.7, respectively (these regression parameters are identical to those used by Setoguchi and colleagues). Setoguchi and colleagues described these seven treatment‐selection models as a model with additivity and linearity (PSA), a model with mild non‐linearity (PSB), a model with moderate non‐linearity (PSC), a model with mild non‐additivity (PSD), a model with mild non‐additivity and non‐linearity (PSE), a model with moderate non‐additivity (PSF), and a model with moderate non‐additivity and non‐linearity (PSG). We retain these seven descriptors in the current study.
Using each treatment‐selection model, we generated a binary treatment status variable for each individual in the super‐population using a Bernoulli distribution with subject‐specific probability of treatment determined using the given treatment‐selection model. The distribution of the propensity score in treated and control subjects in the super‐population under each of the seven propensity score models is described in Figure 1. Propensity scores C and G displayed less overlap between treated and control subjects compared to the other five propensity score models.
2.4. Simulating a Binary Outcome for Each Individual in the Super‐Population
The study by Setoguchi and colleagues considered only a single outcome regression model: a logistic regression model with additivity and linearity (i.e., with no interactions and no non‐linear terms). We wanted to examine a variety of outcome regression models that incorporated non‐additivities and non‐linearities. In the current study, we considered six different outcome models, based on PSB through PSG (we do not consider an outcome model with additivity and linearity, as we wanted to examine the impact of ignoring existing non‐additivities and non‐linearities).
We created six outcome regression models OutcomeB through OutcomeG. Each outcome model was based on the corresponding treatment‐selection model (e.g., OutcomeB was based on PSB). In PSB through PSG, we replaced each occurrence of W5, W6, and W7 (the three variables that are associated with only treatment‐selection) with W8, W9, and W10 (the three variables that are associated with only the outcome), respectively. Furthermore, the regression coefficients were replaced with . The values of were set to −3.85, 0.3, −0.36, −0.73, −0.2, 0.71, −0.19, and 0.26, respectively (these are identical to the regression parameters in the outcome model used by Setoguchi and colleagues). In addition to these modifications to the above regression models, a term for the effect of treatment status was added to each linear predictor, where , as in Setoguchi and colleagues.
Thus, mirroring the treatment‐selection models, we had an outcome model with mild non‐linearity (OutcomeB), an outcome model with moderate non‐linearity (OutcomeC), an outcome model with mild non‐additivity (OutcomeD), an outcome model with mild non‐additivity and non‐linearity (OutcomeE), an outcome model with moderate non‐additivity (OutcomeF), and an outcome model with moderate non‐additivity and non‐linearity (OutcomeG).
Using each of the six outcome models, we generated the two potential outcomes under treatment and control for each individual in the super‐population. Using each pair of potential outcomes we determined the true average treatment effect (ATE) in the super‐population as the average difference in the two potential outcomes. Thus, for each of the six outcome models we determined the true ATE when the risk difference was the measure of treatment effect. In the current study we focused on binary outcomes due to the high frequency with which they are used in clinical research [16]. The true risk difference associated with the six outcome models was −0.007, −0.008, −0.010, −0.008, −0.012, −0.010, for OutcomeB through OutcomeG, respectively. Note that these are the true values of the ATE in the population, prior to any trimming or restriction of the population being applied.
As noted in the previous sub‐section, using each of the seven treatment‐selection models, we generated a treatment status using the appropriate logistic regression model. Then, for each combination of treatment‐selection model and outcome model we generated an outcome. Thus, there were 42 simulated outcomes (7 treatment‐selection models × 6 outcome models) for each individual: one for each combination of treatment‐selection model and outcome model.
In each of the 42 super‐populations defined by all the combinations of the seven treatment‐selection models and the six outcome models, we computed the absolute standardized difference for each of the prognostically important covariates (those main effects, non‐linear terms, and interactions associated with the outcome) [17]. In each of the 42 super‐populations, we computed the median, the 75th percentile, and the maximum absolute standardized difference. Across the 42 super‐populations, the medians of the absolute standardized differences ranged from 0.024 to 0.275. The 75th percentiles of the absolute standardized differences ranged from 0.226 to 0.392. The maximums of the absolute standardized differences ranged from 0.263 to 0.546. Thus, in all 42 super‐populations, there were prognostically important variables that were imbalanced between treated and control subjects.
2.5. Analyses in the Simulated Datasets
We drew a random sample of size N (as noted in Section 2.1, N = 1000, 5000, 10,000, and 20,000) from the super‐population. We used inverse probability of treatment weighting (IPTW) using the propensity score to estimate the risk difference twice in the random sample of size N. First, we estimated the propensity score using the seven variables that were either associated with both treatment‐selection and the outcome (W1, W2, W3, and W4) or that were associated with only the outcome (W8, W9, and W10) (i.e., we included all seven prognostically important variables). This propensity score model included only the main effects and no interactions and no non‐linear terms. Second, we estimated the propensity score using all the terms (both main effects, interactions, and non‐linear terms) that were in the outcome model that had been used to generate the outcomes in the data‐generating process. Using each of these two propensity score models, we computed stabilized inverse probability of treatment weights: , where Z denotes treatment status (Z = 1 for treated vs. Z = 0 for control) and e denotes the estimated propensity score. In some of the scenarios described by Setoguchi and colleagues, there was near violation of the positivity assumption (i.e., there were individuals in the super‐population whose true propensity score was very close to zero or one). To address this, we used propensity score trimming and the method described by Crump and colleagues to identify an optimal subsample in which the ATE can be estimated with the greatest precision [18]. We determined the optimal value of such that only those subjects whose propensity score was in the interval were included. We used weighted standardized differences to assess the balance between treated and control subjects on all main effects, interactions, and non‐linear terms that were in the true outcome model [17]. We then estimated the risk difference and its associated standard error using methods described by Zhou and colleagues [19, 20, 21]. We computed 95% confidence intervals for the estimated risk difference using standard normal‐theory methods. This process was repeated 1000 times for each of the 168 scenarios. In order to assess the effect of trimming on the effective sample size, computed the design effect for the weighted sample both before and after trimming. The design effect was computed as , where N is the size of the sample (either before or after trimming) and denotes the inverse probability of treatment weight [22]. The design effect denotes the reduction in effective sample size compared to an unweighted sample. Thus, a design an effect of 1.1 would indicate that, given a weighted sample of size N, the effective sample size would be N/1.1.
2.6. Performance Metrics
We used three performance metrics to assess the performance of the two different specifications of the propensity score: (i) bias; (ii) relative bias; (iii) empirical coverage rates of 95% confidence intervals. Let RD(true) denote the true risk difference determined in the super‐population (prior to any trimming or restriction being applied to the population), and let RD(i) denote the risk difference estimated in the ith random sample drawn from the super‐population (i = 1, 2, …, 1000). Bias is defined as: , while relative bias is defined . The empirical coverage rates of estimated 95% confidence intervals was computed as the proportion of estimated confidence intervals that contained the true risk difference.
2.7. Software
The simulations were conducted using the R statistical programming language (version 3.6.3) [23]. The propensity score weighted analyses were conducted using the PSweight function from the PSweight package (version 2.1.1) [20]. Simulation results were summarized using the simsum function from the rsimsurv package (version 0.13.0).
3. Monte Carlo Simulation Results
Simulation results are reported in Figures 2 through 9. Figure 2 uses nested loop plots to report the mean absolute weighted standardized difference across the 168 scenarios [24]. Figure 3 reports the mean of Crump's optimal trimming threshold () over the 1000 simulation replicates across the 168 scenarios. Figure 4 reports the mean design effect over the 1000 simulation replicates across the 168 scenarios. Figure 5 (bias), Figure 6 (relative bias), and Figure 7 (coverage rates of 95% confidence intervals) use nested loop plots to report the value of each performance metric for each of the two specifications of the propensity score model in each of the 168 scenarios. Each nested loop plot has two lines (except for Figure 4): a red line for when the propensity score model included only the seven main effects (and did not contain any interactions or non‐linear terms) and a turquoise line for when the propensity score model included all the terms in the outcomes regression model that was used to generate outcomes (Figure 4 has four lines, one for each combination of propensity score specification and whether trimming was used). At the top of each nested loop plot are three step functions in black. The top step function denotes the outer loop and represents the factor denoting the propensity score model. This step function has seven steps: one step for each of the seven treatment‐selection models. The second step function from the top denotes the middle loop and represents the factor denoting the outcome regression model. This step function repeats within each of the steps of the outer loop (thus this step function has 42 steps, one for each combination of treatment‐selection model and outcome model). Within each of the steps of the outer loop, there are seven steps, representing each of the seven outcome regression models. Finally, the third step function from the top denotes the inner loop and represents the sample size of each analytic sample. This step function repeats within each of the steps of the middle step function (thus the third step function has 168 steps). Within each of the steps of the middle loop, there are four steps, representing each of the four sample sizes. For a given scenario of the Monte Carlo simulations, one finds the three steps of the three different steps functions corresponding to the levels of the three factors for the given scenario. Then, the levels (on the vertical axis) of the red and turquoise lines below those three steps denote the value of the given metric for the two specifications of the propensity score for that specific scenario.
FIGURE 2.

Distribution of absolute standardized differences.
FIGURE 9.

Distribution of MCSE of performance metrics across simulation scenarios.
FIGURE 3.

Crump's optimal trimming threshold.
FIGURE 4.

Design effect for different propensity scores with and without trimming.
FIGURE 5.

Bias in estimated risk difference.
FIGURE 6.

Relative bias in estimated risk difference (%).
FIGURE 7.

Coverage of 95% confidence intervals.
For each of the performance metrics, the distribution of the performance metric is summarized across the 168 scenarios using side‐by‐side boxplots in Figure 8, with one boxplot for each of the specifications of the propensity score model. There is one panel for each of the three performance metrics. Finally, the lower right panel of Figure 8 reports the distribution of the mean value of (Crump's optimal threshold for propensity score trimming) across the 168 scenarios. In most scenarios, the mean value of was less than 0.125.
FIGURE 8.

Distribution of performance metrics across simulation scenarios.
In Figure 2, we report the median absolute standardized difference (top left panel), the 75th percentile of the absolute standardized differences (top right panel), and the maximum absolute standardized difference (lower panel). We report specific percentiles of the distribution of standardized differences because the number of variables whose balance was being assessed varied across scenarios, depending on the number of interactions and non‐linear terms in the true outcome model used to generate outcomes in the data‐generating process. In examining Figure 2, the primary observation was, in general, better balance was induced by using the complex specification of the propensity score model compared to when using the propensity score model that only included main effects. The difference (simple specification—complex specification) in the median absolute standardized differences ranged from 0 to 0.018 across the 168 scenarios, with a median of 0.001 (25th and 75th percentiles: 0 and 0.004). The difference in the 75th percentile of the absolute standardized differences ranged from 0 to 0.060, with a median of 0.008 (25th and 75th percentiles: 0 and 0.017). The difference in the maximum absolute standardized differences ranged from −0.006 to 0.201, with a median of 0.010 (25th and 75th percentiles: 0 and 0.084). Thus, in some of the scenarios there were meaningful differences in the balance in some of the prognostically important covariates.
The mean value of Crump's optimal trimming threshold (δ) across the 168 scenarios is reported in Figure 3. In some scenarios (e.g., PSA and OutcomeB), the two different specifications of the propensity score model resulted in essentially identical values of the optimal trimming threshold. In other scenarios (e.g., PSC and OutcomeF or OutcomeG), the use of the simple specification of the propensity score model resulted in a lower value of the trimming threshold compared to the correct specification of the propensity score model. In general, the use of the correctly‐specified propensity score model tended to result in a design effect that was closer to one compared to when the simple specification of the propensity score model was used (Figure 8, lower right panel).
The design effect for the four different methods of analysis (2 specifications of the propensity score model × with/without trimming) across the 168 scenarios are reported in Figure 4. In interpreting this figure, recall that a design effect closer to one indicates a larger effective sample size compared to a design effect further from one. In some scenarios (e.g., PSA and OutcomeB or OutcomeC), all four methods resulted in approximately equal design effects (and thus, approximately equal effective sample sizes). In other scenarios (e.g., PSG and OutcomeF and OutcomeG), trimming resulted in a lower design effect compared to not using trimming, regardless of which specification of the propensity score model was used.
Across 160 (95%) of the 168 scenarios, the specification of the propensity score model had minimal impact on the bias of the estimated risk difference, with the difference in the relative bias being less than 8% in these 160 scenarios. The exceptions were those eight (5%) scenarios when PSC (moderate non‐linearity) or PSG (moderate non‐additivity and non‐linearity) were used to generate treatment status in conjunction with when OutcomeF (moderate non‐additivity) was used to generate outcomes. In these eight scenarios, the difference in relative bias between the two specifications of the propensity score model ranged from 19% to 24%. The distribution of the Monte Carlo standard error across the 168 simulation scenarios is reported in the top two panels of Figure 9 [25]. When focusing on relative bias, most Monte Carlo standard errors were less than 2.5. In Figure 6, one can see that in 95% of scenarios, the difference in relative bias between the two specifications of the propensity score model was likely not meaningful, while it was likely meaningfully different in the remaining 5% of scenarios.
In 95% of the scenarios, the empirical coverage rates did not differ between the two specifications of the propensity score model. Those scenarios in which the empirical coverage rates differed between the two specifications of the propensity score model were the same scenarios for which bias and relative bias differed between the two specifications. The Monte Carlo standard errors for the empirical coverage rates were calculated using methods described by Morris and colleagues: , where N simulations = 1000, the number of iterations in the simulations [25]. Most of the Monte Carlo standard errors were less than 0.0100 (Figure 9, lower left panel). The magnitudes of the Monte Carlo standard errors confirm that the empirical coverage rates did not differ meaningfully between the two specifications of the propensity score in 95% of the scenarios.
Examining the difference in balance induced by the two different formulations of the propensity score model (Figure 2) provides insight into the likely reason for the relatively poor performance of the simple specification of the propensity score model in 5% of the scenarios. Those scenarios in which the relative bias when using the simple specification of the propensity score model was substantially higher than the relative bias when using the complex specification of the propensity score model (and substantially higher absolute bias and substantially lower empirical coverage rates of 95% confidence intervals) were those scenarios in which the maximum absolute weighted standardized difference when using the complex specification of the propensity score model was close to zero, whereas the maximum when using the simple specification of the propensity score model was approximately 0.20. It has been suggested that the standardized differences should be less than 0.10 [26, 27]. Thus, in these scenarios there was substantial imbalance in some of the prognostically important covariates when using the simple specification of the propensity score model, whereas there was excellent balance in all prognostically important covariates when using the complex specification of the propensity score model.
4. Discussion
We examined variable selection for the propensity score model. In particular, we studied the impact of omitting quadratic terms and interaction terms from the propensity score model when these terms were present in the outcome regression model. We compared two formulations of the propensity score model: a formulation that included only main effects and omitted all interactions and quadratic terms and a formulation that included main effects in addition to prognostically important interactions and quadratic terms. We examined 168 scenarios defined by the nature of the true treatment‐selection model, the nature of the true outcome regression model, and by sample size. In the large majority (95%) of scenarios, bias in estimated risk differences, along with empirical coverage rates of estimated 95% confidence intervals, did not differ between the two specifications of the propensity score model when using IPTW to estimate the average treatment effect. In the remaining 5% of scenarios, bias was substantially higher and empirical coverage rates were substantially lower when using the simple specification of the propensity score model compared to when using the complex specification of the propensity score model. The scenarios in which it was better to use the complex specification of the propensity score model were those scenarios in which the true treatment‐selection model had moderate non‐linearity or moderate non‐additivity combined with non‐linearity in conjunction with when the outcome model had moderate non‐additivity.
In applied empirical analyses, analysts will often not know the true nature of the outcome regression model. Based on our findings, we make the following recommendations when using logistic regression to estimate the propensity score. First, that analysts include in the propensity score model all variables that are prognostic of the study outcome. Second, that analysts allow for non‐linear relationships between continuous baseline variables and treatment status when fitting the propensity score model. One method for doing so is to use restricted cubic splines [28]. Third, that applied researchers use balance diagnostics to assess the balance of baseline covariates between treated and control subjects [17, 27]. Several sets of authors have suggested that analysts compare the distribution of interactions and higher order moments of continuous variables between treatment groups [17, 27, 29]. Thus, the assessment of balance should not be restricted to main effects but should include interactions and higher order moments of continuous variables. When assessing balance of interactions, we propose that the focus be on interactions that involve at least one variable that is known to be prognostically important for the outcome. Furthermore, we hypothesize that in most settings, it is sufficient to limit the focus on two‐way interactions, and that incorporating interactions between three or more variables is unlikely to have a meaningful marginal impact. Similarly, as suggested elsewhere, graphical methods can be used to compare the distribution of continuous variables between treated and control subjects. Fourth, following the example of Rosenbaum and Rubin, analysts should use an iterative procedure for specifying the propensity score model [30]. Analysts can assess balance after an initial specification of the propensity score model that includes only main effects for categorical variables and that allows for non‐linear relationships between continuous variables and the log‐odds of treatment. After balance assessment, the initial specification of the propensity score model can be modified by including those interactions that were imbalanced. This procedure can be performed iteratively until acceptable balance is achieved. Note that in doing so, the analyst remains blind to the study outcome and the effect of treatment on the study outcome is not examined until acceptable balance has been achieved.
The primary limitation of the current study is its reliance on Monte Carlo simulations. It is conceivable that different results would be obtained under different data‐generating processes. However, a strength of the current study is that our data‐generating processes were based on ones used by Setoguchi and colleagues [4] and that were subsequently used by Lee and colleagues [5]. In doing so, we considered seven different treatment‐selection models that incorporated differing degrees of non‐linearities and interactions (including one with only main effects). We also considered six different outcome models that also incorporated differing degrees of non‐linearities and interactions. In total, we considered 168 scenarios defined by the treatment‐selection model, the outcome model, and sample size. A secondary limitation of the current study is its restriction to the use of parametric logistic regression models for estimating the propensity score model. Alternative, non‐parametric models from the machine learning literature, such as random forests or generalized boosted models, can also be used to estimate the propensity score model. These models may be better able to capture interactions and non‐linear relationships. Note that when used for estimating the propensity score, these methods would only capture interactions and non‐linear relationships between covariates and treatment status, and would not allow the analyst to capture interactions and non‐linear relationships between covariates and the outcome.
The current study builds on prior work by Rubin and Thomas, by Brookhart and colleagues, and by Austin and colleagues [10, 11, 12]. These authors recommended that the propensity score model should include prognostically important covariates and exclude those that were related only to treatment. In the current study, we expanded the focus from main effects and linear relations to include interactions and non‐linear relations. We found that in some settings, restricting the propensity score model to main effects and linear relations resulted in estimates with increased bias compared to when a more complex propensity score model was used. To the best of our knowledge, only one study on the formulation of the propensity score model has considered settings in which there were prognostically important quadratic terms or prognostically important interactions in the outcome model [27]. This was a study focused on balance diagnostics for propensity score matching. In one set of simulations, the author examined a setting in which there was a single continuous variable that was quadratically related to both treatment‐selection and to the continuous outcome. The simulations showed that fitting a propensity score model with only a linear term for the continuous variable could induce balance on that continuous baseline variable, but not on the square of the continuous baseline variable. Similarly simply simulations with two baseline variables such that both baseline variables and their interaction were associated with both treatment‐selection and the continuous outcome showed that fitting a propensity score model that omitted the interaction could induce balance on the two continuous variables, but not on their interaction. The simulations in the current study are substantially more complex than those in this earlier paper and allow the degree of non‐linearity and non‐additivity to vary across scenarios.
In summary, when using logistic regression to estimate the propensity score model it is not sufficient to include only main effects of prognostically important variables and to allow only linear relations between continuous variables and the log‐odds of treatment. Investigators should allow for non‐linear relations between continuous variables and the log‐odds of treatment and consider the inclusion of prognostically important interactions.
Funding
ICES is an independent, non‐profit research institute funded by an annual grant from the Ontario Ministry of Health (MOH) and the Ministry of Long‐Term Care (MLTC). This study was supported by ICES, which is funded by an annual grant from the Ontario Ministry of Health (MOH) and the Ministry of Long‐Term Care (MLTC). This study also received funding from the Canadian Institutes of Health Research (CIHR) (PJT—183902). The analyses, conclusions, opinions, and statements expressed herein are solely those of the authors and do not reflect those of the funding or data sources; no endorsement is intended or should be inferred.
Ethics Statement
No data on individuals were used in this study. Under section 45 of Ontario's Personal Health Information Protection Act (PHIPA), this study did not require review by a Research Ethics Board.
Consent
The study did not use data on individuals and thus no patient consent is required for this study.
Conflicts of Interest
The authors declare no conflicts of interest.
Data Availability Statement
No empirical data were used in this study and thus no data are available for dissemination. The study consisted of Monte Carlo simulations in which data were generated using a described data‐generating process. The described methods allow the reader to replicate these simulated data.
References
- 1. Rosenbaum P. R. and Rubin D. B., “The Central Role of the Propensity Score in Observational Studies for Causal Effects,” Biometrika 70 (1983): 41–55. [Google Scholar]
- 2. Austin P. C., “A Tutorial and Case Study in Propensity Score Analysis: An Application to Estimating the Effect of In‐Hospital Smoking Cessation Counseling on Mortality,” Multivariate Behavioral Research 46 (2011): 119–151. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3. Austin P. C., “An Introduction to Propensity‐Score Methods for Reducing the Effects of Confounding in Observational Studies,” Multivariate Behavioral Research 46 (2011): 399–424. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4. Setoguchi S., Schneeweiss S., Brookhart M. A., Glynn R. J., and Cook E. F., “Evaluating Uses of Data Mining Techniques in Propensity Score Estimation: A Simulation Study,” Pharmacoepidemiology and Drug Safety 17, no. 6 (2008): 546–555. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5. Lee B. K., Lessler J., and Stuart E. A., “Improving Propensity Score Weighting Using Machine Learning,” Statistics in Medicine 29, no. 3 (2010): 337–346. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6. Luellen J. K., Shadish W. R., and Clark M. H., “Propensity Scores: An Introduction and Experimental Test,” Evaluation Review 29, no. 6 (2005): 530–558. [DOI] [PubMed] [Google Scholar]
- 7. McCaffrey D. F., Ridgeway G., and Morral A. R., “Propensity Score Estimation With Boosted Regression for Evaluating Causal Effects in Observational Studies,” Psychological Methods 9, no. 4 (2004): 403–425. [DOI] [PubMed] [Google Scholar]
- 8. Westreich D., Lessler J., and Funk M. J., “Propensity Score Estimation: Neural Networks, Support Vector Machines, Decision Trees (CART), and Meta‐Classifiers as Alternatives to Logistic Regression,” Journal of Clinical Epidemiology 63, no. 8 (2010): 826–833. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9. Rubin D. B., “The Design Versus the Analysis of Observational Studies for Causal Effects: Parallels With the Design of Randomized Trials,” Statistics in Medicine 26, no. 1 (2007): 20–36. [DOI] [PubMed] [Google Scholar]
- 10. Rubin D. B. and Thomas N., “Matching Using Estimated Propensity Scores: Relating Theory to Practice,” Biometrics 52, no. 1 (1996): 249–264. [PubMed] [Google Scholar]
- 11. Brookhart M. A., Schneeweiss S., Rothman K. J., Glynn R. J., Avorn J., and Sturmer T., “Variable Selection for Propensity Score Models,” American Journal of Epidemiology 163, no. 12 (2006): 1149–1156. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12. Austin P. C., Grootendorst P., and Anderson G. M., “A Comparison of the Ability of Different Propensity Score Models to Balance Measured Variables Between Treated and Untreated Subjects: A Monte Carlo Study,” Statistics in Medicine 26, no. 4 (2007): 734–753. [DOI] [PubMed] [Google Scholar]
- 13. Stuart E. A., “Matching Methods for Causal Inference: A Review and a Look Forward,” Statistical Science 25, no. 1 (2010): 1–21. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14. VanderWeele T. J., “Principles of Confounder Selection,” European Journal of Epidemiology 34, no. 3 (2019): 211–219. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15. Wyss R., Ellis A. R., Brookhart M. A., et al., “The Role of Prediction Modeling in Propensity Score Estimation: An Evaluation of Logistic Regression, bCART, and the Covariate‐Balancing Propensity Score,” AmJEpidemiol 180, no. 6 (2014): 645–655. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16. Austin P. C., Manca A., Zwarenstein M., Juurlink D. N., and Stanbrook M. B., “A Substantial and Confusing Variation Exists in Handling of Baseline Covariates in Randomized Controlled Trials: A Review of Trials Published in Leading Medical Journals,” Journal of Clinical Epidemiology 63, no. 2 (2010): 142–153. [DOI] [PubMed] [Google Scholar]
- 17. Austin P. C. and Stuart E. A., “Moving Towards Best Practice When Using Inverse Probability of Treatment Weighting (IPTW) Using the Propensity Score to Estimate Causal Treatment Effects in Observational Studies,” Statistics in Medicine 34 (2015): 3661–3679. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18. Crump R. K., Hotz V. J., Imbens G. W., and Mitnik O. A., “Dealing With Limited Overlap in Estimation of Average Treatment Effects,” Biometrika 96, no. 1 (2009): 187–199. [Google Scholar]
- 19. Zhou T., Tong G., Li F., Thomas L. E., and Li F., “PSweight: An R Package for Propensity Score Weighting Analysis. arXiv:2010.08893v4 [stat.ME],” 2021.
- 20. Zhou T., Tong G., Li F., Thomas L. E., and Li F., “PSweight: An R Package for Propensity Score Weighting Analysis,” R Journal 14, no. 1 (2022): 282–299. [Google Scholar]
- 21. Zhou Y., Matsouaka R. A., and Thomas L., “Propensity Score Weighting Under Limited Overlap and Model Misspecification,” Statistical Methods in Medical Research 29, no. 12 (2020): 3721–3756. [DOI] [PubMed] [Google Scholar]
- 22. Shook‐Sa B. E. and Hudgens M. G., “Power and Sample Size for Observational Studies of Point Exposure Effects,” Biometrics 78, no. 1 (2022): 388–398. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23. Team RCD , “R: a language and environment for statistical computing [computer program]. Vienna: R Foundation for Statistical Computing,” 2005.
- 24. Rucker G. and Schwarzer G., “Presenting Simulation Results in a Nested Loop Plot,” BMC Medical Research Methodology 14 (2014): 129. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25. Morris T. P., White I. R., and Crowther M. J., “Using Simulation Studies to Evaluate Statistical Methods,” Statistics in Medicine 38, no. 11 (2019): 2074–2102. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26. Mamdani M., Sykora K., Li P., et al., “Reader's Guide to Critical Appraisal of Cohort Studies: 2. Assessing Potential for Confounding,” British Medical Journal 330, no. 7497 (2005): 960–962. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27. Austin P. C., “Balance Diagnostics for Comparing the Distribution of Baseline Covariates Between Treatment Groups in Propensity‐Score Matched Samples,” Statistics in Medicine 28, no. 25 (2009): 3083–3107. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28. F. E. Harrell, Jr. , Regression modeling strategies, Second ed. (Springer‐Verlag, 2015). [Google Scholar]
- 29. Ho D. E., Imai K., King G., and Stuart E. A., “Matching as Nonparametric Preprocessing for Reducing Model Dependence in Parametric Causal Inference,” Political Analysis 15 (2007): 199–236. [Google Scholar]
- 30. Rosenbaum P. R. and Rubin D. B., “Reducing Bias in Observational Studies Using Subclassification on the Propensity Score,” Journal of the American Statistical Association 79 (1984): 516–524. [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Data Availability Statement
No empirical data were used in this study and thus no data are available for dissemination. The study consisted of Monte Carlo simulations in which data were generated using a described data‐generating process. The described methods allow the reader to replicate these simulated data.
