Flexible propensity score estimation strategies for clustered data in observational studies

Abstract

Existing studies have suggested superior performance of nonparametric machine learning over logistic regression for propensity score estimation. However, it is unclear whether the advantages of nonparametric propensity score modeling carry to settings where there is clustering of individuals, especially when there is unmeasured cluster-level confounding. In this work we examined the performance of logistic regression (all main effects), Bayesian additive regression trees and generalized boosted modeling for propensity score weighting in clustered settings, with the clustering being accounted for by including either cluster indicators or random intercepts. We simulated data for three hypothetical observational studies of varying sample and cluster sizes. Confounders were generated at both levels, including a cluster-level confounder that is unobserved in the analyses. A binary treatment and a continuous outcome were generated based on seven scenarios with varying relationships between the treatment and confounders (linear and additive, non-linear/non-additive, non-additive with the unobserved cluster-level confounder). Results suggest that when the sample and cluster sizes are large, nonparametric propensity score estimation may provide better covariate balance, bias reduction, and 95% confidence interval coverage, regardless of the degree of non-linearity or non-additivity in the true propensity score model. When the sample or cluster sizes are small, however, nonparametric approaches may become more vulnerable to unmeasured cluster-level confounding and thus may not be a better alternative to multilevel logistic regression. We applied the methods to the National Longitudinal Study of Adolescent to Adult Health data, estimating the effect of team sports participation during adolescence on adulthood depressive symptoms.

1. INTRODUCTION

Propensity score methods are widely used to evaluate the causal effects of treatments in observational studies. The propensity score, which is the probability of receiving a treatment conditional on a set of covariates,¹ is especially useful when there is a large number of confounders that need to be adjusted for. Conditional on the propensity score, the distribution of the covariates entered in the propensity score model is expected to be similar between treatment groups.¹ Thus, the propensity scores can be used to reduce bias in the treatment effect estimate that arises from differences in the distribution of observed confounders between treatment groups. This bias reduction can be achieved using multiple strategies, including matching subjects on propensity scores, grouping subjects into strata with similar propensity scores, adjusting for propensity scores in the outcome model, or applying propensity score weights (for more details see, e.g., D’Agostino²; Hirano and Imbens³).

Despite the increasing use of propensity score methods in substantive studies over the past two decades,⁴ work on this topic in the context of clustered or multilevel data structures has been relatively limited. However, clustered data is common among many disciplines, especially in medical, behavioral, and educational research settings (e.g., students nested within schools, patients nested within hospitals). Here we consider the simplest case where the data is structured in two levels (individual-level and cluster-level) and treatment is administered at the individual level. The clustered structure adds complexity in conducting propensity score analysis. For example, there may be concerns regarding interference among individuals within clusters as well as possible heterogeneity in treatment effect or treatment implementation across clusters. Moreover, it can be challenging to identify and measure confounders at the cluster level, such as when using survey data that do not record cluster-level covariates. Unmeasured cluster-level confounding would create bias in the treatment effect estimate if unaccounted for. Even when all cluster-level confounders are measured, if the number of clusters is small, there may not be sufficient overlap on these confounders and residual confounding may remain. Treatment effect estimates obtained without consideration of these issues can be misleading, and propensity score methods should be adapted for specific contexts. For the purpose of our study, we will focus exclusively on the issue of unmeasured cluster-level confounding while assuming no unmeasured individual-level confounding, as well as the stable unit treatment value assumption (SUTVA),⁵ which assumes a single version of each treatment level (i.e., no heterogeneity in treatment implementation across clusters) and no interference between individuals, including those that belong to the same cluster. Although SUTVA is often questionable in practical settings with clustered observations, we will generate data based on SUTVA in order to focus on the robustness of propensity score models to unmeasured cluster-level confounding.

When the treatment administered to individuals is binary, to alleviate the concern for unmeasured cluster-level confounding, a logistic regression model with either fixed or random cluster intercepts is often used to estimate propensity scores with two-level clustered data. These models account for unobserved cluster heterogeneity – and thus potential cluster-level unobserved confounding – by allowing the intercept to vary by cluster.⁶ In fixed effects models, the intercept is a fixed constant for each cluster; thus, the number of parameters increases considerably when there are many clusters, and the estimation of parameters may fail if cluster sizes are too small. Random effects models typically assume that the cluster-specific intercept follows a normal distribution. However, unlike fixed cluster intercepts, random intercepts are shrunken toward the mean and thus may not fully capture the remaining effects of unobserved cluster-level covariates.⁷ Random effects models also require independence between the observed covariates and the random intercept.⁸ Both models acknowledge the possibility of unmeasured cluster-level confounding and assume no confounding within each cluster. Existing research has shown that propensity score models that account for the clustering – either through fixed or random cluster intercepts – led to less biased estimates compared to models that do not.^7,9,10

When there are many potential confounders, specification of the propensity score model can become extremely complicated, especially when there is potential interactions between confounders. To allow more flexibility in the propensity score model, Leite et al¹¹ suggested adopting the parsimony principle in building random effects models (i.e., adding random slopes and cross-level interactions step by step until sufficient covariate balance is attained). This approach, though reasonable, is inefficient and still requires a certain level of knowledge of the functional form for the relationship between treatment assignment and covariates. Nonparametric machine learning methods are one promising solution to overcoming model specification challenges due to their general ability to generate flexible models without model specification. Further, there has been evidence that nonparametric propensity score estimation achieves more efficient estimation of the average treatment effect, even when the true propensity score model is known to be parametric.¹² This finding is likely related to the results that show a preference to use estimated propensity scores over known treatment assignment probabilities to adjust for chance imbalances.¹³ Hence, nonparametric methods have gained popularity in propensity score estimation with single-level data – a specific commonly used approach is generalized boosted modeling, which can be used to generate propensity score weights that eliminate most covariate imbalances.¹⁴

Some work has been done to compare parametric versus nonparametric approaches for estimating propensity scores in single-level settings. Setoguchi et al¹⁵ compared machine learning techniques such as recursive partitioning and neural networks to logistic regression with only main effects with respect to propensity score matching. Their simulation study found that neural networks generally yielded the least biased estimates under various scenarios differing by non-linear and/or non-additive relationships between treatment assignment and covariates. Following Setoguchi et al,¹⁵ Lee et al¹⁶ examined the performance of propensity score models based on classification and regression trees (CART) with respect to propensity score weighting. Their simulation results supported that of Setoguchi et al,¹⁵ showing that estimating propensity scores using nonparametric methods, especially boosted regression trees, may offer advantages in propensity score weighting when the relationship between treatment assignment and covariates is non-linear and/or non-additive (and therefore the logistic regression model with main effects only is misspecified). These improvements include better bias reduction and more consistent 95% confidence interval coverage.¹⁶ The simulation designs of Setoguchi et al¹⁵ and Lee et al,¹⁶ however, assume a single-level data structure and no unmeasured confounding.

Motivated by the limited research on nonparametric propensity score estimation with clustered data, and in particular how unobserved cluster-level confounders may influence the methods’ performance, our goal is to examine whether the advantages of the flexible modeling of propensity scores extend to clustered data settings. In this work, we conduct simulations to examine the performance of nonparametric and parametric propensity score models, when used to achieve covariate balance between treatment groups across clusters via weighting. Focus is on estimation of the overall average treatment effect (not cluster-specific effects) in a two-level clustered data context where a binary treatment is administered at the individual level. The paper is organized as follows: Section 2 provides a brief introduction of the statistical methods, including inverse probability of treatment weighting and the nonparametric methods that are used to estimate propensity scores in this work. Section 3 describes the simulation setup and the performance measures for evaluating the performances of different propensity score models. Section 4 presents the simulation results. In Section 5, we apply the methods to data from the National Longitudinal Study of Adolescent to Adult Health (Add Health),¹⁷ estimating the effect of team sports participation during adolescence on depressive symptoms in adulthood. Finally, Section 6 discusses the implications and limitations of our work, as well as potential directions for future research.

2. STATISTICAL METHODS

2.1. Inverse probability of treatment weighting

We first review the basics of treatment effect estimation using propensity score weights. Our definition of the treatment effect is based on the potential outcomes framework, including the SUTVA assumption mentioned in the introduction.^18,19 The SUTVA assumption has two components: 1) an individual’s outcome is unaffected by the level of treatment assigned to another individual; 2) there is only one version of each treatment level. Under this assumption, each individual, indexed by subscript $k$, has two potential outcomes associated with a binary treatment: $Y_k\left(1\right)$ (potential outcome under treatment) and $Y_k\left(0\right)$ (potential outcome under the control condition). The individual treatment effect is defined as the difference between the two potential outcomes, $Y_k\left(1\right)-Y_k\left(0\right)$. Our target estimand is the average treatment effect (ATE) in the population, which is defined as the expected value of the individual treatment effects, $\mathrm{A}\mathrm{T}\mathrm{E}=E\left[Y_k\left(1\right)-Y_k\left(0\right)\right]$.

We use propensity score weighting to obtain covariate balance between the treated and control groups in the full sample, thereby reducing bias in the overall ATE estimate. Specifically, given the model-estimated propensity score for individual $k,{\widehat{e}}_k$ (we add the hat symbol for the estimated propensity score to differentiate it from the true propensity score, $e_k$), the inverse probability weight ${{\widehat{w}}_k=1/\widehat{e}}_k$ is assigned if the individual is in the treated group and ${{\widehat{w}}_k=1/(1-\widehat{e}}_k)$ otherwise. To prevent extreme weights, we use stabilized weights, which replace 1 in the numerator with the marginal probability of being treated $P(Z=1)$ for treated individuals (${{\widehat{w}}_k=P(Z=1)/\widehat{e}}_k$) and the mariginal probability of being in the control group $P(Z=0)$ for control individuals (${{\widehat{w}}_k=P(Z=0)/(1-\widehat{e}}_k)$). The ATE can then be estimated by contrasting the weighted means of the outcome between the two treatment groups:

$$\widehat{ATE}=\frac{\sum _k{Z}_k{Y}_k{\widehat{w}}_k}{\sum _k{Z}_k{\widehat{w}}_k}-\frac{\sum _k(1-Z_k)Y_k{\widehat{w}}_k}{\sum _k(1-Z_k){\widehat{w}}_k},$$

where $Z_k$ and $Y_k$ denote the assigned treatment level (1 if treated and 0 otherwise) and the observed outcome under the assigned treatment, respectively. Because the weights are a direct function of the propensity scores, this inverse probability weighted (IPW) estimator is particularly sensitive to misspecifications of the propensity score model.²⁰

A common metric of covariate balance is the absolute standardized mean difference (ASMD):

$$\text{ASMD}=\left|\frac{{\stackrel{-}{X}}_1-{\stackrel{-}{X}}_0}{s}\right|,$$

where ${\overline{X}}_1$ and ${\overline{X}}_0$ are the weighted sample means of a covariate $X$ (or prevalence if $X$ is binary) for the treated and control groups, respectively; $s$ denotes its standard deviation (SD; usually the pooled SD from the treated and control groups combined). A lower ASMD indicates better covariate balance, and for a covariate to be adequately balanced, a ASMD less than or equal to 0.1 is generally considered acceptable.^21–23 We assess the usefulness of a propensity score model by calculating the ASMD of each covariate after the model-estimated propensity score weights are applied.

To estimate the standard error of the IPW estimator, one may either use a robust (or “sandwich”) estimator or perform bootstrapping. The need for a robust standard error estimator is to account for the within-subject correlation in replications of units caused by the application of propensity score weights.²⁴ In clustered data settings, cluster robust standard errors should be used to account for within-cluster correlation induced by the clustered structure. We note that robust standard errors do not account for the uncertainty in estimating the propensity scores, and some have recommended using bootstrap standard errors for IPW estimators.^7,25 Because bootstrapping would be computationally demanding in our simulations (especially with machine learning involved), we will consider the cluster robust standard error as an approximate for the variability of treatment effect estimates.

The outcome analysis approach described above accounts for the clustered nature of the data in standard error estimation through a sandwich estimator but does not include cluster indicators or other covariates directly. In practice, a “doubly robust” estimator that incorporates the covariates and the clustered structure in both the propensity score and outcome models is recommended (see Li et al⁷ for a doubly robust IPW estimator). Because the goal of our simulations is to compare different strategies for estimating propensity scores, we retain focus on the simple IPW estimator, which does not use a doubly robust outcome model, in order to isolate the performance with respect to propensity score estimation.

2.2. Propensity score estimation using nonparametric methods

For nonparametric estimation of the propensity scores, we introduce two approaches: generalized boosted modeling (GBM) and Bayesian additive regression trees (BART). GBM is a popular method for estimating propensity scores, in part because its covariate balancing ability has been studied extensively and computing tools have been developed in this regard. BART is easy to implement and has great predictive ability, but its use in propensity score estimation is less explored. The mathematical detail of these methods is outside the scope of this paper; hence we provide only a brief introduction to these two methods below. For both, we describe how they can be adapted to the clustered data setting.

Both methods have decision trees underlying the approach. A decision tree is a nonparametric way of partitioning the covariate space into disjoint sets such that each set, which corresponds to a node in the tree, is as similar as possible.²⁶ When the outcome is a class (e.g., treated or untreated), a decision tree is often referred to as a classification tree, and observations falling in the same node of the tree have similar probabilities of class membership. An ensemble method, such as GBM, fits a series of decision trees to a random subset of the data, and it makes a prediction by averaging the predictions of the different trees. The idea of an ensemble method is to combine the predictions of multiple weak classifiers (i.e., trees), each constrained by a shrinkage parameter to prevent overfitting, to improve prediction accuracy. GBM is an ensemble method for which, in each iteration of tree fitting, observations that were incorrectly classified by previous trees are given a higher weight to be selected in the new tree.²⁷ Propensity score estimation using GBM was first proposed by McCaffrey et al¹⁴ and is commonly implemented with the R package twang, which uses an algorithm aimed at achieving optimal covariate balance (e.g., minimizing the mean of the Kolmogorov-Smirnov test statistics).²⁸

Similar to GBM, BART is also a nonparametric ensemble model, introduced by Chipman et al.²⁹ As a Bayesian approach, BART incorporates regularization priors for the model’s residual standard deviation, the tree structure (including tree depth and splitting rules), and the values in the terminal nodes conditional on the corresponding tree. Sampling from the posterior is done by a Bayesian backfitting Markov Chain Monte Carlo approach^29,30; the predicted value can be taken as the average of predictions over many draws from the posterior. The Bayesian framework spares the computational effort of cross-validation in determining model hyperparameters such as maximum tree depth and shrinkage parameter, which is commonly done with non-Bayesian ensemble methods. Although BART was developed for continuous outcomes, it can easily be extended for classification of binary outcomes by the probit or logit transformation and thus can be used to estimate propensity scores (see, e.g., Hill et al³¹; Dorie et al³²). Normally, the estimated propensity score is the average of the values over a default number of posterior draws set by the specific statistical package. Chipman et al²⁹ have demonstrated that BART outperforms several popular machine learning techniques, including GBM, random forest, and neural networks, in terms of in- and out-of-sample predictive performance. In our analysis, though, we are more interested in BART’s covariate balancing (rather than predictive) ability, which is the aim of propensity score methods.

To account for the clustered structure in our nonparametric propensity score models, indicators for cluster membership can be included in the GBM and BART, which is analogous to fitting a parametric regression model with fixed cluster effects, although not all cluster indicators may be used by the nonparametric models. An appealing feature of BART is that it allows random intercepts to be easily added to the model and can be implemented with available statistical software,^29,33 whereas GBM with random effects has not been fully developed. We therefore consider BART with additive random intercepts as a nonparametric counterpart of the logistic regression model with random cluster intercepts.

Standard errors of the IPW estimator with weights estimated from GBM and BART can be obtained via nonparametric bootstrapping, although further research is needed to confirm its validity in the context of propensity score estimation.³⁴ As mentioned in section 2.1, due to the computational burden required to perform bootstrapping with GBM and BART, we estimate standard errors using a cluster adjustment instead, and compare them with empirical standard errors.

3. SIMULATION STUDY

3.1. Setup

Our simulation setup is motivated by that in Setoguchi et al¹⁵ and Lee et al,¹⁶ with extensions to a two-level clustered data structure where a binary treatment is administered at the individual level. Given this two-level structure, we use $h$ to index clusters ($h=\mathrm{1,2},...,H$, where $H$ is the number of clusters in the simulated data set) and $k$ to index individuals within a cluster ($k=\mathrm{1,2},...,n_h$, where $n_h$ is the number of individuals in cluster $h$). The sample size for a given simulated data set is denoted as $N=\sum _{h=1}^H{n}_h$. We consider three cases: 1) a small number of large clusters ($H=20,200\le n_h\le 500$ for $h=\mathrm{1,2},...,20$); 2) a large number of small clusters ($H=100,n_h=50$ for $h=\mathrm{1,2},...,100$); 3) a small number of medium-sized clusters ($H=20,n_h=100$ for $h=\mathrm{1,2},...,20$).

For each simulated data set under each case, six individual-level confounders ($X_i,i=\mathrm{1,2},...,6$), two cluster-level confounders ($V_j,j=\mathrm{1,2}$), and an unmeasured cluster-level confounder ($U$; the confounder is unmeasured in the sense that it is excluded from both the propensity score model and outcome analyses) are independently generated from a standard normal distribution for each individual. Four of the confounders ($X_4,X_5,X_6,V_2$) are subsequently dichotomized with value 1 if the original value is greater than or equal to 0, and 0 otherwise.

The treatment probability $e_{hk}$ i.e., the true propensity score, for individual $k$ in cluster $h$ is generated from the following random effects logistic regression model, which is a function of individual- and cluster-level covariates, including $U$:

$$logit\left(e_{hk}^{\mathrm{*}}\right)=f\left(X_{1,hk},X_{2,hk},\dots ,X_{6,hk},V_{1,h},V_{2,h},U_h\right)+{\beta }_{0,h},$$

$${\beta }_{0,h}~N\left(\mathrm{0,1}\right)$$

with further adjustment $e_{hk}=0.7e_{hk}^{*}+0.15$ to ensure that each cluster has an adequate number of individuals assigned to each treatment level. The specification of the function in the propensity score generating model varies across scenarios that are described below and further detailed in the supplementary material. The treatment assignment $Z_{hk}$ is randomly sampled from a Bernoulli distribution with probability $e_{hk}$; we denote $Z_{hk}=1$ as being assigned to the treated group and $Z_{hk}=0$ as being assigned to the control group.

The continuous outcome $Y_{hk}$ is generated from the following random effects linear regression model (the coefficients are provided in the supplementary material):

$$\begin{gathered} Y_{hk}={\alpha }_{0,h}+{\alpha }_1{X}_{1,hk}+{\alpha }_2{X}_{2,hk}+\dots +{\alpha }_6{X}_{6,hk}+{\alpha }_7{V}_{1,h}+{\alpha }_8{V}_{2,h}+ \\ {\alpha }_9{U}_h+{\tau }_h{Z}_{hk}+\delta Z_{hk}{U}_h^2+{\epsilon }_{hk}, \\ {\alpha }_{0,h}~N\left(\mathrm{0,1}\right),{\tau }_h~N\left(\mathrm{0,1}\right),{\epsilon }_{hk}~N\left(\mathrm{0,0.1}\right) \end{gathered}$$

The interaction term between treatment assignment and the square of $U$ in the outcome model allows non-linear treatment effects in relation to $U$. We set $\delta =2$ and ${\alpha }_9=3$. The value for ${\alpha }_9$ is purposefully chosen to be relatively large to magnify the issue of unmeasured confounding.

Similar to the setup in Setoguchi et al¹⁵ and Lee et al,¹⁶ we consider seven propensity score generating models (scenarios A-G) that differ in degrees of non-linearity or non-additivity (details in the supplementary material). The functional form of the propensity score generating model in each of the seven scenarios has the following components:

• A: Main effects of $X_1,\dots ,X_6,V_1,V_2$ and $U$

• B: Main effects plus three two-way interaction terms between observed confounders ($X_1{X}_4,X_3{V}_2,X_5{V}_2$)

• C: Main effects plus six two-way interaction terms between observed confounders ($X_1{X}_4,X_3{V}_2,X_5{V}_2,X_2{X}_5,X_4{X}_6,X_6{V}_2$)

• D: Main effects plus three two-way interaction terms between $U$ and observed confounders ($X_{1}U,X_{4}U,X_{5}U$)

• E: Main effects plus six two-way interaction terms between $U$ and observed confounders ($X_{1}U,X_{2}U,X_{4}U,X_{5}U,X_{6}U,V_{2}U$)

• F: Main effects plus two cubic terms ($X_1^3,V_1^3$)

• G: Main effects plus four cubic terms ($X_1^3,X_2^3,X_3^3,V_1^3$)

In reality, the functional form of the propensity score generating model is unknown. The addition of scenarios D and E is to investigate the performance of propensity score models when $U$ serves as both an unmeasured confounder as well as a key source of variation in the propensity score. A logistic regression model assuming linear and additive associations between the treatment and confounders (i.e., including only main effects) is misspecified in scenarios B to G. Therefore, we expect the nonparametric propensity score models in general to produce less biased effect estimates compared to the main effects only logistic regression models at least in scenarios B, C, F, and G, in which the nonparametric models have more flexibility to detect non-linear or non-additive associations between the treatment and observed confounders. 1000 datasets are independently generated for each of the seven propensity score scenarios. All simulations are performed using R version 4.0.2.³⁵

3.2. Methods compared

Three general modeling approaches are used to estimate propensity scores: logistic regression, BART and GBM. With each approach, we consider versions that either ignore or incorporate cluster information (through either fixed or random cluster effects). All analyses do not have access to the cluster-level confounder $U$, which is unmeasured. The specific methods are:

• Logistic regression model (abbreviated as PARAM): single-level logistic regression with a main effect for each observed confounder.

• Logistic regression model with fixed cluster effects (PARAM-FE): logistic regression with a main effect for each observed confounder and a fixed intercept for each cluster.

• Logistic regression model with random cluster effects (PARAM-RE): logistic regression with a main effect for each observed confounder and random cluster intercepts.

• Probit BART ignoring clusters (BART): BART model with probit link is implemented using the pbart function in the R package BART with default settings.³⁶ Although the logit version of BART is also available in the BART package, we opt for probit BART due to its computational efficiency.

• Probit BART with cluster indicators (BART-FE): Same as above, except that indicator variables for clusters are used as predictors in addition to the observed confounders.

• Probit BART with random effects (BART-RE): BART model with probit link and additive random intercepts is implemented using the rbart function in the R package dbarts with default settings.³³

• GBM ignoring clusters (GBM): Propensity score estimation using GBM is implemented using the ps function in the R package twang with default settings.²⁸

• GBM with cluster indicators (GBM-FE): Same as above, except that indicator variables for clusters are added to the model.

For each method, we weight individuals by their estimated stabilized propensity score weights, and estimate the ATE using the marginal IPW estimator (as described in section 2.1).

3.3. Performance criteria

To evaluate the performance of the different propensity score estimation methods, we consider the following measures:

• Absolute standardized mean difference (ASMD): a measure of covariate balance. In each of the 1000 simulations, we obtain the post-weighting absolute standardized difference of means between the treated and control groups for each individual-level covariate using the R packages survey³⁷ to apply the estimated weights and tableone³⁸ to calculate the ASMD. The average ASMD is then taken across all six individual-level covariates. In the following sections, we refer to this average as “ASMD” for simplicity (i.e, “ASMD” refers to the average ASMD taken across covariates in the same category, e.g., individual-level covariates), and refer to the mean of the ASMDs over 1000 simulations as “mean ASMD”. Similarly, we do this for the observed cluster-level covariates and the unobserved cluster-level covariate. The ASMD prior to propensity score weighting is also calculated to assess the initial covariate balance.

• Bias: Both the difference between the estimated and true ATEs, $\widehat{ATE}-ATE$, and the absolute percentage difference from the true ATE, $\left|\frac{\widehat{ATE}-ATE}{ATE}\right|$, are calculated.

• Cluster robust standard error: The ATE estimate and cluster robust standard error are obtained using the survey package.³⁷

• 95% confidence interval coverage: In each simulation, the estimated 95% confidence interval is based on the cluster robust standard error. The 95% confidence interval coverage is the percentage of the 1000 estimated 95% confidence intervals that cover the true ATE.

• Weights: distribution of the estimated stabilized propensity score weights for control individuals, $\frac{P(Z=0)}{{1-\widehat{e}}_{hk}}$. Of particular interest is the proportion of extreme weights, which may result in bias and large variance.

4. RESULTS

Simulation results from case 1 ($\text{H}=20\text{and}200\le {\text{n}}_{\text{h}}\le 500$)

Table 1 shows the initial covariate balance in each propensity score scenario, with the mean ASMDs all falling between 0.2 to 0.8, indicating substantial imbalance before propensity score adjustment.

Table 1.

Pre-weighting absolute standardized mean difference (ASMD) averaged over 1000 simulations in case 1 ($H=20,200\le n_h\le 500$).

	Scenario†
	A	B	C	D	E	F	G
ASMD ($\mathit{X}$)*	0.29	0.31	0.32	0.26	0.23	0.28	0.34
ASMD ($\mathit{V}$)**	0.26	0.24	0.24	0.26	0.26	0.36	0.26
ASMD ($\mathit{U}$)***	0.44	0.39	0.35	0.55	0.72	0.29	0.20

^*Average ASMD of six individual-level covariates ($X_1,X_2,\dots ,X_6$).

^**Average ASMD of two observed cluster-level covariates ($V_1,V_2$).

^***ASMD of an unobserved cluster-level covariate ($U$).

^†In the true propensity score model: (A) main effects only; (B) three two-way interaction terms between observed confounders; (C) six two-way interaction terms between observed confounders; (D) three two-way interaction terms between $U$ and observed confounders; (E) six two-way interaction terms between $U$ and observed confounders; (F) two cubic terms; (G) four cubic terms.

Propensity score weighting using propensity scores estimated from nonparametric models generally produced excellent balance of the observed covariates ($\mathit{X}$ and $\mathit{V}$) across the seven scenarios (top and middle panels of Figure 1), though the covariate balancing performance (with respect to $\mathit{X}$ and $\mathit{V}$) of BART and BART-FE declined slightly in scenario G (where the true propensity score model included several cubic terms) with a number of ASMD values greater than 0.1 (Supplementary Figures 1 and 2). When propensity scores were estimated using parametric models, the mean ASMDs of the observed covariates following weighting were mostly acceptable (e.g., the mean ASMD of $\mathit{X}$ ranged from 0.05 in scenario E to 0.15 in scenario G for PARAM-FE; the mean ASMD of $\mathit{V}$ ranged from 0.03 in scenario E to 0.13 in scenario F for PARAM), but large outliers (ASMD≥0.15) were observed across most scenarios, especially in scenarios F and G (where the true propensity score model included cubic terms), with the performance of PARAM being particularly poor (Supplementary Figures 1 and 2).

FIGURE 1.

Post-weighting absolute standardized mean difference (ASMD) averaged over 1000 simulations by propensity score estimation model in each of seven propensity score scenarios (A: main effects only; B and C: interactions between observed covariates; D and E: interactions with an unobserved cluster-level covariate; F and G: cubic terms) in case 1 (20 clusters of size 200–500). ASMD (X) is the average ASMD of six individual-level covariates; ASMD (V) is the average ASMD of two observed cluster-level covariates, and ASMD (U) is the ASMD of an unobserved cluster-level covariate

As to the unmeasured cluster-level covariate ($U$), estimating propensity scores using BART-FE yielded the smallest mean ASMD in scenarios A to F, while GBM-FE yielded the smallest mean ASMD in scenario G (bottom panel of Figure 1). With the clustered structure ignored, GBM on average produced worse balance on $U$ than the other nonparametric models in all scenarios, and performed even worse than PARAM-FE and PARAM-RE in scenarios D and E, where the true propensity score model included interaction terms involving $U$ (Supplementary Figure 3). However, the mean and distribution of the ASMDs of $U$ for BART, which also ignored clustering, were similar to those for nonparametric models that accounted for the clustered structure (Supplementary Figure 3). As expected, PARAM failed to balance $U$, which remained substantially imbalanced with the mean ASMD of $U$ ranging from 0.14 (in scenario G) to 0.58 (in scenario E) across the seven scenarios.

In terms of the ATE estimates, the nonparametric propensity score models outperformed the parametric models with smaller mean absolute biases in all scenarios (top right of Figure 2). Among the nonparametric models, GBM yielded larger mean biases than GBM-FE and the BART-based models except in scenario G, where the covariate balancing performance of BART and BART-FE with respect to the observed covariates declined slightly. The parametric propensity score models performed unsatisfactorily with large absolute biases (>40%) across all seven scenarios. Although PARAM exhibited less bias on average than PARAM-FE and PARAM-RE in many scenarios (top left and top right of Figure 2), the spread of the biases (or absolute biases) over 1000 simulations for PARAM was larger and more extreme values were observed (Supplementary Figures 4 and 5).

FIGURE 2.

Results from case 1 (20 clusters of size 200–500). Bias (estimated—true average treatment effect; top left), absolute bias(%; top right), and cluster robust SE (bottom left) averaged over 1000 simulations by propensity score estimation model in each of seven propensity score scenarios (A: main effects only; B and C: interactions between observed covariates; D and E: interactions with an unobserved cluster-level covariate; F and G: cubic terms). Bottom right: 95% confidence interval coverage (percentage of 1000 estimated 95%confidence intervals that cover the true average treatment effect)

The cluster robust standard errors did not differ greatly across methods, except for PARAM producing noticeably larger standard errors (bottom left of Figure 2). In addition, both PARAM-FE and PARAM-RE had larger standard errors on average than the nonparametric models in scenarios F and G.

The nonparametric propensity score models had higher 95% CI coverage rates than PARAM-FE and PARAM-RE in all scenarios (bottom right of Figure 2). For example, PARAM-FE and PARAM-RE had 54.9% and 58.7% coverage rates, respectively, in scenario F, and only 43.3% and 48.5% coverage rates, respectively, in scenario G. PARAM, however, had a decent coverage rate in all scenarios, ranging from 74.4% in scenario G to 93.1% in scenario A; note, however, that this coverage is likely related to its large standard errors given the substantial bias it produced.

While bias increased with increasing non-additivity or non-linearity in scenarios B, C, F, and G for PARAM-FE and PARAM-RE, in scenarios D and E we saw slight improvements in both bias and coverage rate as the non-additivity involving $U$ increased, which may be a result of $U$ being a continuous variable. To explore this we repeated the simulations but with $U$ dichotomized, thus increasing non-smoothness in the response surface – as expected, we found increasing absolute bias and decreasing coverage rate with increasing non-additivity involving $U$ for PARAM-FE and PARAM-RE (results not shown).

Overall, parametric propensity score models produced a greater number of extreme weights than the nonparametric models (Supplementary Figure 6). For example, in scenario G, the parametric models produced several stabilized weights greater than 50; the proportion of stabilized weights greater than 5 for untreated subjects from 10 random simulated data sets was approximately 2.4% for the parametric models and <1.5% for the nonparametric models.

Simulation results from case 2 ($H=100$ and $n_h=50$)

In a setting with more clusters but each of smaller size, we observed fewer benefits of the nonparametric propensity score models. Weights estimated from the BART-based models and GBM provided considerably better balance on the observed covariates ($\mathit{X}$ and $\mathit{V}$) than those from the parametric models in all scenarios other than D and E, in which the differences in the mean and spread of ASMDs across methods were smaller (top and middle panels of Figure 3, Supplementary Figures 7 and 8). Among the nonparametric models, the performance of GBM-FE was relatively poor as its mean ASMD of $\mathit{X}$ and of $\mathit{V}$ were consistently larger than that of the other nonparametric models, and it provided better balance on $\mathit{X}$ and $\mathit{V}$ compared to the parametric models only in scenarios F and G. On the other hand, for the unobserved cluster-level covariate ($U$), weights generated using PARAM-FE, PARAM-RE, and GBM-FE provided better balance, especially in scenarios D and E (bottom panel of Figure 3; Supplementary Figure 9). $U$ remained largely imbalanced when using PARAM, BART, BART-RE, and GBM in several scenarios, and slightly imbalanced for BART-FE. For example, the mean ASMD of $U$ for BART ranged from 0.13 (in scenario G) to 0.48 (in scenario E) across the seven scenarios, while the mean ASMD of $U$ for BART-FE ranged from 0.07 (in scenario G) to 0.23 (in scenario E).

FIGURE 3.

Post-weighting absolute standardized mean difference (ASMD) averaged over 1000 simulations by propensity score estimation model in each of seven propensity score scenarios (A: main effects only; B and C: interactions between observed covariates; D and E: interactions with an unobserved cluster-level covariate; F and G: cubic terms) in case 2 (100 clusters of size 50). ASMD (X) is the average ASMD of six individual-level covariates; ASMD (V) is the average ASMD of two observed cluster-level covariates, and ASMD (U) is the ASMD of an unobserved cluster-level covariate

With regard to bias, the nonparametric propensity score models as a whole outperformed the parametric ones in scenarios F and G only, with the BART-based models yielding the least bias on average under these scenarios (top left and top right of Figure 4; Supplementary Figures 10 and 11). In scenarios D and E, PARAM-RE had the smallest mean bias, which may relate to its ability to balance $U$.

FIGURE 4.

Results from case 2 (100 clusters of size 50). Bias (estimated—true average treatment effect; top left), absolute bias (%; top right), and cluster robust SE (bottom left) averaged over 1000 simulations by propensity score estimation model in each of seven propensity score scenarios (A: main effects only; B and C: interactions between observed covariates; D and E: interactions with an unobserved cluster-level covariate; F and G: cubic terms). Bottom right: 95% confidence interval coverage (percentage of 1000 estimated 95% confidence intervals that cover the true average treatment effect)

As seen in the previous case, the cluster robust standard errors did not differ greatly across methods, with the exception of PARAM yielding substantially larger standard errors (bottom left of Figure 4). The 95% coverage rates vary greatly, both across models and across scenarios (bottom right of Figure 4). The consistently low coverage rates of GBM-FE were likely the result of a large bias and a small standard error combined; PARAM had a higher coverage rate than GBM-FE despite having larger bias in some scenarios, possibly due to its large standard errors. As in the previous case, the parametric models were more likely to produce extreme weights than the nonparametric models in all scenarios (Supplementary Figure 12).

Simulation results from case 3 ($\text{H}=20$ and ${\text{n}}_{\text{h}}=100$)

Finally, we provide a brief summary of results from the case with a small number of clusters (as in case 1) but where each cluster is smaller. The resulting figures are presented in the supplementary material. With regard to the observed covariates ($\mathit{X}$ and $\mathit{V}$), the BART-based propensity score models yielded the smallest mean ASMD across the seven scenarios, and the GBM-based models yielded better balance than the parametric models in all scenarios except D and E (Supplementary Figure 13). Similar to case 2 ($H=100$ and $n_h=50)$, PARAM-FE, PARAM-RE, BART-FE, and GBM-FE appeared to be better at balancing the unobserved cluster-level covariate $U$. BART-based models yielded the least mean bias across the seven scenarios, and PARAM-RE had comparably small biases in scenarios D and E due to its covariate balancing performance on $U$ (top left and top right of Supplementary Figure 14). As seen in previous cases, the nonparametric models outperformed the parametric models in terms of bias when the true propensity score model included cubic terms of the observed covariates (scenarios F and G); they also produced fewer extreme weights than the parametric models in all scenarios (Supplementary Figure 15).

Assessing the standard error estimation

To assess the validity of cluster robust standard errors, we compared them to the empirical standard errors. The two standard errors were generally similar – the ratio of the cluster robust standard error versus the empirical standard error, averaged across all propensity score scenarios and methods, was 0.96, 1.01, and 0.98 in cases 1, 2, and 3, respectively (comparisons by method and by scenario are presented in Supplementary Table 1). Future work may further explore the performance of cluster robust standard errors – and other variance estimation strategies – under different scenarios and the propensity score estimation method used.

5. APPLICATION

As an illustration, we apply the propensity score estimation methods used in the simulation study to the public-use data sets of The National Longitudinal Study of Adolescent to Adult Health (Add Health). A nationally representative sample of U.S. adolescents who participated in Add Health were followed into adulthood – the first wave was conducted during the 1994–1995 school year when the respondents were in grades 7 through 12; the fourth and most recent wave was conducted in 2008 when the respondents were aged 24–32.¹⁷

Our application is based on the study by Easterlin et al,³⁹ which used the Add Health data to investigate the association of team sports participation during adolescence with adult mental health outcomes among individuals exposed to adverse childhood experiences. For the purpose of demonstration, we use the wave 1 and wave 4 public-use data sets of Add Health to estimate the effect of team sports participation during adolescence on adulthood depressive symptoms. The Add Health public-use data sets contain limited survey data for a subset of the full Add Health sample and are available for access by the general public. The wave 1 and wave 4 public-use data sets contain data for 6,504 and 5,114 respondents, respectively, from 132 schools. To avoid convergence issues with small clusters, we restrict our analysis to the 10 largest schools, resulting in an analytic sample of 617 respondents with the school sizes ranging from 51 to 95 students.

As in Easterlin et al,³⁹ the “treatment” is defined as whether respondents participated in at least one team sport during adolescence, which was captured by the wave 1 in-school questionnaire. Our outcome of interest is the total score on the 10-item subscale of the Center for Epidemiologic Studies Depression scale (CES-D-10) in the wave 4 survey, ranging from 0 to 25 in our analytic sample (the maximum possible score is 30).

We select six individual-level covariates based on components of the propensity score in Easterlin et al³⁹: sex (female and male), race (White, Black, Native American/Indian, Asian, and other), ethnicity (Hispanic and non-Hispanic), parental education (a number between 0–8 where higher values indicate higher education attainment of whichever parent has the higher education level; education level of the mother is used if that of the father is missing, and vice versa), whether the respondent lived in an urban area, and neighborhood connectedness (0–2, the sum of responses to the questions “People in this neighborhood look out for one another” and “Do you usually feel safe in your neighborhood?”⁴⁰; a positive response is coded as 1 and negative response as 0). These covariates are obtained from the wave 1 survey. We also calculate the respondents’ total scores on the Feelings Scale in the wave 1 survey, which mostly consists of items from CES-D (range in analytic sample: 0–38; maximum possible score: 57). School characteristics such as school size and region were also included in the propensity score model in Easterlin et al.³⁹ However, school information is not available in the Add Health public-use data files. Therefore, only individual-level covariates are included in our propensity score models.

We estimate the propensity scores using the eight propensity score models listed in section 3.2. Components of the propensity score models include the aforementioned individual-level covariates, score on the wave 1 Feelings Scale, and school indicators for models with fixed or random cluster effects. We assume that participation in team sports do not affect responses to the Feelings Scale during wave 1 but note that this assumption should be carefully validated if the goal is to make substantive conclusions. The average treatment effect of team sports participation during wave 1 on CES-D-10 score during wave 4 is estimated via inverse probability of treatment weighting.

The left plot of Figure 5 shows the covariate balance of each individual-level covariate before and after weighting. All models yield decent balance (ASMD<0.1) on the individual-level covariates with a few minor exceptions (e.g., the ASMD of parental education from BART-RE is 0.11). Given the relatively small sample and moderate cluster sizes in this example, the covariate balancing performance of the nonparametric models may be more affected by unmeasured cluster-level confounding and potential cross-level interactions compared to the parametric models (similar to scenarios D and E in the simulation study). The right plot of Figure 5 shows the balance on school membership before and after weighting. Within each method, models that include fixed cluster effects (i.e., PARAM-FE, BART-FE, GBM-FE) tend to produce better balance on the school indicators, followed by models with random cluster effects (i.e., PARAM-RE and BART-RE).

FIGURE 5.

Covariate balance of the individual-level covariates (left) and school indicators (right) before and after propensity score weighting by eight propensity score estimation models (PARAM: logistic regression; PARAM-FE: logistic regression with fixed cluster effects; PARAM-RE: logistic regression with random cluster effects; BART: Bayesian additive regression trees; BART-FE: BART with cluster indicators; BART-RE: BART with random effects; GBM: generalized boosted model; GBM-FE: GBM with cluster indicators)

All models yield broadly similar estimates of the average treatment effect (0.30–0.67) and suggest that team sports participation during adolescence may not have an impact on adulthood depressive symptoms among the general U.S. population (Figure 6). The intracluster correlation coefficient of the CES-D score was small (approximately 0.02), which may be a reason why there is no huge difference between methods. We note that the purpose of this application is to demonstrate the use of different propensity score estimation models on real data rather than drawing substantive conclusions. The unavailability of the complete Add Health sample and survey data, as well as unmeasured confounding, may hinder us from obtaining valid causal effect estimates.

FIGURE 6.

Average treatment effect (ATE) estimate, with 95% confidence interval, of team sports participation during adolescence on CES-D-10 score during adulthood by propensity score estimation model (PARAM: logistic regression; PARAM-FE: logistic regression with fixed cluster effects; PARAM-RE: logistic regression with random cluster effects; BART: Bayesian additive regression trees; BART-FE: BART with cluster indicators; BART-RE: BART with random effects; GBM: generalized boosted model; GBM-FE: GBM with cluster indicators)

6. DISCUSSION

Our simulation study extends the findings of Lee et al¹⁶ to clustered data settings, supporting the potential usefulness of nonparametric machine learning techniques in improving propensity score weighting for causal inference. However, we also show that nonparametric propensity scores may lose their advantage under certain settings – in particular, when cluster sizes are not sufficiently larger than the number of clusters and there is strong unmeasured cluster-level confounding, propensity score estimation using BART or GBM models should be proceeded with caution.

The goal of propensity score weighting is to make the treated and control groups as similar as possible with respect to pre-treatment covariates in order to reduce bias in the treatment effect estimate. However, it is essentially impossible to capture the full set of confounders in practice. Our simulation study thus assumes that an unobserved confounder exists at the cluster level, and for both parametric and nonparametric approaches, we consider models that either account for or ignore the clustered structure. Regardless of the approach used, our results show the need to incorporate the clustered structure (through either fixed or random cluster effects) in order to provide better balance on the unobserved cluster-level covariate. While the comparison between fixed versus random effects is not our current focus in this paper, we note that the choice between fixed and random effects should be based not only on the data at hand, but also on the propensity score estimation approach (e.g., logistic regression or BART).

Although the simple IPW estimator is used given the focus on propensity score estimation in this paper, incorporating the clustered structure in both the propensity score and outcome models is recommended in practice.⁷ This can be done by fitting a fixed or random cluster effects outcome model on the weighted dataset or implementing the augmented IPW estimator with outcome models that account for the clustering. To explore the benefits of such an approach, we additionally fit a fixed effects outcome model with a main effect of each observed covariate and found significant improvements for all propensity score estimation strategies in all scenarios (bias plots are presented in the supplementary material, Supplementary Figure 16). The bias reduction was a result of having at least (or additionally) the outcome model correctly specified and accounting for the cluster information.

In our simulation setting with 20 clusters of sizes 200 to 500 (case 1), we found that BART models with random intercepts (BART-RE) provided excellent covariate balance of the observed covariates regardless of the number of interactions or cubic terms in the true propensity score model, while BART and GBM models that included cluster indicators (BART-FE and GBM-FE) were slightly better at balancing the unobserved cluster-level covariate. Further, BART provided decent covariate balance for both the observed and the unobserved cluster-level covariates even with the clustered structure being ignored, whereas GBM without cluster indicators fell short of balancing the unobserved cluster-level covariate. Both the BART-based models (BART, BART-FE, and BART-RE) and GBM with cluster indicators (GBM-FE) provided better balance for all types of covariates than the parametric models on average; this was found not only in scenarios where the logistic regression models were misspecified, but also in the scenario where the true propensity score model was both linear and additive. These findings suggest that (in cases of large cluster and sample sizes):

• BART and GBM models that account for clustering may be better alternatives to parametric models for propensity score estimation;

• including cluster indicators in nonparametric propensity score models is recommended when a strong degree of unmeasured cluster-level confounding is likely;

• BART models with random intercepts may be useful when there is presumed to be little unmeasured cluster-level confounding and the balance of observed covariates is to be prioritized.

Note that the above suggestions may pertain only to cases where the sample and cluster sizes are large (e.g., 20 clusters of sizes 200 to 500). In our simulation setting with 100 clusters of size 50 (case 2), nonparametric propensity score models – especially those without cluster indicators – failed to provide adequate balance of the unobserved cluster-level covariate $U$, which led to increased bias, especially in scenarios D and E, where $U$ interacted with multiple covariates in the true propensity score model. A possible explanation is that because the nonparametric approaches do not force the cluster structure in the model even when clustering is considered, the unobserved cluster-level covariate is not prioritized when the clusters are small with little information for the nonparametric models to detect their importance, while more effort is spent on handling the observed covariates. We further examined the amount of confounding caused by $U$ by including $U$ in the BART model with random effects (i.e., BART-RE) – the performance of the model improved significantly with the coverage rate ranging from 91.8% to 96.3% across the seven propensity score scenarios in case 2. The same issue arised when there were 20 clusters with a smaller sample size of 100 units per cluster, though to a smaller extent (e.g., the average ASMD of $U$ for BART-RE over the seven scenarios was 0.248 and 0.118 in the case of 100 clusters of size 50 and 20 clusters of size 100, respectively). However, with regard to the observed covariates, the covariate balancing performance of BART-based models was generally better than that of the parametric models (even when the parametric model was correctly specified as we have found through results not presented in this paper, which was likely related to the models’ ability to pick up chance imbalances) when there was no cross-level interaction involving $U$ in all clustering settings, suggesting that BART-based models may improve propensity score weighting under a variety of settings if unmeasured cluster-level confounding is minimal.

In our study, only two nonparametric approaches for estimating propensity scores are examined. Other methods such as random forest and neural networks may offer additional insight into nonparametric propensity score estimation in a multilevel context. We note that an ensemble machine learning algorithm called Super Learner has been developed as a method to automatically select among a “library” of candidate models via cross-validation in order to build an optimal model for a given setting; hence, Super Learner has the potential advantage of combining the strengths of a variety of machine learning strategies.⁴¹ It has been shown that estimating propensity scores using Super Learner can improve covariate balance and reduce bias when the main-effects logistic regression model is severely misspecified.³⁴ Evaluation of the effectiveness of Super Learner for propensity score estimation under multilevel contexts could be an avenue for future research.

Our simulation design assumes that all covariates entered in the propensity score estimation model are related to both the treatment and the outcome. However, investigators generally do not know the actual set of confounders and may include redundant covariates in the propensity score estimation model. It has been shown that adding irrelevant covariates to GBM may lead to increased covariate imbalance and bias in the treatment effect estimates.⁴² BART appears to be effective at detecting important predictors when irrelevant ones are added, but its effectiveness in the context of propensity score estimation is unknown.²⁹ Future work may assess the performance of GBM and BART compared to parametric propensity score modeling under more realistic scenarios where irrelevant covariates are included in addition to unmeasured cluster-level confounding.

Lastly, we point out a few properties regarding the implementation of GBM and BART. Note that default parameter settings in the R packages for BART (BART and dbarts) and GBM (twang) are used in our simulation experiment, but the performance of these machine learning algorithms may be enhanced from parameter tuning. As mentioned in Section 2.2, BART has an advantage in that it only requires minimal assumptions regarding the model parameters by placing prior distributions over the tree models. BART is highly robust to small changes in the prior and the choice of the number of trees, and the defaults are usually adequate.²⁹ Thoughtful specification of the GBM parameters may improve its performance by a greater extent. A disadvantage of GBM is that the twang package can be computationally demanding, as evidenced by others as well as our experience in implementing the two methods, where the speed of BART is markedly faster than GBM.⁴³

In conclusion, our results suggest that in observational studies with clustered data, flexible modeling of the propensity score may offer advantages in terms of covariate balance and bias reduction, at least in studies where the sample size is large and cluster sizes are considerably larger than the number of clusters (e.g., 20 clusters of sizes 200 to 500). However, when the cluster sizes are not sufficiently large (e.g., 100 clusters of size 50), nonparametric models may not be particularly useful for propensity score estimation in some cases due to failure to adequately balance unmeasured cluster-level covariates. A major limitation of our study, and all simulation studies in general, is that we are unable to capture all possible cluster/sample sizes and assignment mechanisms that may occur in practice. In particular, we generated clusters such that each has an adequate number of treated and control units in all scenarios, but treatment allocation is often unbalanced in observational datasets. As seen from our simulations and real data application, results are highly dependent on the specific setup and data generating processes. It is thus important to note that the main contribution of our findings lies in offering insight into parametric versus nonparametric propensity score estimation with clustered data. They should not be regarded as definite conclusions and the choice of which model works best will largely depend on the specific data at hand. Future work may extend our study to explore the potential of nonparametric propensity score estimation in other situations (e.g., with extremely unbalanced treatment allocation within clusters).