A Cautionary Note for Plasmode Simulation Studies in the Setting of Causal Inference

ABSTRACT

Plasmode simulation has become an important tool for evaluating operating characteristics of different statistical methods in complex settings, such as pharmacoepidemiological studies of treatment effectiveness using electronic health records (EHR) data. These studies provide insight into how estimator performance is impacted by challenges including rare events, small sample size and so forth that can indicate which among a set of methods performs best in a real‐world dataset. Plasmode simulation combines data resampled from a real‐world dataset with data from a probabilistic model to generate a known truth for an estimand in realistic data. There are different potential plasmode strategies currently in use. We compare two popular plasmode simulation frameworks for the evaluation of methods estimating a point treatment estimand. We provide numerical evidence and a theoretical result that shows one of these frameworks can cause certain estimators to incorrectly appear overly biased. Detailed simulation studies using both model‐generated and real‐world EHR data demonstrate these pitfalls remain at large sample sizes and when analyzing data from a randomized controlled trial. We conclude with guidance for the choice of a plasmode framework that maintains good theoretical properties to allow a fair evaluation of statistical methods while also maintaining the desired similarity to real data.

1. Introduction

Simulation studies are typically used to empirically evaluate the relative performance (e.g., bias, efficiency) of estimation methods [1, 2]. One concern is that simplistic computer‐generated data structures may lead to performance that is different than what would be expected in real‐world data. This limitation motivated a simulation technique known as “plasmode simulation,” in which one or more key elements of the data, such as the treatment effect, is computer‐generated, but others are generated by bootstrap sampling real‐world data.

Several authors have used plasmode simulation to select the best performing method to handle missingness in confounders in electronic health records (EHR) data [3, 4, 5, 6]. It has been used to evaluate machine learning methods to build prediction models [7, 8], to study the performance of different propensity score approaches and other causal methods to estimate treatment effects [9, 10, 11], and to compare statistical methods in a variety settings [12, 13, 14].

Despite the frequency of use, formal mathematical justification to guide the proper application of plasmode simulation has not been fully considered. One of the first uses of the term “plasmode” to describe data where key elements are known to fit some true underlying model was by Cattell and Jaspers [15]. Decades later Gadbury et al. [16] define plasmode “to describe datasets that are derived from real data but for which some truth is known,” and demonstrate the utility of plasmode simulation in high‐dimensional settings.

Franklin et al. [17] described a now popular plasmode simulation procedure to evaluate estimation methodologies for a treatment exposure estimand in the context of longitudinal health care databases. Their method specifies bootstrap sampling covariates and treatment together, then generating the outcome conditional on resampled data. The chosen outcome‐generating model may be based on features in the observed data. They suggest this Sample Treatment framework preserves covariate‐treatment relationships, while still allowing the investigator to know the true (simulated) causal effect. An alternate plasmode simulation approach is to bootstrap sample only the covariates ($W$), then simulate treatment ($A$) conditional on the covariates. In keeping with the plasmode philosophy, one can generate $A$ from a propensity score model trained on the actual data, then generate the outcome, $Y$, conditional on $W$ and the generated $A$, again using a model that could be derived from the observed data. Many papers have used this Generate Treatment plasmode framework in analyses of pharmacoepidemiologic data [5, 9, 18, 19]. In a recent tutorial paper on plasmode simulation, Schreck et al. [20] discuss that in addition to sampling covariates, one can choose to also simulate artificial covariates from a parametric model, but do not go into details as to when one would want to do so.

We compare the common Sample Treatment and Generate Treatment plasmode frameworks and show only the latter approach can be reliably used to evaluate the statistical performance of many typical causal inference methods reliant on the propensity score for evaluating a point treatment effect. We begin by defining a set of causal estimands and common statistical methods whose performance will be evaluated with plasmode simulation. We then describe the two plasmode frameworks and discuss why only the Generate Treatment framework will satisfy the mathematical conditions necessary to guarantee the desirable analytical properties when evaluating performance for several estimators relying on the propensity score. Numerical studies are performed to evaluate the two plasmode simulation procedures for a variety of computer‐generated data settings. We then conduct a series of plasmode simulation studies using real‐world data to further illustrate the superior performance of the proposed Generate Treatment framework. We conclude with a summary of lessons learned regarding plasmode simulation to evaluate statistical methods.

2. Motivating Setting

In observational studies, a principal concern when estimating a treatment effect is confounding bias. A common target of estimation is the average treatment effect (ATE), equivalent to the marginal risk difference (RD) of a binary outcome. The marginal relative risk (RR) or the log of the conditional odds ratio (logcOR) may also be of interest. We show that the problematic issues that arise when doing Sample Treatment plasmode simulations are relevant quite generally for common estimators of causal estimands (e.g., the ATE) reliant on the propensity‐score. A key issue is that not all estimators are affected in the same manner by this plasmode approach, which means the perceived relative performance of the estimators can be misleading. In this section, we formally define a set of treatment effect estimands and estimators to be compared using the two plasmode simulation frameworks.

2.1. Notation and Estimands

Consider data with $n$ i.i.d. observations $O=(W,A,Y)$, $W$ a vector of baseline covariates, $A$ a binary treatment indicator, and outcome $Y$. We define potential outcome $Y(a)$ for treatment exposure $A=a$. Let ${\mu }_a=E[Y(a)]$. We define the marginal ATE = ${\mu }_1-{\mu }_0$ and RR = ${\mu }_1/{\mu }_0$. We also consider the coefficient for treatment $A$ in a logistic regression model for $Y$ adjusted for $W$, namely the log conditional odds ratio (logcOR) for $A$, a common estimand for binary outcomes.

2.2. Estimators

We seek to evaluate the performance of algorithms that require correct specification of the propensity score (PS) for consistency, as well as those that do not. We consider the following common estimators for the ATE:

• – Propensity score matching (Match). This algorithm uses the generalized full optimal matching algorithm with replacement [21, 22] to generate weights for each observation suitable for evaluating the ATE. The outcome model for $E(Y|A)$ is then estimated using a weighted, unadjusted linear regression of the outcome on treatment.
• – Inverse probability of treatment weighting (IPTW). Observation weights are stabilized using the marginal treatment probability [23] and bounded to be no more than $\sqrt{n}\ln (n)/5$ [24]. The outcome model $E(Y|A)$ is then estimated using a weighted, unadjusted linear regression of the outcome on treatment.
• – Doubly robust targeted maximum likelihood estimation (TMLE). The TMLE [25] is fit using the correctly specified working models for the treatment propensity and outcome, bounding the treatment assignment probabilities by $5/(\sqrt{n}\ln (n)).$
• – Generalized linear model, correctly specified (glmCM). The outcome model $E(Y|A,W)$ is fit using the correctly specified regression model. The marginal ATE is then obtained using G‐computation.
• – Generalized linear model, adjusted for propensity score (glmPS). The outcome model is fit regressing $Y$ on $A$ and the PS fit from a correctly specified model $E(A|W)$. The marginal ATE is then obtained using G‐computation.

For each estimator, predicted counterfactual outcomes are obtained when $A$ is set to 0 and to 1 to estimate the ATE (or RR).

A typical estimation approach for the logcOR relies on a marginal structural model (MSM) for the outcome. We define a parametric logistic MSM that includes $A$ and a subset of potential confounders, $\tilde{\mathbf{W}}$. The MSM logcOR is estimated by weighted maximum likelihood using IPTW. Further details are provided in the numerical studies section below.

3. Plasmode Simulation

We define plasmode simulation as a data generation mechanism (DGM) for which at least part of the constructed data is generated by bootstrap sampling a source dataset $\mathcal{D}$. Commonly the source dataset is from a real‐world setting and the goal of plasmode simulation is to create datasets with similar structure to the source data but for which there is a known truth. We assume that an outcome $Y^{\#}$ will be generated from $\mathcal{D}$ so that the treatment estimand of interest (ATE, RR or logcOR) has known value.

For the Sample Treatment plasmode approach, both the treatment and covariates $(A,W)$ are bootstrap sampled from the source data $\mathcal{D}$ and then $Y^{\#}$ is stochastically generated conditional on $(A,W)$ according to an assumed model, defined as $f_Y\left(A,W,U_Y\right)$, where $U_Y$ is an exogenous error term. For the Generate Treatment plasmode approach, we define a propensity score model $P(A|W)=f_A\left(W,U_A\right)$, which may be a fitted model on the source data. We then bootstrap sample only $W$ from $\mathcal{D}$, generate $A^{\#}$ from $P(A|W)$, and then generate the outcome, $Y^{\#}$, conditional on $\left(W,A^{\#}\right)$, from $f_Y\left(A,W,U_Y\right)$ as before. The two approaches are summarized in Table 1.

TABLE 1.

Data generating mechanisms for plasmode simulation approaches.

	Sample Treatment	Generate Treatment
Covariates	Sample $W$ with replacement	Sample $W$ with replacement
Treatment	Sample $A=a$ along with $W$	Generate $A^{\#}\sim f_A\left(W,U_A\right)$
Outcome	Generate $Y^{\#}\sim f_Y\left(A,W,U_Y\right)$	Generate $Y^{\#}\sim f_Y\left(A^{\#},W,U_Y\right)$

4. Theoretical Problems for the Sample Treatment Framework

A frequent goal of a plasmode simulation is to evaluate whether a particular method can accurately and efficiently estimate an estimand of interest. To this end, the statistical performance of an estimator based on data generated by plasmode simulation needs to resemble the statistical performance of the estimator based on the actual data distribution. This does not necessarily mean that plasmode simulations are drawn from an accurate estimator of the data distribution (e.g., think outcome blind simulations), but qualitatively it should be highly informative for comparing estimators and evaluating the performance of a given estimator. We provide theoretical justification for why samples generated from the Sample Treatment framework will not produce accurate estimates of estimator performance, such as bias or confidence interval coverage.

The root of the problem is that the positivity assumption required for identifying a causal estimand, $P(A=a|W)>0$ for all $a$ and observed $W$ in the data, is violated under the Sample Treatment framework. Under this plasmode approach, every time $W_i$ is sampled, the associated value for $A_i$ is fixed at some $a$, its value in the original data for subject $i$; thus, the probability of receiving treatment $a$ given $W_i$ is 1. Importantly, this positivity violation occurs even when there is no violation in the source population.

This induced violation of the positivity assumption under the Sample Treatment framework implies that estimators relying on a correctly specified outcome regression for consistency (e.g., parametric G‐computation, such as glmCM) are fully reliant on extrapolation for the treatment/covariate combinations missed by the Sample Treatment algorithm, whereas estimators relying on propensity score estimation (IPTW, glmPS or Match) will end up having non‐negligible bias in the plasmode samples, even when the propensity score model is correctly specified. We show bias is of order $n^{-1/2}$, which means it is non‐negligible with respect to the SE of the estimator, even as sample size converges to infinity, which affects confidence interval coverage. Because of its double‐robustness property, TMLE will also avoid bias under the Sample Treatment algorithm.

For example, given data $O_i=\left(Y_i,A_i,W_i\right)\sim P_0$, suppose the target estimand is the average outcome $Y$ for treatment $A=1$, namely $\mathrm{\Psi }\left(P_0\right)=E_0\left[E_0(Y|A=1,W)\right]$. For an i.i.d. sample $O_1,\dots ,O_n$ of size $n$ drawn from $P_0$, whose empirical distribution we denote with $P_n$, consider the simple IPTW‐estimator $\widehat{\mathrm{\Psi }}\left(P_n\right)=n^{-1}{\sum }_{i=1}^n{A}_i{Y}_i/\widehat{g}\left(P_n\right)$, where $\widehat{g}\left(P_n\right)$ is a maximum likelihood estimator of the true propensity score ${\bar{g}}_0(W)=P_0(A=1|W)$ based on a correctly specified parametric model. This IPTW estimator is a consistent, asymptotically linear estimator of $\mathrm{\Psi }\left(P_0\right)$, with negligible bias of order $1/n$.

Consider now the Sample Treatment data distribution ${\mathbf{P}}_n$ that draws $(W,A)$ from $P_n$, and then draws $Y$ given $(W,A)$ from a specified conditional distribution, such as the distribution of $Y$ given $(W,A)$ under $P_0$. Under such a data distribution, one would view the true estimand as ${\psi }_n=\mathrm{\Psi }\left({\mathbf{P}}_n\right)=n^{-1}{\mathrm{\Sigma }}_{i=1}^{n}E\left(Y|W_i=w_i,A=1\right)$ (strictly speaking $\mathrm{\Psi }\left({\mathbf{P}}_n\right)$ is not well defined due to the positivity issue, which makes the conditional mean improper for observations $W_i$ with $A_i=0$, and that is the core problem). We show that under the Sample Treatment data distribution ${\mathbf{P}}_n$, the IPTW estimand $\widehat{\psi }\left({\mathbf{P}}_n\right)$ (i.e., the target of the IPTW estimator it ends up fitting when given an infinite sample from ${\mathbf{P}}_n$) is given by $n^{-1}{\mathrm{\Sigma }}_{i=1}^n{A}_i/\widehat{g}\left(P_n\right)E\left(Y|W=W_i,A=1\right)$, which is not equal to ${\psi }_n$, and the difference equals $B_n=\widehat{\psi }\left({\mathbf{P}}_n\right)-{\psi }_n=n^{-1}{\mathrm{\Sigma }}_{i=1}^n\left[\frac{\left\{A_i-{\bar{g}}_n(W)\right\}E\left(Y|W=W_i,A=1\right)}{{\bar{g}}_n(W)}\right]$, where ${\bar{g}}_n=\widehat{g}\left(P_n\right)$. We show that this bias $B_n$ approximates zero only at rate $n^{-1/2}$, but not faster. That means the IPTW estimator acts as an inconsistent estimator of ${\psi }_n$ in the plasmode samples from ${\mathbf{P}}_n$. Even though the bias $B_n$ goes to 0 at rate $n^{-1/2}$, that bias is of the same order as the SE and thus non‐negligible for coverage, wrongly suggesting that the IPTW estimator is an overly biased estimator.

This bias stems from the difference between $P_0$ and $P_n$ with respect to the treatment assignment mechanism, which does not go away asymptotically due to the induced positivity violation. Conversely, the data distribution for the Generate Treatment plasmode approach ${\tilde{\mathbf{P}}}_n$ avoids these theoretical problems by assigning positive probability on $A=1$ and $A=0$ given any possible drawn $W$. A detailed mathematical argument is presented in Supporting Information: Section S1.

5. Numerical Studies

5.1. Overview

We demonstrate how the choice of the plasmode simulation framework impacts the assessment of estimator performance. Four simulation scenarios utilize completely synthetic (computer‐generated) data, varying the outcome model, treatment assignment mechanism, rarity of a binary outcome, and treatment estimand. The fifth scenario mimics a real‐world plasmode simulation study, using a source dataset from the Kaiser Permanente Washington (KPWA) EHR system [5]. We examine estimator performance for the estimators outlined in Section 2.2, as well as an unadjusted estimator, which provides a sense of the degree of confounding. The unadjusted estimate (Unadj) is similar to the glmCM and glmPS methods, except treatment is the only covariate in the outcome model.

The main goal of the simulation study is to demonstrate that even when the plasmode simulation models are correctly specified, bias will occur for the Sample Treatment algorithm for methods that rely on the propensity‐score (i.e., matching, IPTW glmPS). Thus unless otherwise specified (e.g., Scenario 4b) the data generating and analysis models for the plasmode procedure will match the underlying DGM for the source dataset. For each DGM, we generate a data set of size $n$, which is the source dataset for the plasmode simulation. We then conduct 100 000 Monte Carlo replications in which a plasmode data set is generated from the source dataset according to either the Sample Treatment or Generate Treatment framework and evaluate the mean percent (%) bias, empirical standard error (SE), root mean‐squared error (RMSE) and absolute bias:SE ratio. Estimators under study are asymptotically regular (with $\sqrt{n}$‐convergence rate); thus, for consistent estimators, we expect the bias:SE to be proportional to $1/\sqrt{n}$. For asymptotically biased estimators (e.g., Unadj in most scenarios), bias:SE increases with sample size because SE shrinks to 0 while bias does not. We considered $n=100$, $1000$, and $10000$ in Scenarios 1–4. For the real‐data Scenario 5, $n=10000$.

5.2. Plasmode Simulation Set‐Up

5.2.1. Scenarios 1 and 2: Continuous and Binary Outcomes

Scenarios 1 and 2 share the same DGM for ($\mathbf{W}=\left(W_1,W_2,W_3\right)$) and $A$: $W_1\sim N(0,1),W_2\sim N(\mathrm{0.5,1}),W_3\sim \mathit{\text{Bernoulli}}(0.4)$; $P(A=1|\mathbf{W})=\text{expit}\left(-0.48+0.96W_1+0.012W_2+1.08W_3\right)$. For Scenario 1, $Y\sim N(\mu ,1),$ with $\mu =10+2A+W1+0.7W_2+0.02W_3$; for Scenario 2, $P(Y=1|A,\mathbf{W})=\text{expit}(-1.8+1.1A+0.24W_1+0.08W_2+0.8W_3)$. The marginal ATE in Scenario 1 is ${\psi }_{0,\mathit{sim}1}^{\mathit{ATE}}=2$. In Scenario 2, ${\psi }_{0,\mathit{sim}2}^{\mathit{ATE}}=\mathrm{0.2199,0.2171,0.2182},$ when n = 100, 1000, 10 000, respectively.

For Scenarios 1 and 2, we also ran simulations with treatment randomized 1:1 for $n=1000$, Scenarios 1a and 2a, respectively. Estimators again used the correct outcome model and propensity score estimated from the true intercept‐only regression model.

5.2.2. Scenario 3: Rare Binary Outcome

Data with $n=10000$ were generated as follows: $W_1\sim N(0,1),W_2\sim N(\mathrm{0.5,1}),W_3\sim \mathit{\text{Bernoulli}}(0.4)$; $P(A=1|\mathbf{W})=\text{expit}\left(-0.72-0.72W_1+0.18W_2+0.81W_3\right)$, yielding approximately 43% treated, with propensity score ranging from 0.025 to 0.975. $Y$ was generated from the logistic model $\text{logit}(P(Y=1|A,\mathbf{W}))=-4.9-2A+0.4W_1-4W_2-3W_3$, with a 5% event rate. The marginal ATE = −0.0247 and marginal RR = 0.3454.

5.2.3. Scenario 4: Estimating a MSM Regression Coefficient

We consider two MSM scenarios (Scenarios 4a and 4b) to estimate the logcOR, with the same DGM: $W_1\sim N(0,1)$, $W_2\sim N(\mathrm{0.5,1})$, $W_3\sim \mathit{\text{Bernoulli}}(0.4)$, $W_4$ is generated as $W_3+N(0,1)$, and $W_5$ is the binary indicator $W_1>0.2$ (approximately 42% probability). The correlation between $W_3$ and $W_4$ is ${\rho }_{W_3,W_4}=0.44$, and between $W_1$ and $W_5$ is ${\rho }_{W_1,W_5}=0.79$. Other variables are uncorrelated. $A$ is generated according to $\text{logit}(P(A=1|\mathbf{W}))=-0.4+0.8W_1+.01W_2+0.9W_3$, with propensity score range 0.34–0.97 and mean 0.49. The binary outcome (14% event rate) was generated as $\text{logit}(P(Y=1|A,\mathbf{W}))=-2.5+1.1A+0.24W_1+.08W_2+0.8W_3-0.3W_4-0.6W_5$, equivalently expressed as $\text{logit}(P(Y=1|A,\mathbf{W}))=-2.5+1.1A+0.24W_1+.08W_2+0.8W_3-0.3W_3-0.3\mathrm{ϵ}-0.6I\left(W_4<0.2\right),$ with $\mathrm{ϵ}\sim N(0,1)$. We let $W=\left(W_1,\dots ,W_5\right)$ denote the DGM full set of confounders, and $\tilde{\mathbf{W}}$ denote the confounders in the MSM, which may or may not be the same as $\mathbf{W}$. We consider $n=\mathrm{100,1000,10}000$.

In Scenario 4a, the MSM is the correct outcome model, $\text{logit}(P(Y=1|A,\mathbf{W}))={\beta }_0+{\beta }_{1}A+{\beta }_2{W}_1+{\beta }_3{W}_2+{\beta }_4{W}_3+{\beta }_5{W}_4+{\beta }_6{W}_5$. The MSM parameter values are the coefficients in the DGM model, $\beta =(-\mathrm{2.5,1.1,0.24,0.08,0.8},-0.3,-0.6)$, with logcOR = 1.1.

In Scenario 4b, the MSM conditions on all confounders but does not capture the true functional form of the outcome model. This MSM defines a projection of the underlying DGM onto a specified model, $E[Y|\tilde{\mathbf{W}},A]={\gamma }_0+{\gamma }_{1}A+{\gamma }_2{W}_1+{\gamma }_3{W}_2+{\gamma }_4{W}_3$. The MSM parameter values were established through simulation (see Supporting Information: Section S2). The true value for logcOR is 1.084.

Datasets were created under each plasmode framework and analyzed by fitting a weighted logistic regression of $Y$ on the MSM using stabilized IPTW with weights given by ${\mathit{wt}}_{\mathit{\text{stab}},A=1}=p_{A}A/P(A=1|W)$ for treated and ${\mathit{wt}}_{\mathit{\text{stab}},A=0}=\left(1-p_A\right)(1-A)/[1-P(A=1|W)]$ for non‐treated observations.

5.2.4. Scenario 5: KPWA Plasmode Data Simulation

Scenario 5 considers plasmode simulation using EHR data for 50 337 individuals 13 years and older initiating antidepressant medication or psychotherapy for treatment of depression at KPWA. These individuals have full information on the outcome of interest, self‐harm, and key confounders. These data are a subset of data analyzed by Williamson et al. [5] for a plasmode simulation to evaluate methods addressing confounder missingness.

The rate of 5‐year self‐harm or hospitalization is 10.3%. Based on a logistic regression model fit to the data (Table S5), the ATE comparing psychotherapy to antidepressant medication is −0.017 [5]. We created two new plasmode DGMs with varying outcome rate and a larger treatment effect by modifying coefficients in these logistic regression models (Table S5). For each DGM, we generated a new source dataset of size $n=10000$ (details in Supporting Information: Section S3). The ATEs in these two groups are −0.092 for the 15% outcome and −0.079 in the 5% outcome dataset; the RRs are 0.512 (15% outcome) and 0.062, respectively.

We generated 100 000 datasets from these two source datasets under the Sample Treatment and Generate Treatment plasmode frameworks and evaluate the performance for the same estimators of the ATE and RR as described above.

5.3. Results

5.3.1. Scenarios 1 and 2

For Scenario 1 (Table 2), the % bias and bias:SE for IPTW and glmPS are about 10‐fold larger, or worse, for the Sample Treatment versus Generate Treatment approach. In several instances, the bias:SE remains larger than expected for a consistent method (i.e., rate $\propto 1/\sqrt{n}$) for the Sample Treatment but not Generate Treatment approach. For the continuous outcome (Scenario 1), this is more apparent for IPTW than glmPS. For the binary outcome (Scenario 2), the inflation of the bias:SE for IPTW and glmPS under the Sample Treatment approach appears smaller than for Scenario 1; however, the bias:SE does not decrease with sample size (Table 3).

TABLE 2.

Simulation Scenario 1: Estimate ATE for continuous outcome.

	Sample Treatment				Generate Treatment
	% Bias	SE	RMSE	Bias:SE	% Bias	SE	RMSE	Bias:SE
$n=100$
Unadj	$159.29$	$1.337$	$3.455$	$2.382$	$159.60$	$1.344$	$3.463$	$2.376$
Match	$-10.78$	$0.818$	$0.846$	$0.264$	$3.80$	$0.843$	$0.846$	$0.090$
IPTW	$-19.65$	$0.612$	$0.727$	$0.642$	$2.08$	$0.478$	$0.479$	$0.087$
TMLE	$-0.15$	$0.236$	$0.236$	$0.013$	$-0.15$	$0.234$	$0.234$	$0.013$
glmCM	$-0.01$	$0.224$	$0.224$	$0.001$	$0.01$	$0.223$	$0.223$	$0.001$
glmPS	$-2.41$	$0.248$	$0.252$	$0.195$	$0.10$	$0.236$	$0.236$	$0.009$
$n=1000$
Unadj	$19.53$	$0.470$	$0.611$	$0.831$	$19.33$	$0.469$	$0.608$	$0.824$
Match	$-18.01$	$0.548$	$0.655$	$0.658$	$0.85$	$0.419$	$0.419$	$0.041$
IPTW	$-7.28$	$0.235$	$0.276$	$0.620$	$0.18$	$0.221$	$0.221$	$0.016$
TMLE	$-0.13$	$0.076$	$0.076$	$0.034$	$-0.06$	$0.076$	$0.076$	$0.016$
glmCM	$0.00$	$0.071$	$0.071$	$0.001$	$-0.01$	$0.071$	$0.071$	$0.002$
glmPS	$-0.08$	$0.072$	$0.072$	$0.021$	$0.00$	$0.071$	$0.071$	$0.001$
$n=10000$
Unadj	$43.62$	$0.141$	$0.884$	$6.168$	$43.57$	$0.142$	$0.883$	$6.138$
Match	$3.43$	$0.154$	$0.169$	$0.445$	$0.00$	$0.117$	$0.117$	$0.001$
IPTW	$0.09$	$0.056$	$0.056$	$0.033$	$0.01$	$0.058$	$0.058$	$0.003$
TMLE	$0.00$	$0.023$	$0.023$	$0.003$	$0.00$	$0.024$	$0.024$	$0.004$
glmCM	$0.00$	$0.022$	$0.022$	$0.002$	$0.00$	$0.022$	$0.022$	$0.001$
glmPS	$-0.01$	$0.022$	$0.022$	$0.009$	$0.00$	$0.022$	$0.022$	$0.001$

Abbreviations: ATE, average treatment effect; glmCM, generalized linear model correct model; glmPS, generalized linear model propensity score; IPTW, inverse probability of treatment weighted; RMSE, root mean squared error; SE, standard error; TMLE, targeted maximum likelihood estimation; Unadj, unadjusted.

TABLE 3.

Simulation Scenario 2: Estimate ATE for binary outcome.

	Sample Treatment				Generate Treatment
	% Bias	SE	RMSE	Bias:SE	% Bias	SE	RMSE	Bias:SE
$n=100$
Unadj	$29.25$	$0.091$	$0.111$	$0.708$	$29.71$	$0.091$	$0.112$	$0.720$
Match	$1.15$	$0.129$	$0.129$	$0.020$	$1.11$	$0.119$	$0.119$	$0.020$
IPTW	$-2.23$	$0.110$	$0.110$	$0.044$	$0.51$	$0.106$	$0.106$	$0.011$
TMLE	$0.11$	$0.106$	$0.106$	$0.002$	$0.16$	$0.106$	$0.106$	$0.003$
glmCM	$0.10$	$0.101$	$0.101$	$0.002$	$0.18$	$0.101$	$0.101$	$0.004$
glmPS	$-0.34$	$0.101$	$0.101$	$0.007$	$-0.02$	$0.101$	$0.101$	$0.000$
$n=1000$
Unadj	$32.81$	$0.028$	$0.077$	$2.538$	$33.05$	$0.028$	$0.077$	$2.556$
Match	$0.13$	$0.043$	$0.043$	$0.006$	$0.27$	$0.039$	$0.039$	$0.015$
IPTW	$-0.66$	$0.034$	$0.034$	$0.042$	$0.07$	$0.034$	$0.034$	$0.005$
TMLE	$-0.06$	$0.034$	$0.034$	$0.004$	$0.00$	$0.034$	$0.034$	$0.000$
glmCM	$-0.05$	$0.032$	$0.032$	$0.004$	$0.00$	$0.032$	$0.032$	$0.000$
glmPS	$-0.15$	$0.032$	$0.032$	$0.010$	$-0.05$	$0.032$	$0.032$	$0.003$
$n=10000$
Unadj	$32.18$	$0.009$	$0.071$	$7.909$	$32.22$	$0.009$	$0.071$	$7.862$
Match	$0.29$	$0.013$	$0.013$	$0.048$	$0.03$	$0.012$	$0.012$	$0.006$
IPTW	$0.34$	$0.010$	$0.010$	$0.071$	$0.02$	$0.011$	$0.011$	$0.005$
TMLE	$0.02$	$0.010$	$0.010$	$0.003$	$0.02$	$0.011$	$0.011$	$0.004$
glmCM	$0.01$	$0.010$	$0.010$	$0.001$	$0.01$	$0.010$	$0.010$	$0.003$
glmPS	$-0.03$	$0.010$	$0.010$	$0.006$	$0.00$	$0.010$	$0.010$	$0.001$

Abbreviations: ATE, average treatment effect; glmCM, generalized linear model correct model; glmPS, generalized linear model propensity score; IPTW, inverse probability of treatment weighted; RMSE, root mean squared error; SE, standard error; TMLE, targeted maximum likelihood estimation; Unadj, unadjusted.

The magnitude of % bias and bias:SE for the Match estimator is expected to be dominated by the quality of the matches, with increased continuity leading to larger expected bias; however, there were still notable differences in bias:SE between the two plasmode frameworks for the continuous outcome in Scenario 1. In Scenario 1, the bias:SE was again more than 10‐fold worse and not clearly decreasing with sample size for the Sample Treatment approach. For Scenario 2, the performance of Match was more comparable between the two plasmode approaches.

Under the two plasmode approaches, there are no theoretical reasons to expect a different numerical performance from glmCM or TMLE, whose consistency is leveraged from the correctly assumed outcome model, or from the unweighted Unadj estimator that does not adjust for confounding. For these three estimators, the % bias and bias:SE are generally similar across $n$ for Scenarios 1 and 2.

5.3.1.1. Coverage

Wald‐type 95% confidence intervals (CIs) were calculated using the empirical SE. The bias of Unadj leads to poor coverage that tends towards 0 as $n$ increases (data not shown). For the continuous outcome, coverage of Match remained lower than the 95% nominal level for all $n$ and IPTW estimators' coverage suffered at smaller sample sizes under the Sample Treatment approach; coverage remained at the nominal level for all estimators and $n$ with the Generate Treatment approach (Figure S1a). Coverage of all estimators was good under both plasmode sampling schemes for the binary outcome (Figure S1b).

5.3.1.2. Randomized Treatment

The Unadj estimator is known to be consistent in this RCT setting, yet under the Sample Treatment approach, it and the PS‐based estimators (IPTW and glmPS) have a bias:SE larger than expected for a consistent estimator (Tables S1 and S2). The impact is less pronounced when the outcome is binary. Under the Generate Treatment approach, all estimators' performance conforms to theoretical expectations. The large bias:SE under the Sample Treatment approach manifests as poor coverage for the continuous outcome and slightly lower than expected coverage for the binary outcome (Figures S2 and S3).

5.3.2. Scenario 3: Rare Outcome Results

Increased outcome rarity increased the apparent problems for the Sample Treatment framework. There is a larger than expected bias:SE for the Sample Treatment approach for all PS‐based estimators (Table 4). The bias:SE for IPTW is nearly 200 times larger under the Sample Treatment versus Generate Treatment approach. The bias:SE of glm‐PS and Match is 8 times larger under the Sample Treatment approach. The performance of Unadj, glmCM, and TMLE have similar bias and SEs under the two plasmode approaches. A large bias:SE translated into lower than expected CI coverage for the PS‐based estimators. At this sample size, all estimators have 100% power to reject the null (data not shown), but for estimators where CI coverage is below 95%, rejections are sometimes fueled by bias away from the null (Figure 1).

TABLE 4.

Simulation Scenario 3: Estimate ATE for rare binary outcome, $n=10000$.

	Sample Treatment				Generate Treatment
	% Bias	SE	RMSE	Bias:SE	% Bias	SE	RMSE	Bias:SE
Unadj	$41.698$	$0.003$	$0.011$	$3.369$	$34.918$	$0.003$	$0.009$	$2.793$
Match	$8.493$	$0.004$	$0.004$	$0.555$	$-0.103$	$0.003$	$0.003$	$0.007$
IPTW	$9.362$	$0.003$	$0.004$	$0.747$	$-0.052$	$0.003$	$0.003$	$0.004$
TMLE	$-0.001$	$0.003$	$0.003$	$0.000$	$-0.052$	$0.003$	$0.003$	$0.005$
glmCM	$-0.044$	$0.002$	$0.002$	$0.004$	$-0.033$	$0.002$	$0.002$	$0.003$
glmPS	$9.939$	$0.003$	$0.004$	$0.813$	$1.343$	$0.003$	$0.003$	$0.108$

Abbreviations: ATE, average treatment effect; glmCM, generalized linear model correct model; glmPS, generalized linear model propensity score; IPTW, inverse probability of treatment weighted; RMSE, root mean squared error; SE, standard error; TMLE, targeted maximum likelihood estimation; Unadj, unadjusted.

FIGURE 1.

Simulation Scenario 3 (rare binary outcome) coverage for ATE.

Table S3 shows results for the RR. Results are identical for the estimator that uses the correct regression model (glmCM). For all other estimators except the TMLE, the bias:SE is smaller under the Generate Treatment approach. For the TMLE, it is smaller under the Sample Treatment, but the same order of magnitude. Coverage (Figure S4) and power (data not shown) follow a similar pattern as for the ATE.

5.3.3. Scenario 4: MSM Results

Scenario 4a results show that when the specified MSM is the true outcome model, both plasmode frameworks work correctly (Table S4). Scenario 4b results illustrate that the performance of the plasmode approaches differs when the specified MSM is not equivalent to the true outcome model (Table 5). The bias:SE under the Sample Treatment framework does not decrease at the expected rate with sample size, but under the Generate Treatment approach it does.

TABLE 5.

Simulation Scenario 4b: Estimate logcOR when MSM is not equivalent to true outcome regression.

	Sample Treatment				Generate Treatment
	% Bias	SE	RMSE	Bias:SE	% Bias	SE	RMSE	Bias:SE
$n=100$	$60.424$	$2.829$	$2.904$	$0.232$	$50.114$	$2.712$	$2.766$	$0.200$
$n=1000$	$4.189$	$0.228$	$0.232$	$0.199$	$1.323$	$0.229$	$0.229$	$0.063$
$n=10000$	$0.780$	$0.071$	$0.072$	$0.118$	$0.104$	$0.071$	$0.071$	$0.016$

Abbreviations: glmCM, generalized linear model correct model; glmPS, generalized linear model propensity score; IPTW, inverse probability of treatment weighted; logcOR, log conditional odds ratio; MSM, marginal structural model; RMSE, root mean squared error; SE, standard error; TMLE, targeted maximum likelihood estimation; Unadj, unadjusted.

5.3.4. Scenario 5: KPWA Plasmode Simulation Results

Results for the ATE and RR are in Table 6 and Table S6 for the 5% and 15% outcomes, respectively. Other estimands are in Tables S7 and S8. Results are largely similar to those for the purely synthetic data; however, there are two main differences. First, TMLE has a larger bias:SE for the Sample Treatment than the Generate Treatment approach. While in the 15% outcome, the bias:SE is of the expected order of magnitude, it is larger than expected under Sample Treatment for the 5% outcome. Second, coverage is near the nominal level for all estimators besides the unadjusted estimator, regardless of plasmode strategy. These differences may be due to the complex regression models (including interactions with rare categorical variables), and the small number of events. In Figure 2 we show the distribution of correlations between treatment and each covariate over Monte‐Carlo replicates, which are nearly identical for the two plasmode approaches and indicate that sampling $W$ then generating $A$ does not affect the ability of plasmode simulations to capture correlation in the original data.

TABLE 6.

Results of the KPWA‐based plasmode simulations for the 5% outcome (ATE and RR).

Estimand	Estimator	Sample Treatment					Generate Treatment
		% Bias	SE	RMSE	Bias:SE	CP	% Bias	SE	RMSE	Bias:SE	CP
ATE	Unadj	10.964	0.004	0.010	2.130	43.4	11.191	0.004	0.010	2.169	41.8
	Match	0.403	0.005	0.005	0.064	94.9	−0.245	0.005	0.005	0.042	95.0
	IPTW	1.189	0.005	0.005	0.195	95.3	−0.219	0.004	0.004	0.042	95.1
	TMLE	0.571	0.005	0.005	0.096	95.1	0.012	0.004	0.004	0.002	95.1
	glmCM	−0.175	0.004	0.004	0.034	95.3	−0.182	0.004	0.004	0.036	95.1
	glmPS	−2.553	0.004	0.004	0.519	91.9	−2.874	0.004	0.004	0.586	90.9
RR	Unadj	−20.563	0.011	0.017	1.166	77.9	−20.875	0.011	0.017	1.188	77.3
	Match	−3.705	0.019	0.019	0.123	95.6	−1.548	0.018	0.018	0.054	95.5
	IPTW	0.328	0.016	0.016	0.013	95.2	0.340	0.016	0.016	0.014	95.2
	TMLE	−0.555	0.016	0.016	0.022	95.2	0.062	0.016	0.016	0.002	95.1
	glmCM	0.362	0.014	0.014	0.016	95.1	0.326	0.014	0.014	0.014	95.1
	glmPS	4.675	0.014	0.015	0.201	94.6	5.558	0.015	0.015	0.237	94.3

Note: Results for n = 10 000 ($1/\sqrt{n}=0.01$) with 100 000 Monte‐Carlo iterations. The true value of the ATE is −0.079; the true value of the RR is 0.062.

Abbreviations: ATE, average treatment effect; Bias:SE, ratio of bias to standard error; CP, coverage probability; glmCM, generalized linear model, correctly specified; glmPS, generalized linear model, adjusted for propensity score; IPTW, inverse probability of treatment weighting; KPWA, Kaiser Permanente Washington; Match, propensity score matching; RMSE, root mean squared error; RR, relative risk; SE, standard error; TMLE, targeted maximum likelihood estimation; Unadj, unadjusted.

FIGURE 2.

Boxplots of the correlation between treatment and each covariate in the KPWA plasmode simulations. The outcome rate is given in the rows. The boxplots are over the 100 000 Monte Carlo replications. Color denotes the Sample Treatment (light gray) versus Generate Treatment (dark gray) plasmode approach. Age sq., age squared; ANX, anxiety disorder; AUD, alcohol use disorder; Charlson, Charlson comorbidity index; Dx, diagnosis code; MH Hosp, hospitalization with mental health diagnosis code; PHQ8, Patient Health Questionnaire (PHQ) total score; PHQ9, PHQ ninth item; SH, self‐harm.

6. Discussion

When addressing causal questions about treatments using health care databases, analyses often need to address several complexities, including missing data, confounding, and potentially heterogeneous treatment effects. Plasmode simulation is a valuable and practical tool to compare available analytical methods under realistic conditions in order to select the best performing method for the setting at hand; however, care must be taken to use an appropriate plasmode approach.

We showed that the Sample Treatment plasmode framework leads to a biased evaluation of propensity‐score based methods for a binary point treatment. This is because this sampling strategy creates an empirical distribution that does not adequately approximate the data generating distribution with respect to the treatment mechanism. We provided both theoretical justification and numerical evidence demonstrating this bias happens for the Sample Treatment but not Generate Treatment framework. This result readily generalizes to other inverse‐probability of censoring and treatment‐weighted estimators.

Simulation studies were generally performed under correct model specifications to emphasize that the observed discrepancies were driven by underlying deficiencies in the plasmode framework, not by complexities in the data‐generating process or shortcomings of a particular analytical procedure. We showed the Generate Treatment framework had good numerical performance across all scenarios, with the demonstrated expected asymptotic regularity for all estimators under study. Differences between the two plasmode frameworks were more notable for continuous versus binary outcomes, more rare binary outcomes, and for continuous exposures. In some simplified settings, such as those with large samples, only a few binary confounders and a relatively common binary outcome, the observed empirical estimates of the data generating distribution would likely well represent the truth underlying data distribution. In this setting, one would expect there to be little difference between the performance of the two plasmode sampling approaches; however, in practice it would be difficult to reliably predict if one was in a scenario where the Sample Treatment bias was negligible, as it would depend on the prevalence of the binary outcomes and confounders, the number of confounders, and sample size. The numerical studies presented here were done under a limited number of settings. Other issues related to the performance of plasmode simulation may arise in other specific settings. For example, Stolte et al. [26] note that some challenges were observed in plasmode simulation when evaluating classifiers in certain high‐dimensional settings, which could be exacerbated by the type of resampling method used for the plasmode simulation. Further numerical studies would likely be warranted in specialized settings whose inference goal or data characteristics differ greatly from those already studied.

Finally, we note that plasmode datasets that preserve the underlying structure of real‐world data are most relevant for characterizing estimator performance. We recommend avoiding simplistic parametric models for the DGM by flexible modeling or machine learning. These models can be used to generate computer‐generated treatments and outcomes after re‐sampling covariates. We showed in a plasmode simulation based on real‐world data that the Generate Treatment approach preserved complexity present in the original data. Plasmode simulation studies conducted in this manner are grounded in theory and expected to provide a reliable basis for evaluating and comparing estimator performance.

Funding

This project was supported by Task Order 75F40123F19006 under Master Agreement 75F40119D10037 from the US Food and Drug Administration (FDA) and the U.S. National Institutes of Health (NIH) grant R37‐AI131771.

Conflicts of Interest

The authors declare no conflicts of interest.

Supporting information

Supplementary section S1: Mathematical details for the manuscript Section 4: Theoretical problems for the Sample.

Supplementary section S2: Additional details for the synthetic data plasmode simulations.

Supplementary section S3: Additional set‐up details for the electronic health records‐based plasmode simulation.

Supplementary section S4: Supplemental Tables and Figures for the electronic health records‐based plasmode simulations.

Table S1: Simulation Scenario 1a: RCT, estimate ATE for continuous outcome, n = 1000.

Table S2: Simulation Scenario 2a: RCT, estimate ATE for binary outcome, n = 1000.

Table S3: Simulation Scenario 3: Estimate RR for rare binary outcome, n = 10 000.

Table S4: Simulation Scenario 4a: Estimate logcOR when MSM equivalent to true outcome regression.

Figure S1: Nominal coverage of 95% confidence intervals for (a) Simulation Scenario 1: continuous outcome and (b) Simulation Scenario 2 binary outcome.

Figure S2: Estimator bias and coverage under each sampling scheme for simulation Scenario 1a: Continuous outcome in a randomized controlled trial setting (n = 1000).

Figure S3: Estimator bias and coverage under each sampling scheme for simulation Scenario 2a: Binary outcome in a randomized controlled trial setting (n = 1000).

Figure S4: Simulation Scenario 3 (rare binary outcome) Coverage for RR.

Table S5: Logistic regression model coefficients fit to the original KPWA dataset with 50 337 individuals.

Table S6: Results of the KPWA‐based plasmode simulations for the 15% outcome (ATE and RR).

Table S7: Results of the KPWA‐based plasmode simulations for the 15% outcome (EY1, EY0, and logcOR).

Table S8: Results of the KPWA‐based plasmode simulations for the 5% outcome (EY1, EY0, and logcOR).

Data Availability Statement

Code used to generate the synthetic (computer‐generated) data can be obtained by contacting the first author or by going to the GitHub site https://github.com/PamelaShaw/PlasmodeSimulation/. Other data used in this manuscript may be obtained after a data use agreement is in place. For more information contact pamela.a.shaw@kp.org.