A Viable Alternative when Propensity Scores Fail: Evaluation of Inverse Propensity Weighting and Sequential G-estimation in a Two-Wave Mediation Model

Abstract

Two methods from the potential outcomes framework—inverse propensity weighting (IPW) and sequential G-estimation—were evaluated and compared to linear regression for estimating the mediated effect in a two-wave design with a randomized intervention and continuous mediator and outcome. Baseline measures of the mediator and outcome can be considered confounders of the follow-up mediator – outcome relation for which adjustment is necessary to eliminate bias. To adjust for baseline measures of the mediator and outcome, IPW uses stabilized inverse propensity weights whereas sequential G-estimation uses regression adjustment. Theoretical differences between the models are described, and Monte Carlo simulations compared the performance of linear regression; IPW without weight truncation; IPW with weights truncated at the 1^st/99^th, 5^th/95^th, and 10^th/90^th percentiles; and sequential G-estimation. Sequential G-estimation performed similarly to linear regression, but IPW provided a biased estimate of the mediated effect, lower power, lower confidence interval coverage, and higher mean squared error. Simulation results show that IPW failed to fully adjust the follow-up mediator – outcome relation for confounding due to the baseline measures. We then compared the mediated effect estimates using data from a randomized experiment evaluating a steroid prevention program for high school athletes. Implications and future directions are discussed.

Statistical mediation analysis is used to investigate mechanisms through which a randomized intervention causally affects an outcome variable (Baron & Kenny, 1986; Lazarsfeld, 1955; MacKinnon, 2008). However, even when units are randomly assigned to levels of the intervention, the estimated mediated effect may be biased due to participants self-selecting their values on the mediator (Bullock, Green, & Ha, 2010; Holland, 1988; Imai, Keele, & Yamamoto, 2010; MacKinnon, 2008; MacKinnon & Pirlott, 2015; Pearl, 2001; Robins & Greenland, 1992). Even when participants are randomized to either a treatment or a control condition, they are not randomized to their level of the mediator. In this case, investigating the mediator – outcome relation is like conducting a non-randomized study of the relation of the mediator to the outcome (Holland, 1988; MacKinnon, 2008). In non-randomized studies it is important to include baseline covariates in an analysis to help reduce confounding (i.e., selection) bias (Imbens & Rubin, 2015; Rosenbaum, 2002). Theoretically-relevant variables or pretest measures are often good variables to include in an analysis to reduce confounding bias because of their tendency to be highly correlated with treatment and potential outcomes in observational studies (Steiner, Cook, Shadish, & Clark, 2010). Therefore, causal inference can be strengthened by the use of longitudinal data in general (Lepage, Lamy, Dedieu, Savy, & Lang, 2015; MacKinnon, 2008) and especially when investigating mediating processes (Cole & Maxwell, 2003; Gollob & Reichardt, 1991; MacKinnon, 2008; Maxwell & Cole, 2007).

The simplest longitudinal design that can be used to estimate the mediated effect of a randomized intervention on an outcome is the two-wave design and is often referred to as a “half-longitudinal” design (Cole & Maxwell, 2003). The strengths and limitations of traditional analysis of this design using analysis of covariance, difference scores, and residualized change scores, were described by MacKinnon (2008: Chapter 8). The two-wave design uses baseline measures of the mediator and outcome (i.e., measures collected prior to the randomization of participants to levels of an intervention) to adjust for time-invariant common causes of the mediator – outcome relation (Landau, Emsley, & Dunn, 2018; MacKinnon, 2008; Valente & MacKinnon, 2017). We propose here that the baseline measures of the mediator and outcome can be considered measured confounders of the follow-up mediator – outcome relation (i.e., follow-up relation between mediator and outcome) (Mayer, Thoemmes, Rose, Steyer, & West, 2014; VanderWeele, 2015) and that failure to adequately adjust for baseline measures of the mediator – outcome relation may result in biased mediated effect estimates.

Researchers may only have two waves of data—the baseline assessment and one follow-up assessment. For example, in a study of stimulant treatments for children with Attention-Deficit/Hyperactivity Disorder (ADHD), researchers collected baseline measures of cognitive functioning and classroom productivity prior to randomization of participants to a controlled trial involving stimulant medications commonly used to treat ADHD. After the controlled trial, follow-up measures of cognitive functioning and classroom productivity were assessed. Researchers found evidence of a mediated effect of stimulant medication on classroom productivity through its effect on cognitive functioning (Hawk et al., 2018). Longitudinal designs with multiple follow-up waves are generally better suited for establishing temporal precedence of the mediator – outcome relation assuming the correct timing of the mediating process was observed (Cole & Maxwell, 2003; Collins & Graham, 2002; Gollob & Reichardt, 1991; MacKinnon, 2008; Reichardt, 2011). If researchers do not have more than one follow-up wave or have not measured an outcome that clearly occurs after the mediator, the two-wave design is better suited for estimating mediated effects than ignoring the baseline measures altogether although it still must be assumed that the mediator at posttest does occur prior to the outcome at posttest (MacKinnon, 2008; Valente & MacKinnon, 2017).

The potential outcomes framework for causal inference highlights the importance of confounder adjustment for unbiased estimation of mediated effects and has led to the development of many innovative statistical methods for confounder adjustment and mediated effect estimation, including inverse propensity weighting (IPW) and sequential G-estimation (Robins, 2000; VanderWeele, 2009; VanderWeele & Vansteelandt, 2009). IPW and sequential G-estimation are flexible methods used to adjust for baseline confounders, adjust for follow-up confounders affected by treatment, estimate time-varying intervention effects, and estimate mediated effects (Robins, 2000). For example, IPW has recently been applied to evaluate time-varying effects of grade retention in observational studies (Reshetnyak, Cham, & Hughes, 2016; Steiner, Park, & Kim, 2016; Vandecandelaere, Vansteelandt, De Fraine, & Van Damme, 2016), which makes it important to evaluate the finite sample characteristics of this statistical method.

The performance of IPW and sequential G-estimation adjustment for follow-up confounders of the mediator – outcome relation has been investigated elsewhere (Coffman & Zhong, 2012; Goetgeluk, Vansteelandt, & Goetghebur, 2009; Kisbu-Sakarya, MacKinnon, Valente, & Cetinkaya, 2019; Vansteelandt, 2009). However, no simulation study has investigated the finite sample performance of these methods for adjusting for baseline confounders, including baseline status on the mediator and outcome in the widely-used two-wave mediation model with a randomized intervention. Focusing on this simple longitudinal design allows comparison of the performance of IPW and sequential G-estimation to that of linear regression, which has been extensively evaluated in previous literature on mediation analysis. Although IPW and sequential G-estimation are flexible confounder adjustment methods that can be used in complex models (e.g., adjusting for time-varying confounders), they may not perform well in simpler models such as adjusting for baseline measures of the mediator and outcome in a two-wave mediation model. It is important to understand the limitations of these potential outcomes-based methods in finite samples and in the simple case of baseline adjustment in mediation models because of the growing use of these cutting-edge methods in the social sciences.

This research combines separate research literatures on mediation from the regression and potential outcomes frameworks (VanderWeele, 2015) that have rarely been integrated (MacKinnon, 2008), applies them to the estimation of mediated effects in a two-wave design with a randomized intervention, and extends recent work by Valente and MacKinnon (2017), and Landau et al. (2018). Valente and MacKinnon (2017) and Landau et al. (2018) demonstrated adjusting for baseline status on the mediator and outcome through linear regression (referred to as analysis of covariance in MacKinnon, 2008) provides an unbiased estimate of the mediated effect. However, they did not evaluate the statistical performance of IPW or sequential G-estimation – two methods that can be used for baseline adjustment. The purpose of this paper is threefold.

1. Present an empirical example and explain why the baseline measures of the mediator and outcome are confounders of the follow-up mediator-outcome relation.

2. Extend previous simulation work on the two-wave mediation model with randomized intervention by describing and then comparing modern causal inference methods, IPW and sequential G-estimation, to linear regression for estimating mediated effects adjusted for baseline values of the mediator and outcome.

3. Apply these methods to the Athletes Training and Learning to Avoid Steroids dataset (Goldberg et al., 1996) in two empirical examples.

First, an empirical example is presented. Second, the two-wave mediation model and related regression equations are described in general and then in terms of the empirical example. Third, a simulation study is presented to compare the performance of IPW, sequential G-estimation, and linear regression. Fourth, two examples are presented to compare IPW, sequential G-estimation, and linear regression in an empirical dataset. Fifth, a discussion and summary of the overall performance of IPW, sequential G-estimation, and linear regression are presented.

Empirical Example

The Athletes Training and Learning to Avoid Steroids (ATLAS; Goldberg et al., 1996) program was designed to reduce high school football players’ use of anabolic steroids compared to a control group by engaging students in healthy nutrition and strength training alternatives. MacKinnon et al. (2001) investigated 12 mediators of the ATLAS program on three outcomes: intentions to use anabolic steroids, nutrition behaviors, and strength training self-efficacy. It was hypothesized that the ATLAS program increased players’ strength training self-efficacy through its effect on their perceptions of the football team as an information source. Perceptions of their team as an information source (i.e., the mediator) was measured by items such as “Being on the football team teaches players about getting stronger,” and strength training self-efficacy (i.e., the outcome) was measured by items such as “I know how to train with weights to become stronger.” The mediator and outcome were measured prior to randomization to the ATLAS or control condition and immediately following completion of these programs. Therefore there were baseline measures and follow-up measures of the mediator and outcome variables. The purpose of this example is to explain model parameters, creation of propensity scores, and consequences of failing to adjust for baseline values of the mediator and outcome in the following sections.

Two-Wave Mediation Model with a Randomized Intervention

A Directed Acyclic Graph (DAG) of the two-wave mediation model is presented in Figure 1A – B. The DAG in Figure 1A – B encodes the assumed conditional (in)dependencies in the two-wave mediation model nonparametrically and thus does not make any assumptions about the possible linear or additive nature of the model (Pearl, 2009). Figure 1A describes the causal relation between the randomized intervention (X) and the outcome at time 2 (Y₂) (total effect). Figure 1B describes the causal relation between the randomized intervention (X) and the outcome at time 2 (Y₂) adjusted for the mediator at time 2 (M₂) (direct effect), mediator at time 1 (M₁), and outcome at time 1 (Y₁); Figure 1B also describes the causal relation between the mediator at time 2 (M₂) and the outcome at time 2 (Y₂) adjusted for the randomized intervention (X), mediator at time 1 (M₁) and outcome at time 1 (Y₁). The potential outcomes framework is used to formally define the direct and indirect effects for the DAG presented in Figure 1A – B.

Figure 1.

(a) Displays the total effect of randomized intervention assignment, X, on the follow-up measure of the outcome (Y₂). (b) Displays the two-wave mediation model with a randomized intervention with baseline measures of the mediator and outcome (M₁ and Y₁, respectively), a randomized intervention assignment, X, and follow-up measures of the mediator and outcome (M₂ and Y₂, respectively). (c) Displays the pseudo-population that is created via inverse propensity weighting. Inverse propensity weighting removes the effects of M₁ on M₂, and Y₁ on M₂ for estimation of a weighted direct effect in the pseudo-population. (d) Displays the removal of the effect of M₂ on Y₂ adjusted for the other variables in the model (M₁, Y₁, and X) that simulates how sequential G-estimation removes the effect of the mediator on the outcome for estimation of an adjusted direct effect

The mediated (indirect) and direct effects in the potential outcomes framework are defined as the difference between two (sometimes unobserved) potential outcomes (Pearl, 2001; Robins & Greenland, 1992). The natural indirect effect at time 2 (NIE; Equation 1) is the difference between the potential outcome Y₂ had the intervention variable X been fixed at a specific level x (e.g., either 0 or 1 when X represents a randomized intervention) and M₂ been fixed to the level it would have been had X been at level 1 (denoted $Y_2\left(x,M_{2\left(1\right)}\right)$) and the potential outcome Y₂ had X been at level x and M₂ been fixed to the level it would have been had X been at level 0 (denoted $Y_2\left(x,M_{2\left(0\right)}\right)$).

$$NIE=E\left[Y_2\left(x,M_{2\left(1\right)}\right)-Y_2\left(x,M_{2\left(0\right)}\right)\right]$$ (1)

The natural direct effect at time 2 (NDE; Equation 2) is the difference between the potential outcome Y₂ had X been at level 1 and M₂ been fixed to the level it would have been had X been at level x (denoted $Y_2\left(1,M_{2\left(x\right)}\right)$) and the potential outcome Y₂ had X been at level 0 and M₂ been fixed to the level it would have been had X been at level x (denoted $Y_2\left(0,M_{2\left(x\right)}\right)$).

$$NDE=E\left[Y_2\left(1,M_{2\left(x\right)}\right)-Y_2\left(0,M_{2\left(x\right)}\right)\right]$$ (2)

The controlled direct effect at time 2 (CDE; Equation 3) is the effect of X on Y₂ at a fixed level m₂ of M₂. Formally, it is the difference between the potential outcome Y₂ had X been at level 1 and M₂ been fixed to level m₂ (denoted $Y_2\left(1,m_2\right)$) and the potential outcome Y₂ had X been at level 0 and M₂ been fixed to level m₂ (denoted $Y_2\left(0,m_2\right)$).

$$CDE=E\left[Y_2\left(1,m_2\right)-Y_2\left(0,m_2\right)\right]$$ (3)

Causal Assumptions for Mediated Effects

VanderWeele and Vansteelandt (2009) described the following four no-unmeasured-confounding assumptions necessary to nonparametrically identify the CDE and the mediated effect (Imai et al., 2010):

1. No unmeasured confounders of the relation between X and Y₂ conditional on measured baseline confounders.

2. No unmeasured confounders of the relation between M₂ and Y₂ conditional on X and measured baseline confounders.

3. No unmeasured confounders of the relation between X and M₂ conditional on measured baseline confounders.

4. The intervention X does not affect any measured or unmeasured confounder of the relation between M₂ and Y₂ conditional on measured baseline confounders.

Only assumptions 1 and 2 are needed to identify the CDE. It is assumed we have the correct temporal order of the mediator and outcome and the stable unit treatment value assumption holds (SUTVA; Imbens & Rubin, 2015). SUTVA states that there are no hidden variations of treatments and no interference between units; the potential mediator values for each unit do not depend on the treatment status of other units, and the potential outcomes of each unit do not depend on the treatment status or mediator value of other units (Imai, Keele, & Tingley, 2010).

In terms of the empirical example, we assume that conditional on the baseline measures of perception of team as an information source and strength training self-efficacy (M₁ and Y₁), the effect of the ATLAS program (X) on the follow-up measure of strength training self-efficacy (Y₂) and the effect of the follow-up measure of perception of team as an information source (M₂) on the follow-up measure of strength training self-efficacy (Y₂) are not confounded. We also assume we have observed the correct temporal order (i.e., changes in perception of team as an information source precede changes in strength training self-efficacy). Finally, we assume that the potential values of perception of team as an information source at follow-up for each participant do not depend on the treatment status (ATLAS program or control) of the other participants and the potential values of strength training self-efficacy at follow-up for each participant do not depend on the treatment status or potential follow-up values of perception of team as an information source of other participants.

Linear Models and Assumptions

Next, we will assume specific linear models with additive effects to demonstrate how the nonparametric definitions of the potential outcomes framework simplify to specific regression coefficients assuming linear and additive effects.

$$Y_2=i_4+c_{y2x}X+e_4$$ (4)

$$M_2=i_5+a_{m2x}X+s_{m2m1}{M}_1+b_{m2y1}{Y}_1+e_5$$ (5)

$$Y_2=i_6+{c^{\prime }}_{y2x}X+s_{y2y1}{Y}_1+b_{y2m1}{M}_1+b_{y2m2}{M}_2+e_6$$ (6)

Equation 4 represents the total effect of X on Y₂ (c_y2x coefficient; c path). Equation 5 represents the effect of X on M₂ (a_m2x coefficient; a path) adjusted for the baseline measures of the mediator (s_m2m1 coefficient) and outcome (b_m2y1 coefficient). Equation 6 represents the direct effect of X on Y₂ (c’_y2x coefficient; c’ path) and the effect of M₂ on Y₂ (b_y2m2 coefficient; b path) adjusted for the baseline measures of the mediator and the outcome (see Table 1 for a description of each model parameter and Valente & MacKinnon, 2017 for a description of how the single mediator model relates to the two-wave mediation model). i₄, i₅, and i₆ are the respective intercepts and e₄, e₅, and e₆ are the respective regression residuals defined as Y₂-E[Y₂|X], M₂-E[M₂|X,M₁,Y₁], and Y₂-E[Y₂|X,M₂,M₁,Y₁], respectively. Although interaction effects can be included in these models, we assume linearity and additivity. In linear regression analysis, it is typical to assume normally distributed errors when conducting significance testing of the model parameters in Equations 4 – 6 but this assumption can be relaxed and bootstrapping can be used for significance testing of the model parameters. The mediated effect in this model can be equivalently estimated as the product a_m2xb_y2m2 or the difference c_y2x - c’_2yx.

Table 1.

Description of model parameters in Equations 4 – 6 applied to the ATLAS example

Equation	Coefficient	Interpretation
Equation 4	cy2x	Total effect of the ATLAS program on strength training self-efficacy at follow-up.
Equation 5	am2x	Effect of the ATLAS program (X) on perceptions of team as an information source at follow-up (M2) adjusted for effects of perceptions of team as an information source and strength training self-efficacy at pretest (M1 andY1).
Equation 5	sm2m1	Temporal stability of perceptions of team as an information source from baseline to follow-up. Measures the extent to which participants’ perceptions of their team as an information source remain constant over time absent any external influences. High values of the sm2m1 coefficient correspond to a temporally stable mediator.
Equation 5	bm2y1	Cross-lagged effect from strength training self-efficacy at baseline to perceptions of team as an information source at follow-up. Measures the extent to which perceptions of team as an information source at follow-up are affected by strength training self-efficacy at baseline.
Equation 6	c’y2x	Direct effect of the ATLAS program (X) on strength training self-efficacy not through perceptions of team as an information source at follow-up adjusted for baseline values of perceptions of team as an information source and strength training self-efficacy (M1 and Y1).
Equation 6	by2m2	Effect of perceptions of team as an information source at follow-up on strength training self-efficacy at follow-up adjusted for the effect of the ATLAS program (X) and baseline values of perceptions of team as an information source and strength training self-efficacy (M1 and Y1).
Equation 6	sy2y1	Temporal stability of strength training self-efficacy from baseline to follow-up. Measures the extent to which participants’ strength training self-efficacy remain constant over time absent any external influences. High values of the sy2y1 coefficient correspond to a temporally stable outcome.
Equation 6	by2m1	Cross-lagged effect from perceptions of team as an information source at baseline to strength training self-efficacy at follow-up. Measures the extent to which strength training self-efficacy at follow-up is affected by perceptions of team as an information source at baseline.

Assuming linearity, additivity, and no post-treatment confounders affected by X, the potential outcomes-based estimator of the indirect effect (i.e., NIE) is equivalent to the traditional estimator of the mediated effect, a_m2xb_y2m2 = c_y2x – c’_y2x, and the potential outcomes-based estimators of the direct effects (i.e., NDE and CDE) are equivalent to the traditional estimator of the direct effect, c’_y2x (VanderWeele, 2015; VanderWeele & Vansteelandt, 2009). Therefore, assuming the linear models described in Equations 4 – 6 hold, the mediated effect can be estimated as either a_m2xb_y2m2 or as the total effect minus the CDE (c_y2x – CDE). We specifically focus on the c_y2x – CDE estimate of the indirect effect because that corresponds to how the indirect effect is estimated using sequential G-estimation.

To be clear, the assumptions of linear and additive relations, and normally distributed errors are statistical modeling assumptions and not causal assumptions (De Stavola, Daniel, Ploubidis, & Micali, 2015). A researcher must decide what models (parametric or nonparametric) they want to use to estimate the difference between potential outcomes as described in Equations 1 – 3. Statistical assumptions will vary depending on what modeling approach the researcher chooses but the causal assumptions remain invariant to different modeling approaches.

Baseline Measures of the Mediator and Outcome as Confounders

The baseline measures of the mediator and outcome (M₁ and Y₁) can be considered confounders of the follow-up mediator - outcome (M₂ – Y₂) relation because they are both common causes of the follow-up mediator and outcome (Figure 1B; X is also a common cause of M₂ and Y₂ but is not considered a confounder because it serves as a focal predictor). First, the baseline mediator variable is a confounder of the follow-up mediator – outcome relation via the temporal stability of the mediator (i.e., M₁ effect on M₂; s_m2m1) and the cross-lagged effect of past mediator values on follow-up outcome values (i.e., M₁ effect on Y₂; b_y2m1). Second, the baseline outcome variable is a confounder of the follow-up mediator - outcome relation via the temporal stability of the outcome (i.e., Y₁ effect on Y₂; s_y2y1) and the cross-lagged effect of past outcome values on follow-up mediator values (i.e., Y₁ effect on M₂; b_m2y1). In the absence of the effect of past mediator values on follow-up outcome values and the effect of past outcome values on follow-up mediator values (i.e., cross-lagged effects), the baseline mediator and outcome variables form a joint confounder of the follow-up mediator - outcome relation via the temporal stabilities of the mediator (i.e., s_m2m1) and outcome (i.e., s_y2y1) and the relation between the baseline values of the mediator and outcome variables (i.e., b_m1y1) (Mayer et al., 2014). For a statistical method to adequately adjust for the confounding effects of the baseline measures on the follow-up mediator – outcome relation, it must follow one of the strategies below:

1. Remove the effect of the baseline measures of the mediator and outcome on the follow-up mediator variable (i.e., stability of the mediator and the cross-lagged effect of Y₁ on M₂).

2. Remove all effects of the baseline measures on both the follow-up mediator and follow-up outcome variables.

By removing the confounding effects of the baseline measures on the follow-up mediator – outcome relation, we are attempting to balance follow-up mediator scores on the baseline measures. In other words, we are attempting to make the follow-up mediator – outcome relation reflect the relation between the mediator and outcome had participants started out as equivalent on the mediator and outcome at baseline. In summary, failure to control for prior levels of the mediator and outcome may lead to biased estimates of mediated effects because the relation at posttest may be confounded by the prior levels of the mediator and outcome.

In terms of the empirical example, if we could imagine randomizing participants to their observed value of perception of team as information source at follow-up conditional on their treatment assignment, we would notice no systematic differences across values of perceptions of team as information source at follow-up. Because we cannot randomize participants to their observed value of perceptions of team as an information source at follow-up and thus cannot ensure there are no systematic differences across values at follow-up, we attempt to balance participants on their baseline values of perception of team as information source and strength training self-efficacy. That is, we want to ensure that participants who reported high values of perception of team as an information source at follow-up are not disproportionally participants that already started out with high values on perceptions of team as an information source and high values of strength training self-efficacy at baseline.

Linear regression removes the effects of the baseline measures from both the follow-up mediator and the follow-up outcome variables. Including the baseline measures of the mediator and outcome variables in the follow-up outcome regression equation (i.e., Equation 6) removes the variance in the follow-up outcome that is predictable from the baseline measures and removes the baseline effects from the follow-up mediator variable by partialling out the shared variance between the baseline measures and the follow-up mediator variable. We demonstrate in later sections that IPW removes the effects of baseline measures of the mediator and outcome on the follow-up mediator variable and sequential G-estimation removes the effects of the baseline measures from both the follow-up mediator and the follow-up outcome variables.

Because of the algebraic equivalence of the mediated effect a_m2xb_y2m2 = c _y2x – CDE under linearity, additivity, and correctly specified models (i.e., no measured or unmeasured post-treatment confounders affected by X) any confounders biasing the relation between the follow-up mediator – outcome relation (i.e., b_y2m2 path) will also bias the direct effect of the randomized intervention on the outcome (i.e., c’_y2x path). Therefore, unbiased estimation of the mediated effect in the two-wave mediation model with a randomized intervention requires adjustment for baseline status on the mediator and outcome.

Valente and MacKinnon (2017) assessed the finite sample performance of the linear regression, difference score, residualized change score, and cross-sectional models for estimating the mediated effect in the two-wave mediation model with a randomized intervention. However, they did not assess the finite sample performance of emerging methods from the potential outcomes framework for causal inference, including IPW and sequential G-estimation. There exists little information regarding the finite sample performance of either of these methods in general and in particular when the mediator is continuous and confounders of the mediator – outcome relation are measured at baseline.

Inverse Propensity Weighting

IPW is an application of propensity scores which have been described in detail for causal treatment effect estimation (Austin, 2011; Hill, Weiss, & Zhai, 2011; Imbens & Rubin, 2015; Rosenbaum & Rubin, 1983) and for mediation analysis with a binary mediator (Imai, Jo, & Stuart, 2011; Jo, Stuart, MacKinnon, & Vinokur, 2011). VanderWeele (2009) described IPW of marginal structural models (i.e., weighted regression equations) for mediation analysis. The weights for IPW are used to adjust for the influence of confounders in a weighted regression analysis to estimate mediated and direct effects that are free of the confounding effects of measured confounders. For continuous variables or continuous mediators, it is important to estimate stabilized inverse propensity weights (Robins, Hernán, & Brumback, 2000). Stabilized weights consist of a ratio of two propensity scores and ensure finite sampling variability of the IPW estimator (Naimi, Moodie, Auger, & Kaufman, 2014; Robins, 2000; Robins et al., 2000). When applied to the two-wave mediation model with randomized intervention, IPW balances participants’ follow-up mediator values (M₂) on baseline values of the mediator and outcome (M₁ and Y₁) by reweighting the sample of participants to represent a pseudo-population for which the effects of the baseline measures on the follow-up measure of the mediator have been removed. IPW is conducted following these steps:

1. Estimate the predicted probability of M₂ for each participant conditional on their observed level of the intervention, f(M₂|X = x), where x refers to the observed level of the intervention (X) and f refers to the probability density function of a continuous random variable. For example, the observed level x of X would be equal to 0 for a participant in the control condition. This is referred to as the numerator propensity score for use in Step 3.

2. Estimate the predicted probability of M₂ for each participant conditional on their observed level of the intervention and observed values on the baseline mediator and outcome variables, f(M₂|X = x, M₁ = m₁, Y₁ = y₁), where m₁ and y₁ refer to observed levels of the baseline mediator and outcome, respectively. This is referred to as the denominator propensity score for use in Step 3.

3. Form a ratio of the estimated probabilities from Step 1 and Step 2. That is, create the weight for each participant i,$w_i=\frac{f\left(M_2|X=x\right)}{f\left(M_2|X=x,M_1=m_1,Y_1=y_1\right)}$. Participants’ weights are equal to the inverse of the probability of receiving the M₂ score that the participant actually received.

4. Conduct a weighted regression analysis using the estimated weight from Step 3 and regressing the follow-up outcome (Y₂) on the randomized intervention variable (X) and follow-up mediator variable (M₂).

For example, assume we have observed positive relations among all variables and a participant reported a high value on perception of team as an information source at follow-up, meaning they strongly perceived their football team as an information source. Assume the predicted probability that participant A responded this way was .50 because they had responded with a high value on perceptions of team as an information source and strength training self-efficacy at baseline. Focusing on the denominator of the IPW and assuming the numerator is equal to 1, the weight for participant A is 1/.50 which equals 2. Now assume participant B reported a high value at follow-up but their predicted probability of reporting a high value at follow-up was .20 because they had reported low values of perception of team as an information source and strength training self-efficacy at baseline. The IPW for participant B is 1/.20 which equals 5. Participant B is theoretically treated as 5 participants at follow-up whereas participant A is theoretically treated as 2 participants at follow-up to account for the fact that participants who reported high values at follow-up but did not report high values at baseline are underrepresented in the sample. By weighting participants we ensure that participants who reported high values of perception of team as an information source at follow-up are not disproportionally participants that already started out as high on perceptions of team as an information source and strength training self-efficacy at baseline.

Assuming no interaction between the randomized intervention and the mediator at follow-up (i.e., no XM₂ interaction), the mediated effect is estimated as either the difference between the total effect in Equation 4 and the weighted direct effect in Step 4 (c_y2x - c’_{y2x(weighted)}) or the product of a_m2x from Equation 5 and the weighted effect of M₂ on Y₂ from Step 4 b_{y2m2(weighted)} (a_m2xb_{y2m2(weighted)}). Significance testing can be performed by using percentile bootstrapping to create confidence intervals for the mediated effect. Figure 1C displays what the pseudo-population looks like after IPW. The paths from M₁ to M₂ (s_m2m1 from Equation 5) and from Y₁ to M₂ (b_m2y1 from Equation 5) are removed via the denominator propensity score, resulting in a pseudo-population free of the part of the follow-up mediator that is predictable from temporally stable characteristics on the mediator variable and free of the part of the follow-up mediator variable that is predictable from past values of the outcome variable. In other words, IPW removes the baseline effects from the follow-up mediator variable. Therefore, IPW should provide unbiased estimates of follow-up mediator – outcome relation (i.e., b_y2m2 from Equation 5) and the direct effect of the randomized intervention on Y₂ (i.e., c’_y2x path from Equation 6).

Because the follow-up mediator variable is continuous, the numerator and denominator propensity scores are drawn from the normal probability density function (Robins et al., 2000) but may be drawn from other continuous probability density functions if normality is violated (Naimi, et al., 2014). In expectation, the size of the pseudo-population (i.e., the weighted sample) created via stabilized weights equals the size of the observed sample because the numerator ensures that the expected value of the stabilized weight equals one (Hernán & Robins, 2006). It is important to note two properties of the numerator propensity score. First, the numerator propensity score does not play a role in removing the confounding effects of the baseline measures of the mediator and outcome on the follow-up mediator. The numerator propensity score is meant to reduce the sampling variability of the IPW estimator. The conditional probability (i.e., f(M₂|X = x)) was chosen because (1) it results in smaller sampling variability than the marginal probability of M₂ (i.e., f(M₂)) and (2) we are not interested in removing the effect of X on M₂ and Y₂ because X is our focal predictor and not a confounder (Cole & Hernán, 2008; Hernán & Robins, 2006). Second, the numerator propensity score helps to ensure that the weighted sample equals the size of the original sample. That is, the expected value of the weights should equal one unless the propensity score models are misspecified or there was improper adjustment of measured confounders (Cole & Hernán, 2008). Regarding the denominator propensity score, we assume the propensity score model is correctly specified (e.g., relevant confounders are included) and there is a nonzero probability of receiving every level of the mediator at follow-up (i.e., positivity assumption; Robins, 2000; Robins et al., 2000).

Finally, extremely small or extremely large weights increase the sampling variability of the IPW estimator (Cole & Hernán, 2008; Hernán & Robins, 2006). The size of the weights are influenced by the distributional form of the mediator (e.g., continuous mediator; Robins et al., 2000) and by the strength of the predictors of the continuous mediator. Stronger predictors lead to extreme weights more often than weaker predictors (Goetgeluk et al., 2009; Robins et al., 2000; Vansteelandt, 2009). To remedy extreme weights, Cole and Hernán (2008) suggested truncation such that extremely large weights (e.g., > 10) are set equal to the 99^th, 95^th, or 90^th percentile of the distribution of weights and extremely small weights (e.g., < 0.01) are set equal to the 1^st, 5^th, or 10^th percentile of the distribution of weights. Although truncating weights reduces the sampling variability of the IPW estimator, it also increases bias (Cole & Hernán 2008).

In summary, the magnitude of stabilized IPW weights depends on the strength of the predictors of the propensity scores and on the distributional form of the variable (in our case, the mediator at follow-up). Stabilized weights that are larger than 1.00 on average may be indicative of improper adjustment of the confounders included in the propensity score model resulting in biased estimates of the CDE and consequently the mediated effect.

Sequential G-Estimation

Sequential G-estimation was developed to adjust for baseline and follow-up confounders (Moerkerke, Loeys, & Vansteelandt, 2015; Robins, 2000; Vansteelandt, 2009). In the two-wave mediation model, sequential G-estimation adjusts for the confounding effects of the baseline measures of the mediator and outcome through regression adjustment using the following steps:

1. Regress Y₂ on the intervention (X), follow-up mediator (M₂), and baseline measures of the mediator and outcome (M₁ and Y₁), as shown in Equation 6.

2. Save the estimated regression coefficient relating M₂ to Y₂ from Equation 6 (b_y2m2).

3. For each participant, subtract the quantity b_y2m2M₂ from their observed value of Y₂. That is, compute Y_2diff = Y₂ – b_y2m2M₂ for each participant.

4. Regress this residualized outcome variable, Y_2diff, onto the randomized intervention variable, X, to estimate the adjusted direct effect (i.e., c’_{y2x(adjusted)}). This is the G-estimate of the direct effect of the randomized intervention on the follow-up outcome.

The rationale is that once the effect of the follow-up mediator on the follow-up outcome is removed, only a direct effect remains (Goetgeluk et al., 2009; Vansteelandt, 2009). In other words, if the effect of the perception of team as an information source at follow-up is removed from strength training self-efficacy at follow-up, only a direct effect of the ATLAS program on strength training self-efficacy at follow-up will remain. Assuming no interaction between the randomized intervention and follow-up mediator (i.e., XM₂ interaction), the mediated effect can be estimated as the difference between the total effect from Equation 4 and the G-estimate of the direct effect from Step 4 (c_y2x – c’_{y2x(adjusted)}). Significance testing can be performed by using percentile bootstrapping to create confidence intervals for the mediated effect.

Figure 1D graphically depicts sequential G-estimation. Residualizing the follow-up outcome, Y₂, removes the effect of the follow-up mediator, M₂, adjusted for the randomized intervention effect and baseline values of the mediator and outcome, thus resulting in only a direct effect of the randomized intervention on the follow-up outcome, Y₂. Because the follow-up mediator – outcome relation was adjusted for the baseline measures of the mediator and outcome in Equation 6, the remaining direct effect will not be biased by the confounding effects of the baseline measures. In other words, sequential G-estimation is similar to regression adjustment and removes the effects of the baseline mediator and outcome from both the follow-up mediator and the follow-up outcome variables.

Summary

Researchers may consider using linear regression, IPW, or sequential G-estimation when adjusting for baseline confounders in the two-wave mediation model for a few reasons. First, IPW is used to adjust exposure effects in observational studies for measured baseline confounders (Austin, 2011; Imbens & Rubin, 2015) and IPW was described as a method to adjust for baseline confounders of the mediated effect in the single mediator model (VanderWeele, 2009). Second, sequential G-estimation was demonstrated in the context of baseline confounder adjustment for cross-sectional treatment effects (Dukes & Vansteelandt, 2018; Vansteelandt & Joffe, 2014) and is advocated in place of IPW when the mediator variable is continuous due to instability of the propensity weights (Vansteelandt, 2009). Third, linear regression, IPW, and sequential G-estimation are all relatively comparable in the steps taken to estimate the respective models when adjusting for baseline confounders in the linear two-wave mediation model. Researchers may therefore consider using any of these methods when adjusting for baseline measures of the mediator and outcome in the two-wave design.

For the linear regression estimate of the mediated effect, we assume Equations 4 – 6 are correctly specified implying linearity, additivity, and no post-treatment confounding. As is common, we also assume the error terms are mutually independent and are normally distributed. Percentile bootstrapping can be used for statistical inference of any of the regression parameters in Equations 4 – 6 or for the direct and mediated effect to avoid assuming the error terms in Equations 4 – 6 are normally distributed. For IPW, we assume that the propensity score model for the follow-up mediator is correctly specified. We assume that any confounding of the pretest measures of M and Y will be fully adjusted for through the propensity score model for the follow-up mediator. We do not assume the error term of the weighted outcome model follows a normal distribution but we do assume the propensity scores follow a normal distribution and that the probabilities in the denominator of the weights are nonzero.¹ For sequential G-estimation, we assume the outcome regression equation in Step 1 is correctly specified and the residualized outcome equation in Step 4 is correctly specified. Additionally, we assume that the models in Step 1 and Step 4 are congenial (see Vansteelandt, 2012) meaning any variables that are included to reduce confounder bias of the CDE in the outcome regression equation in Step 1 are also included to reduce confounder bias of the CDE in the residualized outcome regression equation in Step 4. In our case of randomized X, the pretest measures of M and Y only need to be included in the outcome regression equation in Step 1 because it is assumed that the pretest measures of M and Y only affect the follow-up measures of M and Y and not X (by design). We do not make any assumptions about the distribution of the error term of the residualized outcome model in Step 4.

Purpose

Previous simulation studies have either focused on a limited number of factors affecting the performance of IPW for continuous variables (e.g., Naimi et al., 2014), the performance of IPW for improving the precision of randomized treatment effects (Williamson, Forbes, & White, 2014), or adjustment for follow-up confounders using IPW or sequential G-estimation (Coffman & Zhong, 2012; Goetgeluk et al., 2009; Kisbu-Sakarya et al., 2019; Vansteelandt, 2009). For example, Naimi et al. (2014) investigated the performance of several IPW estimators for a continuous variable, but only for one sample size (N = 1,500), one set of effect sizes for predictors of the outcome model, one set of effect sizes for predictors of the propensity model, and not for mediated effects. The other simulation studies did not examine the finite sample performance of IPW and/or sequential G-estimation when adjusting for baseline confounders of the mediator – outcome relation, which is the focus of this paper.

We investigated the following four hypotheses regarding the finite sample performance of IPW without weight truncation; IPW with weights truncated at the 1^st/99^th, 5^th/95^th, and 10^th/90^th percentiles; sequential G-estimation; and linear regression.

1. The IPW estimators of the mediated effect will exhibit more finite sample bias when there are large effects of the baseline measures of the mediator and outcome on the follow-up measures of the mediator and outcome (i.e., the stabilized weights are expected to be larger than 1.00 on average).

2. The untruncated IPW estimator of the mediated effect will have higher sampling variability than the truncated IPW estimators and sequential G-estimation but will be less biased than the truncated IPW estimators of the mediated effect.

3. The sequential G-estimator of the mediated effect will be unbiased compared to IPW estimators for all sample sizes and will perform similarly to linear regression because it uses regression adjustment to remove the baseline effects of the mediator and outcome.

4. The linear regression estimator of the mediated effect will be unbiased and have the highest statistical power across the three methods.

Method

Simulation

SAS 9.4 was used to conduct Monte Carlo simulations. The following equations represent the linear regression model used to generate the data where x is an observed value of X and $\tilde{x}$ is the sample median.

$$X~N\left(0,1\right):\left(x\ge \tilde{x}\right)=1;\left(x<\tilde{x}\right)=0$$ (7)

$$M_1~N\left(0,1\right)$$ (8)

$$Y_1=b_{y1m1}{M}_1+e_1$$ (9)

$$M_2=a_{m2x}X+b_{m2y1}{Y}_1+s_{m2m1}{M}_1+e_2$$ (10)

$$Y_2={c^{\prime }}_{y2x}X+b_{y2m1}{M}_1+b_{y2m2}{M}_2+s_{y2y1}{Y}_1+e_3$$ (11)

$${\sigma }_{ej}^2~N\left(0,1\right)$$ (12)

$${\sigma }_{eiej}=0$$ (13)

The simulation data that were used in Valente and MacKinnon (2017) were used for this study. In addition to the simulation data used from Valente and MacKinnon (2017) new simulation data were generated for a sample size of 1,000. Data for a sample size of 1,000 were added to demonstrate the asymptotic trend of the mediated effect estimators. The factors varied were: sample size (N = 50, 100, 200, 500, 1000); effect size of the a_m2x (0, .14, .39, .59), b_y2m2 (0, .14, .39, .59), and c’_y2x (0, .39) paths; effect size of the Y₂ cross-lagged path b_y2m1 (0, .50) and M₂ cross-lagged path b_m2y1 (0, .50); stability (i.e., correlation coefficient) of the mediating variable (s_m2m1) and outcome variable (s_y2y1) (.30, .70); and relation between M₁ and Y₁ (0, .50). All residual terms (e₁, e₂, and e₃) had a standard deviation of one, were uncorrelated with each other and the predictors, and were defined as Y₁ - E[Y₁|M₁], M₂ - E[M₂|X,M₁,Y₁], and Y₂ - E[Y₂|X,M₂,M₁,Y₁], respectively. The effect sizes were chosen to reflect approximately small, medium, and large effect sizes (Cohen, 1988). A full factorial design produced 2,560 conditions, each with 1,000 replications. Percentile bootstrapping with 500 bootstrap samples per simulation replication was used to test the statistical significance of the mediated effect for each model. The following models were compared: IPW without weight truncation; IPW with weights truncated at the 1^st/99^th, 5^th/95^th, and 10^th/90^th percentiles; and sequential G-estimation.

Evaluation Criteria

For each of the 2,560 conditions, sampling variability of the IPW and sequential G-estimators was estimated as the standard deviation of the parameter estimates across the 1,000 replications per condition.

Relative bias was computed by subtracting the true value of the parameter from the parameter estimate, and then dividing this value by the true value of the parameter. An estimator of the mediated effect was considered acceptable in terms of bias if the absolute value of relative bias was less than .10 across replications. When the true mediated effect equaled zero, raw bias was computed (i.e., parameter estimate minus the true value of the parameter).

Mean squared error (MSE) was computed by subtracting the true value of the parameter from the parameter estimate, squaring this value, and then adding the variance of the parameter estimate. MSE combines bias and precision of the estimator in one index. Low values of MSE reflect either low bias, high precision, or some combination of the two.

Significance testing was performed using the percentile bootstrap within each replication. For each replication, 500 bootstrap samples were created and the mediated effect was estimated for each bootstrap sample. The mediated effect was deemed significant if zero was not contained between the 2.5^th and 97.5^th percentile in each replication. The Type 1 error rate was defined as the proportion of replications where an estimate of the mediated effect was statistically significant at the .05 alpha level when the true value of the parameter was equal to zero. Consistent with Bradley’s (1978) liberal criterion, Type 1 error rates were deemed acceptable if they fell within the range of [.025, .075].

Power was the proportion of times across the 1,000 replications per condition an estimate of the mediated effect was statistically significant at the .05 alpha level when the true value of the parameter was not equal to zero. The best performing estimator in terms of statistical power had the highest statistical power within a given condition. Coverage was the proportion of times the true value of the mediated effect fell within the percentile bootstrap confidence intervals. Confidence intervals were deemed acceptable if the interval contained the true value of the mediated effect within the range of [.925, .975].

For each replication, we computed the mean weight from IPW across the N cases. We then averaged the mean weight across the 1,000 replications within each design cell. Because stabilized weights were used, the mean weight should equal 1. A mean weight greater (less) than 1 indicates that the size of the pseudo-population is greater (less) than the size of the observed sample.

For each evaluation criterion except sampling variability, the simulated data were analyzed using analysis of variance. The data set analyzed consisted of 2,560,000 observations (2,560 conditions ✕ 1,000 replications). For sampling variability, the data were aggregated across the 1,000 replications for each condition, resulting in a data set with 2,560 observations. Main and interaction effects that resulted in statistical significance (p < .05) with a semi-partial η² greater than or equal to .01 were considered important predictors (see supplemental materials for tables of all significant predictors of simulation outcomes).

Results

Type 1 Error Rates

For the sequential G-estimator and the linear regression estimator, Type 1 error rates never fell above the upper bound of the robustness interval (i.e., .075) or fell below the lower bound of the robustness interval (i.e., .025) except when a_m2x was zero and b_y2m2 was zero or 0.14; therefore, these results were not reported in tables. For the IPW estimators, Type 1 error rates did not exceed the upper bound of the robustness interval when b_y2m2 was nonzero and a_m2x was zero but exceeded the upper bound of the robustness interval when b_y2m2 was zero and a_m2x was nonzero (Table 2).

Table 2.

Type 1 error rates for IPW estimators when the b path was zero and the a path varied

			Stability
			0.3				0.7
			IPW	IPW 99	IPW 95	IPW 90	IPW	IPW 99	IPW 95	IPW 90
M2 Cross-lag	Y2 Cross-lag	a Path
0	0	0	0.032	0.029	0.025	0.023	0.056	0.045	0.040	0.037
		0.14	0.037	0.033	0.032	0.032	0.057	0.048	0.052	0.065
		0.39	0.045	0.044	0.047	0.055	0.073	0.069	0.120	0.200
		0.59	0.052	0.051	0.057	0.063	0.085	0.092	0.192	0.278
	0.5	0	0.037	0.033	0.029	0.027	0.061	0.048	0.044	0.043
		0.14	0.041	0.039	0.076	0.163	0.068	0.062	0.112	0.196
		0.39	0.054	0.065	0.221	0.420	0.129	0.165	0.439	0.606
		0.59	0.057	0.077	0.283	0.480	0.205	0.284	0.608	0.766
0.5	0	0	0.053	0.044	0.038	0.035	0.058	0.048	0.042	0.041
		0.14	0.056	0.049	0.053	0.068	0.068	0.059	0.078	0.116
		0.39	0.069	0.070	0.118	0.210	0.117	0.132	0.290	0.438
		0.59	0.081	0.087	0.173	0.290	0.177	0.226	0.462	0.610
	0.5	0	0.055	0.044	0.039	0.036	0.062	0.046	0.043	0.045
		0.14	0.061	0.057	0.098	0.175	0.076	0.075	0.145	0.225
		0.39	0.105	0.127	0.343	0.534	0.179	0.248	0.543	0.678
		0.59	0.141	0.194	0.482	0.659	0.312	0.430	0.733	0.863

Note. All Type 1 error rates outside of the robustness interval [.025, .075] are bolded and underlined.

IPW = Inverse propensity weighting with no weight truncation, IPW-99 = Inverse propensity weighting with weights truncated at the 1^st/99^th percentiles, IPW-95 = Inverse propensity weighting with weights truncated at the 5^th/95^th percentiles, IPW-90 = Inverse propensity weighting with weights truncated at the 10^th /90^th percentiles.

IPW with weights truncated at the 90^th percentile had the highest Type 1 error rates followed by IPW with weights truncated at the 95^th percentile, weights truncated at the 99^th percentile and IPW without weight truncation. Generally, IPW estimators had inflated Type 1 error rates for higher values of the baseline correlation, higher stability, cross-lagged paths, a path, and sample size. The highest Type 1 error rate occurred when the baseline correlation = 0.50, stability = 0.70, M₂ cross-lag = 0.50, Y₂ cross-lag = 0.50, a path = 0.59, and N = 1,000.

Sampling Variability

Table 3 displays the sampling variability for the IPW estimators, the sequential G-estimator, and for comparison, the linear regression estimator aggregated over sample size and true value of the mediated effect. IPW without weight truncation had the highest sampling variability across conditions (see supplemental materials for additional results). The sampling variability for all estimators increased as the true value of the mediated effect increased and the sampling variability decreased as sample size increased. Sampling variability for the IPW estimators increased for increasing values of stability, cross-lags, and baseline correlation.

Table 3.

Sampling variability of IPW estimators, sequential G-estimator, and Linear regression estimator

				IPW	IPW 99%	IPW 95%	IPW 90%	Seq-G	Reg
Baseline Corr.	Stability	M2 Cross-lag	Y2 Cross-lag
0	0.3	0	0	0.020	0.019	0.015	0.014	0.013	0.012
			0.5	0.031	0.028	0.021	0.020	0.013	0.012
		0.5	0	0.041	0.035	0.023	0.020	0.015	0.012
			0.5	0.074	0.059	0.034	0.029	0.015	0.013
	0.7	0	0	0.097	0.074	0.037	0.028	0.018	0.013
			0.5	0.180	0.124	0.060	0.049	0.018	0.013
		0.5	0	0.130	0.093	0.049	0.039	0.020	0.013
			0.5	0.257	0.164	0.075	0.065	0.020	0.012
0.5	0.3	0	0	0.020	0.019	0.016	0.015	0.013	0.013
			0.5	0.036	0.032	0.023	0.022	0.013	0.013
		0.5	0	0.062	0.049	0.029	0.025	0.017	0.012
			0.5	0.145	0.102	0.050	0.043	0.017	0.012
	0.7	0	0	0.125	0.092	0.047	0.038	0.018	0.012
			0.5	0.289	0.192	0.083	0.070	0.018	0.012
		0.5	0	0.216	0.141	0.072	0.061	0.025	0.013
			0.5	0.465	0.276	0.122	0.108	0.025	0.013

IPW = Inverse propensity weighting with no weight truncation, IPW-99 = Inverse propensity weighting with weights truncated at the 1^st/99^th percentiles, IPW-95 = Inverse propensity weighting with weights truncated at the 5^th/95^th percentiles, IPW-90 = Inverse propensity weighting with weights truncated at the 10^th /90^th percentiles, Seq-G = sequential G-estimation, Reg = linear regression.

Relative Bias

Because of the large sampling variability of the IPW estimators, semi-partial η² values greater than or equal to 0.001 (rather than 0.01) were used as the significance criterion for relative bias. There were no significant predictors of the relative bias of the sequential G-estimator. The relative bias of the IPW estimators increased as the baseline correlation, stability, and cross-lagged paths increased, but relative bias decreased as the effect size of the mediated effect and the sample size increased (each ${\eta }_p^2\ge 0.001$). IPW without truncation resulted in the smallest magnitude of relative bias across all conditions and resulted in relative bias less than 0.10 for all magnitudes of the mediated effect when (1) baseline correlation = 0.00 and stability = 0.30 and either the M₂ or Y₂ cross-lags = 0.50 but not when both cross-lags = 0.50, (2) when baseline correlation = 0.00 and stability = 0.70 and both cross-lags = 0.00, and (3) when baseline correlation = 0.50, stability = 0.30, M₂ cross-lag = 0.00 and 0.50, and Y₂ cross-lag = 0.00. IPW with weights truncated at the 99^th, 95^th, and 90^th percentiles resulted in higher relative bias than IPW without truncation.

Mean Squared Error

MSE for the sequential G-estimator was higher for higher levels of stability and increased as a function of effect size of the mediated effect and decreased as sample size increased. Generally, the MSE of the IPW estimators increased as baseline correlation, stability, and cross-lagged paths increased. Of IPW estimators, IPW with weights truncated at the 95^th percentile had the lowest MSE; its MSE was close to the MSE of the sequential G-estimator and linear regression.

Power

There was a two-way interaction of sample size and effect size of the mediated effect for IPW with weights truncated at the 95^th percentile and 90^th percentile and the sequential G-estimator (each ${\eta }_p^2\ge 0.01$). Additionally, stability was a significant predictor of power for IPW without weight truncation and for IPW with weights truncated at the 99^th percentile $\left({\eta }_p^2\ge 0.01\right)$. Power was lower for IPW without weight truncation and with weights truncated at the 99^th percentile when stability was high vs. low. Of the IPW estimators, IPW with weights truncated at the 90^th percentile had the highest power.

Confidence Interval Coverage

The results for confidence interval coverage were similar to those for relative bias (figure not shown). There were no significant predictors of confidence interval coverage for the sequential G-estimator. Generally, the confidence interval coverage of the IPW estimators decreased as baseline correlation, stability, and cross-lagged paths increased. The IPW estimator that had confidence interval coverage closest to [.925, .975] was IPW without truncation. The only significant predictor of confidence interval coverage for IPW without truncation was stability.

Weight Results

Stability and M₂ cross-lag were both significant predictors of mean-level untruncated weights and weights truncated at the 99^th percentile $\left({\eta }_p^2\ge 0.01\right)$. Predictors of mean-level weights truncated at the 95^th and 90^th percentile included stability, baseline correlation, M₂ cross-lag and two-way interactions of baseline correlation by M₂ cross-lag and stability by M₂ cross-lag $\left({\eta }_p^2\ge 0.01\right)$. Stability explained close to 50% of the variance of weights truncated at the 95^th and 90^th percentile (${\eta }_p^2=0.49$ and ${\eta }_p^2=0.43$, respectively) and M₂ cross-lag explained over 15% of the variance of weights truncated at the 95^th and 90^th percentile (${\eta }_p^2=0.18$ and ${\eta }_p^2=0.16$, respectively). Truncation of the weights resulted in lower mean weights than the untruncated weights but still resulted in mean weights greater than 1.00 when stability = 0.70, baseline correlation = 0.50, and M₂ cross-lag = 0.50.

The mean weights were affected by factors that appeared in the denominator of the weights but not in the numerator of the weights. Significant predictors of the mean weights (with and without truncation) were stability, baseline correlation, and M₂ cross-lag (Table 4). The only conditions for which the mean weights (with and without truncation) were close to 1.00 occurred when stability = 0.30, baseline correlation = 0.00, and M₂ cross-lag = 0.00. When stability = 0.70, baseline correlation = 0.50, and M₂ cross-lag = 0.50, the mean untruncated weight was 1.647 and truncation at the 90^th percentile reduced the mean weight to 1.133.

Table 4.

Average untruncated weight (Wt), weight truncated at 99^th percentile (Wt-99), weight truncated at 95^th percentile (Wt-95), and weight truncated at 90^th percentile (Wt-90) with minimum and maximum weights in parentheses

		M2 Cross-lag
		0				0.5
		Wt	Wt-99	Wt-95	Wt-90	Wt	Wt-99	Wt-95	Wt-90
Baseline Corr.	Stability	1.057 (0.313, 4.465)	1.051 (0.419, 2.810)	1.030 (0.614, 1.671)	1.018 (0.725, 1.388)	1.200 (0.133, 16.383)	1.167 (0.210, 6.130)	1.095 (0.419, 2.455)	1.058 (0.566, 1.783)
0	0.3
	0.7	1.385 (0.080, 41.629)	1.295 (0.137, 10.141)	1.158 (0.330, 3.135)	1.096 (0.484, 2.088)	1.480 (0.068, 59.121)	1.352 (0.119, 12.124)	1.183 (0.305, 3.396)	1.110 (0.460, 2.199)
0.5	0.3	1.057 (0.313, 4.467)	1.051 (0.419, 2.816)	1.030 (0.614, 1.671)	1.018 (0.725, 1.388)	1.316 (0.094, 30.912)	1.250 (0.156, 8.690)	1.136 (0.355, 2.906)	1.083 (0.508, 1.988)
	0.7	1.387 (0.080, 41.603)	1.297 (0.136, 10.259)	1.158 (0.330, 3.136)	1.096 (0.484, 2.088)	1.647 (0.055, 94.982)	1.445 (0.098, 15.550)	1.221 (0.273, 3.793)	1.133 (0.429, 2.364)

Note. Minimum and maximum weights refer to the average over the 1,000 replications per design cell.

Summary of Simulation Results

The performance of the sequential G-estimator was very similar to that of the linear regression estimator presented in Valente and MacKinnon (2017). The sequential G-estimator was unbiased; had no elevated Type 1 error rates; and had similar power, coverage, sampling variability, and MSE as the linear regression estimator. The sequential G-estimator outperformed the IPW estimators because it does not rely on indirect confounder adjustment via weighting. Instead, sequential G-estimation adjusts for confounders through regression. The bias and sampling variability of the IPW estimators performed as expected based on previous literature (Cole & Hernán, 2008). Truncating the weights resulted in greater bias but lower sampling variability. Among the IPW estimators, IPW without weight truncation resulted in the lowest relative bias, lowest Type 1 error rates, and most adequate confidence interval coverage; IPW with 5^th/95^th percentile weight truncation resulted in the lowest MSE; and IPW with 10^th/90^th percentile weight truncation resulted in the lowest sampling variability.

Empirical Examples

Data from the ATLAS program (Goldberg et al., 1996) were used to demonstrate and compare the mediated effect estimates using linear regression, IPW, and sequential G-estimation. The mediator and outcomes for the two examples were measured prior to randomization to the ATLAS or control condition (M₁ and Y₁) and immediately following completion of these programs (M₂ and Y₂). Seven baseline covariates were measured including age, grade point average (GPA), ethnicity, divorce status of parents, percentage of fathers who graduated college, percentage of mothers who graduated college, and family income. The seven baseline covariates included in this analysis are a subset of the 14 total baseline covariates that were originally hypothesized to be important demographic variables when assessing ATLAS program effects (Goldberg et al. 1996) and were chosen primarily to demonstrate how each of the three methods adjusts for baseline confounders. The model included the pretest measures of the mediator and the outcome and main effects of the seven baseline covariates. For linear regression, the baseline covariates were included in the M₂ and Y₂ equations, for sequential G-estimation the baseline covariates were included in the Y₂ equation in Step 1 as outlined earlier, and for IPW the baseline covariates were included in the denominator propensity score model for M₂. There were 659 observations used in this example after listwise deletion of the original 1,506 observations. The empirical example was used to apply these methods and substantive interpretation of model results should be approached with caution.

Empirical example 1 – Mediated effect adjusted for baseline covariates.

In the first example, the models tested students’ perception of their high school football team as an information source at follow-up as a mechanism through which the ATLAS program improved strength training self-efficacy at follow-up. The mediated effect estimate for linear regression was 0.220, 95% percentile bootstrap confidence interval (CI) [0.147, 0.302] and the mediated effect for sequential G-estimation was 0.279, 95% percentile bootstrap CI [0.176, 0.330]. The mediated effect estimate for IPW without weight truncation was 0.495, 95% percentile bootstrap CI [0.221, 0.905]; the mediated effect for IPW with 1^st/99^th percentile weight truncation was 0.358, 95% percentile bootstrap CI [0.216, 0.502]; the mediated effect for IPW with 5^th/95^th percentile weight truncation was 0.345, 95% percentile bootstrap CI [0.244, 0.454]; and the mediated effect for IPW with 10^th/90^th percentile weight truncation was 0.340, 95% percentile bootstrap CI [0.245, 0.439]. The mean of the weights used for IPW ranged from 1.192 to 1.017 for the untruncated weights to the 90^th percentile truncated weights, respectively. Overall, linear regression and sequential G-estimation provided similar results for the applied example. The IPW estimator provided the largest magnitude estimate with the widest confidence interval. As the weights for IPW were increasingly truncated, the magnitude of the mediated effect (generally) decreased, the confidence intervals became narrower, and the mean of the weights approached one. All methods resulted in a significant mediated effect.

A sensitivity analysis was conducted for the mediation model. The sensitivity analysis was conducted in R using the ‘mediation’ package (Imai et al., 2010a; Imai et al., 2010b; Tingley, Yamamoto, Hirose, Keele, & Imai, 2014). The sensitivity analysis varies the correlation between the residuals of the mediator and outcome equations to determine what magnitude of an unmeasured confounder effect (as indicated by correlated residuals) it would take to eliminate the observed mediated effect estimate. The correlation between the residuals of the mediator and outcome equations that it would take to reduce the estimated mediated effect to zero was 0.40 implying a moderately large unmeasured confounder effect to eliminate the observed mediated effect.

Empirical example 2 –Controlled direct effect on a binary outcome.

The mediator variable (i.e., perception of team as information source) and the baseline covariates remained the same for this empirical example. The outcome variable for the second empirical example was “intention to use anabolic androgenic steroids (AAS)”. In the original papers investigating the program effects and mediated effects of the ATLAS program (Goldberg et al., 1996; MacKinnon, et al., 2001), intention to use AAS was a continuous variable measured with items such as “I intend to try to use anabolic steroids” on a 1 (no-intention) to 7 (strong intention) scale. However, for this example, intention to use AAS was dichotomized at time 1 and time 2 as 0 = “Does not intend to use AAS” and 1 = “Intends to use AAS.” Intention to use AAS was dichotomized based on a median split of the time 1 scores such that substantive interpretation of these results should be approached with caution. The second empirical example was designed to demonstrate how linear regression, sequential G-estimation, and IPW are used to estimate the CDE when there is a binary outcome. This example provides additional information for researchers who may be solely interested in direct effects of treatments on outcomes and for researchers interested in modeling binary outcomes.

G-estimates of causal effects are typically obtained by solving estimating equations (see Vansteelandt, 2010, 2012) that are not currently implemented in most software packages (Dukes & Vansteelandt, 2018; Vansteelandt, 2010, 2012). However, it is possible under certain circumstances to use existing software to obtain G-estimates of causal effects and of the CDE (Vansteelandt & Joffe, 2014). When the outcome and mediator variables are continuous, it is possible to use existing linear regression software to obtain the G-estimate of the CDE (and subsequently the mediated effect) by following the four steps outlined earlier in this paper. Dukes and Vansteelandt (2018) recently demonstrated how to use preexisting generalized estimating equations (GEE) procedure to estimate the CDE when the outcome is either a count or a binary variable. Dukes and Vansteelandt (2018) proposed the following four steps to estimate the CDE when the outcome is either a count or a binary variable:

1. Fit a gamma generalized linear model with a log-link and regress Y₂ on the intervention (X), follow-up mediator (M₂), and baseline covariates.

2. Save the estimated regression coefficient relating M₂ to Y₂ from Step 1 (e.g., b_y2m2).

3. For each participant, compute Y_2diff = Y₂ exp(−b_y2m2M₂).

4. Fit a gamma generalized linear model with a log-link and regress Y_2diff onto the randomized intervention variable, X, to estimate the CDE on the log risk ratio scale.

These four steps essentially “trick” the GEE procedure into solving the sequential G-estimate estimating equation if the outcome is either a count or a binary variable (see Dukes & Vansteelandt, 2018 for technical details). This results in the CDE being interpreted on the log risk ratio or risk ratio scale.

The traditional direct effect (i.e., c’_y2x) was estimated as the effect of X on Y₂ from Step 1 above and the IPW estimates of the CDE were estimated as the weighted effect of X on Y₂ from Step 1 above. In this example we used the geeM package in R to estimate a gamma generalized linear model with a gamma distribution and a log-link for the binary outcome intention to use AAS, but it is also possible to use the “glm” command in Stata (see Dukes & Vansteelandt, 2018). The model contained the pretest measures of the mediator and the outcome and the seven baseline covariates described earlier. The results were reported on the log risk ratio scale.

The traditional direct effect was 0.182 with 95% percentile bootstrap CI [−0.113, 0.482], the sequential G-estimate of the CDE was 0.102 with 95% percentile bootstrap CI [−0.112, 0.311], the CDE for IPW without weight truncation was −0.032 with 95% percentile bootstrap CI [−0.485, 0.371], the CDE for IPW with weights truncated at the 1^st/99^th percentiles was 0.102 with 95% percentile bootstrap CI [−0.170, 0.349], the CDE for IPW with weights truncated at the 5^th/95^th percentiles was 0.106 with 95% percentile bootstrap CI [−0.124, 0.328], and the CDE for IPW with weights truncated at the 10^th/90^th percentiles was 0.119 with 95% percentile bootstrap CI [−0.103, 0.327]. The CDE was not statistically significant across any of the methods. IPW without weight truncation was the only model that resulted in a negative estimate of the CDE. The confidence intervals were the widest for IPW with no weight truncation and the confidence intervals were the narrowest for IPW with weights truncated at the 10^th/90^th percentiles followed by sequential G-estimation. The IPW estimator provided the largest magnitude estimate with the widest confidence interval. As the weights for IPW were increasingly truncated, the magnitude of the CDE (generally) increased, the confidence intervals became narrower, and the mean of the weights approached one.

Summary of empirical examples.

The mediated effect was statistically significant for all of the methods in the first empirical example and the CDE was not statistically significant for any of the methods in the second empirical example. IPW without weight truncation consistently resulted in the widest confidence intervals in both empirical examples. Unfortunately, the ‘mediation’ R package is not currently able to estimate sensitivity analyses for outcome models estimated using generalized estimating equations such that sensitivity analyses were not carried out for this example. Missing data were handled via listwise deletion out of ease of demonstration and because the regression techniques used to estimate the models (regression and propensity score models) automatically use only complete cases. In theory, these methods could be implemented in combination with a multiple imputation procedure to account for missing data. The use of listwise deletion is not an inherent property of the theory of IPW or sequential G-estimation but rather the computational procedures used to implement these methods (e.g., SAS REG procedure).

Discussion

Overall, IPW did not perform as well as either sequential G-estimation or linear regression across all simulation evaluation criteria. The IPW estimators had high sampling variability, high relative bias, high MSE, low confidence interval coverage, and low statistical power compared to sequential G-estimation and linear regression. The IPW estimators also resulted in elevated Type I error rates when the effect of X on M₂ was nonzero and the effect of M₂ on Y₂ was zero. In practice, this combination of effect sizes may be unlikely because mediators are often chosen as the target of interventions because they are correlated with the outcome (MacKinnon, 2008). When the effect of X on M₂ was zero and the effect of M₂ on Y₂ was nonzero, Type 1 error rates for the IPW estimators were within the robustness interval. Although IPW demonstrated finite sample bias for the sample sizes studied in this simulation, the finite sample bias approached acceptable levels of relative bias (i.e., 0.10) as sample size increased for large mediated effect sizes. IPW is a consistent estimator of the CDE and mediated effect; therefore, the finite sample bias is expected to approach zero as the sample size approaches infinity but sample sizes greater than 1,000 are rare in psychology. The sequential G-estimator performed similarly to the linear regression model presented in Valente and MacKinnon (2017) and in the empirical example.

Model Assumptions

Linear regression makes parametric assumptions of the X to M₂ and M₂ to Y₂ relations. This means all relations between the variables X, M₁, Y₁, M₂, and Y₂ must be correctly specified. Both sequential G-estimation and IPW make less parametric assumptions for this two-wave model than linear regression. IPW assumes that the weight model for M₂ is correctly specified but does not assume that the equation for Y₂ is correctly specified. IPW models are doubly-robust because if the weight model for M₂ is misspecified,(i.e., the M₁ or Y₁ to M₂ relation is not specified correctly) the baseline variables can be included in the outcome model for Y₂ as a second chance to remove all effects of the baseline measures on the M₂ – Y₂ relation. Doubly-robust IPW methods were not investigated in this paper because only baseline confounders of the M₂ – Y₂ relation were investigated and the inclusion of the same variables in the propensity score model and the final outcome model will result in estimates identical to linear regression (Robins, Sued, Lei-Gomez, & Rotnitzky, 2007). If researchers adjust for post-treatment confounders (i.e., confounders that occur after treatment and are possibly affected by treatment) in addition to baseline confounders, then it is recommended to include all confounders in the denominator propensity score model but only the baseline confounders in the final outcome model (Cole & Hernán, 2008).

Sequential G-estimation does not make parametric assumptions regarding the M₂ model or any potential post-treatment confounders (Moerkerke et al., 2015; Vansteelandt, 2009). Sequential G-estimation only requires that the observed outcome model for Y₂ is correctly specified, the residualized outcome model is correctly specified, and that the models are congenial (Vansteelandt, 2012). Model congeniality means that if there are confounders of the direct effect of X on Y₂ then these confounders must be included in both the observed outcome model (Step 1) and the residualized outcome model (Step 4) in order to reduce confounder bias. In our two-wave mediation model with randomized X, the pretest measures of the mediator and outcome were not correlated with X (by design); therefore, these pretest measures could have been included in the residualized outcome model to increase the statistical power to detect the CDE but were not necessary to reduce confounder bias. The effect of the addition of the baseline measures of M and Y into Step 4 was investigated and presented in the supplementary materials. The addition of the baseline measures of M and Y into Step 4 increased the statistical power to detect the CDE overall compared to not including the baseline measures and increased the statistical power especially when the stability of the mediator and outcome variables were high, there was a large baseline correlation, and the cross-lag path from the mediator at baseline to the outcome at follow-up was greater than zero (see supplemental materials for more details). In these ways, both IPW and sequential G-estimation provide more flexible alternatives to confounder adjustment than linear regression because of the fewer numbers of parametric modeling assumptions.

Why Did IPW Fail to Eliminate the Confounding Effects of M₁ and Y₁?

IPW removes the effects of the baseline measures of the mediator and outcome on the follow-up mediator via weighting observations such that the baseline measures are no longer common causes of the follow-up mediator – outcome relation. To the extent that IPW does not fully remove the effects of the baseline measures on the follow-up mediator in finite samples, there will be residual confounding of these baseline measures on the follow-up mediator – outcome relation which is exacerbated when weights are truncated. Residual confounding may be overcome by including the baseline measures of the mediator and outcome in the weighted regression model for follow-up outcome (i.e., Y₂) (Cole & Hernán, 2008; Kang & Schafer, 2007; Schafer & Kang, 2008). However, including these baseline measures in the outcome model obviates the need for inverse propensity weights, and the weighted IPW estimates of the predictors on Y₂ will be equivalent to linear regression estimates (Robins et al., 2007).

The sizes of the stabilized weights were affected by the effect size of the predictors of the mediator at follow-up (Goetgeluk et al., 2009; Robins et al., 2000; Vansteelandt, 2009). Cole and Hernán (2008) noted that the mean value of stabilized weights should equal one and that mean values higher than one indicate propensity score model misspecification or improper adjustment of relevant confounders. Because we simulated the data and know the true model for the mediator at follow-up, we know that the propensity score model was correctly specified yet the mean value of the weights was larger than one in many conditions. The denominator propensity score is a function of the randomized intervention and baseline measures of the mediator and outcome and is used to adjust for the confounding effects of the baseline measures. Large weights are caused by very small probabilities in the denominator propensity score, which occur when a participant’s observed value of the follow-up mediator substantially differs from his or her predicted value of the follow-up mediator. This is more likely to occur for large effect sizes of predictors on the follow-up mediator because the variance of the follow-up mediator increases as the effect size of the predictors increases. Therefore the fact that the mean values of the weights were larger than one in our simulation study implies IPW did not completely remove the effects of the baseline mediator and outcome on the follow-up mediator – outcome relation. This result has implications for when IPW can be applied with a continuous mediator and/or longitudinal data. With longitudinal data, the baseline measures often strongly predict the mediator, which appears to be problematic based on the simulations in this paper.

However, there are some benefits of IPW. IPW can be used when there are many potential confounders of the mediator – outcome relation and the functional form of these relations are unknown. Because confounders are adjusted for in a propensity model and not the outcome model, all possible main and interaction effects of the confounders can be included without complicating the interpretation of randomized intervention effects in the outcome model (Imbens & Rubin, 2015). IPW is also recommended over linear regression when there is minimal overlap on baseline confounders between non-randomized treatment groups in observational studies (Dukes & Vansteelandt, 2018; Rubin, 1997). Rubin (1997) demonstrated that if there are confounders of the effect of a non-randomized treatment on an outcome (i.e., the non-randomized treatment groups do not have overlapping distributions on the confounders) and there are differential effects of the non-randomized treatment on the outcome as a function of the confounders (i.e., the confounders moderate the effect of the non-randomized treatment on the outcome), then propensity score methods (e.g., IPW) would provide an unbiased estimate of the average treatment effect whereas linear regression would not. In the two-wave mediation model, this scenario would occur if (1) the baseline measures confounded the effect of perception of team as an information source at follow-up on strength training self-efficacy at follow-up and (2) the baseline measures moderated the effect of perception of team as an information source at follow-up on strength training self-efficacy at follow-up.

When there are measured post-treatment confounders of the mediator – outcome relation, linear regression requires more statistical modeling assumptions to estimate the CDE than either IPW or sequential G-estimation (Moerkerke et al., 2015; De Stavola et al., 2015). Linear regression requires a regression model for the post-treatment confounder and a model for the outcome to be correctly specified which includes how the treatment and any baseline confounders affect the post-treatment confounder and how the post-treatment confounder affects the outcome. IPW requires a model for the mediator to be correctly specified which includes how the post-treatment confounder affects the mediator. Sequential G-estimation requires a model for the outcome to be correctly specified which includes how the post-treatment confounder affects the outcome (Moerkerke et al., 2015; De Stavola, et al., 2015). Assuming no interactions or non-linear effects, the mediated effect can then be estimated as the total effect minus the CDE for all three methods. Therefore, in the presence of post-treatment confounding, both IPW and sequential G-estimation provide researchers with modeling choices that allow for the estimation of direct and indirect effects that make fewer statistical assumptions than linear regression and therefore become more appealing modeling choices in the case of post-treatment confounding.

Future Directions and Conclusions

Because both sequential G-estimation and IPW are able to adjust for time-varying confounders, it would be useful to investigate their performance in more complicated longitudinal mediation models such as when two or more waves of follow-up data for the mediator and outcome are collected and when there are time-varying confounders of these relations in addition to baseline confounders. Future analytical and simulation work might compare the performance of linear regression, IPW, and sequential G-estimation when there are violations of parametric assumptions of the mediator, outcome, and confounders. Both IPW and sequential G-estimation are more flexible than linear regression in this regard because neither IPW nor sequential G-estimation makes strict parametric assumptions.

In conclusion, both sequential G-estimation and IPW from the potential outcomes framework provide researchers with additional flexible alternatives for confounder adjustment when estimating mediated effects. This study provided insight into the finite sample performance of these methods when adjusting for baseline measures of the mediator and outcome in a two-wave longitudinal mediation model. Sequential G-estimation is a strong alternative to traditional methods such as linear regression for estimating the mediated effect. Although IPW may provide more flexible adjustment for types and numbers of confounders than linear regression, it does not perform well based on various statistical criteria in the case of baseline adjustment in the two-wave mediation model.

Figure 2.

Relative bias for IPW estimator with no weight truncation (solid line), sequential G-estimator (short dashed line), and linear regression (long dashed line). Sequential G-estimator ad linear regression results were nearly identical hence the lines describing the results of each estimator overlap. Reference lines at −0.10 and +0.10 mark bounds of relative bias.

Figure 3.

Mean squared errror for IPW estimator with weights truncated at the 95^th percentile (dotted line), sequential G-estimator (short dashed line), and linear regression (long dashed line).

Figure 4.

Power for IPW estimator with weights truncated at the 90^th percentile (dashed-dotted line), sequential G-estimator (short dashed line), and linear regression (long dashed line). Reference line at 0.80 marks nominal level of power.