Analyzing multiple mediators in multiple single-mediator models leads to wrong conclusions

Mediation analysis is widely used to evaluate causal mechanisms. For example, ref. 1 examined whether institutional criticism affected social media engagement via four mediators: anger, worry, surprise, or humor. The author concluded that the effects of criticism attacking the institution’s partisan integrity were fully mediated by anger (“P-values less than 0.002”) with no direct effects (“P-values substantially higher than 0.05”), as highlighted in the abstract.

I reanalyzed the publicly available data and found that anger did not fully mediate the effects. In fact, none of the putative mediators (four emotional responses) fully mediated any effect. All the mediated effect estimates were closer to zero, or even in the opposite direction, than originally suggested.

The key problem with the original mediation analysis was that multiple mediators were evaluated separately, in four single-mediator models. It has been well established that such an approach can lead to wrong conclusions (2, 3). This is because a variable (e.g., worry) causally intermediate between treatment and outcome can simultaneously be a confounder of a different (e.g., anger) mediator-outcome effect. Omitted intermediates can introduce biases that cannot be straightforwardly addressed using sensitivity analyses for unmeasured baseline confounding (4). But these intermediates cannot be merely included as baseline confounders for routine adjustment because they occur after treatment (5, 6). Hence, mediation analytic approaches that can account for confounding by causal intermediates should be used (2, 3). A recently proposed framework is interventional (in)direct effects for multiple mediators (7). Unlike conventional methods, interventional effects make no assumptions about the underlying causal structures or confounding patterns among the mediators (8).

Following recommended practices, I estimated interventional effects with the four emotional responses as multiple mediators, decomposing the total effect into a direct and four indirect effects simultaneously (9), in R (10). The results are shown in Fig. 1. Three observations can be made. First, the indirect effects after adjusting for other mediators were closer to zero, or even in the opposite direction, than from separate single-mediator models as originally used. Second, neither anger nor worry had an indirect effect categorically larger than other emotional responses, to the extent of fully mediating certain effects as originally concluded. Third, the direct effects—along causal pathways bypassing all these emotional responses—could have much larger magnitudes than the indirect effects. This implies that even after accounting for unmeasured confounding (4), the indirect effect could be zero while the direct effect remains nonzero; see Fig. 2 for an example.

Fig. 1.

Interventional indirect and direct effect estimates for the effect of institutional criticism on social media engagement with emotional responses of anger, worry, surprise (“s’prise”), or humor as multiple mediators. Point estimates and 95% bootstrap CIs are indicated by a filled circle and a horizontal line through the circle, respectively. For comparison, indirect effects for each emotional response, estimated using separate single mediation analyses, are indicated by a cross. Each panel corresponds to a combination of agency (AHRQ or CDC), experimental condition (Partisan or Nonpartisan), post type (Criticism or Rebuttal), and engagement outcome (Strength or Polarity), as stated in the heading. In each row, the agency and intervention are the same; in each column, the post type and engagement outcome are the same.

Fig. 2.

Example sensitivity analysis for unmeasured baseline mediator-outcome confounding illustrated using the effect of partisan criticism of the CDC on the strength of social media engagement. Estimates of the interventional indirect effect through anger (red circles) and the direct effect bypassing all emotional responses (black diamonds) are shown. The sensitivity parameter δ encodes an unmeasured confounder’s effect on the outcome, with a value of δ = 0 corresponding to the observed estimates (i.e., assuming no unmeasured confounding).

I commend (1) in using mediation analysis to investigate competing causal pathways. But separate single mediation analyses of multiple mediators can lead to wrong conclusions. I hope that this commentary encourages researchers to explore more suitable analytic methods for etiological investigations of multiple mediators toward more accurate inferences.

Data, Materials, and Software Availability

The R scripts to reproduce the mediation analyses, including for the Republican and Democratic subgroups, are available online: https://osf.io/ajc87/files/osfstorage?view_only=81df8e41fca7474f86db09c68537f899 (11).