MULTIVARIATE DYNAMIC MEDIATION ANALYSIS UNDER A REINFORCEMENT LEARNING FRAMEWORK

Abstract

Mediation analysis is an important analytic tool commonly used in a broad range of scientific applications. In this article, we study the problem of mediation analysis when there are multivariate and conditionally dependent mediators, and when the variables are observed over multiple time points. The problem is challenging, because the effect of a mediator involves not only the path from the treatment to this mediator itself at the current time point, but also all possible paths pointed to this mediator from its upstream mediators, as well as the carryover effects from all previous time points. We propose a novel multivariate dynamic mediation analysis approach. Drawing inspiration from the Markov decision process model that is frequently employed in reinforcement learning, we introduce a Markov mediation process paired with a system of time-varying linear structural equation models to formulate the problem. We then formally define the individual mediation effect, built upon the idea of simultaneous interventions and intervention calculus. We next derive the closed-form expression, propose an iterative estimation procedure under the Markov mediation process model and develop a bootstrap method to infer the individual mediation effect. We study both the asymptotic property and the empirical performance of the proposed methodology, and further illustrate its usefulness with a mobile health application.

1. Introduction.

Mediation analysis is an important analytic tool, which seeks to explain the mechanism or pathway that underlies an observed relationship between a treatment and an outcome variable, through the inclusion of an intermediary variable known as a mediator. It decomposes the effect of the treatment on the outcome into a direct effect and an indirect effect, the latter of which indicates whether the mediator is on a pathway from the treatment to the outcome (Baron and Kenny (1986)). Mediation analysis is widely employed in a range of scientific applications, including psychology (MacKinnon (2008), Rucker et al. (2011)), genomics (Huang and Pan (2016), Bi et al. (2017)), economics (Celli (2022)), social science (Kaufman and Kaufman (2001)), neuroscience (Zhao and Luo (2022)), among many others. See VanderWeele (2016) for a comprehensive review and the references therein.

Mediation analysis has seen considerable progress in recent years. There are two lines of research of particular interest, mediation analysis with multivariate mediators and mediation analysis with time-varying variables. The first line targets the scenario where there are multiple mediators, and one central goal is to evaluate and quantify the contribution of the treatment on the outcome attributed to each individual mediator. There are three main categories of solutions. One category explicitly imposes that the multivariate mediators are conditionally independent given the treatment, which substantially simplifies the analysis (Boca et al. (2014), Huang and Pan (2016), Zhang et al. (2016), Guo et al. (2023), Shuai et al. (2023), Yuan and Qu (2024)). Another category does not impose such a condition, but instead marginalizes each individual mediator, which in effect neglects any potential interaction and dependency among the mediators (Sampson et al. (2018), Djordjilović, Hemerik and Thoresen (2022), Zhao, Li and Initiative (2022), Zhao and Luo (2022)). The last category does allow correlated mediators, characterizes their dependency through some unknown directed acyclic graph (DAG), then carries out mediation analysis based on the estimated DAG (Maathuis, Kalisch and Bühlmann (2009), Chakrabortty, Nandy and Li (2018), Cai, Song and Lu (2021), Shi and Li (2022), Wei et al. (2024)). The second line targets the scenario where the treatment, mediator and outcome are observed over multiple time points or stages. There have been some pioneering works along this line (Selig and Preacher (2009), Preacher (2015), VanderWeele and Tchetgen Tchetgen (2017), Lin et al. (2017), Huang and Yuan (2017), Zheng and van der Laan (2017), Zhao et al. (2018), Hejazi et al. (2023), Cai et al. (2022), Díaz, Williams and Rudolph (2023), Ge et al. (2023)). Nevertheless, they all focus on the case where there is only a single mediator, and most only consider the case where the number of time points, or the time horizon is finite.

In this article, we study the problem of mediation analysis when there are multivariate and conditionally dependent mediators, and when the variables are observed over multiple time points. Our motivation is a mobile health application from the Intern Health Study (IHS, NeCamp et al. (2020)). It is a 26-week prospective longitudinal randomized trial targeting first-year training physicians in the United States (NeCamp et al. (2020)). A key objective of this study is to investigate the effectiveness of in-the-moment mobile prompts in improving an intern’s mood score while minimizing user burden and expense. The prompts, delivered through a customized mobile app, consist of practical tips and life insights, such as reminders to have a break, take a walk or prioritize sleep that aim at promoting healthy behaviors among interns. These prompts not only affect an intern’s mood, but also influence other physiological measurements, including physical activity, sleep duration and heart rate variability, which are closely associated with an individual’s mental state. We can naturally formulate this problem in the framework of mediation analysis, where the treatment is the binary variable that encodes receiving a prompt or not, regardless of the prompt type, the mediators consist of measurements of physical activities, sleep duration and heart rate variability, and the outcome is the mood score. Given the presence of multiple mediators and the longitudinal nature of the data, it calls for a mediation analysis approach that tackles both multivariate mediators and multiple time points.

However, the problem is challenging, for several reasons. First, multivariate and conditionally correlated mediators introduce considerable complexity. Unlike the single mediator or conditionally independent mediator case, the indirect effect of each individual mediator involves not only the path from the treatment to this mediator itself, but also all possible paths pointed to this mediator from its upstream mediators. The total number of potential paths that go through any mediator is superexponential in the number of mediators. It is thus crucial to carefully disentangle the individual mediation effect when the multivariate mediators are interrelated with unknown dependency. Second, time-varying variables introduce another layer of complexity, as there are carryover effects along time. Unlike the single time-point case, the mediator at a given time point could affect both the current and future outcomes, and thus it is crucial to learn both the immediate effect and the delayed effect. Infinite time horizon further complicates the analysis when the mediator effect accumulates over infinite time.

To address those challenges, we propose a novel multivariate dynamic mediation analysis approach. Our proposal consists of four key components. First, drawing inspiration from the Markov decision process (MDP, Puterman (1994)) model that is frequently employed in reinforcement learning (RL, Sutton and Barto (2018)) to handle the carryover effects over time, we introduce a Markov mediation process (MMP) framework to formulate the dynamic mediation analysis problem. We then introduce a system of time-varying linear structural equation models (SEMs) to specifically characterize the relations among the treatment, mediator and outcome variables. Second, within the framework of MMP, we formally define the individual mediation effect, which is built upon the idea of simultaneous interventions and intervention calculus (Pearl (2000)). This individual mediation effect can be further decomposed into a sum of the immediate effect and the delayed effect, the latter of which quantifies the carryover effects of the past treatments and mediators. Third, under the proposed MMP and the linear SEMs, we derive the closed-form expression for the individual mediation effect; see Theorems 1 and 2. These theorems form the basis of our proposed procedure, allow us to express the individual mediation effect using a set of within-stage and cross-stage intermediate quantities that can be estimated through some recursive formulations and consist of the main contributions of our proposal. We next propose to estimate those within-stage quantities through linear regressions with backdoor covariate adjustment, and estimate those cross-stage quantities through the transition equations under SEMs. Finally, we study the asymptotic property of our proposed estimator of the individual mediation effect, develop a bootstrap method to construct its confidence interval (CI) and conduct extensive numerical studies to demonstrate the effectiveness of our method. In summary, our approach allows us to separately evaluate the contribution of each individual mediator, where multiple mediators are interrelated with unknown dependency and are observed over multiple time points.

The rest of the article is organized as follows. We present the Markov mediation process, the structural equation models, and the definition of the individual mediation effect in Section 2. We derive the intermediate quantities, the recursive formulation of the individual mediation effect, and the estimation algorithms in Section 3. We establish the asymptotic properties in Section 3.5. We carry out the simulations in Section 4, and revisit the IHS example in Section 5. We relegate all technical proofs to the Supplementary Material (Luo et al. (2025)).

2. Model and definition.

In this section, we first present our model setup, then formally define the mediation effect of interest in the multivariate dynamic mediation analysis setting.

2.1. Markov mediation process and structural equation models.

We first introduce a Markov mediation process framework to formulate the problem we target; see Figure 1(b) for a graphical illustration. Consider the treatment-mediator-outcome triplets $\left\{\left(A_t,{\mathit{M}}_t,R_t\right):\right.t=1,\dots ,T\}$ over time, where $T$ denotes the number of time points or stages. At each time point or stage $t$, a random treatment $A_t\in \mathbb{R}$ is administered, which subsequently affects a $d$-dimensional vector of potential mediators ${\mathit{M}}_t\in {\mathbb{R}}^d$, and an outcome variable $R_t\in \mathbb{R}$. We assume they satisfy the Markov assumption, in that

$$\left(R_t,{\mathit{M}}_t\right)⫫{\left(R_s,{\mathit{M}}_s,A_{s+1}\right)}_{s<t-1}\mid \left(R_{t-1},{\mathit{M}}_{t-1},A_t\right)\text{for}\text{any}t\ge 1,$$ (1)

where $⫫$ denotes statistical independence. We remark the Markov condition like (1) is widely imposed in sequential data problems (Sutton and Barto (2018)). Suppose the observed data consists of $n$ independent and identically distributed (i.i.d.) realizations of the triplets $\left\{\left(A_t,{\mathit{M}}_t,R_t\right):t=1,\dots ,T\right\}$. We allow the data to be either densely or sparsely observed. Moreover, for simplicity, we assume that all subjects have the same $T$. Nevertheless, our proposed method can be adapted to accommodate the setting when $T$ varies among subjects, and when the time lags between two time points differ. See Section 6.4 for more details.

Fig. 1.

(a) Diagram of Markov decision process (MDP), where treatments depend on current states only, and ($S_t,A_t,R_t$) represents the state-treatment-reward triplet; (b) Diagram of the proposed Markov mediation process (MMP), where current mediators depend on previous mediator-reward pairs, and ($A_t,M_t,R_t$) represents the treatment-mediator-outcome triplet.

In addition to the Markov condition, we further assume that the random treatment assignment is independent of the prior information, in that

$$A_t⫫\left(A_s,{\mathit{M}}_s,R_s\right)\text{for}\text{any}s<t.$$ (2)

Condition (2) holds in the sequentially randomized trials naturally, including our IHS example, which is the main setting we target in this article. Meanwhile, our framework can also be extended to the setting that involves certain static baseline confounders that may influence the treatment selection, thus accommodating the data from observational studies. Specifically, letting $X$ denote the baseline confounders, Condition (2) can be relaxed to $A_t⫫\left(A_s,{\mathit{M}}_s,R_s\right)\mid X$, for any $s<t$. The individual mediation effect that we define later can be similarly derived by incorporating $X$ into the model. For presentation simplicity, however, we choose not to include those confounders in this article.

We next introduce a system of time-varying structural equation models,

$$\left({\mathit{M}}_t-{\mathit{\mu }}_t\right)={\mathit{W}}_t^{\top }\left({\mathit{M}}_t-{\mathit{\mu }}_t\right)+{\mathit{ϵ}}_{wt},$$ (3)

where ${\mathit{\mu }}_t\equiv \mathbb{E}\left({\mathit{M}}_t\mid A_t,{\mathit{M}}_{t-1},R_{t-1}\right)$ is the condition mean function, ${\mathit{W}}_t\in {\mathbb{R}}^{d\times d}$ is the weight matrix, such that ${\left({\mathit{W}}_t\right)}_{ij}\ne 0$ if and only if mediator $M_{ti}$ is a parent of $M_{tj}$, that is, $M_{ti}$ is in the parent set, $M_{ti}\in \text{Pa}\left(M_{tj}\right)\equiv \left\{M_{ti}:{\left({\mathit{W}}_t\right)}_{ij}\ne 0,i\in \{1,\dots ,d\}\setminus \{j\}\right\}$ and ${\mathit{ϵ}}_{wt}={\left({ϵ}_{w1t},\dots ,{ϵ}_{wdt}\right)}^{\top }\in {\mathbb{R}}^d$ is a vector of mean zero random errors. In model (3), the weight matrix ${\mathit{W}}_t$ models the interactions among the mediators ${\mathit{M}}_t$ at each time point, and the conditional mean ${\mathit{\mu }}_t$ characterizes the dynamic dependence over time.

We then consider the linear models for ${\mathit{\mu }}_t$ and $R_t$, in that

$$\begin{gathered} {\mathit{\mu }}_t={\mathit{\alpha }}_{1t}+{\mathit{\delta }}_{1t}{A}_t+{𝚪}_{1t}{\mathit{M}}_{t-1}+{\mathit{\zeta }}_{1t}{R}_{t-1}, \\ {R}_t={\alpha }_{2t}+{\delta }_{2t}{A}_t+{\mathit{\gamma }}_{2t}^{\top }{\mathit{M}}_{t-1}+{\zeta }_{2t}{R}_{t-1}+{\mathit{\kappa }}_t^{\top }{\mathit{M}}_t+{ϵ}_{rt}, \end{gathered}$$ (4)

for some ${\mathit{\alpha }}_{1t},{\mathit{\delta }}_{1t},{\mathit{\zeta }}_{1t}\in {\mathbb{R}}^d,{𝚪}_{1t}\in {\mathbb{R}}^{d\times d}$, and for ${\mathit{\gamma }}_{2t},{\mathit{\kappa }}_t\in {\mathbb{R}}^d,{\alpha }_{2t},{\delta }_{2t},{\zeta }_{2t}\in \mathbb{R}$ and some mean zero errors ${\left\{{ϵ}_{rt}\right\}}_t$ independent over time, respectively. In model (4), ${𝚪}_{1t}$ characterizes the effect of ${\mathit{M}}_{t-1}$ on ${\mathit{M}}_t$. All random variables in (4) are assumed to have finite second moments. Let ${𝚯}_{1t}\equiv \left({\mathit{\alpha }}_{1t},{\mathit{\delta }}_{1t},{𝚪}_{1t},{\mathit{\zeta }}_{1t}\right)\in {\mathbb{R}}^{d\times (d+3)}$ collect all the parameters for ${\mathit{\mu }}_t$, and ${𝚯}_{2t}\equiv {\left({\alpha }_{2t},{\delta }_{2t},{\mathit{\gamma }}_{2t}^{\top },{\zeta }_{2t},{\mathit{\kappa }}_t^{\top }\right)}^{\top }\in {\mathbb{R}}^{(2d+3)\times 1}$ collect all the parameters for $R_t$.

We make a few remarks. First, we draw a connection between the proposed MMP and MDP commonly studied in RL—a powerful machine learning technique for optimal sequential decision making (Murphy (2003), Mnih et al. (2015), Silver et al. (2017), Qin, Zhu and Ye (2021), Zhang, Chen and Yang (2023)). Both capture the carryover effects over time. MDP achieves this by introducing a sequence of time-varying feature variables, referred to as the states. It then models the carryover effects through state transitions, allowing past treatments to affect future outcomes through their impact on future states; see Figure 1(a) for an illustration. This approach has gained substantial attention for policy evaluation, serving to model both immediate and long-term effects of a target policy (Luckett et al. (2020), Hao et al. (2021), Liao, Klasnja and Murphy (2021), Kallus and Uehara (2022), Hu and Wager (2022), Liao et al. (2022), Ramprasad et al. (2023), Shi et al. (2022), Liu et al. (2023), Wang, Qi and Wong (2023)). In a similar vein, our MMP operates by modeling the indirect influence of a preceding mediator via the transitions of mediator-outcome pairs. In essence, a past mediator influences future outcomes by exerting its effects on both the prior outcome and the subsequent mediator; see Figure 1(b) for an illustration. Second, the relation in (3) should be understood as a data generating mechanism, rather than as a mere association. It corresponds to a directed acyclic graph (DAG). Third, to keep the presentation simple, we do not include any time-varying confounders, which may be incorporated into our solution in a relatively straightforward fashion. We do not consider any unobserved confounders either, because we consider random treatment assignments. We leave the case with unobserved confounders as future research. Finally, we consider linear type models in both (3) and (4). The analysis of mediation has been dominated by linear regression paradigms and such linearity assumption is commonly adopted in existing work; see, for example, Nandy, Maathuis and Richardson (2017), Chakrabortty, Nandy and Li (2018), Shi and Li (2022). It is possible to extend to nonlinear type models, under which the definition of individual mediation effect we give later still holds, but is more difficult to evaluate.

In this article, we consider two different settings: the finite-horizon and the infinitehorizon, borrowing the concepts from the RL literature (Sutton and Barto (2018)). For the finite-horizon setting, the number of time points $T$ is fixed and finite. It usually corresponds to the traditional longitudinal studies, where subjects receive a finite number of treatments over a predetermined time period. For the infinite-horizon setting, $T$ approaches infinity theoretically. It often applies to the applications involving continuous monitoring systems such as wearable devices, where the data is densely collected at high frequencies, for example, every second or minute. For the finite-horizon setting, we allow the process $\left\{A_t,{\mathit{M}}_t,R_t\right\}$ to be nonstationary. However, for the infinite-horizon setting, we require the process to be stationary over time, that is, the parameters ${\mathit{W}}_t,{𝚯}_{1t}$ and ${𝚯}_{2t}$ are constant over time. Without stationarity, we need to extrapolate to infer system dynamics beyond the observed time.

In summary, we believe that our model framework provides a reasonable starting point for multivariate dynamic mediation analysis. The methodology under our setting is already complex enough, and deserves a careful investigation.

2.2. Total, direct and indirect effects.

Before we formally define the individual mediation effect, which is the main target of interest in this article, we first introduce the notions of total effect, natural direct effect and natural indirect effect in the finite-horizon setting. This is to facilitate the understanding of our target, and also to establish the connection with the classical mediation analysis literature.

For the total effect, consider a hypothetical intervention applied to the system, where we set all treatments to some value $a$ uniformly over the entire population. This can be realized through Pearl’s do-operator, $\text{do}\left(A_t=a\right)$, which generates an interventional distribution by removing the edges leading into $\left\{A_s:s\le t\right\}$ in the corresponding DAG (Pearl (2000)). We denote the post-interventional expectation of $R_t$ by $\mathbb{E}\left[R_t\mid \text{do}\left(A_s=a,\forall s\le t\right)\right]$. We then define the total effect over $T$ time points as

$$\text{TE}=\sum _{t=1}^T\frac{\partial }{\partial a}\mathbb{E}\left[R_t\mid \text{do}\left(A_s=a,\forall s\le t\right)\right].$$ (5)

We make two remarks. First, each summand $\partial \mathbb{E}\left[R_t\mid \text{do}\left(A_s=a,\forall s\le t\right)\right]/\partial a$ on the right-hand side of (5) measures the total effect of treatment ${\left\{A_s\right\}}_{s\le t}$ on $R_t$ at time $t$. Second, when the treatment takes a discrete value, the derivative in those terms can be substituted by the difference operator. For instance, if the treatment is binary, we obtain that

$$\text{TE}=\sum _{t=1}^T\left\{\mathbb{E}\left[R_t\mid \text{do}\left(A_s=1,\forall s\le t\right)\right]-\mathbb{E}\left[R_t\mid \text{do}\left(A_s=0,\forall s\le t\right)\right]\right\}.$$

Next, we decompose the total effect into the sum of the natural direct effect and the natural indirect effect. Specifically, at each time point $t=1,\dots ,T$, we define the direct effect to be the portion of the total effect of a sequence of treatment variables $\left\{A_s:s\le t\right\}$ on the outcome $R_t$ that does not go through $\left\{{\mathit{M}}_s:s\le t\right\}$. To measure such an effect, we consider the joint intervention on ($A_s,{\mathit{M}}_s$) through $\text{do}\left(A_s=a,{\mathit{M}}_s={\mathit{m}}_s\right)$, and denote the post-interventional expectation of $R_t$ by $\mathbb{E}\left[R_t\mid \text{do}\left(A_s=a,{\mathit{M}}_s={\mathit{m}}_s,\forall s\le t\right)\right]$. We then define the natural direct effect over $T$ time points as

$$\text{DE}=\sum _{t=1}^T{\int }_{{\mathit{m}}_1,\dots ,{\mathit{m}}_T}\frac{\partial }{\partial a}\mathbb{E}\left[R_t\mid \text{do}\left(A_s=a,{\mathit{M}}_s={\mathit{m}}_s,\forall s\le t\right)\right]\times f\left({\mathit{m}}_1,\dots ,{\mathit{m}}_T\mid \text{do}\left(A_t=a,\forall 1\le t\le T\right)\right)d{\mathit{m}}_1\cdots d{\mathit{m}}_T,$$

where $f$ denotes the interventional probability density function of (${\mathit{M}}_1,\dots ,{\mathit{M}}_T$) under the assumption that all treatments are set to $a$, and the term $\partial \mathbb{E}\left[R_t\mid \text{do}\left(A_s=a,{\mathit{M}}_s={\mathit{m}}_s,\forall s\le \right.\right.t\left)\right]/\partial a$ corresponds to the interventional effect of ${\left\{A_s\right\}}_{s\le t}$ on $R_t$ when setting the interventional values of ${\left\{{\mathit{M}}_s\right\}}_{s\le t}$ to constants.

We then define the natural indirect effect as the difference between TE and DE, that is, $\text{IE}=\text{TE}-\text{DE}$, which quantifies the portion of the effect of the treatment sequence on the outcome that goes through the mediator sequence.

We remark that the definitions of TE, DE and IE do not require linear structural equation models like (4). However, imposing such models greatly simplify the analysis. More specifically, in a general Markov mediation process, both $\partial \mathbb{E}\left[R_t\mid \text{do}\left(A_s=a,\forall s\le t\right)\right]/\partial a$ and $\partial \mathbb{E}\left[R_t\mid \text{do}\left(A_s=a,{\mathit{M}}_s={\mathit{m}}_s,\forall s\le t\right)\right]/\partial a$ can depend on $\left(a,{\mathit{m}}_1,\dots ,{\mathit{m}}_t\right)$ in a very complex manner, as also mentioned in Chakrabortty, Nandy and Li (2018). However, (4) simplifies these expressions to be functions of the regression coefficients that are independent of specific values of ${\mathit{m}}_s$ or $a$; see Section 3 for more details. As a result, DE is equivalent to

$$\sum _{t=1}^T\frac{\partial }{\partial a}\mathbb{E}\left[R_t\mid \text{do}\left(A_s=a,{\mathit{M}}_s=\mathit{m},\forall s\le t\right)\right],$$ (6)

for any $\mathit{m}$, where (6) is the controlled direct effect, computed by setting all values of mediators to $\mathit{m}$. Correspondingly, by (5) and (6), the natural indirect effect becomes

$$\sum _{t=1}^T\left\{\frac{\partial }{\partial a}\mathbb{E}\left[R_t\mid \text{do}\left(A_s=a,\forall s\le t\right)\right]-\frac{\partial }{\partial a}\mathbb{E}\left[R_t\mid \text{do}\left(A_s=a,{\mathit{M}}_s=\mathit{m},\forall s\le t\right)\right]\right\}.$$ (7)

In other words, the linear structural equation model avoids the need to estimate the interventional probability density function $f$, thus substantially simplifying the forms of DE, IE and their subsequent estimation and inference procedures.

2.3. Individual mediation effect.

We now formally define the individual mediation effect, the main target in our dynamic mediation analysis. We begin with the finite-horizon setting.

Definition 1 (Individual mediation effect in finite-horizon). In a finite-horizon setting, the individual mediation effect of the $j$th mediator over $T$ time points is defined as

$${\eta }_j^{\left(T\right)}\equiv \sum _{t=1}^T\frac{\partial }{\partial a}\mathbb{E}\left[R_t\mid \text{do}\left(A_s=a,\forall s\le t\right)\right]-\sum _{t=1}^T\frac{\partial }{\partial a}\mathbb{E}\left[R_t\mid \text{do}\left(A_s=a,M_{sj}=m,\forall s\le t\right)\right].$$

When the treatment is binary, ${\eta }_j^{\left(T\right)}$ can be defined as

$${\eta }_j^{\left(T\right)}=\sum _{a=0}^1(-1{)}^{a+1}\left\{\sum _{t=1}^T\mathbb{E}\left[R_t\mid \text{do}\left(A_s=a,\forall s\le t\right)\right]\right.\left.-\sum _{t=1}^T\mathbb{E}\left[R_t\mid \text{do}\left(A_s=a,M_{sj}=m,\forall s\le t\right)\right]\right\}.$$ (8)

We again make a few remarks. First, our definition of the individual mediation effect is related to the definition of IE in (7), and is defined as the difference between the total effect and the interventional effect $\partial \mathbb{E}\left[R_t\mid \text{do}\left(A_s=a,M_{sj}=m,\forall s\le t\right)\right]/{\partial }_a$ over time. However, the key difference is that, whereas IE measures the mediation effect of all mediators, ${\eta }_j^{\left(T\right)}$ focuses on quantifying the portion of the effect that specifically passes through the individual $j$ th mediator. As such, we only intervene the $j$ th mediator as opposed to all mediators in the intervention, and we refer to it as an individual mediation effect.

Second, our definition is consistent with the existing literature. Specifically, when there is only one mediator $M$ and a single time point, our definition aligns with the classical definition of Baron and Kenny (1986), $\partial \mathbb{E}[R\mid \text{do}(A=a\left)\right]/\partial a-\partial \mathbb{E}[R\mid \text{do}(A=a,M=m\left)\right]/\partial a$, where the first term is the regression coefficient by regressing $R$ on $A$, and the second term is the partial regression coefficient by further including $M$ in the regression. This difference measuring the reduction in the total effect due to controlling for $M$ is widely used to quantify the effect mediated through $M$; see also Pearl (2012). When there is a single time point, our definition is consistent with the one proposed by Chakrabortty, Nandy and Li (2018). When there is a single mediator, (8) is similar to the definitions proposed by VanderWeele and Tchetgen Tchetgen (2017) and Ge et al. (2023) under a potential outcome framework.

Third, under the linear structural equation model (4), ${\eta }_j^{\left(T\right)}$ is a constant function with respect to $m$ and $a$. It can depend on ($a,m$) in a more general Markov mediation process, for example, when either the transition model from a given time point to the next or the DAG structural equation model at a given time point is nonlinear. In addition, we observe that, under (4), the second post-interventional expectation term, $\mathbb{E}\left[R_t\mid \text{do}\left(A_s=a,M_{sj}=m,\forall s\le \right.\right.t\left)\right]$, may be analyzed by the path method (Wright (1921)). That is, it can be calculated by summing up the effects along all directed paths from ${\left\{A_s\right\}}_{s\le t}$ to $R_t$ that do not pass through ${\left\{M_{sj}\right\}}_{s\le t}$. However, this approach can be computationally expensive, since the number of paths grows exponentially fast as the number of time points $T$ increases.

Finally, ${\eta }_j^{\left(T\right)}$ in Definition 1 measures the cumulative effect mediated through the $j$ th mediator $M_j$ across all $T$ stages. Alternatively, one may be interested in the incremental effect,

$${\Delta }_j^{\left(t\right)}\equiv \frac{\partial }{\partial a}\mathbb{E}\left[R_t\mid \text{do}\left(A_s=a,\forall s\le t\right)\right]-\frac{\partial }{\partial a}\mathbb{E}\left[R_t\mid \text{do}\left(A_s=a,M_{sj}=m,\forall s\le t\right)\right],$$

which measures the individual mediation effect attributed to $M_j$ at time $t$. By definition, we see that this incremental effect is related to the cumulative effect, in that ${\eta }_j^{\left(T\right)}=\sum _{t=1}^T{\Delta }_j^{\left(t\right)}$.

To better understand our definition of the individual mediation effect, we further decompose ${\Delta }_j^{\left(t\right)}$ into the sum of the immediate individual mediation effect (IIME) and the delayed individual mediation effect (DIME), as defined next.

Definition 2 (Immediate and delayed individual mediation effects in finite-horizon). Define the immediate individual mediation effect (IIME) and the delayed individual mediation effect (DIME) as

$$\begin{gathered} {\text{IIME}}_j^{\left(t\right)}\equiv \frac{\partial }{\partial a}\mathbb{E}\left[R_t\mid \text{do}\left(A_t=a\right)\right]-\frac{\partial }{\partial a}\mathbb{E}\left[R_t\mid \text{do}\left(A_t=a,M_{tj}=m\right)\right], \\ {\text{DIME}}_j^{\left(t\right)}\equiv {\Delta }_j^{\left(t\right)}-{\text{IIME}}_j^{\left(t\right)}. \end{gathered}$$

At a given time point $t,{\text{IIME}}_j^{\left(t\right)}$ can be interpreted as the change in the total effect of $A_t$ on $R_t$ when $M_{tj}$ is knocked out, whereas ${\text{DIME}}_j^{\left(t\right)}$ captures the individual mediation effect of the $j$ th mediator that is carried over from all previous stages $s<t$ up to time $t$.

We next turn to the infinite-horizon setting. In this setting, the individual mediation effect is defined as the average effect over time, so to prevent it from being unbounded.

Definition 3 (Individual mediation effect in infinite-horizon). In an infinite-horizon setting, the individual mediation effect of the $j$ th mediator is defined as ${\eta }_j^{(\infty )}\equiv \underset{T\to \infty }{\text{lim}}{\eta }_j^{\left(T\right)}/T$, provided that the limit exists.

We comment that all above definitions are natural and agree with the intuitions. We next discuss how to evaluate and estimate the individual mediation effect given the data.

3. Evaluation of individual mediation effect.

In this section, we propose approaches to estimate and infer the individual mediation effects given in Definitions 1 and 3. The problem, however, is very challenging, as we have to deal with both the multiple mediators with unknown correlation structure, as well as the carryover effects from the upstream treatments and mediators. Our proposed solution involves deriving a closed-form expression for the individual mediation effect, as detailed in Theorems 1 and 2. This expression is dependent on several intermediate quantities that are computable through a set of recursive relations. Leveraging these theoretical findings, we develop an efficient recursive computation algorithm to estimate these intermediate quantities, which subsequently facilitate both the estimation and inference of the individual mediation effect.

3.1. An illustrative example.

To illustrate our theory, we first consider a simple example as shown in Figure 2, in which there are only $d=2$ mediators and $T=2$ time stages. We then extend our observations to more general cases with $d$ mediators and $T$ stages.

Fig. 2.

An illustrative example with two mediators and two time stages. The left panel shows the DAG in the first stage, and the right panel the first two stages. The dashed lines highlight the paths that go through the intervened mediator, that is, the second mediator.

Suppose we focus on the individual mediation effect for the second mediator, that is, $j=2$. We begin with the first stage $t=1$, as shown in Figure 2, left panel. By Definition 1,

$$\begin{gathered} {\eta }_2^{\left(1\right)}=\frac{\partial }{\partial a}\mathbb{E}\left[R_1\mid \text{do}\left(A_1=a\right)\right]-\frac{\partial }{\partial a}\mathbb{E}\left[R_1\mid \text{do}\left(A_1=a,M_{12}=m\right)\right] \\ \equiv {\theta }_{A_1\to R_1}-{\theta }_{A_1\to R_1}^{M_{12}}, \end{gathered}$$ (9)

where we use ${\theta }_{A_1\to R_1}$ to denote the total effect from $A_1$ to $R_1$, and use ${\theta }_{A_1\to R_1}^{M_{12}}$ to denote the total effect from $A_1$ to $R_1$ when the second mediator is intervened in the first stage.

We compute ${\theta }_{A_1\to R_1}$ by summing up the effects along all directed paths from $A_1$ to $R_1$, namely $A_1\to R_1,A_1\to M_{11}\to R_1,A_1\to M_{12}\to R_1$ and $A_1\to M_{11}\to M_{12}\to R_1$. Meanwhile, we compute ${\theta }_{A_1\to R_1}^{M_{12}}$ by eliminating all paths that go from $A_1$ to $R_1$ through $M_{12}$. This corresponds to subtracting the effects along $A_1\to M_{12}\to R_1$ and $A_1\to M_{11}\to M_{12}\to R_1$, which leads to

$${\theta }_{A_1\to R_1}^{M_{12}}={\theta }_{A_1\to R_1}-{\theta }_{A_1\to M_{12}}{\theta }_{M_{12}\to R_1},$$ (10)

where we use ${\theta }_{A_1\to M_{12}}$ and ${\theta }_{M_{12}\to R_1}$ to denote the effect from $A_1$ to $M_{12}$, and from $M_{12}$ to $R_1$, respectively. The relation in (10) has an intuitive interpretation: to evaluate the total effect from $A_1$ to $R_1$ when $M_{12}$ is intervened, we subtract from ${\theta }_{A_1\to R_1}$ the effects along the paths that go through $M_{12}$. Plugging (10) into (9), we obtain that ${\eta }_2^{\left(1\right)}={\theta }_{A_1\to M_{12}}{\theta }_{M_{12}\to R_1}$, which coincides with the classical product type representation of the individual mediation effect in a single-stage analysis (Nandy, Maathuis and Richardson (2017)).

We next move to the second stage $t=2$, as in Figure 2, right panel. By Definition 1,

$$\begin{gathered} {\eta }_2^{\left(2\right)}={\eta }_2^{\left(1\right)}+\left\{\frac{\partial }{\partial a}\mathbb{E}\left[R_2\mid \text{do}\left(A_1=A_2=a\right)\right]\right.\left.-\frac{\partial }{\partial a}\mathbb{E}\left[R_2\mid \text{do}\left(A_1=A_2=a,M_{12}=M_{22}=m\right)\right]\right\} \\ \equiv {\eta }_2^{\left(1\right)}+{\theta }_{A_1,A_2\to R_2}-{\theta }_{A_1,A_2\to R_2}^{M_{12},M_{22}}, \end{gathered}$$ (11)

where we use ${\theta }_{A_1,A_2\to R_2}$ to denote the cumulative total effect of ($A_1,A_2$) on $R_2$, and use ${\theta }_{A_1,A_2\to R_2}^{M_{12},M_{22}}$ to denote the cumulative total effect of $\left(A_1,A_2\right)$ on $R_2$ when the second mediator is intervened in both the first and second stages.

Because the treatments are randomly assigned and their parent sets are empty, we have

$${\theta }_{A_1,A_2\to R_2}={\theta }_{A_1\to R_2}+{\theta }_{A_2\to R_2},{\theta }_{A_1,A_2\to R_2}^{M_{12},M_{22}}={\theta }_{A_1\to R_2}^{M_{12},M_{22}}+{\theta }_{A_2\to R_2}^{M_{22}},$$ (12)

where we use ${\theta }_{A_1\to R_2}$ and ${\theta }_{A_2\to R_2}$ to denote the total effects from $A_1$ to $R_2$, and from $A_2$ to $R_2$, respectively, use ${\theta }_{A_2\to R_2}^{M_{22}}$ to denote the effect from $A_2$ to $R_2$ when the second mediator is intervened in the second stage, and use ${\theta }_{A_1\to R_2}^{M_{12},M_{22}}$ to denote the effect from $A_1$ to $R_2$ when the second mediator is intervened in both stages.

We compute ${\theta }_{A_1\to R_2}$ and ${\theta }_{A_2\to R_2}$ similarly as that for ${\theta }_{A_1\to R_1}$. We compute ${\theta }_{A_2\to R_2}^{M_{22}}$ similarly as in (10), that is, ${\theta }_{A_2\to R_2}^{M_{22}}={\theta }_{A_2\to R_2}-{\theta }_{A_2\to M_{22}}{\theta }_{M_{22}\to R_2}$. Also, similar to (10), we have

$${\theta }_{A_1\to R_2}^{M_{12},M_{22}}={\theta }_{A_1\to R_2}^{M_{22}}-{\theta }_{A_1\to M_{12}}{\theta }_{M_{12}\to R_2}^{M_{22}},$$ (13)

where we use ${\theta }_{A_1\to R_2}^{M_{22}}$ and ${\theta }_{M_{12}\to R_2}^{M_{22}}$ to denote the effects from $A_1$ to $R_2$, and from $M_{12}$ to $R_2$, when the second mediator is intervened in the second stage. Intuitively, ${\theta }_{M_{12}\to R_2}^{M_{22}}$ can be interpreted as the carryover effect of the second mediator on the outcome from the first stage to the second stage when it is intervened. We again compute ${\theta }_{A_1\to R_2}^{M_{22}}$ similar to (10), that is, ${\theta }_{A_1\to R_2}^{M_{22}}={\theta }_{A_1\to R_2}-{\theta }_{A_1\to M_{22}}{\theta }_{M_{22}\to R_2}$. In addition, we compute ${\theta }_{M_{12}\to R_2}^{M_{22}}$ by eliminating all paths that go from $M_{12}$ to $R_2$ through $M_{22}$, which leads to

$${\theta }_{M_{12}\to R_2}^{M_{22}}={\theta }_{M_{12}\to R_2}-{\theta }_{M_{12}\to M_{22}}{\theta }_{M_{22}\to R_2},$$ (14)

where we use ${\theta }_{M_{12}\to R_2},{\theta }_{M_{12}\to M_{22}}$ and ${\theta }_{M_{22}\to R_2}$ to denote the effects from $M_{12}$ to $R_2$, from $M_{12}$ to $M_{22}$ and from $M_{22}$ to $R_2$, respectively.

Based on the derivations so far, we see that to evaluate the individual mediation effect, it is crucial to calculate those intermediate quantities, such as ${\theta }_{A_1\to R_1},{\theta }_{A_1\to M_{22}},{\theta }_{M_{12}\to M_{22}},{\theta }_{M_{22}\to R_2}$, among others. Next, we briefly discuss how to evaluate those intermediate quantities after imposing the linear structural equation models (3) and (4). We also discuss some recursive relation that facilitates both the computational and statistical efficiencies.

First, we note that, under (3) and (4), we can estimate those intermediate quantities through linear regressions. For instance, we can estimate ${\theta }_{A_1\to R_1}$ as the coefficient of $A_1$ by linearly regressing $R_1$ onto $A_1$. Meanwhile, we can estimate ${\theta }_{M_{22}\to R_2}$ as the coefficient of $M_{22}$ by linearly regressing $R_2$ onto $M_{22}$, however, with some additional covariate adjustment. This is because, unless the two mediators ($M_{21},M_{22}$) are conditionally independent given ($M_{11},M_{12},R_1,A_2$), we need to adjust for a set of covariates that satisfy Pearl’s backdoor criterion (Pearl (2000)). In other words, we should block the effects flowing from ($A_2,M_{11},M_{12},M_{21},R_1$) to $M_{22}$. Therefore, we need to adjust for the covariate set $\text{Pa}\left(M_{22}\right)\cup M_{11}\cup M_{12}\cup R_1\cup A_2$ in this regression.

Second, we note that the set of intermediate quantities involve both within-stage quantities such as ${\theta }_{A_1\to R_1},{\theta }_{M_{22}\to R_2}$, as well as cross-stage quantities such as ${\theta }_{A_1\to M_{22}},{\theta }_{M_{12}\to M_{22}}$. There is some useful recursive relation between the within-stage and cross-stage quantities under the linear structural equation models (3) and (4). For instance,

$${\mathit{\theta }}_{A_1\to {\mathit{M}}_2}={𝚪}_{12}{\mathit{\theta }}_{A_1\to {\mathit{M}}_1}+{\mathit{\zeta }}_{12}{\theta }_{A_1\to R_1},$$ (15)

where ${\mathit{\theta }}_{A_1\to {\mathit{M}}_2}={\left({\theta }_{A_1\to M_{21}},{\theta }_{A_1\to M_{22}}\right)}^{\top }$, and ${\mathit{\theta }}_{A_1\to {\mathit{M}}_1}={\left({\theta }_{A_1\to M_{11}},{\theta }_{A_1\to M_{12}}\right)}^{\top }$. This suggests that the cross-stage carryover effect from $A_1$ to ${\mathit{M}}_2={\left(M_{21},M_{22}\right)}^{\top }$ is a combination of its within-stage effect on ${\mathit{M}}_1={\left(M_{11},M_{12}\right)}^{\top }$ and on $R_1$, respectively, whereas the coefficients $\left({𝚪}_{12},{\mathit{\zeta }}_{12}\right)$ can be viewed as the weights for such a cross-stage transition. In our estimation algorithm, we first estimate the within-stage quantities via linear regressions, then update the cross-stage quantities following (15). As we show later, the relation such as (15) not only expedites the computation, but also improves the estimation efficiency by leveraging more information from the conditional model (4).

In summary, this simple example reveals a number of important relations. First, we see that those intermediate quantities form the building blocks for our evaluation of the individual mediation effect. Second, (10), (13) and (14) suggest some useful recursive representations that in effect reduce the number of mediators in the superscript that are intervened upon. Third, under models (3) and (4), we can estimate those intermediate quantities through linear regressions, with possibly backdoor adjustment. Finally, we can further improve both the computational and statistical efficiencies through another set of recursive relations between the within-stage and cross-stage quantities. All these observations are crucial for our derivation of the expression for the individual mediation effect.

3.2. Intermediate quantities.

We now formally define the set of all intermediate quantities that are needed for the evaluation of the individual mediation effect. We then discuss how to estimate those quantities.

For any $1\le s\le t\le T$, and $j=1,\dots ,d$, define

$$\begin{gathered} {\theta }_{A_s\to R_t}\equiv \frac{\partial }{\partial a}\mathbb{E}\left[R_t\mid \text{do}\left(A_s=a\right)\right]\in \mathbb{R}, \\ {\mathit{\theta }}_{M_{sj}\to {\mathit{M}}_t}\equiv \frac{\partial }{\partial m}\mathbb{E}\left[{\mathit{M}}_t\mid \text{do}\left(M_{sj}=m\right)\right]\in {\mathbb{R}}^d, \\ {\mathit{\theta }}_{A_s\to {\mathit{M}}_t}\equiv \frac{\partial }{\partial a}\mathbb{E}\left[{\mathit{M}}_t\mid \text{do}\left(A_s=a\right)\right]\in {\mathbb{R}}^d, \\ {\theta }_{M_{sj}\to R_t}\equiv \frac{\partial }{\partial m}\mathbb{E}\left[R_t\mid \text{do}\left(M_{sj}=m\right)\right]\in \mathbb{R}. \end{gathered}$$ (16)

These intermediate quantities in (16) can be obtained through linear regressions, either directly, or by some backdoor covariate adjustment. In particular, ${\theta }_{A_s\to R_t}$ can be obtained as the coefficient of $A_s$ by linearly regressing $R_t$ onto $A_s$ with an intercept, and we write this coefficient as ${\beta }_{A_s,R_t}$. Similarly, ${\mathit{\theta }}_{A_s\to {\mathit{M}}_t}$ can be obtained as the coefficient of $A_s$ by linearly regressing ${\mathit{M}}_t$ onto $A_s$, denoted as ${\mathit{\beta }}_{A_s,{\mathit{M}}_t}$. Meanwhile, ${\theta }_{M_{sj}\to R_t}$ can be obtained as the coefficient of $M_{sj}$ by linearly regressing $R_t$ onto $M_{sj}$, along with the adjusted covariate set, $\text{Pa}\left(M_{sj}\right)\cup {\mathit{M}}_{s-1}\cup R_{s-1}\cup A_s$. Similarly, ${\theta }_{M_{sj}\to {\mathit{M}}_t}$ can be obtained as the coefficient of $M_{sj}$ by linearly regressing ${\mathit{M}}_t$ onto $M_{sj}$, along with the adjusted covariate set, $\text{Pa}\left(M_{sj}\right)\cup {\mathit{M}}_{s-1}\cup R_{s-1}\cup A_s$. Putting together, we have

$$\begin{gathered} {\theta }_{A_s\to R_t}={\beta }_{A_s,R_t},{\mathit{\theta }}_{M_{sj}\to {\mathit{M}}_t}={\mathit{\beta }}_{M_{sj},{\mathit{M}}_t\mid \text{Pa}\left(M_{sj}\right)\cup {\mathit{M}}_{s-1}\cup R_{s-1}\cup A_s}, \\ {\mathit{\theta }}_{A_s\to {\mathit{M}}_t}={\mathit{\beta }}_{A_s,{\mathit{M}}_t},{\theta }_{M_{sj}\to R_t}={\beta }_{M_{sj},R_t\mid \text{Pa}\left(M_{sj}\right)\cup {\mathit{M}}_{s-1}\cup R_{s-1}\cup A_s}. \end{gathered}$$ (17)

Moreover, similar to (15), we obtain the following recursive relations between the within-stage quantities and the cross-stage quantities. That is, under models (3) and (4),

$$\begin{gathered} {\theta }_{A_s\to R_t}={\zeta }_{2t}{\theta }_{A_s\to R_{t-1}}+{\mathit{\kappa }}_t^{\top }{\mathit{\theta }}_{A_s\to {\mathit{M}}_t}+{\mathit{\gamma }}_{2t}^{\top }{\mathit{\theta }}_{A_s\to {\mathit{M}}_{t-1}}, \\ {\mathit{\theta }}_{A_s\to {\mathit{M}}_t}={𝚪}_{1t}{\mathit{\theta }}_{A_s\to {\mathit{M}}_{t-1}}+{\mathit{\zeta }}_{1t}{\theta }_{A_s\to R_{t-1}}, \\ {\theta }_{M_{sj}\to R_t}={\zeta }_{2t}{\theta }_{M_{sj}\to R_{t-1}}+{\mathit{\kappa }}_t^{\top }{\mathit{\theta }}_{M_{sj}\to {\mathit{M}}_t}+{\mathit{\gamma }}_{2t}^{\top }{\mathit{\theta }}_{M_{sj}\to {\mathit{M}}_{t-1}}, \\ {\mathit{\theta }}_{M_{sj}\to {\mathit{M}}_t}={𝚪}_{1t}{\mathit{\theta }}_{M_{sj}\to {\mathit{M}}_{t-1}}+{\mathit{\zeta }}_{1t}{\theta }_{M_{sj}\to R_{t-1}}. \end{gathered}$$ (18)

In our implementation, we first estimate the within-stage quantities, $\left\{{\theta }_{A_t\to R_t},{\theta }_{M_{tj}\to R_t}\right.,\left.{\mathit{\theta }}_{A_t\to {\mathit{M}}_t},{\mathit{\theta }}_{M_{tj}\to {\mathit{M}}_t}\right\}$, using (17), then estimate the cross-stage quantities, $\left\{{\theta }_{A_s\to R_t},{\theta }_{M_{sj}\to R_t}\right.,\left.{\mathit{\theta }}_{A_s\to R_t},{\mathit{\theta }}_{M_{sj}\to {\mathit{M}}_t}\right\}$, for $s=1,\dots ,t-1$, using (18). This helps to improve both the estimation efficiency and the computation efficiency.

3.3. Individual mediation effect in a finite-horizon setting.

We next derive the expression of the individual mediation effect through the intermediate quantities in (16) under the finite-horizon setting.

We first sketch the key ideas of the derivation here, and relegate more details to the Supplementary Material. By Definition 1, the individual mediation effect of the $j$ th mediator across $t$ stages, $j=1,\dots ,d,t=1,\dots ,T$, can be expressed as

$${\eta }_j^{\left(t\right)}={\eta }_j^{(t-1)}+{\theta }_{A_1,\dots ,A_t\to R_t}-{\theta }_{A_1,\dots ,A_t\to R_t}^{M_{1j},\dots ,M_{tj}}.$$ (19)

Because all the treatments ($A_1,\dots ,A_t$) are randomly assigned, and thus are independent of each other and all other covariates, similar to (12), we have

$$\begin{gathered} {\theta }_{A_1,\dots ,A_t\to R_t}={\theta }_{A_1\to R_t}+{\theta }_{A_2\to R_t}+\cdots +{\theta }_{A_t\to R_t}, \\ {\theta }_{A_1,\dots ,A_t\to R_t}^{M_{1j},\dots ,M_{tj}}={\theta }_{A_1\to R_t}^{M_{1j},\dots ,M_{tj}}+{\theta }_{A_2\to R_t}^{M_{2j},\dots ,M_{tj}}+\cdots +{\theta }_{A_t\to R_t}^{M_{tj}}. \end{gathered}$$ (20)

Then, similar to (10), (13) and (14), we obtain the following recursive relations that help reduce the number of mediators being intervened upon. That is, for any $s=1,\dots ,t-1$,

$$\begin{gathered} {\theta }_{A_s\to R_t}^{M_{sj},\dots ,M_{tj}}={\theta }_{A_s\to R_t}^{M_{\left(s+1\right)j},\dots ,M_{tj}}-{\theta }_{A_s\to M_{sj}}{\theta }_{M_{sj}\to R_t}^{M_{sj},\dots ,M_{tj}}=\cdots \\ ={\theta }_{A_s\to R_t}-\sum _{i=s}^t{\theta }_{A_s\to M_{ij}}{\theta }_{M_{ij}\to R_t}^{M_{ij},\dots ,M_{tj}}; \end{gathered}$$ (21)

$$\begin{gathered} {\theta }_{M_{sj}\to R_t}^{M_{sj},\dots ,M_{tj}}={\theta }_{M_{sj}\to R_t}^{M_{(s+2)j},\dots ,M_{tj}}-{\theta }_{M_{sj}\to M_{(s+1)j}}{\theta }_{M_{(s+1)j}\to R_t}^{M_{(s+1)j},\dots ,M_{tj}}=\cdots \\ ={\theta }_{M_{sj}\to R_t}-\sum _{i=s+1}^t{\theta }_{M_{sj}\to M_{ij}}{\theta }_{M_{ij}\to R_t}^{M_{ij},\dots ,M_{tj}}. \end{gathered}$$ (22)

Combining (19), (20), (21) and (22), we obtain the following theorem with respect to the identification of individual mediation effect in finite-horizon settings. We relegate a more detailed derivation to the Supplementary Material.

Theorem 1 (Individual mediation effect for finite-horizon). Suppose ${\left\{A_t\right\}}_{t\ge 1}$ are randomly assigned treatments and satisfy (2), and ${\left\{{\mathit{M}}_t,R_t\right\}}_{t\ge 1}$ follow (3) and (4). Then

$${\eta }_j^{\left(t\right)}={\eta }_j^{(t-1)}+\sum _{s=1}^t\sum _{i=s}^t{\theta }_{A_s\to M_{ij}}{\theta }_{M_{ij}\to R_t}^{M_{ij},\dots ,M_{tj}},t=1,\dots ,T,$$ (23)

where ${\theta }_{M_{sj}\to R_t}^{M_{sj},\dots ,M_{tj}}$ is computed following (22), and we set ${\eta }_j^{\left(0\right)}=0$.

As for estimation, based on Theorem 1, we estimate the individual mediation effect in a recursive manner. That is, for stage $t-1$, we first estimate the weight matrix ${\mathit{W}}_t$ in (3), and the parameters ${𝚯}_{1t},{𝚯}_{2t}$ in (4) for stage $t$. Estimation of ${\mathit{W}}_t$ is needed for determining the parent set of each mediator, which in turn is used for backdoor covariate adjustment. There are multiple algorithms available to estimate ${\mathit{W}}_t$, including those developed in single time-point settings (Zheng et al. (2018), Yuan et al. (2019), Bello, Aragam and Ravikumar (2022)), and those developed in time-varying settings (Hyvärinen et al. (2010), Malinsky and Spirtes (2018), Pamfil et al. (2020)). We employ the DAGMA algorithm recently proposed by Bello, Aragam and Ravikumar (2022) for its simplicity and effectiveness. Recall that, in the finite-horizon setting, we allow the DAG to be nonstationary over time. We thus estimate the time-specific DAGs using DAGMA, with data pooled over multiple subjects but restricted to that time point only. We then estimate the within-stage quantities $\left\{{\theta }_{A_t\to R_t},{\theta }_{M_{tj}\to R_t},{\mathit{\theta }}_{A_t\to {\mathit{M}}_t}\right.,\left.{\mathit{\theta }}_{M_{tj}\to {\mathit{M}}_t}\right\}$ using (17), and estimate the cross-stage quantities, $\left\{{\theta }_{A_s\to R_t},{\theta }_{M_{sj}\to R_t},{\mathit{\theta }}_{A_s\to {\mathit{M}}_t}\right.,\left.{\mathit{\theta }}_{M_{sj}\to M_t}\right\}$, for $s=1,\dots ,t-1$, using (18). Finally, we plug-in these estimators into (23) to estimate the individual mediation effect.

As for inference, based on Theorem 3 in Section 3.5, our plug-in estimator is asymptotically normal. This motivates us to employ the nonparametric bootstrap approach (Efron (1979)) to conduct statistical inference. Specifically, we first sample $n$ trajectories from the observed data with replacement. We next refit the weight matrix ${\mathit{W}}_t$ and reestimate the within-stage and cross-stage quantities using the bootstrap samples. During the refitting, there are two options. One is to use the bootstrap samples to refit the coefficients of the nonzero entries in the weight matrix only, without reestimating the structure of ${\mathit{W}}_t$. The other is to refit both the structure of ${\mathit{W}}_t$ and its nonzero entries. When the structure learning algorithm is consistent, the two solutions are asymptotically equivalent. For finite samples, the former is computationally more efficient, while the latter is expected to be more accurate. In our implementation, we adopt the first option due to its simplicity, and our numerical experiments suggest that it works well empirically. We next construct the plug-in estimators using these refitted parameters. Finally, we use the empirical lower and upper $\alpha /2$ quantiles of the plug-in estimators to construct the confidence interval under a given significance level $\alpha$.

Algorithm 1 summarizes our estimation and inference procedure.

3.4. Individual mediation effect in an infinite-horizon setting.

We next derive the expression of the individual mediation effect under the infinite-horizon setting. Unlike the finite-horizon setting, we now require the Markov mediation process ${\left\{A_t,{\mathit{M}}_t,R_t\right\}}_{t\ge 1}$ to be stationary. This is to ensure the existence of the limit in Definition 3. Correspondingly, the parameters $\mathit{W},{𝚯}_1,{𝚯}_2$ in (3) and (4) remain the same across different time stages, which leads to the following relations. For any $1\le s\le t\le T$, and $j=1,\dots ,d$,

$$\begin{gathered} {\theta }_{A_s\to R_t}={\theta }_{A_{s+1}\to R_{t+1}}=\cdots ={\theta }_{A_{s+T-t}\to R_T},{\mathit{\theta }}_{{\mathit{M}}_{sj}\to {\mathit{M}}_t}={\mathit{\theta }}_{{\mathit{M}}_{\left(s+1\right)j}\to {\mathit{M}}_{t+1}}=\cdots ={\mathit{\theta }}_{{\mathit{M}}_{\left(s+T-t\right)j}\to {\mathit{M}}_T}, \\ {\mathit{\theta }}_{A_s\to {\mathit{M}}_t}={\mathit{\theta }}_{A_{s+1}\to {\mathit{M}}_{t+1}}=\cdots ={\mathit{\theta }}_{A_{s+T-t}\to M_T},{\theta }_{{\mathit{M}}_{sj}\to R_t}={\theta }_{M_{\left(s+1\right)j}\to R_{t+1}}=\cdots ={\theta }_{M_{\left(s+T-t\right)j}\to R_T}. \end{gathered}$$

Based on this observation, we obtain a simplified representation for ${\eta }_j^{\left(T\right)}$ as

$$\begin{gathered} {\eta }_j^{\left(T\right)}={\theta }_{M_{1j}\to R_1}\sum _{t=1}^T(T-t+1){\theta }_{A_1\to M_{tj}} \\ +\sum _{t=1}^{T-1}{\theta }_{M_{1j}\to R_{t+1}}^{M_{1j},\dots ,M_{(t+1)j}}\left(\sum _{s=1}^{T-t}(T-t-s+1){\theta }_{A_1\to M_{sj}}\right). \end{gathered}$$

By dividing the right-hand side by $T$ and taking the limit $T\to \infty$, we obtain the following theorem for identifying the individual mediation effect in infinite-horizon settings. We relegate a more detailed derivation to the Supplementary Material.

Algorithm 1.

Estimation and inference for the finite-horizon setting

Input: Observed data $\left\{\left(A_{it},{\mathit{M}}_{it},R_{it}\right):i=1,\dots ,n,t=1,\dots ,T\right\}$, the number of bootstrap samples $B$, and the significance level $\alpha$.
Output: Individual mediation effect ${\widehat{\eta }}_j^{\left(t\right)}$, for $t=1,\dots ,T,j=1,\dots ,d$ and their CIs.
1:	Estimate within-stage quantities $\left({\widehat{\theta }}_{A_1\to R_1},{\widehat{\theta }}_{M_{1j}\to R_1},{\widehat{\mathit{\theta }}}_{A_1\to {\mathit{M}}_1},{\widehat{\mathit{\theta }}}_{M_{1j}\to {\mathit{M}}_1}\right)$ and compute ${\widehat{\eta }}_j^{\left(1\right)}={\widehat{\theta }}_{A_1\to M_{1j}}{\widehat{\theta }}_{M_{1j}\to R_1}$.
2:	for $t=2,\dots ,T$ do
3:	Estimate parameters ${\widehat{\mathit{W}}}_t,{\widehat{𝚯}}_{1t},{\widehat{𝚯}}_{2t}$ in models (3) and (4).
4:	Estimate within-stage quantities $\left({\widehat{\theta }}_{A_t\to R_t},{\widehat{\theta }}_{M_{tj}\to R_t},{\widehat{\mathit{\theta }}}_{A_t\to {\mathit{M}}_t},{\widehat{\mathit{\theta }}}_{M_{tj}\to {\mathit{M}}_t}\right)$,
5:	for $s=1,\dots ,t-1$ do
6:	Estimate cross-stage quantities (${\widehat{\theta }}_{A_s\to R_t},{\widehat{\theta }}_{M_{sj}\to R_t},{\widehat{\mathit{\theta }}}_{A_s\to {\mathit{M}}_t},{\widehat{\mathit{\theta }}}_{M_{sj}\to {\mathit{M}}_t}$) using (18).
7:	end for
8:	Compute ${\widehat{\eta }}_j^{\left(t\right)}$ using (23).
9:	end for
10:	for $b=1,\dots ,B$ do
11:	Sample $n$ trajectories from the observed data with replacement.
12:	Repeat Lines 1 to 9 to compute ${\widehat{\eta }}_j^{(t,b)}$ using the bootstrap samples.
13:	end for
14:	for $t=1,\dots ,T$ do
15:	Construct the CI $\left[{\widehat{\eta }}_j^{(t,L)},{\widehat{\eta }}_j^{(t,U)}\right]$ for ${\eta }_j^{\left(t\right)}$, where ${\widehat{\eta }}_j^{(t,L)}$ and ${\widehat{\eta }}_j^{(t,U)}$ are the empirical lower and upper $\alpha /2$ quantiles of ${\left\{{\widehat{\eta }}_j^{(t,b)}\right\}}_{b=1}^B$, respectively.
16:	end for

Theorem 2 (Individual mediation effect for infinite-horizon). Suppose ${\left\{A_t\right\}}_{t\ge 1}$ are randomly assigned treatments and satisfy (2), ${\left\{{\mathit{M}}_t,R_t\right\}}_{t\ge 1}$ follow (3) and (4), the Markov process ${\left\{A_t,{\mathit{M}}_t,R_t\right\}}_{t\ge 1}$ is stationary, and the parameters $\left\{\mathit{W},{𝚯}_1,{𝚯}_2\right\}$ in (3) and (4) are time-invariant. Then

$${\eta }_j^{(\infty )}\equiv \underset{T\to \infty }{\text{lim}}\frac{{\eta }_j^{\left(T\right)}}{T}={\theta }_{M_{1j}\to R_1}{B}_{1j}+{\left(1+B_{2j}\right)}^{-1}\left(B_{5j}-{\theta }_{M_{1j}\to R_1}{B}_{4j}\right),$$ (24)

where $B_{1j},B_{2j},B_{4j}$ are the $j$ th element of ${\mathit{B}}_1,{\mathit{B}}_2,{\mathit{B}}_4$, respectively,

$$\begin{gathered} {\mathit{B}}_1={\left\{\left(1-{\zeta }_2\right)\left(\mathbf{I}-{\mathit{\Gamma }}_1\right)-{\mathit{\zeta }}_1{\left(\mathit{\kappa }+{\mathit{\gamma }}_2\right)}^{\top }\right\}}^{-1}\left\{\left(1-{\zeta }_2\right){\mathit{\delta }}_1+{\delta }_2{\mathit{\zeta }}_1\right\}\in {\mathbb{R}}^d, \\ \left(\genfrac{}{}{0pt}{}{{\mathit{B}}_2}{B_{3j}}\right)={\left(\mathbf{I}-{\mathit{B}}_6-{\mathit{B}}_7\right)}^{-1}{\mathit{B}}_7\left(\genfrac{}{}{0pt}{}{{\mathit{\theta }}_{M_{1j}\to {\mathit{M}}_1}}{{\theta }_{M_{1j}\to R_1}}\right)\in {\mathbb{R}}^{d+1}, \\ \left(\genfrac{}{}{0pt}{}{{\mathit{B}}_4}{B_{5j}}\right)={\left(\mathbf{I}-{\mathit{B}}_6-{\mathit{B}}_7\right)}^{-1}{\mathit{B}}_7\left(\genfrac{}{}{0pt}{}{{\mathit{\theta }}_{M_{1j}\to {\mathit{M}}_1}}{{\theta }_{M_{1j}\to R_1}}\right)B_{1j}\in {\mathbb{R}}^{d+1}, \\ {\mathit{B}}_6=\left(\begin{array}{cc}{\mathbf{0}}_{d\times d}& {\mathbf{0}}_d\\ {\mathit{\kappa }}^{\top }& 0\end{array}\right)\in {\mathbb{R}}^{(d+1)\times (d+1)}\mathit{\text{and}}{\mathit{B}}_7=\left(\begin{array}{cc}{\mathit{\Gamma }}_1& {\mathit{\zeta }}_1\\ {\mathit{\gamma }}_2^{\top }& {\zeta }_2\end{array}\right)\in {\mathbb{R}}^{(d+1)\times (d+1)}. \end{gathered}$$ (25)

As for estimation, based on Theorem 2, we pool the data across all $T$ stages to estimate the model parameters $\mathit{W},{𝚯}_1,{𝚯}_2$. Recall that, in the infinite-horizon setting, we require the DAG to be stationary over time. We thus apply DAGMA to the data pooled across all time points to learn this time-invariant DAG. This is different from the finite-horizon setting where the model parameters can differ from one stage to another, while in the infinite-horizon setting, they remain the same. We then estimate the quantities ${\mathit{B}}_1$ to ${\mathit{B}}_7$ in (25) by plugging in the estimates of the corresponding model parameters, and we estimate the individual mediation effect using (24). As for inference, we adopt a similar nonparametric bootstrap procedure as in the finite-horizon setting. Algorithm 2 summarizes our estimation and inference procedure.

3.5. Asymptotic theory.

We establish the asymptotic properties of our estimator of the individual mediation effect. We first present a set of regularity conditions.

Assumption 1 (Invertibility). Let ${\mathit{X}}_t\equiv {\left(1,A_t,{\mathit{M}}_{t-1}^{\top },R_{t-1},{\mathit{M}}_t^{\top }\right)}^{\top }\in {\mathbb{R}}^{2d+3}$. There exists some constant $C>0$, such that ${\lambda }_{\text{min}}\left(\mathbb{E}{\mathit{X}}_t{\mathit{X}}_t^{\top }\right)\ge C$ for any $t$, where ${\lambda }_{\text{min}}(\cdot )$ denotes the minimum eigenvalue of a given matrix.

Algorithm 2.

Estimation for the infinite-horizon setting

Input: Observed data $\left\{\left(A_{it},{\mathit{M}}_{it},R_{it}\right):i=1,\dots ,n,t=1,\dots ,T\right\}$, the number of bootstrap samples $B$, and the significance level $\alpha$.
Output: Estimated individual mediation effect ${\widehat{\eta }}_j^{(\infty )}$, for $j=1,\dots ,d$.
1:	Pool data across all $t$ stages, $t=1,\dots ,T$.
2:	Estimate parameters $\widehat{\mathit{W}},{\widehat{𝚯}}_1,{\widehat{𝚯}}_2$ using the pooled data.
3:	Estimate ${\mathit{\theta }}_{M_{1j}\to {\mathit{M}}_1}$ and ${\theta }_{M_{1j}\to R_1}$ using the pooled data.
4:	Compute ${\mathit{B}}_1$ to ${\mathit{B}}_7$ using (25).
5:	Compute ${\widehat{\eta }}_j^{(\infty )}$ using (24).
6:	for $b=1,\dots ,B$ do
7:	Sample $n$ trajectories from the observed data with replacement.
8:	Repeat Lines 1 to 5 to compute ${\hat{\eta }}_j^{(\infty ,b)}$ using the bootstrap samples.
9:	end for
10:	Construct the CI $\left[{\widehat{\eta }}_j^{(\infty ,L)},{\widehat{\eta }}_j^{(\infty ,U)}\right]$ for ${\eta }_j^{(\infty )}$, where ${\widehat{\eta }}_j^{(\infty ,L)}$ and ${\widehat{\eta }}_j^{(\infty ,U)}$ are the empirical lower and upper $\alpha /2$ quantiles of ${\left\{{\widehat{\eta }}_j^{(\infty ,b)}\right\}}_{b=1}^B$, respectively.

Assumption 2 (Error residuals). (i) The error terms ${ϵ}_{w1t},{ϵ}_{w2t},\cdots ,{ϵ}_{wdt}$ in (3) are jointly normally distributed and independent, for $t=1,\dots ,T$. In addition, their variances are constant, in that $\text{Var}\left({ϵ}_{w1t}\right)=\text{Var}\left({ϵ}_{w2t}\right)=\cdots =\text{Var}\left({ϵ}_{wdt}\right)$. (ii) The error terms ${ϵ}_{rt}$ in (4) satisfy that $\mathbb{E}\left({ϵ}_{rt}^4\right)<\infty$, for $t=1,\dots ,T$.

Assumption 3 (Structure learning consistency). The estimated DAG is a consistent estimator of the true underlying DAG; that is, (i) for the finite-horizon setting, for $t=1,\dots ,T,\mathbb{P}\left(\widehat{{𝒢}_t}\ne {𝒢}_t\right)\to 0$ as $n\to \infty$, where ${𝒢}_t$ is the true DAG in stage $t$; (ii) for the infinite-horizon setting, $\mathbb{P}(\widehat{𝒢}\ne 𝒢)\to 0$ as $T\to \infty$, where $𝒢$ is the true time-invariant DAG.

Assumption 4 (Stationarity). For the infinite-horizon setting, the process $\left\{A_t,{\mathit{M}}_t\right.,{\left.R_t\right\}}_{t\ge 1}$ is a strictly stationary $\beta$-mixing process.

We give some remarks about these conditions. Assumption 1 guarantees the identifiability of model parameters in (4). Assumption 2(i) is commonly imposed for identifying DAG structures; see, for example, Peters and Bühlmann (2014), Nandy, Maathuis and Richardson (2017), Yuan et al. (2019), Shi and Li (2022). Assumption 2(ii) requires the fourth moments of the errors to be finite, which is mild. Assumption 3 holds for numerous DAG estimation algorithms, including the one by Bello, Aragam and Ravikumar (2022) that we use in our implementation. Assumption 4 is required for the infinite-horizon setting only, and the $\beta$-mixing condition is again commonly imposed; see, for example, Bradley (2005). Overall, these regularity conditions are reasonable.

We next establish the asymptotic normality of our individual mediation effect estimator for both the finite-horizon and infinite-horizon settings.

Theorem 3 (Asymptotic distribution for finite-horizon). Suppose Assumptions 2 and 3(i) hold. Then, for $t=1,\dots ,T$,

$$\sqrt{n}\left({\widehat{\eta }}_j^{\left(t\right)}-{\eta }_j^{\left(t\right)}\right)\stackrel{d}{\to }𝒩\left(0,{\sigma }_{tj}^2\right)\mathit{\text{as}}n\to \infty ,$$

where ${\sigma }_{tj}^2$ denotes the asymptotic variance of ${\widehat{\eta }}_j^{\left(t\right)}$. Its form is given in the Supplementary Material, and it can be consistently estimated via the bootstrap method.

Theorem 4 (Asymptotic distribution for infinite-horizon). Suppose Assumptions 2, 3(ii) and 4 hold. Then

$$\sqrt{nT}\left({\widehat{\eta }}_j^{(\infty )}-{\eta }_j^{(\infty )}\right)\stackrel{d}{\to }𝒩\left(0,{\sigma }_j^2\right)\mathit{\text{as}}nT\to \infty ,$$

where ${\sigma }_j^2$ denotes the asymptotic variance of ${\widehat{\eta }}_j^{(\infty )}$. Its definition is given in the Supplementary Material, and it can be consistently estimated using the bootstrap method.

We briefly remark that Theorem 3 requires the number of realizations $n$ of ($A_t,{\mathit{M}}_t,R_t$) to diverge to infinity for every $t=1,\dots ,T$, with $T$ being finite. Meanwhile, Theorem 4 requires either $n$ or the number of time points $T$ or both to diverge to infinity.

4. Simulations.

In this section, we investigate the empirical performance of our proposed method through intensive simulations. We also compare with some baseline methods.

4.1. Simulation setup.

We generate $n$ copies of random samples ${\left\{A_t,{\mathit{M}}_t,R_t\right\}}_{t=1}^T$ following models (3) and (4), that is,

$$\begin{gathered} {\mathit{M}}_t={\left(\mathbf{I}-{\mathit{W}}^{\top }\right)}^{-1}{\mathit{ϵ}}_{wt}+{\mathit{\alpha }}_{1t}+{\mathit{\delta }}_{1t}{A}_t+{𝚪}_{1t}{\mathit{M}}_{t-1}+{\mathit{\zeta }}_{1t}{R}_{t-1}, \\ {R}_t={\alpha }_{2t}+{\delta }_{2t}{A}_t+{\mathit{\gamma }}_{2t}^{\top }{\mathit{M}}_{t-1}+{\zeta }_{2t}{R}_{t-1}+{\mathit{\kappa }}_t^{\top }{\mathit{M}}_t+{ϵ}_{rt}. \end{gathered}$$

We generate the sequence of treatments $A_t\stackrel{\mathit{\text{i.i.d.}}}{~}\text{Bernoulli}\left(0.5\right)$, $t=1,\dots ,T$, and the error terms ${\mathit{ϵ}}_{wt}$ and ${ϵ}_{rt}$ from a standard normal distribution. Let ${𝚯}_{1t}={\left({\mathit{\alpha }}_{1t},{\mathit{\delta }}_{1t},{𝚪}_{1t},{\mathit{\zeta }}_{1t}\right)}^{\top }\in {\mathbb{R}}^{(d+3)\times d}$ and ${𝚯}_{2t}={\left({\alpha }_{2t},{\delta }_{2t},{\zeta }_{2t},{\mathit{\gamma }}_{2t}^{\top },{\mathit{\kappa }}_t^{\top }\right)}^{\top }\in {\mathbb{R}}^{(2d+3)}$ collect the model parameters, and we generate the entries of ${𝚯}_{1t},{𝚯}_{2t}$ from a uniform distribution on (−0.5, 0.5). For the finite-horizon setting, we generate different ${𝚯}_{1t},{𝚯}_{2t}$ for different time points $t$, whereas for the infinite-horizon setting, we only generate one copy of ${𝚯}_{1t},{𝚯}_{2t}$, and keep them fixed across all time points. We fix $d=3$, and generate the matrix $\mathit{W}\in {\mathbb{R}}^{3\times 3}$ in two steps. We first begin with a zero matrix, then replace every entry $(\mathit{W}{)}_{ij},i<j$, by the product of two random variables $E_{ij}^{\left(1\right)}\times E_{ij}^{\left(2\right)}$, where $E_{ij}^{\left(1\right)}$ is a Bernoulli variable with probability 0.9, indicating a random directed edge is added from mediator $M_i$ to $M_j$, and $E_{ij}^{\left(2\right)}$ is the edge weight, which is randomly drawn from a uniform distribution on $[-0.9,-0.5]\cup [0.5,0.9]$. Following this generation process, we obtain $\mathit{W}=\left(\begin{array}{ccc}0& -0.80& 0.61\\ 0& 0& -0.82\\ 0& 0& 0\end{array}\right)$. We set the initial values of mediators and outcome all equal to 0.

We recognize that there is no existing solution in the literature for this problem. Instead, we compare with two baseline solutions. One method is termed “independent time points,” which ignores all temporal dependence across different time points. That is, it estimates the individual mediation effect at every single time stage with stage-specific data, without taking into account the dependence to prior stages nor the carryover effects. The other method is termed “independent mediators,” which ignores all dependence among the multivariate mediators, and essentially treats $\mathit{W}$ as a zero matrix. We evaluate all estimation methods using three criteria: the estimation bias, the empirical standard error (SE) and the coverage probability (CP). For inference, we set the number of bootstrapped samples to be 500 and the significance level $\alpha$ to be 0.05. We employ DAGMA with linear models for structural learning, and the corresponding hyperparameters, w_threshold and lambda1, are set to 0.1 and 0.0, respectively.

4.2. Finite-horizon setting.

For the finite-horizon setting, we consider the number of time points $T\in \{10,20,30\}$ and the sample size $n\in \{100,250,500\}$. Note that the true individual mediation effect cannot be derived analytically in our setting, and is computed numerically based on 10 million Monte Carlo samples. Table 1 reports the results based on 500 replications. We observe that, when the number of subjects $n$ increases, the estimation bias and standard error both decrease for our proposed method, which agrees with our theory. Moreover, our method achieves the desired coverage probability across all scenarios. The same is not true for the two baseline methods. For instance, for the independent mediators method, the estimation bias of ${\eta }_2^{\left(T\right)}$ and ${\eta }_3^{\left(T\right)}$ does not decrease. This is because $M_{t1}$ is in the parent set of $M_{t2}$ and $M_{t3}$ for all $t=1,\dots ,T$, and ignoring the effects along the paths $M_{t1}\to M_{t2}$ and $M_{t1}\to M_{t3}$ lead to a larger bias and invalid coverage probability in estimating ${\eta }_2^{\left(T\right)}$ and ${\eta }_3^{\left(T\right)}$. On the contrary, the estimation bias of ${\eta }_1^{\left(T\right)}$ is much smaller, since the parent set of $M_{t1}$ is empty given (${\mathit{M}}_{t-1},R_t,A_t$) in our simulation example.

Table 1.

Simulations for the finite-horizon setting: the bias, the empirical standard error (SE) and the coverage probability (CP) between the estimated and true individual mediator effects with varying number of time points T and number of subjects n

$n$			100			250			500
Method	$T$	Param	Bias	SE	CP	Bias	SE	CP	Bias	SE	CP
Proposed method	10	${\eta }_1^{\left(T\right)}$	0.000	0.594	0.972	0.003	0.351	0.966	0.008	0.249	0.960
		${\eta }_2^{\left(T\right)}$	−0.016	0.533	0.960	−0.036	0.283	0.970	−0.006	0.209	0.952
		${\eta }_3^{\left(T\right)}$	0.007	0.696	0.964	0.023	0.420	0.962	0.011	0.312	0.950
	20	${\eta }_1^{\left(T\right)}$	−0.046	1.960	0.950	0.008	1.215	0.946	−0.023	0.840	0.950
		${\eta }_2^{\left(T\right)}$	−0.037	1.134	0.970	−0.018	0.717	0.944	−0.021	0.490	0.948
		${\eta }_3^{\left(T\right)}$	0.027	1.236	0.956	0.012	0.721	0.954	−0.003	0.491	0.972
	30	${\eta }_1^{\left(T\right)}$	0.044	2.655	0.948	−0.078	1.691	0.938	0.055	1.159	0.940
		${\eta }_2^{\left(T\right)}$	−0.067	1.610	0.962	−0.002	0.915	0.964	−0.020	0.651	0.954
		${\eta }_3^{\left(T\right)}$	−0.067	1.838	0.958	−0.055	1.122	0.960	0.051	0.783	0.946
Independent time points	10	${\eta }_1^{\left(T\right)}$	−0.577	0.373	0.712	−0.558	0.204	0.370	−0.569	0.131	0.132
		${\eta }_2^{\left(T\right)}$	0.199	0.759	0.962	0.171	0.447	0.926	0.175	0.308	0.888
		${\eta }_3^{\left(T\right)}$	−0.768	1.472	0.896	−0.762	0.861	0.866	−0.716	0.656	0.806
	20	${\eta }_1^{\left(T\right)}$	0.823	1.301	0.882	0.844	0.665	0.724	0.852	0.441	0.580
		${\eta }_2^{\left(T\right)}$	6.158	4.075	0.636	6.071	2.479	0.286	6.075	1.797	0.066
		${\eta }_3^{\left(T\right)}$	0.694	3.095	0.952	0.665	1.804	0.936	0.683	1.315	0.914
	30	${\eta }_1^{\left(T\right)}$	1.662	1.491	0.854	1.626	0.831	0.598	1.607	0.530	0.370
		${\eta }_2^{\left(T\right)}$	−10.111	6.992	0.700	−9.898	4.801	0.410	−9.989	3.207	0.126
		${\eta }_3^{\left(T\right)}$	−2.352	1.880	0.860	−2.446	1.081	0.374	−2.392	0.715	0.144
Independent mediators	10	${\eta }_1^{\left(T\right)}$	0.000	0.594	0.972	0.003	0.351	0.954	0.008	0.249	0.964
		${\eta }_2^{\left(T\right)}$	−0.139	0.640	0.944	−0.175	0.342	0.944	−0.142	0.263	0.904
		${\eta }_3^{\left(T\right)}$	0.164	0.603	0.942	0.178	0.368	0.938	0.172	0.276	0.892
	20	${\eta }_1^{\left(T\right)}$	−0.045	1.959	0.948	0.008	1.215	0.950	−0.023	0.840	0.952
		${\eta }_2^{\left(T\right)}$	−0.159	1.370	0.956	−0.135	0.814	0.950	−0.157	0.576	0.930
		${\eta }_3^{\left(T\right)}$	−0.764	1.390	0.926	−0.728	0.830	0.882	−0.779	0.602	0.772
	30	${\eta }_1^{\left(T\right)}$	0.044	2.656	0.946	−0.078	1.691	0.932	0.055	1.159	0.948
		${\eta }_2^{\left(T\right)}$	−1.286	2.087	0.922	−1.154	1.114	0.872	−1.227	0.864	0.692
		${\eta }_3^{\left(T\right)}$	−0.741	2.128	0.934	−0.724	1.279	0.892	−0.632	0.920	0.884

4.3. Infinite-horizon setting.

For the infinite-horizon setting, we consider the number of time points $T\in \{100,250,500\}$ and the sample size $n\in \{20,50,100\}$. Again, the true individual mediation effect cannot be derived analytically in our setting, and is computed numerically based on 10 million Monte Carlo samples with 3000 time points. We also drop the first five steps as a warm up to mitigate the potential influence of the initial conditions. Table 2 reports the results based on 500 data replications. We again observe that the estimation bias and standard error both decrease for our method as $n$ increases, and the desired coverage probability is achieved, across all scenarios. On the other hand, the bias increases sharply for the independent time points method, because ignoring the carryover effects becomes exaggerated with a large $T$. Moreover, the estimation bias of the independent mediators method is much larger than the proposed method for ${\eta }_2^{(\infty )}$ and ${\eta }_3^{(\infty )}$. Both baseline methods fail to achieve the desired coverage probability in most cases, except for the independent mediators method with ${\eta }_1^{(\infty )}$. This is due to the specific simulation setting with an empty parent set given the history. In general, these results demonstrate the importance of taking into account both the dependence among multiple time points and among multiple mediators.

Table 2.

Simulations for the infinite-horizon setting: the bias, the empirical standard error (SE) and the coverage probability (CP) between the estimated and true individual mediator effects with varying number of time points T and number of subjects n

$n$			20			50			100
Method	T	Param	Bias	SE	CP	Bias	SE	CP	Bias	SE	CP
Proposed method	100	${\eta }_1^{\left(\infty \right)}$	0.002	0.070	0.918	0.002	0.046	0.928	0.001	0.029	0.934
		${\eta }_2^{\left(\infty \right)}$	0.003	0.045	0.924	0.001	0.029	0.932	0.001	0.020	0.940
		${\eta }_3^{\left(\infty \right)}$	0.003	0.036	0.924	0.002	0.021	0.946	0.000	0.015	0.946
	250	${\eta }_1^{\left(\infty \right)}$	−0.001	0.042	0.932	−0.002	0.026	0.946	−0.001	0.019	0.930
		${\eta }_2^{\left(\infty \right)}$	−0.003	0.029	0.914	−0.001	0.019	0.938	0.001	0.012	0.958
		${\eta }_3^{\left(\infty \right)}$	0.001	0.020	0.938	0.000	0.014	0.938	−0.001	0.010	0.950
	500	${\eta }_1^{\left(\infty \right)}$	0.000	0.029	0.944	0.000	0.018	0.956	−0.001	0.014	0.932
		${\eta }_2^{\left(\infty \right)}$	0.000	0.020	0.928	−0.001	0.012	0.954	0.000	0.009	0.942
		${\eta }_3^{\left(\infty \right)}$	0.000	0.016	0.924	−0.001	0.010	0.932	0.000	0.007	0.928
Independent time points	100	${\eta }_1^{\left(\infty \right)}$	0.371	0.047	0.000	0.370	0.030	0.000	0.371	0.021	0.000
		${\eta }_2^{\left(\infty \right)}$	−0.065	0.076	0.818	−0.070	0.049	0.658	−0.071	0.034	0.404
		${\eta }_3^{\left(\infty \right)}$	0.132	0.086	0.628	0.133	0.060	0.314	0.132	0.040	0.076
	250	${\eta }_1^{\left(\infty \right)}$	0.372	0.027	0.000	0.371	0.018	0.000	0.371	0.013	0.000
		${\eta }_2^{\left(\infty \right)}$	−0.075	0.048	0.616	−0.073	0.030	0.300	−0.073	0.020	0.052
		${\eta }_3^{\left(\infty \right)}$	0.134	0.055	0.286	0.135	0.034	0.028	0.133	0.024	0.000
	500	${\eta }_1^{\left(\infty \right)}$	0.370	0.021	0.000	0.371	0.013	0.000	0.371	0.010	0.000
		${\eta }_2^{\left(\infty \right)}$	−0.073	0.035	0.368	−0.074	0.021	0.058	−0.073	0.015	0.002
		${\eta }_3^{\left(\infty \right)}$	0.131	0.039	0.074	0.131	0.024	0.000	0.134	0.018	0.000
Independent mediators	100	${\eta }_1^{\left(\infty \right)}$	0.002	0.070	0.918	0.002	0.046	0.934	0.001	0.029	0.936
		${\eta }_2^{\left(\infty \right)}$	0.322	0.031	0.000	0.324	0.019	0.000	0.325	0.015	0.000
		${\eta }_3^{\left(\infty \right)}$	−0.264	0.038	0.000	−0.266	0.024	0.000	−0.265	0.016	0.000
	250	${\eta }_1^{\left(\infty \right)}$	−0.001	0.042	0.934	−0.002	0.026	0.934	−0.001	0.019	0.942
		${\eta }_2^{\left(\infty \right)}$	0.324	0.019	0.000	0.324	0.012	0.000	0.325	0.009	0.000
		${\eta }_3^{\left(\infty \right)}$	−0.265	0.024	0.000	−0.265	0.014	0.000	−0.265	0.010	0.000
	500	${\eta }_1^{\left(\infty \right)}$	0.000	0.029	0.936	0.000	0.018	0.952	−0.001	0.014	0.936
		${\eta }_2^{\left(\infty \right)}$	0.324	0.014	0.000	0.325	0.009	0.000	0.324	0.006	0.000
		${\eta }_3^{\left(\infty \right)}$	−0.265	0.017	0.000	−0.265	0.011	0.000	−0.265	0.007	0.000

5. Data application.

In this section, we revisit the motivating Intern Health Study (NeCamp et al. (2020)). The IHS is a 26-week sequentially randomized trial with the objective of understanding the biological mechanisms of depression, and the ultimate goal of improving the mental health outcomes of medical interns in the United States. The study developed and deployed a mobile app to deliver prompt notifications, such as reminders to have a break, take a walk or prioritize sleep, which aims to improve the well-being of interns who often work under stressful environments. In each week, each intern was randomized into receiving the notifications or not. Meanwhile, wearable devices (Fitbit) recorded daily measurements of step count (Step), sleep duration (Sleep, minutes), resting heart rate (RHR, beats per minute) and heart rate variability (HRV, milliseconds). Interns also self-reported a daily mood score in the app (Shaffer and Ginsberg (2017)). We formulate the problem in the framework of multivariate dynamic mediation analysis, with the binary status of receiving the notification or not as the treatment, the transformed measurements, that is, the cubic-root of step count, the square-root of sleep duration, RHR and HRV as $d=4$ potential mediators and the mood score as the outcome. We average all the measurements within each week, resulting in $T=26$ weeks of data, for $n=1196$ interns undergoing the sequential randomization.

We apply our method to this data adopting the finite-horizon setting. This is partly because we have a relatively limited number of time points $T=26$ in this study, and partly because the data is likely to be nonstationary over time (NeCamp et al. (2020), Li et al. (2022), Wang, Shi and Wu (2023)). Our analysis, as shown in Figure 4, also confirms that the individual mediation effect varies over time. Figure 3(a) shows the estimated DAG structure among the four mediators, while Figures 3(b) and (c) show the estimated individual mediation effects over time, along with the associated 95% confidence intervals, that are smoothed with a natural cubic spline and further decomposed as the immediate and delayed effects.

Fig. 4.

Analysis of IHS mobile health data: the estimated effects from the treatment to the four individual mediators, decomposed as the immediate (first row) and delayed (second row) effects, respectively.

Fig. 3.

Analysis of IHS mobile health data: (a) the estimated DAG among the four mediators, dashed/solid arrows represent negative/positive effects; (b) the estimated immediate individual mediation effects over time; (c) the estimated delayed individual mediation effects over time. The shaded area indicates the 95% confidence interval.

We make a number of observations. First of all, we see from Figure 3(a) that, the four mediators have complex relationships between each other, indicating the importance of accounting for the dependence structure among the mediators. Second, we see from Figure 3(b) and (c) that the magnitude of the delayed effects is generally greater than that of the immediate effects, indicating the importance of accounting for the carryover effects in our understanding of the effects of push notification to intern’s mood. Third, most individual mediation effects are insignificant. This could potentially be attributed to the relatively weak treatment effect, as also noted by NeCamp et al. (2020). Fourth, there is a significantly negative carryover effect from the push notification to mood score mediated by the RHR around week 10. Meanwhile, this carryover effect is predominantly negative across most weeks. To illustrate this effect, Figure 4 further plots the smoothed estimated effects from the treatment to the four individual mediators, decomposed as the immediate and delayed effects, respectively. We see that the push notification leads to an increased RHR after week five, suggesting that the push notification can lead to a lower mood score by increasing the RHR. This agrees with our prior knowledge that an increased RHR usually indicates a more tired and stressful state, thus a worse mood. Finally, the carryover effect from the push notification to sleep duration is mostly negative, and is nearly significant between weeks 15 and 20. As seen in Figure 4, the push notification generally results in reduced sleep hours, aligning with our knowledge that fewer sleep hours lead to poorer mood, too.

6. Conclusion and discussion.

In this section, we discuss the relation between our individual mediation effect and the Granger causality, numerous other mediation effects that may be of potential interest and some additional extensions.

6.1. Connection to Granger causality.

The Granger causality is a concept that leverages the temporal ordering inherent to time series so to draw causal statements restricted to the “past” causing the “future” (Granger (1969)). A variable is said to Granger-cause the other, if including this variable’s past values provides valuable information for forecasting the other variable’s future behavior. The proposed models in (3) and (4) are mathematically similar to those under the Granger causality framework, since $R_t$ may be predicted by its past value $R_{t-1},{\mathit{M}}_t$ and ${\mathit{M}}_{t-1}$ and ${\mathit{M}}_t$ also depends on its past value ${\mathit{M}}_{t-1}$ and $R_{t-1}$.

However, our notion of individual mediation effect differs from the concept of Granger causality, in that a mediator’s individual mediation effect and its ability to Granger-cause the outcome is not equivalent. On the one hand, a mediator can have a zero individual mediation effect, but still Granger-causes the outcome, if it is not influenced by the treatment. Figure 5, Case 1, gives an illustrative example. In this case, we have a single mediator that impacts the outcome at each time point, thereby Granger-causing the outcome. However, since the mediator is unaffected by the treatment, there is no direct causal pathway from the treatment through the mediator to the outcome, resulting in a zero individual mediation effect. On the other hand, a mediator with a nonzero individual mediation effect might not necessarily Granger-cause the outcome, if it does not improve the forecast accuracy of the outcome beyond other mediators. Figure 5, Case 2, gives another example. In this case, the first set of mediators $M_{11}$ and $M_{21}$ directly influence the outcome, thus Granger-causing the outcome. The second mediator $M_{12}$, however, affects the outcome indirectly through its influence on the first mediator $M_{21}$, resulting in a nonzero individual mediation effect. Despite this, $M_{12}$ does not Granger-cause the outcome $R_1$ or $R_2$, as its influence is mediated indirectly.

Fig. 5.

An illustration of the difference between the individual mediation effect and the Granger causality.

6.2. Sequential mediation effect.

As discussed in Section 2.3, the individual mediation effect ${\eta }_j^{\left(T\right)}$ measures the cumulative effect mediated through the $j$ th mediator $M_j$ across all the upstream mediators and all $T$ time points. Meanwhile, it may be of separate interest to analyze the portion of the individual mediation effect of $M_j$ that is attributed to its upstream mediator $M_k$. We denote this effect as ${\eta }_{k\to j}$, and call it the sequential mediation effect, as it quantifies the effect that is sequentially transmitted from the treatment, through the $k$th mediator, to the $j$ th mediator, then to the outcome.

We next formally define the sequential mediation effect. Toward that end, we require the DAG structure to remain stationary over time. That is, if the $k$th mediator is an ancestor of the $j$ th mediator at any time $t$, that is, there exists a direct path from $M_{tk}$ to $M_{tj}$, then the same relation holds at all time points. For the finite-horizon setting, we define the sequential mediation effect over $T$ time points as

$${\eta }_{k\to j}^{\left(T\right)}=\sum _{t=1}^T\left\{\frac{\partial }{\partial a}\mathbb{E}\left[R_t\mid \text{do}\left(A_s=a,\forall s\le t\right)\right]-\frac{\partial }{\partial a}\mathbb{E}\left[R_t\mid \text{do}\left(A_s=a,M_{sj}=m,\forall s\le t\right)\right]\right.-\frac{\partial }{\partial a}\mathbb{E}\left[R_t\mid \text{do}\left(A_s=a,M_{sk}=m,\forall s\le t\right)\right]\left.+\frac{\partial }{\partial a}\mathbb{E}\left[R_t\mid \text{do}\left(A_s=a,M_{sj}=M_{kj}=m,\forall s\le t\right)\right]\right\},$$

if the $k$th mediator is an ancestor of the $j$ th mediator, and set ${\eta }_{k\to j}^{\left(T\right)}=0$ otherwise. In this definition, the first term corresponds to the total treatment effect, the second and third terms measure the effects that do not go through the $j$ th or the $k$th mediator, respectively, and the last term quantifies the effect that does not simultaneously pass both the $j$ th and $k$th mediators. Then, by the principle of inclusion-exclusion, ${\eta }_{k\to j}^{\left(T\right)}$ measures the desired sequential mediation effect. For the infinite-horizon setting, we define ${\eta }_{k\to j}^{(\infty )}={\text{lim}}_{T\to \infty }{\eta }_{k\to j}^{\left(T\right)}/T$, provided the limit exists.

To further illustrate this definition, we revisit the examples in Figure 2. For the single-stage example in Figure 2, left panel, the mediator $M_{12}$ is a child of $M_{11}$ and is affected by $M_{11}$. Its individual mediation effect that passes through $M_{11}$ is

$${\eta }_{1\to 2}^{\left(1\right)}=\frac{\partial }{\partial a}\mathbb{E}\left[R_1\mid \text{do}\left(A_1=a\right)\right]-\frac{\partial }{\partial a}\mathbb{E}\left[R_1\mid \text{do}\left(A_1=a,M_{12}=m\right)\right]$$

$$-\frac{\partial }{\partial a}\mathbb{E}\left[R_1\mid \text{do}\left(A_1=a,M_{11}=m\right)\right]+\frac{\partial }{\partial a}\mathbb{E}\left[R_1\mid \text{do}\left(A_1=a,M_{11}=M_{12}=m\right)\right]$$

$$={\theta }_{A_1\to R_1}-{\theta }_{A_1\to R_1}^{M_{12}}-{\theta }_{A_1\to R_1}^{M_{11}}+{\theta }_{A_1\to R_1}^{M_{11},M_{12}}$$

$$={\theta }_{A_1\to M_{11}}{\theta }_{M_{11}\to M_{12}}{\theta }_{M_{12}\to R_1}.$$

It is clear to see that the term ${\theta }_{A_1\to M_{11}}{\theta }_{M_{11}\to M_{12}}{\theta }_{M_{12}\to R_1}$ corresponds to the effect along the path $A_1\to M_{11}\to M_{12}\to R_1$, which coincides with the results from the path analysis. Similarly, for the two-stage example in Figure 2, right panel, we have that

$$\begin{gathered} {\eta }_{1\to 2}^{\left(2\right)}={\eta }_{1\to 2}^{\left(1\right)}+\frac{\partial }{\partial a}\mathbb{E}\left[R_2\mid \text{do}\left(A_1=A_2=a\right)\right] \\ -\frac{\partial }{\partial a}\mathbb{E}\left[R_2\mid \text{do}\left(A_1=A_2=a,M_{11}=M_{21}=m\right)\right] \\ -\frac{\partial }{\partial a}\mathbb{E}\left[R_2\mid \text{do}\left(A_1=A_2=a,M_{12}=M_{22}=m\right)\right] \\ +\frac{\partial }{\partial a}\mathbb{E}\left[R_2\mid \text{do}\left(A_1=A_2=a,M_{11}=M_{21}=M_{12}=M_{22}=m\right)\right] \\ ={\eta }_{1\to 2}^{\left(1\right)}+{\theta }_{A_1,A_2\to R_2}-{\theta }_{A_1,A_2\to R_2}^{M_{11},M_{21}}-{\theta }_{A_1,A_2\to R_2}^{M_{12},M_{22}}+{\theta }_{A_1,A_2\to R_2}^{M_{11},M_{12},M_{21},M_{22}} \\ ={\eta }_{1\to 2}^{\left(1\right)}+{\theta }_{A_1\to R_2}-{\theta }_{A_1\to R_2}^{M_{11},M_{21}}-{\theta }_{A_1\to R_2}^{M_{12},M_{22}}+{\theta }_{A_1\to R_2}^{M_{11},M_{21},M_{12},M_{22}} \\ +{\theta }_{A_2\to R_2}-{\theta }_{A_2\to R_2}^{M_{21}}-{\theta }_{A_2\to R_2}^{M_{22}}+{\theta }_{A_2\to R_2}^{M_{21},M_{22}}, \end{gathered}$$

where the terms in the last two lines of the above equation can be calculated recursively with the formulas in equations (21) and (22), allowing for the estimation and inference of the sequential mediation effect.

Finally, we extend beyond the above illustrative examples to outline the estimation and inference procedure for the sequential mediation effect under a general setting involving $d$ mediators across $T$ time points. As for estimation, we begin by applying an existing structural learning algorithm to the observed data to learn the DAG structure. From the estimated DAG, if the $k$th mediator is not an ancestor of the $j$ th mediator, we set the estimated sequential effect to zero. Otherwise, we obtain the estimated ${\widehat{\eta }}_{k\to j}$ through the quantities ${\theta }_{A_1,\dots ,A_t\to R_t},{\theta }_{A_1,\dots ,A_t\to R_t}^{M_{1j},\dots ,M_{tj}}$, and ${\theta }_{A_1,\dots ,A_t\to R_t}^{M_{1k},M_{1j},\dots ,M_{kj},M_{tj}}$, which in turn can be recursively updated following the method developed in Section 3. As for inference, we again apply the bootstrap approach in Section 3 to quantify the uncertainty of this plug-in estimator ${\widehat{\eta }}_{k\to j}$.

6.3. Direct, indirect and conditional effects.

We discuss the direct effect DE and indirect effect IE in Section 2.2. We now briefly discuss their estimation and inference, which may be of separate interest. Recall DE in (6) under linear SEMs. It satisfies a recursive equation,

$${\text{DE}}^{\left(t\right)}={\text{DE}}^{(t-1)}{\zeta }_{2t}+{\delta }_{2t},$$

for $t=2,\dots ,T$, with ${\text{DE}}^{\left(1\right)}={\delta }_{21}$. Here, we add the superscript $\left(t\right)$ to DE to indicate the direct effect across $t$ time points. This recursive formulation allows us to employ a similar plug-in method as in Section 3 to estimate DE, as well as a similar bootstrap approach for inference. To estimate IE, we first compute the total effect as a byproduct of the proposed procedure in Section 3. We then estimate IE as the difference between the estimated total effect and DE. Again, we can employ the bootstrap approach to infer IE.

Our definition of the individual mediation effect is concerned with the marginal effect,

$$\frac{\partial }{\partial a}\mathbb{E}\left[R_t\mid \text{do}\left(A_t=a\right)\right]-\frac{\partial }{\partial a}\mathbb{E}\left[R_t\mid \text{do}\left(A_t=a,M_{tj}=m\right)\right],$$

which marginalizes over the history $\left\{A_s,{\mathit{M}}_s,R_s\right\}$ for all $s<t$. This marginalization is appropriate because our interventions on the treatment and mediator at time $t$ do not causally influence those historical variables. Henceforth, the distribution of those historical variables remains unchanged before and after the intervention. Meanwhile, it may be of separate interest to study the conditional effect,

$$\frac{\partial }{\partial a}\mathbb{E}\left[R_t\mid \text{do}\left(A_t=a\right),\left\{A_s,{\mathit{M}}_s,R_s\right\},\forall s<t\right]-\frac{\partial }{\partial a}\mathbb{E}\left[R_t\mid \text{do}\left(A_t=a,M_{tj}=m\right),\left\{A_s,{\mathit{M}}_s,R_s\right\},\forall s<t\right],$$

given the history. Nevertheless, we note that, under the linear structural model, the conditional effect is equivalent to the marginal effect that we target in this article. We next sketch a few lines to show the equivalence. Specifically, the first term is $\frac{\partial }{\partial a}\mathbb{E}\left[R_t\mid \text{do}\left(A_t=\right.\right.a\left)\right]={\theta }_{A_t\to R_t}={\beta }_{A_t,R_t\mid \left\{A_s,{\mathit{M}}_s,R_s\right\},\forall s<t}$ in our current definition. It is the partial coefficient of $A_t$ by linearly regressing $R_t$ on $A_t$ given the historical variables that could affect the distribution of $A_t$. Under the linear model structures (3) and (4), the effects of $A_t$ and the historical variables on $R_t$ are additive. Consequently, the resulting conditional effect $\frac{\partial }{\partial a}\mathbb{E}\left[R_t\mid \text{do}\left(A_t=a\right),\left\{A_s,{\mathit{M}}_s,R_s\right\},\forall s<t\right]={\beta }_{A_t,R_t\mid \left\{A_s,{\mathit{M}}_s,R_s\right\},\forall s<t}$ remains the same as the marginal effect, regardless of the values of the historical variables. The second term is $\frac{\partial }{\partial a}\mathbb{E}\left[R_t\mid \text{do}\left(A_t=a,M_{tj}=m\right)\right]={\theta }_{A_t\to R_t}^{M_{tj}}={\theta }_{A_t\to R_t}-{\theta }_{A_t\to M_{tj}}{\theta }_{M_{tj}\to R_t}$. Again, the value ${\theta }_{A_t\to R_t}$ remains the same whether or not conditioning on the historical variables. Similarly, under (3) and (4), the effects of $M_{tj}$ and the historical variables on $R_t$ are additive. Consequently, the value of the product ${\theta }_{A_t\to M_{tj}}{\theta }_{M_{tj}\to R_t}$ remains the same, regardless of whether the history is included in the conditioning set or not.

6.4. Extension to varying $T$ and time lags.

Finally, we remark that our method can be adapted to accommodate the setting when the number of time points $T$ varies among subjects, and when the time lags between two time points differ.

For the finite-horizon setting, $T$ can be different for different subjects, because the model parameters and intermediate quantities at any given time $t$ are estimated by pooling the data across subjects who have observations available at that time point. Besides, Theorem 3 remains valid when $T$ varies across subjects, provided that the sample size at each time point is sufficiently large. Meanwhile, for the time lag between two time points, we require it to follow the same distribution across all subjects, but it does not need to follow the same distribution across all time points, since we allow the data generating process to be nonstationary over time.

For the infinite-horizon setting, $T$ again can be different, because the data are pooled across both subjects and time points. Similarly, Theorem 4 remains valid when $T$ varies across subjects, provided that the total sample size is sufficiently large. Meanwhile, the time lag between two time points needs to follow the same distribution, due to the requirement for stationarity for the infinite-horizon setting. However, this requirement does not extend to between subjects, and the distributions can vary among different subjects. In that case, we can use each subject’s own data to estimate their subject-specific mediation effects.

Supplementary Material

Supplement to “Multivariate dynamic mediation analysis under a reinforcement learning framework” (DOI: 10.1214/24-AOS2475SUPP; .pdf). The supplement provides the proofs of the main theorems in the paper.