Estimating the effects of treatment regimes over the course of chronic disease: A multi-state causal framework with baseline confounding

Abstract

The development of chronic disease is a long-term process involving multiple endpoints. Although multi-state Cox models can estimate state-specific survival risks over time, they are not well suited for comparing the effectiveness of treatment regimes. A discrete-time split-state framework has been proposed, which divides disease states into substates by conditioning on past history. As this framework is both “memoryless” and “memorable,” the transition rates can be synthesized into summary measures, multimorbidity-adjusted life year (MALY) and substate-specific life year (SSLY). Building on this framework, we propose to investigate the causal effects of static and dynamic treatment regimes over the disease course under the assumptions of constant baseline confounders and instantaneous effects of interventions on transition rates. We identify the optimal treatment regime as the one that maximizes MALY and use SSLY to elucidate the mechanisms of how treatments influence disease progression. In the application, we identified the optimal weight targets in the ARIC study by modeling the disease course in healthy, at-metabolic-risk, coronary heart disease, heart failure, and mortality states. The estimated MALY was 1.80 years higher (95% CI: 0.62, 2.78) under regime “Normal weight initially and change to overweight if age >65 y” compared to regime “Normal weight across all states.” SSLY decomposition indicates that this gain arises from increased life year in all substates except the healthy state. In summary, our method provides a framework to evaluate the health benefits of treatment regimes over the disease course and has the potential to improve the precision prevention of chronic diseases.

1. Introduction

Comparative effectiveness of interventions has been widely studied in medical research. The interventions can be categorized into static regimes, where all treatments during follow-up are predetermined at baseline, and dynamic regimes, where treatment at each time point depends on an individual’s response to prior treatments. The health effects of treatment regimes are commonly measured by estimating the risk of disease development using the parametric g-formula or the marginal structure model (MSM).^1,2 These models first estimate conditional survival risks within each stratum of confounders, and then derive the marginal survival risk for the overall population through standardization (g-formula) or inverse probability weighting (MSM).

Rather than being defined by a single disease endpoint, the development of chronic disease is a long-term process involving multiple endpoints. Taking heart disease as an example, it begins with the development of hypertension, hypercholesterolemia, or hyperglycemia (i.e. an at-metabolic-risk state), followed by the onset of coronary heart disease (CHD), which may progress to heart failure and eventually end in mortality.^3,4 The disease states in this multi-state process are interconnected, and treatment at each state can influence the entire course of disease progression. However, existing methods that assess the effects of treatment regimes based on a single endpoint fail to capture the complexity of the disease course and may therefore misidentify the optimal treatment regime.

Multi-state modeling has been used to describe the course of chronic diseases. For example, multi-state Cox models use multiple Cox models to characterize transitions between disease states. ⁵ However, the estimated transition parameters are state- and time-specific, making it difficult to use them for evaluating the effects of treatment regimes. To address this, a discrete-time split-state framework for multi-state modeling has been proposed, ⁶ which divides disease states into substates by conditioning on past history. As the substates contain information on past history, the time-specific transition parameters can be synthesized into summary measures, multimorbidity-adjusted life year (MALY), and substate-specific life year (SSLY). ⁷ In this article, we extend this into a multi-state causal framework to evaluate the health benefits of treatment regimes over the disease course. Our method identifies the optimal treatment regime that generates the most benefits, as indicated by MALY, and elucidates the mechanisms through which treatment regimes affect disease progression, as indicated by SSLY. Our method is developed under the assumptions of constant baseline confounders and instantaneous effects of interventions on transition rates.

2. Methods

2.1. An introduction to the discrete-time multi-state framework

We applied a discrete-time multi-state framework to estimate the health benefits of treatment regimes over the disease course.^6,7 While this framework can be applied to chronic disease with any number of states, we illustrate it here using five states S₀-S₄ (Figure 1). In brief, the framework divides each disease state into substates by conditioning on past history and estimates transition rates between substates using cause-specific Cox models. For example, S₂ can be divided into substates S ${}_{2|0}$ and S ${}_{2|1\mathrm{\_}0}$ , where S ${}_{2|0}$ represents transitions from S₀ and S ${}_{2|1\mathrm{\_}0}$ represents transitions from S₀ and S₁. The framework has two unique features. First, it is “memoryless.” By conditioning on past states, the newly created substates are independent of past history and exhibit the Markov property, regardless of whether the original process is Markovian. Thus, regardless of whether the original process is Markov, the Aalen–Johansen estimator can be used to estimate state occupation probabilities (or survival risks) based on transition rates. Second, it is “memorable,” as the substates contain information of past states and indicate multimorbidity status. Thus, the state- and time-specific survival risks can be synthesized into summary measures of MALY and SSLY. ⁷ Specifically, MALY is used to identify the optimal treatment regime, and SSLY is used to elucidate the mechanisms.

Figure 1.

Construction of the process of disease substates by conditioning on past states. Note: This figure is cited from our previous publication. ⁶

Suppose we have a population of N participants. Let $m$ denote disease states, and consider a disease process that includes states $S_0,S_1,S_m,\dots$ , and $S_M$ . By conditioning on past history, the five states $S_0$ , $S_1$ , $S_2$ , $S_3$ , and $S_4$ can be divided into nine disease substates, $S_0$ , $S_1$ , $S_{2|0}$ , $S_{2|0\mathrm{\_}1},S_{3|0}$ , $S_{3|0\mathrm{\_}1}$ , $S_{3|0\mathrm{\_}2}$ , $S_{3|0\mathrm{\_}1\mathrm{\_}2}$ , and $S_4$ . Let $E_0,E_1,\dots ,E_{M-1}$ denote the exposures in states $S_0,S_1,S_m,\dots$ , and $S_{M-1}$ at time $t$ (note that there is no exposure in the final state $S_M$ ). After dividing states into substates, the corresponding exposures are denoted as $E_0$ , $E_1$ , $E_{2|0}$ , $E_{2|0\mathrm{\_}1},E_{3|0}$ , $E_{3|0\mathrm{\_}1}$ , $E_{3|0\mathrm{\_}2}$ , and $E_{3|0\mathrm{\_}1\mathrm{\_}2}$ . Let $\mathit{L}$ represent baseline confounders. Our method is developed under two assumptions: (a) $\mathit{L}$ are constant over the follow-up period; (b) The effects of exposure on the transition rates to subsequent states are instantaneous.

2.2. Estimating conditional transition rates

As described in detail in a previous publication, cause-specific Cox models are applied to examine the associations between exposure and transition rates. In brief, the data are divided into subsets by disease state: subset 1 starts from ${\mathrm{S}}_0=1$ and ends with the incidence of S₁, S₂, S₃, S₄, or the end of follow-up, whichever comes first; subset 2 starts from ${\mathrm{S}}_1=1$ and ends with the incidence of S₂, S₃, S₄ or the end of follow up; subset 3 starts from ${\mathrm{S}}_2=1$ and ends with the incidence of S₃, S₄ or the end of follow up; and subset 4 starts from ${\mathrm{S}}_3=1$ and ends with the incidence of S₄ or the end of follow up. Each subset is restructured from wide to long format by age. Cause-specific Cox models (Models 1–4) are fitted to each subset to estimate conditional transition rates. Model 1 in subset 1 starting from S₀ state:

$$h_{(S_m=1|S_0=1)}(\tau ,t,E_0,\mathit{L})=h_{0m}(\tau )\exp (f(t)+E_0+\mathit{L}),$$

where m ranges from 1 to 4, indicating the development of S₁, S₂, S₃, or S₄ states from S₀. $f(t)$ is a function of age. $h_{0m}(\tau )$ is the baseline hazard of developing state m. $\tau$ is time to event in each time interval and ranges from [0, 1] (we use age as the time scale, and thus the maximum length of each time interval is 1). Model 2 in subset 2 starting from S₁ state:

$$h_{(S_m=1|S_1=1,S_0=1)}(\tau ,t,E_1,\mathit{L})=h_{1m}(\tau )\exp (f(t)+E_1+\mathit{L}),$$

where m can be 2–4, indicating the development of S₂, S₃, or S₄ states. Model 3 in subset 3 starting from S₂ state:

$$h_{(S_m=1|S_2=1,J_{0\mathrm{\_}1})}(\tau ,t,J_{0\mathrm{\_}1},E_2,\mathit{L})=h_{2m}(\tau )\exp (f(t)+J_{0\mathrm{\_}1}+E_2+\mathit{L}),$$

where m can be 3 and 4, indicating the development of S₃ or S₄ states from S₂. $J_{0\mathrm{\_}1}$ is a joint indicator of the past states S₀ and S₁, and has two categories, “0” and “0_1.” When $J_{0\mathrm{\_}1}=\text{``0''}$ , $E_2$ corresponds to $E_{2|0}$ , and when $J_{0\mathrm{\_}1}=\text{``0\_1''}$ , $E_2$ corresponds to $E_{2|0\mathrm{\_}1}$ . Model 4 in subset 4 starting from S₃ state:

$$h_{(S_m=1|S_3=1,J_{0\mathrm{\_}2})}(\tau ,t,J_{0\mathrm{\_}2},E_3,\mathit{L})=h_{3m}(\tau )\exp (f(t)+J_{0\mathrm{\_}2}+E_3+\mathit{L}),$$

where m indicates the development of S₄ state from S₃. J ${}_{0\mathrm{\_}2}$ is a joint indicator of the past states S₀, S₁ and S₂, and has 4 categories, “0,” “0_1,” “0_2,” “0_1_2.” When $J_{0\mathrm{\_}2}=\text{``0''}$ , “0_1,” “0_2,” “0_1_2,” $E_3$ corresponds to $E_{3|0}$ , $E_{3|0\mathrm{\_}1}$ , $E_{3|0\mathrm{\_}2}$ , and $E_{3|0\mathrm{\_}1\mathrm{\_}2}$ , respectively.

The discrete-time hazard can be approximated by the age-specific risk at the end of each time interval, ⁸ which can be estimated using the cumulative incidence function (CIF) derived from cause-specific Cox models.^9,10 Specifically, Model 1 estimates ${\lambda }_{01}(t|E_0,\mathit{L})$ , ${\lambda }_{02}(t|E_0,\mathit{L})$ , ${\lambda }_{03}(t|E_0,\mathit{L})$ , and ${\lambda }_{04}(t|E_0,\mathit{L})$ , the conditional transition rates from S₀ to states S₁–S₄, respectively. Model 2 estimates ${\lambda }_{12}(t|E_1,\mathit{L})$ , ${\lambda }_{13}(t|E_1,\mathit{L})$ , and ${\lambda }_{14}(t|E_1,\mathit{L})$ , the conditional transition rates from S₁ to states S₂–S₄, respectively. Model 3 estimates ${\lambda }_{23|0}(t|E_{2|0},\mathit{L})$ and ${\lambda }_{24|0}(t|E_{2|0},\mathit{L}$ ), the conditional transition rates from S ${}_{2|0}$ to S₃ and S₄, respectively, and ${\lambda }_{23|0\mathrm{\_}1}(t|E_{2|0\mathrm{\_}1},\mathit{L})$ and ${\lambda }_{24|0\mathrm{\_}1}(t|E_{2|0\mathrm{\_}1},\mathit{L})$ , the conditional transition rates from S ${}_{2|0\mathrm{\_}1}$ to S₃ and S₄, respectively. Model 4 estimates ${\lambda }_{34|0}(t|E_{3|0},\mathit{L})$ , ${\lambda }_{34|0\mathrm{\_}1}(t|E_{3|0\mathrm{\_}1},\mathit{L})$ , ${\lambda }_{34|0\mathrm{\_}2}(t|E_{3|0\mathrm{\_}2},\mathit{L})$ , and ${\lambda }_{34|0\mathrm{\_}1\mathrm{\_}2}(t|E_{3|0\mathrm{\_}1\mathrm{\_}2},\mathit{L})$ , the conditional transition rates to S₄ from S ${}_{3|0}$ , S ${}_{3|0\mathrm{\_}1}$ , S ${}_{3|0\mathrm{\_}2}$ , and S ${}_{3|0\mathrm{\_}1\mathrm{\_}2}$ , respectively. These conditional transition rates are used to construct the conditional transition rate matrix $\mathit{Q}(t|\mathit{E},\mathit{L})$ as shown below, where $\mathit{E}=(E_0,E_1,E_{2|0},E_{2|0\mathrm{\_}1},E_{3|0},E_{3|0\mathrm{\_}1},E_{3|0\mathrm{\_}2},E_{3|0\mathrm{\_}1\mathrm{\_}2})$ . Each entry in the matrix represents the average transition rate from the substate in the row to the substate in the column, at time $t$ , in a population.

2.3. Static treatment regime and conditional SSLY and MALY

Let $\mathit{A}$ denote the treatment regime, where $\mathit{A}=(A_0,A_1,A_{2|0},A_{2|0\mathrm{\_}1},A_{3|0},A_{3|0\mathrm{\_}1},A_{3|0\mathrm{\_}2},A_{3|0\mathrm{\_}1\mathrm{\_}2})$ .

For a static regime, the hypothetical intervention $\mathit{A}$ is constant over time. Let $\mathit{P}(t|\mathit{A},\mathit{L})$ denote the conditional state occupation probabilities at time $t$ . For a five-state transition, $\mathit{P}(t|\mathit{A},\mathit{L})=c(p_0(t),p_1(t),p_{2|0}(t),p_{2|1\mathrm{\_}0}(t),p_{3|0}(t),p_{3|1\mathrm{\_}0}(t),p_{3|2\mathrm{\_}0}(t),p_{3|2\mathrm{\_}1\mathrm{\_}0}(t),p_4(t))$ , where each component represents the proportion of participants in at time $t$ in substates $S_0$ , $S_1$ , $S_{2|0}$ , $S_{2|0\mathrm{\_}1},S_{3|0}$ , $S_{3|0\mathrm{\_}1}$ , $S_{3|0\mathrm{\_}2}$ , $S_{3|0\mathrm{\_}1\mathrm{\_}2}$ , and $S_4$ , respectively. By using the Aalen–Johansen estimator, ¹¹ the conditional state occupation probability $\mathit{P}(t|\mathit{A},\mathit{L})$ can be estimated as $\mathit{P}(t|\mathit{A},\mathit{L})=\mathit{P}(t_0|\mathit{A},\mathit{L})\times \prod _{t=t_0}^t({\mathit{I}}_{\mathbf{9}}+\mathit{Q}(t|\mathit{A},\mathit{L}))$ , where ${\mathit{I}}_{\mathbf{9}}$ is an identity matrix, and $t_0$ is the start of follow-up.

The estimation of SSLY and MALY based on $\mathit{P}(t|\mathit{A},\mathit{L})$ has been described in detail in a previous publication. ⁷ In brief, SSLY can be estimated as the sum of the probabilities of being in each substate (except S₄) from the starting age until death. Let $SSLY(\mathit{A},\mathit{L})$ denote a $1\times k$ matrix representing life years spent in $k$ substates, where $k=1,\dots ,8$ , corresponding to the substates $S_0$ , $S_1$ , $S_{2|0}$ , $S_{2|0\mathrm{\_}1},S_{3|0}$ , $S_{3|0\mathrm{\_}1}$ , $S_{3|0\mathrm{\_}2}$ , and $S_{3|0\mathrm{\_}1\mathrm{\_}2}$ , respectively. The SSLY in substate $k$ is estimated as $SSLY(\mathit{A},\mathit{L})[1,k]=\sum _{t=t_0}^{t=t_e}\mathit{P}(t|\mathit{A},\mathit{L})[1,k]$ , where $t_0$ is the start of follow-up and $t_e$ is the end.

As these substates indicate multimorbidity status, a disability weight (DW) is assigned to each substate to account for the severity of multimorbidity.^12,13 DW quantify the severity of a disease, ranging from 0 (full health) to 1 (death), ¹² and the DW for each disease can be obtained from the Global Burden of Disease Study. ¹⁴ For disease substates with multimorbidity, DW values can be calculated using a multiplicative approach.^12,13 For example, for the substate S ${}_{2|0\mathrm{\_}1}$ , the DW is calculated as ${\mathrm{DW}}_{2|0\mathrm{\_}1}=1-(1-{\mathrm{DW}}_2)(1-{\mathrm{DW}}_1)(1-{\mathrm{DW}}_0)$ . The weighted SSLY in substate $k$ is then computed as ${\mathrm{DW}}_k\times SSLY(\mathit{A},\mathit{L})[k]$ .

As all substates are mutually exclusive, the conditional MALY can be calculated as the weighted average of SSLY across all $k$ substates. MALY takes the multimorbidity of each substate into consideration and estimates the adjusted life years in full health.

$$\mathrm{MALY}(\mathit{A},\mathit{L})=\sum _{k=1}^{k=8}{\mathrm{DW}}_k\times \mathrm{SSLY}(\mathit{A},\mathit{L})[1,k]=\sum _{k=1}^{k=8}{\mathrm{DW}}_k\times (\sum _{t=t_0}^{t=t_e}\mathit{P}(t|\mathit{A},\mathit{L})[1,k]).$$

2.4. Static treatment regime and counterfactual SSLY and MALY

To compare health benefits of static treatment regimes, we hypothetically assign each regime to all individuals in the target population. The regime yielding the highest mean MALY is the optimal treatment. Theoretically, the MALY used for comparison is the counterfactual MALY, defined as the average MALY in a population had everyone received that treatment regime. This is essentially to standardize the conditional MALY according to the distribution of confounders L in a population. The counterfactual MALY is estimated as

$${MALY}^{\mathit{A}}=\int MALY(\mathit{A},\mathit{L})f(\mathit{L})d\mathit{L},\text{where}f(\mathit{L})\text{is the joint density of confounders}\mathit{L}.$$ (1)

However, in practice, it is difficult to estimate $f(\mathit{L})$ when $\mathit{L}$ contains continuous variables. In fact, (1) is equivalent to an individual-level estimation, with the illustration and proof shown below.

For person $i$ , we only observe the person being in one substate $s_{m(i)}(t)$ with exposure $e_{m(i)}(t)$ at time $t$ , and the estimated conditional transition rate is ${\lambda }_{\mathit{m}\mathbf{(}\mathit{i}\mathbf{)}}(t|{\mathit{e}}_{\mathit{m}\mathbf{(}\mathit{i}\mathbf{)}}(t),{\mathit{L}}_{\mathit{i}})$ , which is estimated from Models 1–4. To construct the counterfactual transition rate matrix ${\mathit{Q}}_i^{{\mathit{A}}_{\mathit{i}}(t)}(t)$ under the hypothetical intervention, four assumptions of consistency and exchangeability are needed.

Assumption 1 (Consistency assumption for treatment) —

The observed transition rate for every treated participant equals the hypothetical transition rate had the person received treatment. For example, for participant $i$ in $s_{2|0}$ observed with $e_{2|0}$ , if the person received intervention $A_{2|0}=e_{2|0}$ , the counterfactual transition rate would be ${\lambda }_{23|0(i)}^{A_{2|0}=e_{2|0}}(t|{\mathit{L}}_{\mathit{i}})$ = ${\lambda }_{23|0(i)}(t|e_{2|0},{\mathit{L}}_{\mathit{i}})$ .

Assumption 2 (Exchangeability assumption for treatment) —

The untreated would have the same transition rate as the treated had the person received treatment. For example, for participant $i$ in $s_{2|0}$ observed with $E_{2|0}\ne e_{2|0}$ and participant $j$ in $s_{2|0}$ observed with $E_{2|0}=e_{2|0}$ , and ${\mathit{L}}_{\mathit{i}}={\mathit{L}}_{\mathit{j}}$ , the counterfactual transition rate would be ${\lambda }_{23|0(i)}^{A_{2|0}=e_{2|0}}(t|{\mathit{L}}_{\mathit{i}})$ = ${\lambda }_{23|0(j)}(t|e_{2|0},{\mathit{L}}_{\mathit{j}})$ , had person $i$ received intervention $A_{2|0}=e_{2|0}$ .

Assumption 3 (Consistency assumption for disease substate) —

The observed transition rate for every participant experienced a disease substate equals the hypothetical transition rate had the person experienced the disease substate. For example, for participant $i$ in $s_{2|0}$ , if the person received intervention $A_{2|0}=e_{2|0}$ , the counterfactual transition rate would be ${\lambda }_{23|0(i)}^{A_{2|0}=e_{2|0}}(t|{\mathit{L}}_{\mathit{i}})$ = ${\lambda }_{23|0(i)}(t|e_{2|0},{\mathit{L}}_{\mathit{i}})$ .

Assumption 4 (Exchangeability assumption for disease substate) —

Participants would have experienced the same transition rate as those who have an alternative disease history, had they experienced the alternative disease history. For example, if person $j$ experiences $S_{2|0}$ and person $i$ does not, and $L_{2|0(i)}=L_{2|0(j)}$ , the counterfactual transition rate would be ${\lambda }_{23|0(i)}^{A_{2|0}=e_{2|0}}\left(t\right|{\mathit{L}}_{\mathbf{2}|\mathbf{0}\mathbf{(}\mathit{i}\mathbf{)}})$ = ${\lambda }_{23|0(j)}\left(t\right|e_{2|0},{\mathit{L}}_{\mathbf{2}|\mathbf{0}\mathbf{(}\mathit{j}\mathbf{)}})$ , had the person $i$ experienced $S_{2|0}$ .

Using these assumptions, the conditional transition rate ${\lambda }_{\mathit{m}(\mathit{i})}(t|{\mathit{e}}_{\mathit{m}(\mathit{i})}(t),{\mathit{L}}_{\mathit{i}})$ estimated from Models 1–4 is used to construct the counterfactual transition rate matrix ${\mathit{Q}}_i^{\mathit{A}}(t)$ .

For person $i$ , the counterfactual state occupation probability is estimated as

$${\mathit{P}}_i^{\mathit{A}}(t|{\mathit{L}}_{\mathit{i}})={\mathit{P}}_{\mathit{i}}(t_0|{\mathit{L}}_{\mathit{i}})\times \prod _{t=t_0}^t({\mathit{I}}_{\mathbf{9}}+{\mathit{Q}}_i^{\mathit{A}}(t|{\mathit{L}}_{\mathit{i}}))$$

For person $i$ , the counterfactual SSLY in substate $k$ is estimated as

$${\mathrm{SSLY}}_i^{\mathit{A}}({\mathit{L}}_{\mathit{i}})[1,k]=\sum _{t=t_0}^{t=t_e}{\mathit{P}}_i^{\mathit{A}}(t|{\mathit{L}}_{\mathit{i}})[1,k],$$

For person $i$ , the counterfactual MALY is estimated as

$${\mathrm{MALY}}_i^{\mathit{A}}({\mathit{L}}_{\mathit{i}})=\sum _{k=1}^{k=K}{DW}_k\times {\mathrm{SSLY}}_i^{\mathit{A}}({\mathit{L}}_{\mathit{i}})[1,k].$$

The counterfactual SSLY in substate $k$ in the whole population is estimated as

$${\mathrm{SSLY}}^{\mathit{A}}[1,k]=E({\mathrm{SSLY}}_i^{\mathit{A}}({\mathit{L}}_{\mathit{i}})[1,k])=\frac{1}{N}\left(\sum _{i=1}^{i=N}{\mathrm{SSLY}}_i^{\mathit{A}}({\mathit{L}}_{\mathit{i}})[1,k]\right).$$

The counterfactual MALY in the whole population is estimated as

$${\mathrm{MALY}}^{\mathit{A}}=E({\mathrm{MALY}}_i^{\mathit{A}}({\mathit{L}}_{\mathit{i}}))=\frac{1}{N}(\sum _{i=1}^{i=N}{\mathrm{MALY}}_i^{\mathit{A}}({\mathit{L}}_{\mathit{i}})).$$ (2)

In fact, the counterfactual MALY and SSLY estimated by standardizing conditional measures are equivalent to those obtained by averaging the corresponding measures over the total population. Take MALY as an example, (1) and (2) are equivalent, with proof shown below.

Proof.

We divide L into $P$ stratum ${\mathit{l}}_{\mathbf{1}},{\mathit{l}}_{\mathbf{2}},\dots ,{\mathit{l}}_{\mathit{P}}$ , with $n_1,n_2,\dots ,n_P$ participants in each stratum.

$$\begin{array}{rl}& \frac{1}{N}\sum _{n=i}^{n=N}{\mathrm{MALY}}_i^{\mathit{A}}({\mathit{L}}_{\mathit{i}})\\ & =\frac{1}{N}\sum _{p=1}^{p=P}{\mathrm{MALY}}^{\mathit{A}}({\mathit{l}}_{\mathit{p}})n_P\\ & =\sum _{p=1}^{p=P}\mathrm{MALY}(\mathit{A}|{\mathit{l}}_{\mathit{p}})\frac{n_P}{N}\text{(Under the consistency and exchangeability Assumptions 1-{-}4)}\\ & =\sum _{p=1}^{p=P}\mathrm{MALY}(\mathit{A}|{\mathit{l}}_{\mathit{p}})f({\mathit{l}}_{\mathit{p}})\\ & =\int \mathrm{MALY}(\mathit{A}|\mathit{L})f(\mathit{L})d\mathit{L}(\text{when}{\mathit{l}}_{\mathit{p}}\text{is very small})\end{array}$$

2.5. Dynamic treatment regime and individual-level estimation of counterfactual MALY

The dynamic treatment regime ${\mathit{A}}_{i(t)}$ can vary over time by person, depending on the decision criteria, such as the occurrence of side effects of treatment. Let $W_{i(t)}$ denote the side effects at time $t$ , which may be affected by ${\mathit{A}}_{i(t)}$ and baseline confounders ${\mathit{L}}_i$ . We assume that the intervention ${\mathit{A}}_{i(t)}$ has only instantaneous effect on $W_{i(t)}$ (i.e. the effect is only at time $t$ and does not persist beyond that time point).

Step 1. For person $i$ starting from $t_0$ , assign treatment ${\mathit{A}}_{i(t0)}$ based on their initial state occupation probability ${\mathit{P}}_{\mathit{i}}(t_0|{\mathit{L}}_{\mathit{i}}$ ).

Step 2. Use the assigned treatment ${\mathit{A}}_{i(t0)}$ to fit Models 1–4 and construct the counterfactual transition rate matrix ${\mathit{Q}}_i^{{\mathit{A}}_{i(t0)}}(t_0|{\mathit{L}}_{\mathit{i}})$ .

Step 3. Estimate the state occupation probabilities at time $t_1$ as

$${\mathit{P}}_i^{{\mathit{A}}_{i(t1)}}(t|{\mathit{Q}}_i^{{\mathit{A}}_{i(t0)}})={\mathit{P}}_{\mathit{i}}(t_0|{\mathit{L}}_{\mathit{i}})({\mathit{I}}_{\mathbf{9}}+{\mathit{Q}}_i^{{\mathit{A}}_{i(t0)}}(t_1|{\mathit{L}}_{\mathit{i}})).$$

Step 4. Estimate the adverse effect at $t_1$ , $W_{i(t1)}$ , as $E(W_{i(t1)}|{\mathit{A}}_{i(t0)},{\mathit{L}}_{\mathit{i}})$ by fitting models regressing $W_{i(t1)}$ on ${\mathit{A}}_{i(t0)}$ and ${\mathit{L}}_{\mathit{i}}$ .

Step 5. Based on the updated state occupation probability, ${\mathit{P}}_i^{{\mathit{A}}_{i(t1)}}$ , and side effect, $W_{i(t1)}$ , apply the decision criteria to define the next treatment ${\mathit{A}}_{i(t1)}|{\mathit{P}}_i^{{\mathit{A}}_{i(t1)}}$ , $W_{i(t1)}$ . Then refit Models 1–4 to construct ${\mathit{Q}}_i^{{\mathit{A}}_{i(t1)}}(t_1|{\mathit{L}}_{\mathit{i}})$ and estimate ${\mathit{P}}_i^{{\mathit{A}}_{i(t2)}}(t|{\mathit{Q}}_i^{{\mathit{A}}_{i(t1)}})$ . Then estimate $W_{i(t2)}$ and determine treatment ${\mathit{A}}_{i(t2)}$ — ${\mathit{P}}_i^{{\mathit{A}}_{i(t2)}}$ . Repeat this process iteratively at each time t until the end of follow-up $t_e$ .

For participant $i$ , the counterfactual SSLY in substate $k$ under the dynamic treatment regime $\overline{{\mathit{A}}_{\mathit{i}({\mathit{t}}_{\mathbf{0}}-{\mathit{t}}_{\mathit{e}})}}$ is estimated as

$${\mathrm{SSLY}}_i^{\overline{{\mathit{A}}_{\mathit{i}({\mathit{t}}_{\mathbf{0}}-{\mathit{t}}_{\mathit{e}})}}}({\mathit{L}}_{\mathit{i}})[1,k]=\sum _{t=t_0}^{t=t_e}{\mathit{P}}_i^{{\mathit{A}}_{i(t)}}(t|{\mathit{Q}}_i^{{\mathit{A}}_{i(t-1)}})[1,k]$$

For participant $i$ , the counterfactual MALY is estimated as

$${\mathrm{MALY}}_i^{\overline{{\mathit{A}}_{\mathit{i}({\mathit{t}}_{\mathbf{0}}-{\mathit{t}}_{\mathit{e}})}}}({\mathit{L}}_{\mathit{i}})=\sum _{k=1}^{k=K}{\mathrm{DW}}_k\times ({SSLY}_i^{\overline{{\mathit{A}}_{\mathit{i}({\mathit{t}}_{\mathbf{0}}-{\mathit{t}}_{\mathit{e}})}}}({\mathit{L}}_{\mathit{i}})[1,k])$$

The counterfactual SSLY in the whole population is estimated as

$${\mathrm{SSLY}}^{\overline{{\mathit{A}}_{({\mathit{t}}_{\mathbf{0}}-{\mathit{t}}_{\mathit{e}})}}}[1,k]=E({\mathrm{SSLY}}_i^{\overline{{\mathit{A}}_{\mathit{i}({\mathit{t}}_{\mathbf{0}}-{\mathit{t}}_{\mathit{e}})}}}({\mathit{L}}_{\mathit{i}})[1,k])=\frac{1}{N}(\sum _{i=1}^{i=N}{\mathrm{SSLY}}_i^{\overline{{\mathit{A}}_{\mathit{i}({\mathit{t}}_{\mathbf{0}}-{\mathit{t}}_{\mathit{e}})}}}({\mathit{L}}_{\mathit{i}})[1,k]).$$

The counterfactual MALY in the whole population is estimated as

$${\mathrm{MALY}}^{\overline{{\mathit{A}}_{({\mathit{t}}_{\mathbf{0}}-{\mathit{t}}_{\mathit{e}})}}}=E({\mathrm{MALY}}_i^{\overline{{\mathit{A}}_{\mathit{i}({\mathit{t}}_{\mathbf{0}}-{\mathit{t}}_{\mathit{e}})}}}({\mathit{L}}_{\mathit{i}}))=\frac{1}{N}\left(\sum _{i=1}^{i=N}{\mathrm{MALY}}_i^{\overline{{\mathit{A}}_{\mathit{i}({\mathit{t}}_{\mathbf{0}}-{\mathit{t}}_{\mathit{e}})}}}({\mathit{L}}_{\mathit{i}})\right).$$

2.6. Compare counterfactual MALY across treatment regimes

The difference in MALY between two treatment regimes can be estimated as

$${\mathrm{MALY}}^{\overline{{A_1}_{({\mathit{t}}_{\mathbf{0}}-{\mathit{t}}_{\mathit{e}})}}}-{\mathrm{MALY}}^{\overline{{A_2}_{({\mathit{t}}_{\mathbf{0}}-{\mathit{t}}_{\mathit{e}})}}}=E({\mathrm{MALY}}_i^{\overline{{\mathit{A}}_{\mathbf{1}\mathit{i}({\mathit{t}}_{\mathbf{0}}-{\mathit{t}}_{\mathit{e}})}}}({\mathit{L}}_{\mathit{i}})-{\mathrm{MALY}}_i^{\overline{{\mathit{A}}_{\mathbf{2}\mathit{i}({\mathit{t}}_{\mathbf{0}}-{\mathit{t}}_{\mathit{e}})}}}({\mathit{L}}_{\mathit{i}}))$$

The 95% confidence interval (CI) can be estimated using bootstrap. We randomly draw samples with replacement to create a new sample that are the same size as the original population. The counterfactual MALY for each treatment regime and the difference in MALY between two regimes can be estimated from each bootstrap sample. Statistically, the resulting parameter estimates across all bootstrap samples represent the empirical distribution and can be summarized with median values and a 95% CI defined by 2.5th to 97.5th percentiles of the empirical distribution.

3. Application

We applied our method to identify optimal body mass index (BMI) range across the course of heart disease in the Atherosclerosis Risk in Communities (ARIC) study. The AHA has recommended maintaining a $\mathrm{BMI}<25$  kg/m ${}^2$ across life course for prevention of CVD. ¹⁵ However, the associations of BMI with CVD-related endpoints vary by disease state. Studies have shown that overweight (BMI 25–30 kg/m ${}^2$ ) and obesity ( $\mathrm{BMI}\ge 30$  kg/m ${}^2$ ) are associated with higher risks of cardiometabolic disease compared to normal weight.^16,17 However, among older people or participants with CVD, elevated BMI is associated with lower risk of mortality compared to those of normal weight.^18–21 Thus, it is imperative to identify disease-state specific BMI range that generates the most health benefits across the course of heart disease.

3.1. Study population

We used data from the ARIC study.^22,23 The enrolled participants underwent a phone interview and clinic visit at baseline in 1987 and were followed up by telephone calls and re-examinations until 2019. Participants were contacted periodically by phone and interviewed about interim hospital admissions, cardiovascular outpatient diagnoses, and deaths. We obtained ARIC data through Biologic Specimen and Data Repository Information Coordinating Center (BioLINCC), an open repository managed by the National Heart, Lung, and Blood Institute (NHLBI). The study was deemed exempt by the Institutional Review Board (IRB) of the University of North Carolina (UNC) at Chapel Hill.

3.2. Assessment of exposure, outcome, and covariates

1. Exposure. BMI was measured on weight scales by medical professionals. Normal weight is defined as $\mathrm{BMI}<25$  kg/m ${}^2$ , overweight is defined as BMI between 25–30 kg/m ${}^2$ , and obesity is defined as $\mathrm{BMI}\ge 30$  kg/m ${}^2$ .

2. Variables used to define states of heart disease. Low density lipoprotein cholesterol (LDL-c), high density lipoprotein cholesterol (HDL-c), triglycerides, and blood glucose were assessed in laboratories at each visit. Blood pressure was measured using electronic monitor by medical professionals at each visit. Participants who reported CVD-related events were asked to provide medical records that were reviewed by physicians. ²³ Deaths were identified from systematic searches of vital records in states and the National Death Index, supplemented by reports from family members and postal authorities. ²⁴

3. Covariates. Age, sex, race, education level, smoking status, and family history of heart disease were measured using questionnaires at baseline.

3.3. Definition of the course of heart disease

We modeled the course of heart disease in five states: Healthy, at metabolic risk, coronary heart disease (CHD), heart failure, and mortality. The at-metabolic-risk state was defined as the development of hypertension, hyperlipidemia, or diabetes. Hypertension was defined as blood pressure $\ge$ 140/90 mmHg or a history of hypertension or use of blood pressure medications. ²⁵ Hyperlipidemia included primary hypertriglyceridemia ( $\ge$ 175 mg/dL) or primary hypercholesterolemia (LDL-c 160–189 mg/dL, and/or non-HDL-c 190–219 mg/dL). ²⁶ Diabetes was defined as fasting glucose $\ge$ 7.7 mmol/L. ²⁷

3.4. Regimes of BMI range

We compared four BMI regimes as below. Regimes 1–3 were static regimes which predefined BMI range before intervention. Regime 4 was a dynamic regime which defined BMI range by a person’s characteristics that can change over time.

• Regime 1: Normal weight ( $\mathrm{BMI}<25$  kg/m ${}^2$ ) in all disease states.

• Regime 2: Normal weight ( $\mathrm{BMI}<25$  kg/m ${}^2$ ) in healthy, at-metabolic-risk, and CHD states, and overweight (BMI 25–30 kg/m ${}^2$ ) if the person develops heart failure.

• Regime 3: Normal weight ( $\mathrm{BMI}<25$  kg/m ${}^2$ ) in healthy and at-metabolic-risk states, and overweight (BMI 25–30 kg/m ${}^2$ ) if the person develops CHD or heart failure.

• Regime 4: Normal weight ( $\mathrm{BMI}<25$  kg/m ${}^2$ ) at the start of intervention, and change to overweight (BMI 25–30 kg/m ${}^2$ ) if age >65y.

By coding normal weight, overweight, and obesity as 0, 1, and 2, respectively, the four regimes can be rewritten as follows.

• Regime 1: ${\overline{A}}_{r1}=c(A_0=0,A_1=0,A_{2|0}=0,A_{2|1\mathrm{\_}0}=0,A_{3|0}=0,A_{3|1\mathrm{\_}0}=0,A_{3|2\mathrm{\_}0}=0,A_{3|2\mathrm{\_}1\mathrm{\_}0}=0)$

• Regime 2: ${\overline{A}}_{r2}=c(A_0=0,A_1=0,A_{2|0}=0,A_{2|1\mathrm{\_}0}=0,A_{3|0}=1,A_{3|1\mathrm{\_}0}=1,A_{3|2\mathrm{\_}0}=1,A_{3|2\mathrm{\_}1\mathrm{\_}0}=1)$

• Regime 3: ${\overline{A}}_{r3}=c(A_0=0,A_1=0,A_{2|0}=1,A_{2|1\mathrm{\_}0}=1,A_{3|0}=1,A_{3|1\mathrm{\_}0}=1,A_{3|2\mathrm{\_}0}=1,A_{3|2\mathrm{\_}1\mathrm{\_}0}=1)$

• Regime 4: For ${\overline{A}}_{r4}$ , the intervention was 0 at each time interval in each of the substate if age $\le$ 65 years, and changed to 1 if age >65 years.

3.5. Statistical analysis

We applied our method to examine the effects of BMI regimes on the course of heart disease. In brief, the transition rates between substates were estimated using cause-specific Cox models, adjusting for age (continuous), sex (women, men), race (White, Black), education (less than high school, high school/vocational school, more than high school [college or professional school]), family history of heart disease (yes, no), and baseline smoking status (never, past, current). For each treatment regime, the age-specific transition rates and state occupation probabilities were estimated, which were used to synthesize MALY and SSLY. Specifically, for the estimation of MALY, we used DWs derived from the Global Burden of Disease Study 2019. ¹⁴ The DWs were 0.049, 0.072, and 0.074 for at-metabolic-risk, CHD, and heart failure, respectively. For states with multimorbidity, a multiplicative approach was used to assign disability weights. For example, the DW for CHD transitioning from healthy and at metabolic risk states was calculated as: ${\mathrm{DW}}_{2|0\mathrm{\_}1}=1-(1-0.049)(1-0.072)$ . The most beneficial BMI regime was identified as the one associated with the highest counterfactual MALY. More details of the statistical analysis of the application study can be found in the supplemental materials.

3.6. Results

Figure 2 shows the associations of BMI with the rates of transitioning to subsequent states. Starting from the healthy state, overweight and obesity were associated with higher rate ratios of transitioning to at-metabolic-risk states (1.21 [95% CI: 1.10, 1.32] for overweight; 1.48 [95% CI: 1.32, 1.65] for obesity). The positive associations of overweight with transitions to the next states were attenuated to null starting from the at-metabolic-risk or CHD state. The association became inverse starting from the heart failure state, where overweight was associated with a lower rate ratio of transitioning to death (0.82 [95% CI: 0.73, 0.92]).

Figure 2.

Hazard ratios (95% CIs) for state transitions by obesity status across disease substates in the Atherosclerosis Risk in Communities (ARIC) study, 1987–2019 ( $n=15027$ ). Normal weight: body mass index (BMI) < 25 kg/m ${}^2$ ; Overweight: 25 kg/m ${}^2$   $\le$ BMI < 30 kg/m ${}^2$ ; Obesity: BMI $\ge$ 30 kg/m ${}^2$ . Cause-specific Cox models adjusted for age (continuous), race (White, Black), sex (women, men), education (less than high school, high school/vocational school, more than high school [college or professional school]), family history of cardiovascular disease (yes, no), and baseline smoking status (never, past, current).

The cardiovascular benefits of the four BMI regimes are presented in Table 1. For example, the counterfactual MALY was 35.08 years (95% CI: 22.63, 47.11) under the hypothetical Regime 1, meaning that the mean healthy life expectancy accounting for multimorbidity would be 35.08 years had everyone in the population maintained a normal weight across all disease states. Compared to Regime 1, the MALY was 0.38 years (95% CI: 0.07, 0.90) higher under Regime 3, and 1.80 years higher (95% CI: 0.62, 2.78) under Regime 4. These results suggest that starting with a normal weight and changing to overweight after age 65y (Regime 4) yields the most cardiovascular benefits.

Table 1.

Difference in multimorbidity-adjusted life years (MALY) across body mass index (BMI) regimes in the Atherosclerosis Risk in Communities (ARIC) study, 1987–2019 ( $n=15027$ ).

	MALY,life year (95% CI)	Difference in MALY,life year (95% CI)
Regime 1	35.08 (22.63, 47.11)	Reference
Regime 2	35.52 (23.02, 47.39)	0.37 (0.10, 0.86)
Regime 3	35.54 (23.04, 47.40)	0.38 (0.07, 0.90)
Regime 4	36.95 (24.28, 48.56)	1.80 (0.62, 2.78)

To elucidate the mechanisms by which the BMI regimes affect disease progression, we estimated SSLY under each treatment regime. Compared to Regime 1, the SSLY of the optimal regime (Regime 4) was higher in all substates except the healthy state, with the largest gain observed in the at-metabolic-risk state (Table 2). This finding suggests that maintaining a BMI between 25 and 30 kg/m ${}^2$ may elongates life expectancy among older adults, even for those who start from the at-metabolic-risk state.

Table 2.

Estimated substate-specific life years (SSLY) by body mass index (BMI) regimes in the Atherosclerosis Risk in Communities (ARIC) study, 1987–2019 ( $n=\mathrm{15,027}$ ).

Disease substates	Life year (95% CI)
	Regime 1	Regime 2	Regime 3	Regime 4
Healthy (S0)	27.73 (17.60, 35.48)	27.73 (17.60, 35.48)	27.73 (17.60, 35.48)	27.67 (17.69, 35.38)
At metabolic risk (S1)	28.22 (3.20, 44.27)	28.22 (3.20, 44.27)	28.22 (3.20, 44.27)	29.16 (3.69, 45.44)
CHD without metabolic risk factors (S $2|0$ )	0.33 (0.05, 1.83)	0.33 (0.05, 1.84)	0.33 (0.05, 1.86)	0.36 (0.05, 1.97)
CHD with metabolic risk factors (S $2|0\mathrm{\_}1$ )	0.98 (0.13, 2.61)	0.98 (0.13, 2.63)	0.99 (0.13, 2.67)	1.15 (0.17, 3.04)
Heart failure with no metabolic risk factors or CHD (S $3|0$ )	1.40 (0.47, 3.33)	1.65 (0.54, 3.84)	1.65 (0.54, 3.84)	1.82 (0.64, 4.22)
Heart failure only with metabolic risk factors (S $3|0\mathrm{\_}1$ )	1.94 (0.29, 3.52)	2.29 (0.35, 4.16)	2.30 (0.35, 4.16)	2.65 (0.45, 4.67)
Heart failure only with CHD (S $3|0\mathrm{\_}2$ )	0.08 (0.01, 0.48)	0.09 (0.01, 0.56)	0.09 (0.01, 0.56)	0.10 (0.01, 0.64)
Heart failure with metabolic risk factors and CHD (S $3|0\mathrm{\_}1\mathrm{\_}2$ )	0.20 (0.02, 0.65)	0.23 (0.02, 0.79)	0.23 (0.02, 0.79)	0.27 (0.03, 0.93)

3.7. Discussion

The proposed method is developed based on a discrete-time split-state framework for multi-state modeling. ⁶ In brief, this framework splits disease states into substates by conditioning on past history. The newly created substates have two unique features. First, they are “memoryless,” meaning that they exhibit the Markov property, regardless of whether the original process is Markovian. Second, they are “memorable, as being in a substate indicates multimorbidity status. The features of being “memoryless” and “memorable” enable the synthesis of the substate- and time-specific transition rates into summary measures (i.e. SSLY and MALY) to characterize the entire disease course. In this article, building on this framework, we propose a multi-state causal approach to identify the optimal treatment regime that generates the most benefits, as indicated by the maximum MALY. Our method also provides insights into the underlying mechanisms of treatment regime on disease progression, as indicated by SSLY.

From a methodology perspective, our method is a causal inference approach that evaluates the counterfactual effect of a treatment regime on the disease course, assuming all participants receive the treatment. Existing methods that examine the effects of treatment regimes on disease risk include the parametric g-formula and the MSM.^1,2 Both methods control for time-varying confounding by conditioning on past confounders under treatment and estimate conditional survival probabilities using the Kaplan–Meier estimator. They derive counterfactual effects in the entire population through standardization (g-formula) or inverse probability weighting (MSM). However, these methods focus on the risk of a single endpoint and ignore the entire disease course. This limitation may result in the misidentification of optimal treatment regimes, especially when treatments exert long-term effects across multiple disease states. Our method incorporates the entire disease course and provides a more comprehensive evaluation of treatment effectiveness. It can be considered as an extension of the g-formula to the multi-state setting. In addition to causal inference methods, Sequential Multiple Assignment Randomized Trials (SMARTs) have been used to evaluate the causal effect of treatment regimes on an outcome. ²⁸ In SMART designs, participants are randomized at multiple decision points to construct and compare dynamic treatment regimes. However, SMART trials are often impractical for chronic diseases, given the extreme long follow-up needed for multiple disease states to develop. By estimating the counterfactual effect under hypothetical interventions, our method emulates a sequential trial that would otherwise be infeasible to implement in real-world settings.

From a public health perspective, our method has the potential to advance the precision prevention of chronic diseases by identifying the optimal treatment for each person at each (sub)state that maximizes the precise benefits over the disease course. First, our method enables the identification of (sub)state-specific optimal treatments. Traditional chronic disease prevention is generally divided into primary prevention—aimed at reducing disease incidence—and secondary prevention—focused on preventing complications once a disease has developed. ²⁹ However, due to the complex and heterogeneous progression of chronic diseases, prevention strategies may need to be further tailored to specific disease states. Our method identifies the optimal substate-specific strategies for precision prevention. Second, our method allows for dynamic, individualized interventions. Conventional public health prevention strategies are typically applied to the general population, with limited consideration for individual variability. In this article, our dynamic treatment regime accounts for individual heterogeneity, enabling treatment decisions to be made based on a person’s current disease state and response over time. This facilitates the development of personalized prevention strategies that are most beneficial across the entire disease course. ² Third, because MALY incorporates multimorbidity and quality of life over the disease course, our method provides a more precise estimation of prevention benefits. This is particularly valuable for cost-effectiveness analyses and public health decision-making. Moreover, the estimated SSLY offer insights into the mechanisms through which treatment impacts disease progression. As suggested in the application section, starting with a normal weight and changing to overweight after age 65y (Regime 4) achieved significantly higher MALY compared to other regimes. This was due to prolonging time spent in all disease substates, particularly the at-metabolic-risk substate.

We acknowledge that our method needs the assumption that the effects of treatments on transition rates are instantaneous. For treatments that had cumulative effects, to estimate the cumulative treatment effect at time t, we would need to trace the transitions between substates from time 0 to t to calculate the time spent in each substate from time 0 to t, as the treatments may vary by disease substate. The disease substates created by our framework record multimorbidity status at each time point and are used to estimate MALY, however, they do not trace the transitions between substates from time 0 to t. Although accommodating cumulative effects is beyond the scope of this article, our proposed method can still be applied under two specific scenarios involving cumulative treatments: a static treatment regime with same treatment applied to all disease states, and a two-state transition that is equivalent to a conventional single time-to-event outcome.

Another assumption we assume in this article is that confounders are constant over follow-up. Time-varying confounders can be influenced by treatment and disease state. Under the assumption of instantaneous effect of treatment on confounders, time-varying confounders can be modeled by tracing the transition between time $t-1$ and $t$ . It is of high interest to extend our causal framework to allow for time-varying confounding for future research.

In summary, we developed a multi-state causal framework to estimate the health benefits of both static and dynamic treatment regimes over the disease course. This method enables the identification of the optimal treatment regime that maximizes health benefits, as measured by MALY, and reveals the mechanisms through which treatment regimes influence disease progression, as captured by SSLY. Our method has the potential to advance the precision prevention of chronic diseases.