Estimating optimal dynamic treatment strategies under resource constraints using dynamic marginal structural models

Abstract

Methods for estimating optimal treatment strategies typically assume unlimited access to resources. However, when a health system has resource constraints, such as limited funds, access to medication, or monitoring capabilities, medical decisions must account for competition between individuals in resource usage. The problem of incorporating resource constraints into optimal treatment strategies has been solved for point exposures (1), that is, treatment strategies entailing a decision at just one time point. However, attempts to directly generalize the point exposure solution to dynamic time-varying treatment strategies run into complications. We sidestep these complications by targeting the optimal strategy within a clinically defined subclass. Our approach is to employ dynamic marginal structural models to estimate (counterfactual) resource usage under the class of candidate treatment strategies and solve a constrained optimization problem to choose the optimal strategy for which expected resource usage is within acceptable limits. We apply this method to determine the optimal dynamic monitoring strategy for people living with HIV when resource limits on monitoring exist using observational data from the HIV-CAUSAL Collaboration.

1 |. INTRODUCTION

In resource limited settings, where health system constraints prevent immediate initiation of treatment in all individuals, the optimal treatment strategy is the strategy which, if implemented by all doctors, would lead to the best population health outcomes while ‘respecting’ the system’s resource constraints (in a sense we will make rigorous below). In many care settings, physicians repeatedly assess patients and make treatment decisions based on patient history at each assessment. In such settings, ‘dynamic treatment strategies’ mapping a patient’s treatment and covariate history to a treatment decision at each time point are of interest. (Here ‘treatment’ refers to any intervention, including monitoring through lab tests to inform future decisions.) In this article, we propose a method to identify optimal dynamic treatment strategies (from a subclass of all possible strategies) respecting population-level resource constraints.

Luedtke and van der Laan¹ considered the problem of estimating optimal resource constrained treatment strategies in the case of a point treatment—that is, when there is only one time point at which a treatment decision is made (see also Wang et al² and Huang et al³). They considered resource constraints that place an upper limit κ on the expected proportion of treated patients, and defined the set of strategies which respected resource constraint κ as all strategies for which the expected proportion of treated patients in the population under that strategy is less than κ. The optimal point treatment strategy was then identified as the optimal strategy among those respecting the system level constraint. However, optimal point treatment strategies are often of limited utility in clinical decision-making, especially in the context of chronic disease or long-term therapy. Such strategies cannot recommend, for example, “Come back next month, and if the problem has progressed then begin treatment”.

Here, we consider the optimal resource-constrained dynamic strategy and present a method for estimating the optimal resource-constrained dynamic strategy from a parameterized subclass of all strategies. For example, suppose we restrict our attention to the class of monitoring strategies $\{g_x:g_x(L_t)=1\{L_t<x\}\}$ (where 1{} denotes the indicator function) that monitors at time t if and only if the covariate L_t is less than x. Such a class of strategies might be approximately appropriate, for example, for the decision of how often to monitor people living with HIV, where L_t represents CD4 count at time t.

In the absence of resource constraints, Orellana et al⁴ and Robins et al⁵ describe how to estimate the optimal strategy from a parameterized class of strategies using a dynamic Marginal Structural Model. A dynamic MSM models the expected counterfactual outcomes under strategies parameterized by x as a function b(x; β) of x. With an estimate $\widehat{\beta }$ of the dynamic MSM parameter β, estimating the optimal strategy simply reduces to finding x^opt maximizing $b(x;\widehat{\beta })$ (assuming larger values of b(x; β) are preferable). To accommodate resource constraints, we propose a fairly straightforward extension of this procedure that entails fitting two dynamic MSMs-one estimating the expected counterfactual clinical outcome b(x; β) and one estimating expected counterfactual treatment utilization h(x; θ). The optimal resource-constrained dynamic strategy is then $x_{rc}^{opt}$ maximizing b(x; β) subject to h(x; θ) < κ.

Several methods have been proposed to find dynamic treatment strategies that respect constraints using observational data.^6–10 Sometimes the constraint is a maximum acceptable level for an adverse outcome rather than a resource constraint, for example, prevalence of a side effect, but the resulting methodology is similar. These methods employ the parametric g-formula¹¹ to estimate effects of candidate treatment strategies and then limit attention to those that satisfy constraints. The well-known advantages and drawbacks of the g-formula compared with dynamic MSMs in the unconstrained setting extend to the resource constrained setting. A commonly used algorithm to estimate the parametric g-formula requires correctly specified models for the conditional distribution of the outcome and each of the time-varying covariates over time, whereas to fit a dynamic MSM by inverse probability weighting requires correct specification of a model for treatment and a structural model for the counterfactual outcome. When all models are correctly specified, the g-formula is statistically more precise and can be used to explore a collection of strategies that may not fit neatly into a class indexed by a low dimensional parameter. However, the parametric g-formula requires correct specification of more models, is computationally intensive, and does not directly estimate an optimal strategy within a class (though this latter point will often be of minor practical significance). Thus, the dynamic MSM based approach we propose provides a useful complement to methods previously employed for this problem. We also note that there exists a somewhat related literature in adaptive trial planning.^12–16 In this setting, the constrained resource is sample size and the intervention to be optimized is the rate at which patients are assigned to treatment arms in order to obtain a maximally precise effect estimate.

We apply this approach to estimate the optimal resource-constrained dynamic strategy for CD4 cell count and HIV-RNA monitoring in people living with HIV who have achieved viral suppression. CD4 cell count and HIV-RNA tests are used to monitor an individual’s response to ART. More frequent monitoring has been shown to be associated with a lower risk of virologic failure (HIV RNA levels > 200 copies/mL) at two years after viral suppression.¹⁷ Guidelines recommend dynamic monitoring strategies in which virologically suppressed individuals on ART may be monitored less frequently once their CD4 cell count crosses above a certain threshold. However, the optimal point at which to decrease monitoring frequency is unclear.

In a health system with resource constraints, such as limited funds for monitoring or stock-outs of CD4 cell count and HIV-RNA tests, the CD4 cell count at which monitoring may be decreased must be chosen from a subset of strategies where the average number of tests does not exceed the available resources. Depending on the setting, the cost of a single HIV-RNA test may range from 18 to 36 US dollars,¹⁸ whereas the cost of a single CD4 cell count test may range from 15 to 24 US dollars.^19,20 While limited funds may be more likely to impact low- and middle-income countries than high-income countries, stock-outs can occur throughout the world, and are often unexpected. In a survey of national AIDS programs in 12 countries in Latin America conducted in 2011, 50% of countries reported stock-outs of HIV laboratory supplies, including no rapid tests in 41% of cases, no CD4 cell count tests in 35%, and no HIV-RNA tests in 18%. This resulted in the discontinuation of services in 88% of cases.²¹ In a more recent study of gaps in the HIV treatment cascade in 11 West African countries conducted in 2018, ART stock-outs were recorded during 23% of health facility visits and stock-outs of HIV-RNA tests were recorded in 17% of visits.^22,23 In 2020, 73 countries reported risk of ARV disruptions in response to the COVID-19 pandemic (including 13% of European countries), and 24 of these countries had critically low stocks of ARVs. Stock-outs were not only reported for ARVs—38 countries reported facing disruption in HIV testing and 23 countries reported facing disruption in HIV-RNA monitoring.²⁴

2 |. NOTATION

Let:

• t ∈ {0, …, K} denote time, assumed discrete, with K the end of the study;

• N_t be a variable indicating the whether an individual is monitored at time t;

• Y denote health outcome we aim to optimize;

• L_t denote covariates at time t that may influence monitoring decisions;

• D denote the total number of monitoring tests received by an individual;

• ${\overline{X}}_t$ denote X₀, …, X_t and X_t denote X_t, …, X_K for arbitrary time varying variable X.

We assume that we observe independent and identically distributed (iid) realizations of the random vector O = (L₀, …, L_K, N₀, …, N_K, Y). We use capital letters to denote random variables and corresponding lower case letters to denote specific values that random variables might take.

A treatment strategy is a rule or function that determines the value to which N_t will be set for a given observed history, that is, a function $g:({\overline{L}}_t,{\overline{N}}_{t-1})\mapsto N_t$. A strategy is said to be static if its recommendation for the present does not depend on past covariate and treatment values, and can therefore be specified from baseline. An example of a static strategy would be ‘monitor every 3 months’. A strategy is said to be dynamic if it does depend on past covariate and treatment values. An example of a dynamic strategy would be ‘monitor if time since last monitoring ≥ 6 months or if last observed viral load > 100 copies/mL and time since last monitoring ≥ 2 months’. Most realistic and clinically relevant strategies are dynamic. Compared with the above examples of deterministic strategies, random strategies output a distribution over the treatment space instead of a single treatment value,¹⁷ which capture deviations from rigid strategies that occur in realistic clinical practice. Random strategies can also capture lottery treatment assignments, which might be of interest particularly in resource constrained settings.

We denote arbitrary strategies by g and we adopt the counterfactual framework of Robins¹¹ in which corresponding to each possible strategy g are counterfactual random variables Y(g), L_t(g), and D(g) that would have been observed had strategy g been followed, possibly contrary to fact. Implicit in the notation for counterfactuals (eg, Y(g)) is the assumption that the treatment strategy followed by one patient does not influence any other patient. This implicit assumption is called the ‘No Interference Assumption’ by Rubin.²⁵ Note that in defining our resource constraint, we are optimizing based on the (counterfactual) average number of treatment or monitoring events per individual over a defined time period, but assuming no competition between individuals to access care under the optimal resource-constrained dynamic strategy. We also make the additional assumptions:²⁶

$$SequentialPositivity:\text{If}f({\overline{l}}_t,{\overline{n}}_{t-1})>0,f(N_t=n|{\overline{L}}_t={\overline{l}}_t,{\overline{N}}_{t-1}={\overline{n}}_{t-1})>0\forall n,t$$ (1)

$$Consistency:\text{For any strategy}g,\text{if for a given individual}{N}_t=g({\overline{L}}_t,{\overline{N}}_{t-1})\text{for each}t,\text{then}Y=Y(g)\text{and}{\overline{L}}_K={\overline{L}}_K(g)\text{for that individual}$$ (2)

$$SequentialExchangeability:Y(g)\perp N_t|{\overline{L}}_{t-1}={\overline{l}}_t,{\overline{N}}_{t-1}={\overline{n}}_{t-1}\forall t,g,\overline{n},\overline{l}$$ (3)

We define resource constraints as caps on the expected number of doses or monitoring events per patient over a defined time period. We say that strategy g respects resource constraint κ if

$$\text{E}[D(g)]<\kappa$$ (4)

We consider parameterized classes of strategies {g_x} and seek to estimate

$$x_{\kappa }^{opt}\equiv {argmax}_x\text{E}[Y(g_x)]\text{subject to E}[D(g_x)]<\kappa$$ (5)

3 |. REVIEW OF DYNAMIC MARGINAL STRUCTURAL MODELS

A dynamic MSM is a model for expected counterfactual outcomes under a class of strategies {g_x} parameterized by x as a function of x, that is,

$$\text{E}[Y(g_x)]=b(x;\beta ).$$ (6)

To estimate β, first note that each individual might follow multiple strategies from the class {g_x}. Let ∧_i denote the number of strategies that individual i follows. Generate an artificial dataset with ∧_i contributions from each individual: (Y_i, x_i1), …, (Y_i, x_i∧i). Using the artificial dataset, we can fit by weighted least squares the regression model

$$\text{E}[Y|x]=b(x;\gamma )$$

with weights

$$W(x)\equiv \prod _{k=0}^K\frac{f^{\ast }(x)}{f(N_k|{\overline{L}}_{k-1},{\overline{N}}_{k-1})}$$

to obtain $\widehat{\gamma }$, where $f(\cdot |\cdot )$ denotes conditional density and f*(x) is an arbitrary function of x. (When treatment strategies in {g_x} are stochastic, the numerator of W(x) is set to $f_x^{\ast }(N_k|{\overline{L}}_{k-1},{\overline{N}}_{k-1})$, the conditional probability of receiving observed treatment at time k under g_x conditional on patient history, as described in Cain et al¹⁷). When treatment or monitoring probabilities are unknown, as they are in our application, a consistent estimator $\widehat{f}(N_k|{\overline{L}}_{k-1},{\overline{N}}_{k-1})$ of $f(N_k|{\overline{L}}_{k-1},{\overline{N}}_{k-1})$ can be plugged into W(x). Under sequential exchangeability (3) and consistency (2), Orellana et al⁴ show that the weighted regression parameter estimate $\widehat{\gamma }$ approaches the causal estimand β in the limit.

4 |. ESTIMATING OPTIMAL TREATMENT STRATEGIES WITH RESOURCE CONSTRAINTS

To estimate the optimal resource-constrained dynamic strategy, we simply estimate the parameters of two dynamic MSMs:

$$\text{E}[Y(g_x)]=b(x;\beta )$$ (7)

$$\text{E}[D(g_x)]=h(x;\theta )$$ (8)

and then estimate the x indexing the optimal strategy as

$$\begin{gathered} {\widehat{x}}_{\kappa }^{opt}\equiv argmi{n}_{x}b(x;\widehat{\beta }) \\ \text{subject to}h(x;\widehat{\theta })\le \kappa \end{gathered}$$ (9)

(9) can be solved using software for non-linear optimization with non-linear inequality constraints such as the NLopt²⁷ implementation of the method-of-moving-asymptotes.²⁸ Optimization algorithms are not generally theoretically guaranteed to find global minima when constraints are nonlinear, but in most practical cases (see our application) a simple grid search would suffice. Standard errors for $\widehat{\beta }$, $\widehat{\theta }$, and certain derived quantities such as E[Y(g_x)] and E[D(g_x)] for any x can be computed by bootstrap or analytically using formulas in Reference 4. We note that confidence intervals for $x_{\kappa }^{opt}$ and $\text{E}[Y(g_{x_{\kappa }^{opt}})]$ are not generally available in the resource constrained setting through standard methods. This is due to complications arising from an estimator obtained from a constrained optimization where the constraint itself contains a plugin estimator.

4.1 |. An illustrative simulation

In this section, we illustrate the approach with a simple simulated example. We also compare the performance of the strategy learned by our proposed method (which incorporates the resource constraint) with an alternative approach that applies the optimal unconstrained strategy until the resource limit is reached and then administers no further treatment.

We first generated an “observational” data set of 25 000 subjects according to the following data generating process:

$$L_0\sim Normal(0,1);N_0\sim Normal(logi{t}^{-1}(-0.5+|L_0|)$$

$$L_1(0)\sim Normal(L_0,1);L_1=L_1(0){0.5}^{N_0};N_1\sim Bernoulli(logi{t}^{-1}(-0.5+|L_1|))$$

$$Y(0)\sim Normal(L_1,1);Y=|Y(0){0.5}^{N_1}|+.5(N_0+N_1);D=N_0+N_1.$$

We set ourselves the goal of finding the treatment rule g setting N₀ and N₁ that would minimize Y subject to the arbitrary constraint that E[D(g)] ≤ κ = 0.25 for D = N₀ + N₁. In the absence of treatment, Y would be the absolute value of the third step in a normal random walk in which L₀ and L₁ are the first two steps. Setting N_t = 1 applies a multiplicative shrinking factor of .5 to step t + 1 of the random walk with a countervailing additive effect of 0.5 on Y. There is time varying confounding as N_t is more likely to be 1 when |L_t| is large under the observational regime.

We considered strategies {g_x} of the form “set N_t = 1 if |L_t| > x_t” for x₁, x₂ ∈ [0, 3]. Note that {g_x} allows the threshold to vary by time, so with two time points x is two dimensional. We specified flexible dynamic marginal structural models for outcome and doses:

$$\begin{gathered} \text{E}[Y(g_x)]=b(x;\beta )={\beta }_0+{\beta }_1{x}_1+{\beta }_2{x}_1^2+{\beta }_3{x}_1^3+{\beta }_4{x}_1^4+{\beta }_5{x}_1^5+{\beta }_6{x}_2+{\beta }_7{x}_2^2+{\beta }_8{x}_2^3+{\beta }_9{x}_2^4+{\beta }_{10}{x}_2^5+ \\ {\beta }_{11}{x}_1{x}_2+{\beta }_{12}{(x_1{x}_2)}^2+{\beta }_{13}{(x_1{x}_2)}^3+{\beta }_{14}{(x_1{x}_2)}^4 \end{gathered}$$

$$\begin{gathered} \text{E}[D(g_x)]=h(x;\theta )={\theta }_0+{\theta }_1{x}_1+{\theta }_2{x}_1^2+{\theta }_3{x}_1^3+{\theta }_4{x}_1^4+{\theta }_5{x}_1^5+{\theta }_6{x}_2+{\theta }_7{x}_2^2+{\theta }_8{x}_2^3+{\theta }_9{x}_2^4+{\theta }_{10}{x}_2^5+ \\ {\theta }_{11}{x}_1{x}_2+{\theta }_{12}{(x_1{x}_2)}^2+{\theta }_{13}{(x_1{x}_2)}^3+{\theta }_{14}{(x_1{x}_2)}^4. \end{gathered}$$

We then estimated parameters β and θ via weighted regression as described in Section 3, plugging probability of treatment estimates from a correctly specified logistic regression model into the weights. Given $\widehat{\beta }$ and $\widehat{\theta }$, we used the L-BFGS-B optimization algorithm²⁹ implemented in the R function optim() to estimate the unconstrained optimal strategy ${\widehat{x}}^{opt}$, and we used the NLopt²⁷ implementation of the method-of-moving-asymptotes²⁸ for optimization under nonlinear inequality constraints through the R package nloptr to solve (9) and estimate the resource constrained optimal strategy ${\widehat{x}}_{\kappa }^{opt}$. The unconstrained optimal strategy ${\widehat{x}}^{opt}$ was (3,0.61), meaning that the strategy “set N₁ = 1 if |L₁| > 3 and set N₂ = 1 if |L₂| > 0.61” leads to the optimal expected outcome of all strategies of the form “set N₁ = 1 if |L₁| > x₁ and set N₂ = 1 if |L₂| > x₂” in the absence of resource constraints. In contrast, the resource-constrained optimal strategy ${\widehat{x}}_{\kappa }^{opt}$ was (3,1.54), reflecting that it was never optimal to give treatment at the first time point and treatment would only be administered under more extreme values of the covariate at the second time point in the resource constrained case. The expected outcome under our estimated unconstrained optimal strategy from the class was $\text{E}[Y(g_{{\widehat{x}}^{opt}})]=1.16$, and the expected outcome under the estimated optimal strategy from the class respecting resource constraints was $\text{E}[Y(g_{{\widehat{x}}_{\kappa }^{opt}})]=1.20$, reflecting a small but inevitable sacrifice to accommodate resource constraints. (We computed these counterfactual mean outcome values via simulation as the mean outcome of N = 1 000 000 subjects generated under $g_{{\widehat{x}}^{opt}}$ and $g_{{\widehat{x}}_{\kappa }^{opt}}$. The values therefore reflect ground truth but are also very close to model generated estimates.)

To assess the gains that would be realized from implementing the resource-constrained optimal strategy as opposed to a realistic alternative, we also computed the mean Y obtained when treatment is administered following the unconstrained optimal strategy $g_{{\widehat{x}}^{opt}}$ until the resource limit is reached and then no further treatment is administered. This resource naive strategy resulted in a mean outcome of 1.3, which was over three times as far away from the unconstrained optimal outcome of 1.16 as the optimal resource constrained strategy $(E[Y(g_{x_{\kappa }^{opt}})]=1.2)$, illustrating the utility of our approach. R code for this simulation is available with supplementary materials, and at https://github.com/CausalInference/Resource_Constrained_dynMSM_Simulation.

5 |. APPLICATION TO MONITORING OF HIV PATIENTS

We apply the method described above to estimate the optimal resource-constrained dynamic strategy for monitoring CD4 cell count and HIV-RNA in people living with HIV using data from the HIV-CAUSAL collaboration.³⁰ The HIV-CAUSAL collaboration combines data from prospective cohorts of people living with HIV enrolled in universal health care systems in Brazil, Canada, France, Greece, Netherlands, Spain, Switzerland, UK, and USA. Due to privacy concerns, the data are available from HIV-CAUSAL only through an approval process.

We have previously reported on the optimal dynamic monitoring strategy in this cohort and showed that decreasing monitoring when CD4 cell count >200 cells/μL compared to >500 cells/μL does not worsen short-term clinical and immunologic outcomes in virologically suppressed individuals living with HIV but may increase the risk of virological failure.³¹ We now extend these results to identify the optimal resource-constrained dynamic strategy under constraints on the average number of monitoring events over a two-year period.

First we briefly describe the eligibility criteria and monitoring strategies under consideration. We then describe the estimation of the optimal resource-constrained dynamic strategy.

Eligibility criteria:

Previously antiretroviral therapy naive HIV-positive individuals who initiate antiretroviral therapy in 2000 or later and achieve confirmed virologic suppression (2 consecutive HIV-RNA ≤200 copies/mL) within 12 months of initiating therapy are eligible for inclusion in the study. Individuals must meet the following additional eligibility criteria at baseline (date of confirmed virologic suppression): 18 years of age or older, CD4 cell count measurement within the previous 3 months, no history of an AIDS-defining illness, and no pregnancy (when information was available).

Monitoring strategies:

We consider 31 dynamic monitoring strategies, based loosely on current clinical guidelines. Under each strategy, CD4 cell count and HIV-RNA are monitored every 3 to 6 months when CD4 is below the strategy’s threshold and every 9 to 12 months when CD4 is above the threshold. Each strategy corresponds to a CD4 threshold ranging from 200 to 500 cells/μL in increments of 10 cells/μL. All of the monitoring strategies further require individuals to be monitored once every 3 to 6 months when HIV-RNA >200 copies/mL or after diagnosis of an AIDS-defining illness, and that CD4 cell count and HIV-RNA be monitored concurrently.

Follow-up period and outcome:

Individuals are followed from baseline until death, pregnancy (if known), loss to follow-up, or the administrative end of follow-up. The outcome of interest is virologic failure (HIV-RNA >200 copies/mL) at 24 months of follow-up.

Statistical methods:

We compare the 31 monitoring strategies using the replication and censoring approach. Briefly, we create an expanded dataset by making 31 exact replicates of each individual (1 per strategy). If and when an individual’s data are no longer consistent with a given strategy, we artificially censor the corresponding replicate at that time. We compute inverse probability weights to adjust for the potential selection bias induced by the artificial censoring.

We then fit an inverse-probability weighted Poisson regression model to estimate the risk ratio of virologic failure at 24 months of follow-up among those with measurements at 24 ± 2 months. The model includes a flexible functional form of the strategy variable (restricted cubic splines) and the baseline covariates: sex, CD4 cell count (<200, 200-349, 350-499, ≥500 cells/μL), years since HIV diagnosis (<1, 1 to 4, ≥5 years, unknown), race (white, black, other or unknown), geographic origin (N. America/W. Europe, Sub-Saharan Africa, other, unknown), acquisition group (heterosexual, homosexual or bisexual, injection drug use, other or unknown), calendar year (restricted cubic splines with 3 knots at 2001, 2007, and 2011), age (restricted cubic splines with 3 knots at 25, 39, and 60), cohort, and months from cART initiation to virologic suppression (2-4, 5-8, ≥9). Under the assumptions described above, the parameters of the regression model consistently estimate the parameters of a dynamic marginal structural model $logPr(Y(g_x)=1|V)={\beta }_0+{\beta }_{1}h(x)+{\beta }_2^{\prime }V$. The model’s estimated parameters are used to estimate the standardized risk of virologic failure at 24-months for each monitoring strategy.

Next, we fit an inverse-probability weighted log-linear regression model to estimate the mean number of measurements at 24 months of follow-up. As above, the model includes a flexible functional form of the strategy variable and the baseline covariates. Again, under the assumptions described above, the parameters of the regression model consistently estimate the parameters of a dynamic marginal structural model $\text{E}[D(g_x)|V]={\theta }_0+{\theta }_{1}f(x)+{\theta }_2^{\prime }V$. The predicted values are used to estimate the standardized mean number of measurements at 24-months for each monitoring strategy.

After estimating the counterfactual risk of virologic failure at 24 months and counterfactual mean number of measurements at 24 months, we perform a 3-step algorithm to rank the dynamic strategies and selection the best strategy among the subset of strategies that satisfy a hypothetical resource constraint. First, we rank the strategies by the counterfactual mean number of measurements at 24 months (highest to lowest), obtained from the dynamic MSM E[D(g_x)] = h(x; θ). Second, we restrict our consideration to the strategies that satisfy the resource constraint characterized by a maximum average number of tests per patient. For example, if the resource constraint states that CD4 cell count and HIV-RNA may be measured a maximum of once every 6 months on average, we would only consider the strategies that lead to an average of 4 or fewer measurements per person over 24 months of follow-up. Third, we choose the strategy with the smallest risk of virologic failure among the strategies that satisfy the resource constraint, obtained from the dynamic MSM E[Y(g_x)] = b(x; β). In the third step, we use a grid search to find the best strategy among the subset of strategies corresponding to discrete CD4 cell counts rather than solving the optimization problem to find the optimal continuous CD4 cell count. This is the optimal strategy for minimizing the risk of virologic failure among the strategies that satisfy the constraint.

Results:

47 635 individuals initiated antiretroviral therapy in 2000 or later and met the eligibility criteria for our study. At baseline, we made 31 copies of each eligible individual, for a total sample size of 1 476 685. During follow-up, CD4 cell count and HIV-RNA were measured on average every 4 and 3.8 months, respectively. Figure 1 shows the estimates obtained from the dynamic MSMs for virologic failure at 24 months and for mean number of monitoring events over 24 months across the range of CD4 cell count thresholds considered. In this example, the estimated risk of virologic failure is monotonically increasing and the estimated mean number of measurements is monotonically decreasing as the CD4 cell count threshold increases, so the optimal resource-constrained dynamic strategy can be identified graphically as the lowest CD4 cell count threshold for which the mean number of monitoring events over 24 months is below the resource constraint, κ. Table 1 gives the same information with 95% confidence intervals obtained via bootstrapping.

FIGURE 1.

Risk of virologic failure and mean number of measurements per person at 24 months of follow-up by CD4 threshold strategy. The grey line represents one potential resource constraint—a cap on the average per person number of measurements over 24 months of follow-up of 4. Strategies in the green shaded area meet this restriction, and the CD4 threshold 320 strategy is the optimal resource-constrained dynamic strategy [Colour figure can be viewed at wileyonlinelibrary.com]

TABLE 1.

Risk of virologic failure and cumulative number of measurements at 24 months for CD4 threshold

CD4 threshold* (cells/μL)	Virologic failure		Cumulative # Measurements
	Risk (%)	95% CI	Expected value	95% CI
500	6.91	(3.99, 9.82)	4.94	(4.82, 5.06)
490	6.85	(4.09, 9.61)	4.89	(4.78, 5.00)
480	6.80	(4.17, 9.43)	4.84	(4.73, 4.94)
470	6.74	(4.21, 9.27)	4.78	(4.69, 4.88)
460	6.69	(4.23, 9.15)	4.73	(4.64, 4.83)
450	6.64	(4.22, 9.05)	4.68	(4.59, 4.78)
440	6.59	(4.18, 8.99)	4.63	(4.54, 4.73)
430	6.54	(4.13, 8.95)	4.58	(4.49, 4.68)
420	6.52	(4.08, 8.95)	4.53	(4.43, 4.63)
410	6.51	(4.03, 8.99)	4.48	(4.38, 4.58)
400	6.54	(4.02, 9.05)	4.43	(4.32, 4.53)
390	6.60	(4.05, 9.15)	4.37	(4.27, 4.48)
380	6.71	(4.14, 9.28)	4.31	(4.21, 4.42)
370	6.87	(4.26, 9.48)	4.25	(4.15, 4.36)
360	7.08	(4.41, 9.76)	4.19	(4.08, 4.30)
350	7.33	(4.55, 10.12)	4.13	(4.02, 4.24)
340	7.62	(4.67, 10.57)	4.07	(3.95, 4.18)
330	7.93	(4.79, 11.08)	4.01	(3.89, 4.13)
320	8.27	(4.91, 11.62)	3.96	(3.84, 4.08)
310	8.61	(5.09, 12.14)	3.92	(3.79, 4.04)
300	8.97	(5.31, 12.61)	3.88	(3.75, 4.00)
290	9.33	(5.59, 13.06)	3.84	(3.71, 3.97)
280	9.70	(5.91, 13.49)	3.81	(3.68, 3.93)
270	10.08	(6.22, 13.94)	3.78	(3.65, 3.91)
260	10.48	(6.52, 14.43)	3.75	(3.62, 3.88)
250	10.88	(6.78, 14.99)	3.73	(3.59, 3.86)
240	11.31	(6.98, 15.64)	3.70	(3.55, 3.85)
230	11.75	(7.12, 16.38)	3.67	(3.52, 3.83)
220	12.21	(7.21, 17.72)	3.65	(3.48, 3.82)
210	12.68	(7.25, 18.12)	3.62	(3.44, 3.80)
200	13.18	(7.23, 19.13)	3.60	(3.41, 3.79)

Note: The monitoring strategies falling in the bold values meet the restriction that CD4 cell count and HIV-RNA may only be monitored every six months. Among the monitoring strategies that meet the restriction, the 320 threshold strategy is the optimal strategy in bold italic.

^*The CD4 Threshold corresponds to the CD4 cell count at which monitoring frequency changes from once every 2 to 7 months (if CD4 cell count is below the threshold) to once every 8 to 13 months (if CD4 cell count is above the threshold). Each strategy also includes monitoring once every 2 to 7 months when HIV-RNA>200 copies/mL or after diagnosis of an AIDS-defining illness.

In our example, we consider the case of κ = 4 and identify the optimal threshold for switching monitoring frequency as 320 cells/μL. The optimal resource-constrained dynamic strategy is then ‘monitor CD4 cell count and HIV-RNA every 3 to 6 months when CD4 is below 320 cells/μL and every 9 to 12 months when CD4 is above 320 cells/μL’.

6 |. CONCLUSIONS

Dynamic treatment strategies are a better representation of real-world clinical decision-making processes than static or point intervention strategies. However, resource utilization of dynamic strategies is difficult to assess, since the number of individuals requiring intervention over time under a given strategy cannot be straightforwardly determined at baseline. When a health system faces resource constraints that prohibit implementing the true optimal dynamic treatment strategy, the optimal resource-constrained dynamic strategy is instead required.

Here we propose a method to identify the optimal resource-constrained dynamic strategy within a parameterized class of strategies of interest by estimating the counterfactual resource usage. We apply this method to estimate the optimal resource-constrained dynamic strategy for monitoring frequency among individuals living with HIV who achieve virologic suppression.

Our choice of κ = 4 was somewhat arbitrary. If we had instead chosen κ = 3, we would have found that none of the strategies under consideration would satisfy this resource constraint. Interestingly, if we had chosen κ = 4.7, we would have identified the optimal threshold for switching monitoring frequency as 410 cells/μL, even though all the strategies in the range from 200 to 450 cells/μL would have satisfied the resource constraint (however, the confidence intervals around our estimates are quite wide).

In reality, determining the number of CD4 cell count and HIV-RNA measurements a setting is willing to allocate depends on a complex assessment of the costs and health benefits of monitoring. In our illustrative application, we imposed a constraint on the number of tests, which we imagined was derived from a hypothetical corresponding cost constraint. In other applications, it might be useful to directly bound cost instead. For example, since HIV-RNA tests cost more than CD4 tests, an optimal strategy satisfying a total cost constraint may be a more flexible joint strategy that allows CD4 cell count and HIV-RNA to be monitored with different frequencies. In addition, a constraint on the number of tests could be caused by stock-outs, an increasing global problem given the current COVID-19 pandemic. Future studies should also assess other health outcomes such as quality-adjusted life years associated with various monitoring strategies. Finally, even in the absence of a single hard resource constraint, examining outcomes of optimal strategies over a range of hypothetical cost constraints could allow for computation of incremental cost-effectiveness ratios, which could be useful for key stakeholders and decision-makers.

DATA AVAILABILITY STATEMENT

Data from each participating study in HIV-CAUSAL can be requested from the sponsoring institution in accordance with the applicable laws or regulations in each country.