Implementing Generalized Additive Models to Estimate the Expected Value of Sample Information in a Microsimulation Model: Results of Three Case Studies

Abstract

Background

The expected value of sample information (EVSI) can help prioritize research, but its application is hampered by computational infeasibility, especially for complex models. We investigated an approach by Strong and colleagues to estimate EVSI by applying generalized additive models (GAM) to results generated from a probabilistic sensitivity analysis (PSA).

Methods

For three potential HIV prevention and treatment strategies, we estimated life expectancy and lifetime costs using the Cost-Effectiveness of Preventing AIDS Complications (CEPAC) model, a complex patient-level microsimulation model of HIV progression. We fitted a GAM, a flexible regression model that estimated the functional form as part of the model fitting process, to the incremental net monetary benefits obtained from the CEPAC PSA. For each case study, we calculated expected value of partial perfect information (EVPPI) using both the conventional nested Monte Carlo approach and the GAM approach. EVSI was calculated using the GAM approach.

Results

For all three case studies, the GAM approach consistently gave similar estimates of EVPPI compared to the conventional approach. The EVSI behaved as expected: it increased and converged to EVPPI for larger sample sizes. For each case study, generating the PSA results for the GAM approach required 3–4 days on a shared cluster, after which EVPPI and EVSI across a range of sample sizes were evaluated in minutes. The conventional approach required approximately 5 weeks for the EVPPI calculation alone.

Conclusion

Estimating EVSI using the GAM approach with results from a PSA dramatically reduced the time required to conduct a computationally intense project, which would otherwise have been infeasible. Using the GAM approach, we can efficiently provide policy makers with EVSI estimates, even for complex patient-level microsimulation models.

Introduction

Computer simulation models of disease are used to compare the expected incremental net benefits of alternative strategies based on a set of input parameters; however, these parameters are rarely known with certainty. The expected value of perfect information (EVPI) and the expected value of partial perfect information (EVPPI) represent the economic benefit of eliminating uncertainty in all parameters or a subset of parameters; the EVPI targets all model parameters whereas the EVPPI relates to a specified subset. As this would require a study of infinite sample size, a more practical metric is the expected value of sample information (EVSI), which represents the benefit of reducing parameter uncertainty by collecting study outcomes in a finite sample, as done in a clinical trial. EVSI can determine whether further research will demonstrate good value and which study designs and sizes will be optimal.

The conventional approach to EVPPI and EVSI uses a nested Monte Carlo procedure.¹ While considered the gold standard, the conventional approach is computationally intensive, especially for complex patient-level microsimulation models. Though alternative methods have recently been developed to facilitate and expedite EVPPI and EVSI estimation,^2–7 no practical applications to a complex model exist to support them. To address this research gap, we applied one of these methods, a flexible regression approach developed by Strong and others,⁵ to three different case studies of human immunodeficiency virus (HIV) prevention and treatment. The first two, for the South Africa context, examined the value of long-acting pre-exposure prophylaxis (LA-PrEP) for high-risk women⁸ and a hypothetical HIV cure strategy.⁹ The third considered the use of a generic version of efavirenz (EFV) as part of initial antiretroviral therapy (ART) in the US.¹⁰ Costs and benefits for these case studies were grounded on recently published or conducted cost-effectiveness reports using the Cost-Effectiveness of Preventing AIDS Complications (CEPAC) model, a complex patient-level microsimulation of HIV disease and treatment.^11–13

We begin by summarizing the general theoretical evaluation of EVPPI and EVSI. Then, we describe the practical calculation of EVPPI and EVSI using the conventional and alternative approaches. Next, we outline the three case studies used to illustrate the benefits of the alternative approach applied to the CEPAC model. Finally, we compare the EVPPI results between the two approaches, report the EVSI results from the alternative approach, and discuss implications of our analysis.

Methods

Theoretical evaluation of EVPPI and EVSI

Suppose we have a health economic model that, based on a set of uncertain parameters, θ, estimates incremental net benefit, INB(d, θ), for each of D different strategies (d = 1, …, D), such that d = 1 is the reference strategy (incremental net benefit of zero). The value of a decision made with current information is simply the maximum expected incremental net benefit across all strategies, where the expectation for each strategy is taken over the prior distribution of θ (Table 1, Equation 1A). For ease of comparison across the different approaches, all equations are presented in Table 1.

Table 1.

Terms and equations to calculate EVPPI and EVSI.

	Description	Target expectation	Conventional nested Monte Carlo approach	GAM approach
Equation #		A	B	C
1	Value of decision made with CURRENT information	$${\mathit{\text{max}}}_d{E}_{\theta }\mathit{\text{ INB}}(d,\theta )$$	$${\mathit{\text{max}}}_d\frac{1}{N}\sum _{n=1}^N\mathit{\text{INB}}(d,{\theta }^n)$$	$${\mathit{\text{max}}}_d\frac{1}{M}\sum _{m=1}^M\mathit{\text{INB}}(d,{\theta }^m)$$
2	Value of decision made with PARTIAL PERFECT information	$$E_{{\theta }_T}{\mathit{\text{ max}}}_d{E}_{\theta |{\theta }_T}\mathit{\text{ INB}}(d,\theta )$$	$$\frac{1}{K}\sum _{k=1}^K{\mathit{\text{max}}}_d\frac{1}{J}\sum _{j=1}^J\mathit{\text{INB}}(d,{\theta }_C^{k,j},{\theta }_T^k)$$	$$\frac{1}{M}\sum _{m=1}^M{\mathit{\text{max}}}_d{\hat{g}}^m(d,{\theta }_T)$$
3	EVPPI	$$E_{{\theta }_T}{\mathit{\text{ max}}}_d{E}_{\theta |{\theta }_T}\mathit{\text{ INB}}(d,\theta )-{\mathit{\text{max}}}_d{E}_{\theta }\mathit{\text{ INB}}(d,\theta )$$	$$\frac{1}{K}\sum _{k=1}^K{\mathit{\text{max}}}_d\frac{1}{J}\sum _{j=1}^J\mathit{\text{INB}}(d,{\theta }_C^{k,j},{\theta }_T^k)-{\mathit{\text{max}}}_d\frac{1}{N}\sum _{n=1}^N\mathit{\text{INB}}(d,{\theta }^n)$$	$$\frac{1}{M}\sum _{m=1}^M{\mathit{\text{max}}}_d{\hat{g}}^m(d,{\theta }_T)-{\mathit{\text{max}}}_d\frac{1}{M}\sum _{m=1}^M{\hat{g}}^m(d,{\theta }_T)$$
4	Value of decision made with SAMPLE information	$$E_X{\mathit{\text{ max}}}_d{E}_{\theta |X}\mathit{\text{ INB}}(d,\theta )$$	$$\frac{1}{K}\sum _{k=1}^K{\mathit{\text{max}}}_d\frac{1}{J}\sum _{j=1}^J\mathit{\text{INB}}(d,{\theta }_C^{k,j},{\theta }_T|X^k)$$	$$\frac{1}{M}\sum _{m=1}^M{\mathit{\text{max}}}_d{\hat{g}}^m(d,X)$$
5	EVSI	$$E_X{\mathit{\text{ max}}}_d{E}_{\theta |X}\mathit{\text{ INB}}(d,\theta )-{\mathit{\text{max}}}_d{E}_{\theta }\mathit{\text{ INB}}(d,\theta )$$	$$\frac{1}{K}\sum _{k=1}^K{\mathit{\text{max}}}_d\frac{1}{J}\sum _{j=1}^J\mathit{\text{INB}}(d,{\theta }_C^{k,j},{\theta }_T|X^k)-{\mathit{\text{max}}}_d\frac{1}{N}\sum _{n=1}^N\mathit{\text{INB}}(d,{\theta }^n)$$	$$\frac{1}{M}\sum _{m=1}^M{\mathit{\text{max}}}_d{\hat{g}}^m(d,X)-{\mathit{\text{max}}}_d\frac{1}{M}\sum _{m=1}^M{\hat{g}}^m(d,X)$$

Abbreviations: EVPPI: expected value of partial perfect information, EVSI: expected value of sample information, GAM: generalized additive model.

Note: The uncertain parameters (θ) are divided into target parameters (θ_T) and complementary parameters (θ_C) and the incremental net benefit of strategy d given true values of the uncertain parameter is given by INB (d, θ). The expected value over potential realizations of uncertain parameters is denoted E_θ (similarly denoted for other expectations). The entire set of potential study outcomes is denoted X. In the conventional approach, iterations are denoted by K for the number of realizations of target parameters and J for the number of sets of complementary parameters, with N = K × J total results, while in the GAM approach, iterations are denoted by M. Parameters and study outcomes are indexed with superscript lower case letters: n, m, j, and k. In the conventional approach, the updated portion of the joint conditional distribution involving the target parameters, conditional on the study outcomes, is denoted by θ_T|X^k. In the GAM approach, ĝ^m(d, θ_T) is the m^th fitted value obtained from the GAM of strategy d based on the entire set of target parameters used to fit the model, while ĝ^m(d, X) is the m^th fitted value obtained from the GAM of strategy d based on the entire set of study outcomes used to fit the model.

First, we assume we can eliminate all uncertainty about a subset of the unknown parameters by carrying out a theoretical study that gives us perfect information. We refer to the subset of parameters for which information would be gained from this study as target parameters, denoted θ_T, and the other uncertain parameters that would not be measured in this study as complementary parameters, θ_C. As an example, we might learn the efficacy (θ_T) of a drug in the study; however, the drug cost (θ_C) would not be informed from the study and thus remains a complementary parameter. The value of a decision made with partial perfect information (perfect information about θ_T, but not θ_C) is the maximum expected incremental net benefit across strategies, where the expectation for each strategy is taken with respect to the conditional distribution of θ given θ_T, denoted θ|θ_T. Since we do not yet know the true value of θ_T, we take the expectation of the maximized incremental net benefit over potential realizations of θ_T (Table 1, Equation 2A). By definition, the EVPPI is the maximum price a decision maker should pay for additional perfect information on this subset of parameters. Mathematically, the EVPPI is the difference between the value of a decision made with partial perfect information and with current information (Table 1, Equation 3A). While obtaining perfect information is rarely possible, the EVPPI does provide an upper bound on the value that a real study could provide.

Study outcomes, X, collected from a finite sample size reduce, rather than entirely eliminate, the uncertainty about θ_T, with uncertainty reduced as the size of the study increases. The value of a decision made with sample information is the maximum expected incremental net benefit across strategies, where the expectation for each strategy is taken with respect to the conditional distribution of θ|X. Again, as we have not yet observed these study outcomes, we average over potential realizations of X (Table 1, Equation 4A). The EVSI is the difference between the value of a decision made with sample information and with current information (Table 1, Equation 5A). The EVSI is recalculated for each potential sample size and study design considered.

Conventional approach to estimating EVPPI and EVSI

The gold standard approach to EVPPI developed by Ades and others uses a nested Monte Carlo procedure.¹ Although this nested procedure is not necessary for some simple cases (e.g., linear models), more complex models, such as the CEPAC model—which involves thousands of input parameters, nonlinear relationships, and parameter interactions—require this more computationally intensive approach.

To implement the conventional nested Monte Carlo approach, we sample K realizations of the target parameters, and J sets of complementary parameters for each of the k = 1, …, K realizations of the target parameters, sampling the complementary parameters conditional on the target parameters. Each set of parameters is then used as input into the CEPAC model, resulting in N = J × K model evaluations for each strategy. The value of a decision made with current information is estimated by taking the maximum mean incremental net benefit across strategies (Table 1, Equation 1B).

To estimate the value of a decision made with partial perfect information, we calculate the mean of the incremental net benefits across the J sets of complementary parameters for each set of target parameters, resulting in K mean incremental net benefit values for each of the D strategies. Then, for each kth set of values across strategies, we take the maximum, resulting in a single set of K maximum mean incremental net benefit values. The value of a decision made with partial perfect information is estimated by taking the mean of this set (Table 1, Equation 2B) and the conventional EVPPI is calculated as the difference (Table 1, Equation 3B).

We provide the conventional equations for EVSI in Table 1 (Equations 4B and 5B). We did not calculate conventional EVSI in our analysis, as it was not feasible given the computational requirements of the CEPAC model. To calculate EVSI for a specific sample size, after drawing a sample θ_T, one would draw a hypothetical result for the study conditional on the value of θ_T, then update and draw from the conditional distribution, θ|X^k, based on the hypothetical result of the study.

Generalized additive model approach

An alternative approach to EVPPI or EVSI calculation developed by Strong and others⁵ uses a generalized additive model¹⁴ (GAM) to regress the simulation model-based incremental net benefits, obtained from PSA results, on the target parameters (for EVPPI) or study outcomes (for EVSI). The expected value of the incremental net benefits is assumed to be some smooth function of the target parameters in the GAM approach. The GAM does not impose a specific functional form, but rather estimates the functional form as a part of the model fitting process. Any variation due to the complementary parameters is accounted for in the error term of the model.⁵ This regression model can also include higher order terms, such as interactions between target parameters. In essence, a GAM is simply an extension of the multivariable linear regression model in which the linear form is replaced by a sum of unspecified smooth functions.

We obtained incremental net benefits for each strategy from the results of a standard probabilistic sensitivity analysis (PSA), which jointly sampled M potential realizations of the target and complementary parameters and used these realizations as input parameters in the CEPAC model. Following the algorithm outlined by Strong and others, we use a GAM to regress incremental net benefits on the target parameters and obtain the fitted values of incremental net benefit, denoted ĝ^m (d, θ_T) for the m^th fitted value obtained from the GAM of strategy d based on the entire set of target parameters, θ_T. Using these fitted values, the difference between a decision made with partial perfect information and with current information estimates the EVPPI (Table 1, Equation 3C). Note that in Table 1, Equation 3C, we estimate the value of the decision made with current information using fitted values obtained from the GAM, ${\mathit{\text{max}}}_d\frac{1}{M}{\sum }_{m=1}^M{\hat{g}}^m(d,{\theta }_T)$, instead of those based on the PSA, ${\mathit{\text{max}}}_d\frac{1}{M}{\sum }_{m=1}^M\mathit{\text{INB}}(d,{\theta }_m)$, since this leads to an increased precision of the EVPPI estimate due to a reduction in Monte Carlo sampling variability.⁵

Estimating EVSI using the GAM approach is a two-step process. In the first step, for each of the M iterations of the PSA, a study outcome (X^m) is simulated conditional on the target parameter value ( ${\theta }_T^m$) in that iteration. For example, for a target parameter of the probability of HIV virologic suppression of an HIV therapy at 48 weeks, the outcome is the number of participants in a study suppressed at 48 weeks. In the second step, we regress incremental net benefits on the study outcomes. We then obtain the fitted values and calculate the difference between the value of a decision made with sample information and with current information to estimate the EVSI (Table 1, Equation 5C). For increased precision, we again estimate the value of a decision made with current information using the fitted values obtained from the GAM of the study outcomes X^m, rather than those based on the PSA.

Case Studies

We chose three case studies to illustrate results. Cases selected were CEPAC analyses, recently published^8–10 and are representative of ongoing HIV-related clinical trials or active fields of investigation.^{15, 16} During the analysis of the EVSI results, we found interesting technical aspects unique to each case that highlight a spectrum of possible issues that might arise when using the GAM to estimate EVSI and therefore included all three. The studies were assumed to update beliefs about target parameters but not to update beliefs about complementary parameters.

In the CEPAC model, individual patients are generated from random draws of user-defined cohort characteristics. Natural history of HIV disease progression is modeled as a series of monthly transitions between health states characterized by CD4 count (a marker of immune function) and HIV RNA viral load (which measures burden of HIV infection). Users also define HIV prevention or treatment strategies specific to the questions and setting of interest. Cost outcomes are generally assessed from the health system perspective. A more detailed technical specification of the CEPAC model is available at http://www.massgeneral.org/mpec/cepac/.

Case study 1: LA-PrEP for high-risk South African women

Standard oral pre-exposure prophylaxis (Std-PrEP) is one of the few effective HIV prevention strategies currently available for high-risk South African women; however, its effectiveness is inherently linked to, and often limited by, daily pill adherence.^17–24 Long-acting injectable PrEP (LA-PrEP), currently under development in a Phase 2B/3 clinical trials, provides sustained drug levels, which could improve adherence and therefore effectiveness of PrEP.²⁵ A recent CEPAC analysis demonstrated that even with an increased relative cost, LA-PrEP would prove cost-effective compared to Std-PrEP at its anticipated effectiveness benefit.⁸ Since a placebo-controlled trial would be unethical, we considered the value of a two-arm trial with two-year follow-up to reduce the uncertainty of two target parameters: annual HIV incidence if LA-PrEP was implemented and annual HIV incidence if Std-PrEP was implemented. Complementary parameters included drug costs of LA-PrEP and laboratory monitoring costs for the two strategies. Incremental net benefits were calculated with Std-PrEP as the reference strategy using a primary willingness-to-pay threshold of $6,600/year of life saved (YLS), the 2014 South African per capita Gross Domestic Product (GDP).²⁶ Two alternative willingness-to-pay thresholds of $1,650/YLS and $19,800/YLS (25% and 300% of per capita GDP, respectively) were considered to examine the sensitivity of our results.

Case study 2: A hypothetical cure strategy in South Africa

While the elimination of HIV from individuals has been demonstrated as possible based on a single case of HIV eradication using hematopoietic stem cell transplantation,²⁷ ongoing research is actively pursuing other more viable cure strategies.^{16, 28, 29} We examined the performance benchmarks required for a hypothetical cure strategy to be a cost-effective alternative to currently available antiretroviral therapy (ART) for the treatment of HIV infection by South African standards.⁹ For this analysis, we examined the value of a single-arm two-year study among HIV patients with durable virologic suppression on ART to obtain estimates of two target parameters of a cure strategy: cure efficacy and probability of relapse after cure. Complementary parameters included the cost of a cure strategy and the one-time probability of fatal toxicity related to a cure strategy. We assessed the value of a single-arm trial along the lines of a hypothetical cure strategy in this case study, so the study only updated beliefs about the cure strategy. Efficacy of the reference strategy, currently available ART, was not updated by this study. Incremental net benefits were calculated with ART as the reference strategy and used the same willingness-to-pay thresholds as the first case study.

Case study 3: Use of generics for efavirenz (EFV) in the US

In the US, at the time of this project, the current standard of care for first-line ART was most commonly a branded, single-pill, co-formulated regimen, e.g. tenofovir disoproxil fumarate (TDF)/emtricitabine (FTC)/efavirenz (EFV). With the anticipated approval of generic efavirenz (EFV) in the US,³⁰ decision makers would face the dilemma of whether and even more importantly what to recommend for a switch to a multi-pill, generic-based regimen option. Thus, we compared the value of two hypothetical two-arm trials, each with a two-year follow-up: a 3-pill trial, comparing a 3-pill generic (generic EFV + generic lamivudine [3TC] + branded TDF) regimen to a single-pill branded (TDF/FTC/EFV) regimen, and a 2-pill trial, comparing a 2-pill generic (generic EFV + branded TDF/FTC) regimen to a single-pill branded regimen. For both studies, target parameters were the probabilities of 24-week HIV-RNA suppression and subsequent ART failure (after 24 weeks and through 96 weeks) specific to each strategy. Complementary parameters included mean CD4 cell count at ART initiation, costs of the generic drug components, monthly probability of lost to follow-up, and upon being lost, the monthly probability of returning to care. Incremental net benefits (EVPPI and EVSI) were calculated assuming the frequently-cited US-based willingness-to-pay threshold of $100,000/YLS.^31–33 Again, two alternative willingness-to-pay thresholds of $25,000/YLS and $300,000/YLS were examined.

Parameter distributions for each case study can be found in Supplementary Table 1.

Computational and Statistical Methods for Value of Information Calculations

For the conventional approach, we used K = 5,000 realizations of the target parameters, J = 400 sets of complementary parameters, with 100,000 patients for each set of inputs, to estimate EVPPI using the conventional approach for each of our examples. This resulted in 2×10¹¹ (200 billion) patient lifetimes evaluated in CEPAC per strategy. For EVSI, this procedure would be repeated for each sample size considered. The number of K and J simulations was pragmatic. A run with K = 1,000 realizations of the target parameters and J = 400 sets of complementary parameters required over a week to complete, and was used as they generally could be completed successfully. Larger runs tended to hit either storage capacity or time limits on the institutional research computing cluster. We recognize, however, that the relatively low number of sets of complementary parameters could lead to an upward bias in these calculations, but decided to accept this potential problem rather than make each of the projects even more computationally intensive. We combined the results from five separate runs to obtain the 5,000 realizations of the target parameters for each case study.

In contrast, when using the GAM approach, the 50,000 PSA simulations with 100,000 patients for each set of inputs required only 5×10⁹ (5 billion) patient lifetime evaluations of the CEPAC model per strategy to evaluate EVPPI, 1/40th of the number of evaluations used in the conventional approach. Though there was no theoretical justification for the number of PSA simulations used, we selected 50,000 as a large, yet feasible, number. No additional evaluations were required to estimate EVSI for different sample sizes because EVSI uses the same PSA runs as the EVPPI estimation. Therefore, the only additional sampling needed was for the study outcomes that were calculated almost instantaneously.

For all three case studies, after calculating the mean net benefit for each strategy, we calculated the mean incremental net benefit for each non-reference strategy. For the conventional approach to EVPPI, we averaged over the J = 400 conditional complementary parameter values in each set and then averaged over the maximum incremental net benefit over the K = 5,000 realizations of the target parameters. For the GAM approach, we regressed incremental net benefits on the target parameters using the “gam” function within the R package “mgcv” (v 1.8–9).³⁴ The statistical analyses of the CEPAC results to obtain EVPPI and EVSI were performed using R version 3.2.2 (www.r-project.org) and conducted on a standard windows-based desktop computer.

To estimate EVSI, a single hypothetical study outcome was generated conditional on the target parameter drawn for each PSA iteration. This process was repeated for each study sample size of interest. As all target parameters were probabilities in our case studies, we assumed a binomial likelihood for data that are informative for the target parameters. Table 2 illustrates the process involved in determining the results for EVSI for a two-arm study with a primary outcome of suppression, as in case study 3. This data simulation step was repeated for a range of hypothetical sample sizes to obtain the EVSI results for each case study.

Table 2.

Illustration of data used to calculate EVPPI and EVSI using GAM approach.

Iteration	Simulated results from PSA*			Potential outcome of a study with 500 participants per arm (conditional on target parameters)
	Target parameters (θT): Suppression rate		Maximum incremental net benefit	Study outcomes (X) : Number suppressed
	Control arm	Experimental arm		Control arm	Experimental arm
1	87%	83%	0	442	422
2	86%	91%	83,212	427	465
3	84%	93%	244,616	414	462
4	86%	86%	0	433	419
…
50,000	86%	87%	9,498	426	432

Abbreviations: EVPPI: expected value of partial perfect information, EVSI: expected value of sample information, GAM: generalized additive model, PSA: probabilistic sensitivity analysis.

Note: This table illustrates the basic data setup for the GAM approach to EVPPI and EVSI. Both begin with a large number of PSA iterations, jointly sampling all uncertain parameters (both the target parameters of the study, θ_T, and the complementary parameters, θ_C) and calculating the corresponding incremental net benefit values based on our patient-level simulation model. In our case studies, there were 50,000 iterations of the PSA. This illustration assumes a study estimating only the suppression rate in a control and experimental arm. Using the GAM approach, the value of a decision made with partial perfect information is estimated by the mean maximum value when incremental net benefits are predicted using a GAM based on the entire set of target parameters, θ_T (Table 1, Equation 2C). The value of a decision made with sample information is estimated as the mean maximum value when incremental net benefits are predicted using a GAM based on the entire set of study outcomes, X (Table 1, Equation 4C).

^*While we draw for complementary parameters (column not shown), they are not specified in the table, as they are not involved in the GAM estimation of either EVPPI or EVSI; however, they are important for the estimation of incremental net benefit.

Standard errors of the conventional EVPPI estimates were calculated by taking the standard deviation of 10,000 bootstrapped EVPPIs. For each iteration of the bootstrapping procedure, we sampled 5,000 incremental net benefit values with replacement and applied Equation 3B (Table 1) to calculate EVPPI. Standard errors of the GAM-based estimates were calculated by first drawing 1,000 sets of GAM coefficients from a multivariate normal distribution (using the coefficients and covariance matrix output from the GAM). We then applied the 1,000 new coefficients to the data from the PSA to obtain 1,000 new sets of fitted values (using the prediction matrix obtained from the GAM). Using these 1,000 sets of fitted values, we calculated 1,000 corresponding EVPPIs or EVSIs (using Equation 3C or 5C, respectively). The standard deviation of these results was used to estimate the standard error of the EVPPI/EVSI, as suggested by Strong and others.⁵

GAM diagnostic plots were produced to ensure that modeling assumptions (and, thus, standard error estimates) were valid. First, partial residual plots—with the model-based component smooth functions overlaid—were used to assess whether the expectation of the incremental net benefits was a smooth function of the target parameters (EVPPI) or study outcomes (EVSI). Then, we plotted the fitted values against the GAM residuals and the simulated incremental net benefits to assess whether any unmodeled structure existed. For all three case studies, model diagnostics did not reveal any violations to the underlying assumptions for the final models used for the analysis.

We considered tensor-products between target parameters in the second and third case studies as we anticipated that there would be an interaction between parameters on incremental net benefit: such interactions are an inherent feature in the CEPAC model. Tensor-products of terms treat the interaction as an additional bivariate smooth function in the GAM model. For example, in the second case study if a cure strategy has a higher efficacy, then the relapse rate would have a more pronounced effect on the incremental net benefit than it would if there was low efficacy.

Results

Case study 1: LA-PrEP for high-risk South African women: behavior and precision across a variety of sample sizes

Assuming a willingness-to-pay threshold of $6,600/YLS, mean net benefit was $159,100/person for LA-PrEP and $157,800/person for Std-PrEP; mean incremental net benefit of LA-PrEP was $1,300/person. Using the conventional Monte Carlo approach, we calculated the EVPPI of removing the uncertainty of the two efficacy target parameters to be $103/person. We obtained nearly the same estimate of EVPPI ($105/person) using the GAM approach (Table 3, top). The conventional nested approach led to a relatively precise estimate of EVPPI (standard error $3); the GAM approach, however, led to even more precision, with standard errors <$1. As expected, EVSI estimates increased monotonically and approached EVPPI as the sample size increased from 25 to 50,000 patients per study arm (Table 3, bottom). The increase in EVSI was drastic initially and then began to reach its asymptote for sample sizes above 5,000 patients per arm.

Table 3.

Results for the three case studies.

	Case study 1* LA-PrEP for high-risk South African women		Case study 2† A cure strategy in South Africa		Case study 3‡ Use of generics for EFV in the US
					2-pill trial		3-pill trial
	EVPPI (SE), $§
Conventional Approach	103	(3)	151	(6)	6,900	(443)	5,500	(517)
GAM Approach	105	(<1)	151	(1)	7,100	(19)	5,900	(25)
	EVSI (SE), $§
Hypothetical trial sample size per arm
50	3	(<1)	95	(2)	5,300	(40)	4,200	(44)
100	10	(<1)	117	(2)	5,900	(37)	4,800	(37)
200	25	(<1)	133	(1)	6,400	(31)	5,200	(35)
500	54	(1)	143	(1)	6,800	(26)	5,600	(30)
1,000	71	(1)	147	(1)	6,900	(23)	5,700	(28)
5,000	95	(<1)	150	(1)	7,100	(20)	5,800	(27)
10,000	100	(<1)	151	(1)	7,100	(20)	5,800	(26)
50,000	104	(<1)	151	(1)	7,100	(19)	5,800	(25)

Abbreviations: EFV: efavirenz, EVPPI: expected value of partial perfect information, EVSI: expected value of sample information, GAM: generalized additive model, LA-PrEP: long-acting pre-exposure prophylaxis, PSA: probabilistic sensitivity analysis, SE: standard error.

^*Incremental net benefits were calculated with standard oral pre-exposure prophylaxis as the reference strategy using a primary willingness-to-pay threshold of $6,600 per year of life saved (100% of the 2014 South African per capita GDP).

^†Incremental net benefits were calculated with currently available ART as the reference strategy using a primary willingness-to-pay threshold of $6,600 per year of life saved (100% of the 2014 South African per capita GDP).

^‡Incremental net benefits were calculated with the generic as the reference strategy using a US-based willingness-to-pay threshold of $100,000 per year of life saved (typical threshold used in the US).^31–33

^§Standard errors of the conventional EVPPI estimates were calculated by taking the standard deviation of 10,000 bootstrapped EVPPIs. Standard errors of the GAM-based estimates were calculated using the approach suggested by Strong and others.⁵

Case study 2: A hypothetical cure strategy in South Africa: parameter interactions and non-linear functions

Assuming a willingness-to-pay threshold of $6,600/YLS, mean net benefit was $116,300/person for the cure strategy and $114,100/person for the current ART strategy; mean incremental net benefit of cure was $2,200/person. The conventional approach estimated the EVPPI of cure efficacy and probability of relapse to be $151/person. The GAM approach gave the same value ($151, Table 3, top). Again, the standard error of the GAM approach ($1) was smaller than the standard error of the conventional approach ($6). EVSI again increased monotonically and approached EVPPI as sample size increased (Table 3, bottom).

Figure 1 illustrates the inherent interaction between efficacy and relapse rate on incremental net benefit in the CEPAC model. On the left, we stratified relapse rate into two equal sized groups: low relapse (bottom quartile of simulations) and high relapse (top quartile of simulations). As cure efficacy increased, the rate of increase in the incremental net benefit was faster (steeper positive slope) in simulations with low vs. high relapse. Intuitively, this reflects that the economic benefit of higher efficacy is enhanced when the relapse rate is lower. Figure 1 also demonstrates the flexible nature of the GAM approach, which was able to fit the non-linear relationship between cure relapse rate and incremental net benefit (Figure 1, right).

Figure 1.

Parameter interaction and non-linear function in case study 2: a hypothetical cure strategy in South Africa. This figure demonstrates the interaction between the two target parameters by plotting each against incremental net benefit, colored by lower (light grey) and upper (dark grey) quartile of the other target parameter. For example, high rate of cure relapse decreased the positive effect efficacy of cure had on incremental net benefit (i.e., shallower positive slope with high relapse; dark on left panel) compared to a low relapse rate (light on left panel). Conversely, high efficacy increased the negative effect of a higher relapse rate on incremental net benefit (i.e., steeper negative slope with high efficacy; dark on right panel) compared to low efficacy (light on right panel). The right panel also demonstrates the nonlinear relationship between cure relapse rate and incremental net benefit, especially for the high efficacy group, such that as relapse increased, incremental net benefit decreased, but at a progressively slower rate. For clarity, only 5,000 randomly selected draws from the 50,000 PSA iterations were included in Figure 1.

Case study 3: Use of generics for EFV in the US: scenario limiting pre-trial decision to non-optimal reference strategy

Assuming a US-based willingness-to-pay threshold of $100,000/YLS, mean net benefit was $981,400/person for the single-pill branded strategy, $1,010,200/person for the 2-pill generic strategy, and $1,022,700/person for the 3-pill generic strategy; mean incremental net benefit was $28,800/person for the 2-pill versus single-pill branded strategy and $41,300/person for the 3-pill versus single-pill branded strategy. Using the conventional approach, we calculated the EVPPI for both hypothetical parallel group trials of generic EFV against SOC: $6,900/person for the 2-pill trial and $5,500/person for the 3-pill trial scenarios. The GAM approach resulted in similar EVPPI estimates of $7,100/person for the 2-pill trial and $5,900/person for the 3-pill trial. Although the EVPPI estimate for the 3-pill trial obtained using the GAM approach was about 7% different from that obtained using the conventional approach, it was still within the confidence interval for the conventional Monte Carlo estimate. Again, the standard errors of the estimates were much larger for the conventional approach than for the GAM approach (Table 3, top). EVSI increased monotonically and approached EVPPI as sample size increased. Similar to the relapse term in the second case study, the GAM approach identified a non-linear relationship between subsequent late ART failure and incremental net benefit (data not shown). Based on the relative value of the two trials, it would appear that the 2-pill trial offered the greater potential benefit than the 3-pill trial (e.g., $6,800/person vs. $5,600/person for the information gained from a study with 500 participants) (Table 3).

Using two alternative willingness-to-pay thresholds for each case study led to consistent EVPPI values (between conventional and GAM approach) and similar behavior and precision of EVSI (Supplementary Table 2).

Analysis Time

Although we did not systematically record the computing requirements for the various case studies, the final analysis for the conventional approach to EVPPI took approximately 5 weeks per case study on an institutional shared research computing cluster. Using the conventional approach would require a similar amount of computing time to calculate EVSI for each sample size considered. Once we obtained the results of the PSA runs (generated after 3–4 days per case study), no additional PSA runs were required for estimating EVPPI or EVSI across a range of sample sizes. The time to generate the study outcomes used in estimating the EVSI, fit the GAM to the study outcomes, and calculate EVSI took less than 1 minute for all the sample sizes presented once the PSA results were available.

Discussion

The GAM approach facilitated EVSI calculation for three case studies using the CEPAC model, turning analyses that would have required months into ones that could be done in days. Compared to the conventional approach, this alternative method drastically reduced the number of evaluations of the CEPAC model, making EVSI calculations computationally feasible. We found that for EVPPI, the GAM approach and the conventional approach gave similar results (within 7%). The conventional approach was not feasible for EVSI. We also demonstrated the broad applicability of this method by including examples of non-linear functions and parameter interactions.

There has been a general calling for applied examples and suggestions that this EVSI approach should be tested in more complex patient-level models.⁵ We provide the first pragmatic application of this method to a complex patient-level model. The CEPAC model has thousands of input parameters, requiring weeks of computational runtime to calculate EVSI for a single trial and sample size, making the conventional approach to EVSI computationally infeasible. Moreover, with the CEPAC model, there is no closed form solution to the joint conditional parameter distribution, which created the need for a nested Monte Carlo procedure in the conventional approach; the GAM approach, however, required no closed form solution and a much more modest computational burden using standard PSA, which have become the norm in many health economic analyses.³⁵

It has been recommended that value of information research should investigate how this method will perform under different structural assumptions.³⁶ Previous applications of this method have considered only simple hypothetical economic models in which parameters are multi-linear and independent. We intentionally selected scenarios that contained non-linear functions between parameters and incremental net benefits as well as parameter interactions, and also compared the value of multiple trials to aid trial design. This analysis demonstrated that the GAM approach was able to adapt nicely to these structural irregularities.

The results of this analysis should be taken in the context of its limitations. First, there is no standard software available to calculate EVSI using this alternative GAM approach. To conduct this analysis, we expanded upon R code that has been previously developed for calculating EVPPI, so that it would also provide EVSI results; sample code for the first case study is in the Supplemenary Material. Second, while we have validated our EVPPI results, there is no gold standard by which to validate our EVSI estimates. We must rely on the face validity of our findings—that EVSI estimates initially increase rapidly as our sample size increases and then converge to EVPPI for larger samples. Third, others have proposed different alternatives to calculating EVSI;^2–7 we focused on the GAM approach because of its flexibility in considering a non-linear functional form and in considering the interaction between terms (as highlighted in case study 2).

In considering each of the cases, we were not exhaustive in the target parameters considered; we selected several important target parameters as a means of illustrating the viability of this approach. Thus, a more complete value of information analysis for each case might also include examination of other target parameters, unequal treatment allocations, and a more comprehensive consideration of additional (more than 2) intervention arms. Finally, we have estimated in this analysis the incremental net benefit for an individual. The next step to a complete value of information analysis – which would examine whether the overall anticipated benefit is worth the cost of the trial itself—must expand results from an individual to the population level. Other factors would then merit investigation, including: the total number of individuals anticipated to benefit, the diffusion of the intervention among those who might benefit, the time horizon of the expected benefit, and the potential for future interventions that might limit this horizon.

In the realm of HIV research, as in other areas of medical research, financial resources are limited. Given the large number of potentially worthwhile studies, funding agencies must have a streamlined approach to prioritize HIV trials based on their economic value. EVSI is a critical method to answer that call; however, its computational complexity is a strong limitation on its practical impact. We demonstrated that the GAM approach to EVSI addresses this challenge, as this fast and flexible regression approach makes estimation of EVSI feasible even for complex patient-level simulation models. The GAM approach provides a rapid means of informing policy makers which strategies should be adopted or whether more research is needed before making that decision.

Glossary

ART

antiretroviral therapy

CEPAC

Cost-effectiveness of Preventing AIDS Complications Microsimulation Model

Complementary parameters (θ_c)

other uncertain parameters

Conventional approach

nested Monte Carlo approach to EVPPI/EVSI introduced by Ades and others¹

EVPI

expected value of perfect information

EVPPI

expected value of partial perfect information

EVSI

expected value of sample information

GAM

generalized additive model, as described by Hastie/Tibshirani¹⁴

GAM approach

approach to EVPPI/EVSI estimation using generalized additive model (GAM) and probabilistic sensitivity analysis (PSA) introduced by Strong and others⁴

HIV

human immunodeficiency virus

PSA

probabilistic sensitivity analysis

Study outcomes (X)

a simulated data set that could be obtained from a study of a specified size, conditional on the target parameters

Target parameters (θ_T)

parameters for which information would be obtained from a trial

Uncertain parameters (θ)

parameters which are uncertain in the model estimating incremental net benefit, which is the union of target parameters and complementary parameters.

YLS

year of life saved