Value of Information: Sensitivity Analysis and Research Design in Bayesian Evidence Synthesis

ABSTRACT

Suppose we have a Bayesian model that combines evidence from several different sources. We want to know which model parameters most affect the estimate or decision from the model, or which of the parameter uncertainties drive the decision uncertainty. Furthermore, we want to prioritize what further data should be collected. These questions can be addressed by Value of Information (VoI) analysis, in which we estimate expected reductions in loss from learning specific parameters or collecting data of a given design. We describe the theory and practice of VoI for Bayesian evidence synthesis, using and extending ideas from health economics, computer modeling and Bayesian design. The methods are general to a range of decision problems including point estimation and choices between discrete actions. We apply them to a model for estimating prevalence of HIV infection, combining indirect information from surveys, registers, and expert beliefs. This analysis shows which parameters contribute most of the uncertainty about each prevalence estimate, and the expected improvements in precision from specific amounts of additional data. These benefits can be traded with the costs of sampling to determine an optimal sample size. Supplementary materials for this article, including a standardized description of the materials available for reproducing the work, are available as an online supplement.

1. Introduction

Bayesian modeling is a natural paradigm for decision making, in the presence of uncertainty, based on multiple sources of evidence. However, as more data sources, parameters, and assumptions are built into a model, it becomes harder to see the influence of each input or assumption. The modeling process should involve an investigation of where the weak parts of the model are, to identify which uncertainties in the model inputs contribute most to the uncertainty in the final result or decision (sensitivity analysis). We might then want to assess and compare the potential value of obtaining datasets of specific designs or sizes to strengthen different parts of the model. Furthermore, we may want to formally trade-off the costs of sampling with the resulting expected improvement to decision making.

Annual estimation of HIV prevalence in the United Kingdom has, for several years, been based on a Bayesian synthesis of evidence from various surveillance systems and other surveys (Goubar et al. 2008; Presanis et al. 2010; De Angelis et al. 2014; Kirwan et al. 2016). This is an example of a class of problems called multiparameter evidence synthesis (MPES) (e.g., Ades and Sutton 2006), where the quantities of interest cannot be estimated directly, but can be inferred from multiple indirect data sources linked through a network of model assumptions that can be expressed as a directed acyclic graph. Markov chain Monte Carlo (MCMC) is typically required to estimate the posterior. The HIV MPES model is used to inform health policies, thus it is crucial to be able to assess sensitivity to uncertain inputs and to indicate how the model could be strengthened with further data.

These dual aims can be achieved with value of information (VoI) analysis, a decision-theoretic framework based on expected reductions in loss from future information. The concepts of VoI were first set out in detail by Raiffa and Schlaifer (1961), while Parmigiani and Inoue (2009) give a more recent overview. The expected value of partial perfect information (EVPPI) is the expected reduction in loss if the exact value of a particular parameter or parameters ${\mathit{\theta }}_0$ were learnt, also interpreted as the amount of decision uncertainty that is due to ${\mathit{\theta }}_0$. The expected value of sample information (EVSI) is the expected reduction in loss from a study of a specific design. The EVSI can be traded off with the costs of data collection to give the expected net benefit of sampling (ENBS). Therefore, as well as recommending a policy based on minimising expected loss under the current model and data, the decision-maker may also recommend collecting further data according to a design which minimises the ENBS.

These concepts have been applied in various forms in three distinct areas: health economics, computer modeling and Bayesian design. In health economic modeling, there is a large literature on calculation and application of VoI, see, for example, Felli and Hazen (1998); Willan and Pinto (2005); Claxton and Sculpher (2006); Welton et al. (2008). The model output is then the expected net benefit of each alternative policy, a known deterministic function $g(\mathit{\theta })$ of uncertain inputs $\mathit{\theta }$, and the decision problem is the choice of policy that minimises $g(\mathit{\theta })$. In computer modeling, see, for example, Oakley and O’Hagan (2004) and Saltelli et al. (2004), the influence of a particular element ${\mathit{\theta }}_0$ of $\mathit{\theta }$ is calculated as the expected reduction in $\text{var}(g(\mathit{\theta }))$, if we were to learn ${\mathit{\theta }}_0$ exactly. This is equivalent to the EVPPI for ${\mathit{\theta }}_0$ under a decision problem defined as point estimation of $g(\mathit{\theta })$ with quadratic loss (Oakley and O’Hagan 2004). The decision-theoretic view of Bayesian experimental design also has a long history, see, e.g. Lindley (1956); Bernardo and Smith (1994); Chaloner and Verdinelli (1995); Berger (2013), and a recent review of the computational challenges by Ryan et al. (2016).

However, the current tools in any one of these three areas cannot be applied directly to MPES. First of all, it is not always feasible or desirable to make a discrete decision with a quantifiable loss, as in health economic modeling. Instead, the aim of evidence synthesis is often to estimate one or more quantities. For a scalar quantity of interest, we might then define the “loss” as the posterior variance of this quantity, as Oakley and O’Hagan (2004) described in the computer modeling context. In computer modeling, however, tools to estimate the expected value of a proposed study to learn a particular ${\mathit{\theta }}_0$ more precisely have not been developed, and it is not clear what an appropriate loss for a vector of model outputs would be. Challenges also arise with computation. Current methods for computing the expected variance reduction in the computer modeling field (Sobol’ 2001; Saltelli et al. 2004) assume the output is an explicit function $g(\mathit{\theta })$ of the inputs, therefore do not apply in MPES, where this function is unknown and the outputs must be estimated by MCMC. For Bayesian design, Ryan et al. (2016) reviewed methods where evaluating the expected utility of a design (equivalent to the EVSI) is relatively inexpensive, so that maximizing the utility over a complex design space is feasible. However, this can again be difficult with MCMC. Given a sample from the posterior $p(\theta |\mathbf{x})$, potential future datasets $\mathbf{y}$ under a specific design can be simulated cheaply from the posterior predictive distribution, but then to obtain the expected utility, the posterior $p(\theta |\mathbf{x},\mathbf{y})$ needs to be repeatedly updated for different $\mathbf{y}$, which is feasible with Monte Carlo only for smaller problems (e.g., Han and Chaloner 2004).

Here, we use and extend methods from health economics, computer modeling, and Bayesian design to devise a new VoI framework for sensitivity analysis and research design in evidence syntheses based on graphical models fitted by MCMC. This is a broader class of models than those typically used in health economics or computer modeling, since the model “output” is not necessarily a known function of the inputs, but depends on the model parameters $\mathit{\theta }$ and observed data $\mathbf{x}$ through a network of statistical models or deterministic functions, potentially with hierarchical relationships. We apply this new VoI framework to the part of the HIV prevalence estimation model that estimates prevalence in men who have sex with men (MSM), in London. Here, the decision problem is point estimation of a single scalar or a vector of parameters, followed by the choice of what extra data should be collected in the future. We use ideas from Bayesian design to choose appropriate loss functions in this context. We also generalize methods of computing EVPPI (Strong, Oakley, and Brennan 2014) and EVSI (Strong et al. 2015), developed for finite choices in health economics, to a broader class of decision problems, including point estimation. The method for computing EVSI enables the expected utility over all potential $\mathbf{y}$ to be estimated cheaply without an additional level of simulation, assuming only that the information provided by $\mathbf{y}$ can be represented as a low-dimensional sufficient statistic $T(\mathbf{y})$.

In Section 2, we describe the general MPES model, and define the expected VoI under different decision problems and loss functions, and in Section 3, we present methods to compute them. In Section 4, we describe the model for HIV prevalence estimation, and in Section 5, we use VoI to identify the areas of greatest uncertainty in this model and determine what specific data should be collected to improve the precision of the estimates of various subgroup-specific prevalences. Finally, we discuss potential extensions to the methods and application and the associated challenges.

2. Theory and Methods

2.1. Bayesian Graphical Modeling for Evidence Synthesis

In our motivating applications, the general model can be represented as a directed acyclic graph (Figure 1) in the standard way, see, for example, Lauritzen (1996). Nodes in the graph may represent scalar or vector quantities. A set of datasets $\mathbf{x}=\{x_1,\dots ,x_n\}$ is observed, most generally from n different sources. These data are assumed to arise from statistical models with parameters ${\mu }_1,\dots ,{\mu }_n$ respectively, collectively denoted $\mathit{\mu }$. The “founder nodes” of the graph are denoted $\mathit{\varphi }=({\varphi }_1,\dots ,{\varphi }_p)$ and given a joint prior distribution $\mathit{\varphi }\sim p(.)$ which may also include substantive information. The full set of unknowns is denoted $\mathit{\theta }$. Most simply, the $\mathit{\mu }$ could equal the $\mathit{\varphi }$ or be related to the $\mathit{\varphi }$ through deterministic functions, so that $\mathit{\theta }=\mathit{\varphi }$. More generally, some of the relationships in the graph could be stochastic, defining a hierarchical model, where the $\mathit{\mu }$ themselves arise from a distribution with parameters given by the $\mathit{\varphi }$ or descendants of $\mathit{\varphi }$. The vector of unknowns $\mathit{\theta }$ would then comprise $\mathit{\varphi }$ and the stochastic descendants of $\mathit{\varphi }$ such as random effects.

Fig. 1.

Directed acyclic graph for Bayesian evidence synthesis.

We further denote $\mathit{\alpha }$ as an intermediate node in the graph, the model “output,” which is used for decision making. This could be any unknown quantity, including one of the $\mathit{\mu }$ or $\mathit{\varphi }$, a function of these, or a prediction of new data. We may also plan to collect additional data, either from the same source as one of the existing datasets (e.g., y₁ in Figure 1), or from a new source informing a parameter ${\mu }_{n+1}$ on which no direct data (y₂) were available.

This DAG (Figure 1) is a generalization of the typical structure (Figure 2) used in computer modeling (Oakley and O’Hagan 2004) where the output $\mathit{\alpha }$ is a known (usually complicated) deterministic function of uncertain model inputs $\mathit{\varphi }$, which are given substantive priors that may be derived separately from data.

Fig. 2.

Graph representing a known deterministic model.

2.2. Expected VoI: Definitions

In a general decision-theoretic framework, the purpose of the model is to choose a decision or action d from a space of possible decisions $\mathcal{D}$, to minimise an expected loss $E_{\mathit{\theta }}(L(d,\mathit{\theta }))$, with the expectation taken with respect to the posterior distribution of $\mathit{\theta }$. Let $\mathit{\alpha }=\mathit{\alpha }(\mathit{\theta })$ be the minimal subset or function of $\mathit{\theta }$ necessary to make the decision, so that $E_{\mathit{\theta }}(L(d,\mathit{\theta }))=E_{\mathit{\alpha }}(L(d,\mathit{\theta }))$, $\forall d\in \mathcal{D}$. For example, the purpose could be the choice of decision d among a finite set $\mathcal{D}=\{1,\dots ,D\}$ expected to minimize a loss defined as a function of the parameters, so that $\mathit{\alpha }$ would be a vector with D components ${\alpha }_d=f_d(\mathit{\theta })=L(d,\mathit{\theta })$. This is the typical situation in health policy decisions (e.g., Claxton and Sculpher 2006), where a treatment d is chosen to maximize a measure of utility such as expected quality-adjusted survival. Alternatively, as in our examples, the decision could simply be the choice of a point estimate $\widehat{\mathit{\alpha }}$ of some parameter $\mathit{\alpha }$, in which case the decision space $\mathcal{D}$ is the support of $\mathit{\alpha }$ (see Section 2.3). Alongside making a decision, we wish to also determine where further research should be prioritized to reduce the uncertainty about the decision, and given the costs of data collection, to determine the optimal design of further research (see Section 2.4).

For general decision problems, let $d^{*}=\text{arg} {\text{min}}_d{E}_{\mathit{\theta }}(L(d,\mathit{\theta }))$ be the optimal decision under current knowledge about $\mathit{\theta }$, represented by the posterior distribution $p(\mathit{\theta }|\mathbf{x})$. Suppose now we are in a position to collect new information. Let $d_{\mathbf{y}}^{*}$ be the optimal decision given further knowledge of a quantity $\mathbf{y}$ (either parameters or potential data) that informs $\mathit{\alpha }$, so that the updated posterior would be $p(\mathit{\theta }|\mathbf{x},\mathbf{y})$. We define the following quantities.

1. The expected value of perfect information (EVPI) is the expected loss of the decision $d^{*}$ under current information, minus the expected loss for the decision $d_{\mathit{\alpha }}^{*}$ we would make if we knew the true $\mathit{\alpha }$ (Raiffa and Schlaifer 1961).

   $$\text{EVPI}=E_{\mathit{\theta }}\left(L\right(d^{*},\mathit{\theta }\left)\right)-E_{\mathit{\theta }}\left(L\right(d_{\mathit{\alpha }}^{*},\mathit{\theta }\left)\right)$$

   Since additional information is always expected to reduce the expected loss of the optimal decision (Parmigiani and Inoue 2009), the EVPI is an upper bound on the expected gains from any new information.

2. The expected value of partial perfect information (EVPPI) for a particular parameter $\mathit{\varphi }$ is the expected reduction in loss if $\mathit{\varphi }$ were to be learnt precisely. Since this precise value is not yet known, an expectation must be taken over all possible values.

   $$\text{EVPPI}\left(\mathit{\varphi }\right)=E_{\mathit{\theta }}\left(L\right(d^{*},\mathit{\theta }\left)\right)-E_{\mathit{\varphi }}\left[E_{\mathit{\theta }|\mathit{\varphi }}\right(L(d_{\mathit{\varphi }}^{*},\mathit{\theta })\left)\right],$$ (1)

   where $d_{\mathit{\varphi }}^{*}$ is the optimal decision if $\mathit{\varphi }$ were known. This is an upper bound on the potential value of data $\mathbf{y}$ which inform only $\mathit{\varphi }$. In a graphical model, this means data $\mathbf{y}$ that are conditionally independent of $\mathit{\theta }$ given $\mathit{\varphi }$, for example $\mathbf{y}=y_1$ and $\mathit{\varphi }={\mu }_1$ in Figure 1.

3. The expected value of sample information $\text{EVSI}(\mathbf{y})$ is the reduction in loss we would expect from collecting an additional dataset $\mathbf{y}$ of a specific design

   $$\text{EVSI}\left(\mathbf{y}\right)=E_{\mathit{\theta }}\left(L\right(d^{*},\mathit{\theta }\left)\right)-E_{\mathbf{y}}\left[E_{\mathit{\theta }|\mathbf{y}}\left(L\right(d_{\mathbf{y}}^{*},\mathit{\theta })\right].$$ (2)

The inner expectation is now with respect to the updated posterior distribution of $\mathit{\theta }|\mathbf{y}$, after learning $\mathbf{y}$ as well as the existing data $\mathbf{x}$, or “preposterior” (Berger 2013).

2.3. VoI in Different Decision Problems

Finite-Action Decisions

For a choice of d among a finite set $\{1,\dots ,D\}$ with loss $L(d,\mathit{\theta })={\alpha }_d$ and $\mathit{\alpha }=\{{\alpha }_1,\dots ,{\alpha }_d,\dots ,{\alpha }_D\}$, the expected loss with current information is ${\min }_d\{E_{\mathit{\alpha }}({\alpha }_d)\}$, so (Raiffa and Schlaifer 1961)

$$\begin{array}{c}\text{EVPI}=\underset{{}_d}{\min }\left\{E_{\mathit{\alpha }}\right({\alpha }_d\left)\right\}-E_{\mathit{\alpha }}\underset{{}_d}{\min }\left\{{\alpha }_d\right\},\\ \text{EVPPI}\left(\mathit{\varphi }\right)=\underset{{}_d}{\min }\left\{E_{\mathit{\alpha }}\right({\alpha }_d\left)\right\}-E_{\mathit{\varphi }}\underset{{}_d}{\min }\left\{E_{\mathit{\theta }|\mathit{\varphi }}\right({\alpha }_d\left)\right\},\\ \text{EVSI}\left(\mathbf{y}\right)=\underset{{}_d}{\min }\left\{E_{\mathit{\alpha }}\right({\alpha }_d\left)\right\}-E_{\mathbf{y}}\underset{{}_d}{\min }\left\{E_{\mathit{\alpha }|\mathbf{y}}\right({\alpha }_d\left)\right\},\end{array}$$ (3)

Point estimation

When the decision is the choice of a point estimate $\widehat{\mathit{\alpha }}$ of a vector of parameters $\mathit{\alpha }$, with quadratic loss

$$L(\widehat{\mathit{\alpha }},\mathit{\alpha })={(\widehat{\mathit{\alpha }}-\mathit{\alpha })}^{T}H(\widehat{\mathit{\alpha }}-\mathit{\alpha })$$ (4)

for a symmetric, positive-definite H, the optimal estimate with current information is the posterior mean, $\widehat{\mathit{\alpha }}=E_{\mathit{\alpha }}(\mathit{\alpha })$.

For a scalar $\mathit{\alpha }=\alpha$ and H = 1, the expected loss is $\text{var}(\alpha )$ under current information and zero under perfect information, so that $\text{EVPI}=\text{var}(\alpha )$ and

$$\text{EVPPI}\left(\mathit{\varphi }\right)=\text{var}(\alpha )-E_{\mathit{\varphi }}\left[{\text{var}}_{\alpha |\mathit{\varphi }}(\alpha |\mathit{\varphi })\right]$$ (5)

$$\text{EVSI}\left(\mathbf{y}\right)=\text{var}(\alpha )-E_{\mathbf{y}}\left[{\text{var}}_{\alpha |\mathbf{y}}(\alpha |\mathbf{y})\right]$$ (6)

the expected reduction in variance given new information. Expression (5) is used by Oakley and O’Hagan (2004) and Saltelli et al. (2004) as a measure of sensitivity of the output of a deterministic model $\alpha =g(\varphi ,\dots )$ to an uncertain input $\varphi$, termed the main effect of $\varphi$, but this has not been extended to the EVSI of potential data $\mathbf{y}$ in a point estimation context.

When $\mathit{\alpha }$ is a vector, the typical situation where a MPES of the form in Figure 1 is carried out, we could conduct independent VoI analyses for each component of $\mathit{\alpha }$. In more formal decision analyses, we may want a scalar loss for the overall vector $\mathit{\alpha }$. There are various alternatives based on generalizations $v(\mathit{\alpha })$ of the variance, which can be used instead of the scalar variance $\text{var}(\alpha )$ in Equations (5) and (6) to define the expected VoI. These have been applied in the context of Bayesian study design, and we explain two examples that can be adopted for EVPPI and EVSI in our context as follows.

1. If $H=\mathbf{c}{\mathbf{c}}^T$ in the quadratic loss (4), for some vector of weights $\mathbf{c}$, then the expected loss is $v(\mathit{\alpha })={\mathbf{c}}^T\text{cov}(\mathit{\alpha })\mathbf{c}=\text{var}({\mathbf{c}}^T\mathit{\alpha })$, corresponding to optimal (under squared error loss) estimation of the weighted sum of the parameters, ${\mathbf{c}}^T\mathit{\alpha }$. For example, when the elements α_s of $\mathit{\alpha }$ are weighted equally, the goal is to minimise the sum of all elements (r, s) of the covariance matrix, $v(\mathit{\alpha })=\sum _{r,s}\text{cov}{(\mathit{\alpha })}_{r,s}$, or, if the α_s are also independent of each other, $v(\mathit{\alpha })=\mathit{tr}(\text{cov}(\mathit{\alpha }))=\sum _s\text{var}({\alpha }_s)$. The same absolute reductions in variance for different components of $\mathit{\alpha }$ would then be valued equally. More generally, if $\mathbf{c}$ is given a prior, then loss (4) also arises (see Chaloner and Verdinelli 1995 and references therein). Designs that minimize (4) are Bayesian analogs of classical A-optimal designs. See also Lamboni, Monod, and Makowski (2011) for similar measures of sensitivity for multivariate outputs in deterministic computer models.

2. A Bayesian D-optimal design, on the other hand, minimizes the determinant $v(\mathit{\alpha })=\det (\text{cov}(\mathit{\alpha }))$ (Chaloner and Verdinelli 1995; Ryan et al. 2016). This simplifies to the product of the $\text{var}({\alpha }_s)$ when the α_s are independent and equally weighted. Equivalently, a standardized version $\det {(\text{cov}(\mathit{\alpha }))}^{1/S}$, where S is the number of components of $\mathit{\alpha }$, represents a geometric average variance of the α_s, adjusted for their covariance.

Here, the same relative reductions in variance for different components of $\mathit{\alpha }$ would then be valued equally, which would be more appropriate when the output of interest $\mathit{\alpha }$ comprises quantities on very different scales and/or with different interpretations.

2.4. Maximising the Expected Net Benefit of Sampling

The EVSI measures the expected benefits from sampling. The costs of sampling should also be considered. The decision-maker can then choose the design and sample size for data $\mathbf{y}$ to maximize the expected net benefit of sampling $E_{\mathbf{y}}(b(\mathbf{y})-c(\mathbf{y}))$, where $b(\mathbf{y})=\text{EVSI}(\mathbf{y})$ is the benefit and $c(\mathbf{y})$ is the cost of obtaining data $\mathbf{y}$ (Parmigiani and Inoue 2009). This requires benefits and costs to be measured on the same scale, which can be achieved in different ways. Improved precision of point estimates might be valued in monetary terms, as described below and illustrated in Section 5.4. Alternatively, the better knowledge given by the new data could lead to indirect benefits which could be valued, for example, improved health from better-informed health-related decision making, as discussed in Section 6. We will assume $c(\mathbf{y})$ depends only on the design and sample size, thus is known in advance of observing $\mathbf{y}$, so that $E_{\mathbf{y}}(c(\mathbf{y}))=c(\mathbf{y})$.

To directly translate improved precision to a monetary benefit, the decision-maker should specify the amount they are willing to pay to reduce the posterior variance by a certain amount. This willingness to pay may depend on the original posterior variance. Formally, the benefit function $b(\mathbf{y})=f(v_0,v_{\mathbf{y}})$, specified by the decision-maker, places a value on a reduction in variance (or its multivariate analog as in Section 2.3) from $v_0=\text{var}(\mathit{\alpha })$ to $v_{\mathbf{y}}$, the variance after collecting new data $\mathbf{y}$. For example, if any absolute variance reduction is valued the same way (as in A-optimal design, see Section 2.3), $f(v_0,v_{\mathbf{y}})=\lambda (v_0-v_{\mathbf{y}})$, where λ is the constant willingness-to-pay for one unit of variance reduction. The expected benefit is then $E_{\mathbf{y}}(b(\mathbf{y}))=\lambda (v_0-E_{\mathbf{y}}({\text{var}}_{\mathit{\alpha }|\mathbf{y}}(\mathit{\alpha }|\mathbf{y}))$, which equals $\text{EVSI}(\mathbf{y})$ using the quadratic loss function (4) multiplied by a constant λ. Alternatively, if the same relative gains are valued equally (as in D-optimal design), the decision-maker could specify λ as the amount they are willing to pay to (e.g.,) halve the variance, so that $f(v_0,v_{\mathbf{y}})=f(v_0,v_0/2^k)=k\lambda$, for $k=\hspace{0.17em}\text{log}\hspace{0.17em}(v_0/v_{\mathbf{y}})/\hspace{0.17em}\text{log}\hspace{0.17em}(2)$.

3. Computation of VoI

3.1. Partial Perfect Information

Computation of the EVPPI in general is not straightforward. Given a sample from the posterior distribution, the first term in Equation (1) can be calculated by a Monte Carlo mean. The double expectation in the second term is more challenging. While it can be evaluated using nested Monte Carlo, this is expensive. Strong, Oakley, and Brennan (2014) proposed a method for estimating the EVPPI in the special case of finite choice decisions (Equation 3) which uses only a single Monte Carlo loop. To estimate EVPPI (1) in a broader class of decision problems, which also includes point estimation, the method needs to be generalized.

Strong, Oakley, and Brennan (2014) estimated formula (3) by expressing

$${\alpha }_d=E_{{\alpha }_d|\mathit{\varphi }}\left({\alpha }_d\right|\mathit{\varphi })+ϵ=g_d(\mathit{\varphi })+ϵ$$ (7)

for each $d=1,\dots ,D$, where $ϵ$ is an error term with mean zero. Then, $g_d(\mathit{\varphi })$ is estimated by regression of α_d on $\mathit{\varphi }$, fitted to a Monte Carlo sample of $({\alpha }_d^{(k)},{\mathit{\varphi }}^{(k)}):k=1,\dots ,K$. If $\mathit{\varphi }$ comprises p parameters that could be learned simultaneously, the regression will have p predictors. Since the functional form of $g_d()$ will not be known in general, nonparametric regression methods are used. This produces a fitted value ${\widehat{g}}_d({\mathit{\varphi }}^{(k)})$ for each k, which allows the second term in Equation (3) to be estimated by a Monte Carlo mean

$$\begin{array}{c}{E}_{\mathit{\varphi }}\left[\underset{{}_d}{\min }\right(E_{\mathit{\theta }|\mathit{\varphi }}\left({\alpha }_d\right)\left)\right]=E_{\mathit{\varphi }}\left[\underset{{}_d}{\min }\right(E_{{\alpha }_d|\mathit{\varphi }}\left({\alpha }_d\right|\mathit{\varphi }\left)\right)]\\ \approx \frac{1}{K}\sum _{k=1}^K\underset{{}_d}{\min }\left({\widehat{g}}_d\right({\mathit{\varphi }}^{\left(k\right)}\left)\right).\end{array}$$ (8)

Our generalization of this approach computes EVPPI (Equation 1) in a broader class of decision problems defined as follows. Given a state of knowledge about the decision-relevant quantities $\mathit{\alpha }$ represented by a distribution $\psi (·)$, the expected loss under the optimal decision should be a known function h of the mean of $\mathit{\alpha }$ under that distribution

$$E_{\psi }\left(L\right(d_{\psi }^{*},\mathit{\theta }\left)\right)=h\left(E_{\psi }\right(\mathit{\alpha }\left)\right).$$ (9)

If $\psi (·)$ is the current posterior, this is $h(E_{\mathit{\alpha }}(\mathit{\alpha }))$, and if we were to learn the value of $\mathit{\varphi }$, the expected loss would be $h(E_{\mathit{\alpha }|\mathit{\varphi }}(\mathit{\alpha }|\mathit{\varphi }))$. The method of Strong, Oakley, and Brennan (2014) only applies to the special case, where $\mathit{\alpha }$ is a vector and $h(E(\mathit{\alpha }))={\min }_d\{E({\alpha }_d)\}$. To estimate $E_{\mathit{\alpha }|\mathit{\varphi }}(\mathit{\alpha }|\mathit{\varphi })$ in more general problems, we use a similar principle to (7–8), by expressing

$$\mathit{\alpha }=E_{\mathit{\alpha }|\mathit{\varphi }}\left(\mathit{\alpha }\right|\mathit{\varphi })+ϵ=g(\mathit{\varphi })+ϵ$$ (10)

then fitting a regression model $g()$ of $\mathit{\alpha }$ on $\mathit{\varphi }$ allows us to estimate

$$E_{\mathit{\varphi }}\left[E_{\mathit{\theta }|\mathit{\varphi }}\right(L(d_{\mathit{\varphi }}^{*},\mathit{\theta })\left)\right]=E_{\mathit{\varphi }}\left[h\right(E_{\mathit{\alpha }|\mathit{\varphi }}\left(\mathit{\alpha }\right|\mathit{\varphi }\left)\right)]\approx \frac{1}{K}\sum _{k=1}^{K}h(\widehat{g}\left({\mathit{\varphi }}^{\left(k\right)}\right)).$$

Point estimation problems are also a special case of Equation (9), for example, for estimation of a scalar α with quadratic loss, $h(E_{\alpha }(\alpha ))=E[{(\alpha -E_{\alpha }(\alpha ))}^2]=\text{var}(\alpha )$. Therefore to calculate EVPPI in this case (Equation (5)), we estimate $\text{var}(\alpha |{\mathit{\varphi }}^{(k)})$ by the squared residual ${(\alpha -\widehat{g}({\mathit{\varphi }}^{(k)}))}^2$, substitute this for $h(\widehat{g}({\mathit{\varphi }}^{(k)}))$ and estimate $E_{\mathit{\varphi }}\left[{\text{var}}_{\alpha |\mathit{\varphi }}(\alpha |\mathit{\varphi })\right]$ as the mean, over k, of the squared residuals. Equivalently, we can estimate $\text{var}(\theta )-E_{\mathit{\varphi }}\left[{\text{var}}_{\alpha |\mathit{\varphi }}(\alpha |\mathit{\varphi })\right]={\text{var}}_{\mathit{\varphi }}(E_{\alpha |\mathit{\varphi }}(\alpha |\mathit{\varphi }))$ as the variance, over k, of the fitted values. Similarly, for vector $\mathit{\alpha }$ and loss functions based on $\text{cov}(\mathit{\alpha })$, we can fit regressions to get the marginal mean for each component α_d, and calculate the empirical covariance matrix of the residuals.

Several methods of nonparametric regression have been suggested. For small p, Strong, Oakley, and Brennan (2014) used generalized additive models, with tensor products of splines to represent interactions between components of $\mathit{\varphi }$. Where $\mathit{\varphi }$ included about p = 5 or more components, Gaussian process regression was recommended as a more efficient way of modeling interactions, though the resulting matrix computations rapidly become impractical as the MCMC sample size K increases. Heath, Manolopoulou, and Baio (2016) developed an integrated nested Laplace approximation for fitting Gaussian processes more efficiently where $p\ge 2$. For the application in Section 4 (with K = 150, 000, $p\le 3$), we have found multivariate adaptive regression splines (Friedman 1991) via the earth R package (Milborrow 2011) to be more efficient. Standard errors for the EVPPI estimates can be calculated in general by simulating from the asymptotic normal distribution of the regression coefficients (Mandel 2013).

3.2. Sample Information

The regression method above can also be used to estimate the expected value of sample information $\text{EVSI}(\mathbf{y})$. This again requires a generalization of the approach described by Strong et al. (2015) from finite decision problems to any problem satisfying condition (9), including point estimation. The method requires that the information provided by the data $\mathbf{y}$ can be expressed as a low-dimensional sufficient statistic $T(\mathbf{y})$, so that $E_{\alpha |\mathbf{y}}(\mathit{\alpha }|\mathbf{y})=E_{\alpha |\mathbf{y}}(\mathit{\alpha }|T(\mathbf{y}))$. This could be a point estimator of the parameter μ (as in Figure 1) that $\mathbf{y}$ gives direct information on. As in (10), we can write

$$\mathit{\alpha }=E_{\mathit{\alpha }|\mathbf{y}}\left(\mathit{\alpha }\right|T\left(\mathbf{y}\right))+ϵ=g(T\left(\mathbf{y}\right))+ϵ$$

and estimate $g()$ using a regression fitted to a Monte Carlo sample of $({\mathit{\alpha }}^{(k)},T({\mathbf{y}}^{(k)})):k=1,\dots ,K$, where ${\mathbf{y}}^{(k)}$ are drawn from their posterior predictive distribution. Then, the fitted values $\widehat{g}(T({\mathbf{y}}^{(k)}))$ enable the double expectation to be estimated as

$$E_{\mathbf{y}}\left[E_{\theta |\mathbf{y}}\right(L(d_{\mathbf{y}}^{*},\mathit{\theta })\left)\right]=E_{\mathbf{y}}\left[h\right(E_{\alpha |\mathbf{y}}\left(\mathit{\alpha }\right|\mathbf{y}\left)\right)]\approx \frac{1}{K}\sum _{k=1}^{K}h(\widehat{g}\left(T\right({\mathbf{y}}^{\left(k\right)}\left)\right)).$$

Then, for example, for point estimation with quadratic loss, this is the estimated residual variance from the regression, as in Section 3.1.

4. The HIV Prevalence MPES Model

We consider the submodel of the full HIV burden model (De Angelis et al. 2014; Kirwan et al. 2016) that estimates HIV prevalence in men who have sex with men (MSM), in London. We define three subgroups of MSM: those who have attended a genitourinary medicine (GUM) clinic in the past year (GMSM), those who have not (NGMSM), and previous MSM (PMSM), men who no longer have sex with men. We denote the proportion of all men who are in these subgroups by ρ_G, ρ_N, and ρ_P respectively. For each group $g\in (G,N,P)$, we aim to estimate simultaneously these subgroup proportions ρ_g, prevalence of HIV in this group π_g and the proportion of infections that are diagnosed, δ_g. Given these parameters, further important quantities are easily derived: the prevalence of diagnosed (${\pi }_g{\delta }_g={(\pi \delta )}_g$) and undiagnosed (${\pi }_g(1-{\delta }_g)={\overline{(\pi \delta )}}_g$) infection; and the numbers of MSM living with diagnosed (${\mu }_{\mathit{Dg}}={\mu }_{\text{pop}}{\rho }_g{(\pi \delta )}_g$) and undiagnosed (${\mu }_{\mathit{Ug}}={\mu }_{\text{pop}}{\rho }_g{\overline{(\pi \delta )}}_g$) infection, where ${\mu }_{\text{pop}}$ is the number of men (MSM and non-MSM) living in London. Since the prevalence among PMSM is much lower, this subgroup is not examined in detail.

We construct a Bayesian model to link the unknown ${\rho }_g,{\pi }_g,{\delta }_g$ with the available evidence provided by various routinely-collected and survey datasets as well as expert belief. Figure 3 shows a directed acyclic graph representing this model, in the form of Figure 1, distinguishing founder nodes, observed data, and outputs of interest. The following sections explain in detail the quantities and relationships illustrated in Figure 3. All data and estimates refer to the year 2012 (unless indicated) and the Greater London area.

Fig. 3.

Directed acyclic graph for HIV prevalence estimation model.

4.1. Subgroup Membership

The total male population of London, ${\mu }_{\text{pop}}$, is informed by published data $y_{\text{pop}}$ (Office for National Statistics 2012), assumed to be a Poisson count: $y_{\text{pop}}\sim \mathit{Po}({\mu }_{\text{pop}})$. A log-normal prior for ${\mu }_{\text{pop}}$ is assumed, $\text{log}\hspace{0.17em}({\mu }_{\text{pop}})\sim N(0,{1000}^2)$. The number of people in each group g is estimated as $r_g={\rho }_g{\mu }_{\text{pop}}$. Estimates of the subgroup proportions ρ_g are informed by data from the National Survey of Sexual Attitudes and Lifestyles (Mercer et al. 2013): y_G = 7, $y_N=38,y_P=10$, out of $y_{\text{NAT}}=824$ men, which we assume to come from a multinomial distribution with probabilities ${\rho }_G,{\rho }_N,{\rho }_P$ given a uniform Dirichlet prior. Thus, the expected number of people with HIV (diagnosed or undiagnosed) in group g is ${\mu }_g={\pi }_g{r}_g$.

4.2. Registry of Diagnosed Infections and Diagnosed Prevalence

Individuals diagnosed with HIV and accessing care in the UK are reported to the HIV and AIDS Reporting System (Kirwan et al. 2016). From the 2012 version of this dataset, known as SOPHID (Surveillance of Prevalent HIV Infections Diagnosed), we obtain the reported number of HIV diagnoses for MSM, $y_M\sim \mathit{Po}({\mu }_M)$, with y_M = 8390. A reporting bias of unknown direction is assumed, through $\text{log}\hspace{0.17em}({\mu }_M)=a_S+\hspace{0.17em}\text{log}\hspace{0.17em}({\mu }_D)$ where $\text{exp}\hspace{0.17em}(a_S)\sim N(1,{0.018}^2)$, giving a prior 90% interval of about $(-3\%,3\%)$ for the adjustment to the number of MSM HIV diagnoses μ_M. After adjustment, ${\mu }_D={\mu }_{\mathit{DG}}+{\mu }_{\mathit{DN}}+{\mu }_{\mathit{DP}}$ is the expected number of diagnoses among MSM, summed from the expected numbers of diagnoses among GMSM, NGMSM, and PMSM, respectively. The following sections explain where ${\mu }_{\mathit{DG}},{\mu }_{\mathit{DN}}$ come from; μ_DP is modeled using similar techniques.

Since SOPHID did not record GUM clinic attendance, to strengthen the evidence on diagnosed prevalence in GMSM, we include data from the HIV and AIDS New Diagnoses and Deaths Database (HANDD) (Kirwan et al. 2016), recording how many of the y_M prevalent diagnosed MSM were newly diagnosed in 2012 and reported to have been diagnosed initially in a GUM clinic. These new diagnoses, y_H = 630, are modeled as $y_H\sim \text{Bin}(y_M,p_H)$, where p_H is assumed to be a lower bound for the proportion of prevalent diagnosed MSM who have attended a GUM clinic in 2012. This bound is expressed through $p_H=a_H{\mu }_{\mathit{DG}}/{\mu }_D$, where $a_H\sim U(0,1)$ is the unknown probability that a prevalent diagnosed MSM who has attended a GUM clinic in 2012 was newly diagnosed that year. y_H therefore gives us additional indirect information on μ_DG, the number of prevalent diagnosed GMSM.

The number of diagnosed infections is related to the total number of infections in each group g as ${\mu }_{\mathit{Dg}}={\delta }_g{\mu }_g$. The proportion of infections that are diagnosed δ_g is not known, but given our inferences about the undiagnosed prevalence ${\overline{(\pi \delta )}}_g={\pi }_g(1-{\delta }_g)$ (explained in the subsequent sections), we can exploit the implicit constraint $1-{\delta }_g>{\overline{(\pi \delta )}}_g$. Therefore, we define ${\delta }_g=a_{\delta g}(1-{\overline{(\pi \delta )}}_g)$, with $a_{\delta g}\sim U(0,1)$, and the diagnosed prevalence ${(\pi \delta )}_g={\pi }_g{\delta }_g$ in each group follows.

4.3. Undiagnosed Prevalence Among GMSM

Information about undiagnosed infections in GMSM is obtained from GUMCAD (Genitourinary Medicine Clinic Activity Dataset) (Kirwan et al. 2016) a registry of attendance episodes in GUM clinics. HIV tests are offered routinely to previously undiagnosed patients. Thus, we have a sequence of observations g_i, representing firstly the number of GUM clinic visitors ($g_1=35,121$) and then the number of patients with no previous HIV diagnosis ($g_2=34,187$), HIV tests offered ($g_3=30,570$), HIV tests accepted ($g_4=29,529$), and HIV diagnoses made ($g_5=855$). For $i=2,\dots ,5,g_i\sim \text{Bin}(g_{i-1},{\gamma }_{i-1})$, with priors ${\gamma }_1,{\gamma }_2,{\gamma }_3\sim U(0,1)$ and ${\gamma }_4\sim U(0,0.15)$ (see below). An HIV infection may therefore remain undiagnosed if either a test is not offered or the patient opts out of testing. We can then decompose the prevalence of undiagnosed infection ${\overline{(\pi \delta )}}_G$ into “unoffered” ${\pi }^{(\text{UN})}$ and “opt-out” ${\pi }^{(\text{OP})}$ components.

$${\overline{\left(\pi \delta \right)}}_G={\pi }^{\left(\text{UN}\right)}+{\pi }^{\left(\text{OP}\right)}.$$ (11)

Both of those require strong prior assumptions to estimate, which will later be relaxed in a sensitivity analysis (Section 4.5). First, the prevalence of infection that remains undiagnosed due to an unoffered test is

$${\pi }^{\left(\text{UN}\right)}={\gamma }_1(1-{\gamma }_2)p^{\left(\text{UN}\right)},$$

where ${\gamma }_1(1-{\gamma }_2)$ is the proportion of clinic attenders that are undiagnosed but not offered a test, and $p^{(\text{UN})}$ is the probability that a test would be positive for these people. We assume the prevalence in this group is between 0.5 and 1.5 times the prevalence in people actually tested, and $\text{logit}(p^{(\text{UN})})=\text{logit}({\gamma }_4)+a^{(\text{UN})}$, with $a^{(\text{UN})}\sim U(\hspace{0.17em}\text{log}\hspace{0.17em}(0.5),\hspace{0.17em}\text{log}\hspace{0.17em}(1.5))$

Secondly, the prevalence of infection remaining undiagnosed due to refusing a test is

$${\pi }^{\left(\text{OP}\right)}={\gamma }_1{\gamma }_2(1-{\gamma }_3)({\gamma }_4+a^{\left(\text{EX}\right)})$$

where ${\gamma }_1{\gamma }_2(1-{\gamma }_3)$ is the proportion of clinic attenders that are undiagnosed and offered a test but opt out. We assume this group has an underlying HIV prevalence higher than those given tests, but not more than 15%, so that the excess prevalence in this group is $a^{(\text{EX})}=a^{(\text{OP})}(0.15-{\gamma }_4)$, where $a^{(\text{OP})}\sim U(0,1)$, and the prior on γ₄ is truncated above at 0.15.

A small amount of additional evidence on ${\overline{(\pi \delta )}}_G$ is available from another dataset, GUM Anon (Public Health England, London 2012), a convenience survey of men not previously diagnosed with HIV who had attended a GUM clinic in the previous year. This gives direct information about the prevalence of HIV among previously undiagnosed GMSM,

$${\pi }^{\left(\text{GA}\right)}=({\overline{\left(\pi \delta \right)}}_G+{\pi }^{\left(\text{GD}\right)})/{\gamma }_1,$$ (12)

where ${\pi }^{(\text{GD})}={\prod }_1^4{\gamma }_r$ is the prevalence of newly diagnosed infection among clinic attenders. The data in GUM Anon are $g^{(A)}\sim \text{Bin}(g^{(\text{AN})},{\pi }^{(\text{GA})})$, where $g^{(A)}=4$ and $g^{(\text{AN})}=85$.

4.4. Undiagnosed Prevalence Among NGMSM

To inform undiagnosed HIV prevalence in NGMSM, we use data from the Gay Men’s Sexual Health Survey (GMSHS) (Aghaizu et al. 2016), based on face-to-face interviews in selected venues where participants were offered anonymous HIV tests. While this group is likely to have a higher HIV prevalence than the general population of MSM, it is assumed that the relative odds of having HIV between NGMSM and GMSM is the same as in the general population. The GMSHS data provide the numbers $y_g^{(\text{GM})}$ out of $n_g^{(\text{GM})}$ previously undiagnosed people in group g who tested positive for HIV (20 out of 493 GMSM and 20 out of 452 NGMSM) so that $y_g^{(\text{GM})}\sim \text{Bin}(n_g^{(\text{GM})},p_g^{(\text{GM})})$, with $p_g^{(\text{GM})}\sim U(0,1)$. Defining the odds $o(p)=p/(1-p)$, we apply the resulting odds ratio $o{r}^{(\text{GM})}=o(p_N^{(\text{GM})})/o(p_G^{(\text{GM})})$ to the baseline estimated from GUMCAD (Section 4.3), giving $o({\overline{(\pi \delta )}}_N)$ = $o({\overline{(\pi \delta )}}_G)o{r}^{(\text{GM})}$.

4.5. Alternative Assumptions

The results presented in Section 5 are for the above model assumptions, unless specified otherwise. Two alternative assumptions are also explored.

1. Undiagnosed prevalence from GUM Anon only

   To avoid the strong prior assumptions on prevalence among those not offered a test or refusing a test, which are necessary to use the GUMCAD data to infer ${\overline{(\pi \delta )}}_g$, we could infer ${\overline{(\pi \delta )}}_g$ from GUM Anon alone. To construct this model, we replace Equation (11) by a U(0, 1) prior on ${\overline{(\pi \delta )}}_g$, although the GUMCAD data are still used to estimate the parameters ${\pi }^{(\mathit{GD})}$ and γ₁ relating the prevalence in GUM Anon to ${\overline{(\pi \delta )}}_g$.

2. GUMCAD also informs diagnosed prevalence

   Instead of being inferred indirectly through the graph, the diagnosed prevalence can be modeled directly as

   $${(\pi \delta )}_G=(1-{\gamma }_1)+{\gamma }_1{\gamma }_2{\gamma }_3{\gamma }_4,$$ (13)

where $1-{\gamma }_1$ is the probability of a previous diagnosis, and ${\gamma }_1{\gamma }_2{\gamma }_3{\gamma }_4$ is the probability of a new diagnosis, in GUMCAD. This is not done in the base case due to concerns about inconsistencies in reporting between GUMCAD and SOPHID/HANDD.

5. VoI Results in the HIV Model

The model outputs of interest (as in Figures 1, 3) are $\mathit{\alpha }=({(\pi \delta )}_G,{(\pi \delta )}_N,{\overline{(\pi \delta )}}_G,{\overline{(\pi \delta )}}_N,{\mu }_{\text{DG}},{\mu }_{\text{DN}},{\mu }_{\text{UG}},{\mu }_{\text{UN}},\mu$), the diagnosed and undiagnosed prevalences among both GMSM and NGMSM, and the corresponding absolute numbers of people living with HIV (or “case-counts”), and the total number of MSM with HIV $\mu ={\mu }_{\text{DG}}+{\mu }_{\text{DN}}+{\mu }_{\text{UG}}+{\mu }_{\text{UN}}$. Samples from the posterior are generated using Hamiltonian Monte Carlo methods in the Stan software (Stan Development Team 2016). These are illustrated in Figure 4 along with the overall prevalence ${\pi }_g={(\pi \delta )}_g+{\overline{(\pi \delta )}}_g$ in each group g, and each of these quantities summed over the two groups g. The estimates of diagnosed prevalence in all MSM (top panel) are reasonably precise, while the corresponding estimates for NGMSM and GMSM are more uncertain. Estimates of undiagnosed prevalence are lower and more precise. Full results under the two alternative assumptions are presented in the supplementary figures.

Fig. 4.

Posterior distributions of HIV prevalence (top) and numbers of MSM living with HIV/AIDS (bottom), London 2012. Darkness within each strip proportional to posterior density, with 95% credible intervals indicated.

5.1. Partial Perfect Information (EVPPI) for Single Outputs

Defining the decision problem as point estimation of $\mathit{\alpha }$ with quadratic loss, we use EVPPI formula (5) to determine which parameters $\varphi$ contribute most to the uncertainty about each component of $\mathit{\alpha }$, thus which $\varphi$ may be worth learning more precisely. We will take $\varphi$ to include the founder nodes of the graph illustrated in Figure 3. Since they are related to the $\mathit{\alpha }$ through a network of deterministic functions, perfect knowledge of these implies perfect knowledge of $\mathit{\alpha }$. Each of the $\varphi$ are either directly informed by data or given a substantive prior distribution based on belief. In the former case, EVPPI measures the maximum potential value of collecting more data from the same source. In the latter case, it will not necessarily be feasible to collect data to improve the precision of the belief, but EVPPI is still useful as a measure of how much of the uncertainty in $\mathit{\alpha }$ is explained by the uncertainty in the parameter.

The results are presented in Figure 5 as a grid whose r, s entry is colored according to ${\text{EVPPI}}_{{\alpha }_s}({\varphi }_r)/\text{var}({\alpha }_s)$, the proportion of variance in α_s which would be reduced if we learnt ${\varphi }_r$. The lighter cells correspond to ${\varphi }_r$ with greater EVPPI. Standard errors in these and all following EVPPI and EVSI estimates, arising from uncertainty in the coefficients of the regression (10), were negligible, at less than 1% of the EVPPI or EVSI estimates.

Fig. 5.

Expected value of partial perfect information in the HIV prevalence model.

The parameters $a_{\delta G}$ and $a_{\delta N}$, governing the proportions of HIV infections that are diagnosed in each of the two groups, and the probability a_H that a GMSM is newly diagnosed in a GUM clinic, explain most of the uncertainty in the diagnosed prevalences ${(\pi \delta )}_G,{(\pi \delta )}_N$ and the corresponding numbers of people diagnosed ${\mu }_{\text{DG}},{\mu }_{\text{DN}}$. Direct data on any of these parameters would be difficult to obtain. However, if we were willing to make the assumption in Equation (13), the estimates of diagnosed prevalence would become more precise, for example the posterior median (SD) of ${(\pi \delta )}_G$ would change from 0.06 (0.13) to 0.051 (0.001), though the extent of uncertainty around ${(\pi \delta )}_N,{\mu }_{\mathit{DN}}$ would not change substantively.

For the undiagnosed prevalences ${\overline{(\pi \delta )}}_G,{\overline{(\pi \delta )}}_N$ and undiagnosed case count ${\mu }_{\text{UG}}$, Figure 5 shows that more GUM-Anon data (via ${\pi }^{(\mathrm{G}A)}$), more GMSHS data (via $o{r}^{(\mathrm{G}M)}$) and more NATSAL data (via ${\rho }_{\text{UG}}$), respectively, would give the greatest uncertainty reductions. These outcomes, however, are already precisely estimated in absolute terms (Figure 4). The number of NGMSM ${\mu }_{\text{UN}}$ with undiagnosed HIV is more uncertain, with 95% CrI (279,1442), and more GMSHS data would be potentially valuable to reduce this uncertainty.

If ${\overline{(\pi \delta )}}_G$ were informed only from the 4 infections out of 85 people observed in GUM Anon (alternative assumption (a)), the estimates of undiagnosed prevalence or case counts become extremely uncertain, for example, $\text{var}({\mu }_{\text{UN}})$ increases from 302² to 2872². We could reduce this uncertainty by collecting more GUM Anon data—since ${\text{EVPPI}}_{{\mu }_{\text{UN}}}({\pi }^{(\text{GA})})$ is $p=62\%$ of $\text{var}({\mu }_{\text{UN}})$, more GUM Anon data could reduce $\text{var}({\mu }_{\text{UN}})$ to a minimum of ${2872}^2(1-p)={1777}^2$ (note that the square root of the expected variance after learning data is not the same as the expected standard deviation).

5.2. Partial Perfect Information for Multiple Outputs

Staying with alternative assumption (a), suppose we wish to calculate the maximum potential value of extra GUM Anon data for jointly reducing the uncertainty about the number of GMSM, NGMSM, and PMSM with undiagnosed HIV, so that $\mathit{\alpha }$ is the vector $({\mu }_{\text{UG}},{\mu }_{\text{UN}},{\mu }_{\text{UP}})$. As described in Section 2.3, we could simply calculate the standard EVPPI based on a scalar output α redefined as their sum, ${\mu }_U={\mu }_{\text{UG}}+{\mu }_{\text{UN}}+{\mu }_{\text{UP}}$, the total number of MSM with undiagnosed HIV, whose posterior median is 5149 (SD 3280). This would ensure that any data expected to reduce the variance of any of these three outputs by the same (additive) amount would be valued equally. From this, we find that extra GUM Anon data would be expected to reduce $\text{var}({\mu }_U)$ from 3280² to a minimum of 1801². Since μ_U is dominated by NGMSM (posterior median of ${\mu }_{\text{UN}}$ is 4185), this is mostly explained by an expected reduction in $\text{var}({\mu }_{\text{UN}})$ from 2864² to a minimum of 1770².

Alternatively, suppose both the prevalences and the case counts are of interest, for example in NGMSM, so that $\mathit{\alpha }=({\overline{(\pi \delta )}}_N,{\mu }_{\text{UN}})$. Since these two components are on very different scales, the Bayesian “D-optimality” criterion $v(\mathit{\alpha })=\det (\text{cov}(\mathit{\alpha }))$ would be a preferable measure of overall expected loss due to uncertainty. We use this criterion to compare the maximum expected value of extra GUM Anon data and extra GMSHS data, which combine to estimate the outcomes for NGMSM as described in Section 4.4. The EVPPI is interpreted as the expected reduction in the product of $\text{var}({\overline{(\pi \delta )}}_N)$ and $\text{var}({\mu }_{\text{UN}})$ given by extra GUM Anon or GMSHS data, adjusted for their covariance. This is 425 and 135, respectively, favoring extra data from GUM Anon. Though in this example, examining expected reductions in $\text{var}({\overline{(\pi \delta )}}_N)$ or $\text{var}({\mu }_{\text{UN}})$ separately would lead to the same conclusion, since ${\overline{(\pi \delta )}}_N$ is defined as the proportion ${\mu }_{\text{UN}}/r_N$ of NGMSM with HIV, and GUM Anon and GMSHS are not informative about the number r_N of NGMSM, thus extra data informs ${\mu }_{\text{UN}}$ entirely through information on ${\overline{(\pi \delta )}}_N$ (or vice versa).

5.3. Sample Information (EVSI)

We now estimate the expected value of data with specific sample sizes for improving the precision of the estimated number of people μ_U with undiagnosed HIV. Using the GUMCAD data and associated strong prior assumptions, the posterior median of μ_U is 804 (SD 320), compared to 5149 (SD 3280) with this information excluded (a). We compare the value of additional data from GUM Anon and additional data from GMSHS (on top of their original sample sizes of 85 and 945, respectively) for reducing these posterior standard deviations.

The EVSI is computed for a series of sample sizes n using the method in Section 3.2. For GUM Anon (Section 4.3), the sufficient statistic $T(\mathbf{y})$ consists of the empirical HIV prevalence $\mathbf{y}/n$ from an additional survey $\mathbf{y}\sim \text{Bin}(n,{\pi }^{(\text{GA})})$. For GMSHS (Section 4.4), given a sample size n, $\mathbf{y}=(N_G^{(\mathrm{G}M)},Y_G^{(\mathrm{G}M)},Y_N^{(\mathrm{G}M)})$, where $N_G^{(\text{GM})}$ is the number of previously undiagnosed MSM in the future sample of n who attend GUM clinics (the equivalent of the observed $n_G^{(\text{GM})}=493$). Then $Y_G^{(\text{GM})}$ and $Y_N^{(\text{GM})}$ are the numbers of men out of denominators $N_G^{(\text{GM})}$ and $N_N^{(\text{GM})}=n-N_G^{(\text{GM})}$ (GMSM and NGMSM, respectively) who test positive for HIV, the equivalents of the observed $y_G^{(\mathrm{G}M)}=20,y_N^{(\mathrm{G}M)}=492$. We take $T(\mathbf{y})=o({\widehat{p}}_N^{(\mathrm{G}M)}(\mathbf{y}))/o({\widehat{p}}_G^{(\mathrm{G}M)}(\mathbf{y}))$, a point estimator of the odds ratio, where ${\widehat{p}}_G^{(\mathrm{G}M)}(\mathbf{y})$ is an estimator of the proportion of MSM in group g who have HIV. To avoid zeros in the denominator $o({\widehat{p}}_G^{(\text{GM})}(\mathbf{y}))$, we use a Bayesian estimator ${\widehat{p}}_G^{(\text{GM})}(\mathbf{y})=(Y_G^{(\text{GM})}+0.5)/(N_G^{(\text{GM})}+1)$, the posterior mean of a binomial proportion under a Jeffreys Beta(0.5,0.5) prior, rather than the empirical proportion $Y_G^{(\text{GM})}/N_G^{(\text{GM})}$.

Figure 6 shows $\text{var}({\mu }_U)-\mathit{EVSI}(\mathbf{y})$, the expected variance remaining after data collection, under the two alternative assumptions. With the strong priors, μ_U is relatively well informed, and extra data from GUM Anon at realistic sample sizes (1000 or less) would not noticeably reduce $\text{var}({\mu }_U)$. GMSHS data would be more valuable, through improving the estimate of ${\mu }_{\text{UN}}$, the more uncertain contributor to ${\mu }_U={\mu }_{\text{UG}}+{\mu }_{\text{UN}}$. 1000 extra observations from GMSHS would be expected to reduce $\text{var}({\mu }_U)$ from 320² to 279².

Fig. 6.

EVSI: value of additional data from GUM Anon or GMSHS for reducing the variance of the total number of MSM with undiagnosed HIV, ${\mu }_U={\mu }_{\text{UG}}+{\mu }_{\text{UN}}$. The x-axis is on the log scale. The y-axis is the variance, with the labels as SD².

Without the strong prior information, $\text{var}({\mu }_U)={3280}^2$ is substantially greater, and μ_U is only directly informed by the 85 observations from GUM Anon. Extra data from this source would be valuable, for example, another 500 observations would be expected to reduce this variance to 2184². Relative to these improvements, GMSHS data of the same size would be much less valuable. GMSHS data, however, would be expected to give around the same absolute reductions in $\text{var}({\mu }_U)$, whether or not the strong priors are included.

5.4. Net Benefit of Sampling

The benefits from improved precision of estimates of μ_U must be traded off with the costs of data collection, to determine an optimal sample size for extra survey data. Consider the scenario which excluded the GUMCAD data and associated strong priors. In the GUM Anon survey, there was a cost of around £17 per participant, which is assumed to be the same for collecting further data from this source. The cost $c(\mathbf{y})$ is illustrated against sample sizes of $\mathbf{y}$ from 1 to 400 by the straight line in Figure 7. Suppose also that the decision-maker is willing to pay £5000 to reduce the variance of μ_U by $\mathit{dv}={3271}^2-{2771}^2$, which in this case would reduce the standard deviation by 500, from 3271 to 2771. The willingness to pay per unit variance is then $\lambda =5000/\mathit{dv}$.

Fig. 7.

Expected cost, benefit, and net benefit of sampling up to 400 extra participants from GUM Anon, if we wish to reduce the variance of μ_U, the number of MSM with undiagnosed HIV. The optimal sample size is illustrated as a dotted line.

Collecting extra data $\mathbf{y}$ will give an expected reduction in $\text{var}({\mu }_U)$ of $\text{EVSI}(\mathbf{y})$, as illustrated in Figure 7. The resulting expected (monetary) benefit $E_{\mathbf{y}}(b(\mathbf{y}))$ (Section 2.4) is shown to be a nonlinear function of the sample size of $\mathbf{y}$, with an asymptote representing the expected value of partial perfect information on ${\pi }^{(\text{GA})}$. Hence, the expected net benefit of sampling $E_{\mathbf{y}}(b(\mathbf{y})-c(\mathbf{y}))$ is illustrated in the bottom panel of Figure 7. The expected benefits of sampling always exceed the costs, and the net benefit is maximized at a sample size of 166. Also illustrated are the benefit and net benefit that would result if the decision maker was willing to pay twice or half the original amount, $2\lambda$ or $\lambda /2$. The corresponding optimal sample sizes would be 315 or 81, respectively.

6. Summary and Potential Further Work

We have presented tools to find the most influential sources of uncertainty in MPES models and determine the expected value of extra data. We generalized methods, previously applied only in deterministic models, to complex graphical models, a class which also includes hierarchical models. We have shown how VoI methods developed for formal finite-choice decision problems can be extended to deal with estimation of single or multiple quantities.

While the purpose of our model was to estimate a quantity of interest to policy-makers, the same methods could be used for models to compare specific health policies. Sections 2.4 and 5.4 illustrated how the benefits from more precise estimates of HIV prevalence could be converted directly into a monetary value. An alternative approach would be to value the indirect health gains that would result from better data, through better-informed health policies. This would allow standard health economic principles to be used (see, e.g., Briggs, Sculpher, and Claxton 2006). For example, the National Institute for Health and Care Excellence in the UK recommends that a new health-care intervention is funded by the National Health Service if the cost per quality-adjusted life year (QALY) gained, compared to current practice, is less than around $\lambda =$£20,000, implying a willingness to pay of λ per QALY. This is a choice between two actions $d\in$ “$\{ \text{accept}, \text{reject} \}$ intervention” (as in Section 2.3) with loss $L(d,\theta )=c_d(\theta )-\lambda q_d(\theta )$, where $q_d(\theta )$ is the expected QALY and $c_d(\theta )$ is the expected cost for a person under action d, from a health economic model with parameters θ. See, for example, Carmona, O’Rourke, and Robinson (2016); Baggaley et al. (2017), for how such models might be built for HIV testing interventions to increase the proportion δ of people who are diagnosed. Briefly, any QALY gains strongly depend on the underlying prevalence of HIV among the population receiving the intervention. Thus, improved estimates of prevalence will lead to more precise estimates of the QALY gains, and a better-informed decision about whether to implement the intervention, which may result in a better use of health service resources. VoI methods may then be used to decide whether further information should be collected to support the decision.

In the HIV application, we found that structural assumptions, such as whether to include a particular piece of information, were influential to both the parameter estimates and the VoI. Such uncertainties might be parameterized (see, e.g., Strong, Oakley, and Chilcott 2012), for example a particular prior or dataset of uncertain relevance could be discounted using an unknown weight (e.g., Neuenschwander, Branson, and Spiegelhalter 2009). The EVPPI of the extra parameter would then quantify this uncertainty in the context of all other uncertainties, referred to as the “expected value of model improvement” by Strong and Oakley (2014).

Note that VoI refers to the expected value of potential future information, which differs from the observed value of a dataset x_i currently included in the model. The latter could be computed as the observed reduction in loss when the model is refitted without x_i. This could demonstrate the value of past data to the policymaker responsible for funding the collection of future data of the same type. For surveys or longitudinal studies conducted at regular intervals, VoI might be used to determine the expected value of future surveys or follow-up, although a full analysis would require modeling the expected changes through time in the quantities, such as disease prevalence or incidence, informed by the data.

While our method is broadly applicable, the details of computation for different decision problems and loss functions may be different. We discussed finite-action decisions and point estimation. A more general decision problem is to estimate the entire uncertainty distribution of $\mathit{\theta }$. The standard posterior $p(\mathit{\theta }|\mathbf{y})$ is then optimal under a log scoring rule (Bernardo and Smith 1994), and (following Lindley 1956) standard Bayesian design theory aims to maximize the information gain from new data $\mathbf{y}$, which we can write as $\text{EVSI}(\mathbf{y})=E_{\mathit{\theta }}(-\hspace{0.17em}\text{log}\hspace{0.17em}(p(\mathit{\theta })))+E_{\mathbf{y}}{E}_{\theta |\mathbf{y}}\{\hspace{0.17em}\text{log}\hspace{0.17em}(p(\mathit{\theta }|\mathbf{y}))$. Under linear models (Chaloner and Verdinelli 1995), this is equivalent to minimising $\det (\text{cov}(\mathit{\theta }))$, but more generally this is challenging to compute (Ryan et al. 2016).

Note that the VoI approach to sensitivity analysis is an example of the “global” approach, which examines the changes in model outputs given by varying parameters within the ranges of their belief distributions. The “local” approach is based on examining the posterior geometry resulting from small parameter perturbations around a base case, for example, Roos et al. (2015) assess the robustness of hierarchical models to prior assumptions in this way. While the global approach is easier to interpret, as discussed by Oakley and O’Hagan (2004) and Roos et al. (2015), it conditions on one particular prior specification, and parameterising all potential prior beliefs or structural assumptions would be impractical.

The regression method for VoI computation that we described requires only a MCMC sample from the joint distribution of parameters of interest $\mathit{\varphi }$ and outputs $\mathit{\alpha }$. Additionally for EVSI it requires that the information in the new data $\mathbf{y}$ can be condensed into an analytic sufficient statistic $T(\mathbf{y})$. Alternative methods which exploit particular analytic structures of $g()$, where α is a known function $g(\mathit{\varphi })$, thus avoiding a regression approximation, were discussed by Madan et al. (2014) for EVPPI and Ades et al. (2004) for EVSI. Menzies (2016) also presented an importance resampling method for EVSI computation which needs only a single MCMC sample and not a sufficient statistic.

In conclusion, the consideration of future evidence requirements is an often-neglected part of statistical analysis. The VoI methods we have presented provide a practicable set of tools for achieving this aim in the context of Bayesian evidence synthesis.

Supplementary Materials

A supplementary document provides estimates of HIV prevalence and expected value of partial perfect information under the alternative assumptions described in Section 4.5.

Funding Statement

This work was funded by the Medical Research Council, grant code U105260566, and from Public Health England (funding DDA and SC).

Acknowledgments

The authors are grateful to the HIV department of Public Health England for providing the data and permission to use the example, to the NATSAL team for providing data, and Louise Logan for advice on costs of data collection in GUM Anon.