Cross-sectional HIV Incidence Estimation in an Evolving Epidemic

SUMMARY

The cross-sectional approach to HIV incidence estimation overcomes some of the challenges with longitudinal cohort studies and has been successfully applied in many settings around the world. However, the cross-sectional approach does rely on an initial training data set to develop and calibrate the statistical methods to be used in cross-sectional surveys. The problem addressed in this paper is that the initial training data set may over time not reflect the current target population of interest because of evolution of the epidemic. For example, the mismatch between the target population and the initial data set could occur because of increasing use of anti-retroviral therapy (ART) among HIV infected persons throughout the world. We developed methods to adjust the initial training data set with the goal that the adjusted data sets better reflect the target population. These adjustment procedures could help avoid the time and expense of collecting a completely new training data set from the current target population. We report the results of a simulation study to evaluate the procedures. We applied the methods to a dataset of HIV subtype B infection. The adjustment procedures could be applicable in situations other than cross-sectional incidence estimation where complex statistical analyses are to be conducted using an initial data set but those results may not be directly transportable to a new target population of interest. The approach we have proposed could offer a practical and cost-effective way to apply cross-sectional incidence methods to new target populations as the epidemic evolves.

1. Introduction

Accurate and timely estimates of HIV incidence are critical for tracking the spread of the HIV/AIDS epidemic and for evaluating interventions.¹ The traditional approach to estimating incidence rates is based on longitudinal follow-up of cohorts. However, there are important disadvantages of the cohort approach for HIV incidence estimation. Cohort studies are expensive to conduct because of the need to follow large numbers of subjects for an extended time, and follow-up rates may be low especially among marginalized populations at risk for HIV infection. The incidence results from cohort studies may not be available for years and thus may not provide timely information for assessing the current growth rate of the epidemic. Cohort studies may suffer from selective attrition especially in populations at risk for HIV infection. Selective attrition would arise, for example, if those participants who are more likely to become infected over the course of follow-up are also those more likely to be lost to follow-up.

An alternative approach, called the cross-sectional approach to HIV incidence estimation, overcomes some of the disadvantages of the longitudinal cohort approach. The cross-sectional approach is based on a single representative cross-sectional survey of the target population whose incidence rate we aim to measure.² Survey participants provide biological specimens which are tested for biomarkers. The specific biomarkers chosen are those that have some ability to help distinguish persons who were recently infected from those who have been infected for a long time. The cross-sectional estimator of the HIV incidence rate $\widehat{I}$ is V/μN⁻ where V is the number of infected individuals in the survey whose biomarker measurements indicate the infection occurred recently, N⁻ is the number of individuals who are HIV uninfected, and μ is the “mean window period.”³ The mean window period is the expected duration of time that a newly HIV-infected person would be classified as positive by the biomarkers indicating an infection that occurred “recently.” The mean window period μ depends on the specific biomarkers, as well as the operational definition for classifying an individual as a “recent infection” based on the biomarker values.

A requirement of the cross-sectional method before it can be applied in practice is: (1) the biomarkers have been identified; (2) it has been established how to combine those biomarkers to create an operational definition of “recent infection”; and (3) an estimate of μ based on that biomarker definition of a recent infection is available. These three critical pieces of information are typically determined from an initial training data set of HIV-infected individuals. The key elements of the initial training data set include biomarker measurements on biological specimens from HIV infected persons and their durations of infection, which may well be interval censored. The analysis of the training data set is the key calibration step prior to applying cross-sectional incidence methods in the field. Once the calibration step has been completed, a cross-sectional method involves selecting a random sample of the population, which would include both HIV-uninfected and HIV-infected individuals.

The target population refers to the population whose HIV incidence we desire to estimate. An important complication is that over time the initial data set may eventually not be representative of the target population of interest because of evolution of the epidemic.⁴ The mismatch between the target population and the initial data set could occur for several reasons. First, there has been and continues to be increasing use of anti-retroviral therapy (ART) for HIV infected persons throughout most parts of the world.⁵ Initiation of ART therapy is occurring earlier in the course of infection. ART induces viral suppression, and viral suppression may modify some biomarker levels because antibody titers would tend to decrease, thereby making longstanding infections resemble recent infections.⁶ An initial data set collected years before widespread ART use may no longer be applicable to the current target population. Second, the subtypes of HIV that are circulating in a population may evolve over time.⁷ HIV subtype may affect biomarker levels and their relationship to duration of infection.^8,9 The changing mix of subtypes can create mismatches between the initial data set and the target populations. The statistical analyses that determined the operational definition of recent infection and the mean window period based solely on the initial training data set may not be transportable to the current target population.

One simple solution to address the mismatch between the initial data set and the target population is to collect a new training data set that is representative of the current target population for the statistical analyses. However, that approach is expensive and could take considerable time, obviating the advantages of the cross-sectional approach to HIV incidence estimation. The objective of this paper is to develop methods that address the mismatch between the initial training data set and the target population for cross-sectional HIV incidence estimation.

In this paper, we show how an initial training data set which is no longer representative of the target population because of changes in the characteristics of the epidemic can be adjusted and still utilized for calibrating methods to perform cross sectional HIV incidence estimation in the target population. We develop methods to identify the operational biomarker definition of recent infection for the target population. We show how to calculate the mean window period applicable to the target population.

In section 2 we describe our general approach. In brief, the approach is to obtain multiple adjusted data sets that more closely match the target population than the initial data set, perform the necessary statistical analyses and calibrations on each of them, and then pool the results across the multiply adjusted data sets. We propose two approaches for obtaining “adjusted” data sets to address the mismatch between the initial training data set and the target population: a resampling approach (section 3) and a counterfactual modeling approach (section 4). We discuss the application of the approach to cross-sectional incidence estimation in section 5. We perform a simulation study to evaluate the methods in section 6. The methods are applied to an HIV data set in section 7 and discussed in section 8.

2. Approach

The initial training data set D consists of data on biological specimens from infected individuals. On the i^th biological sample we have measurements on k biomarkers, which we denote by the k-dimensional vector Y_i. We have recorded the duration of infection associated with the i^th biological sample, which we denote as t. In addition, we have a binary variable X whose distribution we are concerned may have changed over time. For example, among infected persons X can indicate if persons are virally suppressed. Alternatively, X could indicate if persons are infected with a particular pathogen subtype. The changes in the distribution of X create a mismatch between the initial data set and the target population. It is possible that the probability that X = 1 depends on other variables which are also recorded in D. The methods developed here can be applied whether or not we have such variables. In this paper we present the methods and notation for the more general situation when we do have such variables. We assume each of these variables takes discrete values and that a discrete-valued variable Z defines these strata. For example, one value of Z might correspond to females who have been infected for less than 2 years. Let P_T(X|Z) and P_I(X|Z) represent the probabilities in stratum Z that X = 1 in the target population and initial data set, respectively. The issues we are addressing in this paper arise when P_I(X|Z) from the target population and P_T(X|Z) from the initial data set are different for one or more values of Z.

We make a number of assumptions. First, we assume that P_T(X|Z) is known. In practice, P_T(X|Z) could sometimes be estimated from simple surveys measuring X and Z in the target population. Such surveys would not need the detailed information on biomarkers as required for the initial training data set. If surveys are not available, it is still useful to perform sensitivity analyses to examine how results would change under various assumed scenarios about P_T(X|Z) in the target population. In this article, we will examine the consequences of error in the assumed values of P_T(X|Z) for our methods.

Second, we assume that the initial training dataset covers the entire domain of values in the target population. This assumption is analogous to the positivity assumption in causal inference settings.¹⁰ For example, if the initial training data set included only samples collected within the first six months after infection occurs, or did not include any virally suppressed samples, but the target population did, then it would not be possible to obtain reliable estimates from either of our adjustment procedures.

Third, we assume that the distribution of biomarker values, conditional on infection duration t, stratum Z, and covariate X, does not change between the training data set and the target population; that is, we assume that P_T(Y|X,Z,t) = P_I(Y|X,Z,t) for all values of X,Z,t. This assumption implies that all of the variables that describe the relevant differences between the initial and target population are identified. It is analogous to the conditional exchangeability and consistency assumptions in causal inference settings.^11,12

Fourth, we assume that P_T(X|Z,t) = P_T(X|Z). This assumption implies that the stratifying variable Z completely captures the dependence of X on t.

Fifth, we assume that the distribution of stratifying variable Z, conditional on infection duration t, does not change between the calibration data set and the target population; that is, we assume that P_T(Z|t) = P_I(Z|t); this is trivially true when Z is a function of time, such as an indicator variable for duration of infection less than two years.

A sixth assumption is made for the counterfactual modeling approach. For that approach we assume that the form of P_T(Y|X,Z,t) is known, including any necessary transformations of the biomarker scales, and the functional form of the linear predictor model. The parametric assumptions are not required for the resampling approach.

Our proposed approach can be summarized in three steps. In step 1, we adjust the initial data set. Two methods are proposed for adjusting the data set and they are described in sections 3 and 4. As discussed in sections 3 and 4, the observations included in the adjusted data sets with either approach have a random component. Thus, we obtain K adjusted data sets called ${\tilde{D}}_1,{\tilde{D}}_2,\dots ,{\tilde{D}}_K$. These K adjusted data sets will be somewhat different from each other due to a random element of how the adjusted data sets are constructed. In step 2, instead of performing the statistical analyses and calibrations on the initial database D, we perform them on each of the adjusted data sets. In step 3, we pool the results across the adjusted data sets. Figure 1 is a schematic illustration of the approach. Suppose μ is the parameter of interest. We obtain estimates ${\tilde{\mu }}_1,{\tilde{\mu }}_2,\dots ,{\tilde{\mu }}_K$ corresponding to each of the K adjusted data sets ${\tilde{D}}_1,{\tilde{D}}_2,\dots ,{\tilde{D}}_K$. In step 3, we pool the results to obtain an adjusted estimate $\widehat{\mu }$. Here we use $\widehat{\mu }=\mathrm{m}\mathrm{e}\mathrm{d}\mathrm{i}\mathrm{a}\mathrm{n}\left({\tilde{\mu }}_1,{\tilde{\mu }}_2,\dots ,{\tilde{\mu }}_K\right)$. In our simulation study (section 6), we consider K = 10.

Figure 1:

Schematic illustration of procedure to multiply adjust an initial training data set D

We use bootstrapping to estimate the standard error of $\widehat{\mu }$.¹³ We resample with replacement from the initial data set D to construct B bootstrapped data sets $D_1^{*},\dots ,D_B^{*}$, each with the same number of observations as the initial data set. From the j^th bootstrapped data set, we obtain K adjusted data sets ${\tilde{D}}_{j1}^{*},{\tilde{D}}_{j2}^{*},\dots {\tilde{D}}_{jK}^{*}$, and from those data sets we obtain K bootstrap adjusted estimates ${\tilde{\mu }}_{j1}^{*},{\tilde{\mu }}_{j2}^{*},\dots {\tilde{\mu }}_{jK}^{*}$. The pooled estimate corresponding to the j^th bootstrap data set is ${\widehat{\mu }}_j^{*}=\mathrm{m}\mathrm{e}\mathrm{d}\mathrm{i}\mathrm{a}\mathrm{n}\left({\tilde{\mu }}_{j1}^{*},{\tilde{\mu }}_{j2}^{*},\dots {\tilde{\mu }}_{jK}^{*}\right)$. Then the bootstrapped standard error of the estimate $\widehat{\mu }$ is $\widehat{SE}\left(\widehat{\mu }\right)=\sqrt{\frac{1}{B-1}\left({\sum }_{j=1}^B{\left({\widehat{\mu }}_j^{*}-\stackrel{-}{{\widehat{\mu }}^{*}}\right)}^2\right)}$ where $\stackrel{-}{{\widehat{\mu }}^{*}}={\sum }_{j=1}^B{\widehat{\mu }}_j^{*}/B$.

3. Resampling Adjustment Approach

The main idea of the resampling approach is to sample observations from the initial training data set D with probability weights based on P_T(X|Z) to create an adjusted training data set that reflects the target population. The sampling is performed at each level of Z. Let n(z) represent the number of biological specimens in D with Z = z. We would expect n(z)P_T(X = 1|Z = z) biological specimens to have the values X = 1 and Z = z if P_T(X|Z) was applicable to the initial data set. Accordingly, we sample with replacement n(z)P_T(X = 1|Z = z) biological specimens (to the nearest integer) from the set of biological specimens in D that have the values X = 1 and Z = z. Similarly, if P_T(X|Z) was applicable to the initial data set, then we would expect n(z)(1 − P_T(X = 1|Z = z)) biological specimens to have the values X = 0 and Z = z. Accordingly, we sample with replacement n(z)(1 − P_T(X = 1|Z = z)) biological specimens (to the nearest integer) from the set of biological specimens in D that have the values X = 0 and Z = z.

For example, suppose there has been an increase in the use of ART resulting in higher levels of viral suppression since the time of the collection of the initial training data set. Specifically, suppose the probability of viral suppression among persons infected one year or less increased from 0.15 in the initial data set to 0.30 in the target population; and among persons infected more than one year, the probability of viral suppression increased from 0.30 in the initial training data set to 0.60 in the target population. In this example, we would have two levels for Z, the first (Z = 1) corresponding to persons infected less than one year, and the second (Z = 2) for persons infected more than one year. Suppose n(1) = 1000 with 150 and 850 of those biological specimens having values X = 1 and X = 0 respectively. Further suppose n(2) = 2000 with 600 and 1400 of those biological specimens having values X = 1 and X = 0 respectively. Then we would resample with replacement: 1000 × 0.3 = 300 specimens from the 150 specimens with Z = 1, X = 1; resample 1000 × 0.7 = 700 specimens from the 850 specimens with Z = 1 and X = 0; resample 2000 × 0.6 = 1200 specimens from the 600 specimens with Z = 2 and X = 1; and, resample 2000 × 0.4 = 800 specimens from the 1400 specimens with Z = 2 and X = 0.

This method is a resampling equivalent of the Horvitz-Thompson estimator from survey sampling.¹⁴ Since it treats the data set as an imbalanced sample from the population of interest, it will have a design effect that inflates the variance relative to a simple random sample of the same size from that population.¹⁵

4. Counterfactual Modeling Adjustment Approach

The main idea of the counterfactual modeling approach is to randomly flip (i.e., change) a subset of the values of X in the initial training data (e.g. X = 1 is flipped to X = 0; alternatively X = 0 is flipped to X = 1) in order to guarantee that the probability that X = 1 in the adjusted data set matches P_T(X|Z). Then, a parametric model is fit using the initial training data set D to predict what the values of the biomarkers would have been if in fact the flipped values of X were the true X values. These biomarker predictions and the associated flipped values of X are substituted into the initial training data set D to create an adjusted training data set $\tilde{D}$. This approach is similar to the potential outcomes framework in the causal inference literature.^16,17

The first step in creating $\tilde{D}$ is to select the biological specimens for which the value of X will be flipped. The selection of which values to flip is based on Bernoulli trials. For observations with values X = 0 in the initial training data set D, the probability that value is flipped to X = 1 in the adjusted training data set is set to

$${\alpha }_{01}\left(z\right)=\text{max}\left\{0,\frac{P_T\left(X=1|Z=z\right)-P_I\left(X=1|Z=z\right)}{1-P_I\left(X=1|Z=z\right)}\right\}$$

For observations with values X = 1 in the initial training data set D, the probability that value is flipped to X = 0 in D is set to

$${\alpha }_{10}\left(z\right)=\text{max}\left\{0,\frac{P_I\left(X=1|Z=z\right)-P_T\left(X=1|Z=z\right)}{P_I\left(X=1|Z=z\right)}\right\}$$

Then, the probability X = 1 in the adjusted training data set is

$$P_I\left(X=1|Z=z\right)\left(1-{\alpha }_{10}\left(z\right)\right)+\left(1-P_I\left(X=1|Z=z\right)\right)\left({\alpha }_{01}\left(z\right)\right)$$

and after substituting the expressions for α₀₁(z) and α₁₀(z) in the above equation, it can be shown that the probability X = 1 in the adjusted training data set is equal to P_T(X = 1|Z = z), the probabilities in the target population.

The second step is to adjust the biomarker levels Y for those biological specimens for which the value of X was flipped. Let ${\tilde{X}}_i$ represent the possibly flipped values of X in $\tilde{D}$. Using the initial training data set D, we propose fitting a parametric model

$$\mathrm{E}\left({\mathbf{Y}}_i\right)=f\left(t_i,x_i,z_i;\mathbf{\beta }\right)$$

with multivariate normal error where β are regression coefficients and t_i is the duration of infection associated with the i^th biological specimen. Let ${\widehat{\mathbf{Y}}}_i\left(x\right)$ be the predicted value of Y_i with X = x holding t_i and z_i fixed at their values as given in D. If the value of X for the i^th biological specimen was flipped, we change the value of Y_i to the value ${\tilde{\mathbf{Y}}}_i$ in the adjusted training data set where:

$${\tilde{\mathbf{Y}}}_i={\mathbf{Y}}_i+\left({\widehat{\mathbf{Y}}}_i\left(1\right)-{\widehat{\mathbf{Y}}}_i\left(0\right)\right)\mathrm{ }\mathrm{i}\mathrm{f}\mathrm{ }{X}_i=0\mathrm{ }\mathrm{a}\mathrm{n}\mathrm{d}\mathrm{ }{\tilde{X}}_i=1$$

$${\tilde{\mathbf{Y}}}_i={\mathbf{Y}}_i+\left({\widehat{\mathbf{Y}}}_i\left(0\right)-{\widehat{\mathbf{Y}}}_i\left(1\right)\right)\mathrm{ }\mathrm{i}\mathrm{f}\mathrm{ }{X}_i=1\mathrm{ }\mathrm{a}\mathrm{n}\mathrm{d}\mathrm{ }{\tilde{X}}_i=0$$

For example, consider a simple multivariate normal linear regression model of the form

$$\mathrm{E}\left[{\mathbf{Y}}_i\right]={\mathbf{\beta }}_0+{\mathbf{\beta }}_1{t}_i+{\mathbf{\beta }}_2{x}_i$$

Then, the adjusted biomarker values ${\tilde{\mathbf{Y}}}_i$ for those biological specimens for which the x values were flipped are:

$${\tilde{\mathbf{Y}}}_i={\mathbf{Y}}_i+{\mathbf{\beta }}_2\mathrm{ }\mathrm{i}\mathrm{f}\mathrm{ }{X}_i=0\mathrm{ }\mathrm{a}\mathrm{n}\mathrm{d}\mathrm{ }{\tilde{X}}_i=1$$

$${\tilde{\mathbf{Y}}}_i={\mathbf{Y}}_i-{\mathbf{\beta }}_2\mathrm{ }\mathrm{i}\mathrm{f}\mathrm{ }{X}_i=1\mathrm{ }\mathrm{a}\mathrm{n}\mathrm{d}\mathrm{ }{\tilde{X}}_i=0$$

If the value of X for the i^th biological specimen was not flipped, the value of ${\tilde{\mathbf{Y}}}_i$ in the adjusted training data set $\tilde{D}$ remains the same as Y_i in the initial training data set D.

5. Considerations in Applying Adjustment Procedures for Cross-sectional Incidence Estimation

The cross-sectional approach to incidence estimation requires operational definitions of recent infection. These definitions have been based on hierarchical step-wise testing algorithms using biomarkers. Hierarchical step-wise testing algorithms, also termed multi-assay algorithms (MAAs), have been studied as an approach for combining values from multiple biomarkers to classify observations as recent infections or not.¹⁸ At each step of the algorithm, testing is performed with another assay on the subset of specimens provisionally classified as a recent infection at the prior step. Such an approach is generally less costly than other approaches because not all specimens are tested with all assays and because less expensive, high-throughput assays could be performed in the initial steps and more expensive or labor-intensive assays performed in the later steps. Multi-assay algorithms have been successfully developed and applied for both subtype B and subtype C epidemics.^19–21 These methods and applications have all assumed that there is no mismatch between the training data set and the target population. We are able to address such mismatches using the methods described in sections 2–4. Once the algorithm and mean window period have been established, cross-sectional incidence estimation can be conducted in the target population with confidence intervals.^22,23

Statistical methods have been developed to identify multi-assay algorithms that have certain optimality criteria; for example, one criterion is that the incidence estimator should have minimal standard error, subject to the constraint that it is estimating incidence within one year of the cross-sectional survey.¹⁸ These methods require estimation of several quantities including the mean window period and a quantity termed the shadow that measures how far back in time incidence is being estimated.^24,25 Here we briefly explain how the mean window period is estimated in adjusted data sets.¹⁸ The MAA is applied to each specimen in the data set, producing a vector W of binary classifications (W_i = 1 if all biomarker values for specimen i indicate recent infection, W_i = 0 otherwise). Then, the probability Pr(W_i = 1) is estimated as a function of time $\widehat{\varphi }\left(t\right)$ by using a logistic regression model with time since infection t_i as the covariate. The estimate of the mean window period is $\widehat{\mu }={\int }_{t=0}^{T^{*}}\widehat{\varphi }\left(t\right)dt$, where T* is a time point beyond which the probability of a positive MAA result is essentially zero. One can then search through multi-assay algorithms by changing discrete values of biomarkers cutoffs to find the algorithm that meets the optimality criteria.

A number of statistical complexities arise in these analyses. For example, the duration of infection may be interval censored. Multiple biological specimens may come from the same individual at different times. Methods have been developed for dealing with these issues in the context of cross sectional incidence estimation.^18,26 An advantage of the multiple adjustment procedures proposed here, is that whichever statistical methods are chosen to address these issues, they can be applied to each of the adjusted data sets. The multiple adjustment procedure allows incorporation of complex or non-standard statistical analyses.

6. Simulation Study

6.1. Design of Simulation Study

The objective of the simulation study was to compare the performance of the counterfactual modeling and resampling adjustment approaches as well as a naïve approach of using the unadjusted initial data set D for cross-sectional incidence estimation. Their performances were evaluated under various assumptions about the target population. The design of the simulation study and methods of data generation were motivated by a study of multi-assay algorithms for cross-sectional HIV incidence estimation in populations with subtype C infection.²⁰ Two biomarkers were considered for recent infection: the LAg-Avidity assay and the BioRad-Avidity assay. We evaluated the performance of the initial data sets for estimating the mean window period in the target population when there was a mismatch between the levels of viral suppression in the initial data set and the target population.

The model for generating the simulation data was based on a multivariate normal regression model for the two biomarkers. The vector of dependent variables included the biomarker LAg-Avidity (Y₁) and a transformation of the original BioRad-Avidity assay (Y₂). The regression model included an intercept term, logarithm (base 10) of infection duration (t), and an indicator (X) for viral suppression defined as binary indicator which was set to 1 if viral load ≤ 400 copies/ML. The specific parameter values of the regression coefficients, transformations of variables, covariance matrix and other details are given in the Supplementary Material.

The probability of viral suppression was allowed to depend on a binary classification of the duration of infection: Z = 1 corresponded to biological specimens that were infected less than or equal to one year, and Z = 2 corresponded to the biological specimens that were infected greater than one year. We considered two scenarios for the values of P_T(X|Z) and P_I(X|Z) (see Table 1). We were specifically interested in situations where the level of viral suppression was increased in the target population compared to the initial training data set due to increase uptake and earlier initiation of ART. Scenario A corresponded to a doubling in levels of viral suppression at both levels of Z in the target population compared to the initial training data set. Scenario B corresponds to a situation where viral suppression doubled in the target population at Z = 1 while suppression levels in the initial data set at Z = 2 were already high and did not change. We considered samples sizes of 250, 500, and 1000 observations for the initial training data sets. Viral suppression (X_i) for a specimen at level Z was generated by Bernoulli random trials with probabilities based on P_I(X|Z). The time since infection (t_i) was generated according to uniform random variable between 0 and 12 years.

Table 1:

Simulation results comparing the performance of adjustment procedures for estimating the mean window period in a target population. Bias, standard errors, and root mean squared error are given in days^*

Scenario						unadjusted					resampling adjustment					counterfactual modeling adjustment
	PI(X|Z)		PT(X|Z)		Sample size	Bias	% Bias	Empirical SE	Mean Bootstrap SE	RMSE	Bias	% Bias	Empirical SE	Mean Bootstrap SE	RMSE	Bias	% Bias	Empirical SE	Mean Bootstrap SE	RMSE
	Z=1	Z=2	Z=1	Z=2
A	.15	.30	.30	.60	250	−22.7	−16.3	34.8	35.1	41.5	−5.3	−3.8	49.8	42.2	50.0	−4.7	−3.3	35.3	37.0	35.6
					500	−22.0	−15.8	25.0	24.9	33.3	−4.3	−3.1	35.7	32.8	35.9	−2.3	−1.6	25.7	25.8	25.8
					1000	−21.0	−15.1	17.6	17.7	27.5	−1.4	−1.0	23.7	23.9	23.8	−0.1	−0.1	18.8	18.4	18.8
B	.15	.60	.30	.60	250	−12.8	−9.2	38.7	37.5	40.7	−3.5	−2.5	47.4	40.0	47.5	−4.6	−3.3	38.3	37.1	38.6
					500	−11.9	−8.6	25.4	26.9	28.1	−3.0	−2.2	30.8	31.1	30.9	−1.7	−1.2	24.7	26.4	24.7
					1000	−11.0	−7.9	19.6	19.2	22.4	−0.7	−0.5	22.5	22.2	22.5	−0.2	−0.1	19.2	18.8	19.2

^*Bias = $\stackrel{-}{\widehat{\mu }}-\mu$; Percent Bias = $\left(\overline{\widehat{\mu }}-\mu \right)/\mu \times 100\%$; Empirical SE = $1/499{\sum }_{l=1}^{500}{\left({\widehat{\mu }}_l-\stackrel{-}{\widehat{\mu }}\right)}^2$; mean bootstrap SE = $1/500{\sum }_{l=1}^{500}{\widehat{SE}}_l\left(\widehat{\mu }\right)$; RMSE = $\sqrt{1/500{\sum }_{l=1}^{500}{\left({\widehat{\mu }}_l-\mu \right)}^2}$.

We evaluated the performance of each of the approaches for estimating the mean window period in the target population. We considered a previously developed MAA where MAA positivity (MAA positivity indicates a recent infection) was defined as LAg < 2.8 OD-n and BioRad < 40% Avidity Index.²¹ Each set of simulations were repeated 500 times; for each simulated training data set, we created K = 10 modified data sets to construct the point estimates, and used B = 200 bootstrap iterations to estimate standard errors.

For each set of simulations, we obtained 500 estimates of μ using each approach. We compared these estimates to the true value μ for the target population, which we approximated by generating a data set of 100,000 observations using P_T(X|Z). For each approach, we calculated the empirical bias $\stackrel{-}{\widehat{\mu }}-\mu$, percent bias $\left(\stackrel{-}{\widehat{\mu }}-\mu \right)/\mu \times 100\%$, empirical standard error $1/499{\sum }_{l=1}^{500}{\left({\widehat{\mu }}_l-\stackrel{-}{\widehat{\mu }}\right)}^2$, mean bootstrap-estimated standard error $1/500{\sum }_{l=1}^{500}{\widehat{SE}}_l\left(\widehat{\mu }\right)$, and root mean squared error (RMSE) $\sqrt{1/500{\sum }_{l=1}^{500}{\left({\widehat{\mu }}_l-\mu \right)}^2}$ of the window period estimates based on each of the training data sets.

We investigated by simulation the robustness of the counterfactual modeling approach to incorrect modeling assumptions. We considered the performance of the counterfactual modeling adjustment approach (section 4) when Y₁ was not correctly transformed when fitting the multivariate normal regression model (see Supplementary material). We also considered the performance when time since infection (t) was not correctly transformed to log time in model fitting. In addition, we considered the situation when both model modeling errors were committed.

We also investigated by simulation the consequences of error in the assumed values of P_T(X|Z) for our methods. We repeated the simulations of the unadjusted, resampling, and counterfactual modeling approaches, using incorrect values $P_T^{*}\left(X|Z\right)$ for P_T(X|Z) where $P_T^{*}\left(X|Z\right)=P_T\left(X|Z\right)-.10$. We calculated the true value of μ using the true P_T(X|Z), and examined the resulting bias, standard errors, and RMSE for the three approaches.

6.2. Simulation Results

Table 1 shows simulation results comparing the performance of the three approaches for estimating the mean window period in the target population. The counterfactual modeling approach is employed using the correct form of the multivariate normal regression model in the set of simulations in Table 1. If the initial data set is small (250), we find that the resampling approach reduces the bias relative to the unadjusted analysis, by more than 70% in both scenarios. The counterfactual modeling approach reduces the bias by more than 60%. The biases were similar in magnitude for the two methods across both scenarios and all sample sizes.

The standard errors of the estimates based on the counterfactual modeling approach were similar to the standard errors of the unadjusted analysis. The standard error based on the resampling adjustment approach was generally larger. The standard error of the mean window period averages approximately 30% greater with resampling adjustment than counterfactual modeling adjustment across sample sizes and scenarios. The standard error of the mean window period using either adjustment procedure did not vary much between scenarios holding the sample size fixed.

Because the biases of the two methods were similar, and were always much smaller than the standard errors, RMSE was dominated by the standard error component. Therefore, the counterfactual modeling approach had better performance according to this metric.

Table 2 shows the simulation results evaluating the robustness of the counterfactual modeling approach to biomarker model misspecification. The bias increased appreciably when Y₁ was not correctly transformed especially for scenario A. The bias also increased somewhat when t was not correctly transformed. When both modeling errors were present, the bias increased even further than when only Y₁ was incorrectly transformed.

Table 2:

Simulation results evaluating the robustness of the counterfactual modeling adjustment approach to model misspecification: Y₂ and infection duration (t) not correctly transformed. Bias, percent bias, empirical standard error, mean bootstrap-estimated standard error, and root mean squared error are given for each estimator of the mean window period (in days)^*

Scenario						Y2 not transformed					t not transformed					Both Y2 and t not transformed
	PI(X|Z)		PT(X|Z)		Sample size	Bias	% Bias	Empirical SE	Mean Bootstrap SE	RMSE	Bias	% Bias	Empirical SE	Mean Bootstrap SE	RMSE	Bias	% Bias	Empirical SE	Mean Bootstrap SE	RMSE
	Z=1	Z=2	Z=1	Z=2
A	.15	.30	.30	.60	250	−18.2	−13.1	34.5	35.6	39.0	−7.4	−5.3	34.9	37.0	35.6	−18.9	−13.5	34.4	35.6	39.3
					500	−15.4	−11.0	25.1	25.1	29.4	−5.2	−3.7	25.8	25.8	26.3	−16.7	−12.0	25.2	25.1	30.2
					1000	−13.5	−9.7	17.9	17.9	22.4	−3.3	−2.3	18.6	18.4	18.9	−14.6	−10.4	17.9	17.9	23.1
B	.15	.60	.30	.60	250	−10.2	−7.3	39.0	37.4	40.3	−7.3	−5.2	38.6	37.2	39.3	−11.4	−8.2	38.7	37.5	40.3
					500	−8.0	−5.7	25.3	26.7	26.5	−4.8	−3.4	24.9	26.4	25.4	−9.7	−7.0	25.3	26.7	27.1
					1000	−6.3	−4.5	19.3	19.0	20.3	−3.1	−2.2	19.0	18.8	19.3	−8.1	−5.8	19.3	19.0	20.9

^*Bias = $\stackrel{-}{\widehat{\mu }}-\mu$; Percent Bias = $\left(\overline{\widehat{\mu }}-\mu \right)/\mu \times 100\%$; Empirical SE = $1/499{\sum }_{l=1}^{500}{\left({\widehat{\mu }}_l-\stackrel{-}{\widehat{\mu }}\right)}^2$; mean bootstrap SE =$1/500{\sum }_{l=1}^{500}{\widehat{SE}}_l\left(\widehat{\mu }\right)$; RMSE = $\sqrt{1/500{\sum }_{l=1}^{500}{\left({\widehat{\mu }}_l-\mu \right)}^2}$. See Supplementary Material for further details on model misspecifications

Table 3 shows the simulation results evaluating the effects of error in the assumed values of P_T(X|Z). We find that the bias of our methods increases in this situation. RMSE increases accordingly; due to its higher standard error, the resampling method still has higher RMSE than the counterfactual modeling approach, especially with small sample sizes.

Table 3:

Simulation results evaluating the effect of error in the assumed values of P_T(X|Z) on the performance of adjustment procedures for estimating the mean window period in a target population. In this sensitivity analysis, $P_T\left(X|Z\right)=P_T^{*}\left(X|Z\right)+.10$. Bias, standard errors, and root mean squared error are given in days^*

Scenario						unadjusted					resampling adjustment					counterfactual modeling adjustment
	PI(X|Z)		${\mathit{P}}_{\mathit{T}}^{\mathbf{*}}\left(\mathit{X}|\mathit{Z}\right)$		Sample size	Bias	% Bias	Empirical SE	Mean Bootstrap SE	RMSE	Bias	% Bias	Empirical SE	Mean Bootstrap SE	RMSE	Bias	% Bias	Empirical SE	Mean Bootstrap SE	RMSE
	Z=1	Z=2	Z=1	Z=2
A	.15	.30	.30	.60	250	−33.5	−22.3	34.8	35.1	48.3	−16.1	−10.7	49.8	42.2	52.2	−15.5	−10.3	35.3	37.0	38.5
					500	−32.8	−21.8	25.0	24.9	41.2	−15.1	−10.0	35.7	32.8	38.7	−13.1	−8.7	25.7	25.8	28.8
					1000	−31.8	−21.2	17.6	17.7	36.4	−12.2	−8.1	23.7	23.9	26.7	−10.9	−7.2	18.8	18.4	21.7
B	.15	.60	.30	.60	250	−23.6	−15.7	38.7	37.5	45.3	−14.3	−9.5	47.4	40.0	49.5	−15.4	−10.3	38.3	37.1	41.3
					500	−22.7	−15.1	25.4	26.9	34.1	−13.8	−9.2	30.8	31.1	33.7	−12.5	−8.3	24.7	26.4	27.6
					1000	−21.8	−14.5	19.6	19.2	29.2	−11.5	−7.7	22.5	22.2	25.3	−10.9	−7.3	19.2	18.8	22.1

^*Bias = $\stackrel{-}{\widehat{\mu }}-\mu$; Percent Bias = $\left(\overline{\widehat{\mu }}-\mu \right)/\mu \times 100\%$; Empirical SE = $1/499{\sum }_{l=1}^{500}{\left({\widehat{\mu }}_l-\stackrel{-}{\widehat{\mu }}\right)}^2$; mean bootstrap SE = $1/500{\sum }_{l=1}^{500}{\widehat{SE}}_l\left(\widehat{\mu }\right)$; RMSE = $\sqrt{1/500{\sum }_{l=1}^{500}{\left({\widehat{\mu }}_l-\mu \right)}^2}$.

7. Application

In this section we apply the methods to an initial training data set of Subtype B infection in the United States.^19,21 The data set included 1782 observations from 709 individuals, collected in three cohort studies in the US, with estimated durations of infection ranging from one month to greater than 8 years. The data, from individuals with a documented HIV negative test and a subsequent documented HIV positive test, had biomarker values including LAg-Avidity, BioRad-Avidity, and viral load measurements. The exact durations of infection were interval-censored and were multiply imputed (see Supplementary Material).

The goal of this analysis was to estimate the properties of various MAAs, and identify an optimal algorithm, for a target population with higher levels of viral suppression than the initial data set, which had 14.5% viral suppression (viral load ≤ 400 copies/mL) among samples with estimated duration of infection less than one year, and 23.4% viral suppression among samples with estimated duration of infection greater than one year. We considered two target populations: first, a target population with double the viral suppression of the initial data set (“Scenario A”); and second, a population with triple viral suppression levels compared to the initial dataset (“Scenario B”).

We applied the resampling adjustment approach because it did not require assuming a parametric biomarker model. For each scenario, we applied this approach to construct estimates and confidence intervals for the mean window and shadow for each of 110 MAAs, consisting of combinations of cutoffs for LAg-Avidity and BioRad Avidity (see Supplementary Material). We generated K = 100 multiply adjusted datasets. We performed B = 1000 bootstraps. We then identified the optimal algorithm with the largest mean window period, among those algorithms with a shadow whose upper confidence interval limit was less than 365 days.

The optimal algorithms for each suppression scenario are shown in Figure 2, along with the optimal algorithm based on the unadjusted data set. For the unadjusted data set, the optimal algorithm was BioRad Avidity < 40% Avidity Index, LAg-Avidity < 2.8 OD-n; the estimated mean window period was 116 days (95% CI 95–142;). In scenario (A), the optimal algorithm was still BioRad Avidity < 40% Avidity Index, LAg-Avidity < 2.8 OD-n and the estimated mean window period increased to 128 days (95% CI 98–158). In scenario (B), the optimal algorithm shifted to BioRad Avidity < 37% Avidity Index, LAg-Avidity < 2.8 OD-n and the estimated mean window period was 114 days (95% CI 83–147). We find, as expected, that with more viral suppression, we need either stricter (lower) biomarker cutoffs for labeling an infection as an infection that occurred recently, or we estimate larger values of μ, or both. The intuitive explanation for that finding is that with more viral suppression long-term infections will have biomarker profiles that resemble recent infections and as such, to compensate either stricter biomarker cutoffs are needed or a large mean window period is determined.

Figure 2:

Multi-assay algorithms for Subtype B dataset, adjusting for higher levels of viral suppression in the target population. Panel A: unadjusted; Panel B: adjusted for doubling of viral suppression levels; Panel C: adjusted for tripling of viral suppression levels.

8. Discussion

HIV prevention and evaluation research rely on accurate estimates of HIV incidence. Methodological challenges arise because of the difficulty in determining HIV incidence from cohorts of uninfected persons and documenting HIV acquisition.²⁷ The cross-sectional approach addresses these challenges because incidence can be estimated without requiring longitudinal follow-up of persons. However, the cross-sectional approach does rely on an initial training data set to develop and calibrate the statistical methods to be used in cross-sectional surveys. The problem addressed in this paper is that the initial training data set may over time not reflect the current target population. We developed methods to multiply adjust the initial training data set with the goal that the adjusted data sets better reflect the target population. These adjustment procedures could help avoid the time and expense of collecting a completely new training data set from the current target population. A critical assumption of the methods is that the variables that describe the relevant differences between the initial and target population are identified. In our application the relevant variable was anti-retroviral treatment which results in viral suppression.

We proposed two methods for adjusting the data set, a resampling approach and a counterfactual modeling approach. We found that the biases were similar, as long as the counterfactual modeling approach was correctly specified. The standard errors from the resampling approach were larger than those from the counterfactual modeling approach. The counterfactual modeling approach makes parametric assumptions not shared by the resampling approach; these assumptions result in lower standard errors, but that is at the cost of vulnerability to bias when those assumptions are violated.

Surprisingly, we found that the standard errors of the counterfactual modeling approach were similar to, rather than larger than, those of the unadjusted analysis. This phenomenon can be understood from the following perspective. If the regression coefficient β₂ were known precisely, then counterfactually adjusted data sets would have the same distribution as data sampled directly from the target population. Therefore the variability of $\widehat{\mu }$ from the counterfactual modeling approach would be similar to that from estimating μ directly from the target population. Furthermore, since we are dichotomizing the biomarker values into MAA-positive and MAA-negative and the sample sizes of our initial training data sets were large, our simulation results show that estimation of β has minimal effects on the standard errors of $\widehat{\mu }$.

Due to the larger standard errors, we found that the resampling approach had larger RMSE than the correctly-specified version of the counterfactual modeling approach in the situations considered. However, as expected, with model misspecification, the modeling approach’s bias, and accordingly RMSE, increased substantially. The counterfactual modeling approach requires model assumptions. Our robustness study showed that considerable bias can occur with the counterfactual approach if the modeling assumptions are incorrect. These findings suggest that the resampling approach may be the method of choice in situations where the correct functional form for the biomarker model is unclear. However, if the sample sizes in the initial training data set are small in some strata, then the counterfactual modeling approach with sensible modeling assumptions may provide better precision than the resampling approach. We found that a modest choice for the number of adjusted data sets in the resampling approach (K) performed well in simulations. Further work to guide the choice of K would be useful.

We assumed that only the levels of viral suppression differed between the training data set and the target population. In reality, there may be several such relevant covariates, such as HIV subtype, calendar time, and demographic factors that differ between the training and target populations. Further, the model for the biomarkers may involve interactions among these covariates. Future work could include extending these methods to such situations, and would involve assuming or estimating a joint model for the covariates as a function of time, that is, replacing P_T(X|Z) with P_T(X₁, …X_p|Z), and using this joint model to either resample the training data set or generate counterfactually adjusted data sets.

In the application example, we considered an MAA that did not include viral suppression status as a biomarker. However, it is possible to apply our methods to MAAs that do include viral suppression status, even when viral suppression is also the covariate X whose distribution needs to be adjusted.

A critical assumption of our methods for transporting results from the initial training dataset to the target population is the exchangeability assumption, that is, P_T(Y|X,Z,t) = P_I(Y|X,Z,t). An epidemic could evolve to violate this assumption. For example, if the virus mutates, new strains with different biological signatures could be introduced that might invalidate the assumption. Changes in clinical practice or in population characteristics could also invalidate this assumption; for example, if the mixture of causes of viral suppression (from anti-retroviral treatment versus innate resistance) evolves over time, then the statistical relationship between suppression and biomarker values may also shift. It is important to consider the validity of the exchangeability assumption using expert knowledge and any additional data that may be available from the target population.

While we have discussed the multiple adjustment procedure in the context of cross-sectional HIV incidence estimating, the proposed procedure is more general. It could be applicable in other situations where complex statistical analyses are conducted using an initial data set but those results may not be directly transportable to the new target population of interest.

Cross-sectional HIV incidence methods have been successfully applied in many settings around the world.^28,29 These methods require training data sets to develop and calibrate the methods. The approach we have proposed could offer a practical and cost-effective way to apply cross-sectional incidence methods to new target populations as the epidemic continues to evolve.