Assessing the reliability of the CD4 depletion model in the presence of Ending the HIV Epidemic initiatives

Abstract

Background:

Accurate estimates of HIV incidence are necessary to monitor progress towards Ending the HIV Epidemic (EHE) initiative targets (90% decline by 2030). US incidence estimates are derived from a CD4 depletion model (CD4 model). We performed simulation-based analyses to investigate the ability of this model to estimate HIV incidence when implementing EHE interventions that have the potential to shorten the duration between HIV infection and diagnosis (diagnosis delay).

Methods:

Our simulation study evaluates the impact of three parameters on the accuracy of incidence estimates derived from the CD4 model: rate of HIV incidence decline, length of diagnosis delay, and sensitivity of using CD4 counts to identify new infections (recency error). We model HIV incidence and diagnoses after the implementation of a theoretical prevention intervention and compare HIV incidence estimates derived from the CD4 model to simulated incidence.

Results:

Theoretical interventions that shortened the diagnosis delay (10–50%) result in overestimation of HIV incidence by the CD4 model (10–92%) in the first year and by more than 10% for the first six years after implementation of the intervention. Changes in the rate of HIV incidence decline and the presence of recency error had minimal impact on the accuracy of incidence estimates derived from the CD4 model.

Conclusion:

In the setting of EHE interventions to identify persons living with HIV earlier during infection, the CD4 model overestimates HIV incidence. Alternative methods to estimate incidence based on objective measures of incidence are needed to assess and monitor EHE interventions.

Introduction

HIV incidence, the number of new infections that occur in one year, cannot be directly measured as infections are often asymptomatic until later stages and universal diagnostic testing is not performed annually. As a result, HIV diagnosis rates, which are influenced by testing efforts, may not provide an accurate snapshot of the HIV epidemic. In spite of this, in early 2019, the United States (US) Department of Health and Human Services launched the ambitious Ending the HIV Epidemic (EHE) initiative to target a 75% decline in HIV incidence by 2025 and a 90% decline by 2030 [1]. The initiative tasks jurisdictions with implementing prevention programs tailored to local epidemiology [2]. Local HIV transmission dynamics often require jurisdictions to prioritize their EHE prevention efforts among specific sub-populations [3]. For example, in San Diego County, HIV health disparities among Latinx men who have sex with men (MSM) highlighted the need for improved testing capacity, community engagement and benefits navigation [4]. Accurate estimates of changes in HIV incidence in these sub-populations (i.e., 100–300 annual new HIV diagnoses) within short timeframes is essential to evaluating the effectiveness of these prevention efforts and to track the progress towards achieving regional EHE targets.

Estimating HIV incidence poses several challenges that current methods were not originally designed to address. The CD4 depletion model (CD4 model) is a method widely used to estimate HIV incidence nationally and locally in the US [5]. The CD4 model estimates population incidence by first estimating the duration of infection (DoI) and diagnosis delay (elapsed time between infection and diagnosis) for each individual [6]. DoI estimation uses each individuals’ date of diagnosis and first antiretroviral therapy (ART)-naïve CD4+ cell count (CD4 count) after diagnosis. By aggregating these values across the population of interest into a diagnosis delay distribution, the CD4 model estimates population HIV incidence (Supplement Figure 1). However, the CD4 model assumes that the diagnosis delay is stable over an eight-year period [6,7]. Many EHE interventions that promote earlier HIV diagnosis (e.g., universal opt-out testing programs) are expected to shorten the diagnosis delay and would challenge this key assumption of the CD4 model (see Supplemental Materials for a detailed description). Additionally, significant intra- and inter-person variability in longitudinal CD4 counts raises concerns about the accuracy of DoI estimates, potential impacts on diagnosis delay distribution, and ultimately incidence estimates derived from the CD4 model; we define this error due to using CD4 depletion to detect recent infection as recency error [8–10]. The combined error from these sources is magnified in small populations, as demonstrated by substantial uncertainty (>30% relative standard error) in CD4 modeled incidence in more than 60% of target EHE jurisdictions in the 2019 CDC surveillance report [5].

In this manuscript, we demonstrate that the CD4 model could require up to six years after an intervention to accurately estimate incidence (i.e., impact of an intervention). Our simulation-based analysis highlights limitations of the CD4 model when used to estimate HIV incidence in the presence of theoretical interventions that shorten the diagnosis delay. Our analysis also evaluates how recency error impacts CD4 modeled incidence. Given these limitations, in the Discussion we discuss potential solutions to monitor incidence based on objective measures of incidence to assess EHE interventions.

METHODS

Study design

To evaluate the performance of the CD4 model to estimate HIV incidence, we conduct an extensive simulation study. The study investigates the impact of three parameters on incidence estimates from the CD4 model. Parameter 1 is the rate of HIV incidence decline over time. Parameter 2 is the length of diagnosis delay, which many EHE interventions shorten. Parameter 3 is recency error. In our simulation study, we vary the level of each parameter. Using the results from the simulation, we assess the accuracy of HIV incidence estimates from the CD4 model by comparing estimated and simulated incidence for each simulation scenario. Accuracy is quantified using the percentage difference of estimated incidence compared to simulated incidence.

Model Description

All simulation scenarios use illustrative data from a CDC presentation as a pre-intervention phase from 2014–2020 [7]. These data represent HIV incidence in a large US jurisdiction and provide the baseline incidence decline and diagnosis delay distribution. Next, we assume the instantaneous implementation of a theoretical intervention at the end of 2020 through 2028, representing the first eight years of the EHE initiative (2021–2028). During the intervention phase we vary the values for the three parameters independently; details for each parameter are provided below. Our approach enables us to directly input the incidence and diagnosis delay for each year. Therefore, we do not incorporate an epidemic model to simulate yearly incidence; however, our approach to assessing the CD4 model can be easily extended to these models [11–13]. As changes in incidence and diagnosis delay are fully integrated within the first eight years after an intervention, results beyond this time point are not shown.

Parameter 1: Rate of HIV incidence decline over time.

During the intervention phase, incidence is simulated to follow three levels of decline over time; these declines represent the potential impact of HIV interventions. Specifically, we simulate scenarios where the HIV incidence has a stable (5% annual) decline, no (0% annual) decline, and a rapid (24% annual) decline (Figure 1). These rates of decline emulate a range of potential incidence decline outcomes, with the rapid decline rate calculated to meet the 75% decline goal at 2025 set by the EHE initiative.

Figure 1:

Simulated HIV incidence, 2014–2028. The vertical line separates the pre-intervention (years 2014–2020) and intervention (years 2021–2028) periods. The blue line simulates no change in the annual HIV incidence in the intervention period. Simulations illustrate theoretical Ending the HIV Epidemic prevention interventions representing a stable (5% per year) decline in red and a rapid (24% per year) decline in green in annual HIV incidence in the intervention phase.

Parameter 2: Length of the diagnosis delay.

For each level of incidence decline, four levels of decrease in diagnosis delay are simulated. The diagnosis delay is simulated to remain stable (0% decrease) or decrease by 10%, 25%, or 50% during the intervention phase to depict the potential impact of testing initiatives. Our model assumes the time to diagnosis shortens by each level independent of incidence decline and calculates how this impacts the proportion of individuals diagnosed within a given timeframe post-infection and incidence estimates from the CD4 model.

Parameter 3: Recency error.

For each of the 12 combinations of rate of incidence decline and decrease in diagnosis delay, HIV incidence from the CD4 model is estimated with and without recency error. We first estimate the sensitivity of CD4 depletion to detect incident infection using data from the San Diego Primary Infection Resource Consortium (PIRC) [14–16]. PIRC participants are newly diagnosed with HIV infection; those with incident infection have a DoI calculated using virologic and serologic data (Supplement Table 1). Using the derived sensitivity (from PIRC), we adjust the diagnosis delay distributions for recency error. Functionally, recency error impacts estimated DoI (eDoI) but does not change simulated DoI (sDoI), or date of diagnosis. To incorporate recency error into diagnosis delay distributions, diagnoses made in year y, which are simulated as incident infections (sDoI < 1 year) but misclassified as prevalent infections (eDoI ≥ 1 year, based on CD4 depletion sensitivity), are redistributed to the prior years of infection based on the distribution observed in PIRC (Supplement Table 3).

In total, 24 simulation scenarios (Table 1) are used to evaluate the impact of the three parameters on HIV incidence estimates from the CD4 model.

Table 1:

Summary of the simulation scenarios

Rate of Incidence Decline	Diagnosis Delay Decrease	Recency Error
Stable decline (5%/year)	0% (no change)	Not included
		Included
	10% decrease	Not included
		Included
	25% decrease	Not included
		Included
	50% decrease	Not included
		Included
No decline (0%/year)	0% (no change)	Not included
		Included
	10% decrease	Not included
		Included
	25% decrease	Not included
		Included
	50% decrease	Not included
		Included
Rapid decline (24%/year)	0% (no change)	Not included
		Included
	10% decrease	Not included
		Included
	25% decrease	Not included
		Included
	50% decrease	Not included
		Included

Estimating the sensitivity of using CD4 depletion to detect incident infection

PIRC is an observational cohort of people newly diagnosed with HIV infection with well-characterized dates of infection [14,16]. Recruitment to the cohort began 26 years ago, thus many participants enrolled in the remote past chose to delay initiation of ART. The availability of well-characterized, longitudinal follow-up in persons who remained ART-naïve make this an ideal group for estimating the sensitivity of CD4 depletion to detect incident infection. See Supplemental Materials including Supplement Table 2 for further details.

Estimating sensitivity of the CD4 model– a bootstrapping approach

Among individuals who opted to delay initiation of ART, the potential for CD4 depletion to differentiate between incident and prevalent infection is fully evaluated (Supplement Figure 2). We use a bootstrapping procedure [17] with 1000 person sample populations, sampling with replacement from the total population of CD4 counts and dates available to derive a sensitivity estimate. The diagnosis delay is redistributed using this estimate as described above.

Results

In study simulations that include the implementation of an intervention that shortens the diagnosis delay by 50% and decreases HIV incidence by 5% annually, the CD4 model overestimates simulated incidence by at least 10% for the first six years after the intervention (Figure 2A, purple solid line). The percentage difference for this scenario is illustrated without recency error. In addition, in this scenario the CD4 model overestimates simulated incidence by >90% the first year after intervention implementation. In each subsequent year, the percentage difference between CD4 modeled incidence and simulated incidence decreases. Eight years after implementation of the intervention, there is no difference between the CD4 modeled incidence and simulated incidence.

Figure 2:

HIV incidence estimates using CD4 model compared with simulated incidence. Panel A demonstrates scenarios where there is a 5% annual decline in simulated incidence. Panel B demonstrates scenarios where there is a 0% annual decline in simulated incidence. Panel C demonstrates scenarios where there is a 24% annual decline in simulated incidence. Lines are labeled according to their delay between infection and diagnosis – [DD(0), blue] for no change and [DD(50), purple] for 50% decrease compared to baseline (purple) – and inclusion of recency error (RE, dashed lines) that results from using CD4 depletion to estimate the duration of infection at diagnosis.

Parameter 1: Rate of incidence decline

The rate of incidence decline only modestly impacts the error in incidence estimates from the CD4 model. For example, in all three settings where the decline in incidence rate varies (0%, 5%, and 24% decline/year; Figure 2 A, B, and C, purple solid lines), the CD4 model overestimates simulated incidence by the same amount (91%) in the first year after the implementation of an intervention that reduces the diagnosis delay by 50%. Across all incidence rate levels evaluated, the difference between CD4 modeled and simulated incidence decreases in each subsequent year, reaching a nadir in year eight (0%). It takes six years after intervention implementation before CD4 model estimated incidence is within 10% of the simulated incidence.

Parameter 2: Decrease in diagnosis delay

Shortening the diagnosis delay leads to overestimation of HIV incidence using the CD4 model (Figure 2). When the diagnosis delay is unchanged, the CD4 modeled incidence estimate closely approximates the simulated incidence (Figure 2 A, B, C, blue solid lines). In the setting of a stable diagnosis delay (0% change), across all incidence decline scenarios and years after implementation, estimates of HIV incidence derived by the CD4 model are within 10% of simulated incidence. By contrast, as diagnosis delay decreases (by 10%, 25% and 50%; Figure 3), there is increasing overestimation of HIV incidence by the CD4 model. Regardless of the degree of shortening in the diagnosis delay, the difference between simulated HIV incidence and CD4 modeled incidence decreases as the time since implementation of the intervention increases. See Supplement Figure 4 for additional scenarios.

Figure 3:

Simulated incidence by decreasing diagnosis delay. There is an incremental shortening of the diagnosis delay (DD) by 0% [DD(0), blue], 10% [DD(10), green], 25% [DD(25), red] and 50% [DD(50), purple]. All simulations have a 5% annual decline in simulated HIV incidence.

Parameter 3: Recency error

The sensitivity of the CD4 model to detect incident infection is estimated as 55% (95% CI 50%−59%). When the diagnosis delay is redistributed based on sensitivity, the CD4 modeled incidence is primarily impacted under scenarios when there is an associated decline in incidence (Figure 2). For example, in scenarios with 50% decrease in diagnosis delay and 24% HIV incidence decline (Figure 2C), recency error leads to an overestimation of HIV incidence by 95% in year one compared to 92% without recency error. In these scenarios, recency error leads to lower incidence estimates after the first year, resulting in a 14% underestimation of incidence at year eight (compared to 0% without recency error). Similarly, in scenarios with 50% decrease in diagnosis delay and 5% HIV incidence decline, recency error leads to lower incidence estimates, beginning two years after the intervention, and underestimates incidence by 4% in year eight of the intervention (Figure 2A). For scenarios with no change in diagnosis delay, and ≥ 5% decline in HIV incidence, recency error again leads to lower incidence estimates. This results in a gradual increase in the magnitude of HIV incidence underestimation with recency error, reaching 14% in the 24% decline scenario and 4% in the 5% decline scenario. In contrast, for scenarios with 0% decline in HIV incidence, recency error had minimal impact on the estimation of incidence by the CD4 model, regardless of the degree of diagnosis delay decrease (Figure 2A–C).

Discussion

Accurate estimates of HIV incidence are critical to monitoring the HIV epidemic trend, and for evaluating the efficacy of EHE programs. Our simulation study found that interventions resulting in a 50% decrease in diagnosis delay led to large overestimations in CD4 modeled HIV incidence in the first six years after the intervention. Song and colleagues previously acknowledged that testing increases may lead to overestimates in CD4 modeled incidence, however, the potential magnitude was not quantified [6]. Increased testing services, the first pillar in EHE, is a fundamental component of any EHE plan to decrease incidence [3,18]. Prior simulation models estimate that persons living with undiagnosed HIV account for nearly 40% of new HIV transmissions [19]. Improved scale up of HIV testing could support earlier diagnosis and linkage to care, which when coupled with rapid initiation of effective ART, would prevent onward transmission of HIV and decrease HIV incidence [20,21]. As recent estimates show a median diagnosis delay of three years in the US [22], a decline to a median diagnosis delay of 1.5 years seems feasible. However, overestimates in incidence derived from the CD4 model while implementing testing interventions may lead to the impression that these interventions are failing, which could lead to the inappropriate termination of successful programs.

The sensitivity of the CD4 model to detect recent infection was only marginally higher than the odds of flipping a coin. This aligns with observations of significant intra- and inter-person variability in longitudinal CD4 counts [8–10,23–25]. Surprisingly, while the addition of recency error falsely lengthens diagnosis delay, recency error only leads to decreases in CD4 modeled incidence when combined with a decline in HIV incidence. When there is no change in HIV incidence, the recency error is offset by the corresponding change in proportion diagnosed within a given duration after infection, leading to minimal impact on incidence estimates. However, in scenarios where recency error is combined with a decrease in HIV incidence, the CD4 model produces lower estimates compared to scenarios without recency error. This occurs because in these scenarios, the CD4 model assumption that the proportion of individuals diagnosed within eight years of infection equals the proportion of individuals infected within eight years of diagnosis is incorrect. Therefore, the further into the intervention, the greater the magnitude of the incidence difference associated with use of the CD4 model. These observations are specific to the CD4 model developed by the US CDC; although alternative CD4 methods to estimate DoI have not been officially compared on their accuracy, our results should still be valid as the underlying assumptions are similar [26,27].

While we currently lack ideal methods for estimating changes in HIV incidence in response to decreases in diagnosis delay, objective measures of recency will be key moving forward. Objective measures of recency include recency assays [28–32], identification of Stage 0 HIV infection (new diagnosis with a negative HIV test in the prior 6 months prior), and identification of acute HIV infection (antigen positive/antibody negative) [33]. First, as suggested by Song and colleagues, an adjustment for recency data could decrease the recency error observed in our simulations [6]. This would improve the accuracy of diagnosis delay measures, resulting in more robust incidence estimates derived from the CD4 model. Second, monitoring for recency could serve as a proxy for total incidence until methods are developed that more accurately estimate incidence in the setting of EHE interventions. A decrease in observed recent infections in the setting of increased HIV testing services would provide compelling evidence that incidence is decreasing. Finally, incidence estimates derived from these objective measures should be robust to a decreasing diagnosis delay. Objective measures of incidence will be key to developing accurate incidence estimates to evaluate whether such focused EHE interventions are effective.

The monitoring of objective measures of incidence should be integrated into our HIV screening algorithms. Multiple groups, including the World Health Organization and Joint United Nations Programme on HIV/AIDS and the US President’s Emergency Plan for AIDS Relief, have endorsed the routine use of recency tests [34,35]. The simplest strategy for improving estimation of HIV incidence involves reporting of Stage 0 infection and acute HIV infection [36,37]. While the current laboratory HIV screening algorithm supports identification of acute HIV infection, reporting of this information to public health departments may not be required. However, both acute HIV infection and Stage 0 infection have short recency window periods, leading to large uncertainty in incidence estimates derived from these measures [38]. Alternatively, public health policies could be revised to ensure collection of additional or remnant blood samples for recency testing in all persons newly diagnosed with HIV using a recent infection testing algorithm (RITA). One recency test, the signal to cutoff ratio from the ARCHITECT 4^th generation HIV Ab/Ag combo assay, has been validated, and is available with routine operation of this FDA approved 4^th generation HIV Ab/Ag combo assay [32,39]. The routine use of HIV recency assays (Asante HIV-1 Rapid Recency Assay and Swift Recent Infection Assay) to improve incidence detection has been successfully implemented in more than 25 countries since 2017 through the Tracking with Recency Assays to Control the Epidemic Initiative [35,40–42]. These methods may provide recency information for a substantial proportion of newly diagnosed individuals without imposing an undue burden related to cost or patient discomfort.

Development of novel methods for estimating incidence is an active area of research [43–46]. Estimating HIV incidence requires the ability to (1) identify who among newly diagnosed individuals are recently infected and (2) estimate the proportion of recent infections that were diagnosed. These two items can be combined to provide an incidence estimate, similar to the approach used by the CD4 model, but without the need for a stable diagnosis delay. For identifying recently infected individuals, we propose the use of RITAs, which are well validated for the identification of recent infections [28–30,32,39,47–49]. For estimating the proportion of recent infections diagnosed, traditional and non-traditional epidemiologic metrics should be investigated including the total number of tests performed, and HIV phylogenetic characteristics [50–53]. HIV molecular epidemiology, which is integral to EHE surveillance strategies to identify growing clusters, is underutilized in estimating incidence [1,4,54]. Additional research is needed to convert this approach to a practical and validated tool to rapidly evaluate jurisdiction-level HIV prevention programs.

Our study has several limitations. First, the PIRC cohort was designed to identify and enroll individuals with acute and early HIV infection to longitudinal follow up. As a result, our bootstrapped cohort used to estimate CD4 depletion sensitivity has a high proportion of incident infections (91%, Supplement Figure 3), and does not represent a population-based convenience sample. However, CD4 depletion sensitivity should be independent of the prevalence of incident infection in the observed population, thus our sensitivity estimates are likely still accurate. Conversely, CD4 depletion specificity from PIRC would likely underestimate population CD4 depletion specificity, thus we made a conservative estimate that CD4 depletion specificity is 100% for our analysis. Second, our statistical model did not include a transmission model or an internal CD4 depletion model. Similarly, our analyses did not include scenarios with gradual roll-out of interventions. Our methodology was chosen to best understand the limitations of the CD4 model and the theoretical limits of bias in estimating incidence, without introducing additional variables. While using a gradual roll out of interventions may reduce the magnitude of effect seen, we believe that an effect would still be present. Additionally, while one time “shock” interventions are generally unrealistic, the EHE campaign has funded multiple simultaneous interventions in priority jurisdictions across multiple organizations. These efforts are necessary to meet the ambitious incidence goals set out by EHE. Analyses that incorporate these components are an area for potential future research.

In summary, we found that the EHE initiative and associated prevention interventions create a novel challenge for estimating HIV incidence. Interventions that are associated with decreases in diagnosis delay will overestimate HIV incidence estimates derived from the CD4 model, particularly in the first six years after the intervention. The inability to rely on accurate incidence estimates will challenge the interpretation of prevention program efficacy by public health departments and may lead to inappropriate termination of successful programs and reallocation of resources. Thus, HIV incidence estimates derived from the CD4 model should be interpreted with caution. New strategies to estimate incidence in the setting of dynamic changes in regional HIV incidence and prevention strategies are needed to support ongoing EHE efforts.