Poor Performance of the Determine HIV-1/2 Ag/Ab Combo Fourth-Generation Rapid Test for Detection of Acute Infections in a National Household Survey in Swaziland

Abstract

Fourth-generation HIV rapid tests (RTs) claim to detect both p24 antigen (Ag) and HIV antibodies (Ab) for early identification of acute infections, important for targeting prevention and reducing HIV transmission. In a nationally representative household survey in Swaziland, 18,172 adults, age 18 to 49 years, received home-based HIV rapid testing in 2010 and 2011. Of the 18,172 individuals, 5,822 (32.0%) were Ab positive (Ab⁺) by the Determine HIV-1/2 Ab/Ab combo test, and 5,789 (99.4%) of those were confirmed to be reactive in the Uni-Gold test. Determine combo identified 12 individuals as having acute infections (Ag⁺/Ab negative [Ab⁻]); however, none had detectable HIV-1 RNA and 8 of 12 remained HIV negative at their 6-week follow-up visit (4 were lost to follow-up). All RT-nonreactive samples were pooled and tested by nucleic acid amplification testing (NAAT) to identify acute infections. NAAT identified 13 (0.1%) of the 12,338 HIV antibody-negative specimens as HIV RNA positive, with RNA levels ranging from 300 to >10,000,000 copies/ml. However, none of them were Ag⁺ by Determine combo. Follow-up testing of 12 of the 13 NAAT-positive individuals at 6 months demonstrated 12 seroconversions (1 individual was lost to follow-up). Therefore, the Determine combo test had a sensitivity of 0% (95% confidence interval, 0 to 28) and positive predictive value of 0% for the detection of acute infections. The ability of the 4th-generation Determine combo to detect antigen was very poor in Swaziland. Thus, the Determine combo test does not add any value to the current testing algorithm; rather, it adds additional costs and complexity to HIV diagnosis. The detection of acute HIV infections may need to rely on other testing strategies.

INTRODUCTION

The development of HIV rapid tests (RTs) has facilitated the massive scale-up of HIV testing and counseling at thousands of testing venues, especially in sub-Saharan Africa, allowing millions of individuals to receive their HIV diagnosis outside a primary care facility (1). HIV RTs have relied on the detection of HIV antibodies (Ab) after seroconversion, and the sensitivities of various RTs have approached nearly 100% (2, 3). However, most RTs are unable to detect the acute phase of infection, during which HIV Ab are absent and only viral nucleic acid or p24 antigen (Ag) may be detectable. Identification of individuals in the acute phase is considered important in curbing new infections, since the acute phase is characterized by a high viral load, a founder virus capable of efficient infection, and the absence of HIV antibodies, resulting in a greater risk of transmission compared to the risk later in infection (4–6). Individuals in the acute phase of infection are considered drivers of HIV transmission, accounting for 10 to 50% of new HIV infections (7–9). Thus, it has been suggested that when these individuals are identified and paired with risk reduction behavioral counseling, treatment, and other intervention and prevention strategies, a major gain in public health by reduction of the overall HIV incidence can result (10, 11).

The current methods for detection of acute infection are laboratory based and require the detection of viral RNA or p24 antigen using complex, laboratory-based methods. The standard approach for nucleic acid amplification testing (NAAT) employs PCR to detect viral RNA in either individual or pooled HIV-seronegative samples (12–15), while the occurrence of p24 Ag during the acute phase is detected by enzyme-linked immunosorbent assays (ELISAs), a less commonly used method. Although the sensitivity of detection of acute infection by NAAT is high, it requires screening a large number of HIV-seronegative individuals due to the short acute-infection window (approximately 3 to 4 weeks [5, 11]). Moreover, such screening is costly, labor-intensive, time-consuming, and thus far, impractical to implement on a large scale. ELISAs that detect only p24 Ag have also had limited use, especially in the United States, where there is no FDA-approved test for p24 Ag only. However, 4th-generation HIV diagnostic ELISAs have incorporated p24 antigen detection, resulting in simultaneous detection of both p24 Ag and viral antibodies (16–18). Fourth-generation assays can detect acute infections and have been shown to perform well in clinical settings (17); however, additional testing of reactive specimens is required to distinguish specimens that are positive only for antigen. Moreover, these ELISAs require a venous blood draw, a laboratory setting, and skilled laboratory technicians, and there are issues related to turnaround times and patient follow-up limiting their practical use for the detection of acute infections.

To address these issues, p24 Ag detection has recently been incorporated into a rapid test format to detect both p24 Ag and HIV antibodies as distinct lines and thereby, theoretically, shorten the diagnostic window period. A rapid test capable of detecting acute infections as well as seropositive infections would allow individuals to know of their infection earlier and in real time (11). Additionally, an Ag/Ab rapid test can be performed by nonlaboratorians in testing and counseling facilities where HIV testing currently occurs. The Determine HIV-1/2 Ag/Ab combo rapid test is the first such test that detects both Ag and Ab in two separate indicator lines. Early data on the performance of the Determine combo test to detect acute HIV infections in seroconversion panels suggested that it had a good performance in detecting early infections (19, 20). However, a number of recent publications indicate that the Determine combo test performs poorly in accurately detecting acute infections (Ag positive [Ag⁺]/Ab negative [Ab⁻]), both in laboratory and in field evaluations (21–28). Several of the studies were limited by small sample size and failure to follow up potential acute infections for seroconversion.

We present results on the performance of the Determine combo test in detecting acute infections in the Swaziland HIV Incidence Measurement Survey (SHIMS), a large, nationally representative household survey conducted from late 2010 to 2011. Swaziland has the highest HIV prevalence (>30% among adults 18 to 49 years old) in the world, along with a high HIV incidence which was estimated to be 2.4% in 2011 (29). The detection of acute infections by the Determine combo test was compared to NAAT results, and acute infections identified by either the Determine combo or NAAT were followed up to confirm seroconversion.

MATERIALS AND METHODS

Study design.

Participants from this study were recruited as part of the Swaziland HIV Incidence Measurement Survey (SHIMS), a longitudinal incidence measurement cohort in which a nationally representative household survey was conducted to measure baseline HIV prevalence and incidence in Swaziland. To identify eligible participants for the longitudinal cohort, a cross-sectional household survey was conducted from December 2010 to June 2011, with key biologic, demographic, and risk behavior information collected from adults from 18 to 49 years old, including HIV test results. Blood draws and HIV tests were conducted by trained nurses, and interviews were conducted by trained counselors/interviewers. The study design, sample size, eligibility criteria, and other survey details have been described elsewhere (30). Briefly, a two-stage cluster sampling scheme was used to achieve a nationally representative sample of households; a total of 14,891 households were approached, and 12,571 households participated in the baseline cross-sectional survey (94.3% response rate). From these households, 18,172 individuals consented to participate in the cross-sectional study.

Specimen collection and processing.

A phlebotomy-trained nurse collected approximately 10 ml of whole blood from each participant in 2 separate EDTA tubes (Greiner Bio-One, Germany), a 9-ml and a 2-ml tube, respectively. The sample in the 2-ml tube was used for HIV rapid testing in the household and for quality assurance (QA) testing at the National Reference Laboratory (NRL) in Mbabane, while the sample in the 9-ml tube was processed into plasma, aliquoted into 1.2-ml aliquots, and stored at −70°C at the NRL for further testing. All specimens were collected, transported in coolers with freezer packs, processed, and stored within 24 h of collection time to ensure specimen integrity for molecular testing. Quality control checks, such as temperature monitoring and time spent in transit, were performed throughout the specimen transportation process to ensure the integrity of the whole-blood samples. Specimens that were unable to meet the 24-h rule were either discarded if the sample was heavily hemolyzed or processed but excluded from NAAT.

HIV testing algorithm.

For SHIMS, an HIV diagnosis was determined by using a serial testing algorithm approved by the Swaziland Ministry of Health for use in the study (Fig. 1). All HIV rapid testing was performed in the household at the time of blood collection by well-trained testers according to the manufacturers' instructions. Testers were also required to use quality control specimens and participate in a proficiency testing program to ensure their competency. The Determine HIV-1/2 Ag/Ab combo rapid test (Inverness Medical, Japan) was used as the screening test. There were four possible outcomes for the Determine combo test, as follows: (i) Ag⁺/Ab⁻, (ii) Ag⁺/Ab⁺, (iii) Ag⁻/Ab⁺, and (iv) Ag⁻/Ab⁻. All specimens with an antibody-positive result (2nd and 3rd outcomes) were further tested using the Uni-Gold Recombigen HIV rapid test (Trinity Biotech, Ireland). Specimens with an antigen-positive-only result (1st outcome) were subsequently tested for viral load to confirm an acute infection, while negative specimens (4th outcome, Ag⁻/Ab⁻) had additional pooled NAAT performed to detect acute infections. Specimens with Ab results that were discordant in the Determine combo and Uni-Gold rapid tests were resolved using a testing algorithm with two HIV enzyme immunoassays (EIAs), with Bio-Rad Genscreen HIV-1/2 V2 (Hercules, CA) as the screening EIA and Vironostika HIV-1 uniform II plus O (bioMérieux, France) as the confirmatory EIA. All EIA testing was performed at the National Institute of Communicable Disease (NICD), National Health Laboratory Services, Johannesburg, South Africa, according to the manufacturers' instructions.

FIG 1.

Flow chart showing summary of SHIMS HIV testing algorithm and results. Determine HIV-1/2 Ag/Ab RT was used as the screening test, followed by Uni-Gold RT as the confirmatory test for all specimens found to be Ab⁺ by Determine combo. Specimens with discordant results were further tested by EIA to confirm HIV status. Samples with Determine combo Ag⁺-only results had viral-load testing performed to confirm acute infections, and participants were followed up at 6 weeks for confirmation of HIV seroconversion. All Determine combo Ag⁻/Ab⁻ specimens were combined in pools of 10 and tested by NAAT for detection of acute infections. Positive pools were deconstructed, and individual specimens were tested for viral RNA. All NAAT-positive samples were retested by Determine combo, and all participants with positive NAAT results had a 6-month follow-up visit to confirm seroconversion.

Viral load (VL) quantification was performed on all specimens showing acute infection (Ag⁺/Ab⁻) in the Determine combo test, using 1.2 ml of undiluted plasma on the COBAS AmpliPrep/COBAS TaqMan system (CAP/CTM) platform and the COBAS AmpliPrep/COBAS TaqMan HIV-1 test, version 2.0, according to the manufacturer's instructions; the limit of detection (LOD) for the assay was 20 copies/ml. To detect acute infections among HIV-negative specimens, 10 plasma samples were pooled (120 μl plasma/sample) and tested with the CAP/CTM HIV-1 test, version 2.0. The LOD for the pooled testing was 200 copies/ml. Positive pools were then deconstructed, and individual samples diluted 1:2 with NAAT-confirmed HIV-negative human plasma were tested to identify the NAAT-positive sample with an LOD of 40 copies/ml. Specimens identified as displaying acute infections by NAAT were retested in the laboratory with the Determine combo test for Ag and Ab reactivity.

Follow-up testing.

All individuals identified as acutely infected by either the Determine combo rapid test (Ag⁺/Ab⁻) or NAAT had follow-up visits to confirm seroconversion. Those testing as acutely infected by the Determine combo test had a follow-up visit 6 weeks after the initial visit and had a venous blood draw (9-ml tube). Those who tested as acutely infected by NAAT had a follow-up visit approximately 6 months after the initial visit and had a 9-ml venous blood sample collected. In both cases, the whole-blood sample was transported to the NRL for testing according to the SHIMS testing algorithm. After rapid testing, blood samples were processed into plasma and stored for additional testing.

Ethical considerations.

All study participants provided written informed consent prior to the collection of data and blood samples. The study was approved by the Swaziland Ethics Committee and the Institutional Review Boards (IRBs) of Columbia University and the U.S. Centers for Disease Control and Prevention before study initiation.

RESULTS

Demographic characteristics of the SHIMS study population.

Details about the SHIMS study population have been presented elsewhere (31); a summary is presented in Table 1. Briefly, a total of 18,172 individuals from 12,571 households agreed to participate in the cross-sectional survey, of whom 7,130 (39.2%) were men and 11,042 (60.8%) were women. Of the total participants, 58.7% were between the ages of 18 and 29, while 24.5% were 30 to 39 and 16.8% were 40 to 49; the age distributions were similar for men and women. The majority of the study participants were single (55.8%), and most participants lived in urban parts of Swaziland (71.0%). Overall, 5,802 (31.9%) individuals in the study population were HIV positive, and 13 (0.1%) of 12,370 negative individuals were acutely infected at the time of the study according to NAAT. Women had a higher HIV prevalence (38.3%) than men (22.0%), and there were more acute infections in women (10 [0.1%] among the HIV-negative individuals) than in men (3 [0.05%] among the HIV-negative individuals), suggesting a higher incidence in women than in men.

TABLE 1.

Swaziland HIV Measurement Survey study population demographics and HIV results from 2010 and 2011

Characteristic	No. (%) of participantsa
	Men	Women	Total
Gender	7,130 (39.2)	11,042 (60.8)	18,172 (100.0)
Age (yr)
18–29	4,400 (61.7)	6,269 (56.8)	10,669 (58.7)
30–39	1,730 (24.3)	2,720 (24.6)	4,450 (24.5)
40–49	1,000 (14.0)	2,053 (18.6)	3,053 (16.8)
Marital status
Married/partnered	2,385 (33.7)	5,601 (51.0)	7,986 (44.2)
Single	4,701 (66.3)	5,380 (49.0)	10,081 (55.8)
Area
Rural	2,066 (29.0)	3,195 (28.9)	5,261 (29.0)
Urban	5,064 (71.0)	7,847 (71.1)	12,911 (71.0)
HIV status
Positive	1,571 (22.0)	4,231 (38.3)	5,802 (31.9)
Negative	5,559 (77.9)	6,811 (61.7)	12,370 (68.1)
Acute infection	3 (0.05)	10 (0.1)	13 (0.1)

^aCounts may not sum to the column totals due to missing data. Reported percentages exclude those with missing data.

HIV serology.

Of the 18,172 participants tested with the Determine combo test, 5,822 (32.0%) were reactive for Ab (Ag⁺/Ab⁺ and Ag⁻/Ab⁺) (Fig. 1). Uni-Gold confirmed 5,789 as Ab reactive, while 33 specimens were indeterminate and required additional testing by EIA. Thus, there was a 99.4% agreement between the Determine combo Ab detection and Uni-Gold rapid test. Of the 33 indeterminate specimens, 13 were confirmed as HIV positive using the two-EIA algorithm, and 20 were HIV negative by EIA and NAAT.

Detection of acute HIV infections.

The performance of the Determine combo test for detection of acute HIV-1 infections is summarized in Table 2. The Determine combo test classified 12 participants as acutely infected (Ag⁺ only). To confirm acute infection, specimens from these 12 participants were tested for viral load. All 12 had no detectable viral load, and 8 of the 12 participants had a 6-week follow-up visit (4 were lost to follow-up) and were found to be HIV seronegative according to the testing algorithm. However, follow-up samples from 4 of the 8 participants were repeatedly Determine combo Ag⁺/Ab⁻ reactive but were nonreactive by Uni-Gold, EIA, and NAAT, indicating false Ag⁺ results.

TABLE 2.

Ag detection performance characteristics of the Determine 4th-generation combo rapid test in the Swaziland HIV Measurement Survey from 2010 and 2011

Parameter	Value(s)
Diagnostic group (no. of participants)
True positive	0
False positive	12
True negative	12,345
False negative	13
Performance metric [% (95% CI)]a
Sensitivity	0 (0, 23)
Specificity	99.9 (99.8, 99.9)
PPV	0 (0, 24)
NPV	99.9 (99.8, 99.9)

^a95% CIs are based on the Wilson score bounds. PPV, positive predictive value; NPV, negative predictive value.

Plasma samples from 12,325 Ag⁻/Ab⁻ nonreactive Determine combo specimens were pooled and tested for viral RNA by NAAT, and positive pools were deconstructed and retested individually by NAAT. This identified 13 NAAT-positive specimens, with HIV RNA values ranging from 300 to >10,000,000 copies/ml. Repeat testing by both Determine combo and Uni-Gold in the laboratory confirmed these 13 acute specimens as Ag and Ab nonreactive by RTs. Follow-up testing of 12 of the 13 NAAT-positive individuals at 6 months demonstrated 12 seroconversions (1 individual was lost to follow-up). Therefore, the Determine combo test had a sensitivity of 0% (95% confidence interval [CI], 0, 23) and a positive predictive value (PPV) of 0% (95% CI, 0, 24) for detecting acute infections.

DISCUSSION

We present here the performance results of the Determine combo RT in detecting acute and seropositive HIV infections in a large field evaluation as part of the Swaziland HIV Incidence Measurement Survey (SHIMS). Our results confirm previous findings by others that showed the Determine combo test is unable to detect acute infections with high sensitivity (8, 21–23, 26, 32–34). In fact, in our large field evaluation, the Determine combo test was not able to identify any true acute infections (Ag detection sensitivity of 0%); rather, all the Ag⁺/Ab⁻ specimens identified were falsely positive for acute infection (PPV of 0%). Although the calculated specificity of the Determine combo was 99.9% (12 false positives from 12,350 tested), this can be misleading. The specificity (false positivity) should be evaluated in the context of sensitivity (ability to detect true acute infections), which was 0.0%, and of PPV, which was also 0%. It is important to note that EIAs that detect p24 routinely require a neutralization step to confirm positive results because of the likelihood of false positives.

Unlike its Ag detection, the Determine combo test was able to detect the Ab component for HIV infection postseroconversion with good performance, since 5,789 (99.4%) of the 5,822 reactive samples were confirmed by Uni-Gold (35–37), suggesting a high positive predictive value in a high-prevalence setting. The high concordance of the two-serial-RT algorithm to accurately identify HIV infections in a household-based survey is a reflection of the intense training of the field staff on performing the rapid tests and the QA processes. This study provides documented evidence that these QA measures can be successfully implemented in the field to ensure quality testing and accurate diagnosis (38).

The utility of a rapid test to detect acute infections remains questionable due to the very short window of acute infection and the extremely low prevalence of acute infections at any point in time. Even in Swaziland, where the HIV prevalence was nearly 32% and incidence was at 2.4% (29), only 13 (0.1%) NAAT-positive acute infections were detected from the 12,338 HIV-seronegative individuals (approximately 1 per 1,000 tested). In order for an Ag-detecting rapid test to be valuable in point-of-care testing, it must be able to detect Ag with extremely high sensitivity and high PPV to identify the few acute infections in the population. Currently, the Determine combo test does not meet these criteria.

The inability of the Determine combo test to detect true acute infections in this field study may be dependent on a number of factors. First, total IgG levels are significantly higher in Africans than in individuals from Western countries due to exposure to multiple pathogens (39). This increased IgG level may interfere with the detection of p24 antigen due to nonspecific binding to the antigen. Second, the test may lack the ability to detect non-subtype B infections. Although the p24 antigen is more conserved than envelope proteins, subtle subtype differences in Africa may lead to suboptimal performance of the test. Although we did not subtype all the HIV-positive specimens, more than 100 specimens were subtyped (data not shown) and were all found to be subtype C; this was expected, since subtype C is the predominant infecting strain in Swaziland and the surrounding region (40).

Unlike the RT format, immunoassays that detect p24 antigen incorporate a signal amplification step and have a much greater sensitivity, such that they are able to detect Ag concentrations as low as 0.1 pg/ml (41, 42). This equates to an approximately 100-fold increase in sensitivity over the manufacturer's reported lower limit of detection (LLD) of 12.5 to 25 pg/ml for the Determine combo test (19). Other groups have reported that, in fact, the LLD of the Determine combo test is 20 to 50 pg/ml (17, 18), and the test seems to perform well in laboratory evaluations, especially with subtype B panels and cell culture supernatants (4, 18). Faraoni et al. found the assay to perform optimally when viral load was >10⁷ copies/ml (33). However, among the 13 acutely infected individuals in our study, 3 with a viral load of >10⁷ copies/ml and 8 with a viral load of >10⁵ copies/ml were not detected as acutely infected by the Determine combo test.

The major strengths of our study include (i) a nationally representative household survey with a large sample size, (ii) high HIV prevalence and incidence with a high probability of finding acute infections, (iii) testing of whole-blood specimens, which represent the major specimen type for a point-of-care test such as the Determine combo test, (iv) identification of true acute infections by NAAT pooling and retesting of specimens from positive pools for Ag and Ab by both the Determine combo and Uni-Gold RT, and (v) follow-up testing of individuals with acute infections identified by both NAAT and the Determine combo test to confirm seroconversion.

Before 4th-generation rapid tests that incorporate Ag detection, such as the Determine combo test, are approved for use in HIV testing algorithms, field evaluations should be done in both high- and low-prevalence settings to evaluate the performance of these tests in detecting acute infections. The good performance in the laboratory evaluations and the poor performance in the field evaluations conducted thus far suggest that the acceptability and implementation of 4th-generation rapid tests that detect Ag should be based on both laboratory and field evaluations, rather than archived retrospective specimens, spiked specimens, and cell supernatant cultures alone. Recently, the U.S. Food and Drug Administration approved the Determine combo test for use in the United States with the claim that the test “can distinguish acute HIV-1 infection from established HIV-1 infection when the blood specimen is positive for HIV-1 p24 antigen but is negative for HIV-1 and HIV-2 antibodies” (43). However, the performance of the Determine combo in its current form suggests that acute-infection results are more likely to be false positive, especially in non-subtype B settings, requiring additional viral RNA or follow-up testing to confirm the results, with no added value. Moreover, our results demonstrate that the yield of true acute infections (NAAT positive) even in a high-incidence population is very low (13 out of 12,325 tested). Therefore, it should be recognized that attempts to detect acute infections are very resource intensive and not very productive, even if accurate.

In summary, the antigen component of the Determine combo test in a high-prevalence, high-incidence setting was unable to detect any acute infections in this subtype C population. Our substantial data add to the overwhelming body of evidence that the Determine combo test has extremely poor sensitivity to detect Ag-positive acute infections, and its PPV was very poor (0.00%). The current 4th-generation Determine combo test does not offer any advantages over the Determine 3rd-generation test; rather, it is more expensive in terms of the test kit itself, as well as the cost of the additional testing required to confirm the acute infection, which is likely to be HIV negative. A simple test that can detect acute infections with high sensitivity and accuracy would certainly be a breakthrough, but its impact on preventing transmission is questionable due to the rarity of true acute infections that can be detected, even in high-incidence settings.