Evaluation of Performance of Two Rapid Tests for Detection of HIV-1 and -2 in High- and Low-Prevalence Populations in Nigeria

Abstract

The availability of reliable human immunodeficiency virus types 1 and 2 (HIV-1/2) rapid tests in resource-limited settings represents an important advancement in the accurate diagnosis of HIV infection and presents opportunities for implementation of effective prevention and treatment interventions among vulnerable populations. A study of the potential target populations for future HIV vaccine studies examined the prevalence of HIV infections at six selected sites in Nigeria and evaluated the use of two rapid diagnostic tests (RDTs) for HIV. The populations included market workers at sites adjacent to military installations and workers at highway settlements (truck stops) who may have a heightened risk of HIV exposure. Samples from 3,187 individuals who provided informed consent were tested in parallel using the Determine (DT) and Stat-Pak (SP) RDTs; discordant results were subjected to the Uni-Gold (UG) RDT as a tiebreaker. The results were compared to those of a third-generation enzyme immunoassay screen with confirmation of repeat reactive samples by HIV-1 Western blotting. One participant was HIV-2 infected, yielding positive results on both RDTs. Using the laboratory algorithm as a gold standard, we calculated sensitivities of 98.5% (confidence interval [CI], 97.1 to 99.8%) for DT and 98.1% (CI, 96.7 to 99.6%) for SP and specificities of 98.7% (CI, 98.3 −99.1%) for DT and 99.8% (CI, 99.6 to 100%) for SP. Similar results were obtained when the sites were stratified into those of higher HIV prevalence (9.4% to 22.8%) versus those of lower prevalence (3.2% to 7.3%). A parallel two-test algorithm requiring both DT and SP to be positive resulted in the highest sensitivity (98.1%; CI, 96.7 to 99.6%) and specificity (99.97%; CI, 99.9 to 100%) relative to those for the reference laboratory algorithm.

INTRODUCTION

UNAIDS estimates that 35 million people are living with HIV worldwide, with nearly 2.1 million new HIV infections added in 2013 (1). The highest rates of infection, 24.7 million, occur in sub-Saharan Africa, where countries with high HIV burdens and low treatment coverage have shown little or no decline in new HIV infections (2, 3). Three countries, South Africa, Nigeria, and Uganda, accounted for nearly half of all new infections in sub-Saharan Africa in 2013. Although 90% of individuals who tested positive for HIV in these areas seek treatment and 76% have achieved successful viral suppression, fewer than 40% of HIV-infected individuals are aware of their HIV status (1, 4–6). More effective HIV screening strategies are critical to reduction of HIV transmission rates and a prerequisite to elimination of the spread of AIDS in these areas.

Rapid diagnostic tests (RDTs) provide an affordable, point-of-service approach for wide-scale HIV testing of populations in low-income, high-HIV-burden countries, which lack the financial and technological resources to perform more sophisticated laboratory-based assays (7, 8). These tests are extensively used at various levels of health care in the rapid scale-up of HIV prevention and treatment services under the U.S. President's Emergency Plan for AIDS Relief (PEPFAR) in Nigeria (9, 10). The Determine (DT), Stat-Pak (SP), and Uni-Gold (UG) rapid HIV diagnostic tests have been shown to meet the World Health Organization (WHO) minimum sensitivity (99%) and specificity (95%) requirements for HIV screening (11–14). Nevertheless, unacceptably high rates of false-positive (FP) results in RDT screening of large populations have led to many individuals still being misdiagnosed (15–18). The WHO guidelines, therefore, recommend stringent selection and adaptation of HIV testing algorithms to retest positive specimens with an independent HIV screening test (19, 20). Improved specificity with a significant reduction of false-positive rates has been achieved with the use of orthogonal algorithms based on combinations of two or more RDTs which target a different set of antigens/antibodies or employ different test mechanisms to minimize the probability of sharing factors that lead to false-positive or false-negative (FN) results (14, 21–23).

Due to their low cost, ease of use, and reliable performance, whole-blood RDTs are the standard of care and basis of national testing algorithms in many resource-constrained countries. Combinations of the SP, DT, and UG RDTs are used in Uganda, Malawi, Zambia, Nigeria, and other countries (4, 24, 25). Ideally, the suitability of HIV control strategies and algorithms for screening specific populations should be evaluated in the appropriate context of test availability, antigen/antibody targets of each test, local seroprevalence, genetic diversity, and risk of infection in the populations being examined as well as local operational factors and laboratory qualifications (26–28).

We examined a convenience sample set from an institutional review board (IRB)-approved study, originally designed to evaluate the suitability of populations and sites for HIV vaccine cohort development. The RDT data generated by the study were used to evaluate the performance of an orthogonal RDT screening algorithm in areas of high and low HIV prevalence in Nigeria where subtypes in circulation differ from those in neighboring countries and consist largely of subtype G and CRF02_AG and G/CRF02_AG recombinants (28–30). Field testing results from a parallel DT and SP algorithm, employing UG as a tiebreaker, were compared to results with use of a more rigorous reference laboratory algorithm consisting of a third-generation enzyme immunoassay (EIA) reflexed to Western blotting and nucleic acid testing.

MATERIALS AND METHODS

The study sites included workers in markets contiguous to Nigerian Military Medical Centers at four locations in the north and south of the country, representing a cross-section of local populations. Two highway settlements (truck stops), which included female commercial sex workers, hotel workers, bar workers, and food sellers who come in regular close contact with long-distance drivers represented the populations at higher HIV risk. The sites were selected to be reflective of the HIV subtype distribution throughout various regions of the country and included (i) Makurdi, (ii) Abuja, (iii) Enugu, (iv) Kaduna, (v) Tafa, and (vi) Ojo Lagos. A total of 3,187 volunteers, representing more than 500 individuals per site, who provided informed consent were included in this study. Although the number of individuals receiving antiretroviral therapy (ART) in this population is expected to be low, the original study design did not collect information concerning prior HIV test history, infection status or therapy. A subsequent modification, which was implemented only at site 6 (a high-prevalence site), revealed that out of 500 individuals who completed the study at that site, 44 had been previously tested for HIV infection; of these, 4 individuals were HIV-1 infected and indicated they were receiving ART.

All samples were originally screened in parallel for HIV by two independent RDTs: the Determine (DT) HIV-1/2 (Alere Medical Company Limited, Chiba, Japan) and the HIV-1/2 Stat-Pak (SP) dipstick (HIV-1; Chembio Diagnostic Systems, Medford, NY). Discordant results were resolved using a tiebreaker test, the Uni-Gold (UG) HIV (HIV-1; Trinity Biotech, Bray, Ireland). Although all 3 kits use antigen targets to the same general regions of the HIV-1 (gp41/gp120) and HIV-2 (gp36) genomes (31), specific targets were independently selected by the manufacturers and are not expected to be identical. Whereas the Stat-Pak uses specific peptides, Determine uses a combination of peptides and recombinant antigens for capture of antibodies, which are expected to allow for independent confirmation of signal response. This algorithm was consistent with the approved HIV screening algorithms that were used in Nigeria at the time of the study (25). All RDTs were conducted on fresh EDTA anticoagulated whole-blood samples, according to the manufacturers' directions. Serum and EDTA plasma samples collected from each participant at the time of the RDT screen were processed, reposed at −140°C, and forwarded to the HIV Diagnostics and Reference Laboratory (HDRL), U.S. Military HIV Research Program, Walter Reed Army Institute of Research (WRAIR) (Silver Spring, MD, USA) for comprehensive reference laboratory testing.

The reference laboratory test algorithm employed an initial screen with a third-generation HIV-1/HIV-2 Plus O EIA (Bio-Rad Laboratories, Redmond, WA). Reactive samples were repeated in duplicate, and repeat reactive (RR) samples were reflexed to supplemental HIV-1 confirmatory testing using Genetic Systems (GS) HIV-1 Western blotting (WB) (Bio-Rad Laboratories). The HIV-1/HIV-2 MultiSpot (MS) rapid test (Bio-Rad Laboratories) was used for serotype differentiation of HIV-1 from HIV-2 specimens. EIA RR specimens that were negative (Neg) or indeterminate (IND) on HIV-1 WB were reflexed to the Aptima HIV-1 RNA qualitative assay (Gen-Probe/Hologic, San Diego, CA) or the HIV-1 real-time quantitative HIV RNA (Abbott, Chicago, IL) assay for resolution of the HIV-1 infection status classification and to rule out acute HIV infection (AHI). An HIV-2 RNA real-time laboratory-developed test (LDT) (limit of quantification, 2.0 log₁₀ [100] copies/ml) clinically validated for HIV-2 infection monitoring was used to confirm HIV-2 infection. An EIA nonreactive specimen was classified as an HIV-negative case. An HIV-1-positive (Pos) case (the gold standard) was defined as RR by an EIA and either HIV-1 WB positive or HIV-1 WB Neg/IND but positive for HIV-1 RNA by the Roche Cobas AmpliPrep/Cobas TaqMan HIV-1 test (>20 copies/ml) or reactive by the Aptima qualitative RNA assay (>5 copies/ml). An HIV-2-positive case was defined as RR by an EIA, as HIV-2 reactive by MultiSpot, and as HIV-2 RNA positive (>100 copies/ml) by the HIV-2 RNA real-time LDT. The performance of the individual RDTs and their use in combination algorithms were evaluated in terms of sensitivity, specificity, positive (PPV) and negative predictive values (NPV), and positive and negative likelihood ratios relative to those of the laboratory test standard for testing populations with lower and higher prevalences of HIV.

Ethical considerations.

All study participants were enrolled after completing an IRB-approved informed consent process, and HIV counseling and testing were offered after enrollment and an baseline interview. The study was approved by the WRAIR IRB and by the National Health Research Ethics Committee of Nigeria (NHREC). Specimens were transported outside Nigeria for further testing as specified in the IRB-approved study protocol under an NHREC-approved materials transfer agreement (MTA).

RESULTS

Bio-Rad HIV-1/2/O EIA-reactive samples confirmed by Western blotting at the 6 sites ranged from 3.2% to 22.8% (Table 1). The number of DT-reactive samples tended to run higher than those reactive with SP or by EIA confirmed by WB (Fig. 1). The numbers of EIA-confirmed positive samples shown in Fig. 1 include 3 samples (one each from sites 1, 2, and 5) which were WB indeterminate but RNA positive, indicative of acute HIV infection (AHI). Only a single participant was identified as HIV-2 infected by MultiSpot HIV-2 positivity; the HIV-2 viral load was 5.18 log₁₀ copies/ml. This sample was DT and SP positive, EIA repeat reactive, and HIV-1 WB positive.

TABLE 1.

HIV positivity observed at six test sites in Nigeria as detected by EIA repeat reactivity, HIV-1 WB, or RNA confirmation or reactive DT or SP rapid tests

Site	No. tested	HIV positivity with:
		EIA/WB/RNA		DT		SP
		No.	%	No.	%	No.	%
1	534	50	9.4	60	11.2	50	9.4
2	549	40	7.3	44	7.9	38	6.9
3	567	18	3.2	28	4.9	18	3.2
4	554	27	4.9	30	5.4	30	5.4
5	491	112	22.8	116	23.6	111	22.6
6	492	77	15.7	78	15.6	76	15.4
Total	3,187	324	10.2	356	11.4	323	10.1

FIG 1.

Percent repeat reactivity by the Bio-Rad HIV-1/2 O EIA as confirmed by HIV-1 Western blotting or HIV-1 or HIV-2 RNA detection (gold standard) compared to Determine (DT) or Stat-Pak (SP) test results at sites 1 to 6 and at all sites combined.

An additional analysis of RDT performance relative to the reference laboratory algorithm was conducted by stratifying the study population into either high-prevalence (HP) sites (sites 1, 5, and 6 with 15.7% prevalence; confidence interval [CI], 13.9 to 17.7%) or lower prevalence (LP) sites (sites 2, 3, and 4 with 5.08% prevalence; CI, 4.10 to 6.28%) (Fig. 3). Of 1,517 HP site participants, 236 (15.6%) were confirmed as HIV-1 infected by either WB or RNA detection. Eighteen samples positive by DT and two by SP were not reactive by EIA. Included among the confirmed positive samples were two samples (one each from sites 1, 2, and 5) which were HIV-1 WB IND with detectable HIV-1 RNA (318, 41, 365, and >10,000,000 copies/ml, respectively), thus, presumably within the acute HIV infection stage. Both the DT and SP RDTs misclassified the site 1 acute-stage sample as negative. The site 2 and 5 acute-stage samples were positive by DT but negative by SP. UG, used as a tiebreaker, was negative for the two discordant samples. All 20 EIA-negative/RDT-positive samples tested negative for HIV RNA and therefore were classified as false positive (FP), yielding 1.3% FPs for DT and 0.31% FPs for SP. Figure 2 depicts the analysis of results if a two-test algorithm is used, wherein both the DT and SP rapid tests must be positive to confirm HIV infection. Since there was no overlap of FP samples by the parallel two-test algorithm, the effective FP rate for this sample set relative to that for the reference laboratory test result was 0%. The use of UG as a tiebreaker would not have increased the number of true positives (TPs) detected but would have contributed one FN since the test scored positive for a DT-Pos/ST-Neg sample, which was WB Neg and RNA Neg.

FIG 3.

Number of false-positive and false-negative results, when DT and SP are used independently or in combination (DT-SP). False-positive samples are those not confirmed by HIV WB or RNA detection. False-negative samples are those that were EIA RR and confirmed as HIV positive by HIV-1 WB or RNA detection but misclassified by the RDT.

FIG 2.

Results of laboratory tests and RDTs for the higher and lower prevalence sites. Results of DT or SP independently or in combination are compared to those of the reference laboratory algorithm. Calculations of sensitivity, specificity, positive likelihood ratio, and negative likelihood ratio of each test or in combination for correct classification of HIV infection are shown.

A similar analysis of the low-prevalence sites revealed that 82/1,670 samples were true positives as confirmed by HIV-1 Western blotting and RNA detection. DT and SP identified 102 and 86 positive samples, respectively. Twenty DT-positive samples and 4 SP-positive samples, including 1 sample that was positive by both RDTs, were negative by the HIV-1/2/O EIA and therefore were not tested by Western blotting. All 24 samples were negative for HIV-1 RNA detection by the qualitative RNA assay (sensitivity of 5 copies/ml) and were therefore classified as FPs. DT misclassified 3 samples that were EIA RR and HIV-1 WB positive as negative. The same 3 samples were also misclassified as HIV negative by the SP test; thus, the overall FN rate was 3.5% for this sample set. The low-prevalence population included a single acute/early HIV infected case, which was EIA RR and WB IND, with detectable RNA. The sample was classified as positive by both DT and SP. The tiebreaker test (UG) was Neg for all DT/SP discrepant samples from the low-prevalence sites.

The statistical analysis of results in Fig. 2 also shows the sensitivity, specificity, PPV, NPV, positive and negative likelihood ratios, pretest probability, and posttest positive and negative probability calculations of the two RDTs used separately or in combination relative to those for the reference laboratory algorithm. DT sensitivity was higher than that of the Stat-Pak at both the HP (98.7% [CI, 98.0 to 100%] and 98.3% [CI, 97.3 to 100%], respectively), and LP (96.5% [CI, 92.6 to 100%] versus 96.5% [CI, 92.6 −100%]) sites. The specificities of SP at the HP and LP sites were 99.8% (CI, 99.6 to 100%) and 99.7% (CI, 99.5 to 100%), respectively, which were better than those for DT (98.6% [CI, 98.1 to 99.2%] and 98.7% [CI, 98.2 to 99.3%]). The PPVs for DT and SP were 92.9% (CI, 89.8 to 96.1%) and 99.2% (CI, 98.8 to 100%), respectively, at HP sites, with lower values observed for the LP sites (80.4% [CI, 72.7 to 88.1%] and 95.3% [CI, 90.9 to 99.88%]), respectively. The PPV was greatly improved when the parallel two-test algorithm (DT + SP) was employed at HP sites (100% [CI, 98.0 to 100%]) or at LP (98.8% [CI, 96.5 to 100%]) sites. Similarly, the likelihood ratios positive for DT increased from 70 and 77 at the HP and LP sites and 631 and 382 for SP at the HP and LP sites to infinity or 1,529 when both tests were used in combination. The NPVs for both assays were >99.7%, with very low negative likelihood ratios (<0.04), indicative of excellent ability to detect a true negative.

Figure 3 provides an alternative analysis of the results in terms of misdiagnosis as defined by the rates of false-positive and false-negative results from the DT and SP screens compared to those from the two-test RDT algorithm at HP and LP sites. DT gave false-positive rates of 1.4% at the HP sites and 1.3% at the LP sites, while SP false-positive rates were 0.16% at HP sites and 0.25% at LP sites. The false-negative results for DT were 1.26% and 1.67% at the HP and LP sites, with those for SP at 3.53% at all sites. The orthogonal two-test algorithm results in a dramatic decrease in the false-positive results: no false positives were detected at the HP site and a single false positive result was detected at the LP site corresponding to false positives of 0 and 0.06%, respectively. The false-negative sample rates at the higher and lower prevalence sites were 1.67% and 3.53%, respectively. These results further emphasize the value of utilizing a two-test orthogonal algorithm as a highly reliable measure of infection in these populations.

DISCUSSION

The present study, which was designed for identifying populations for HIV vaccine cohort development, provided the opportunity to evaluate performance of RDTs on a convenience sample from 6 sites within Nigeria. Based on these results, we identified 3 sites with high HIV prevalences of 9.4% to 22.8% where populations were considered at higher risk for HIV acquisition and 3 sites with HIV prevalences of 3.2% to 7.3% in populations at somewhat lower risk. The two-rapid-test algorithm, consisting of DT as the initial screen, with SP, a more specific test for confirmation, demonstrated overall specificities of 100% and 99.9%, respectively, in both high-risk and lower risk populations. This orthogonal algorithm eliminated all FPs in the high-prevalence group (0%) and all but one (0.1%) FP in the lower prevalence group. Thus, the orthogonal combination of DT and SP demonstrated excellent sensitivity (97.8%) and specificity (>99.9%) with a high level of confidence that individuals with positive results were correctly diagnosed, approximating the results of a reference laboratory employing third-generation EIA screening with more specific supplemental serological and nucleic acid confirmation.

The FN samples, representing HIV-infected individuals who were misclassified by RDT, included 6 individuals misclassified by DT and 7 by SP, yielding a 1.7% overall FN rate for the higher and 3.5% for the lower prevalence groups. The two-rapid-test algorithm showed very little increase in FN results over that of either test alone, with only a single true positive (an individual with an acute HIV infection [AHI]) detected by DT alone eliminated. One of the samples misclassified by SP was an early AHI sample. It should be pointed out that the laboratory reference algorithm used in these studies was based on a very sensitive antibody assay, but very early infection prior to evolution of the antibody, which would only be detectable by RNA or antigen assays, would have been missed. As such, a true FN could not be accurately assessed in this study.

As all EIA RR samples not confirmed by supplemental HIV-1 WB as well as all EIA-nonreactive samples reactive by DT or SP were subjected to the highly sensitive qualitative nucleic acid HIV-1 RNA assay and were found to be RNA negative, FP samples were highly unlikely to have been improperly classified in this study. However, since only third-generation antibody-based assays were used by the reference laboratory algorithm, AHIs where only HIV-1 RNA and/or HIV-1 p24 antigen is detected would have been missed in this study. Thus, we were not able to accurately access the number of true FN samples, an important limitation of this study.

A single HIV-2 infection (0.03%) was identified in this study. Of note, the HIV-2 viral load was high (5.17 log₁₀ copies/ml) for this benign infection. The appearance of HIV-2 on HIV-1 Western blots with profiles consistent with those of HIV-1-positive specimens underscores the utility of MultiSpot and the HIV-2 viral load to confirm HIV-2 infection. This sample was correctly identified as HIV positive by both rapid tests.

The results reported here are consistent with those of several similar studies of the use of RDTs for HIV screening of high-prevalence populations in Africa. Previous studies with DT and ST in sub-Saharan Africa also revealed higher sensitivity for DT relative to that for ST (11), UG (15), or OraQuick (17) but lower specificity. In the present study, the UG tiebreaker resolved only a single discrepant sample, which was WB IND with undetectable HIV-1 RNA, and therefore not confirmed by the laboratory algorithm. As such, the use of UG as a tiebreaker would have incorrectly classified this sample as positive, thereby increasing the number of FPs and decreasing the specificity of the two-test algorithm. In this analysis, there was no advantage to the inclusion of UG as a tiebreaker to resolve discrepancies. These results are consistent with those previously published in which the use of UG for resolution of discrepant results resolved only a small subset (29 of 291 = 10%) of discrepant samples (15). The original study design, however, did not permit us to evaluate the utility of UG as a second-line test after the DT primary screen with SP as the tiebreaker as is currently used in a number of countries, including Nigeria.

For cost considerations, implementation of a two-test algorithm is usually in the form of serial screening, in which the second test is used only for those samples which were classified positive by the first test. The higher sensitivity of DT positions this RDT most appropriately as the initial screening test, while the higher specificity of SP positions it as a more suitable second-line test. However, if positivity on both DT and SP is a required criterion for classification as RDT positive and UG is not being used as a tiebreaker, the order of the tests is less important.

In line with the WHO guidelines, this study supports the use of DT as the initial screening test with SP for independent confirmation in a serial orthogonal algorithm (14, 17, 20). In this study, comparison of the RDT results with those of well-established laboratory tests showed excellent agreement; DT was slightly more sensitive (fewer false negatives) but less specific (more false positives) than SP. The orthogonal parallel test algorithm employed in this study significantly reduced FPs with a minimum increase in FNs, thus minimizing the number of people who were incorrectly diagnosed as HIV infected. This study also serves to highlight the fact that knowledge of the principle of a test and/or the antigens/antibodies employed for target detection is critically important for the development of highly sensitive, specific, and effective test strategies.