The accuracy of HIV rapid testing in integrated bio-behavioral surveys of men who have sex with men across 5 Provinces in South Africa

Abstract

We describe the accuracy of serial rapid HIV testing among men who have sex with men (MSM) in South Africa and discuss the implications for HIV testing and prevention.

This was a cross-sectional survey conducted at five stand-alone facilities from five provinces.

Demographic, behavioral, and clinical data were collected. Dried blood spots were obtained for HIV-related testing. Participants were offered rapid HIV testing using 2 rapid diagnostic tests (RDTs) in series. In the laboratory, reference HIV testing was conducted using a third-generation enzyme immunoassay (EIA) and a fourth-generation EIA as confirmatory. Accuracy, sensitivity, specificity, positive predictive value, negative predictive value, false-positive, and false-negative rates were determined.

Between August 2015 and July 2016, 2503 participants were enrolled. Of these, 2343 were tested by RDT on site with a further 2137 (91.2%) having definitive results on both RDT and EIA. Sensitivity, specificity, positive predictive value, negative predictive value, false-positive rates, and false-negative rates were 92.6% [95% confidence interval (95% CI) 89.6–94.8], 99.4% (95% CI 98.9–99.7), 97.4% (95% CI 95.2–98.6), 98.3% (95% CI 97.6–98.8), 0.6% (95% CI 0.3–1.1), and 7.4% (95% CI 5.2–10.4), respectively. False negatives were similar to true positives with respect to virological profiles.

Overall accuracy of the RDT algorithm was high, but sensitivity was lower than expected. Post-HIV test counseling should include discussions of possible false-negative results and the need for retesting among HIV negatives.

1. Introduction

The HIV epidemic among men who have sex with men (MSM) in South Africa is characterized by high prevalence,^[1–3], high incidence,^[4,5] and a relatively large proportion of undiagnosed HIV infection in communities across the country.^[3] While current World Health Organization recommendations suggest that key populations at a high risk for HIV infection seek HIV testing services (HTS) at least quarterly, HIV surveillance data have shown that relatively low proportions of MSM test regularly, even at half-yearly intervals.^[3,6] Consequently, a primary focus of HIV programming for MSM in South Africa is increasing regular uptake of HTS as a component of the universal test-and-treat strategy being implemented in the country.^[7,8] At present, MSM HTS programming initiatives in South Africa aim to expand access and improve uptake of HTS through mobile clinics and alternative testing sites, such as at MSM community events including sports contests or talent shows with accelerated linkage to HIV care and treatment with the advent of the test-and-treat strategy.^[9] Another key intervention targeting the MSM population is the provision of pre-exposure prophylaxis (PrEP).^[6,8] PrEP involves the provision of antiretroviral drugs to eligible and willing HIV-negative individuals in order to prevent HIV acquisition.^[10] HTS is required before initiation and on a regular basis during PrEP. Consequently, the number of HIV rapid tests conducted among MSM is expected to increase dramatically, as the country strives to achieve its 90–90–90 benchmarks for HIV treatment and expand the use of biomedical prevention strategies including PrEP. It is therefore critical that rapid testing kits employed demonstrate optimal sensitivity and specificity to minimize error in screening and diagnosing HIV across diverse HTS modalities.

Most rapid HIV tests currently in use in the field are third-generation antibody tests capable of detecting antibodies to the HIV virus 4 to 6 weeks following infection. On the contrary, fourth-generation HIV tests—both laboratory based and rapid—are capable of detecting both HIV 1/2 antibodies and the HIV-1 p24 antigens. As a result, they are expected to be more sensitive during acute and early HIV infections than the third-generation assays. However, the rapid combination antibody/antigen tests such as the Alere HIV 1/2 Combo or SD Bioline HIV Ag/Ab Combo has been found to have reduced sensitivity in acute infection, although they may still increase the number of individuals with acute and early infections detected over and above what would be detected by third-generation testing alone.^[11–13]

Opportunities to evaluate sensitivity and specificity of rapid HIV testing in the field are rare; integrated biological and behavioral surveillance surveys (IBBS) employing both rapid and laboratory-based enzyme-linked immunosorbent assay (ELISA) testing offer one such opportunity. In 2014 to 2015, we conducted the South Africa Health Monitoring Survey with MSM (SAHMS-MSM) in 5 MSM communities: Johannesburg, Cape Town, and Mangaung metros, and in Capricorn (Polokwane) and NM Molema (Mahikeng) district municipalities. The primary aims of this IBBS survey were to estimate the prevalence of HIV and risk behaviors, and assess utilization of available HTS and HIV treatment services in each of the communities; these results will be fully reported in a forthcoming publication. A secondary aim was to assess the sensitivity and specificity of the rapid diagnostic tests (RDTs) employed at the survey sites for those MSM who elected to know their HIV status, against the reference-standard (third and fourth-generation ELISA assays), which established HIV infection in samples. In this paper, we report and discuss the results of the sensitivity and specificity analyses of the HIV rapid testing employed at these 5 survey sites.

2. Methods

2.1. The SAHMS-MSM survey

The SAHMS-MSM survey collected behavioral and biological data from all participants in each of the 5 stand-alone survey sites. Participants were aged 16 years or older; born biologically male; and reported anal or oral sex with another male within the preceding 6 months. Sexual orientation (e.g., gay, bisexual) and gender identity or expression (e.g., feminine, transgender) were neither inclusion nor exclusion criteria.

SAHMS-MSM recruited participants between June 2015 and April 2016. Local preparations determined the exact launch date for recruitment; once launched, recruitment continued over 24 weeks at each site following standard respondent-driven sampling procedures described by Heckathorn,^[14,15] Salganik and Heckathorn,^[16] and employed in multiple IBBS surveys with MSM and other key populations throughout the world. Briefly, each site selected “seed” participants to begin a chain-referral process that was limited and tracked by means of recruitment coupons that were valid for up to 4 weeks. Seeds were given 3 recruitment coupons to distribute to MSM peers whom they knew. These individuals were then asked to present to the survey site with their coupon. Staff verified coupon validity and participant eligibility before enrollment in a single survey visit for behavioral and biological assessment (described below). At the conclusion of the visit, participants were given 3 coupons to distribute to their peers to further propagate the recruitment chains. Participants were invited to return to the survey site after 4 weeks to collect any secondary incentives for successfully recruiting their peers to which they were entitled. The participants received a primary incentive of ZAR 100 (US $15) cash at the conclusion of their enrollment visit and survey participation, and a secondary incentive of ZAR 30 (US $4) in the form of grocery store vouchers for each successful referral (up to 3 referrals). In addition, participants were reimbursed up to ZAR 30 (US $4) for travel costs to the survey site. The protocol stipulated that recruitment at each site would continue through the enrollment of at least 500 participants enrolled, or until such time as investigators determined that the survey samples had reached saturation, that is, the point at which further recruitment would not impact the composition of the sample with respect to key demographic or behavioral indicators.

2.2. Data collection procedures

Demographic, clinical, and behavioral data were collected using a standardized interviewer-administered, computer-assisted questionnaire based on model instruments successfully used across sub-Saharan African countries and adapted to the South African context. Survey domains comprised data on demographics, behaviors potentially correlated with HIV infection and other sexually transmitted infections (STIs), symptoms of STIs among MSM, as well as on HIV-related knowledge, attitudes, and practices; experiences of stigma and discrimination; perceptions of sexual risk and risk behaviors; and HIV testing and care-seeking behaviors.

2.3. Staff training for HIV testing and counseling

All counselors and site coordinators received standard 1-day training from the National Institute for Communicable Diseases (NICD) before the commencement of recruitment. The training covered all aspects of quality assurance and quality control for HIV rapid testing using a standardized curriculum. The training facilitated by 2 trainers commenced with a written pre-training assessment test for each participant that provided the trainers an assessment on the participants’ knowledge of HTS issues. Besides the theoretical aspects of HTS, the training also included a practical component where counselors worked on HTS, quality assurance, and quality control measures under the supervision of the trainers. All staff wrote an end-of-training assessment test.

2.4. Rapid HIV testing procedures

At the conclusion of the behavioral questionnaire, all participants were offered rapid HIV testing with same-day results return. All those consenting to rapid testing provided a blood sample obtained through a lancet fingerprick conducted by lay HTS counselors. Rapid HIV testing followed the National Department of Health recommended testing algorithm comprising 2 RDTs in series (see Fig. 1).^[7] Samples were first tested on Alere Determine HIV-1/2 Rapid Test (Alere Medical Co.Ltd, Matsudo-Shi Chiba, Japan) [RDT1]. RDT1 nonreactive samples were reported to participants as an HIV-negative result. RDT1 reactive samples were confirmed with Abon HIV 1/2/0 Tri-Line Rapid Test (Abon Biopharm, Hangzhou, R.R China) [RDT2]. Confirmed RDT1 and RDT2 reactive samples were reported to participants as an HIV-positive result, and participants were referred to a local clinic for HIV care. RDT2 nonreactive samples were reported to the participants as discrepant results. Participants with discrepant rapid HIV test results were referred to a local clinic for a whole blood specimen collection and diagnostic laboratory based ELISA testing.

Figure 1.

HIV testing algorithms.

2.5. Rapid HIV testing quality control and assurance

A site coordinator was responsible for implementing quality assurance and control measures at each site. Specimens with known antibody reactivity to HIV [internal quality control (IQC)] were routinely used to validate that the test devices were working before testing clients. Failure of the IQC would prevent further testing until the problems were resolved. The NICD provided the samples for IQC. The positive and negative IQC contained heat-inactivated human plasma that had been defibrinated with CaCl₂ and preserved with bronidox. Multiple ELISA and western blot testing identified each control sample as either “Negative” or “Positive.” At each site, IQC for rapid screening and confirmatory test devices was done under the supervision of the Site Coordinator once a week, for example, every Monday or at any other time when new shipment of test kits or control materials were received at the testing site, beginning with a new lot number, opening a new test kit box, and environmental conditions exceed range needed for stability of test kits, for example, temperature. All the results of the IQC process were recorded on the Quality Control Record Sheet. Also, daily room temperature logs were also recorded at site level and reviewed for any possible effects on test-kits.

2.6. Laboratory HIV testing and quality assurance

Regardless of whether participants accepted rapid testing, all were requested to provide a dried blood sample via lancet fingerprick collected on dried blood spot (DBS) cards (Whatman 903 Protein Saver Cards; Sigma-Aldrich, St Louis, MI) and sent for laboratory testing in batches. The effect of moisture on DBS was closely monitored on a daily basis through checking the humidity cards and corrective measures carried out before transportation and submission of the DBS to the laboratory. The DBS cards were tested for HIV-1 using a third-generation HIV ELISA (Genscreen^T HIV1/2 version 2; Bio-Rad, Marnes-la-Coquette, France) as the screen test (EIA1). If the result was nonreactive, it was regarded as HIV negative. A fourth-generation ELISA, Vironostika, HIV Ag/Ab Assay; bioMérieux, Marcy-I’Etoile, France, was used to confirm any reactive ELISA result (EIA 2)—see Fig. 1. ELISAs were optimized for DBS specimens to ensure optimal performance. For Genscreen, the cut-off value for a nonreactive or reactive result was based on the (mean of 3 controls)/10. For Vironostika, the cut-off value for a nonreactive or reactive result was based on the (mean of 3 controls) + 0.100. A grey zone was defined as being 10% below or above the cut-off value. All specimens within this cut-off were termed equivocal and retested. If the result remained equivocal, the specimens were retested on a different assay. If the results were nonreactive on the confirmatory test, the final call was discrepant. All discrepant results were tested by western blot. NICD had in place all required quality assurance measures and controls and participated in recognized proficiency testing program for DBS for all tests listed. Due to the potential of increased false-reactive results, especially for DBS specimens, optimization of ELISAs with DBS and confirmation of testing using GS HIV-1 Western Blots (Bio-Rad Laboratories, Redmond, WA) was performed on specimens that were reactive by both ELISAs and those that were discrepant as described above. The laboratory staff were blinded to the site RDT results.

2.7. Variables and outcomes

The main outcome of this substudy was the accuracy of the RDT algorithm and this was defined as the overall proportion of individuals tested with an HIV RDT on site who had the correct HIV result on the reference fourth-generation EIA. Other outcomes determined were the sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), false-positive rate, and false-negative rate of the HIV RDT algorithm compared with the reference standard of laboratory-based EIA algorithm. As the RDTs used in the survey were antibody-only HIV testing assays and were expected to be less sensitive compared with fourth-generation EIA that detect for antibody and p24 antigen, their performance against an alternative reference standard, western blot assay (GS HIV-1 Western Blot assay; Bio-Rad Laboratories, Redmond, WA) was also evaluated.

2.8. Data analysis and statistical methods

The population tested by an HIV RDT was described using descriptive statistics—median and interquartile ranges (IQRs) for continuous variables as well as counts and proportions for categorical variables. The outcomes as described were determined as proportions with 95% confidence intervals (CIs) around the estimates. In order to assess any potential effects of virological status on the sensitivity of RDT, participants who tested false negative on RDT were compared with the true positives who were correctly diagnosed by RDT with respect to median viral load and proportions with viral loads >10,000 copies/mL. Wilcoxon rank sum and Chi-squared tests were used to assess statistically significant differences between these groups.

2.9. Ethical considerations

The SAHMS-MSM protocol was approved by the University of Cape Town Human Research Ethics Committee, the University of California, San Francisco Committee on Human Research, and the Associate Director of Science at the US Centers for Disease Control and Prevention, Atlanta. All participants provided written informed consent to enroll in the survey and to have rapid HIV testing done on site. Relevant permissions were also obtained from the provincial departments of health.

3. Findings

3.1. Description of survey participants tested and testers

A total of 2503 participants were enrolled in the survey across the 5 sites. Of these, 2343 (93.6%) tested were for HIV by RDT (Fig. 2). The median age of the participants tested was 25 years (21–29 years). The number of participants tested ranged between 359 (15.3%) at the Limpopo site and 545 (23.3%) at the Gauteng site. From laboratory testing, overall HIV positivity in this population was 23.4% (see Table 1). Before the commencement of recruitment, a total of 18 lay counselors and interviewers were trained across the 5 sites. All counselors passed the posttraining assessments and none had failed IQC procedures during the recruitment period.

Figure 2.

Study flow.

Table 1.

Characteristics of participants who tested by Rapid Diagnostic Algorithm (N = 2343).

3.2. HIV results

Of the 2343 tested by RDT, 533 were reactive while 1810 were nonreactive (Fig. 2). From the 533 who were reactive and required a confirmatory rapid HIV test, 138 were not tested with RDT2 [135 of these were from the Gauteng site and were excluded because they were known HIV positives and self- reported taking antiretroviral therapy (ART)]. Of the 395 RDT1-reactive specimens tested with RDT2, 389 of the second tests were reactive and therefore considered HIV positive, while 6 were nonreactive and were therefore discrepant rapid HIV test results. Discrepant rapid HIV testing results totaled 1.1% of the RDT1-positive tests and 0.3% of all tested.

As Fig. 2 shows, of 2343 tested by a RDT, 2322 (99.1%) had a DBS specimen tested for HIV in the laboratory (388 of the 389 RDT positive, all 6 who were RDT discrepant, all 138 who did not have RDT 2 done, and 1790 of the 1810 RDT negative). The proportion of participants who had a DBS specimen tested in the laboratory was greater than 99% at all sites except the NW location (97.9%). Of the 2322 DBS specimens tested by EIA, 579 were reactive and 10 were equivocal on EIA1 and eligible for confirmatory EIA2 testing, with the remaining 1733 nonreactive and considered HIV negative by EIA. On EIA2 testing, 548 (23.4%) were HIV positive (376 of the 388 RDT2 positives tested by EIA in the laboratory, 5 of the 6 were RDT discrepant, 137 of the 138 not tested by RDT 2 on site, and 30 of the 1790 who were RDT negative on site) and were considered true positives. There were 41 who were discrepant on EIA testing (2 out of 388 tested RDT 2 positives who were reactive on EIA 1 and nonreactive on EIA 2 and 29 of the 1790 RDT negative tested were reactive or equivocal by EIA 1 and non-reactive on EIA 2 and 10 who were EIA1 indeterminate). Western blot testing confirmed that the true positives were positive and 37 of those with discrepant results were either negative or indeterminate and likely HIV negative.

3.3. Performance of RDT algorithm compared with laboratory EIA

In order to determine the performance of the RDT algorithm on site against the laboratory-based EIA, 2137 participants who had complete results on both RDT and EIA algorithms were included in the analysis [excluding 206 participants with i) incomplete testing on RDT (n = 138), ii) discrepant RDT results (n = 6), iii) not tested by EIA (n = 21), and iv) discrepant EIA results (n = 41)]. This represented 91.2% of the 2343 tested by RDT on site and 92% of the 2322 tested by EIA in the laboratory. This proportion was greater than 94% for all sites except Gauteng where this proportion was 73% of those tested by RDT1.

The overall accuracy of the RDT against fourth-generation EIA algorithm was high at 98.1% (95% CI 97.5–98.6) and varied slightly across the sites ranging from 97.1% at the North West site to 99.3% at the Gauteng site (X²P = .245). The sensitivity and specificity were 92.6% (95% CI 89.6–94.8) and 99.4% (95% CI 98.9–99.7), respectively, while the PPV and the NPV were 97.4% (95% CI 95.2–98.6) and 98.3% (95% CI 97.6– 98.8), respectively. The false positive rates and false negative rates were 0.6% (95% CI 0.3–1.1%) and 7.4% (95% CI 5.2–10.4%) respectively (see Table 2). The performance of the RDT algorithm was similar when an alternative reference standard of laboratory EIA and western blot testing (equivalent to third-generation HIV testing) was used (Table 2).

Table 2.

Performance of the rapid test algorithm against fourth-generation EIA assay.

3.4. Comparison of false negatives to true positives

Although the false-negative rate was 7.4% overall, this rate ranged from 3.0% at the Gauteng site to 14.5% at the North West site (Fischer exact P = .06). On comparison of participants who were falsely classified as HIV negative by RDT algorithm (n = 30) to the ones who were correctly classified as HIV positive by RDT (n = 376), the false negatives did not differ from the true positives with respect to the median viral load, proportions with detectable virus as well as proportions with viral load greater than 10,000 copies/ml (Table 3).

Table 3.

Immunological and virological profiles of false negatives compared with true positives.

4. Discussion

This paper describes the performance of a serial rapid testing algorithm in a population of MSM who participated in an integrated behavioral and biological survey at 5 surveillance sites in South Africa. The overall accuracy of the RDT algorithm (the proportion of individuals who received the correct positive or negative result) was high at 98%, but sensitivity was low at 92.6% with a false-negative rate of 7.4%. The false-positive rate was low at 0.6%. RDT performance was similar across all facilities, although the proportion of participants who completed both laboratory and site RDT algorithms was lower at 1 site than the others. Participants who were falsely negative on rapid HIV testing did not differ from those who were correctly identified as HIV positive, suggesting that virological status was unlikely to be a significant contributor to the diminished sensitivity observed.

The need for an accurate HIV diagnosis among MSM and other high incidence populations cannot be overstated. MSM are a priority population for HTS and accelerated entry into HIV care and treatment in South Africa, given the low rates of testing, linkage, and retention in care. The Department of Health in South Africa has targeted MSM as a priority population for the scale up of pre-exposure prophylaxis (PrEP) and has set up demonstration sites in collaboration with implementation partners.^[8] PrEP requires HIV testing to exclude HIV infection before initiation and regular ongoing testing to avoid dual therapy and the development of drug resistance that may arise if newly infected individuals continue taking PrEP. A false-negative diagnosis of HIV infection may therefore lead to delays in accessing care and treatment, risk of drug resistance on PrEP, and inadvertent transmission to others given an HIV-negative diagnosis. A systematic review of HIV rapid test performance compared with fourth-generation HIV testing found a pooled sensitivity of 94.5% slightly higher than what was observed in our study.^[17] This reduced sensitivity is concerning and more needs to be done to ensure greater accuracy of rapid HIV testing in these settings. In addition to continued implementation of quality assurance measures, the use of nucleic acid amplification tests for those who test rapid HIV test negative should also be considered.

A few factors may have contributed to the reduced sensitivity observed during rapid HIV testing at the survey sites. First, the sites used the serial testing algorithm without a third test as tiebreaker. The absence of a tiebreaker has been associated with low false-positive rates but marginally higher false-negative rates.^[18,19] Second, testing errors may have occurred. There were documented errors in following the algorithm with 138 participants positive on the first rapid HIV test (RDT1) not getting a confirmatory test (RDT2). There may have been other testing errors that were not documented.^[20] In the literature, clerical errors and not allowing enough time before reading off results on the testing devices have been shown to be associated with misdiagnosis of HIV infection.^[18,19] The implementation of strong quality assurance measures, including practical training, IQC, and on site supervision in the survey, is likely to have minimized this. Third, because of high HIV incidence documented among MSM, some participants may have recently acquired HIV infection and may have had low antibody titers, hence the low sensitivity. This was unlikely given that there were no acute infections detected among those who were falsely negative. Lastly, some participants enrolled in the survey may have been on ART. Although ART exposure data were not available, the high proportions of both false negatives and true positives who had undetectable viral load suggest that some participants may have been known HIV positives on ART. The highest sensitivity was observed at the one site that had known HIV-positive clients on ART excluded from the analysis. A few studies have documented low antibody titers and even seroreversion with long-term ART initiated during acute or early infection and continued long term with sustained viral suppression.^[21,22]

Our substudy was based on a survey of MSM at 5 surveillance sites in the country. The survey used respondent-driven sampling until saturation in order to enroll a large group of MSM. Our findings are therefore likely representative of the MSM populations seeking care within public health services in the country. HIV testing was conducted at facilities by lay counselors who were well trained and supervised. However, our substudy also had some limitations. The lack of consistently self -reported or laboratory-verified antiretroviral exposure data limits the extent to which the low sensitivity can be attributed to ART use. In addition, lack of detailed information on outcomes of quality assurance measures and errors detected during such activities also means that these cannot be entirely excluded as factors contributing to the low sensitivity.

In conclusion, although the rapid HIV testing algorithm was able to identify most HIV positive participants, there was low sensitivity with a high rate of false-negative results. Nucleic acid based HIV testing could be considered for excluding HIV infection among MSM populations taking PrEP and who test negative on rapid HIV testing. Post-HIV test counseling should also include discussion of the false-negative results and the need for retesting among the HIV negatives. Research into the effect of long-term ART on rapid HIV test performance should be encouraged.

Acknowledgment

The authors acknowledge Ushmita Patel, Zinhle Brukwe, Centre for HIV and STI.