Field Evaluation of Performance of Alere and Cepheid Qualitative HIV Assays for Pediatric Point-of-Care Testing in an Academic Hospital in Soweto, South Africa

ABSTRACT

Point-of-care (POC) technologies for HIV diagnosis in infants have the potential to overcome logistical challenges that delay treatment initiation and prevent improvements in morbidity and mortality. This study aimed to evaluate the performance of two POC technologies against the current standard-of-care (SOC) laboratory-based assay in South Africa, when operated by nurses in a hospital environment. Children <18 months of age who were treatment naive (excluding prophylaxis) and in whom an HIV PCR test was indicated were eligible for the study. To increase the rate of enrollment of HIV PCR-positive children, HIV-exposed neonates at high risk of mother-to-child transmission and children requiring confirmatory HIV testing were preferentially enrolled. The two POC technologies demonstrated excellent concordance, with 315 (97.8%) results consistent with the SOC result. The POC technologies yielded 102 positive and 220 negative tests each. The SOC assay had 101 positive, 214 negative, 4 indeterminate, 1 invalid, and 2 specimen-rejected results. To include the indeterminate results in sensitivity/specificity calculations, a sensitivity analysis was performed, which yielded a simulated sensitivity of 0.9904 (interquartile range [IQR], 0.9808 to 0.9904) and a specificity of 0.9954 (IQR, 0.9954 to 1.0). This study confirmed that both POC technologies can be successfully used outside the laboratory environment to yield precise sensitivity/specificity values for pediatric, including neonatal, HIV testing.

INTRODUCTION

Early infant diagnosis of HIV (EID) is essential to overcome high morbidity and mortality rates associated with perinatal HIV transmission (1). Early diagnosis enables prompt initiation of combination antiretroviral treatment (cART) for infected infants. HIV infection progresses rapidly in infants, and early initiation of cART improves survival and reduces morbidity (2, 3). There have been challenges to implementing EID and initiating infants on cART, which are largely due to logistical issues, including long turnaround times (TATs) for test results, lack of result receipts due to loss to follow-up of mothers and infants, and inadequate linkage to care (4–6). Consequently, it is estimated that only 50% of HIV-exposed infants in low-resource settings access testing (7), and many infants who are diagnosed HIV positive either experience delays in ART initiation or are not initiated at all (8, 9).

EID point-of-care (POC) technologies were developed to facilitate early diagnoses and are now commercially available. These are attractive tools that have the potential to overcome logistical challenges (9, 10). EID POC devices, operated by nontechnical staff and without laboratory infrastructure, can provide HIV test results in less than 2 h in a clinic, enabling same-day result receipt and linkage to care (11). Alere Inc. and Cepheid Inc. technologies are European Conformity-In Vitro Diagnostics (CE-IVD) marked and World Health Organization (WHO) prequalified since June 2016 (12, 13). The availability of two different POC assays for HIV detection would enable identification and subsequent confirmation of infection in a single day (14). This supports a WHO recommendation that a positive HIV test require a second HIV virological assay on another sample (15).

Birth testing of all HIV-exposed neonates was introduced in South Africa in 2015, in addition to HIV testing at 10 weeks, at 6 weeks after cessation of breastfeeding, and at 18 months of age (16). In South Africa, EID is currently conducted using molecular diagnostics at nine central laboratories in the National Health Laboratory Service (NHLS), with approximately 500 000 results issued per annum (17). While there is laboratory capacity to manage the increasing demand for EID, access to testing remains challenging in remote areas. Molecular diagnostic laboratories are not easily established, as they require complex infrastructure, sophisticated equipment, costly operations, and skilled staff. Point-of-care testing (POCT) is a potential alternative; however, performance of these EID POC technologies in South Africa requires a field evaluation before implementation can be considered. Evidence from laboratory and field evaluations is limited, and those including prototype devices are not comparable. Laboratory-based evaluations conducted on the Alere technology in South Africa (18) and the Cepheid technology in Botswana (19) and Malawi (20) indicated high sensitivity (93.3% to 100%) and specificity (99.8% to 100%) in specimens from children aged <18 months. Error rates ranged from 2% to 6% (18–20). Field evaluation studies conducted in Mozambique (21) and South Africa (22) on the Alere technology and in Tanzania (23) and South Africa (24) on the Cepheid technology reported sensitivities and specificities ranging from 92% to 100% and 95% to 100%, respectively. Higher error rates have been reported in the field evaluations, ranging from 4.8% to 12% (22–24). Two laboratory evaluations, each on approximately 90 neonates <7 days old, reported lower POCT sensitivity (93%) than those reported in older infants for Alere (18) and Cepheid (19). Similarly, a field evaluation of Alere on 291 neonates, <1 day old, reported a sensitivity of 92% (22). The sensitivities in these evaluations were based on very low numbers of enrolled infected neonates (n = 9 to 16). A recent field evaluation on neonates tested at a median age of 1 day using Cepheid POCT demonstrated 100% sensitivity based on a larger number of infected neonates (n = 30) (24). With the new WHO and South Africa recommendation to introduce birth HIV testing into existing EID approaches, the performance of the POCT in neonates is highly relevant. There is currently no head-to-head evaluation of the two regulatory agency-approved POC devices in the same field setting. The aim of this study was to evaluate the performance of two EID POC technologies against the current standard-of-care (SOC) laboratory-based assay for children <18 months of age, including neonates, when operated by nurses in a hospital environment in Johannesburg, South Africa.

RESULTS

From 1 December 2015 to 17 February 2017, 332 participants were enrolled from the 361 screened for the study. The reasons for not enrolling 29 participants are described in Fig. 1. Fourteen eligible participants refused consent due to lack of interest in the study or not wanting their child's blood to be tested or without disclosing any reason. Ten participants were enrolled and subsequently excluded from the study due to lack of blood samples taken, and incorrect consent form being signed, and the SOC test being rejected and not followed up.

FIG 1.

STARD diagram indicating the flow of participants through the study. The index test represents both POC tests operated by the study nurses. The reference standard is the standard-of-care Roche CAP/CTM v2 HIV qualitative assay performed by laboratory staff (Chris Hani Baragwanath Academic Hospital [CHBAH]). Samples with no reference standard result (a), incomplete test requisition form (n = 2) and an invalid reference result (n = 1). All three samples had negative results on follow-up testing.

Neonates <7 days old comprised 70% (225/322) of the study population, with 9% (28/322) aged between 7 and <28 days. Infants aged 28 days to <12 months and children older than 12 months comprised 16% (52/322) and 5% (17/322), respectively. From the 225 neonates <7 days old, 42/225 (19%) were considered high risk (with the mother being on antiretrovirals [ARVs] for <12 weeks). Fourteen percent (6/42) of these high-risk neonates and 4% (7/183) of low-risk neonates tested HIV PCR positive. The study population demographics are summarized in Table 1.

TABLE 1.

Demographics of the study population^a

Parameter	Value for group
	Total	HIV POCT positive	HIV POCT negative
Neonates and children
Total no. enrolledb	322	102	220
Median age, days (IQR)	1 (1–13)	79 (11–222)	1 (1–1)
Age (no.)
<7 days	225	13	212
7–<28 days	28	24	4
28 days–<12 mo	52	49	3
12–<18 mo	17	16	1
Gender (no.)
Female	168	58	110
Male	154	44	110
Mothers
Total no. enrolledb	311	102	209
Median age, yr (IQR)	30 (25–35)	27 (25–33)	31 (26–36)
PMTCT (no.)
Yes	261	65	196
No	46	33	13
Unknown	4	4	0
Median PMTCT duration, wk (IQR)	18 (0–30)	13 (0–27)	21 (14–34)
cART (no.)
<12 wk	82	46	36
Preconception	61	14	47

^aStudy participants were grouped according to their point-of-care HIV result. Four participants with inconclusive SOC results were included based on their POC result. IQR, interquartile range; PMTCT, prevention of mother-to-child transmission; cART, combination antiretroviral therapy.

^bEleven sets of twins were enrolled in the study.

The Alere and Cepheid POC devices had perfect concordance, with 315 (97.8%) results consistent with the SOC HIV PCR. POC testing yielded 102 positive and 220 negative tests each. The SOC results were 101 positive, 214 negative, 4 indeterminate, and 1 invalid, with 2 specimens being rejected. Participants whose SOC results were invalid or rejected were followed up for repeat testing. Table 2 summarizes the comparison of POC and SOC results, indicating 1 indeterminate result which was a potential false positive and 3 indeterminate results which were potential false negatives. The potential false positive had an indeterminate SOC result and a positive POC result. The potential false negatives had indeterminate SOC results and negative POC results. If the worst-case scenario is assumed, in which all of the indeterminate results are discordant with the POC results as described in Table 2, the sensitivity would be 0.9712 (95% confidence interval [CI], 0.9186 to 0.9901) and the specificity 0.9953 (95% CI, 0.9741 to 0.9992). However, on follow-up of these participants, subsequent SOC testing was concordant with the POC tests (Table 3), resulting in perfect sensitivity and specificity, which would be the best-case scenario. Excluding the indeterminate results from the analyses would also have resulted in perfect sensitivity and specificity.

TABLE 2.

Performance of POC testing^a

Scenariob	POC result	No. with SOC result:					POC sensitivity (95% CI)	POC specificity (95% CI)
		Positive	Negative	Indeterminate	Invalid/rejectedc	Total
A	Positive	101	0	1	0	102
	Negative	0	214	3	3	220
	Total	101	214	4	3	322
B	Positive	102	0			102	1.0	1.0
	Negative	0	217			217
	Total	102	217			319
C	Positive	101	1			102	0.9712 (0.9186–0.9901)	0.9953 (0.9741–0.9992)
	Negative	3	214			217
	Total	104	215			319

^aThe two POC technologies had perfect concordance.

^bA, performance of point-of-care (POC) testing compared to standard-of-care (SOC) testing; B, best-case scenario in which all POC results are assumed to be concordant for the indeterminate SOC cases; C, worst-case scenario in which all POC results are assumed to be discordant for the indeterminate SOC cases.

^cExcluded from sensitivity/specificity calculations.

TABLE 3.

Summary of indeterminate SOC cases and follow-up testing

Case	Blood sample	Age at test	Test	Result
1	1	0 days	SOC	Positive
	2	8 days	SOC	Indeterminate (CT, 39.2; RFI, 1)
			POC	Negative
	3	15 days	SOC	Negative
	4	1.4 mo	SOC	Negative
	5	7 mo	SOC	Negative
2	1	0 days	SOC	Indeterminate (CT, 35.0; RFI, 4)
			POC	Negative
	2	2.5 mo	SOC	Negative
	3	5 mo	SOC	Negative
3	1	1 day	SOC	Indeterminate (CT, 35.3; RFI, 1)
			POC	Negative
	2	1.3 mo	SOC	Negative
	3	3.3 mo	SOC	Negative
4	1	1 day	SOC	Positive
	2	21 days	SOC	Indeterminate (CT, 27.1; RFI, 4)
			POC	Positive
			VL (copies/ml)	1,810
	3	3.8 mo	SOC
			VL (copies/ml)	1,350

A sensitivity analysis was performed, in which each of the indeterminate results was randomly assigned to either positive or negative with equal probability and the sensitivity and specificity and their corresponding 95% confidence intervals were calculated. This was repeated 1,000 times, and the mean values obtained were a sensitivity of 0.9904 (interquartile range [IQR], 0.9808 to 0.9904) and specificity of 0.9954 (IQR, 0.9954 to 1.0).

The HIV viral loads (VLs) tested within 7 days of POCT ranged from 230 copies/ml to >10⁷ copies/ml (Fig. 2). In the birth testing age group (<7 days), the median log VL around the time of POCT was 5.28 (IQR, 4.45 to 5.38). In the older neonates (7 to <28 days), the median was 4.49 (IQR, 3.37 to 5.35). In infants aged 28 days to <12 months, the median log VL was 6.24 (IQR, 6.03 to 6.67), and in children older than 12 months, it was 5.80 (IQR, 5.65 to 6.02). The HIV VLs that were determined within 24 h of the POC test were compared to the POC cycle threshold (C_T) values obtained from the Cepheid technology (Fig. 3). The linear regression of the data indicates an r² value of 0.513, which is associated with a strong correlation coefficient of 0.71.

FIG 2.

Viral load results around the time of point-of care testing in participants <7 days old (n = 7) (A), 7 to <28 days (n = 19) (B), 28 days to <12 months (n = 37) (C), and 12 months to <18 months (n = 12) (D).

FIG 3.

Comparison of Cepheid POC C_T values and viral load results, where POC and viral load samples were collected within 24 h of each other (n = 53).

Errors were observed with both technologies and included device and operator errors. These errors returned no result rather than an incorrect result. Alere had an error rate of 3.3% (n = 14) and Cepheid of 2.1% (n = 7). Operator errors on Alere included no sample being detected (n = 3), the cartridge not being properly locked (n = 3), cartridge misalignment (n = 2), and too little sample in the cartridge (n = 2). Device errors on Alere included a connection error between the controller and processor (n = 3) and failure to read the barcode (n = 1). Cepheid operator errors comprised only sample volume inadequacies (n = 3). Device errors on Cepheid included power outages (n = 3), which are common at the hospital, and a device optics error (n = 1). All specimens with errors were retested, and valid results were obtained.

The two nurses who operated the POC devices for the duration of the study reported similar experiences. For both technologies, the device and test were considered easy to use and the results easy to interpret. Advantages of the Alere technology were the shorter run times and the backup power system. Disadvantages included the capacity to run only one sample at a time and difficulties with using the capillaries to load samples. The Cepheid technology allowed four samples to run simultaneously, and the printout of results was more detailed. However, it lacked a backup power source, and the run time per test was longer.

DISCUSSION

Two WHO-prequalified POC technologies for HIV diagnosis in children demonstrated excellent performance outside a laboratory environment and operated by nontechnical staff. The two technologies were tested in parallel and had perfect concordance across 102 HIV PCR-positive and 220 HIV PCR-negative specimens from children younger than 18 months. These POC technologies showed high sensitivity and specificity compared to the SOC HIV PCR test in South Africa, supporting previous studies on the Alere q HIV-1/2 Detect (18, 21) and Cepheid Xpert HIV-1 Qual (19, 20, 23, 24) assays. Over 100 PCR-positive specimens were tested, which enabled a precise estimate of POCT sensitivity. Prior to this study, no more than 65 positive specimens had been evaluated in the field for POCT, and these did not include specimens from neonates (21). The POC assays detected HIV PCR-positive specimens in all age groups, particularly in neonates, where the VL concentrations are often low. This is evident from the comparison of VL concentrations tested within 7 days of POCT, where the median VL was lowest in the older neonates (7 to <28 days). The large number of negative results at birth (212) indicated excellent specificity in this age group, as there were no false-positive results by POCT. The VL data suggest a strong relationship between Cepheid POC C_T values and log VL. However, there were high VLs (≥log 5.8) which were associated with high C_T values (>34). This could be due to a number of factors, including PCR inhibition from high nucleic acid template concentrations or inhibitors in the sample that were not adequately extracted.

Considering the simulated sensitivity/specificity as the most objective scenario where the indeterminate results are assumed positive 50% of the time over 1,000 replicates of the analysis, the sensitivity/specificity of the POC technologies ranges from 97.12%/99.53% to 100%/100% in the case of worst and best scenarios, respectively. One hundred percent sensitivity and specificity has been reported for the Cepheid POC technology in laboratory (18) and field (20, 23) evaluations. Follow-up SOC testing on the four indeterminate cases suggests that the POC results were correct, indicating perfect concordance between POC and SOC. To further support this, the POC-negative results were associated with SOC-indeterminate C_T values of ≥35 and relative fluorescence increase (RFI) values of <5. The POC-positive result was associated with an SOC-indeterminate C_T value of 27.1 but an RFI value of 4. Reported C_T values of <32 and RFI values of ≥4 are indicative of HIV-infected infants (25). All 4 indeterminate results occurred in the neonatal age group, a time when prevention of mother-to-child transmission (PMTCT) infant prophylaxis may reduce the sensitivity of PCR tests (14), which may account for the reduced sensitivity in the neonatal period noted in previous evaluations (18, 19, 22).

Error rates were lower than reported in previous POC field evaluations (18, 23, 24). This may be due to problems experienced in earlier studies being resolved prior to initiation of this study. POCT training may also have improved since the earlier studies, to include common errors and how to avoid them. The Alere device had a slightly higher error rate than the Cepheid device; however, the frequency of errors with the Alere technology decreased when the device at the maternity unit was replaced.

The study nurses who ran the POC devices had predominantly positive feedback, with both POC technologies being considered easy to use. There were advantages and disadvantages for each device; however, the choice would depend on prior assessment of the required TAT and testing volume.

Limitations of this study include logistical challenges involved with two study nurses enrolling participants in a busy hospital environment. POCT was not offered to all eligible participants due to the high volume of HIV-exposed neonates in the maternity unit. The study did not assess the implementation or impact of POCT on TATs from specimen collection to result or time to ART initiation.

The performance and ease of use of two POC devices were evaluated. This study confirms that both WHO-prequalified POCT technologies can be successfully utilized outside the laboratory environment. The sensitivity and specificity of both POC technologies in screening for HIV infection in children <18 months of age were at least 0.9904 (IQR, 0.9808 to 0.9904) and 0.9954 (IQR, 0.9954 to 1.0), respectively. The same sensitivity and specificity applies to neonates, where both POCT technologies yielded results in keeping with the infant's final HIV infection status as determined by multiple repeat testing following the indeterminate SOC result. Implementation of POCT in the field so as to be most impactful in shortening TATs and decreasing time to ART initiation requires assessment. Placed at sites where the necessary clinical skills to initiate ART are present, the impact on HIV-positive infants' morbidity and mortality could be significant. Further confidence on test performance can be achieved through the introduction of external controls to maintain quality control.

MATERIALS AND METHODS

Study design.

This prospective study was conducted from 1 December 2015 to 17 February 2017 during offices hours on weekdays only at Chris Hani Baragwanath Academic Hospital (CHBAH) in Johannesburg, South Africa. CHBAH is a large hospital located in Soweto, southwest of Johannesburg. There are approximately 480 live births of HIV-exposed children per month at CHBAH, and neonates born by normal vaginal delivery are discharged on average at 6 h postdelivery. According to program data, the estimated intrauterine transmission rate at birth at CHBAH before study initiation, emanating almost exclusively from the maternity unit, was 1%. The POCT devices were placed adjacent to the maternity unit and the pediatric HIV clinic. The maternity unit site enrolled neonates primarily from the postnatal and neonatal wards, while the pediatric HIV clinic did so from pediatric inpatients admitted to the hospital wards and outpatients from the clinic. A single study nurse at each site enrolled participants, once informed consent from parents was obtained, and operated the POCT devices.

Ethics approval.

The study (M150326) was approved by the Human Research Ethics Committee (Medical) of the University of the Witwatersrand.

Enrollment criteria.

In the maternity unit, all HIV-exposed neonates were eligible for the study. Neonates at high risk of mother-to-child transmission (MTCT), defined as a maternal history of <12 weeks of cART prior to delivery, were preferentially enrolled. These subjects included women first diagnosed as HIV infected at delivery or who did not attend antenatal care and neonates born before arrival at the hospital. At the pediatric HIV clinic site, children aged <18 months who were treatment naive (excluding prophylaxis), who had a previous positive HIV PCR test result, and for whom a confirmatory HIV PCR test was indicated were eligible for the study.

Sample size.

The challenge for a diagnostic test study in a setting of low-prevalence disease is testing sufficient participants with the disease to achieve a precise estimate of the test sensitivity. Thus, in order to capture sufficient HIV-positive participants, the sampling strategy purposefully targeted infants and young children who were more likely to be HIV PCR positive. The target sample size for the study was 100 HIV PCR-positive and 100 HIV PCR-negative children <18 months old. With this sample size, if the test were to have a sensitivity and/or specificity of 0.97 or higher, the lower limit of the 95% CI would be 0.05 or less. Precision would increase if a larger number of subjects were enrolled.

Study methods.

An additional 300 μl of EDTA-anticoagulated whole blood was collected during routine blood draws for HIV PCR or viral load (VL) testing on eligible participants. The specimens were stored at 4°C until informed consent was obtained from the child's parent or primary caregiver and was run on both POC devices within 72 h of collection by the study nurses. The study nurses documented maternal and infant data on data collection forms that were entered onto an Excel worksheet. Participant demographics for the mother (including PMTCT history) and infant were obtained. A laboratory record review of HIV VL tests performed within 7 days of the POCT was conducted. Each nurse operated one Alere q HIV-1/2 Detect (Alere Inc., Waltham, MA, USA) and one Cepheid Xpert HIV-1 Qual (Cepheid Inc., Sunnyvale, CA, USA) POC device according to the manufacturer's instructions, outside a laboratory environment. The Alere q HIV-1/2 Detect assay was performed using an input volume of 25 μl of whole blood, with a lower limit of detection of 1,758.68 copies/ml (13). The assay has a run time of 52 min (13). The Cepheid Xpert HIV-1 Qual assay was performed using an input volume of 100 μl of whole blood, with a lower limit of detection of 350 copies/ml (12). The assay has a run time of 90 min (12). The results of the current SOC HIV PCR test in the South African EID program, the laboratory-based Roche CAP/CTM HIV qualitative version 2.0 assay (Roche Molecular Systems Inc., Pleasanton, CA, USA), were used as the reference standard against which POCT results were compared. The Roche CAP/CTM assay requires an input volume of 100 μl and has a limit of detection of 20 copies/ml, and testing was performed at an accredited HIV PCR reference laboratory on the hospital grounds by laboratory staff. Only laboratory-based HIV PCR test results were returned to participants. The study nurses collated POCT results with the laboratory result and notified pediatric staff of any discrepancies to ensure that follow-up testing and linkage to care of HIV-infected children took place. Both POC assays contain built-in internal controls for quality control. Where an internal control failed or any error occurred, retesting was done if sufficient specimen remained. Where the laboratory-based assay failed to yield a negative or positive result (for example, an indeterminate result), SOC required that the participant be recalled for an additional sample to be collected and tested. The NHLS national EID standard operating protocol defines an indeterminate HIV PCR result as a result with a detected target that has a cycle threshold (C_T) value of >33 and/or a relative fluorescence increase (RFI) of <5. These are considered valid inconclusive results requiring additional HIV testing (26).

Statistical data analysis.

Descriptive statistics were used to analyze participant demographics, using STATA version 12 (StataCorp LLC, College Station, TX, USA). POCT sensitivity and specificity were calculated with 95% confidence intervals (CI), using the Newcombe-Wilson method without continuity correction. Error rates for the POC tests were estimated as a proportion of the total number of tests run on the device. Errors were defined as POC tests that yielded no result. Reporting of results from this study adheres to the STARD 2015 guidelines for reporting diagnostic accuracy studies (27).

User questionnaire.

A user questionnaire was distributed to each study nurse to obtain information regarding their user experience with both POC devices.