Laboratory evaluation of a smartphone-based electronic reader of rapid dual point of care tests for antibodies to human immunodeficiency virus and Treponema pallidum infections

Abstract

Background

Dual point of care tests for antibodies to human immunodeficiency virus (HIV) and Treponema (T.) pallidum allow for same-day testing and treatment and have been demonstrated to be cost-effective in preventing the adverse outcomes of HIV infection and syphilis. By recording and transmitting data as they are collected, electronic readers address challenges related to the decentralization of point of care testing.

Methods

We evaluated a smartphone-based electronic reader using 201 sera tested with two dual rapid tests for detection of antibodies to HIV and T. pallidum in Los Angeles, USA and Lima, Peru. Tests were read both visually and with the electronic reader. Enzyme immunoassay followed by Western blot and T. pallidum particle agglutination were the reference tests for HIV and T. pallidum, respectively.

Results

The sensitivities of the two rapid tests for detection of HIV were 94.1% and 97.0% for electronic readings. Both tests had specificity of 100% for detection of HIV by electronic reading. The sensitivities of the two rapid tests for detection of T. pallidum were 86.5% and 92.4% for electronic readings. The specificities for detection of T. pallidum were 99.1% and 99.0% by electronic reading. There were no significant differences between the accuracies of visual and electronic readings, and the performance did not differ between the two study sites.

Conclusions

Our results show the electronic reader to be a promising option for increasing the utility of point of care testing programs.

Introduction

Despite the success in the treatment of human immunodeficiency virus (HIV) infections, there are still approximately 2 million new HIV infections annually [1]. Although not garnering similar attention, Treponema pallidum is responsible for 5.6 million new syphilis infections in adults each year [2]. While syphilis is a curable disease, when left untreated, it can cause long-term sequelae such as neurological problems, heart abnormalities, increased risk of HIV infection, and death [3]. For pregnant women, syphilis infection can lead to miscarriage, stillbirth, neonatal death, premature birth, low birth weight, and maternal to child transmission [4]. The global burden of HIV and syphilis remains high in part due the difficulty of reaching people with testing and treatment in remote areas, which precludes the prevention of further transmission to the sexual partners and children of those who are infected. Point of care (POC) testing presents an opportunity to reach those people with diagnostics that are affordable, easy to use, and rapid, allowing for same day testing and treatment. POC testing at rural clinics has been shown to correctly diagnose and treat more patients than either onsite or offsite laboratory testing for syphilis [5]. In addition to remote areas, POC testing is important in any clinical setting where rapid results are necessary but strong laboratory systems are not in place. Dual testing for HIV and syphilis antibodies has been demonstrated to be particularly cost-effective in treating and preventing the adverse outcomes of those infections [6]. Point of care HIV/Syphilis dual tests have been found useful in many settings, including antenatal care [7], among female sex workers [8,9], among high risk men who have sex with men [9,10], among transgender women [9,10], and in rural clinical settings [7].

Smart-phone based electronic readers have recently emerged as a promising technology to improve the utility of POC tests. While other imaging technologies may be expensive or difficult to transport, smartphones are small and light, relatively inexpensive, and ubiquitous in the modern world. Furthermore, electronic readers have the potential to address one of the main challenges of POC testing: decentralization. Challenges with large scale data collection are likely to be common in the same environments where POC testing will be particularly useful. By recording and transmitting data as they are collected, electronic readers can help monitor quality assurance, supply chain management, and real time disease surveillance [11]. Smartphone based imaging has been demonstrated for detection of HIV infection, syphilis, leprosy, malaria, Hepatitis B, helminthic infections, and giardia [12–17].

The Cellmic smartphone electronic reader consists of an opto-mechanical attachment and an Android application (HRDR-200, Cellmic, LLC., California, USA) [18, 19]. See Figure 1 for an image of the Cellmic smartphone reader. The attachment illuminates rapid tests, and a photograph is taken using the smartphone camera. The application then reads and quantifies the image to deliver results. Those results are available immediately to the user and also transmitted wirelessly to a cloud-based database. Unlike some other electronic readers, the HRDR-200 is not associated with any specific test, and can be adapted to any lateral or downward flow assay.

Figure 1.

Cellmic HRDR-200 smartphone based electronic reader.

Methods

We evaluated the Cellmic reader’s accuracy using two lateral flow assays, the SD BIOLINE HIV/Syphilis Duo (SD Bioline) (Standard Diagnostics, Korea) and First Response HIV/Syphilis Combo Card Test (First Response) (Premier Medical Corporation, Ltd., India). We tested 201 stored (−80°C) sera with First Response and 199 of those sera with SD Bioline tests. One hundred thirty-eight specimens were from a previous study evaluating rapid diagnostics in Los Angeles, California [20]. Sixty-three specimens were from high-risk patients enrolled in a longitudinal study on HIV and syphilis infection among Peruvian men who have sex with men and transgender women in Lima, Peru [21]. Sera were selected to generate a relatively equal amount of positive and negative results from reference tests, but were randomly chosen within those categories. T. pallidum particle agglutination (TP-PA) and rapid plasma reagin (RPR) titers were not considered for sera selection, nor were they analyzed in the evaluation. All sera from Lima had RPR titers of 1:4 or 1:8. Data on RPR titers were not available for the sera from Los Angeles. Both tests were performed and read on sera at both sites by one researcher. Other team members performed the reference tests. Researchers were blinded to the previous results on the samples they were testing.

Reference testing for T. pallidum was performed using TP-PA (SERODIA-TPPA, Fujirebio Diagnostics, Japan). Reference testing for HIV was performed using 4th generation enzyme immunoassays (EIAs) (Siemens Advia Centaur HIV 1/O/2 enzyme immunoassay, Siemens, Tarrytown, NY for tests performed in Los Angeles and Genscreen ULTRA HIV Ag-Ab, Bio-Rad, France for tests performed in Lima) followed by confirmatory Western blots for EIA positive sera (GS HIV-1 Western blot kit, Bio-Rad, Hercules, CA for tests performed in Los Angeles and NEW LAV BLOT I, Bio-Rad, France for tests performed in Lima).

All rapid tests were read both visually and with the electronic reader. The electronic reader produces both quantitative and qualitative readings of the rapid test for each infection: a numerical value indicating the intensity of the test line and a positive or negative determination based on whether this test value is above or below a cut-off value. Because the HRDR-200 can be used with any lateral flow-based assay, a calibration is needed to set the specific cut-off values for each rapid test. Once a cut-off value has been determined, it can be programmed into any Cellmic reader that will read the same rapid test. For this study there were no pre-determined cut-off values because no previous tests had been performed with the electronic reader on the SD Bioline and First Response tests, and we wanted to find the optimal performance of the reader. To do so, we recorded each test value and generated Receiver Operating Characteristic (ROC) curves using Stata/IC 14.1 (StataCorp, College Station, TX) to determine cut-off values for each test and infection to maximize sensitivity and specificity. Using the reference tests as the comparison, sensitivity and specificity were calculated for the visual and electronic readings of each rapid test for each infection. We used the laboratory reference tests for both visual and electronic readings in order to determine whether one reading style was more accurate than the other. We used the binomial method to calculate 95% confidence intervals.

Results

Of the 400 rapid tests, one First Response test had an unclear control line, which the electronic reader read correctly as invalid. The reader incorrectly read two First Response tests and three SD BIOLINE tests as having invalid controls. The sensitivities and specificities for the tests are listed in Table 1, stratified by infection and reading type. Tests which were invalid are excluded from the table. There were no significant differences between the accuracy of the visual and electronic readings, although the sensitivity of visual readings was higher than their electronic counterparts. There was one specimen for which the visual reading gave a false negative and the electronic reading gave a true positive. All other false rapid tests results were either in both the visual and electronic readings or only the electronic reading. Both tests had specificities of 100% for both infections read visually. Two false positives were read by the electronic reader, one for T. pallidum indication on the SD Bioline test and one for T. pallidum indication on the First Response test, resulting in specificities of 99.0% and 99.1%, respectively. Among sera for which RPR titer data was available, there was no significant difference between sensitivity or specificity of sera with titers or 1:4 and those with titers of 1:8.

Table 1.

Sensitivities and specificities of the SD BIOLINE HIV/Syphilis Duo and the First Response HIV/Syphilis Combo Card Test with visual and electronic readings

Test	Antibody detected	Reading type	True positives	True negatives	Sensitivity (95% CI)	Specificity (95% CI)
SD BIOLINE HIV/Syphilis Duo	HIV*	Visual (n=199)	102	97	97.1% (91.6–99.4%)	100% (96.3–100%)
		Electronic (n=196)***	101	95	97.0% (91.6–99.4%)	100% (96.2–100%)
	Treponema pallidum**	Visual (n=199)	92	107	93.5% (86.3–97.6%)	100% (96.6–100%)
		Electronic (n=196)***	92	104	92.4% (85.0–96.9%)	99.0% (94.8–100%)
First Response HIV/Syphilis Combo Card Test	HIV	Visual (n=200)***	103	97	95.2% (89.0–98.4%)	100% (96.3–100%)
		Electronic (n=198)***	102	96	94.1% (87.6–97.8%)	100% (96.2–100%)
	Treponema pallidum	Visual (n=200)***	91	109	91.2% (83.4–96.1%)	100% (96.7–100%)
		Electronic (n=198)***	89	109	86.5% (77.6–92.8%)	99.1% (95.0–100%)

^*HIV reference standard: 4th generation enzyme immunoassays (EIAs) (Siemens Advia Centaur HIV 1/O/2 enzyme immunoassay, Siemens, Tarrytown, NY for tests performed in Los Angeles and Genscreen ULTRA HIV Ag-Ab, Bio-Rad, France for tests performed in Lima) followed by confirmatory Western blots for those EIA positive (GS HIV-1 Western blot kit, Bio-Rad, Hercules, CA for tests performed in Los Angeles and NEW LAV BLOT I, Bio-Rad, France for tests performed in Lima).

^**T. pallidum reference standard: T. pallidum particle agglutination (SERODIA-TPPA, Fujirebio Diagnostics, Japan).

^***Sample size excludes tests read as invalid, by either visual or electronic reading. Three SD Bioline tests were incorrectly read as invalid by the electronic reader. One First Response test was read as invalid by both visual and electronic reading. Two First response tests were read as invalid by electronic reading only.

There were also no significant differences in the accuracy of the visual and electronic readings between Los Angeles and Lima. The electronic reader cut-off values generated by the ROC curves did not shift between analyzing the first 138 tests from Los Angeles and the following 63 from Lima, for neither test nor infection. See Figures 2a–d for ROC curves for each infection and rapid test combination.

Figure 2.

Figure 2a: Receiver Operating Characteristic curve for detection of antibodies to HIV with an electronic reading of the SD BIOLINE HIV/Syphilis Duo.

Figure 2b: Receiver Operating Characteristic curve for detection of antibodies to Treponema pallidum with an electronic reading of the SD BIOLINE HIV/Syphilis Duo.

Figure 2c: Receiver Operating Characteristic curve for detection of antibodies to HIV with an electronic reading of the First Response HIV/Syphilis Combo Card Test.

Figure 2d: Receiver Operating Characteristic curve for detection of antibodies to Treponema pallidum with an electronic reading of the First Response HIV/Syphilis Combo Card Test.

Discussion

We found both the SD Bioline test and the First Response test to have very good sensitivities and specificities for detection of antibodies to HIV and T. pallidum when using an electronic reader. The electronic reader demonstrated no significant difference compared with visual readings, indicating that the reader could be used in the field for accurate, timely and streamlined data collection. While use of the electronic reader did not increase case detection, other advantages of an accurate electronic reader may include the ability to perform automatic real-time epidemiological surveys, consistency in test use and reading across locations, and supply chain management for testing and treatment.

The optimal electronic reader cut-off values remained constant between the two locations, indicating that the cut-off values generated by a smaller sample size will hold true for larger data sets. It is especially important to note that the cut-off values remained the same despite sera being collected from two different demographic groups and processed in different laboratories in different countries. Those findings suggest that multiple HRDR-200 readers can have utility in many settings without a need for recalibration for each location or population.

Although not the focus of our study, these data confirm previous studies evaluating the laboratory performance of the SD BIOLINE HIV/Syphilis Duo which found sensitivities and specificities for HIV antibody detection in sera ranging from 97.9–100% and 99.0–100%, respectively and T. pallidum antibody detection sensitivities and specificities ranging from 93.0–100% and 96.0–100%, respectively [7, 8, 10, 20, 22–25]. Although no previous evaluations of the First Response HIV/Syphilis Combo Card Test have been published, our results are well within the ranges found for other dual HIV/syphilis point of care tests using sera with sensitivities and specificities for HIV antibody detection ranging from 95.7–100% and 91.9–100%, respectively, and for T. pallidum antibody detection ranging from 46.4–97.0% and 93.8–100%, respectively [7–10, 13, 20, 22–27].

Three other point of care electronic readers for rapid tests have been evaluated. The DPP® Micro Reader (ChemBio Diagnostics Inc., Medford, NY) is specifically designed to accompany Chembio’s Dual Path Platform (DPP) rapid tests, including tests for STIs and tropical diseases. The Chembio DPP HIV-syphilis assay was evaluated using visual and electronic readings on stored sera [13]. The Chembio reader demonstrated sensitivities and specificities of 100% and 100%, respectively, for HIV antibody detection and 94.0% and 99.7%, respectively, for T. pallidum antibody detection compared to laboratory tests. There was no significant difference between the accuracies of the visual and electronic readings. The Deki Reader (Fio Corporation, Toronto, Canada) is similar to the Cellmic reader in that it can be used with a variety of rapid tests, including those for HIV and syphilis. The Deki reader has been evaluated with the SD Bioline Malaria Antigen Pf/Pv (Standard Diagnostics, Korea) for detection of Plasmodium falciparum and P. vivax indicating malaria [28]. There was no significant difference between the accuracies of the visual and electronic readings for the Deki reader. A smartphone “dongle” was designed by bioengineers at Colombia University that also has utility for detection of HIV and syphilis infection. It is an enzyme linked immunosorbent assay (ELISA)-like test for HIV antibodies, treponemal antibodies, and non-treponemal antibodies for T. pallidum [16]. The dongle is different than the Cellmic, Deki, and Chembio readers in that it performs the testing and imaging on one device without the option for visual interpretation. The dongle demonstrated sensitivities and specificities of 100% and 87% for HIV antibody detection, 92% and 92% for treponemal antibody detection, and 100% and 79% for non-treponemal antibody detection using diluted whole blood. The performance we found for the Cellmic reader is similar to that of the Chembio and Deki readers as all three did not have significantly different results than a visual interpretation of the tests they read. Although the dongle is a different style of reader with 100% sensitivity, the rapid tests and Cellmic reader performed with better specificity.

Our evaluation is limited by the samples and population studied. Rapid tests were performed using sera, which had been stored and previously characterized. It is possible the thawing and refreezing of the samples may have affected the likelihood of detecting antibodies. Additionally, while the tests are designed for use with sera or whole blood, their optimal use is with finger prick whole blood to allow for real-time field-based testing. Few evaluations of dual tests have been performed with finger prick whole blood [9, 10]. An evaluation of the rapid tests and electronic reader should be repeated in a field setting using finger prick whole blood. Although we found the cut-off points to be the same for the two populations we studied, another study with more study sites and a larger sample size would better evaluate whether pre-determined, uniform cut-off points would be feasible for large scale use of the reader. A larger evaluation using samples with varying RPR titers may also reveal additional information about the accuracy of the rapid tests and reader compared to activity of syphilis infection.

We were also limited by the narrow focus on measuring accuracy. Further research is needed to determine the performance, utility and optimal features of electronic readers for point of care tests. Further research should also be done on the feasibility and acceptability of the Cellmic reader among providers and patients in clinical and field settings.

In conclusion, the SD Bioline and First Response tests demonstrated very good sensitivities and specificities for the detection of antibodies to HIV and syphilis with both visual and electronic readings. The HRDR-200 reader interpreted the rapid tests with similar accuracy as a visual reading, indicating its potential as a tool for accurate large-scale data collection and results monitoring. The cut-off values were the same for both study sites in Los Angeles and Lima, suggesting that Cellmic readers could be pre-programmed with cut-off values for various tests for large-scale use. Our results show the Cellmic reader to be a promising option for increasing the utility of point of care testing programs.

Summary.

Evaluation of an electronic reader for point of care dual HIV/syphilis testing in Los Angeles, USA and Lima, Peru found the electronic reader to be as accurate as visual readings.