Diagnostic accuracy of a point‐of‐care syphilis test when used among pregnant women in Bolivia

Abstract

Objective

To evaluate the performance of a point‐of‐care (POC) syphilis test when used in urban Bolivian maternity hospitals.

Methods

We tested 8892 pregnant women for syphilis using the Abbott Determine Syphilis TP rapid POC test and rapid plasma reagin (RPR) in the laboratory of four large urban maternity hospitals where national statistics reported a syphilis prevalence of at least 3%. Sera were stored and transferred to the national reference laboratory (INLASA) where RPR testing was repeated. When the reference laboratory staff observed a positive RPR result, a Treponema pallidum particle agglutination assay (TPPA) was performed to confirm these findings. We calculated test performance characteristics for the POC test and hospital RPR using RPR performed at the reference laboratory confirmed by TPPA as the reference standard. Participants received treatment during their initial visit based on the POC test results.

Results

The sensitivity, specificity, negative predictive value and positive predictive values of the POC syphilis test were: 91.8% (95% confidence intervals 88.4% to 94.5%), 98.5% (98.2% to 98.8%), 71.0% (66.6% to 75.2%), and 99.7% (99.5% to 99.8%), respectively. The RPR values were 75.7% (70.8% to 80.2%), 99.0% (98.9% to 99.3%), 76.9% (72.0% to 81.3%), and 99.0% (98.8% to 99.2%), respectively.

Conclusion

The Abbott Determine Syphilis TP test proved to be more sensitive than routine RPR and had comparable specificity. POC testing may be a simple way to expand syphilis screening to clinics with no laboratory facilities, improve case detection, and facilitate treatment delivery.

The World Health Organization (WHO) recommends syphilis screening at least once during pregnancy, preferably during the first or second trimester.¹ Regrettably many countries in Latin America do not routinely offer syphilis testing to women attending antenatal care (ANC), especially in rural areas. Large urban hospitals in Bolivia have laboratories and are supplied with rapid plasma reagin (RPR) and venereal diseases research laboratory (VDRL) tests to diagnose maternal syphilis. Nevertheless, the quality of results is sometimes compromised by competing operational considerations such as the difficulty implementing stringent quality control measures or concerns about containing operating costs through the purchase of low‐cost reagents. In addition, RPR testing does not always occur, and when testing is attempted only 50% of laboratory results may be returned to the providers, discouraging those women who do return and are unable to learn their results.² The Bolivian Ministry of Health (MOH) documented 3.1 symptomatic cases of neonatal syphilis per 1000 live births in 1994.³ Although the government has added syphilis diagnosis to its universal maternal and child health care insurance that guarantees free care to pregnant women and children, it is looking for a more efficient maternal syphilis diagnostic method to prevent congenital syphilis.

Traditional syphilis screening tests include non‐treponemal RPR and VDRL. These tests are generally processed in batches and require laboratory infrastructure, supplies, and trained personnel. Providers at rural health clinics with no laboratory capacity send samples to external laboratories and depend on unreliable and expensive transport and communication systems to obtain results. In some cases, weeks may go by before the provider obtains the results and can notify the patient.

Rapid syphilis point‐of‐care (POC) tests represent an opportunity to improve access to syphilis diagnosis, especially in locations where laboratory facilities and trained personnel are in short supply. Unlike RPR or VDRL, POC syphilis tests detect antibodies specific for Treponema pallidum which tend to persist despite successful treatment. However, the advantage of these POC tests is that they can be performed individually, without a centrifuge or rotary platform, and results are available in 20 minutes. The RPR reagent requires refrigeration while the POC tests can be stored at room temperature for up to two years. Introducing and utilising POC tests for syphilis screening could simplify testing procedures, decrease waiting times, and facilitate treatment delivery during the initial ANC visit.

One such POC test (Abbott Determine Syphilis TP, Abbott Laboratories, Abbott Park, Illinois, USA) is already commercially available in Latin America. According to the manufacturer, this test has a sensitivity of 92% and specificity of 100% when used with blood.⁴ Previous studies have measured sensitivity from 93.7% to 100% and specificity from 94.1% to 100%.⁵,⁶,⁷,⁸ A study in Brazil also noted low inter‐reader variability: 98% agreement among three different readers.⁵

We sought to document the performance characteristics of the Abbott Determine Syphilis TP test and routine RPR when used for pregnant women attending urban maternity hospitals in Bolivia, as compared to RPR confirmed by Treponema pallidum particle agglutination assay (TPPA) performed in a reference laboratory. Assessing the performance of these tests would allow us to consider the appropriateness of a POC test for ANC syphilis screening in urban hospitals as well as implementation in rural areas where there is no laboratory infrastructure. We hypothesised that use of the Abbott Determine Syphilis TP test could improve syphilis diagnostic coverage and treatment uptake among pregnant women across the country.

METHODS

Study population

Pregnant women attending prenatal care services at one of the four largest urban maternity hospitals in Bolivia during the morning shift (the busiest of three shifts) between January 2004 and April 2005 were invited to participate. Inclusion criteria included: (1) being pregnant; (2) 18 years of age or older; (3) seeking care at a participating urban maternity hospital; (4) not tested for syphilis previously during the current pregnancy; (5) consented to participate in the larger feasibility and acceptability study; and (6) consented to undergo a blood draw via venous puncture in addition to a finger‐stick.

Study sites

The maternity hospitals where the study took place included Hospital de La Mujer in La Paz, Hospital Los Andes in El Alto, Maternidad Germán Urquidi in Cochabamba, and Maternidad Percy Boland in Santa Cruz. Recruitment was conducted at these four maternity hospitals because they: (1) were public maternity hospitals that offer prenatal care; (2) had laboratory infrastructure and already performed VDRL or RPR tests; (3) had a minimum of 2000 prenatal care visits annually; and (4) a syphilis prevalence of at least 3% (using 2002 National Health Information System data).

Testing procedures

Upon consenting to participate, hospital laboratory staff conducted a blood draw from participants by finger‐stick using an autolancet and we put directly a drop of blood from the finger on the Abbott Determine Syphilis TP test. The reading and interpretation of results were carried out according to the manufacturer's instructions (fig 1). They collected a second blood sample via venous puncture and extracted serum. Serum was used to perform RPR (RPR_Hosp) on‐site, and a separate aliquot was frozen and sent to the national reference laboratory (INLASA). INLASA staff performed another RPR (RPR_Ref) (Bio‐Rad Laboratories, Hercules, California, USA) and confirmed positive RPR_Ref findings by TPPA (Fujirebio Diagnostics Inc, Japan).

Figure 1 Laboratory flowchart. POC, point‐of‐care; RPR, rapid plasma reagin; TPPA, Treponema pallidum particle agglutination assay.

Participants received their POC results during the recruitment visit and, when relevant, immediately offered treatment according to WHO guidelines. All pregnant women who had a positive result with the Abbott Determine Syphilis TP test or those who were found to have a positive TPPA by the reference laboratory received treatment. The TPPA results were typically available one or two weeks after samples were collected. The study protocol called for women with intermediate or inconclusive results with the Abbott Determine Syphilis TP test to also receive treatment.

The Determine Syphilis TP test was administered and pregnant women waited 15 minutes for their results according to the manufacturer's instructions. Results were read to the woman, and for positive cases, women were given a brief counselling session on syphilis and the need to undergo treatment for themselves and their newborns. Women who agreed to participate in the study were given a baby blanket to thank them for their participation.

Training and quality assurance procedures

A two‐day training was provided for all participating laboratory staff. During this training, staff reviewed RPR and TPPA procedures, learned how to conduct the Abbott Determine Syphilis TP test, and received instruction regarding data collection and management.

Laboratory specific documentation was collected and reviewed monthly from each study site.

INLASA has a quality assurance programme in place for all laboratory procedures, including RPR and TPPA.

Analysis

The sample size calculated for the larger feasibility and acceptability trial of 8924 women was determined to be more than adequate to assess the test performance characteristics of the Abbott Determine Syphilis TP rapid test. A total of 6367 women in a population with a prevalence of 3.0% is sufficient to show that the test is at least 80% as good as the reference tests, with 90% power and α of 0.05.

The “gold standard” used to determine the test performance characteristics (sensitivity, specificity, positive predictive value and negative predictive value) of the Abbott Determine Syphilis TP test and RPR_Hosp was RPR_Ref confirmed by TPPA conducted at INLASA. Syphilis cases were defined as those with positive RPR (RPR+) and positive TPPA (TPPA+) results determined by trained technicians at INLASA. This is consistent with the Bolivian MOH's definition.

Prevalence, sensitivity, specificity, negative and positive predictive values and 95% confidence intervals (CIs) were obtained using Stata 8 (StataCorp LP, College Station, Texas, USA). Exact binomial confidence intervals were used, as these are appropriate when not all specimens are tested by both reference tests. For all calculations, true positives and true negatives were defined according to the gold standard definition of a syphilis case: positive RPR performed by the reference laboratory confirmed by positive TPPA performed by the same laboratory.

Ethical considerations

Before beginning this study, procedures were reviewed and approved by the Population Council's ethics committee and permission was obtained from the Ministry of Health. The study was explained before obtaining written informed consent. Specimens were labelled with a study identification number rather than name, and specimen data, including test results, were kept in locked files.

RESULTS

Table 1 shows the accuracy measures for the Abbott Determine Syphilis TP test and for the RPR test performed in the hospital laboratories. Of note, there were no cases of indeterminate or inconclusive results using the Abbott test. The sensitivity of the Abbott test was 91.8% (95% CI 88.4% to 94.5%), while that of the RPR performed in the hospital was significantly lower (75.7%, 95% CI 70.8% to 80.2%).

Table 1 Accuracy measures for the Abbott Determine Syphilis TP test and hospital RPR.

		Gold standard*		Sensitivity (%) (95% CI§)	Specificity (%) (95% CI)	Positive predictive value (%) (95% CI)	Negative predictive value (%) (95% CI)
		True positive n = 342	True negative n = 8550
Determine Syphilis TP†	+	314	128	91.8 (88.4 to 94.5)	98.5 (98.2 to 98.8)	71.0 (66.6 to 75.2)	99.7 (99.5 to 99.8)
	−	28	8422
Hospital RPR‡	+	259	78	75.7 (70.8 to 80.2)	99.0 (98.9 to 99.3)	76.9 (72.0 to 81.3)	99.0 (98.8 to 99.2)
	−	83	8469

*Gold standard definition of a syphilis case: positive rapid plasma reagin (RPR) performed by the reference laboratory confirmed by positive Treponema pallidum particle agglutination assay (TPPA) performed by the same laboratory.

†Specimens for 8 participants were not available (lost or insufficient volume) for testing at the reference laboratory.

‡Three additional specimens were not available for testing by RPR in the hospital.

§Exact binomial confidence intervals.

The reference tests identified 342 syphilis cases among 8892 pregnant women. (Eight specimens were lost in transit or had insufficient volume upon arrival at INLASA.) Based on this definition, syphilis prevalence was 3.84% (95% CI 3.5% to 4.3%). The Abbott Determine Syphilis TP test identified 442 syphilis cases among 8892 women and estimated a higher prevalence of 5.0% (95% CI 4.8% to 5.2%). All women with a positive result using the Abbott Determine Syphilis TP test or who were found to have a positive RPR and confirmatory TPPA by the reference laboratory underwent standard treatment for late latent syphilis. No adverse events related to testing with the Abbott Determine Syphilis TP test or to treatment were observed.

Table 2 compares the accuracy measures for the Abbott Determine Syphilis TP test to those reported in the literature. The accuracy measures we calculated do not differ dramatically from those observed in previous studies. These studies also found the Determine Syphilis TP test's sensitivity and specificity to be greater than 90%.

Table 2 Publicly available accuracy measures of Determine Syphilis TP point‐of‐care test.

Location	Study design	Study sample	Sample type used with POC test	Reference standard	Sensitivity (95% CI)	Specificity (95% CI)
Bolivia (present study)	Prospective	8889 pregnant women	Whole blood from finger‐stick	RPR, positives confirmed by TPPA	91.8 (88.4 to 94.5)	98.5 (98.2 to 98.8)
Brazil*†5	Retrospective	250 TPHA+; 300 TPHA−	Stored serum	TPHA only	97.6 (94.6 to 98.8)	97.3 (94.6 to 98.6)
					98.4 (95.6 to 99.4)	95.7 (92.9 to 97.7)
					95.6 (93.2 to 96.2)	96.3 (93.4 to 98.3)
Brazil*†5	Retrospective	195 VDRL and TPHA+; 300 VDRL and TPHA−	Stored serum	VDRL and TPHA	97.9 (94.4 to 99.2)	97.3 (94.6 to 98.6)
					98.5 (95.1 to 99.4)	95.7 (92.9 to 97.7)
					96.9 (93.6 to 98.9)	96.3 (93.4 to 98.3)
Brazil6	Retrospective	63 syphilis cases; 24 with other STD; 38 syphilis seronegative	Serum	Clinical evaluation, VDRL, FTA‐ABS, TPHA, and ELISA	93.7 (84.4 to 97.9)	95.2 (86.4 to 98.7)
WHO SDImulti‐country evaluation7	Retrospective	399 TPHA or TPPA+;390 TPHA or TPPA−	Stored serum	TPHA or TPPA	97.2 (95.6 to 98.8)	94.1 (91.8 to 96.4)
Vietnam8	Prospective	291 participants: 71 syphilis cases; 92 negative cases; 128 potential cross reacting (by hepatitis A and B, HIV, malaria, pregnancy, syphilis, TB and high risk of STI acquisition)	Venous whole blood, plasma and serum	RPR confirmed by ELISA (FTA‐ABS used to resolve discordant confirmatory results)	100 (95.0 to 100.0)‡	98.6 (96.0 to 99.7)

*Performance characteristics were obtained from the same study using two different reference standards.

†Performance characteristics were calculated for three different readers.

‡Calculated using diagti commands in Stata version 9 (StataCorp LP, College Station, Texas, USA).

CI, confidence interval; ELISA, enzyme‐linked immunosorbent assay; FTA‐ABS, fluorescent treponemal antibody; RPR, rapid plasma reagin; STI, sexually transmitted infection; TB, tuberculosis; TPHA, Treponema pallidum haemagglutination assay; TPPA, Treponema pallidum particle agglutination assay; VDRL, venereal diseases research laboratory.

Discussion

The Bolivian MOH recommends a minimum sensitivity of 81% for screening tests for detecting active syphilis in pregnant women. The Determine Syphilis TP test is 91.85% sensitive and 98.5% specific in pregnant women when compared to RPR confirmed by TPPA performed by a reference laboratory. The performance of the Determine rapid syphilis test therefore meets the national standards for it to be used as a prenatal screening tool. Previous retrospective studies observed similar or slightly higher measures of accuracy,⁵,⁶,⁷,⁸ particularly sensitivity. The manufacturer notes that use of whole blood may lower sensitivity from that observed with serum,⁴ which may account for this small difference.

Furthermore, we found that the Determine Syphilis TP test was more sensitive than RPR performed in a hospital (75.7%). This means if 20 people have syphilis, five will not be detected or receive treatment using the hospital‐based RPR. It is not clear why the sensitivity of the RPR performed in the hospital was so low compared to the gold standard tests performed by the reference laboratory. We hypothesise that the laboratories in the hospitals might use outdated or poor quality reagents or suboptimal equipment, and the staff might not be adequately trained to perform the test or might have insufficient experience performing the test. This suggests that a simple POC test such as the Determine Syphilis TP test may be appropriate even in settings with laboratory capacity by minimising the dependence on reagents, equipment and staff training.

The positive predictive value of the Determine Syphilis TP test was 71.0%, suggesting that the test may over‐diagnose syphilis cases. While this is true, it is useful to put this into perspective by considering the number of congenital syphilis cases averted versus the potential risks of over‐diagnosis. In our study, there were 342 cases of maternal syphilis diagnosed by the reference laboratory, of which 259 were detected by the hospital laboratory RPR. The Bolivian surveillance system estimates that 50% of pregnant women never return to the clinic to obtain their RPR results,² meaning that with standard testing only 130 of the hospital‐RPR‐positive women would obtain treatment. Therefore, with standard RPR testing, only 38% (130/342) of maternal syphilis cases would have been detected and treated. In comparison, with the Determine Syphilis TP test, 92% (314/342) of the true syphilis cases were detected and immediately treated. Assuming that congenital syphilis occurs in approximately 30% of cases,⁹ an additional 61 cases ((314−130)/3) of congenital syphilis were averted by using the Determine Syphilis TP test compared to using hospital‐based RPR testing.

At the same time, 128 (1.4% of the tested population) were falsely diagnosed as syphilis‐positive using the Determine Syphilis TP test when RPR confirmed by TPPA was used as the reference standard. The rate of biological false positive treponemal tests has been estimated at 1% or less. Thus, among these 128 women, it is likely that only 1–2 were true false positive cases; the remainder were women with a past history of syphilis. Re‐treatment of pregnant women with a history of syphilis may be beneficial as these women may have been inadequately treated or re‐infected. The risks of a misdiagnosis are primarily adverse reaction to treatment, including anaphylaxis, and stigma and other social consequences, including possible domestic violence from their partner. While it is difficult to estimate the risk of the social sequelae, the incidence of anaphylaxis after parenteral penicillin has been reported to be 32 per 100 000 exposed patients.¹⁰ Based on these limited data, it seems that the benefits of the Determine Syphilis TP test's improved sensitivity and ability to provide treatment immediately outweigh the risks of over‐treatment.

Treponemal tests including the Abbott Determine Syphilis TP test continue to detect antibodies and react positively even after successful treatment. This limits the use of treponemal tests as an accurate screening tool, especially in high‐risk populations that have access to adequate care. This study shows that the treponemal POC test is a reasonable alternative testing method, particularly if the priorities are to: (1) increase uptake of testing by simplifying the process; (2) introduce testing in locations with no laboratory infrastructure; and (3) ensure treatment delivery. Use of this test can facilitate implementation of WHO's recommendation to build capacity on‐site for ANC screening,¹ particularly in locations that do not have the capital to construct laboratories or conduct routine technical training.

This study showed that using Abbott's Determine Syphilis TP test can improve case detection when committed laboratory staff in urban hospitals performed each test. A study assessing performance of a different POC syphilis test observed a 97.2% agreement between POC test results obtained by clinic staff in basic health facilities and POC test results obtained by reference laboratory staff.¹¹ This suggests that these accuracy measures may also represent test performance when the Abbott test is conducted by clinic staff in simple clinic settings such as rural Bolivia.

Our study had several limitations. One is that we defined the gold standard for syphilis diagnosis as a reactive RPR confirmed by TPPA. While this makes sense from a clinical standpoint, it is possible that there were false negative cases among the RPR results performed by the reference laboratory that would have been detected if we had used TPPA alone as the gold standard. We performed TPPA on a random sample of 3% of the RPR‐negative samples (216 cases) and found only one positive result. This suggests that the results would not have been significantly different if we had used TPPA as the gold standard. Another limitation of our study relates to how generalisable the results are. In other settings with highly efficient and accessible laboratories that perform RPR, the performance of the Abbott Determine Syphilis TP might not be superior to standard testing procedures. In general, however, the results are likely to be applicable to many low‐resource settings in the developing world where hospital laboratories frequently are under‐funded, understaffed and generally concentrated in urban centres.

This study has shown that Abbott Determine Syphilis TP rapid tests can perform well when used among pregnant women, for whom prevention of congenital syphilis is a primary consideration. This study has also shown that use of traditional screening tests in locations with laboratory capacity may result in substandard results. Introducing a POC test is one option. Addressing the inadequacies (insufficient supplies, poor quality reagents, irregular quality control and assurance opportunities, and lack of routine technical training) is another option, particularly in locations with laboratory capacity. Taking this indirect approach may improve the quality of syphilis screening as well as enhance overall laboratory capacity.

Key messages

• When used in a real‐world setting in antenatal clinics in Bolivia, the Abbott Determine Syphilis TP test was more sensitive than standard rapid plasma reagin performed in an urban hospital and equally specific

• The accuracy of the Abbott Determine Syphilis TP test in this prospective study was similar to that reported in other retrospective studies

• Because the Abbott Determine Syphilis TP test is a simple, point‐of‐care test, its use in low‐resource settings could dramatically increase the proportion of pregnant women tested and treated for syphilis and could have a significant impact on the prevention of congenital syphilis

Based on our findings, several strategies can be taken to improve ANC syphilis screening. We understand that the contexts for delivery of syphilis diagnosis and treatment are heterogeneous and require a portfolio of complementary solutions and recommend the following actions to the Bolivian MOH:

• Implement a POC syphilis test to improve the diagnostic coverage, especially in health care facilities where traditional syphilis tests cannot be performed.

• Improve the quality of RPR results in maternity hospitals by implementing quality control and quality assurance procedures.

• Refine the purchasing system for RPR and VDRL kits and supplies so they fulfill a minimum standard of quality and accuracy.

New developments in POC syphilis tests are on the horizon. Of particular interest is a single test that has non‐treponemal and treponemal reactivity. Availability of such a test will broaden the populations for which POC syphilis tests are appropriate. Future research interests include evaluation of such new technologies as well as the review, standardisation, and implementation of operational procedures that improve overall laboratory capacity.

Abbreviations

ANC - antenatal care

MOH - Ministry of Health

POC - point‐of‐care

RPR - rapid plasma reagin

TPPA - Treponema pallidum particle agglutination assay

VDRL - venereal diseases research laboratory

WHO - World Health Organization