Evaluation of the diagnostic performance and operational characteristics of four rapid immunochromatographic syphilis tests in Burkina Faso

Abstract

Background and objective

Little information is available on the rapid diagnostic testing for syphilis in Burkina Faso. The objectives of the study were (i) to assess the sensitivity and specificity of four on site rapid tests in comparison with Treponema pallidum haemagglutination assay (TPHA) as a gold standard and (ii) to evaluate the operational characteristics of those tests among health workers in a maternity unit.

Methods

Four rapid syphilis tests commercially available in Burkina Faso were evaluated using archived serum samples and Treponema pallidum hemagglutination assay (TPHA) as the gold standard. Blood samples were collected between November 2011 and June 2012 from blood donors at the Regional Blood Transfusion Center of Ouagadougou. The sensitivity and specificity of the tests were calculated. Evaluation of operational characteristics such as clarity of pamphlet, complexity of technique, duration, was conducted in a first-level healthcare center with health workers in maternity unit.

Results

Alere DetermineTM Syphilis was the most sensitive of the four rapid syphilis tests evaluated. It was followed by SD Bioline Syphilis 3.0, Cypress Diagnostics Syphilis Quick test and Accu-Tell ® Rapid Anti-TP, which was the least sensitive. The four tests demonstrated a good diagnostic specificity for syphilis (95–98%), and healthcare workers found them easy to use.

Conclusions

The study allowed confirming the good performance of three of four rapid syphilis tests in Burkina Faso. More research will be conducted to assess the feasibility of introducing selected rapid tests for syphilis in antenatal care services.

Introduction

Syphilis in pregnant women remains a major public health problem. The World Health Organization (WHO) estimates that 90% of syphilis cases occur in low-income countries¹. The prevalence in developing countries ranges from less than 1% to 10%. In a recent review among studies in 1990–2011, prevalence estimates were 4.5% (3.9%–5.1%) in East and southern Africa and 3.5% (1.8%–5.2%) in West and Central Africa². In Burkina Faso, prevalence in pregnant women is low, with notable regional variations^3,4. According to WHO estimates, each year, maternal syphilis is responsible for at least 50,000 spontaneous abortions or stillbirths and 500,000 premature births of babies infected with congenital syphilis or who have low birth weight ¹. However, rates of congenital syphilis are generally underestimated⁵. Congenital syphilis is an avoidable disease that prenatal testing and early treatment of infected pregnant women could eliminate⁶.

In response to this situation, the scientific and technical consulting group of the WHO's Department of Reproductive Health and Research adopted a strategy for the global elimination of congenital syphilis by 2015¹. Consequently, many countries have included syphilis testing as part of a minimum package of tests conducted during prenatal visits.

Unfortunately syphilis diagnosis in peripheral clinics (CSPSs) in Burkina Faso is conducted using a syndromic approach, while the majority of syphilis cases are asymptomatic. In hospital laboratories (Medical Centers with Surgical Services (CMA), regional hospitals (CHR), national hospitals (CHU)), testing is conducted with a venereal diseases research laboratory (VDRL) test or with a Treponema pallidum hemagglutination assay (TPHA). Although these tests present certain advantages in that they allow for the differentiation between an old or treated syphilis infection and active syphilis, as well as an analysis of treatment adherence, their use requires qualified personnel, laboratory equipment, and a source of electricity, which limits their utility to peripheral clinics (CSPSs).

Currently, there are several available specific, rapid syphilis tests that are simple to use and could be implemented in CSPSs. When compared to the diagnostic tests currently being used (VDRL and TPHA), rapid treponemic tests have several advantages, including the rapid availability of results ( less than 30 minutes) and the fact that their usage does not require electricity or highly qualified laboratory staff⁷. Simultaneous pointof- care treponemal and non-treponemal are available with good performance¹¹ and are undergoing wide adoption for their benefits. However, despite reports of diagnostic performance provided by the manufacturers of rapid syphilis tests, data on test effectiveness and operational characteristics in the field remain limited in West Africa and non-existent in Burkina Faso. Against that background, we conducted a study to evaluate the diagnostic laboratory performance and operational characteristics of four rapid tests for Treponema pallidum available in Burkina Faso. The objectives of the study were; to assess the sensitivity and specificity of four on site rapid tests in comparison with Treponema pallidum haemagglutination assay (TPHA) as a gold standard and to evaluate the operational characteristics of the tests.

Methods

Study populations

We evaluated rapid syphilis tests commercially available in Burkina Faso using archived serum samples and TPHA as the gold standard. Rapid syphilis tests were defined as Treponema pallidum tests (i) capable of giving a result within 30 minutes and (ii) that could be used at service delivery points without any need for special storage or transport⁷. Only rapid syphilis tests with market authorization in Burkina Faso (currently valid or in the process of renewing validity with the Ministry of Health) were considered. In total, four tests were selected. These were 1) Accu-Tell® Rapid Anti-TP (Treponema pallidum / Syphilis) (Accu Biotech Co Ltd, China), 2) AlereTM DetermineTM Syphilis TP (AlereTM Médical Co Ltd, UK), 3) Syphilis Cypress Diagnostics (Cypress Diagnostics, Belgium), and 4) SD Bioline syphilis 3.0 (Standard Diagnostics INC, Korea). The characteristics of the tests according to the manufacturers are listed in Table 1.

Table 1.

Characteristics of the tests according to the manufacturers.

Characteristics	Accu-Tell ® Rapid Anti-TP	Alere Determine™ Syphilis TP	Cypress Diagnostics Syphilis Quick test Diagnostics	SD Bioline Syphilis 3.0 test
Manufacturer	Accu Biotech Co Ltd	Alere ™ Medical Co Ltd	Cypress Diagnostics	Standard Diagnostics INC
Product code	ABT-STD-B17	7D2443	353 Syphilis Quick Test	06FK11
Lot used	2011071312	14448k100	25094	32052
Type of antigen	Immunochromatographic TP Ag1, Ag2	Immunochromatographic Ag Treponema pallidum	Immunochromatographic Ag 17, 15 KDa Treponema pallidum	Immunochromatographic Ag recombinant pale (17, 15 kDa)
Specimen	Serum or plasma	Serum, plasma or whole blood	Serum, plasma or whole blood	Serum, plasma or whole blood
Time needed for results	15 minutes	15 minutes 24 hours maximum	5–20 minutes	5–20 minutes
Stable result	20 minutes	24 hours	20 minutes	20 minutes
Storage conditions	2–30°C	2–30°C	8–30°C	2–30°C

Blood samples were collected between November 2011 and June 2012 from blood donors at the Regional Blood Transfusion Center (CRTS) of Ouagadougou, Burkina Faso. A total of 120 serum samples were considered for the evaluation. There were 60 samples positive for syphilis (50 samples were TPHA and VDRL positive: active syphilis, and 10 were TPHA positive and VDRL negative: previous or primary case of syphilis) and 60 samples negative for syphilis (50 samples were TPHA negative and VDRL negative, and 10 were TPHA negative and VDRL positive). The comparative analysis was conducted in the laboratory of the Institut de Recherche en Science de la Santé (IRSS) in Ouagadougou, Burkina Faso.

For the evaluation of the operational characteristics, the study was conducted in a primary healthcare centre (CSPS) located in Ouagadougou the capital city of Burkina Faso. In total sixteen healthcare professionals (midwives, birth attendants, and auxiliary birth attendants) working in the maternity ward of an urban first-level healthcare were trained in the use of the four tests. Their seniority in the maternity ward was between 6 months and 18 years, although none had previously used a rapid syphilis test. However, they had used HIV and malaria rapid tests.

Sample collection

After obtaining the blood donor's consent, approximately 10 mL of blood was taken from the blood collection bag, put in a dry BD Vacutainer® tube (Becton, Dickinson and Company; USA) without anticoagulant and kept at +4°C for approximately four hours while waiting for the results of the syphilis test.

For the operational characteristics study, during the routine prenatal consultations, each healthcare worker used each rapid syphilis test on 5 pregnant women who consented to a blood-draw by finger prick. A self-administered questionnaire was made available to the healthcare workers for the data collection. The operational characteristics were appreciated following: (i) the clarity of the manufacturers' pamphlets, mainly the description of use, in terms of level of comprehension, (ii) the complexity of technique, (iii) the length of time required to complete the entire testing process, (iv) the time of apparition of line result, and (v) the interpretation of result. For each criterion, measures were defined.

The clarity of the manufacturer's pamphlets was appreciated regarding comprehension of the text if it was very clear, moderately clear or difficult to understand.

The complexity of technique was looked if the technique for utilization (from blood collection to the availability of the result) was complex (very difficult), moderately complex (difficult) or very easy in comparison with other rapid tests they had already used such as HIV test. The length of time required for a test (from blood collection to the availability of the result) was compared to the duration of other rapid tests they had used following the modalities of long, equivalent or short duration. The time for apparition of line result was compared to what was written in the pamphlet of each test. Finally the ease in interpretation of the result, in terms of visibility and readability of line result, was compared to other used rapid tests following the modalities of very easy or moderately easy to interpret. All these criteria were used to appreciate if a test has good operational characteristics or not.

Laboratory methods

After blood collection from blood donors, syphilis test was conducted by CRTS using the ARCHITECT Syphilis TP automated treponemal antibody test (Abbott Diagnostic, USA). This is a two-step immunoassay for the qualitative detection of anti-TP antibodies in human serum or plasma. It uses chemiluminescent microparticle immunoassay (CMIA) technology with flexible dosage protocols called Chemiflex. The microparticles are covered with recombinant TP antigens (TpN15, TpN17, and TpN47). Once the syphilis status of the blood donor was determined by CRTS, we chose positive and negative samples for syphilis for our evaluation. At the end of the collection day, the samples were centrifuged at 3000 rpm for 10 minutes and then submitted to combine VDRL/TPHA (BIOLABO SA, France) testing in the serologic laboratory of the Institut de Recherche en Sciences de la Santé (IRSS) to confirm the syphilis status. After confirmation, the serum was aliquoted in two cryotubes of 2 mL, labeled, and stored at −20°C until beginning of the next stage of the evaluation. Hemolytic samples were excluded.

The evaluation of the rapid syphilis tests performance was conducted at the laboratory of IRSS in July 2012 using the stock of serum samples from CRTS. The evaluation followed the manufacturers' instructions and used good laboratory practices. First the frozen serum samples were brought to ambient temperature before use. Then each test was used in series. To avoid the comparison of results between tests during the laboratory analysis, each rapid test was used on all samples before moving to the next test. There was a blind interpretation of test results, independent of the results of the reference test (TPHA). TPHA (BIOLABO SA, http://www.biolabo.fr/pdfs/noticesFR/Syphilis) was used as the gold standard (reference test). It is an indirect hemagglutination assay (IHA) for the identification of Treponema pallidum antibodies circulating in human plasma and regularly used for the diagnosis of syphilis in a laboratory.

Statistical data analysis

For each test, the laboratory evaluation results were compared to the reference test (TPHA) and categorized as true positive, false positive, true negative, or false negative. The data were entered with Epidata and analyzed using SPSS version 15 and R version 2–12.1. The performance characteristics, such as sensitivity and specificity were calculated relative to the reference standard TPHA results obtained for each serum specimen.

Ethical issues

The research protocol received the approval of the Health Research Ethics Committee (CERS), Ministry of Health, Burkina Faso.

Results

Performance of diagnostic tests

A total of 120 samples were tested for the evaluation of the performance. The sensitivity of the tests was compared to the gold standard used (TPHA). Analysis of the sensitivity of the tests showed that Alere DetermineTM syphilis TP had the highest sensitivity (93%) among the four selected rapid tests. The sensitivity of Cypress syphilis and SD Bioline Syphilis 3.0 was 90%. Accu-Tell ® Rapid Anti-TP had a sensitivity of 78%. In terms of specificity, Alere DetermineTM syphilis TP, Accu-Tell ® Rapid Anti-TP and SD Bioline Syphilis 3.0 had the same result (98%), while Cypress syphilis had a specificity of 95%. (Table 2)

Table 2.

Results of the diagnostic performances of rapid syphilis tests.

	Accu-Tell ® Rapid Anti-TP (Accu Biotech Co Ltd)	Alere Determine™ Syphilis TP (Alere™ Médical Co Ltd)	Cypress Diagnostics Syphilis Quick test (Cypress Diagnostics)	SD Bioline Syphilis 3.0 test (Standard Diagnostics INC)
Sensitivity (95% CI)	78% (66 –87)	93% (84–97)	90% (79–95)	90% (80–95)
Specificity (95% CI)	98% (91–100)	98% (91–100)	95% (86–98)	98% (91–100)

CI = confidence interval

Operational characteristics of the four tests according to healthcare professionals

The results of the evaluation of the operational characteristics of the four tests by 16 healthcare professionals are summarized in Table 3. The pamphlet of the test was clearly comprehensible by 12 of the 16 respondents for Accu-Tell ® Rapid Anti-TP, by 11 respondents for SD Bioline syphilis 3.0, by 9 respondents for Cypress Diagnostics Syphilis Quick test, and by 7 respondents for Alere DetermineTM syphilis TP.

Table 3.

Operational characteristics of syphilis diagnostic tests.

Characteristics evaluated	Accu-Tell ® Rapid Anti-TP	Alere Determine™ Syphilis TP (Alere™ Médical Co Ltd)	Cypress Diagnostics Syphilis Quick test	SD Bioline Syphilis 3.0 test (Standard Diagnostics INC)
Clarity of the test pamphlet	(n=16)	(n=16)	(n=16)	(n=16)
Difficult to understand	2	2	1	1
Moderately clear	2	7	6	4
Very clear	12	7	9	11
Complexity of technique	(n=16)	(n=16)	(n=16)	(n=16)
Complex	1	0	0	0
Moderately complex	1	2	1	1
Very easy	14	14	15	15
Time needed for performing	(n=16)	(n=16)	(n=16)	(n=16)
Long	0	4	0	0
Equivalent	9	10	10	8
Short	7	2	6	8
Time need for results	(n=16)	(n=16)	(n=16)	(n=16)
Time indicated in pamphlet	15	14	15	15
Time indicated plus 5 minutes	1	2	1	1
Ease of result interpretation	(n=16)	(n=16)	(n=16)	(n=16)
Moderately easy	2	4	7	2
Very easy	14	12	9	14

As a whole, the healthcare professionals judged the technique for utilization of the tests to be very easy. Indeed, of the 16 healthcare professionals who participated in the evaluation, 15 found Cypress Diagnostics Syphilis Quick test and SD Bioline Syphilis 3.0 to be very easy to use, while 14 said the same for Accu-Tell ® Rapid Anti-TP and Alere DetermineTM syphilis TP. Overall, the time needed to obtain results was determined to be similar to other tests already used for HIV and malaria. Specifically, 10 respondents found Alere DetermineTM syphilis TP and Cypress Diagnostics Syphilis Quick test to be similar to the previously used tests, and approximately 50% found this for Accu-Tell ® Rapid Anti-TP (9/16) and SD Bioline syphilis 3.0 (8/16).

For the majority of respondents, the time required to obtain results matched the time indicated by the manufacturer for Accu-Tell ® Rapid Anti-TP, Cypress Diagnostics Syphilis Quick test and SD Bioline syphilis 3.0 (Table 3).

The interpretation of results was found to be very easy by 14 of 16 respondents for Accu-Tell ® Rapid Anti-TP and SD Bioline syphilis 3.0 and by 12 of 16 respondents for Alere DetermineTM syphilis TP. As for Cypress Diagnostics Syphilis Quick test, only 9 of 16 found the results to be easy to interpret.

The realization of the tests requires the use of equipment not provided in the kit. Specific and non-specific equipment are needed. Specific equipment, such as capillary tubes with EDTA for Alere Determine™ Syphilis TP and pipettes for Cypress Diagnostics Syphilis Quick test and SD Bioline syphilis 3.0 Accu-Tell® are needed. Rapid Anti-TP does not require specific equipment. Non-specific equipment, including alcohol swabs and lancets, was required for the four tests.

Discussion

This is the first evaluation of the performance and the characteristics of commercially available rapid diagnostic syphilis tests for Burkina Faso, and it allows the identification of the test that is adapted to the context of the country. To our knowledge, two of the tests, Cypress Diagnostics Syphilis Quick test and Accu-Tell ® Rapid Anti-TP, were for the first time compared to other rapid tests for syphilis in West Africa.

All the tests used an immunochromatographic detection of anti-Treponema pallidum antibodies. These tests used one or more of several similar recombinant Treponema pallidum antigens, although their diagnostic performances in our study were different. With a sensitivity of 93%, Alere DetermineTM Syphilis was the most sensitive of the four rapid syphilis tests. It was followed by SD Bioline Syphilis 3.0 and Cypress Diagnostics Syphilis Quick test (Se = 90%), with Accu-Tell ® Rapid Anti-TP being the least sensitive (Se =78%). In general, the four tests demonstrated a good diagnostic specificity for syphilis (95 — 98%). Taking into account the two diagnostic criteria of sensitivity and specificity, our study showed that Alere DetermineTM Syphilis had higher diagnostic performance than the four evaluated tests (Se = 93%, Sp = 98%) followed by SD Bioline Syphilis 3.0 (Se = 90%, Sp = 98%). The good diagnostic performance of these rapid tests compared to other tests has been reported in the literature^10–13. However, few data exist regarding the performances of Cypress Diagnostics Syphilis Quick test and Accu-Tell ® Rapid Anti-TP.

The performance of the tests was different than what was indicated in the manufacturers' pamphlets. Our results showed lower values than those reported for Cypress Diagnostics Syphilis Quick test (99.3% and 99.5%), Alere DetermineTM syphilis TP (100% with serum), and SD Bioline syphilis 3.0 (99.3% and 99.5%). The pamphlet for Accu-Tell ® Rapid Anti-TP did not give any indication of performance. This data, which is crucial in the choice of diagnostic tests, is unavailable in the current published literature. This result calls into question the criteria used for obtaining market authorization from the Ministry of Health to sell rapid tests in Burkina Faso. Additionally, this result emphasizes the need to evaluate available tests on the local market to properly inform policy decision makers.

The performances of Cypress Diagnostics Syphilis Quick test, Alere DetermineTM syphilis TP, and SD Bioline syphilis in this study were in accordance with the threshold set by the WHO⁷, which recommends a minimum sensitivity of 85% and a minimum specificity of 95%. The low sensitivity of Accu-Tell ® Rapid Anti-TP (78%) when compared to the WHO threshold shows that this test is inappropriate for syphilis testing in Burkina Faso.

The sensitivities found for Alere DetermineTM syphilis TP and SD Bioline syphilis 3.0 are below those found by Herring et al.¹⁴ using stored serum samples in Gambia (100%), Tanzania (96% and 94%), and South Africa (96% and 94%). However, their specificities are higher than those found by Herring et al. and Mabey et al.¹³ and similar to the results obtained in China. This variation in performance could be explained not only by ambient storage conditions and use of the different tests but also by the use of fluorescent treponemal antibody, absorbed (FTA-ABS) as reference test and the type of specimen used. This study, whose objective was to evaluate rapid syphilis tests commercially available in Burkina Faso, used tests bought or received by intermediaries of local distributors. A direct order of rapid tests from the manufacturer would have permitted the traceability of the tests evaluated.

Regarding the evaluation of operational characteristics, no test was found to be easier to use than any other. We found all four tests to have very good operational characteristics. In fact, the majority of respondents found that the results were very easy to interpret, and the time required to use the tests was similar to that of existing tests for HIV and malaria. These results are corroborated by Herring et al. for Alere DetermineTM syphilis TP and SD Bioline syphilis 3.0. Other studies have also shown rapid diagnostic syphilis tests to be easy to use¹⁵ but this criterion is not the most important in choosing a diagnostic test. Although perceived favorably by the healthcare professionals, the four test kits were missing equipment. This equipment, such as alcohol swabs, lancets, pipettes, and capillary tubes, is necessary to use the tests in the field. We recommend that the distributors include them in the kits to ensure correct use of the tests.

Of the four tests, three can be performed using whole blood. Studies have documented the performance of Alere DetermineTM syphilis TP in both the laboratory as well as in real-world situations¹³. Given its simplicity and its good performance in our study, which is in agreement with data collected in previous studies by other research teams, Alere DetermineTM syphilis TP seems to be adapted to syphilis testing in Burkina Faso. While our study presented the laboratory performance of these tests when used with serum samples, their performance with whole blood will be different¹⁶. In addition, providing diagnostic tests to CSPSs would allow for the collection of more information on the effects of storage conditions on test performance when used in rural settings. Moreover, we did not evaluate the effects that additional infections, such as HIV or malaria, could have on the performance of these tests in real-world situations with whole blood.

Conclusion

Rapid syphilis tests in limited-resource countries can help diagnose syphilis in CSPSs that, until now, have used a syndromic approach to STI diagnosis. Our study allowed us first to confirm the good performance of three of four rapid syphilis tests in Burkina Faso. Additionally, it allowed us to identify Alere DetermineTM Syphilis TP as the test that is adapted to Burkina Faso. More research on the feasibility and acceptability of these rapid syphilis tests in first-level healthcare centers should allow for the effective implementation of the recommendation for systematic testing of pregnant women.