Evaluation of the Determine Rapid Syphilis TP Assay Using Sera

Abstract

The Abbott Determine Rapid Syphilis TP assay is a treponemal test that can be used in resource-poor settings that lack laboratory facilities. However, this test has not been extensively evaluated. We measured its sensitivity and specificity by using stored serum specimens (n = 567) from all persons who tested Treponema pallidum hemagglutination assay (TPHA) positive (n = 250) or TPHA indeterminate (n = 17) in the year 2001 and the first 300 patients in 2001 who tested TPHA negative at the Evandro Chagas Research Institute in Rio de Janeiro, Brazil. This rapid assay was independently interpreted by three different observers. With TPHA results as the reference, sensitivity ranged between readers from 95.6 to 98.4% and specificity ranged from 97.3 to 95.7%. There was little interreader variability in the interpretation of results, with approximately 98% agreement for all reader combinations. Of samples from persons with human immunodeficiency virus (HIV) infection (n = 198), sensitivity was 96.9 to 99.2% and it was 94.4 to 96.3% among HIV-negative persons (n = 127). Specificity was 92.4 to 95.5% among HIV-positive persons and 97.2 to 100% among HIV-negative persons. We found this test to have high sensitivity and specificity and little interreader variability, indicating that it may be easily used in resource-poor settings without laboratory facilities. Further studies are needed using this test on whole blood and under the clinical conditions for which it is intended.

Diagnosis of syphilis infection is neither easy nor rapid. Serodiagnosis of syphilis is usually based on detection of antibodies against cardiolipin or against the causative organism, Treponema pallidum (6, 10, 11). In the United States, screening is usually performed with the Venereal Disease Research Laboratory (VDRL) test (13) or the rapid plasma reagin (RPR) test (7). These cardiolipin-based tests are confirmed with a test for detection of T. pallidum antibodies, such as the T. pallidum hemagglutination assay (TPHA; not currently available in the United States) (10, 14), the T. pallidum particle agglutination test (12), or the more time-consuming fluorescent treponemal antibody adsorption test (9).

In developing countries and areas with limited resources, laboratory facilities are often unavailable for standard syphilis tests. Blood samples may need to be sent to distant laboratories, delaying diagnosis. As a result, infected individuals may go home untreated. A rapid serologic test could greatly enhance public health efforts to decrease the spread of this infection.

The availability of individual T. pallidum antigens through recombinant DNA techniques (12) has resulted in the use of these antigens for serologic tests by lateral-flow technology. These tests use one or multiple recombinant antigens, such as the 47-, 17-, or 15-kDa antigen. Although several different manufacturers developed rapid tests using lateral-flow technology and recombinant antigens in the late 1990s, there are few published evaluations of these tests (17).

In 1997, the Abbott Determine Rapid Syphilis TP assay was approved for use in Brazil. This assay is an in vitro, visually read, qualitative immunoassay for the detection of antibodies to the antigen of T. pallidum. Antibodies bind to an antigen-selenium colloid that is captured by immobilized antigen and forms a red line on the test strip. The test can be used with either serum or whole blood with a finger stick for collection; no specialized equipment is required. This test would allow health clinics without a phlebotomist to screen for syphilis; screening could also occur in non-health facility settings such as prisons and drug treatment or community centers.

In this study, we determined the sensitivity and specificity of the Abbott Determine Rapid Syphilis TP assay by using stored sera from 567 patients in an infectious disease research center in Brazil. We used VDRL test and TPHA results as the reference. We also determined interreader variability in the interpretation of results.

MATERIALS AND METHODS

Study design.

This was a comparative study using stored sera to evaluate a diagnostic test.

Study site.

The Evandro Chagas Research Institute (IPEC), located in Rio de Janeiro, Brazil, is part of the Oswaldo Cruz Foundation of the Brazilian Ministry of Health. This institute serves patients who are part of infectious disease research studies. During the year 2000, an estimated 1,000 patients with human immunodeficiency virus (HIV) infection, 800 with human T-lymphotropic leukemia virus type I (HTLV-I) and HTLV-II infections, 600 with Chagas' disease, 150 with tuberculosis, and 500 with leishmaniasis were treated at this hospital. This population provided not only patients with syphilis but also patients with other conditions that may cause false-positive serologic tests for syphilis. In the year 2001, of 1,213 TPHA-negative samples, 23 (1.9%) were read as VDRL test positive, all with titers less than or equal to 1:8.

Patients are referred by other physicians or come to this institute because of its reputation as an infectious disease research center. They are triaged and tested for multiple infections (including syphilis) to determine if they meet the eligibility criteria for different studies. Patients meeting the criteria are offered participation in a study and become patients at the hospital. Those not meeting the criteria are treated or referred for treatment of any conditions they may have but do not become patients at the hospital. Patients in the latter group do not have medical records.

All studies at this institution require ethical committee approval and must obtain informed consent if the study involves contact with human subjects. This study received IPEC and Centers for Disease Control and Prevention Institutional Review Board approval.

Source of samples.

Serum specimens (n = 567) from all persons who tested TPHA positive (n = 250) or TPHA indeterminate (n = 17) in the year 2001 and those from the first 300 patients in 2001 who tested TPHA negative were stored at −20°C in the Evandro Chagas immunodiagnostic laboratory.

Laboratory procedures.

The immunodiagnostic laboratory of this center performs syphilis testing on 120 to 150 patients per month. Standard international procedures for VDRL (Laborclin, Paraná, Brazil) testing and TPHA (Biolab, BioMérieux, Rio de Janeiro, Brazil) were used (10).

A technologist in the laboratory performed the Abbott Determine Rapid Syphilis TP assay (Dainabut Co. Ltd., Tokyo, Japan) in accordance with the manufacturer's instructions. This technologist interpreted the result and recorded the findings on a form. Within 10 min, another laboratory technologist independently interpreted the same test strip. After recording the result on a separate form, the second technologist gave the assay to one nurse (also within 10 min), who also independently interpreted the assay.

Data collection.

Data were recorded at the laboratory on a standardized form and included the medical record number, the date of the test, and the results of the VDRL test, the TPHA, and the Abbott Determine Rapid Syphilis TP assay (separately for each observer).

A standardized data abstraction form was used by infectious disease fellows to abstract data from the medical record of each patient. Data abstracted included the medical record number, sex, age, symptoms, whether the patient had been treated for syphilis, stage of syphilis, and the presence of other concurrent infections (e.g., HIV infection).

These two data sets were linked by first assigning a random number to each medical record number, placing this number on the laboratory and medical record data form, removing the medical record number, and then merging the data on the basis of this random number. No identifying information (such as name or birth date) was collected on either data form.

Analysis.

Because this rapid test is based on detection of treponemal antibody, sensitivity and specificity were determined with the TPHA as the reference. The TPHA used in this study has been reported to have had a sensitivity of 100% and a specificity of 99% when sera from 85 patients with known primary (n = 15), latent (n = 40), and tertiary (n = 28) syphilis were used (5). The sensitivity and specificity of the Abbott Determine Rapid Syphilis TP assay were also calculated with standard criteria for diagnosis of syphilis (both TPHA and VDRL test positive or both TPHA and VDRL test negative). Clinical data were available for 340 (60%) patients and were not available for 217 (38%) who presented at triage for possible selection for a clinical trial but were considered ineligible for a trial and thus had no medical record and for 10 (2%) patients who were deceased and whose medical records were no longer available. Of the 340 serum samples with available clinical data, 15 had indeterminate TPHA results. For the remaining 325 samples, sensitivity and specificity were estimated separately by HIV infection status. Analysis by stage of syphilis infection was not possible, because there were sufficient data to classify patients by stage of syphilis disease for only 71 (28%) of the 250 TPHA-positive samples.

Interreader variability in the interpretation of the results was computed with the kappa statistic including 95% confidence intervals.

RESULTS

Of the 567 samples, 250 were TPHA positive, 17 were TPHA indeterminate, and 300 were TPHA negative. The majority of the TPHA-positive samples (195 of 250) were VDRL reactive, while most of the TPHA-indeterminate samples (14 of 17) and all of the TPHA-negative samples were VDRL nonreactive.

The sensitivity of the Abbott Determine Rapid Syphilis TP assay ranged, between readers, from 95.6 to 98.4% (Table 1). The specificity ranged from 95.7 to 97.3%. All 17 TPHA-indeterminate serum samples were read as negative in the Abbott Determine Rapid Syphilis TP assay by the laboratory technologists, and 15 were read as negative by the nurse.

TABLE 1.

Sensitivity and specificity of the Determine Rapid Syphilis TP assay by reader type with the TPHA as the reference standard^a

Reader	TPHA positive (n = 250)		TPHA negative (n = 300)
	No. determine positive	% Sensitivity (95% CIb)	No. determine negative	% Specificity (95% CI)
Laboratorian 1c	244	97.6 (94.6-98.8)	292	97.3 (94.6-98.6)
Laboratorian 2c	246	98.4 (95.6-99.4)	287	95.7 (92.9-97.7)
Nursed	241	95.6 (93.2-96.2)	289	96.3 (93.4-98.3)

^aFive hundred fifty samples were tested; 17 TPHA-indeterminate results are not included.

^bCI, confidence interval.

^cAll TPHA-indeterminate results were read as negative.

^dFifteen of 17 indeterminate TPHA results were read as negative.

With positive VDRL test and TPHA results for the reference, sensitivity ranged (between readers) from 96.9 to 98.5% (Table 2). The sensitivity estimates for samples with positive TPHA and negative VDRL test results (most likely representing past infection) were 96.9 to 97.9% and were not significantly different from those for samples with both TPHA- and VDRL test-positive results. Specificity was the same as that reported when using TPHA alone as the reference because all TPHA-negative samples were also VDRL nonreactive.

TABLE 2.

Sensitivity and specificity of the Determine Rapid Syphilis TP assay by reader type with the TPHA and the VDRL test as the reference standard^a

Reader	% sensitivity (95% CIb)		% Specificity (95% CI), TPHA and VDRL nega- tive (n = 300)
	TPHA and VDRL positive (n = 195)	TPHA positive VDRL negative (n = 53)
Laboratorian 1	97.9 (94.4-99.2)	96.2 (85.0-99.0)	97.3 (94.6-98.6)
Laboratorian 2	98.5 (95.1-99.4)	98.1 (86.3-99.7)	95.7 (92.9-97.7)
Nurse	96.9 (93.6-98.9)	94 (82.5-98.0)	96.3 (93.4-98.3)

^aFive hundred fifty samples were tested; 17 TPHA-indeterminate results were not included.

^bCI, confidence interval.

Among the 325 samples with clinical data, we found that other infections were common. Of the 145 TPHA-negative samples, 93% revealed other infections, including hepatitis C (n = 25), Hansen's disease (n = 4), and HIV (n = 67). Of the 186 TPHA-positive samples, 93% also revealed other infections, including hepatitis C (n = 18), Hansen's disease (n = 5), and HIV (n = 131). Additionally, of the TPHA-positive samples, 4 (3%) revealed primary, 22 (17%) revealed secondary, 33 (25%) revealed latent, and 12 (9%) revealed tertiary syphilis; for 61 patients (33%), there was not sufficient information in the medical record for classification of the disease stage.

Because of the large number of HIV-infected patients, we were able to calculate the sensitivity and specificity separately for HIV-infected (n = 198) and HIV-negative (n = 127) sera with TPHA as the reference (Table 3). Sensitivity ranged (between readers) from 97.7 to 99.2% for HIV-positive persons and from 94.4 to 96.3% for HIV-negative persons. Specificity ranged from 92.4 to 95.5% among HIV-positive persons and from 97.3 to 100% among HIV-negative persons. Of the eight Abbott Determine Rapid Syphilis TP assay false-positive persons, all were VDRL negative, all were read as weakly positive by at least one reader, and all had infections other than syphilis, the most frequent being HIV infection. Of the eight Rapid Abbott Determine Syphilis TP assay false-negative persons, five were VDRL reactive (titers ranged from 1:1 to 1:8) and all except one had an infection other than syphilis, primarily HIV.

TABLE 3.

Sensitivity and specificity of the Determine Rapid Syphilis TP assay by reader type and HIV status using the TPHA as the reference standard^a

Reader	HIV positive (n = 198)		HIV negative (n = 127)
	% Sensitivity (95% CIb)	% Specificity (95% CI)	% Sensitivity (95% CI)	% Specificity (95% CI)
Laboratorian 1	98.5 (93.7-99.6)	95.5 (86.3-98.5)	94.4 (83.3-98.2)	100 (88.6-100%)
Laboratorian 2	99.2 (94.2-99.8)	92.5 (82.5-96.8)	96.3 (85.2-99.0)	97.3 (88.8-99.3)
Nurse	97.7 (92.4-99.2)	92.5 (82.2-98.9)	94.4 (83.1-98.1)	98.6 (89.8-99.8)

^aThree hundred twenty-five samples were tested, excluding samples without clinical data (n = 227) and those TPHA indeterminate (n = 15).

^bCI, confidence interval.

For the entire sample (n = 567), there was little interreader variability in the interpretation of the results (Table 4). There were no significant differences in interpretation between the reader pairs analyzed. Additionally (data not shown), there was no significant difference in agreement between HIV-positive and -negative samples.

TABLE 4.

Interobserver variability in interpretation of the Determine Rapid Syphilis TP assay

Reader combination	% Agreement	Kappa coefficient	95% Confidence interval
Laboratorian 1-nurse	98.0	0.96	0.94-0.98
Laboratorian 2-nurse	98.0	0.96	0.94-0.98
Laboratorian 1-laboratorian 2	98.6	0.97	0.95-0.99

DISCUSSION

We found that the Abbott Determine Rapid Syphilis TP assay had a high sensitivity (95.6 to 98.4%) and specificity (95.7 to 97.3%) with stored sera and TPHA as the reference test. Additionally, we found high agreement between readers (two laboratory technologists and a nurse), suggesting that the test is easy to read. Although serum may provide higher sensitivity than whole blood, these data suggest that the Abbott Determine Rapid Syphilis TP assay could be used for diagnosis when laboratory facilities are not available or when results are needed at the point of care.

There is only one peer-reviewed article in the medical literature regarding the evaluation of the Abbott Determine Rapid Syphilis TP assay (17). However, no fluorescent treponemal antibody absorption testing or other treponemal antibody testing was done. The investigators found that when they compared this test to the VDRL and RPR tests, with 72 known positive and 219 known negative samples, there was 99.3% agreement. In that analysis, all of the persons positive by the RPR test were positive (72 of 72) by the Abbott Determine Rapid Syphilis TP assay and there were 3 false-positive persons (3 of 219). The manufacturer, on the basis of studies done in Japan, has reported the specificity of the rapid test to be 100% (325 of 325) and its sensitivity to be 92.3% (48 of 52) in whole blood, 100% (52 of 52) in serum, and 100% (52 of 52) in plasma compared with a unspecified commercially available test (Abbott Determine Rapid Syphilis TP assay insert instructions).

We found that, with TPHA as the reference standard, the specificity of this test was slightly lower and its sensitivity was higher among HIV-positive persons than among HIV-negative persons, although the differences were not statistically significant. Other studies have found false-positive treponemal syphilis tests among HIV-infected persons (8). One study found that false-negative treponemal tests occurred only among HIV-infected persons in their sample (4). Studies with larger sample sizes of HIV-positive and -negative persons need to be conducted to clarify this performance difference in HIV-infected persons.

Currently, the accepted method for syphilis screening in the United States (1), Brazil (2), and other countries is detection of antibodies to a nontreponemal antigen with either the VDRL test or the RPR test. If a newly positive result by a nontreponemal assay is detected, a treponemal test is usually performed to confirm infection, as the treponemal assay is highly specific for syphilis infection. In other countries, use of the treponemal assay as the first screening assay, or as the sole test, may be a feasible option (3).

One limitation of treponemal tests is that they can remain positive even after treatment (6, 10, 16), so that someone previously treated for syphilis may be misdiagnosed as having a new, untreated case of syphilis if only treponemal tests are used and overtreatment could occur. Nontreponemal tests usually become nonreactive after effective therapy (10, 11, 15). In our study, we had a small number of serum samples with positive treponemal tests and negative nontreponemal tests, suggestive of previously treated syphilis. The Abbott Determine Rapid Syphilis TP assay was read as positive for 94 to 98% of these samples.

In Brazil, the Abbott Determine Rapid Syphilis TP assay has been used to screen pregnant women at delivery who have not been tested previously during pregnancy. However, the sensitivity and specificity of the test in this setting have not been evaluated (Fabio Mahoredaui, personal communication). We had no pregnant women in our sample of patients with clinical information. In South Africa, a feasibility study was done to assess whether the Abbott Determine Rapid Syphilis TP assay can be used to screen pregnant women. This study found the test to be acceptable to women and used correctly by nurses and midwives when they are appropriately trained. Sensitivity and specificity were similar to what we found with stored sera (Centers for Disease Control and Prevention findings from a study of feasibility, acceptability, performance, cost, and cost-effectiveness of on-site testing for syphilis in rural antenatal clinics, Eastern Cape Province, Republic of South Africa, 2003 report, pages 10 and 11). Therefore, one potential use of this rapid test is to screen pregnant women for syphilis, especially in developing countries, where access to prenatal laboratory testing may be difficult.

Our study had two major limitations. First, we relied on stored sera that could result in higher sensitivity than with whole blood according to the manufacturer. Second, we did not have complete clinical data on all samples. Therefore, we were unable to classify many persons by stage of syphilis infection to determine if performance varied by stage of disease. Treponemal tests have been shown to be less sensitive during primary syphilis and are therefore not recommended for diagnosis of primary syphilis (10). Additionally, we could not determine if we had any definitive cross-reactive sera and thus were unable to evaluate the test under these conditions.

We found this test to have high sensitivity and specificity, with TPHA as the reference, and low interreader variability in the interpretation of results, suggesting that this test may be easily used in resource-poor settings without laboratory facilities. However, even with this specificity, in a low-prevalence population, the positive predictive value would be low. Several similar rapid diagnostic tests for syphilis are commercially available outside of the United States. Evaluations of these tests also are needed to allow decisions to be made about their use. Further studies are needed to evaluate the practical applications of these tests, specifically to clarify their sensitivity and specificity with whole blood from persons with and without HIV infection and to explore their feasibility, acceptability, and cost-effectiveness in the field.