Laboratory Evaluation of a Commercially Available Rapid Syphilis Test

Serological diagnosis of syphilis depends on assays that detect treponemal and nontreponemal antibodies. Laboratory certification and trained personnel are needed to perform most of these tests, while high costs and long turnaround time can hinder treatment initiation or linkage to care.

ABSTRACT

Serological diagnosis of syphilis depends on assays that detect treponemal and nontreponemal antibodies. Laboratory certification and trained personnel are needed to perform most of these tests, while high costs and long turnaround time can hinder treatment initiation or linkage to care. A rapid treponemal syphilis test (RST) that is simple to perform, accessible, and inexpensive would be ideal. The Syphilis Health Check (SHC) assay is the only Food and Drug Administration (FDA)-cleared and Clinical Laboratory Improvement Amendments (CLIA)-waived RST in the United States. In this study, 1,406 archived human serum samples were tested using SHC and traditional treponemal and nontreponemal assays. Rapid test results were compared with treponemal data alone and with a laboratory test panel consensus defined as being reactive by both treponemal and nontreponemal assays for a given specimen, or nonreactive by both types of assays. The sensitivity and specificity of the SHC assay compared with treponemal tests alone were 88.7% (95% confidence interval [CI], 86.2 to 90.0%) and 93.1% (95% CI, 90.0 to 94.9%), respectively, while comparison with the laboratory test panel consensus showed 95.7% (95% CI, 93.6 to 97.2%) sensitivity and 93.2% (95% CI, 91.0 to 95.1%) specificity. The data were further stratified based on age, sex, pregnancy, and HIV status. The sensitivity and specificity of the SHC assay ranged from 66.7% (95% CI, 46.0 to 83.5%) to 91.7% (95% CI, 87.7 to 94.7%) and 88% (95% CI, 68.8 to 97.5%) to 100% (95% CI, 47.8 to 100%), respectively, across groups compared to traditional treponemal assays, generally increasing for all groups except the HIV-positive (HIV⁺) population when factoring in the laboratory test panel consensus. These data contribute to current knowledge of the SHC assay performance for distinct populations and may guide use in various settings.

INTRODUCTION

Syphilis cases and rates are increasing across all regions of the United States (1, 2). Recent estimates indicate rising rates among men and women in every age group ≥15 years, particularly among men who have sex with men (MSM) (1, 2). Early and accurate diagnosis of syphilis are key for treatment and prevention. Direct detection of the causative agent of syphilis, Treponema pallidum, is limited to darkfield microscopy examination of lesion exudate or tissues, and no commercial molecular detection tests are currently Food and Drug Administration (FDA) cleared for diagnosis. While some laboratories provide locally developed and validated PCR tests, serology-based methods that are FDA cleared are recommended for antibody detection and screening purposes and are currently used to diagnose and manage syphilis (1, 3). Serodiagnosis of syphilis involves two categories of tests, nontreponemal and treponemal (1, 3–8). Nontreponemal tests detect IgG and IgM antibodies that are not specific to T. pallidum, instead targeting lipoidal antigens released from damaged host cells caused by T. pallidum, or from a damaged T. pallidum cell wall, and host immune responses (9, 10). Antibodies specific to T. pallidum antigen(s) are also produced, and these are detected by treponemal tests (10). Nontreponemal and treponemal serological tests can yield false-positive or false-negative results in part due to the complex pathology of syphilis and individual assay limitations (1, 4–21). Thus, to identify syphilis, both types of tests must be performed, and for many years, testing has followed a traditional algorithm that involves nontreponemal testing first and then a treponemal test(s). Due to cost and workflow factors, a reverse algorithm has been adopted by some laboratories, which first uses automated treponemal tests to facilitate high throughput and quick turnaround, followed by a nontreponemal test and reflex testing to confirm discordant results (1, 3). In addition to a patient's clinical and immune status, specimen type (blood versus serum versus plasma) and quality can also affect downstream laboratory test results (6, 7, 10). Regardless of the algorithm used, clinicians must also consider epidemiological risk, patient history, and clinical signs to determine whether treatment or follow-up testing is the next appropriate step (1, 3, 10, 11).

Current FDA-cleared nontreponemal tests include laboratory-based manual assays, such as the rapid plasma reagin (RPR) and Venereal Disease Research Laboratory (VDRL) tests. FDA-cleared laboratory treponemal assays also include manual procedures, such as the fluorescent treponemal antibody absorbed (FTA-ABS) test and T. pallidum passive particle agglutination (TP-PA) test, as well as automated or automatable enzyme immunoassays (EIA), chemiluminescence immunoassays (CIA), and immunoblots (3, 5, 7, 10, 11). In recent years, rapid syphilis tests (RST) have been gaining interest globally, with one FDA-cleared and CLIA-waived qualitative lateral flow immunochromatographic assay available for use in the United States (Syphilis Health Check [SHC]; Trinity Biotech USA, Inc., NY) (22, 23). Unlike traditional laboratory-based syphilis treponemal tests, RST are portable for use in the field, are relatively inexpensive, do not involve complex training, and yield near-instantaneous results that allow for immediate point-of-care (POC) treatment and/or linkage to care (15, 24–30). The manufacturer's insert for the SHC test, for example, indicates that results can be read within 10 to 15 minutes (22). Traditional serological testing can still be performed at well-equipped off-site laboratories, but risks of delayed treatment and/or loss of patient follow-up could be minimized with RST in the interim. Other RST have been evaluated in international settings (31–39), while only three reports have described SHC assay performance in the United States and in Uganda since its release in 2011 (40–42). The goal of the current study is to therefore gather additional data for this RST within the scope of existing serological tests and recommendations. The study design involves comparative performance evaluation of SHC (treponemal test) and traditional treponemal (TP-PA, EIA, and CIA) and nontreponemal (RPR) assays using a collection of archived clinical and commercially procured human serum samples. Further analyses of serological test results among distinct populations were also done to gain insight on the SHC assay performance among specific group(s).

MATERIALS AND METHODS

Serum specimens.

A total of 1,406 archived unlinked (deidentified) frozen (−20°C to −80°C) serum specimens spanning the years 2014 to 2016 were provided by Bio-Rad Laboratories (Hercules, CA), health care organizations, and public health laboratories to the Centers for Disease Control and Prevention (CDC, Atlanta, GA). Specifically, 455 and 200 samples were provided by the Association of Public Health Laboratories (Silver Spring, MD) and Georgia Public Health Laboratory (Decatur, GA), respectively, through joint collaborative efforts with the CDC. Bio-Rad Laboratories donated 506 specimens that were initially procured commercially by Bio-Rad based on FDA recommendations for their internal FDA 510(k) study. Their remnant specimens were gifted to the CDC. Kaiser Permanente Northern California (KPNC), Kaiser Permanente Southern California (KPSC), and the San Francisco Department of Public Health (SFDPH) provided 245 samples to the CDC. A combination of screening and diagnostic specimens was provided by KPNC and KPSC, whose laboratories utilize reverse-sequence screening. Specimens from the SFDPH were from the city's municipal sexually transmitted disease (STD) clinic, which performs point-of-care nontreponemal RPR testing at the clinic, followed by treponemal testing in the laboratory; specimens sent to the CDC included those with reactive serology and clinical diagnosis of syphilis. There were no preselection sample criteria used by the CDC, with all specimens being accepted as is from these suppliers and later tested at the CDC using treponemal and nontreponemal tests in parallel. Serology results from the suppliers were also blinded until after testing was completed at CDC. If available, information pertaining to age, sex, pregnancy status, and HIV status was provided by each supplier. For a given population category, not all information was available for the remaining categories. Negative/nonreactive specimens were obtained from normal healthy subjects, though information related to history and/or treatment of diseases other than syphilis and HIV was not available. Approvals in accordance with federal regulations, state laws, ethics guidelines, and CDC policies were obtained prior to CDC laboratory testing. Samples received at the CDC were stored at −80°C until ready for testing, at which time they were maintained at 4°C until all testing was complete to ensure that analyses were performed in the same freeze-thaw cycle. Specimens had undergone up to four freeze-thaw cycles from the time of collection by suppliers to the time of testing at the CDC.

Syphilis serological assays.

Of the 1,406 serum samples, 754 samples were previously tested in a different CDC study as treponemal reactive, while 652 samples were treponemal nonreactive based on TP-PA (Fujirebio Diagnostics, Inc., Malvern, PA), EIA, and CIA (our unpublished data). Nontreponemal results, obtained from testing by RPR (Arlington Scientific, Inc., Springville, UT), were provided for 752 of the 1,406 samples by the serum suppliers. Repeat RPR testing for these specimens could not be performed at the CDC due to insufficient sample volume, but RPR tests were conducted for the remaining 654 samples at the CDC. All 1,406 samples were tested using a rapid lateral flow immunochromatographic assay, Syphilis Health Check (SHC), during the same freeze-thaw cycle as the other treponemal tests, according to the manufacturer's instructions (Trinity Biotech USA, Inc., Jamestown, NY). Samples and SHC test cassettes were allowed to equilibrate to room temperature before testing. Briefly, for each specimen, 25 μl of serum was dispensed into the sample well of the test cassette, followed by 200 μl of the kit's diluent containing saline buffer, detergent, and 0.1% sodium azide. Following a 10-min incubation at room temperature (15-min maximum), the results were read by evaluating the development and intensity of indicator lines in the cassette's control and test windows. On-site training for CDC laboratorians on the use of the SHC assay was provided by Trinity Biotech USA, Inc. CDC laboratorians performing the RPR and SHC tests were blinded to the results from each test and to previous results obtained at the CDC and provided by the suppliers. The nontreponemal and treponemal tests described above were performed in parallel at the CDC, and an algorithm was not followed, since the goal was to test all specimens by all assays regardless of outcome in order to fully evaluate SHC test performance.

Data and statistical analyses.

Data obtained from TP-PA, EIA, and CIA tests were considered treponemal (TR) variables, while data from SHC testing at CDC were considered SHC variables. SHC and TR variables were converted to 0 if recorded as nonreactive (−) and 1 if recorded as reactive (+). A laboratory test panel consensus (LT) variable was also created in this study to consider specimens reactive by both treponemal and nontreponemal tests or nonreactive by both tests. Based on this laboratory-derived definition, 194 samples with discordant treponemal and nontreponemal results (i.e., TR+/NTR− and vice versa) were excluded from LT variable analyses. The nontreponemal (NTR) variables were converted to 0 if recorded as nonreactive and 1 if recorded as reactive or minimally reactive (Rm). If both TR and NTR variables were recorded as 1, the LT is 1, and if both TR and NTR variables were recorded as 0, the LT is 0. Statistical analysis was performed using the R statistical programming software (R Foundation, Vienna, Austria) (43, 44). The R package popEpi was used to convert the factor variables into numerical values. Sensitivity and specificity were determined by comparing SHC assay results to either treponemal results alone or to the LT variable described above. The 95% confidence intervals for both the sensitivity and specificity were obtained using the R package epiR.

Population stratification.

The age variable in the data set was grouped into three different categories, as follows: subjects (i) age 11 to 20 years, (ii) 21 to 30 years, and (iii) ≥31 years. No restrictions (sex, HIV status, or pregnancy status) were considered for the three age stratifications. Information regarding pregnancy status is as reported by the serum supplier(s).

RESULTS

Overall analysis.

A total of 1,406 serum specimens were tested by the rapid SHC assay, and the results were first compared with standard laboratory treponemal assay results (i.e., reactive or nonreactive to TP-PA, EIA, and CIA) previously determined at the CDC. As shown in Table 1 (left), the sensitivity and specificity of the SHC assay were 88.7% (95% confidence interval [CI], 86.2 to 90.9%) and 93.1% (95% CI, 90.0 to 94.9%), respectively, compared with the treponemal-only test as a reference standard. When performing analysis based on the laboratory test panel consensus defined in Materials and Methods, 1,212 samples that met these criteria showed that sensitivity increased to 95.7% (95% CI, 93.6 to 97.2%) (Table 1, right). However, specificity remained relatively unchanged at 93.2% (95% CI, 91.0 to 95.1%).

TABLE 1.

Comparative analysis of SHC testing of 1,406 archived human serum samples with treponemal (TR) testing only, and with the laboratory test panel consensus that includes both TR and NTR results

SHC	Assay results
	Treponemal onlya		Test panel consensusb
	TR+	TR−	TR+/NTR+	TR−/NTR−
+	669	45	550	43
−	85	607	25	594

^aTreponemal assays include TP-PA, EIA, and CIA platforms. Sensitivity was 0.887 (95% CI, 0.862, 0.909), and specificity was 0.931 (95% CI, 0.909, 0.949).

^bTest panel consensus refers to results that were either reactive by both treponemal and nontreponemal (RPR) tests or nonreactive by both treponemal and nontreponemal tests; discordant treponemal and nontreponemal results were excluded from analysis. Sensitivity was 0.957 (95% CI, 0.936, 0.972), and specificity was 0.932 (95% CI, 0.910, 0.951).

Population-specific analysis.

With the availability of information pertaining to age, sex, pregnancy status, and HIV status for some of the samples, serological test results were collated for these archived specimens, and the data set was stratified into four distinct groups (Table 2). Analyses similar to those described above were then performed for each population. As shown in Table 2, SHC assay sensitivity and specificity ranged from 84.3 to 91.7% and 88.0 to 93.3%, respectively, among the three age groups compared to the reference treponemal assays, with the highest values being noted for the age group of ≥31 years. Consistent with findings obtained for the overall analysis described above, all parameters generally improved when defining positives and negatives based on the laboratory test panel consensus that includes both treponemal and nontreponemal assay results. Similar trends were observed for the populations stratified based on sex, pregnancy, and HIV infection status (Table 2). Of note, SHC testing of the HIV-positive sample set showed the lowest sensitivity compared to the treponemal results (66.7%). Although the results for the male and female populations were similar for the rapid SHC test and the traditional assays, most of the SHC assay performance parameters were lower for the female population than for the males; indeed, 100% specificity was observed for male sample testing in a comparison of the SHC assay results to laboratory test panel consensus. However, high specificity values of 100% were observed for females when this population was further stratified based on pregnancy status, while sensitivity was higher for the pregnant group, regardless of whether SHC assay results were compared to the reference treponemal assays only or both treponemal and nontreponemal test results. In summary, the majority of the data set showed sensitivity and specificity above 90% across all groups, and confidence intervals often overlapped among the groups. With the exception of the HIV-positive population, the performance parameters evaluated here generally improved further when considering the agreement of results among all three tests (the rapid SHC test and the reference treponemal and nontreponemal assays).

TABLE 2.

Population-specific determination of sensitivity and specificity of the SHC test performance compared to TR assays only or with a laboratory test panel consensus that includes both TR and NTR assays

Populationa	Assay results					Sensitivity (95% CI)		Specificity (95% CI)
		Treponemal onlyb		Test panel consensusc		Treponemal only	Test panel consensusc	Treponemal only	Test panel consensusc
	SHC	TR+	TR−	TR+/NTR+	TR−/NTR−
Age (yrs)
11–20	+	181	4	141	4	0.896 (0.845, 0.934)	0.972 (0.931, 0.992)	0.927 (0.824, 0.980)	0.922 (0.811, 0.978)
	−	21	51	4	47
21–30	+	134	3	103	3	0.843 (0.777, 0.896)	0.963 (0.907, 0.990)	0.880 (0.688, 0.975)	0.875 (0.676, 0.973)
	−	25	22	4	21
≤31	+	191	34	166	33	0.917 (0.877, 0.947)	0.946 (0.910, 0.971)	0.933 (0.908, 0.953)	0.934 (0.909, 0.954)
	−	18	474	9	468
Sex
Female	+	182	10	119	10	0.831 (0.775, 0.878)	0.937 (0.880, 0.972)	0.926 (0.868, 0.964)	0.924 (0.865, 0.963)
	−	37	125	8	122
Male	+	301	1	269	0	0.907 (0.870, 0.936)	0.968 (0.939, 0.985)	0.900 (0.555, 0.997)	1.000 (0.478, 1.000)
	−	31	9	9	5
Pregnancy status
Nonpregnant	+	46	0	42	0	0.780 (0.653, 0.877)	0.875 (0.749, 0.953)	1.000 (0.478, 1.000)	1.000 (0.292, 1.000)
	−	13	5	6	3
Pregnant	+	96	10	39	10	0.814 (0.731, 0.879)	0.951 (0.835, 0.994)	0.923 (0.863, 0.962)	0.922 (0.862, 0.962)
	−	22	120	2	119
HIV positive	+	18	4	3	4	0.667 (0.460, 0.835)	0.600 (0.147, 0.947)	0.923 (0.815, 0.973)	0.922 (0.811, 0.978)
	−	9	48	2	47

^aA total of 1,406 sample results were stratified based on age, sex, pregnancy, and HIV status. Populations may overlap across categories. For a given category, not all information was available for the remaining categories.

^bTR assays include TP-PA, EIA, and CIA platforms.

^cTest panel consensus refers to results that were either reactive by both treponemal and nontreponemal (RPR) tests or nonreactive by both treponemal and nontreponemal tests; discordant treponemal and nontreponemal results were excluded from analysis.

DISCUSSION

In our study, testing and analysis of 1,406 archived human serum samples showed that overall, the SHC assay had >85% sensitivity and >90% specificity compared to the traditional treponemal assays (TP-PA, EIA, and CIA), with sensitivity increasing further to >95% when the laboratory test panel consensus was considered (Table 1), while specificity remained relatively unchanged. These findings are similar to two previous reports (40, 41) that utilized smaller sample sizes (<250), albeit one of the studies (40) included a comparison with a single treponemal test and differences in sample matrix (whole blood versus serum) that limit direct comparisons with the current study. Nonetheless, Nakku-Joloba et al. (41) demonstrated that both the specificity and sensitivity of the SHC assay improved considerably when compared to both treponemal (TP-HA) and nontreponemal (RPR) standards, which is consistent with our findings. A caveat of the laboratory test panel consensus used in this study is that discordant results (TR+/NTR− and vice versa) were excluded from analysis. The CDC was not privy to information regarding patient history or confirmed clinical syphilis diagnosis for these sample sets; thus, it is possible that true positives or true negatives were missed. However, since TR+/NTR+ individuals are typically the most likely to require treatment, high sensitivity among this subset, as shown in this study, would be beneficial for identifying those who need treatment.

Our analysis included a large number of serum samples, allowing for stratified analyses. These stratified analyses identified variability in performance characteristics based on four categories, age, sex, pregnancy, and HIV status, with sensitivity and specificity ranging from 60.0% to 97.2% and 88.0 to 100%, respectively, depending on population and reference tests. Confidence intervals were also noted to overlap among groups and should be taken into consideration when interpreting these findings. Parameters, with the exception of specificity, improved for most of the examined populations when the laboratory test panel consensus was considered, which is consistent with patterns obtained for the overall analysis and with previous reports (41). A limitation of the stratified analyses is that differences in sample source or screening criteria among the suppliers, and their effect, if any, on downstream test performance comparisons cannot be ruled out. In addition, for a given category (age, sex, pregnancy, or HIV status), information was available for some or none of the remaining categories, meaning that there is overlap across groups in some cases, with no overlap in others. Thus, there could be other characteristics to consider for a given population, which might influence sensitivity and specificity outcomes. Analysis of samples from the HIV-positive population showed the lowest sensitivity (<70%); however, it is important to note that the sample size is relatively small for the HIV-positive data sets that include a comparison of SHC test results to treponemal results only (n = 79) and both treponemal and nontreponemal results (n = 56). The confidence intervals for sensitivity were also noted to be wide (46.0 to 83.5%), and so there is a need for a study with larger test groups to determine if this trend holds true. The reason(s) for the lower sensitivity of the SHC assay observed for the HIV-positive population in this study are unclear; however, missed or underestimated syphilis diagnoses in HIV-positive individuals have been reported for other serological assays and also depend on the algorithm used (7, 12, 45, 46). Several confounding factors that include HIV viremia, CD4 T-cell count, and subsequent impact on inflammatory and/or humoral immune responses may influence treponemal antibody levels, as well as the affinity and/or avidity of antibodies. The reactivity of these antibodies in treponemal assays may vary further depending on the test platform and T. pallidum antigen(s) used in the assays, while also recognizing that reference assays themselves may yield a certain degree of false-negative and false-positive results. The disease stage and immune status of the HIV-positive population considered in this study are unknown. The scope of our analysis is also limited, in that other high-risk groups that include immunocompromised or immunosuppressed individuals and men who have sex with men (MSM) were not evaluated. Given the increased rate of reported primary and secondary syphilis cases among MSM in the United States in recent years (2), additional SHC test evaluations for these populations are needed. Specimens from populations with a history of other treponemal/spirochete infections, or diseases other than syphilis and HIV, would also help further delineate test sensitivity and specificity in a future study.

Due to variations in sensitivity and specificity among the overall and stratified data sets, multiple measures of association that include concordance, phi coefficient, and equivalence were also calculated to gain additional perspective on test similarities (data not shown). The results suggested that while the SHC test is not equivalent to the reference treponemal and nontreponemal assays evaluated in this study, test performance did not differ by a significant degree and indeed showed strong positive relationships. However, agreement among assays based on these parameters is not necessarily accurate or conclusive due to the impact of epidemiologic factors, such as prevalence (47). There is also an additional study limitation to consider, as the archived serum samples had undergone up to four freeze-thaw cycles. Of note, a previous CDC study has shown that both IgG and IgM treponemal antibodies show no change in reactivity for up to 10 freeze-thaw cycles (48).

Despite study limitations, the current findings show that the rapid SHC test performance aligns with traditional treponemal and nontreponemal tests, though performance varies among specific populations and can also change depending on the serological assay(s) used as reference standards, as shown in previous reports. Furthermore, the influence of differences in sample type or quality, potential assay modifications, and subjective interpretations across laboratories cannot be ruled out (28). As noted above, the SHC tests in this study were performed by laboratorians who received specific training by the test manufacturer, and serum instead of whole blood was used as the test medium. Additional studies would be needed to address test performance in point-of-care settings. It is also important to note that patients with a history of treated syphilis could still test as seroreactive, as observed with other treponemal tests. Thus, appropriate diagnostic and follow-up steps should be considered. Regardless, RST, such as the SHC assay, address an important need in the syphilis diagnostics sphere, particularly in the United States, where the SHC test is currently the only FDA-cleared RST. Rapid tests, like the SHC, have the potential to improve access to immediate diagnosis and treatment, particularly during outreach programs, in resource-limited settings, or among highly vulnerable populations, such as pregnant women, where immediate treatment is an urgent concern so as to prevent loss to follow up and potential fetal complications. Indeed, outreach screening has been reported for at least four states in the United States at the community level, STD clinics, and other facilities that include high-risk populations, with treatment being administered in those showing reactive RST results while confirmatory laboratory tests are performed (28). More research is needed to determine if this approach effectively addresses loss to follow-up and reduces syphilis transmission in high-risk groups. Importantly, data from this study are consistent with previous reports and contribute to our understanding of the SHC assay efficacy and utility as an alternative serodiagnostic method, while reaffirming existing recommendations that laboratory-based reference serological assays should not be replaced (1–3, 28). Continued independent study of the SHC assay performance in various laboratory and field settings will add to a body of data intended to provide perspectives on new and existing syphilis screening practices, as well as guide future RST development.