Performance of the Syphilis Health Check in Clinic and Laboratory-Based Settings

A study to evaluate clinical and laboratory performance of the Syphilis Health Check, found improved sensitivity when testing with whole blood compared to serum, in detecting likely active syphilis.

Abstract

Background

In this study, we evaluate the performance of the Syphilis Health Check (SHC) in clinical and laboratory settings using fingerstick whole blood and serum.

Methods

Fingerstick whole blood and serum specimens from adult patients (n = 562) without prior syphilis history presenting at 2 county health department STD clinics in North Carolina were tested. Fingerstick specimens were tested with the SHC in clinic, and serum specimens were tested at the North Carolina State Laboratory of Public Health with: (1) qualitative rapid plasma reagin, (2) treponemal EIA, and (3) SHC. Sensitivity and specificity were calculated with 95% confidence intervals.

Results

The fingerstick whole blood had a sensitivity of 100% (7 of 7) and specificity of 95.7% (531 of 555), compared with consensus reference testing (CRT) (rapid plasma reagin and EIA reactive), but a sensitivity of 50% (8 of 16), and specificity of 95.9% (523 of 546), when compared with the treponemal EIA. Both laboratory-based SHC on serum and whole-blood SHC performed similarly, compared with CRT, and the treponemal EIA alone. Twenty-four specimens SHC reactive on whole blood were nonreactive by CRT. In 8 of these 24 cases, STD clinic staff reported difficulty reading the test line for the SHC. Of the fingerstick whole-blood SHC reactive specimens, only 14 of 31 were also serum SHC reactive.

Conclusions

The SHC on whole blood appears to be sensitive at detecting patients likely to have syphilis and could be an option for testing among high-risk populations. However, given challenges in interpreting SHC test results, adequate training of persons performing testing and ongoing quality assurance measures are key.

Syphilis is a curable sexually transmitted infection caused by the spirochete Treponema pallidum subspecies pallidum. In the United States, there has been a notable increase in the rates of primary and secondary syphilis in the past 2 decades, with a significant burden of disease impacting gay, bisexual, and other men who have sex with men. In recent years, there have also been concerning increases in syphilis among women of reproductive age and in congenital syphilis.¹ In the 2017 STD surveillance report by the Centers for Disease Control and Prevention (CDC), increase in incidence rates of 72.77% and 153.3% was reported for primary and secondary syphilis and congenital syphilis, respectively, when compared with the 2013 reported data.¹

The diagnosis of syphilis using laboratory methods is challenging, as accurate diagnosis requires 2 types of serological tests used in combination, along with clinical examination and a sexual history. Nontreponemal assays (eg, rapid plasma reagin, RPR) detect nonspecific serum antibodies directed against lipoidal antigens released from damaged host cells and may be used to monitor response after therapy. Treponemal assays (eg, TP-PA Treponema pallidum particle agglutination assay) detect specific serum antibodies directed against T. pallidum proteins, and generally remain reactive for life after treatment.²

Standard serological tests for syphilis require trained laboratory staff, equipment and reagents that may not be readily accessible in many clinic sites, particularly in resource poor settings or remote areas. In these cases, specimens must be transported to central facilities for testing, sometimes resulting in protracted turnaround times. Patients may experience delays in treatment, leading to continued transmission of infection.

The use of rapid point-of-care testing for syphilis could help address some of these issues by facilitating faster linkage to treatment and partner notification efforts. Although several assays are already available on the global market, only one is currently approved by the United States Food and Drug Administration and waived under US Clinical Laboratory Improvement Amendments (CLIA). CLIA waived tests are cleared by the Food and Drug Administration and approved for waiver under CLIA. The CLIA waived tests must be simple enough to be used by nonlaboratory staff and have a low risk for erroneous results. The Syphilis Health Check (SHC) (Trinity Biotech, Jamestown, NY) is a lateral flow immunochromatographic rapid test that detects treponemal antibodies, which can be performed on serum, plasma, or fingerstick whole blood, with results available in under 20 minutes. There are little published data on use of SHC in real-world clinical settings, including test performance when compared with other standard laboratory tests.

The main objectives of this study were to compare the performance of SHC in a clinical setting using fingerstick whole-blood specimens (fingerstick whole-blood SHC) versus 3 comparators: (1) a consensus reference standard of RPR plus treponemal EIA (to examine SHC's performance in patients likely to have current syphilis); (2) laboratory-based treponemal EIA (to compare SHC's performance against a widely used treponemal screening test); and (3) SHC in the laboratory on serum (to identify issues related to specimen type or operator). We also compared performance of the laboratory-based SHC on serum (serum SHC) versus the: (1) consensus reference standard (RPR plus EIA) or (2) laboratory-based treponemal EIA.

MATERIALS AND METHODS

Study Population

Participants were enrolled consecutively from 2 study sites in North Carolina, both county health department STD clinics (county A and county B). Study sites were selected to represent counties with a moderate and high incidence of primary and secondary syphilis, as defined by the CDC.³ In 2015, county A's case rate was 5.6 cases per 100,000, whereas county B's was 17.6 cases per 100,000.³

All adults (male and female) presenting to either study site for syphilis testing (for routine screening or symptoms) between May 2015 and July 2015 were approached to enroll in the study. Patients with a known history of past syphilis infection were excluded. For the purposes of the study, history of syphilis infection was determined by patient self-report, chart review of existing medical records at the STD clinic where the patient presented for care, and review of prior treponemal serology results at the North Carolina State Laboratory of Public Health. All participants provided informed consent. The study protocol was approved by the North Carolina Department of Health and Human Services, Division of Public Health Institutional Review Board before initiation of the study.

Testing/Treatment Procedures: County STD Clinics

All participants provided fingerstick whole-blood specimens for the SHC performed at the clinic. Serum specimens obtained through blood draw and sent to the laboratory for testing (Fig. 1). The distributor of the SHC provided on-site training for staff at participating study sites and the laboratory on how to perform and interpret the SHC according to the manufacturer's instructions. County STD clinic staff were instructed not to share SHC results with the treating physician or the enrollee since the test was still under evaluation. STD clinic staff provided qualitative data when submitting results for the study if the person performing the SHC had difficulty interpreting the results. Treatment for syphilis followed the standard operating protocols for each clinic and was based on the results of standard laboratory-based testing, not the SHC results.

Figure 1.

Specimen collection and testing flow plan at county clinics and at North Carolina State Laboratory of Public Health.

Testing Procedures: North Carolina State Laboratory of Public Health

All serum specimens sent to the laboratory were tested with: (1) qualitative RPR (Sure-Vue RPR; biokit/INOVA Diagnostics, Inc, San Diego, CA) plus quantitative RPR on reactive sera to determine antibody titer, (2) Trep-Sure EIA (Trinity Biotech, Jamestown, NY), and (3) SHC. Laboratory staff were blinded to the fingerstick whole-blood SHC results obtained at the clinic until the conclusion of all of the laboratory testing.

Data Analysis

Sensitivity and specificity for the fingerstick whole-blood SHC were calculated against 3 different comparators.

Analysis 1: Fingerstick whole-blood SHC compared to consensus reference standard of RPR and treponemal EIA (serum). True positives were defined as RPR reactive, EIA reactive. True negatives were defined as: (1) RPR nonreactive, EIA nonreactive; (2) RPR reactive, EIA nonreactive (biologic false positive RPR); and (3) RPR nonreactive, EIA reactive (old syphilis or possible false positive EIA).

Analysis 2: Fingerstick whole-blood SHC compared with treponemal EIA (serum). True positives were defined as EIA reactive. True negatives were defined as EIA nonreactive.

Analysis 3: Fingerstick whole-blood SHC compared to serum SHC (serum). True positives were defined as serum SHC reactive. True negatives were defined as serum SHC nonreactive.

Sensitivity and specificity for the serum SHC were calculated against the consensus reference tests (RPR plus EIA), and treponemal EIA alone, using the same case definitions as described above.

Sensitivity and specificity confidence intervals were calculated using exact Clopper-Pearson confidence intervals. Confidence intervals for the predictive values are the standard logit confidence intervals given by Mercaldo et al.⁴ Analyses were performed using STATA/IC Version 11 (Stata Corp., College Station, TX).

RESULTS

A total of 562 participants were enrolled in the study, 279 (49.6%) at county A and 283 (50.4%) at county B. Demographic characteristics and testing results for the participants are described in table 1. The median age of the participants was 26 years (IQR 22–31 years) and the majority were female (58.9%). Among the 562 specimens, 7 (1.25%) were reactive by the consensus reference standard (RPR reactive, EIA reactive), 16 (2.9%) were reactive with the treponemal EIA, 18 (3.2%) were reactive with serum SHC, and 31 (5.5%) were reactive with the fingerstick whole-blood SHC.

TABLE 1.

Demographic and Summary Data

Performance of the Fingerstick Whole-blood SHC

Compared to the consensus reference standard (ie, RPR plus EIA), the fingerstick whole-blood SHC had a sensitivity of 100% (7 of 7) and specificity of 95.7% (531 of 555) (table 2A). A total of 24 specimens were reactive with the fingerstick whole-blood SHC but consensus reference standard negative (RPR nonreactive and EIA nonreactive). In one-third of these cases (8 of 24) STD clinic staff reported difficulty reading the test line for SHC. Compared to the treponemal EIA alone, the fingerstick whole-blood SHC had a sensitivity of 50% (8 of 16) and specificity of 95.9% (523 of 546) (table 2B). Compared to the serum SHC, the fingerstick whole-blood SHC had a sensitivity of 77.8% (14 of 18) and specificity of 96.9% (527 of 544) (table 2C).

TABLE 2A.

SHC in Clinic (Fingerstick Whole Blood) Versus CRT at Laboratory (Serum)

TABLE 2B.

SHC in Clinic (Fingerstick Whole-Blood Versus Treponemal EIA at Laboratory (Serum)

TABLE 2C.

SHC at Clinic (Fingerstick Whole-Blood) Versus SHC at Laboratory (Serum)

Performance of Laboratory SHC on Serum

Compared to the consensus reference standard, SHC on serum had a sensitivity of 100% (7 of 7) and specificity of 98% (544 of 555) (table 3A). Compared with the treponemal EIA, the serum SHC had a sensitivity of 43.8% (7 of 16) and specificity of 98% (535 of 546 (table 3B). Nine specimens were treponemal EIA reactive, serum SHC nonreactive, and RPR nonreactive.

TABLE 3A.

SHC at Laboratory (Serum) Versus CRT at Laboratory (Serum)

TABLE 3B.

SHC at Laboratory (Serum) Versus Treponemal EIA at Laboratory (Serum)

DISCUSSION

To our knowledge, this is the first assessment of the performance of the SHC in clinic and laboratory settings on 2 different specimen types (fingerstick whole blood and serum). In our analysis, the SHC using both specimen types had high sensitivity and acceptable specificity for detecting patients likely to have syphilis (RPR reactive and EIA-reactive). Although there are no US-based standards for rapid test performance for syphilis, the World Health Organization Sexually Transmitted Diseases Diagnostics Initiative (SDI) suggests acceptable sensitivity ranges of 85% to 98% and specificity ranges of 93% to 98%, when compared with a reference standard.⁵ The fingerstick whole-blood SHC and the serum SHC fulfilled these criteria when compared to consensus reference testing (RPR and EIA). Previous reports of other point-of-care tests (POCT) outside of the United States have yielded mixed results, with some tests not meeting Sexually Transmitted Diseases Diagnostics Initiative performance criteria.^6,7

However, SHC in both fingerstick whole blood and serum had poor sensitivity compared to a treponemal EIA (Trep-Sure), which is used as an initial test in the reverse sequence algorithm. In a prior analysis of SHC by Matthias et al,⁸ the authors reported a sensitivity of 71.4% and specificity of 91.5% for the SHC performed on fingerstick whole blood compared with treponemal EIA performed on serum. The SHC had a greater sensitivity in Matthias et al. as compared with our study (71.4% vs. 50%) but a lower specificity (91.5% vs. 95.8%). The poor sensitivity of SHC in both studies is concerning, however, there are conflicting data regarding the treponemal EIA (Trep-Sure) test performance and its suitability as a reference standard. In studies by Wong et al⁹ and Busse et al,¹⁰ Trep-Sure demonstrated 99% and 94% specificity, compared to TP-PA and FTA-ABS as the reference standard. In contrast, analyses by the CDC in both high and low seroprevalence populations demonstrated 18.6% to 25.2% of Trep-Sure EIA reactive specimens were TP-PA nonreactive, possibly reflecting false-positive EIA results.¹¹ In a recent study by Park et al,¹² Trep-Sure was significantly less specific (82.8%) than other treponemal immunoassays (94.5–98.5%). Given the concerning specificity data with Trep-Sure, it may not be the most appropriate reference standard for other treponemal tests such as the SHC.

When comparing the SHC on fingerstick whole-blood versus laboratory-based SHC on serum, we found positive and negative discordant specimens. Only 14 (45.1%) of 31 reactive fingerstick whole-blood SHC specimens were also reactive with the SHC on serum. Matthias et al also found that 16 (61.5%) of 26 whole-blood SHC reactive specimens did not confirm in the laboratory with either a treponemal or nontreponemal test.⁸ Multiple factors could contribute to differences in performance between the 2 specimen types. Whole blood could result in background coloration that made interpretation of the colored bands more challenging as compared to the nearly colorless serum samples. Clinic staff reported vague/faint lines and difficulties in reading results with fingerstick whole-blood samples in some cases, which may have led to more reactive SHC readings. Timing of interpretation may have played a role, as fingerstick whole blood was tested immediately and the serum was tested after transport to the laboratory. Although clinic and laboratory staff were both trained by the manufacturer before initiation of the study, background familiarity with similar lateral flow devices likely varied by operator. More data are needed to evaluate the performance of POCTs in the field and laboratory with different specimen types and operators to identify factors associated with differential test performance.

Our study is subject to several limitations. We analyzed the performance of SHC against several laboratory-based reference standards and assays, but we did not have clinically characterized sera to be able to determine true disease status. Although we attempted to exclude patients with a prior history of syphilis, misclassification is possible and could bias our test performance estimates, particularly for participants where syphilis history was provided by self-report only. Although we observed differences in SHC performance based on specimen type, we are unable to determine whether time from specimen collection to testing or other operator issues are responsible for these differences.

Like all treponemal tests, the SHC alone cannot distinguish between current or past infections or be used to monitor treatment. A reactive SHC result should be followed by a nontreponemal assay (eg, RPR) to confirm an active syphilis infection.¹³ If there is concern that the patient could be lost to follow-up or the patient is considered high risk for infection (ie, symptomatic or the sexual partner to a confirmed case), then current recommendations are to presumptively treat for syphilis.¹⁴ In addition to the limitations already listed, it should be added that the sample size of fingerstick whole-blood and serum samples that were both SHC+ and CRT+ was small (n = 7), therefore, although the sensitivity is listed as 100%, the confidence intervals are wide, so these results should be interpreted with caution.

In conclusion, the CLIA-waived version of the SHC performed in a clinic setting on fingerstick whole blood appears to be sensitive at detecting patients likely to have syphilis (RPR and EIA-reactive). However, positive SHC results should be confirmed with laboratory-based testing and adequate training of persons performing SHC testing and ongoing quality assurance are key. Given the lack of data on its performance and the wide availability of other laboratory-based treponemal assays, it is premature to use the SHC for screening or as the confirmatory treponemal test in a laboratory setting. Further studies should be conducted in poor resource areas, or areas outside of standard laboratory capabilities, to determine which settings would benefit the most from the use of POCTs such as SHC. The data from such studies would provide more data on performance of POCTs in these settings in the United States. These are areas where prompt identification of likely syphilis cases, prompt treatment, linkage to care and partner notification would be highly beneficial.