Comparison of Manual and Fully Automated AIX1000 Rapid Plasma Reagin Assays for Laboratory Diagnosis of Syphilis

The analytical performance of the AIX1000 system, a fully automated and recently FDA-cleared rapid plasma reagin (RPR) system, was evaluated by comparison to a manual RPR test in a traditional syphilis testing algorithm. A total of 1,028 consecutive serum samples submitted for syphilis testing to the University of North Carolina Hospitals Clinical Immunology Laboratory were tested per each manufacturer's instructions.

ABSTRACT

The analytical performance of the AIX1000 system, a fully automated and recently FDA-cleared rapid plasma reagin (RPR) system, was evaluated by comparison to a manual RPR test in a traditional syphilis testing algorithm. A total of 1,028 consecutive serum samples submitted for syphilis testing to the University of North Carolina Hospitals Clinical Immunology Laboratory were tested per each manufacturer's instructions. Among those samples, 996 were nonreactive and 20 were reactive using both the ASI RPR card system and the AIX1000 system. Of the 12 discrepant specimens, 11 were AIX1000 reactive and ASI card nonreactive whereas 1 specimen was ASI card reactive and AIX1000 nonreactive. The sensitivity and specificity of the manual ASI card were 76.0% and 99.8%, respectively, while the sensitivity and specificity of the AIX100 were 100.0% and 99.4%, respectively (sensitivity P = 0.03). Among the 20 concordant reactive specimens, 68.4% of the titers agreed within ±1 dilution between methods. Reproducibility testing of the AIX1000 system demonstrated qualitative and semiquantitative (within ±1 dilution) agreement between specimens tested on different days of 96.0% and 76.0%, respectively, and 100.0% agreement between replicates within the same run. One of 24 samples analyzed under other disease conditions was reactive on both the AIX1000 system and the ASI card. Overall, the fully automated AIX1000 system demonstrated significantly enhanced sensitivity and specificity similar to that of the manual ASI RPR card test, making the AIX1000 system suitable for the laboratory diagnosis of syphilis in a clinical laboratory setting.

INTRODUCTION

Syphilis is a chronic bacterial infection caused by the spirochete Treponema pallidum subsp. pallidum that manifests in different stages, including primary (chancre), secondary (rash, malaise), and tertiary (neurological, cardiovascular, and gummatous) syphilis. Congenital syphilis can cause stillbirth or infant death, and untreated infections can enter a latent stage that lasts for months to many years. In the United States, syphilis was at the lowest reported rate in 2000, and yet the rate has increased steadily in the years since (1, 2), and syphilis remains a significant public health burden globally (3, 4). Genital ulcers have been associated with an increased rate of HIV acquisition (5); therefore, all individuals with a syphilis diagnosis are now recommended to be screened for HIV infection (6). Along with the resurgence of syphilis, the strong link with HIV highlights the critical need for accurate identification and successful treatment of individuals with syphilis.

The variable nature of syphilis symptoms makes clinical diagnosis alone challenging. Dark-field and fluorescence microscopy, as well as PCR, can be used for definitive diagnosis in the primary stage of disease, though presumptive diagnosis is more commonly performed by serological methods. The Centers for Disease Control and Prevention (CDC) recommends a tiered serology-based testing approach to minimize false-positive results and increase detection sensitivity (6). Serological screening for syphilis has traditionally been performed with nontreponemal tests such as the rapid plasma reagin (RPR) and the Venereal Disease Research Laboratory (VDRL) tests, which utilize nonspecific cardiolipin, cholesterol, and lecithin antigen mixtures and require manual visualization of flocculation reactions. To confirm a syphilis diagnosis, reactive specimens are then tested with a qualitative assay containing T. pallidum-specific antigens such as an automated chemiluminescent immunoassay (CLIA) or a manual agglutination assay such as the T. pallidum passive particle agglutination (TP-PA) assay. Nontreponemal tests are additionally performed in a semiquantitative manner (by measuring the titer) at the time of diagnosis to establish baseline reactivity. Semiquantitative serological follow-up testing to monitor treatment effectiveness is now recommended at 3, 6, 9, 12, and 24 months for HIV-infected persons and at 6 and 12 months for HIV-negative persons (6). Monitoring is performed using nontreponemal tests, as the reactivities of these typically decline following treatment. However, some individuals may remain serofast following successful treatment (7–9). Most patients with a reactive treponemal test remain reactive for life; therefore, treponemal assays are not recommended for treatment monitoring.

Results of the manual RPR test are subjectively interpreted, can be easily misinterpreted, and are subject to person-to-person variation. The sensitivity of the RPR assay ranges from 73% in latent syphilis to 100% in secondary syphilis (10, 11), though during primary syphilis, which is associated with the highest likelihood of transmittal of infection, the RPR is 86% sensitive (10, 11). Some high-volume laboratories have switched to a reverse algorithm approach starting with a treponeme-specific test (12), such as an automated CLIA, to improve sensitivity during latent syphilis and enhance throughput capabilities (13–15). However, this approach can lead to increased false-positive rates in low-prevalence populations (16, 17), difficult-to-interpret discordant results (12), and potentially higher costs and increased unnecessary treatment (18).

Thus, a need remains for a high-throughput and objectively interpreted nontreponemal screening assay. In the current study, a newly FDA-approved and fully automated RPR system, the AIX1000 system, was evaluated and its analytical performance compared to that of a manual RPR test for use in a traditional algorithm testing sequence.

MATERIALS AND METHODS

Study specimens.

We first performed a retrospective pilot study, consisting of analysis of 100 stored specimens that had been tested with a manual rapid plasma reagin (RPR) test performed in the Clinical Immunology Laboratory at the University of North Carolina Hospitals (UNCH). These samples were tested with a Gold Standard Diagnostics AIX1000 instrument (Davis, CA). Retrospective specimen selection was based on manual RPR test results obtained with 50 nonreactive (NR) specimens and 50 reactive (RX) specimens with low (1:1 to 1:2; n = 10), intermediate (1:4 to 1:8; n = 20), and high (≥1:16; n = 20) titers. An additional panel of specimens that had positive cytomegalovirus IgG (n = 4), rubella IgG (n = 5), toxoplasma IgG (n = 5), herpes simplex virus 2 (HSV2) (n = 5), and antinuclear antibody (n = 5) results was selected to evaluate the specificity of the AIX1000 data. Serum was stored at −80°C until testing. All specimens had been stored for less than 3 months.

We next performed a prospective study using consecutive serum specimens (n = 1,028) submitted for syphilis screening to UNCH. All samples were tested by the use of an ASI RPR card (Arlington Scientific Inc., Springville, UT) and of the AIX1000 RPR assay. All reactive results were confirmed by Serodia TP-PA (Fujirebio Inc., Malvern, PA). All specimens from both the retrospective and prospective studies were remnant specimens from UNCH patients over the age of 18, and approval was obtained from the University of North Carolina Institutional Review Board.

ASI RPR card test.

The ASI RPR card test is a nontreponemal flocculation assay for the detection of reagin antibodies in serum and plasma used as a screening test for serological evidence of syphilis. Manual ASI RPR card testing was performed according to the manufacturer's package insert instructions. Briefly, 0.05 ml of serum was dropped onto a separate circle of the test card for each assay and evenly spread over the circle area. One free-falling drop of the prepared antigen suspension was added to each specimen, and cards were then rotated at 100 rpm for 8 min in a humidified chamber. Results were read macroscopically under a light source following brief rotation by hand. The titers of specimens with RX results were then determined starting at 1:1, and endpoint titers were determined by serial dilution in phosphate-buffered saline (PBS) at up to a 1:16 dilution or were serially diluted in a 1:50 NR serum–PBS solution for dilutions greater than 1:16 before being tested as described above. The endpoint titer was determined as the highest dilution in which visible aggregation occurred.

AIX1000 RPR automated test system.

The AIX1000 RPR system consists of a fully automated nontreponemal flocculation reaction instrument for the qualitative and semiquantitative detection of reagin antibodies in serum to aid in the diagnosis of syphilis. Briefly, the analyzer delivers serum to microtiter plate wells, an antigen mixture is added, and plates are incubated while shaking. A camera captures an image, and a proprietary software algorithm determines the result. Reagents, microtiter plates, and controls are provided with the test kits. The titers of reactive specimens were determined according to package instructions using PBS diluent at up to a 1:16 dilution and a 1:50 NR serum–PBS diluent for dilutions over 1:16. AIX1000 titer dilutions automatically start at 1:2; therefore, screens were repeated to include 1:1 retesting to match the ASI RPR card procedure. For the retrospective study, determinations of titers were performed on all RX specimens and screens were rerun only if the titer was NR. Prospective specimens were screened, and RX specimens were then rescreened and titers were determined simultaneously.

Serodia TP-PA.

The Serodia TP-PA assay (U. S. Fujiribio and P. A. Malvern) is a qualitative gelatin particle agglutination test intended for the detection of T. pallidum antibodies in serum or plasma to aid in the diagnosis of syphilis. All testing was performed according to manufacturer's package insert instructions. Briefly, serum samples were diluted in sample diluent in microtiter plate wells to 1:20 or 1:40 and 25 μl of unsensitized or sensitized gelatin particles was added to the respective wells. Plates were shaken for 30 s and then incubated under stationary conditions for 2 h at room temperature. Results were then determined manually according to the assay instructions.

Traditional testing algorithm.

For the purpose of determining sensitivity, specificity, and positive predictive values and negative predictive values (PPV and NPV, respectively), verified positives were considered those specimens with a RX RPR result (by either method), a RX TP-PA result, and 1 additional RX treponeme-specific chemiluminescent immunoassay result (Architect Syphilis TP assay [Abbott Laboratories, Abbott Park, IL] or Liaison Treponema assay [DiaSorin Inc., Stillwater, MN]). All samples in the prospective cohort were tested with both chemiluminescent assays (A. Sanfilippo, K. Freeman, and J. L. Schmitz, submitted for publication).

Reproducibility testing.

AIX1000 quantitative intra-assay precision was assessed by repeated testing of 2 specimens (ASI card titers of 1:2 and 1:32) in 5 separate runs performed on the same day; interassay precision was assessed by testing 5 specimens (ASI card titers of 1:1, 1:2, 1:8, 1:32, and 1:64) once per day for 5 nonconsecutive days. AIX1000 qualitative precision was evaluated by screening 5 specimens (2 RX and 3 NR) once per day for 5 nonconsecutive days.

Statistical analysis.

Κ (Kappa) coefficients to estimate concordance and data from an unpaired t test performed for analysis of the time required per test were calculated using Graphpad Prism 7 (GraphPad Software, Inc. La Jolla, CA). McNemar's test was used to compare the sensitivities of the ASI card and AIX1000 RPR screen using SAS version 9.4 (SAS Institute Inc. Cary, NC). P values of <0.05 were considered statistically significant.

RESULTS

A retrospective assessment of a panel of 50 nonreactive (NR) and 50 reactive (RX) specimens previously tested at UNCH was performed to assess initial concordance between the AIX1000 system and the ASI RPR card test. Results for 48 RX and 48 NR specimens were concordant, while those for 4 specimens were discordant, providing positive percent agreement (PPA) and negative percent agreement (NPA) values of 96.0% (Κ = 0.920; 95% confidence interval [CI], 0.843 to 0.997) (Table 1).

TABLE 1.

Summary of RPR screen concordance between AIX1000 system and ASI card^a

AIX1000 parameter	ASI card value(s)
	RX (%)	NR (%)	Total (%)	% PPA	% NPA	K (95% CI)
Retrospective specimens
RX	48 (48)	2 (2)	50 (50)
NR	2 (2)	48 (48)	50 (50)
Total	50 (50)	50 (50)	100 (100)	96.0	96.0	0.920 (0.843–0.997)
Prospective specimens
RX	20 (1.9)	11 (1.1)	31 (3.0)
NR	1 (0.1)	996 (96.9)	997 (97.0)
Total	21 (2.0)	1,007 (98.0)	1,028	95.2	98.8	0.763 (0.634–0.893)

^aCI, confidence interval; Κ, kappa coefficient; NPA, negative percent agreement; NR, nonreactive; PPA, positive percent agreement; RX, reactive.

To further evaluate the analytical performance of the AIX1000 system in a representative cohort, 1,028 consecutive sera submitted for ASI RPR testing to UNCH were tested prospectively with the AIX1000 system. Initial screening identified 55 of 1,023 (5.4%) AIX1000 RX specimens, 35 of which were discordant with the ASI card, providing initial screen PPA and NPA values of 95.2% and 96.5%, respectively (Κ = 0.512; 95% CI, 0.375 to 0.649). Due to the higher-than-expected discordance rate in the prospective cohort compared to the retrospective cohort, a repeat AIX1000 screening test was added at the time that the titers were determined for the RX specimens. Following repeat testing, 31 of 55 (56.4%) AIX1000 initial RX specimens repeated as RX for a final total of 31 of 1,028 (3.0%) AIX1000 RX specimens compared with 21 of 1,028 (2.0%) ASI card RXs, providing overall RPR screen PPA and NPA values of 95.2% and 98.9%, respectively (Κ = 0.763; 95% CI, 0.634 to 0.893) (Table 1).

In comparisons of each method's screening results to verified positives identified as RPR and TP-PA and at least 1 additional positive treponeme-specific test result (Sanfilippo et al., submitted), the ASI card test identified 19 of 25 verified positive specimens whereas two specimens were ASI card RX and treponemal NR for a false-positive rate of 0.2%. Six verified positive specimens were ASI card negative, yielding a false-negative rate of 24.0%. The ASI card test had a sensitivity of 76.0% (95% CI, 54.9 to 90.6%) (Table 2) and specificity, PPV, and NPV of 99.8% (95% CI, 99.3 to 100.0%), 90.5% (95% CI, 69.6 to 98.8%), and 99.4% (95% CI, 98.7 to 99.8%), respectively (Table 2). The AIX1000 identified 25 of 25 verified positives for a sensitivity of 100.0% (95% CI, 86.3 to 100.0%; P = 0.03 compared to the ASI card). Six specimens were AIX1000 RX and treponemal NR for a false-positive rate of 0.6%, yielding a specificity of 99.4% (95% CI, 98.7 to 99.8%) and PPV and NPV of 80.7% (95% CI, 62.5 to 92.6%) and 100.0% (95% CI, 99.6 to 100.0%), respectively. One specimen was falsely positive using both the ASI card and the AIX1000 system (Table 3).

TABLE 2.

Overall prospective traditional testing algorithm comparison^a

Parameter	No. (%) of verified positives			% Sensitivity (95% CI)	% Specificity (95% CI)	% PPV (95% CI)	% NPV (95% CI)
	RX	NR	Total
ASI card
RX	19 (1.8)	2 (0.1)	21 (2.0)
NR	6 (0.6)	1,001 (97.4)	1007 (98.0)
Total	25 (2.4)	1,003 (97.6)	1028	76.0 (54.9–90.6)	99.8 (99.3–100.0)	90.5 (69.6–98.8)	99.4 (98.7–99.8)
AIX1000 system
RX	25 (2.4)	6 (0.6)	31 (3.0)
NR	0 (0.0)	997 (97.0)	997 (97.0)
Total	25 (2.4)	1,003 (97.6)	1,028	100.0 (86.3–100.0)b	99.4 (98.7–99.8)	80.7 (62.5–92.6)	100.0 (99.6–100.0)

^aCI, confidence interval; NPV, negative predictive value; NR, nonreactive; PPV, positive predictive value; RX, reactive. Verified positives were defined as those determined with any RX RPR or RX TP-PA assay and 1 additional RX treponeme-specific chemiluminescent immunoassay. (See the “Traditional testing algorithm” section in Materials and Methods.)

^bP = 0.03 compared to ASI card system data using McNemar's test.

TABLE 3.

Discordant prospective specimen summary^a

Specimen	ASI card		AIX1000		Treponemal antibody status	Interpretation
	Screen	Titer	Screen	Titer
464	NR	−	RX	1:4	RX	ASI card false negative
471	NR	−	RX	1:2	RX	ASI card false negative
613	NR	−	RX	1:1	NR	AIX false positive
629	NR	−	RX	1:8	RX	ASI card false negative
701	NR	−	RX	1:1	RX	ASI card false negative
850	NR	−	RX	1:1	RX	AIX false positive
1002	NR	−	RX	1:1	NR	AIX false positive
1009	NR	−	RX	1:1	RX	ASI card false negative
1205	RX	1:4	RX	1:1	NR	Dual false positive
1241	NR	−	RX	1:1	NR	AIX false positive
1247	NR	−	RX	1:1	NR	AIX false positive
1300	RX	1:1	NR	−	NR	ASI card false positive
1412	NR	−	RX	UNDb	RX	ASI card false negative

^aNR, nonreactive; RX, reactive. Treponemal antibody status was defined as detection of a TP-PA RX result plus 1 additional RX treponeme-specific chemiluminescent immunoassay result. (See the “Traditional testing algorithm” section in Materials and Methods.)

^bUND, unable to determine (insufficient specimen volume for titer).

The reproducibility of the AIX1000 system results was also evaluated. The level of agreement of endpoint titers of reactive specimens in the retrospective study was 94.0%, as the AIX1000 analysis resulted in 3 of 50 specimens being outside a range of ±1 2-fold dilution compared to the ASI card (Fig. 1A). Among the concordant, RX prospective specimens, AIX1000 titers matched the ASI card titer within ±1 2-fold dilution for 12 of 18 specimens for an overall prospective titer concordance of 66.7% (Fig. 1B). AIX1000 low and high titers were concordant in 2 of 2 (100.0%) specimens tested in 5 separate runs performed on the same day. Nineteen of 25 (76.0%) total replicates were quantitatively concordant and 24 of 25 (96.0%) screens were qualitatively concordant in testing performed on five nonconsecutive days.

FIG 1.

Comparison of AIX1000 titer to ASI titer. Differences in the AIX1000 titer dilutions compared to the ASI card titer dilutions of retrospective (A) and prospective (B) specimens are shown. The dotted lines represent a ±1-dilution-factor (2-fold) difference from the ASI card manual titer result. Specimens that gave discordant RPR screen results and/or were not confirmed by treponeme-specific testing are excluded.

AIX1000 specificity was assessed using a retrospective panel of serum with positive test results determined under a variety of different disease conditions (See Materials and Methods). One antinuclear-antibody-positive specimen among 24 total specimens (4.2%) was RX by both tests, suggesting good specificity of both RPR methods. Overall, the retrospective results demonstrated good qualitative concordance but various levels of semiquantitative concordance between the AIX1000 system and the ASI card.

Amounts of hands-on time seen per specimen test were also compared between the AIX1000 system and ASI card. The total time from start to result, excluding the time required to perform endpoint titer determinations, was lower for the AIX1000 system, averaging 0.31 min per specimen (954 specimens tested in 6 separate runs) compared to 0.48 min per specimen for the ASI card (490 specimens tested in 7 separate runs, P = 0.02).

DISCUSSION

In the present study, the AIX1000 system, a fully automated nontreponemal RPR screening assay recently cleared by the FDA, was evaluated and its analytical performance compared to that of a manual RPR assay for use in a traditional testing algorithm. The qualitative concordance between the manual ASI RPR card and the AIX1000 system was ≥96.0% in both the retrospective and prospective studies. In the prospective study, an additional AIX1000 screen was included when the endpoint titer determination was performed, which resulted in an initial screening false-positive rate of 43.6%. This was unexpectedly high, but including the rescreen during the titering process improved the concordance to 98.8%. Inclusion of the second screen at the time of titer determination is not part of the procedure described in the AIX1000 package insert; however, this addition adequately detected AIX1000 false-positive screens and greatly increased the concordance with the ASI card test.

The sensitivity of the manual RPR has been reported to range from 73% for late or tertiary syphilis to 100% for secondary syphilis; specificities have been reported as 93% or greater (10, 11). Using the AIX1000 system for the initial screen in a traditional testing algorithm sequence and comparing the results to treponemal-assay-verified positives resulted in improved sensitivity (100.0% versus 76.0%) and NPV (100.0% versus 99.4%) compared to the manual ASI card results. The specificity of the AIX1000 results was similar to that of the manual RPR assay; however, the PPV was lower (80.7% versus 90.5%). Considering that positive and negative predictive values are dependent on the prevalence of syphilis in the population tested, it is not surprising that the AIX1000 system demonstrated a lower PPV in the current study; likewise, enhanced sensitivity at the expense of specificity is acceptable for a screening assay.

Automated treponeme-specific serological assays are being utilized more frequently as an initial screening test in reverse algorithm syphilis screening sequences (12); however, the majority of College of American Pathologists-accredited clinical laboratories still utilize the traditional algorithm (19). While potentially identifying more patients with latent or previously treated syphilis, the reverse algorithm may yield difficult-to-interpret results with reportedly higher false-positive rates in low-prevalence populations (16, 17). Therefore, a need remains for high-throughput, accurate nontreponemal screening tests.

The CDC recommends using quantitative nontreponemal antibody titers following diagnosis to monitor treatment effectiveness (6). A 4-fold (2-dilution) titer change, determined using the same nontreponemal test, compared to the titer at the time of diagnosis is necessary to demonstrate clinical effectiveness (6). Nontreponemal titers are also used to diagnose congenital syphilis with a greater than 4-fold difference in serum titers (maternal > infant), again using the same assay, suggestive of congenital infection, though the sensitivity of this approach is considered to be low (20, 21). The levels of quantitative concordance between the AIX1000 system and the manual ASI card test were 76.7% in the retrospective study and 63.2% in the prospective study using the criterion of a ±1 2-fold dilution range as acceptable. Specimens with titers of ≥1:16 demonstrated the lowest concordance in both studies. Considering that manual RPR testing is subject to variation between technologists performing the tests, a more objective measurement would be advantageous to minimize person-to-person variations. How well the AIX1000 titer function performs for treatment effectiveness monitoring or congenital syphilis diagnostics will require further evaluation.

Manual RPR has various sensitivities depending on the disease stage, with primary and late or tertiary syphilis having the lowest sensitivities (11). A limitation of this study was the lack of detail about the patients' symptoms, reasons for testing, syphilis disease stage, and other clinical factors that may impact the interpretation of results. False-positive nontreponemal test results are reported to occur in 1% to 2% of the U.S. population and have been associated with autoimmune disease, injection drug use, pregnancy, HIV and hepatitis B virus infection, and various other conditions; titers are typically 1:8 or lower in these instances (22, 23). False negative nontreponemal tests can be attributed to the prozone effect but have also been associated with HIV infection (22). In the current study, the AIX1000 system found 10 more reactive specimens than the manual ASI card test; 6 of these were false positives based on treponeme-specific confirmatory testing. The titers of the AIX1000 false positives were all 1:1, while the titers of the ASI card false negatives ranged from 1:1 to 1:8 on the AIX1000 system. Considering that the AIX1000 system requires at least 300 μl of serum for each test, unfortunately, resolution of discordant specimens by repeated testing was not possible, as the majority of the subjects who were the sources of these specimens had had multiple tests performed for clinical purposes before being enrolled in studies.

The AIX1000 saved hands-on time compared to the manual test, excluding start-up and shut-down maintenance, which adds additional time pre- and postrun. Reagent costs were similar between the AIX1000 system and ASI card RPR, with the AIX1000 instrument representing the major cost. Considering that the time saving offered by the AIX1000 system, while consistent, was relatively small and that any financial savings would be specimen volume dependent, each individual laboratory will need to determine the specimen volume threshold that results in workflow and cost improvements over manual RPR methods.

Automated RPR-based systems are in use outside the United States. The Mediace RPR assay was reportedly less sensitive than a manual RPR (24) and was deemed unsuitable for quantitative titer analysis (25). The HiSens Auto RPR assay was also less sensitive (26, 27) though more specific (27) than a manual nontreponemal assay. An additional approach to provide more consistency in RPR evaluation is through the use of a digital flocculation reader, which in one study was found to have >98% sensitivity and specificity compared to manual evaluation for both qualitative and quantitative RPR results (28). As the AIX1000 system is the only fully automated RPR system approved for clinical use in the United States, direct comparison with other automated RPR-based systems will be of interest.

In conclusion, the AIX1000 was significantly more sensitive than the ASI card and identified 5 additional treponemal-test-confirmed reactive specimens that were missed by the manual method. As a screening assay used in a traditional algorithm setting, the AIX1000 system is an effective option to automate testing in order to improve result reporting time and to provide more-objective results than a manual RPR method. Nevertheless, additional research is needed to determine the AIX1000 performance in different syphilis disease stages and in well-defined patient populations with various levels of disease prevalence. Additionally, further assessment of AIX1000 quantitative titer function, specifically, for the diagnosis of congenital syphilis and treatment monitoring over time, is needed to fully evaluate how the AIX1000 can be utilized in high- and low-volume clinical settings.