Evaluation of Three Automated Nontreponemal Rapid Plasma Reagin (RPR) Tests for the Laboratory Diagnosis of Syphilis

ABSTRACT

Automated nontreponemal rapid plasma reagin (RPR) tests were recently introduced in the United States for syphilis testing and limited performance data are available. In collaboration with the Association of Public Health Laboratories, three public health laboratories (PHL) were chosen through a competitive selection process to evaluate the performance of three FDA-cleared automated RPR test systems: BioPlex 2200 Syphilis Total & RPR assay (Bio-Rad Laboratories), AIX 1000 (Gold Standard Diagnostics), and ASI Evolution (Arlington Scientific). Panels prepared at the CDC included: a qualitative panel comprised of 734 syphilis reactive/nonreactive sera; a quantitative panel of 50 syphilis reactive sera (RPR titer 1:64 to 1:1,024); and a reproducibility panel of 15 nonreactive and reactive sera (RPR titer 1:1 to 1:64). Panels were shipped frozen to the PHL and tested on the automated RPR systems following manufacturers’ instructions. Prior test results were blinded to all laboratories. When compared to manual RPR (Arlington Scientific) performed at the CDC as a reference test, the qualitative panel results demonstrated an overall concordance of 95.9% for AIX 1000, 94.6% for ASI Evolution, and 92.6% for Bioplex RPR; quantitative panel showed within range titer of 2-fold for 94% of specimens for AIX 1000, 68% for ASI Evolution, and 64% for BioPlex RPR, and the reproducibility testing panel demonstrated point estimates ranging from 69 to 95%. Automated RPR instruments could reduce turnaround time and minimize interpretation errors. However, additional evaluations with more specimens could assist laboratories with implementing automated RPR tests and understanding their limitations.

INTRODUCTION

Syphilis is a systemic disease caused by a spiral-shaped bacterium, Treponema pallidum subspecies pallidum (T. pallidum) that is primarily transmitted via sexual contact; it can also be acquired congenitally (congenital syphilis) and rarely by blood transfusion or other nonsexual contact. Untreated syphilis advances through multiple stages: primary, secondary, latent, and tertiary. Syphilis in general is on the rise in the United States. In 2021, CDC’s most recent sexually transmitted disease (STD) surveillance reported 176,713 cases of syphilis (all stages and congenital syphilis), that included 53,767 cases from primary and secondary stages, a rise of 28.6% during 2020 to 2021 (1). Congenital syphilis cases are increasing, with 2,855 recorded cases during 2021 including 220 stillbirths and infant deaths (1). This rise in total and congenital syphilis underscores the need for timely testing and adequate treatment to curb resurging syphilis.

Diagnosing syphilis is challenging as determining the correct stage requires using laboratory results along with clinical presentations of the disease. The two types of serological tests most commonly used to support syphilis diagnosis are nontreponemal and treponemal tests. Nontreponemal tests detect antibodies to lipoidal material released from damaged host cells as well as to a lipoprotein-like material, and possibly cardiolipin released from the treponemes (2, 3). The Venereal Disease Research Laboratory (VDRL) slide test, Unheated Serum Reagin (USR), Rapid Plasma Reagin (RPR), and Toluidine Red Unheated Serum Test (TRUST) are examples of nontreponemal tests offering qualitative results and/or quantitative titers for reactive specimens. Titers obtained using a nontreponemal test might correlate with the disease activity and are useful for monitoring a response to syphilis treatment (4). Multiple medical conditions and factors not related to syphilis (e.g., HIV) can give false-positive results with a nontreponemal test requiring follow up testing with a treponemal test for confirmation (4). Treponemal tests detect antibodies to whole T. pallidum bacterium or its purified recombinant proteins or peptides and produce qualitative results. Examples of treponemal tests include enzyme immunoassay (EIA), chemiluminescent immunoassay (CIA), T. pallidum particle agglutination (TP-PA), and rapid immunochromatographic syphilis tests. There are Food and Drug Administration (FDA)-cleared automated treponemal tests available and being increasingly used to improve laboratory workflow.

Minor modifications have been made with the improvement of manual nontreponemal tests design since their inception; hence, there is broad interest and incentive for researchers to expedite syphilis testing. Manual nontreponemal tests are useful though have a few practical limitations, including availability of skilled staff to set up, perform, and interpret results which may in turn affect specimen processing/turnaround time (5, 6). Recently, three fully automated FDA-cleared nontreponemal tests were introduced in the United States: BioPlex 2200 Syphilis Total & RPR (Bio-Rad Laboratories, Inc., CA, hereafter referred to as BioPlex RPR because only the nontreponemal component was evaluated in this work), AIX 1000 agglutination RPR analyzer (Gold Standard Diagnostics, Inc., CA, hereafter referred to as AIX 1000), and ASI Evolution automated RPR syphilis test (Arlington Scientific, Inc., UT, hereafter referred to as ASI Evolution). BioPlex RPR is based on a flow immunoassay principle whereas AIX 1000 and ASI Evolution are based on the flocculation reaction that uses nontreponemal carbon antigen. These advanced RPR platforms automate sample and regent addition, incubation, and washing steps (where required), and circumvent the subjective interpretation of the manual RPR tests. Additionally, all three automated RPR tests have an overall turnaround time of approximately 200 tests per an hour for qualitative testing that may help with improving routine workflow. However, limited performance data are available evaluating automated nontreponemal RPR tests for syphilis testing. Hence, the U.S. Centers for Disease Control and Prevention (CDC) and the Association of Public Health Laboratories (APHL) collaborated to evaluate the performance of three automated RPR test systems in comparison to manual RPR focusing reproducibility, qualitative characteristics, and titer reporting.

MATERIALS AND METHODS

CDC/APHL collaboration for automated RPR evaluation.

A request for proposal (RFP) was jointly developed by the CDC and APHL investigators, posted publicly on the APHL website (aphl.org/rfp), and used to identify three competent sites by a competitive selection process. Laboratories with experience in syphilis testing, unrestricted access to an automated RPR platform of interest, adequate resources, and an appropriate budget for the proposed work were considered. Eight public health laboratories responded to the RFP, and their proposals were reviewed, scored, and ranked by six independent qualified reviewers. Criteria such as the laboratory’s experience with the syphilis testing methods, technical merits, competency of staff with automated RPR system based on the training received from the test manufacture/PHL and respective institution’s training evaluation procedure were used to evaluate and rank RFPs. The Alabama Department of Public Health (Mobile, AL) was selected for BioPlex RPR, Oklahoma State Department of Health (Oklahoma City, OK) for AIX 1000, and Georgia Public Health Laboratory (Decatur, GA) for ASI Evolution. During January 2020 to mid-March 2020, specialized specimen panels were prepared (described below) at the STD Laboratory Reference and Research Branch (SLRRB) of the CDC and were shipped frozen on dry ice to the participating sites in three batches. All three sites completed testing within the specified time frame (February 2020 to July 2020). The PHLs sent data to CDC and APHL investigators, including laboratory-generated results along with raw data, information on repeat testing due to technical errors, observations, and feedback.

Automated RPR tests.

The BioPlex 2200 Syphilis Total & RPR is a fully automated platform that is based on the flow immunoassay principle and detects treponemal and nontreponemal antibodies. This automated test uses T. pallidum fusion protein (rTP47/rTP17) and cardiolipin coated fluoromagnetic beads and offers qualitative results for treponemal assay and qualitative/titer for nontreponemal assay (7). Only the nontreponemal component of this test system was evaluated in this study because the aim is to compare available FDA-cleared automated RPR tests. Of note, in February 2022, BioPlex RPR was recalled by the manufacturer due to concerns about false reactive results following COVID-19 vaccination and reagent instability issues contributing to elevated RPR reactivity (8–10). Because our evaluation was conducted prior to the availability of COVID-19 vaccines, its impact (potential false-positive result) on our data are negligible.

The other two fully automated RPR tests, AIX 1000 and ASI Evolution, are based on the flocculation reaction (like manual RPR) that employ cameras, proprietary hardware, and software tools to capture and interpret automated RPR test results (11–14). Key features of these automated RPR tests, such as turnaround time, specimen vial/volume requirements, and titer range are summarized in Table 1. More details on the automated RPR test platforms can be obtained from the respective manufacturers and the FDA website (11–16).

TABLE 1.

Key features of the automated RPR tests

Parameters	Automated RPR
	BioPlex RPR	AIX 1000	ASI evolution
Turnaround (tests/hour)a	200	192	190
Specimen vialb
Type	Polystyrene / polypropylene	Polystyrene / polypropylene	Polystyrene / polypropylene
Dimension (mm)	12 × 75	16 × 100	12 × 75
Specimen volb
Qualitative / Screening (μL)	240c	305d	300f
Quantitative low titer (μL)	285c	394d	290f
Quantitative high titer (μL)	405c	220e	140f
Titer rangea
Low titer	1:4 to 1:64g	1:1 to 1:16	1:1 to 1:64
High titer	1:128 to 1:2,048	1:16 to 1:256	1:128 to 1:2,048

^aDerived from product instructions for use. Refer to respective manufacturer’s instructions for more information.

^bReported based on participating public health laboratories’ inputs.

^cIncludes 230 μL of dead volume for BioPlex Syphilis Total & RPR.

^dIncludes 300 μL of dead volume for qualitative/screening and quantitative low titer for AIX 1000.

^eIncludes 200 μL of dead volume for quantitative high titer specimen for AIX 1000.

^fIncludes 190 μL of dead volume for ASI Evolution.

^gA reactive specimen with titer less than 1:4 is reported as <1:4, and for greater than 1:64 as >1:64, requiring further testing following instructions from the manufacturer’s package insert.

Specimen source and panel preparation.

Three specialized testing panels (reproducibility, quantitative, qualitative) described below were prepared at the CDC SLRRB. Four sets of each panel were prepared; three were shipped, and the fourth set was kept at the CDC for reference testing. Upon receipt, the PHLs were asked to visually verify the frozen condition of every specimen and to store all specimens frozen (−40 to −80°C) until the day of testing.

(i) Reproducibility testing panel. A reproducibility testing panel comprised of 13 syphilis reactive (RPR titer 1:1 to 1:64) and two syphilis nonreactive donor specimens (tested nonreactive by syphilis serological tests) was prepared using sera available from a commercial source in 2019 (Plasma Service Group [PSG], PA). Specimens with RPR titers of 1:128 and above were requested but unavailable when this panel was prepared. In total, 15 specimens with a volume in the range of 50 mL to 80 mL were purchased from PSG. Specimens were prepared as 5 mL aliquots per cryotube; two aliquots per specimen were sent to each PHL to perform reproducibility testing in a total of 10 separate runs; two times a day for 5 consecutive days. The laboratories were asked to run end-titer of a reactive specimen using their respective automated RPR test.

(ii) Quantitative panel. A quantitative panel comprised of 50 reactive specimens (titer of 1:64 and above) was prepared using APHL collected sera (described under qualitative panel below) to evaluate titer capability of the automated RPR test systems. State and local public health laboratories reported nontreponemal test (RPR or VDRL) titers and specimen data were used to select high titer specimens. At the time of testing, all sites, including the PHLs and SLRRB lab were blinded to the prior results (described below). Four sets of 0.5 mL aliquots were prepared and stored frozen at −80°C.

(iii) Qualitative panel. A qualitative panel comprised of 734 syphilis sera (including both reactive and nonreactive) was prepared. The sera came from a CDC/APHL collaboration where APHL regularly releases a call for residual specimens to their member PHLs to provide specimens to the CDC/APHL syphilis serum repository (17). From 2017 to 2019, 796 specimens were collected from 11 state and/or local PHLs by APHL and sent to CDC with supporting data on reported disease stage for syphilitic sera and prior serological test results (when available). Of the 796 sera submitted, 495 had volume of 2 mL or more and were included in the qualitative panel (n = 445) and quantitative panel (n = 50) development. Sera with less than 2 mL of volume (n = 301) were transferred for use in the CDC/APHL syphilis serum repository (17). Additional sera (n = 289) with a 2 mL or greater volume were included from the CDC’s Network Epidemiology of Syphilis Transmission (NEST, OMB: 0920-1248) study (18). Four sets of 0.5 mL aliquots were prepared and stored frozen at −80°C. Prior to specimen collection and use, approvals in accordance with federal regulations, state laws, and ethics guidelines were obtained.

Testing.

(i) Automated RPR testing. All specimens were kept frozen (−40 to −80°C) until the day of testing. Once thawed, sites were instructed to store specimens at 2°C to 8°C and complete testing within 5 consecutive days or 120 h. Two to five testing personals from each PHL were trained on the use of automated RPR tests and participating sites successfully evaluated competency and proficiency of staff involved in this testing. PHLs followed respective manufacturer’s instructions for the automated RPR testing method for qualitative and quantitative testing. For test run verification, controls were run each time per respective manufacture/a testing laboratory’s standard procedure. If a specimen required endpoint titration beyond the range of the automated RPR system, PHLs were asked to follow manufacturer’s guidance and continue testing. For BioPlex RPR, the titration range is 1:4 to 1:64. For reactive specimens with titer less than 1:4 or greater than 1:64, a manual dilution and additional testing is required per manufacturer’s instructions. The titration range for BioPlex RPR could be extended to 1:2,048. ASI Evolution titration range extends from 1:1 to 1:2,048. The titer range for AIX 1000 is 1:1 to 1:256. Of note, participating PHL that evaluated AIX 1000 followed manufacturer’s recommended testing method until 1:256 dilution; thereafter sera were manually diluted, tested, and read using a qualitative screening protocol on the instrument. Due to issues obtaining reagents for BioPlex RPR, the site was unable to test 208 specimens from the qualitative panel. All remnant specimens from the testing sites were shipped back to CDC on dry ice. PHLs completed all the testing as planned and sent each instrument generated raw data along with imported data captured on Excel file templates for analysis to the CDC.

(ii) Reference testing. Specimens from all of the prepared panels had undergone at least two freeze-thaw cycles during testing at the CDC and the PHLs. Panels were thawed in batches and specimens were tested using manual RPR (qualitative and quantitative, Arlington Scientific, UT) following manufacturers’ instructions. Additionally, testing panels were subjected to TP-PA (qualitative, Serodia TP-PA, Fujirebio, TX) following manufacturers’ instructions to have both nontreponemal and treponemal results. Two experienced testing personnel performed all the testing at the SLRRB using a single lot of reagents/kits for RPR and TP-PA tests. Obtained data were recorded in Excel file for further analysis/comparison. Of note, reactive sera from the reproducibility panel were tested five times to determine an endpoint titer and obtained results were used for comparison. Sera from the qualitative and quantitative panels were only tested once.

Data analysis.

(i) Analysis of reproducibility testing. Each PHL repeated testing of specimens from the reproducibility testing panel 10 times to ascertain reproducibility of the respective automated RPR test system. For RPR reactive sera, an endpoint titer was determined based on the respective automated RPR testing method. The titers obtained from the automated RPR tests were compared to the manual RPR titers. The titer of an automated RPR test that was within 2-fold (1 dilution) relative to manual RPR titer was considered within range, and the remainder were considered out-of-range. This criterion was kept as a standard for both the reproducibility and quantitative panel data analyses. A point estimate for reproducibility was calculated with 95% confidence intervals (CI). Obtained results were analyzed using R software tool with an assistance from an experienced statistician (19).

(ii) Analysis of quantitative testing. Titers obtained using automated RPR tests were compared to manual RPR titers, and Spearman’s correlation coefficient was calculated. Spearman's rank-order correlation assessed a monotonic association of each automated RPR system with manual RPR; the correlation coefficient ranges from −1 (strong negative correlation) to 1 (strong positive correlation) including 0 (no correlation). Results were analyzed using R software tool (19).

(iii) Analysis of qualitative testing. The sample size for the qualitative panel was calculated to capture 95% and above sensitivity and specificity of automated RPR tests. The number and percentage of sera tested using automated RPR tests that showed consistent qualitative results compared to manual RPR tests was calculated. Obtained specimen data were stratified in context to reported status for analysis. Of the 734 sera, 185 specimens were staged for syphilis, 140 had unknown syphilis stage, and 409 were nonreactive. The overall percent agreement, positive percent agreement (PPA) and negative percent agreement (NPA) along with 95% CI, and kappa correlation coefficient (chance-adjusted agreement measure) were calculated for automated RPR tests using R software (19). Obtained automated RPR results were compared to combined manual RPR, and TP-PA results to confirm syphilis status of the specimens and to further evaluate automated RPR tests performance with possible discordant results.

RESULTS

Reproducibility performance.

Table 2 summarizes reproducibility testing data obtained for three automated RPR tests. The panel for reproducibility testing was repeatedly tested five times in manual RPR at SLRRB and obtained results are shown in Table 2. Based on manual RPR as a reference test, the titers obtained for reactive sera from the reproducibility panels were grouped into three categories: low titer (1:1 to 1:2), moderate titer (1:4 to 1:16), and high titer (1:64). Testing of low titer reactive sera (n = 4) yielded 60% to 100%, 100%, and 30% to 100% agreement for BioPlex RPR, AIX 1000, and ASI Evolution, respectively. Except for AIX 1000, the other two automated RPR systems gave nonreactive results with low titer sera as shown in Table 2. For testing of moderate titer reactive sera (n = 7), BioPlex RPR showed 40% to 100% agreement for five specimens; however, two specimens (A1 and A15) were out of range with 4- to 8-fold higher titers relative to manual RPR titer for all of 10 repeated runs. AIX 1000 yielded 100% agreement with the manual RPR for six specimens; however, one specimen (A14) with a manual RPR titer of 1:16 had a 4-fold less titer (out of range) for seven out of 10 repeats. ASI Evolution gave 70% to 100% agreement for this group of specimens. Testing of high titer sera (n = 2) gave 100%, 90% to 100%, and 80% agreement, respectively, for BioPlex RPR, AIX 1000, and ASI Evolution. Testing of RPR nonreactive sera (n = 2) yielded 100% agreement for all three automated RPR systems. The point estimates calculated were 68.7% (CI = 60.9% to 75.5%), 94.7% (CI = 89.8% to 97.3%), and 86.0% (CI = 79.5% to 90.7%) for BioPlex RPR, AIX 1000 and ASI Evolution, respectively.

TABLE 2.

Results of the reproducibility panel testing for the automated RPR tests

Automated RPRa tests	Reproducibility testing panel (n = 15)
	Specimens	A7	A10	A6	A2	A8	A9	A3	A12	A11	A1	A4	A14	A15	A5	A13	Point estimate % (95% CI)
	RPR	Nonreactive (n = 2)		Low titer (n = 4)				Moderate titer (n = 7)							High titer (n = 2)
	RPR titer	NR	NR	1:1	1:2	1:2	1:2	1:4	1:4	1:8	1:16	1:16	1:16	1:16	1:64	1:64
	TP-PA	NR	NR	R	R	R	R	R	R	R	R	R	R	R	R	R
Out of total 10 repeats, ‘X’ times a specimen gave an end point titer as shown below
BioPlex RPR	NR	10b	10	4	-	4	4	-	-	-	-	-	-	-	-	-	68.7 (60.9 to 75.5)
	1:1	-c	-	4	-	-	-	-	-	-	-	-	-	-	-	-
	1:2	-	-	2	10	6	6	-	-	-	-	-	-	-	-	-
	1:4	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
	1:8	-	-	-	-	-	-	7	4	-	-	-	-	-	-	-
	1:16	-	-	-	-	-	-	3	6	7	-	-	6	-	-	-
	1:32	-	-	-	-	-	-	-	-	3	-	7	4	-	-	6
	1:64	-	-	-	-	-	-	-	-	-	10	3	-	5	-	2
	1:128	-	-	-	-	-	-	-	-	-	-	-	-	5	10	2
	% Agr	100	100	60	100	60	60	70	40	70	0	70	100	0	100	100
AIX 1000	NR	10	10	-	-	-	-	-	-	-	-	-	-	-	-	-	94.7 (89.8 to 97.3)
	1:1	-	-	7	6	8	6	-	-	-	-	-	-	-	-	-
	1:2	-	-	3	4	2	4	8	9	-	-	-	-	-	-	-
	1:4	-	-	-	-	-	-	2	1	3	-	-	7	-	-	-
	1:8	-	-	-	-	-	-	-	-	6	1	7	3	-	-	-
	1:16	-	-	-	-	-	-	-	-	1	9	3	-	10	-	-
	1:32	-	-	-	-	-	-	-	-	-	-	-	-	-	3	6
	1:64	-	-	-	-	-	-	-	-	-	-	-	-	-	4	4
	1:128	-	-	-	-	-	-	-	-	-	-	-	-	-	2	-
	1:256	-	-	-	-	-	-	-	-	-	-	-	-	-	1	-
	% Agr	100	100	100	100	100	100	100	100	100	100	100	30	100	90	100
ASI Evolution	NR	10	10	5	3	-	1	-	-	-	-	-	-	-	-	-	86.0 (79.5 to 90.7)
	1:1	-	-	1	2	1	1	-	-	-	-	-	-	-	-	-
	1:2	-	-	2	-	4	5	3	2	-	-	-	-	-	-	-
	1:4	-	-	2	5	5	3	3	2	2	1	3	1	-	-	-
	1:8	-	-	-	-	-	-	3	6	4	5	2	5	-	-	1
	1:16	-	-	-	-	-	-	1	-	4	3	2	4	3	2	1
	1:32	-	-	-	-	-	-	-	-	-	1	3	-	7	1	4
	1:64	-	-	-	-	-	-	-	-	-	-	-	-	-	7	4
	1:128	-	-	-	-	-	-	-	-	-	-	-	-	-	-	-
	% Agr	100	100	30	70	100	90	90	100	100	90	70	90	100	80	80

^aR = reactive; NR = nonreactive; RPR = rapid plasma reagin; TP-PA = T. pallidum particle agglutination; %Agr = percentage agreement; CI = confidence interval.

^bGray cells represent within-range specimens that had titers within 2-fold (or 1 dilution) to manual RPR titers, and white cells represent out-of-range specimens that had titers outside within-range acceptance criterion. All RPR nonreactive sera by manual RPR gave nonreactive results for automated RPR tests and are shown in gray background.

^c-, no data.

Quantitative performance.

Testing of a quantitative panel on all three automated RPR systems yielded titers that were within range (2-fold, or 1 dilution) for 64% to 94% of specimens when compared to manual RPR data (Table 3). Out-of-range titers occurred in 36% of specimens for BioPlex RPR (4- to 8-fold higher titer); 6% for AIX 1000 (4-fold lower titer); and 32% for ASI Evolution (4- to 8-fold higher and lower). For ASI Evolution, 13 specimens (26%) were repeatedly tested due to reported technical errors (see Table 3 and footnotes for details). Spearman’s correlation coefficients calculated were 0.746, 0.893, and 0.162 for BioPlex RPR, AIX 1000, and ASI Evolution, respectively.

TABLE 3.

Results of the quantitative panel testing for the automated RPR tests

Automated RPRf tests		Specimens (n = 50)
	n	23	16	8	2	1
	RPR	1:64	1:128	1:256	1:512	1:1,024
	TP-PA	R	R	R	R	R
BioPlex RPR (titer)
1:32		1	-	-	-	-
1:64		8	-	-	-	-
1:128		8	2	-	-	-
1:256		5	8	1	-	-
1:512		1	5	2	2	-
1:1,024		-g	1	4	-	-
1:2,048		-	-	-	-	-
Above 1:2,048		-	-	1	-	1
Within rangea
n (%)		17 (73.9)	10 (62.5)	3 (37.5)	2 (100)	0 (0)
Total n (%)		32 (64)
Out-of-range n (%)b
n (%)		6 (26.1)	6 (37.5)	5 (62.5)	0 (0)	1 (100)
Total n (%)		18 (36)
Spearman's correlation coefficient (p-value)c		0.746 (p = 5.2 × 10−10)
AIX 1000 (titer)
1:32		16	2	-	-	-
1:64		7	8	-	-	-
1:128		-	4	5	-	-
1:256		-	2	3	1	1
1:512		-	-	-	1	-
1:1,024		-	-	-	-	-
1:2,048		-	-	-	-	-
Within rangea
n (%)		23 (100)	14 (87.5)	8 (100)	2 (100)	0 (0)
Total n (%)		47 (94)
Out-of-range n (%)b
n (%)		0 (0)	2 (12.5)	0 (0)	0 (0)	1 (100)
Total n (%)		3 (6)
Spearman's correlation coefficient (p-value)c		0.893 (p = 3.1 × 10−18)
ASI evolution (titer)
1:32		-	-	-	-	-
1:64		17d	11e	7	1	-
1:128		1d	4e	1	1	1
1:256		1	-	-	-	-
1:512		2d	1	-	-	-
1:1,024		1d	-	-	-	-
1:2,048		1d	-	-	-	-
Within rangea
n (%)		18 (78.3)	15 (93.8)	1 (12.5)	0 (0)	0 (0)
Total n (%)		34 (68)
Out-of-range n (%)b		5 (21.7)	1 (6.2)	7 (87.5)	2 (100)	1 (100)
n (%)		5 (21.7)	1 (6.2)	7 (87.5)	2 (100)	1 (100)
Total n (%)		16 (32)
Spearman's correlation coefficient (p-value)c		0.162 (p = 0.262)

^aWithin range (gray cell) = specimen showed titer within 2-fold (or 1 dilution) to manual RPR titer.

^bOut-of-range = Specimen showed titer outside within range acceptance criterion.

^cSpearman's rank-order correlation test assessed a monotonic association of each automated RPR system with manual RPR; the correlation coefficient ranges from −1 (strong negative correlation) to 1 (strong positive correlation) including 0 (no correlation).

^dTotal 10 specimens reported technical errors and gave nonreactive results that upon repeated gave 1:64 (n = 6), 1:128 (n = 1), 1:512 (n = 1), 1:1,024 (n = 1), 1:2,048 (n = 1) titer and were included in respective category as indicated.

^eA total of three specimens reported technical errors that gave nonreactive results that upon repeated testing gave 1:64 (n = 2), 1:128 (n = 1) titer and were included in respective category as indicated.

^fR = reactive; RPR = rapid plasma reagin; TP-PA = T. pallidum particle agglutination; n = number; % = percentage.

^g-, no data.

Qualitative performance.

As shown in Table 4, testing of 185 syphilis staged specimens in all three automated RPR tests and comparison to manual RPR showed concordance ranging from 79% to 100% for reactive specimens. Twenty syphilis staged specimens gave nonreactive results in manual RPR and had concordance ranging from 40% to 100% to automated RPR tests; AIX 1000 detected slightly more cases for primary and secondary sera compared to other two automated RPR tests. Testing of 140 syphilis specimens of unknown status in all three automated RPR tests gave concordance of 82% to 100% for reactive and 79% to 86% for nonreactive sera. For manual RPR nonreactive specimens, all automated RPR tests showed concordance of 96% to 100% as shown in Table 4.

TABLE 4.

Results of the qualitative panel testing for the automated RPR tests and comparison to manual RPR data

Status	Specimen	Manual RPR		Automated RPR tests
				BioPlex RPR		AIX 1000		ASI evolution
	n	Status	n	R n (%)	NR n (%)	R n (%)	NR n (%)	R n (%)	NR n (%)
Syphilis staged specimens
Primary	24	R	19	18a (100)	0a (0)	18 (94.7)	1 (5.3)	15k (78.9)	4k (21.1)
		NR	5	1b (25)	3b (75)	3 (60)	2 (40)	1l (20)	4l (80)
Secondary	43	R	41	37c (100)	0c (0)	40 (97.6)	1 (2.4)	39m (95.1)	2m (4.9)
		NR	2	0 (0)	2 (100)	2 (100)	0 (0)	1 (50)	1 (50)
Early latent	38	R	34	33 (97.1)	1 (2.9)	33 (97.1)	1 (2.9)	28 (82.4)	6 (17.6)
		NR	4	1 (25)	3 (75)	1 (25)	3 (75)	1 (25)	3 (75)
Early NPNSq	14	R	14	11d (84.6)	2d (15.4)	14 (100)	0 (0)	13 (92.9)	1 (7.1)
Late latent	66	R	57	47e (83.9)	9e (16.1)	55 (96.5)	2 (3.5)	53n (93)	4n (7)
		NR	9	1 (11.1)	8 (88.9)	1 (11.1)	8 (88.9)	0 (0)	9 (100)
Syphilis specimens with unknown stage	140	R	64	41f (82)	9f (18)	64 (100)	0 (0)	60o (93.8)	4o (6.2)
		NR	76	7g (14)	43g (86)	16 (21.1)	60 (78.9)	11p(14.5)	65p (85.5)
Nonreactive or not diagnosed with syphilis	409	R	2	1h (100)	0h (0)	1 (50)	1 (50)	2 (100)	0 (0)
		NR	407	8i,j (3.2)	240i,j (96.8)	1 (0.2)	406 (99.8)	5 (1.2)	402 (98.8)
Total	734	-r	-	206a-i (39.2)	320a-i (60.8)	249 (33.9)	485 (66.1)	229 (31.2)	505 (68.8)
Overall concordance % (95% CI)				92.6 (90.3 to 94.8)		95.9 (94.5 to 97.3)		94.6 (92.9 to 96.2)
PPA % (95% CI)				90.6 (87.7 to 93.5)		93.6 (91.5 to 96)		91.3 (88.6 to 94)
NPA % (95% CI)				93.9 (92 to 95.8)		97 (95.9 to 98)		96.0 (94.8 to 97.2)
Kappa correlation % (95% CI)				84.5 (79.8 to 89.2)		90.7 (87.5 to 94)		87.3 (83.5 to 91.2)

^a-iReagents were not available to test 208 specimens under described categories as “a” (n = 1), “b” (n = 1), “c” (n = 4), “d” (n = 1), “e” (n = 1), “f” (n = 14), “g” (n = 26), “h” (n = 1), and “i” (n = 159) were excluded from data calculation.

^jTwo specimens gave errors during testing; were repeatedly tested, gave nonreactive results and were added into “NR” (238 + 2 = 240).

^kOne specimen gave NR result; upon repeated testing it again gave NR result (repeated due to technical error).

^lOne specimen gave NR result; upon repeated testing it again gave NR result (repeated due to technical error).

^mThree specimens gave NR results; upon repeated testing two gave R result and were added into “R” (37 + 2 = 39) (repeated due to technical error).

ⁿThree specimens gave NR results; upon repeated testing two gave R and were added into “R” (51 + 2 = 53) (repeated due to technical error).

^oFour specimens gave NR results; upon repeated testing three gave R results and were added into “R” (57 + 3 = 60) (repeated due to technical error).

^pFive specimens gave NR results; upon repeated testing one gave R results and were added into “R” (10 + 1 = 11) (repeated due to technical error).

^qEarly NPNS = early nonprimary nonsecondary syphilis; RPR = rapid plasma reagin; R = reactive; NR = nonreactive; n = number; % = percentage; PPA = positive percent agreement; NPA = negative percent agreement.

^r-, no data.

The overall concordance of automated RPR tests to manual RPR was 92.6% (CI = 90.3% to 94.8%) for BioPlex RPR, 95.9% (CI = 94.5% to 97.3%) for AIX 1000, and 94.6% (CI = 92.9% to 96.2%) for ASI Evolution. Positive percent agreement (PPA) and negative percent agreements (NPA) calculated were: for BioPlex RPR, 90.6% (CI 87.7% to 93.5%) and 93.9% (CI 92.0% to 95.8%); for AIX 1000, 93.6% (CI = 91.5% to 96%) and 97% (CI = 95.9% to 98%); and for ASI Evolution, 91.3% (CI = 88.6% to 94%) and 96.0% (CI = 94.8% to 97.2%). Kappa correlation coefficients (95% CI) determined were 84.5% (CI = 79.8% to 89.2%) for BioPlex RPR, 90.7% (CI = 87.5% to 94%) for AIX 1000, and 87.3% (CI = 83.5% to 91.2%) for ASI Evolution.

Qualitative results for the automated RPR tests were compared to the combined manual RPR/TP-PA results as shown in Table 5. Specimens with manual RPR reactive results were confirmed reactive in TP-PA for staged and syphilis specimens with unknown stage, except for one specimen from late latent and two with reported status as nonreactive or not diagnosed with syphilis (shaded light gray, RPR+/TP-PA−). Two specimens from the reported primary stage yielded nonreactive results for manual RPR, TP-PA, and all three automated RPR tests. No other data were available for review for these two primary specimens. All three automated RPR tests detected RPR/TP-PA reactive sera in range of 90% to 98% as shown in Table 5. RPR−/TP-PA+ results were noted (shaded in dark gray) for both syphilis staged and unknown stage syphilis specimens, of which 14.7% were reactive in BioPlex RPR, 24.5% for AIX 1000, and 14.9% for ASI Evolution. For nonreactive or not diagnosed with syphilis specimens, comparison of manual RPR-/TP-PA- to automated RPR tests showed concordance ranging from 97% to 100%.

TABLE 5.

Results of the qualitative panel testing for the automated RPR tests and comparison to combined manual RPR/TP-PA data

Status	Specimen n	Manual tests			Automated RPR tests
					BioPlex RPR		AIX 1000		ASI evolution
		RPR	TP-PA	n	R n (%)	NR n (%)	R n (%)	NR n (%)	R n (%)	NR n (%)
Syphilis staged specimens (n = 185)
Primary	24	R	R	19	18a (100)	0a (0)	18 (94.7)	1 (5.3)	15k (78.9)	4k (21.1)
		NRr	R	3	1 (33.3)	2 (66.7)	3 (100)	0 (0)	1l (33.3)	2l (66.7)
		NR	NR	2	0b (0)	1b (100)	0 (0)	2 (100)	0 (0)	2 (100)
Secondary	43	R	R	41	37c (100)	0c (0)	40 (97.6)	1 (2.4)	39m (95.1)	2m (4.9)
		NR	R	2	0 (0)	2 (100)	2 (100)	0 (0)	1 (50)	1 (50)
Early latentq	38	R	R	34	33 (97.1)	1 (2.9)	33 (97.1)	1 (2.9)	28 (82.4)	6 (17.6)
		NR	R	4	1 (25)	3 (75)	1 (25)	3 (75)	1 (25)	3 (75)
Early NPNS	14	R	R	14	11d (84.6)	2d (15.4)	14 (100)	0 (0)	13 (92.9)	1 (7.1)
Late latent	66	R	R	56	47 (83.9)	9 (16.1)	54 (96.4)	2 (3.6)	52n (92.9)	4n (7.1)
		NR	R	9	1 (11.1)	8 (88.9)	1 (11.1)	8 (88.9)	0 (0)	9 (100)
		R	NR	1	0e (0)	0e (0)	1 (100)	0 (0)	1 (100)	0 (0)
Syphilis specimens with unknown stage	140	R	R	64	41f (82)	9f (18)	64 (100)	0 (0)	60o (93.8)	4o (6.2)
		NR	R	76	7g (14)	43g (86)	16 (21.1)	60 (78.9)	11p (14.5)	65p (85.5)
Syphilis cases combined (staged and unknown)	325	R	R	228	187a,c,d,f (89.9)	21a,c,d,f (10.1)	223 (97.8)	5 (2.2)	207 (90.8)	21 (9.2)
		NR	R	94	10g (14.7)	58g (85.3)	23 (24.5)	71 (75.5)	14 (14.9)	80 (85.1)
		R	NR	1	0e (0)	0e (0)	1 (100)	0 (0)	1 (100)	0 (0)
		NR	NR	2	0b (0)	1b (100)	0 (0)	2 (100)	0 (0)	2 (100)
Nonreactive or not diagnosed with syphilis	409	R	NR	2	1h (100)	0h (0)	1 (50)	1 (50)	2 (100)	0 (0)
		NR	R	1	1 (100)	0 (0)	0 (0)	1 (100)	0 (0)	1 (100)
		NR	NR	406	7 (2.8)a,j	240i,j (97.2)	1 (0.2)	405 (99.8)	5 (1.2)	401 (98.8)
Total	734	-s	-	734	206a-i (39.2)	320a-i (60.8)	249 (33.9)	485 (66.1)	229 (31.2)	505 (68.8)

^a-iReagents were not available to test 208 specimens under described categories as “a” (n = 1), “b” (n = 1), “c” (n = 4), “d” (n = 1), “e” (n = 1), “f” (n = 14), “g” (n = 26), “h” (n = 1), and “i” (n = 159) were excluded from data calculation.

^jTwo specimens gave errors during testing; were repeatedly tested, gave nonreactive results and were added into “NR” (238 + 2 = 240).

^kOne specimen gave NR result; upon repeated testing it again gave NR result (repeated due to technical error).

^lOne specimen gave NR result; upon repeated testing it again gave NR result (repeated due to technical error).

^mThree specimens gave NR results; upon repeated testing two gave R result and were added into “R” (37 + 2 = 39) (repeated due to technical error).

ⁿThree specimens gave NR results; upon repeated testing two gave R and were added into “R” (50 + 2 = 52) (repeated due to technical error).

^oFour specimens gave NR results; upon repeated testing one gave R results and were added into “R” (57 + 3 = 60) (repeated due to technical error).

^pFive specimens gave NR results; upon repeated testing one gave R results and were added into “R” (10 + 1 = 11) (repeated due to technical error).

^qEarly NPNS = early nonprimary nonsecondary syphilis; RPR = rapid plasma reagin; TP-PA = T. pallidum particle agglutination; R = reactive; NR = nonreactive; n = number; % = percentage; PPA = positive percent agreement; NPA = negative percent agreement.

^rGray cells represent discordant RPR/TP-PA results.

^s-, no data.

DISCUSSION

Here, we describe the performance evaluation of three FDA-cleared automated RPR tests that were recently introduced for syphilis testing in the United States. To our knowledge, this is the first evaluation that compared three automated RPR tests to manual RPR using developed specialized specimen panels to assess reproducibility, qualitative, and quantitative reporting simultaneously. A higher test reproducibility was recorded for the AIX 1000 (94.7%) compared to other two automated RPR test systems (86% for ASI Evolution and 68.7% for BioPlex). Sanfilippo et al. performed reproducibility testing of AIX 1000 and reported 76% agreement between specimens tested on different days and 100% within the same run of replicates using a limited number of specimens (20). In our analysis, we expanded the reproducibility testing of automated RPR tests to collect comprehensive evaluation data. Testing of low RPR titer sera (1:1 and 1:2) in BioPlex RPR and ASI Evolution gave nonreactive results for three out of four sera but all were reactive in AIX 1000 with expected tiers. Likewise, AIX 1000 continued to demonstrate within range titers for moderate (1:4 to 1:16) and high titer sera (1:64) except for the one that had RPR titer of 1:16 (4-fold less than expected) for seven repeats and may be due to equipment’s limitations to generate expected titers. BioPlex RPR yielded out-of-range, elevated titers of 4- to 8-fold compared to manual RPR for moderate reactive sera though two sera of high titers were within acceptable range when compared to manual RPR. Noted elevated titer for BioPlex RPR may align with the published work from Arbefeville et al. where higher titers recorded for few reactive specimens in comparison to manual RPR (21). Variations in titer reporting (slightly lower or higher titer than expected) were noted for ASI Evolution when tested for reproducibility. Notably, only few published data are available on reproducibility testing for BioPlex RPR and ASI Evolution limiting our comparison analysis, and, in this regard, our work adds new performance data to the scientific literature.

Our analysis of the quantitative testing panel for automated RPR tests and comparison to manual RPR showed a strong positive Spearman’s coefficient correlation for AIX 1000 compared to the BioPlex RPR and ASI Evolution systems. Like the reproducibility panel data, BioPlex RPR demonstrated elevated titers for reactive sera from the quantitative panel. Published data from Tesfazghi et al. for BioPlex RPR showed 78% concordance to manual RPR (2-fold or 1 dilution) though most specimens evaluated in their work had a titer of 1:4 or less (22). The titer concordance data for BioPlex RPR from the quantitative panel showed 64% agreement which is less than previously reported (22). ASI Evolution had titers that were out-of-range and gave a lower Spearman’s correlation coefficient for the tested quantitative panel.

Overall, the variations in RPR titers noted for automated RPR platforms may be due to the differences in test sensitivity and a result reporting mechanism (21). Although most sera (94%) tested from the quantitative panel had RPR titers of up to 1:256 or less, our generated data provide valuable insights on the titer potential of the automated RPR tests and determine their concordance to manual RPR test. The limited number of RPR reactive specimens were tested at the lower end (1:4 and below) and the higher end (1:512 and above) from the panels warranting further evaluation of automated RPR systems. The titration of an RPR reactive serum to determine an endpoint/reporting on an automated RPR test should concord to the widely established manual RPR (quantitative) test. The more robust the correlation between these two methods, the greater the benefit to laboratories considering a transition from a manual RPR to an automated RPR platform for both qualitative and quantitative testing and the greater confidence clinicians will have in the results when managing syphilis.

A greater concordance of the AIX 1000 to the manual RPR test was recorded for qualitative testing and aligns with prior published work from Sanfilippo et al. (20). The performance of BioPlex RPR reported by Tesfazghi et al. had PPA and NPA of 85% and 98%, respectively (22). Our evaluation data showed a higher PPA of 91% but a lower NPA of 94% for BioPlex RPR. Of note, the sensitivity of BioPlex RPR appeared to be reduced for latent staged specimens over AIX 1000 and ASI Evolution systems. Data for ASI Evolution yielded a high overall concordance to manual RPR, though lower sensitivity was recorded for primary and early latent staged specimens. No prior published study is available for ASI Evolution for comparison and in this regard, our evaluation adds new data to the literature.

Testing of sera from the qualitative panel using RPR and TP-PA confirmed the reported syphilis status of for staged and unstaged specimens; AIX 1000 showed a higher sensitivity for RPR/TP-PA reactive sera compared to other two automated RPR tests. Review of discordant results as RPR−/TP-PA+ representing past or current (potentially early) syphilis and comparison to automated RPR data showed more reactive cases detected for AIX 1000, indicating a higher test sensitivity. Our obtained data confirmed an earlier observation by Sanfilippo et al. that AIX 1000 detected more “treponemal” reactive specimens that otherwise were missed by a manual RPR method (20). The BioPlex RPR and ASI Evolution showed less sensitivity for this discordant group (RPR−/TP-PA+) of specimens, yet are in agreement to manual RPR data. Specimens with RPR+/TP-PA− are typically, biologically false reactive specimens and most yielded reactive results for all three automated RPR tests evaluated. There were two specimens reported as primary stage syphilis (Table 5) but gave nonreactive results for manual and automated RPR as well as TP-PA tests. These specimens might have been collected from the early primary stage at where the sensitivity of a serological test is reduced or patients have yet to seroconvert with antibodies (3). No other relevant data were provided for these two primary specimens limiting further investigation.

This evaluation was designed to collect performance data for automated RPR tests that have the potential to support and improve routine workflows for syphilis testing. Our evaluation data and prior published reports collectively show promising performance of automated RPR tests, particularly for qualitative testing. For quantitative testing, variabilities in titer reporting were recorded for automated RPR tests. A variability in nontreponemal test titers of 4-fold (2 dilutions) or greater to baseline visit (prior to treatment) could have significant clinical implications as titer declines of 4-fold or greater imply adequate treatment, while increases of 4-fold or greater indicative of reinfection or treatment failure (4, 23). Laboratories considering a switch from manual to automated RPR should ensure that they communicate with clinicians about the potential differences between automated and manual RPR titers. Ongoing dialogue between clinicians and laboratorians has been helpful when implementing automated RPR systems. Further evaluation of a larger number of specimens, particularly at the lower and higher end of the titer ranges would inform and strengthen our findings. Given that each automated RPR test has a predefined titer range, laboratories must factor this in while diluting a reactive specimen. Due to clinical implications, all reactive specimens should be diluted until an endpoint titer is achieved, either by following manufacturer’s instructions for an automated RPR test, or a manual RPR test if an endpoint titer cannot be determined on an automated RPR platform due to the instrument’s titer range limitation. Likewise, inaccurate titer reporting as <1:4 or > 1:256 could lead to unnecessary treatment, or follow-up lab visits for a patient (24).

This evaluation has a few limitations. The frozen sera used at the SLRRB and PHLs went through two freeze-thaw cycles. The impact of freeze-thaw cycles on assay performance warrants further testing using freshly collected sera. The syphilis staging of specimens is based on the records submitted by jurisdiction and its accuracy could not be verified. Prozone specimens were not available at the time of panel preparation, limiting our investigation in this arena. At the time of sample collection and this evaluation, the COVID-19 vaccines were not available; hence, the potential interference on assay performance is negligible. The issue of reagent stability for BioPlex RPR as reported on the FDA website for Class 2 device recall was not evaluated (9, 10).

The three automated RPR tests evaluated have a relatively rapid turnaround for qualitative testing that may help a clinical laboratory with improving routine workflow. Given the fact that manual RPR, VDRL and/or other nontreponemal tests have well-documented limitations of labor-intensive testing methods and subjective interpretation, an automated platform could address such challenges. In addition, automated RPR tests record results electronically; either in terms of antibody index for BioPlex RPR or images of test wells for AIX 1000 and ASI Evolution, enabling a laboratory to review and verify results. This evaluation provides necessary data on the performance of automated RPR tests and determines concordance to manual RPR for qualitative testing. Additional testing and evaluation with a larger specimen pool could establish a greater concordance to manual RPR testing and assist laboratories with implementing automated RPR tests for syphilis testing.