Evaluation of Orthogonal Testing Algorithm for Detection of SARS-CoV-2 IgG Antibodies

Abstract

Background

Severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) antibody testing is an important tool in assessment of pandemic progress, contact tracing, and identification of recovered coronavirus disease 2019 (COVID-19) patients. We evaluated an orthogonal testing algorithm (OTA) to improve test specificity in these use cases.

Methods

A two-step OTA was applied where individuals who initially tested positive were tested with a second test. The first-line test, detecting IgG antibodies to the viral nucleocapsid protein, was validated in 130 samples and the second-line test, detecting IgG antibodies to the viral spike protein in 148 samples. The OTA was evaluated in 4333 clinical patient specimens. The seropositivity rates relative to the SARS-CoV-2 PCR positivity rates were evaluated from our entire patient population data (n = 5102).

Results

The first-line test resulted in a clinical sensitivity of 96.4% (95% CI; 82.3% to 99.4%), and specificity of 99.0% (95% CI; 94.7% to 99.8%), whereas the second-line test had a sensitivity of 100% (95% CI; 87.1% to 100%) and specificity of 98.4% (95% CI; 94.2% to 99.5%). Using the OTA, 78/98 (80%) of initially positive SARS-CoV-2 IgG results were confirmed with a second-line test, while 11/42 (26%) of previously diagnosed COVID-19 patients had no detectable antibodies as long as 94 days post PCR diagnosis.

Conclusion

Our results show that an OTA can be used to identify patients who require further follow-up due to potential SARS CoV-2 IgG false positive results. In addition, serological testing may not be sufficiently sensitive to reliably detect prior COVID-19 infection.

Introduction

Severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2), a novel coronavirus that originated in the city of Wuhan, China in late 2019, has spread rapidly throughout the world resulting in pandemic of coronavirus disease 2019 (COVID-19). Despite the relatively short timeframe of this outbreak, a considerable number of laboratory tests targeting some aspect of the virus or disease state have been developed (1). As with other viruses, nucleic acid amplification tests (NAAT) have been employed as a sensitive marker of virus presence within various clinical samples, most notably in respiratory specimens. The development of such assays has had a positive impact on clinical diagnosis and epidemiologic endeavors. In addition to diagnostic NAATs, serological tests increasingly are being employed in the populations to assess the progress of the pandemic and in individuals to identify recovered cases of COVID-19 in previously undiagnosed and/or asymptomatic individuals (2, 3).

One of the early challenges with serological testing was that, due to the initial lack of regulatory oversight from US Federal Drug Administration (FDA), several serological tests with inadequately verified performance characteristics were marketed (4, 5). This led to reporting of inaccurate results. Some early serological tests reported high rates of false positive results due to assay cross-reactivity with other antibodies, such as those against the other endemic human coronaviruses (6, 7). Fortunately, automated serology tests developed by reputable in vitro diagnostic (IVD) vendors are now readily available and are being evaluated by clinical laboratories (8–12).

Given that current utility of serological testing is mostly for epidemiological studies, proper evaluation of the clinical sensitivity and specificity of these tests within the tested population is very important. The test specificity directly affects its positive predictive values (PPV) and thus the reliability of a positive result.

One of the testing strategies recommended by Center for Disease Control (CDC) to improve PPV of a test result is an orthogonal approach, a two-step algorithm where individuals who initially test positive are tested with a second test (13). We evaluated an orthogonal testing approach utilizing two immunoassays validated clinically at the Medical University of South Carolina (MUSC) clinical laboratories. The first-line test is Abbott SARS-CoV-2, targeting nucleocapsid (N) protein. The second-line test is a laboratory-developed enzyme-linked immunosorbent assay (ELISA), targeting spike (S) protein. With this strategy involving testing for antibodies against two different immunogenic targets, we aimed to improve the specificity of a positive test, thus providing important information to patients and public health officials.

Materials and Methods

Patient Cohorts

All method validation specimens were collected from apparently healthy volunteers or derived from de-identified, residual clinical specimens. Per MUSC Institutional Review Board (IRB) policy, this project was classified as a clinical laboratory quality improvement project and was exempted from IRB review and approval.

Clinical specificity samples were a combination of residual clinical specimens sent to our laboratory before the COVID-19 outbreak (prior to December 2019) and sera collected from asymptomatic volunteer donors in April 2020. The sensitivity samples were either de-identified, residual, serum specimens collected from patients at least 14 days following polymerase chain reaction (PCR) diagnosis of COVID-19 or prescreened convalescent plasma donors provided by Vitrologic. Most specimens were available in sufficient quantities to use in evaluation of both first and second-line tests.

The two-step algorithm was evaluated using de-identified, residual serum specimens received in our laboratory for SARS-CoV-2 IgG testing from mid-April through May 2020.

First-Line Test

The first-line test in our algorithm is the automated Abbott SARS-CoV-2 IgG assay (Abbott Laboratories). This assay has been granted Emergency Use Authorization (EUA) by the US Food and Drug Administration (FDA). It is a two-step chemiluminescent microparticle immunoassay, designed to detect IgG antibodies to the viral N protein. The test requires 75 µL of serum or plasma and is completed within approximately 29 min. This is a qualitative test, reporting a signal to cutoff (S/C) index, with the cut-off of 1.4.

To verify performance characteristics of Abbott SARS-CoV-2 IgG assay established by a vendor, a total of 130 samples were analyzed. Clinical sensitivity was determined using 28 known seropositive samples, collected from patients with previously documented COVID-19 infection (n = 18) and provided by Vitrologic (n = 10). Seronegative (specificity) samples consisted of 102 specimens collected from patients with no evidence of SARS-CoV-2 infection, collected either prior to the global COVID-19 outbreak or from asymptomatic volunteers in South Carolina.

Second-Line Test

Antibodies that bind to the surface glycoprotein of the virus, termed spike (S) protein, were determined using the serological assay described earlier (14). Briefly, the 96-well microtiter plates were coated overnight at 4.0°C using the commercially available S-protein (LakePharma, Cat # 46328) at 2 µg/mL. Plates were then washed thrice and blocked for 1 h with PBS-Tween plus 3% milk powder (weight/volume). The diluted samples were then added to each well and incubated for 2 h, after which the plates were washed and anti-human IgG (Fab specific) HRP labeled secondary antibody 1:3000 in PBS-T containing 1% milk was added for 1 h. Subsequently, plates were washed and substrate (o-phenylenediamine dihydrochloride, OPD solution; Sigma-Aldrich) was added. The reaction was stopped by adding 50 µL of 3 mol/L HCl to all wells before reading at 490 nm to record the data. Inactivated human AB serum was used as a negative control, while monoclonal antibody CR3022 was used as a positive control to ensure assay reproducibility.

This ELISA test was validated using a total of 148 samples. Clinical sensitivity was assessed using 26 known seropositive samples. All positive samples with sufficient quantities were shared between the two assays. Clinical specificity was determined using a total of 122 specimens collected from patients from pre-COVID-19 or with no evidence of SARS-CoV-2 infection.

Real Time RT- PCR Tests

Three emergency use authorization (EUA) real-time RT-PCR tests were used to evaluate the patients included in this study for detection of SARS-CoV-2 RNA in nasopharyngeal swab samples collected from individuals suspected of having COVID-19 by their healthcare provider.

The Abbott RealTime SARS-CoV-2 EUA test using the m2000 sp/rt system was performed at MUSC (15). Real-time RT-PCR amplification and detection of three targets, SARS-CoV-2 RdRp gene, SARS-CoV-2 N gene, and an internal control (hydroxypyruvate reductase gene from the pumpkin plant), took place simultaneously in the same reaction. The test has a reported limit of detection of 100 copies/mL (15).

The Centers for Disease Control (CDC) 2019-nCoV Real-time RT-PCR Diagnostic Panel (16) performed at a commercial referral laboratory and the South Carolina State Public Health Laboratory was also used to evaluate some of the individuals included in this study. The test is performed on the Applied Biosystems 7500 Fast Dx Real-time PCR instrument. The panel consisted of 3 separate reactions targeting 2 regions within the SARS CoV-2 N gene, N1 and N2, and the human RNase P gene (RP) as an internal control. This test has a reported LOD of 1000 copies/mL.

Finally, a subset of 5 specimens with documented COVID-19 infection was provided by University of Chicago. They used the Roche cobas SARS-CoV-2 assay on the 6800 system and the Cepheid Xpert Xpress SARS-CoV-2 assay, as previously described (10).

In2 patients reported to be positive for SARS-CoV-2 RNA, the test and performing laboratory could not be determined.

Orthogonal (Two-Step) Testing Algorithm

All negative results obtained by the first-line test were considered negative and no further action was taken. All initially positive results by the first-line test were followed up with the second-line test. Only if both tests were positive (i.e., if IgG antibodies against both N and S proteins of SARS-CoV-2 were detected), was the result considered positive. The testing algorithm was evaluated using the initial 100 first-line test positive patient results. Of these, the second-line test could not be performed for 2 specimens due to insufficient sample quantity. Thus, the total number of patients initially positive by the first-line test was 98. Total number of individual patients tested was 4333. Testing occurred April through May 2020. All discrepant samples were sent to a major referral laboratory (RL) for adjudication. A RL that offers a test to detect IgG antibodies against SARS-CoV-2 N-protein by an alternate methodology (RL N) was selected. In addition to discrepant specimens, a subset of 20 positive samples confirmed by orthogonal testing was sent to RL to confirm positive agreement between the 2 different immunoassays targeting N protein. Finally, clinical sensitivity of serological testing was verified by monitoring seroconversion rates in patients with known COVID-19 diagnosis by PCR, at ≥14 days post-diagnosis. This analysis was based on all patients tested between the clinical testing launch (late April) through late June 2020. Total number of patients analyzed during this time was 5102.

Data Analysis

Clinical sensitivities and specificities along with corresponding Wilson 95% confidence intervals (CI) were calculated using a clinical method validation software package (Analyze-it for Microsoft Excel). Patient demographics (gender and age), PCR, and serology testing statistics were provided by MUSC Health Analytics using Tableau software. All serology results including S/C index values and final interpretative results were extracted from our laboratory information system (Cerner Millennium). The paired analysis of discordant results was performed using univariate distribution plots (Analyze-it software).

Results

Test Performance Characteristics Evaluation

Clinical sensitivity and specificity characteristics for the first-line test (Abbott SARS-CoV-2 IgG) are summarized in Table 1. Of the samples tested (n = 130), there was one false negative and one false positive, resulting in clinical sensitivity of 96.4% (95% CI, 82.3% to 99.4%) and clinical specificity of 99.0% (95% CI, 94.7% to 99.8%).

Table 1.

Clinical sensitivity and specificity for Abbott ARCHITECT SARS-CoV-2 IgG test.

	Test positive	Test negative	Total
SARS-CoV-2 Infection	27	1	28
Asymptomatic/Pre-Pandemic	1	101	102
Total	28	102	130
% Sensitivity (95% CI)	96.4% (82.3% to 99.4%)
% Specificity (95% CI)	99.0% (94.7% to 99.8%)

Clinical sensitivity and specificity characteristics for the second-line, S protein, ELISA are summarized in Table 2. Of 148 samples tested, no false negatives and 2 false positives were observed, resulting in clinical sensitivity of 100% (95% CI, 87.1% to 100%) and clinical specificity of 98.4% (95% CI, 94.2% to 99.5%).

Table 2.

Clinical sensitivity and specificity for ELISA SARS-CoV-2 IgG test.

	Test positive	Test negative	Total
SARS-CoV-2 Infection	26	0	26
Asymptomatic/Pre-Pandemic	2	120	122
Total	28	120	148
% Sensitivity (95% CI)	100% (87.1% to 100%)
% Specificity (95% CI)	98.4% (94.2% to 99.5%)

Given high clinical sensitivities and specificities of both tests, selecting a test with higher specificity as a first-line test and a test with higher sensitivity as the second-line test minimized the number of discordant results reported, without affecting the number of positive results and PPV of the test (13).

Two-Step Orthogonal Testing Algorithm Evaluation

The two-step testing algorithm was evaluated using a total of 4333 individual clinical patient specimens analyzed April through May 2020. The age distribution in our population was ≤ 18 years (2%), 19–49 (49%), 50–59 (18%), 60–69 (17%), 70–79 (12%) and ≥ 80 (2%). Of the patients tested, 98 were initially positive by a first-line test, resulting in 2.3% seroprevalence rates. We were able to confirm 78/98 (80%) of seropositive results using the two-test approach. The resulting discordances in 20 samples indicate that our true seroprevalence rates may be <2%. The distribution of all initially positive specimens (Abbott test S/C of 1.4) is shown in Fig. 1. As the figure shows, all but one of the discordant samples were below Abbott SARS-CoV-2 IgG S/C index of 3.5. The distribution of all specimen results tested at RL N is shown in Fig. 2. Following the analysis of 20 discordant specimens by RL N (Fig. 2, A), 2 were confirmed as positive for IgG against SARS-CoV-2 N-protein, 3 were indeterminate, and a total of 15 discordant results were not confirmed by an alternate RL N IgG test. However, 4/20 of results positive by the MUSC two-step testing approach were negative by RL N, and 1/20 was indeterminate (Fig. 2, B). All discrepant MUSC positive patient results were retested at MUSC and confirmed as positives by ELISA targeting the receptor binding domain (RBD) of S protein (see Supplemental Materials for assay characteristics) as well as the entire S protein. Figure 2, A also shows that all 3 tests show excellent concordance for samples with high Abbott S/C index values (>3.5). Virtually all MUSC discordant samples, with discrepant RL N results had Abbott S/C index values <3.5 (Fig. 2, B).

Fig. 1.

Distribution of all initially positive (n = 98) Abbott SARS-CoV-2 IgG results (S/C > 1.4). Discordant samples (n = 20) were negative on MUSC ELISA, and Positive (n = 78) were positive on both tests.

Fig. 2.

Distribution of all MUSC results sent to RL for analysis: (A) all MUSC discordant results (n = 20) and (B) a subset of results positive by both MUSC tests (n = 20). All results are shown relative to Abbott SARS-CoV-2 IgG Index results (S/C > 1.4).

Finally, within the total tested population (n = 5102), a record of COVID-19 diagnosis by PCR was available for 42 patients. Of those, 11 patients (26%) had no detectable SARS-CoV-2 IgG antibodies and one patient had discordant antibody results (Table 3). The seronegative patients were tested at 14–94 days post-diagnosis.

Table 3.

Verification of clinical sensitivity of 2-step orthogonal SARS-CoV-2 IgG serological testing on 5102 clinical samples collected by MUSC from South Carolina residents for clinical testing April–late June 2020.

		SARS-CoV-2 PCR results
		PCR positive ≥14 days prior to antibody test	No prior positive PCR
SARS-CoV-2 IgG Results	Positive	30	67
	Negative	11	4977
	Discordant	1	16

Discussion

The current outbreak of SARS-CoV-2, as well as the global response efforts aimed at controlling the spread of the virus, has significantly impacted daily life worldwide. Although there are uncertainties that still remain regarding utility of antibody testing—namely duration, strength, and specificity of the immune response—it is a valuable tool that can help monitor disease progress and determine disease prevalence. Our testing strategy and experience provide data from a large cohort of patients in South Carolina, a state that has experienced a low burden of disease from SARS-CoV-2 as of late May 2020. Based on our data, seroprevalence in our population sample is 1.8% (78/4333). Of note, this population sample is susceptible to selection bias among a relatively healthy, predominantly Caucasian group of healthcare workers in a health system with exceptionally low numbers of documented COVID-19 infections from low-incidence areas of South Carolina. Analysis of population data revealed that virtually all discordant results were among the patients without prior positive PCR diagnosis (Table 3). Surprisingly, our data analytics also revealed that approximately a quarter of patients with prior COVID-19 PCR diagnosis had no detectable antibodies 14–94 days post diagnosis, which raises a question whether current serology tests are sensitive enough to detect low levels of SARS-CoV-2 IgG antibodies or whether serologic response to SARS-CoV-2 wanes quickly in some recovered patients. We did not perform follow-up serological testing of seropositive individuals in our cohort to assess for the latter possibility. It also may reflect some false-positive PCR test results since the true diagnostic accuracy of none of the many molecular tests for SARS-CoV-2 RNA currently in use under EUA is known and is likely to vary between different tests and laboratories (17). Of note, at this time only symptomatic patients were tested by PCR.

Using the orthogonal testing approach, with a highly specific first-line test and a more sensitive second-line test, we were able to confirm 80% of positive SARS-CoV-2 IgG results. The potential false positive rate of 15%–20% is to be expected in low-prevalence population and given our first-line test sensitivity and specificity. The side-by-side analysis of the Abbott test, MUSC ELISA, and RL N revealed that the Abbott SARS-CoV-2 IgG test results with S/C Index >3.5 are likely true positive results and would not require additional follow-up (see Fig. 1). Patient results with an S/C Index value between 1.4 and 3.5 (about 50% of all initially positive results for our institution) require additional follow-up to exclude a possibility of discordance due to difference in the kinetics of antibody response. Unfortunately, the RL N test could not effectively adjudicate MUSC discordant results, most likely due to its inferior sensitivity. As shown in Fig. 2, B, we were not able to confirm 5/20 MUSC positive results with the RL N assay. One was indeterminate, and 4 samples were negative by the RL N test. All had Abbott S/C Index value <3.5. This suggests that sensitivity of the RL N is not sufficient to detect antibodies at these levels, since the likelihood of false positive in these samples is low. Given that all but one MUSC discordant results had Abbott S/C Index value <3.5, most (18/20) were either indeterminate or negative by RL N assay (Fig. 2, A). Based on seroconversion studies reported in the literature (18–20) and studies conducted by IVD vendors (12), most patients seroconvert 14 days following the infection and can be detected by serological tests directed against both N and S proteins. Thus, it is reasonable to conclude that, in the absence of PCR data, if the second-line test is still negative following a re-test at 14 days, the original first-line positive test result may be a false positive. In our institution, we report all discordant results with an interpretive comment stating that the antibodies against N protein were detected and antibodies against S protein were not detected and recommend repeat testing.

In conclusion, we were able to show that the orthogonal testing strategy is a useful tool that can be used to help identify potentially false positive SARS-CoV-2 IgG serology results, particularly in populations with low disease prevalence. While the advantage of this approach is clear, one of the caveats is that the false positive serological results still cannot be definitively characterized. In addition, this approach may be not feasible for many clinical laboratories, due to costs associated or lack of appropriate tests.

Author Contributions

All authors confirmed they have contributed to the intellectual content of this paper and have met the following 4 requirements: (a) significant contributions to the conception and design, acquisition of data, or analysis and interpretation of data; (b) drafting or revising the article for intellectual content; (c) final approval of the published article; and (d) agreement to be accountable for all aspects of the article thus ensuring that questions related to the accuracy or integrity of any part of the article are appropriately investigated and resolved.

S.R. Curry, provision of study material or patients.

Authors’ Disclosures or Potential Conflicts of Interest

Upon manuscript submission, all authors completed the author disclosure form. Disclosures and/or potential conflicts of interest:

Employment or Leadership

None declared.

Consultant or Advisory Role

None declared.

Stock Ownership

None declared.

Honoraria

F.S. Nolte, Abbott Molecular.

Research Funding

S.R. Curry, NIAID K23 AI25607.

Expert Testimony

None declared.

Patents

None declared.

Role of Sponsor

The funding organizations played no role in the design of study, choice of enrolled patients, review and interpretation of data, preparation of manuscript, or final approval of manuscript.