Analytical and clinical performances of five immunoassays for the detection of SARS-CoV-2 antibodies in comparison with neutralization activity

Abstract

Background

Reliable high-throughput serological assays for SARS-CoV-2 antibodies are urgently needed for the effective containment of the COVID-19 pandemic, as it is of crucial importance to understand the strength and duration of immunity after infection, and to make informed decisions concerning the activation or discontinuation of physical distancing restrictions.

Methods

In 184 serum samples from 130 COVID-19 patients and 54 SARS-CoV-2 negative subjects, the analytical and clinical performances of four commercially available chemiluminescent assays (Abbott SARS-Cov-2 IgG, Roche Elecsys anti-SARS-CoV-2, Ortho SARS-CoV-2 total and IgG) and one enzyme-linked immunosorbent assay (Diesse ENZY-WELL SARS-CoV-2 IgG) were evaluated and compared with the neutralization activity achieved using the plaque reduction neutralization test (PRNT).

Findings

Precision results ranged from 0.9% to 11.8% for all assays. Elecsys anti-SARS-CoV-2 demonstrated linearity of results at concentrations within the cut-off value. Overall, sensitivity ranged from 78.5 to 87.7%, and specificity, from 97.6 to 100%. On limiting the analysis to samples collected 12 days after onset of symptoms, the sensitivity of all assays increased, the highest value (95.2%) being obtained with VITRO Anti-SARS-CoV-2 Total and Architect SARS-CoV-2 IgG. The strongest PRNT₅₀ correlation with antibody levels was obtained with ENZY-Well SARS-CoV-2 IgG (R²adj = 0.569).

Interpretation

The results confirmed that all immunoassays had an excellent specificity, whereas sensitivity varied across immunoassays, depending strongly on the time interval between symptoms onset and sample collection. Further studies should be conducted to achieve a stronger correlation between antibody measurement and PRNT₅₀ in order to obtain useful information for providing a better management of COVID-19 patients, effective passive antibody therapy, and developing a vaccine against the SARS-CoV-2 virus.

Funding

None.

Research in context.

Evidence before this study

We searched Pubmed (NCBI) on July 13, 2020, for articles published with the keywords ("SARS-CoV-2" OR "COVID-19") AND ("antibody" OR "antibodies") AND ("neutralization" OR "neutralisation" OR "neutralizing" OR "PRNT") AND ("performance" OR "performances" OR "evaluation" OR "clinical" OR "comparison") in humans. We did not restrict our search to a publication language. This search retrieved a total of 61 papers, some of them dealing with the comparison of the clinical performances of anti-SARS-CoV-2 Ab immunoassays to neutralization titers. One report compares IgG or total antibodies (AbT) measurement of three enzyme-linked immunosorbent assay (ELISA), two CLIA chemiluminescent assays (CLIA) and one enzyme-linked immunosorbent assay (ELISA), in a total of 100 SARS-CoV-2 convalescent plasma donors and found a good correlation (rho > 0.700) between ELISA (Euroimmun IgG and Wantai Total antibodies) and neutralization titer. Another study evaluated the performances of six commercial immunoassays for the detection of IgG, IgA and IgM antibodies, including four CLIA automated assays [Abbott SARS-COV-2 IgG, Diasorin Liaison IgG and Euroimmun SARS-COV-2 IgG and IgA], and two RLF [Acro Biotech 2019-nCoV IgG/IgM and Xiamen Biotime Biotechnology SARS-COV-2 IgG/IgM] with a microneutralization test (MNT). Evaluating 70 included sera from PCR confirmed COVID-19 patients, a panel of 81 sera from negative subjects and patients with autoimmune disease and with respiratory virus, a total of forty-one out of 62 COVID-19 patients showed neutralizing antibodies. Another study evaluated two ELISA assays (Euroimmun SARS-CoV-2 IgG and Vircell COVID-19 ELISA IgG), one LFA (FaStep COVID-19 IgG/IgM Rapid Test Device) and two in-house developed assays. For all examined assays, the sensitivity ranged from 58.8 to 76.5% for the early phase of infection (days 5-9) and from 93.8 to 100% for the later period (days 10-18). Four automated immunoassays (Abbott Architect, Roche Cobas, LIAISON, VIRCLIA automation system) in comparison to two ELISA assays (Euroimmun SARS-CoV-2 IgG and Virotech SARS-CoV-2 IgG ELISA) an in-house developed plaque reduction neutralization test (PRNT) were tested with serum/plasma samples of followed up individuals with PCR-diagnosed COVID-19. A highest sensitivity of 93.3% was achieved, whilst the specificity of the examined assays was ≥ 97%.

Added value of this study

In this study, a series of four CLIA and one ELISA immunoassays were examined for clinical performances and for the correlation of anti-SARS-CoV-2 Ab serum levels with the gold standard method for determining PRNT. According to suggestions reported in some recent meta-analyses, this study evaluated immunoassays performances in different time frames, including the early Ab response. Furthermore, different groups of subjects were included to derive robust estimations (negative subjects included autoimmune patients and pregnant women, while SARS-CoV-2 positive included patients with asymptomatic/paucisymptomatic, moderate and severe diseases). Besides to evaluate the clinical performances of the examined immunoassays, this study was designed to collect evidence on the correlation between neutralization activity and anti-SARS-CoV-2 Ab levels.

Implications of all available evidences

Comparative data for immunoassays is needed as a basis for the production of convalescent plasma, for potential interpretations COVID-19 immunity and for planning national and international strategies to reduce viral spread. To these purposes, data collected from multiple studies are required to obtain an unbiased estimate of the current evidences, whilst careful study design is requested for pooling data. Taking all together, findings showed that commercial immunoassays performances are comparable and high positive or negative predictive values are achievable, especially from samples collected 12 days post symptoms onset. Furthermore, the poor non-linear response between neutralization activity and anti-SARS-CoV-2 Ab levels suggests that immunoassays can be mainly developed for detecting positive/negative subjects and for improving rRT-PCR diagnosis of COVID-19. These results can therefore be helpful for in vitro diagnostic device manufacturers, to develop more specific immunoassays.

Alt-text: Unlabelled box

1. Introduction

The continuing spread of coronavirus disease 2019 (COVID-19) caused by severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) has prompted concern worldwide, leading the World Health Organization (WHO) to declare COVID-19 a pandemic on 11 March 2020 [1]. The accurate and timely diagnosis of the disease is crucial to the effective management of patients, control of the pandemic and the establishment of appropriate infection control measures. Although real-time reverse transcription polymerase chain reaction (rRT-PCR) allows the diagnosis of the disease in most patients, including asymptomatic carriers, it has some analytical and clinical limitations. Analytical pitfalls in both the pre- and analytical steps have been described [2] and negative molecular test results have been reported in the later stages of infection, thus being misleading from a clinical viewpoint. Therefore, rRT-PCR precludes the identification of individuals who have been infected, but have had only minor, or no, symptoms and therefore have not sought medical attention. A wide range of immunoassays to detect SARS-CoV-2 antibodies (Ab) have been developed to complement rRT-PCR, with different antigen targets and formats [3], [4], [5]. Although not well suited for allowing an early diagnosis, serological assays for SARS-CoV-2 may play an important role in diagnosing COVID-19 disease in individuals who present late, in understanding the virus epidemiology in the general population, and in identifying the disease prevalence in categories at higher risk of infection (e.g. healthcare workers) [6]. In addition, they should be used to ascertain the efficacy of containment measures both locally and globally, to screen convalescent sera for therapeutic and prophylactic purposes, and to improve knowledge of the immune response to the novel virus as the degree and duration of the response of specific antibodies is as yet poorly understood [7,8]. Like infections from other pathogens, SARS-CoV-2 infection elicits development of IgM and IgG specific Ab which are the most available antibodies for assessing response, while little is known about IgA response in the blood.

The aim of this paper is to evaluate the performance characteristics and diagnostic specificity, sensitivity of four chemiluminescent assays (CLIA) and one enzyme-linked immunosorbent assay (ELISA) for SARS-CoV-2 antibodies and in comparison with neutralizing activity.

2. Methods

2.1. Patients

A total of 184 leftover serum samples from 130 COVID-19 patients (8 asymptomatic/paucisymptomatic who recovered at home with supportive care and isolation, and 122 hospitalized classified with moderate or severe disease, following WHO interim guidance [9]) and 54 SARS-CoV-2 negative subjects (33 healthcare workers, 21 autoimmune patients, 8 pregnant women) were included in the study (Table 1). All subjects underwent at least one nasopharyngeal swab test, analyzed by rRT-PCR. Healthcare workers were considered negative (NHW) on the basis of at least three negative sequential molecular test results obtained between February 26th and May 29th, 2020. Raw data of the study is available at 10.6084/m9.figshare.12928832.

Table 1.

Demographic characteristics of the 184 subjects included in the study.

Groups	N (%)	Gender		Age (mean±SD)
		Females N (%)	Males N (%)
Negative Healthy Workers (NHW)	33 (17.9%)	25 (75.7%)	8 (24.3%)	40.1±11.8
Autoimmune Patients and Pregnant women (AI/Pr)	21 (11.4%)	19 (90.5%)	2 (9.5%)	43.8±16.2
Asymptomatic / Paucisymptomatic SARS-CoV-2 positive Patients (Asympt/Pauci)	8 (4.4%)	7 (87.5 %)	1 (12.5%)	45.4±17.9
Moderate SARS-CoV-2 positive patients (Mod)	56 (30.4%)	23 (41.1%)	33 (58.9%)	60.6±17.4
Severe SARS-CoV-2 positive patients (Sev)	66 (35.9%)	13 (19.7%)	53 (80.3%)	67.9±15.3
Overall	184 (100%)	87 (47.3%)	97 (52.7%)	56.9±19.1

2.2. Analytical systems under evaluation

In this study, an evaluation was made of four commercially available chemiluminescent immunoassays (CLIA) [Anti-SARS-CoV-2 IgG (ref 6199919, insert: GEM1292, v4) and Total (6199922, insert: GEM1293, v1), Ortho Clinical Diagnostics, Raritan, NJ, USA; Elecsys Anti-SARS-CoV-2 (ref 09203095190, insert: 2020-06 v2), Roche Diagnostic GmbH, Mannheim, Germany; SARS-CoV-2 IgG (ref 6R86, insert: H07891R02 rev April 2020), Abbott Laboratories, IL, USA] and one enzyme-linked immunosorbent assay (ELISA) [ENZY-WELL SARS-CoV-2 IgG (ref 91400, insert IO09/440 v04-23-2020), Diesse Diagnostica Senese, Siena, Italy]. Table 2 provides a summary of the specific features of each immunoassay. Moreover, Liaison SARS-CoV-2 S1/S2 IgG (ref 311450, insert 200/007-797 v04-2020) (Diasorin, Sallugia-VC, Italy), ENZY-Well SARS-CoV-2 IgA (ref 91402, IO09/442 v04-24-2020) and IgM (ref 91401, insert IO09/441 v04-24-2020) were evaluated for the correlation with the neutralization results.

Table 2.

The five SARS-CoV-2 antibody assays investigated: characteristics specified by the manufacturers.

Manufacturer	Ortho Clinical Diagnostics		Roche Diagnostics	Abbott	DIESSE Diagnostics
Commercial name	Anti-SARS-CoV-2 IgG	SARS-CoV-2 Total	Elecsys Anti-SARS-CoV-2	SARS-CoV-2 IgG (also referred to as CoV-2 IgG)	ENZY-WELL SARS-CoV-2 IgG
Platform	VITROS ECi/ECiQ/3600 and VITROS 5600/XT 7600	VITROS ECi/ECiQ/3600 and VITROS 5600/XT 7600	Cobas e 411, Cobas e 601 and Cobas e 602	All Architect systems	96 wells microplate, automatable
Method	Chemiluminescent immunoassay (CLIA)	Chemiluminescent immunoassay (CLIA)	Electro-ChemiLuminescent (ECLIA)	Chemiluminescent Microparticle Immunoassay (CMIA)	Enzyme-linked immunosorbent assay (ELISA)
Detection	IgG Antibodies against SARS-CoV-2	Total Antibodies (Included IgG, IgA and IgM)	Antibodies (included IgG) against SARS-CoV-2	IgG antibodies against SARS-CoV-2	IgG antibodies against SARS-CoV-2
Antigen target	Spike Protein	Spike protein S1	Nucleocapsid protein	Nucleocapsid protein	Native antigen (Vero E6 cells infected with SARS-CoV-2)
Results	Signal/Cut-off (S/C)	Signal/Cut-off (S/C)	Signal Sample/Cut-off (COI)	Index (S/C)	OD/Cut-off (Index)
Interpretation	< 1.0 Negative ≥ 1.0 Positive	< 1.0 Negative ≥ 1.0 Positive	< 1.0 Negative ≥ 1.0 Positive	< 1.4 Negative ≥ 1.4 Positive	< 0.9 Negative 0.9 - 1.1 Equivocal ≥ 1.1 Positive
Sensitivity	12-15 d: 83.3% ≥ 8 d: 90.0%	0-8 d: 100.0% ≥ 8 d: 100.0%	0-6 d: 65.5% 7-13 d: 88.1% ≥ 14 d: 100%	< 3 d: 0.0% 3-7 d: 25.0% 8-13 d: 86.4% ≥ 14 d: 100%	92.5%
Specificity	100.0%	100.0%	99.80%	Pre-COVID-19: 99.6% Other Respiratory Illness: 100.0%	95.8%

d = Days.

2.3. Repeatability and intermediate precision evaluation

Precision estimation was performed on CLIA assays using two human serum sample pools with different values, by means of quintuplicate measurements of same pool aliquots, performed for a total of four consecutive days. Nested analysis of variance was used to estimate precision, following the CLSI EP15-A3 protocol [10]. The results for precision were compared to those claimed by the manufacturer when available, using the procedure recommended by EP15-A3. Repeatability and within-laboratory precision were in accordance with the repeatability and intermediate precision conditions specified in the international vocabulary of metrology (VIM, JCGM 100:2012) for precision estimation within a four-day period.

2.4. Linearity assessment

Linearity was assessed using serial dilution of two samples pools, prepared with two different levels of SARS-CoV-2 antibodies (high and low level pools), as specified in the CLSI EP06-A, guideline (paragraph 4.3.1) [11]. In brief, the following high-level serum pools were prepared: 5.2 signal to cut-off (S/CO) ratio for VITROS Anti-SARS-CoV-2 IgG, 53 S/CO ratio for VITROS Anti-SARS-CoV-2 total, 5.4 S/CO ratio for Elecsys Anti-SARS-CoV-2, and 3.76 S/CO ratio for Architect SARS-CoV-2 IgG. The pools were serially diluted with the corresponding low-level serum pools (0.174 S/CO ratio for Elecsys Anti-SARS-CoV-2, 0.01 S/CO for VITROS Anti-SARS-CoV-2 IgG, 0.2 S/CO for VITROS Anti-SARS-CoV-2 total, 0.04 S/CO ratio for Architect SARS-CoV-2 IgG). All measurements were performed in triplicate.

2.5. Plaque reduction neutralization test (PRNT)

For a subgroup of 68 samples from SARS-CoV-2 positive individuals, PRNT test was performed. These samples were randomly selected from the 130 SARS-CoV-2 positive individuals included in the study and independently from disease severity (Fisher's exact test, p = 0.103). Each sample was obtained from a different individual. On the same samples, the ENZY-WELL SARS-CoV-2 IgA and IgM (Diesse Diagnostica Senese, Siena, Italy) assays were further performed. The high-throughput method for PRNT was developed for the fast and accurate quantification of neutralizing antibodies in plasma samples collected from patients exposed to SARS-CoV-2. Samples were heat-inactivated by incubation at 56 °C for 30 min and 2-fold dilutions were prepared in Dulbecco modified Eagle medium (DMEM). The dilutions, mixed to a 1:1 ratio with a virus solution containing 20–25 focus-forming units (FFUs) of SARS-CoV-2 (self-obtained from patient isolate), were incubated for 1 h at 37  °C. Fifty microliters of the virus–serum mixtures were added to confluent monolayers of Vero E6 cells, in 96-wells plates and incubated for 1 h at 37 °C, in a 5% CO₂ incubator. The inoculum was removed and 100 µl of overlay solution of Minimum essential medium (MEM), 2% fetal bovine serum (FBS), penicillin (100 U/ml), streptomycin (100 U/ml) and 0.8% carboxy methyl cellulose was added to each well. After 26 h’ incubation, cells were fixed with a 4% paraformaldehyde (PFA) solution. Visualization of plaques was obtained with an immunocytochemical staining method using an anti-dsRNA monoclonal antibody (J2, 1:10,000; Scicons) for 1 hour, followed by 1 h incubation with peroxidase-labeled goat anti-mouse antibodies (1:1000; DAKO) and a 7 min incubation with the True Blue™ (KPL) peroxidase substrate. FFUs were counted after acquisition of pictures at a high resolution of 4800 × 9400 dpi, on a flatbed scanner. Biosafety Level 3 laboratory setting was used for PRNT tests. The serum neutralization titer was defined as the reciprocal of the highest dilution resulting in a reduction of the control plaque count >50% (PRNT₅₀). To define a seropositive threshold and the specificity of the PRNT test, we included in our analyses 43 negative control sera collected in 2018, before the COVID-19 pandemic. To infer the inter-operator reproducibility of the PRNT assay, a subset of 29 sera from the 68 obtained from SARS-CoV-2 positive subjects was tested three times, by three different couples of operators, 1 to 2 weeks apart. An intraclass correlation coefficient was inferred as previously described elsewhere [12]. Liaison SARS-CoV-2 S1/S2 IgG (Diasorin, Sallugia, VC, Italy) [5] assay was further performed for 52 out of these 68 samples tested neutralization activity.

2.6. Statistical analyses

For evaluation of precision, an in-house developed R (R Foundation for Statistical Computing, Vienna, Austria) script for implementing the CLSI EP15-A3 protocol was used for ANOVA and for calculating the upper verification limit [10]. The GraphPad Prism version 8.4.1 for Windows (GraphPad Software, LLC) was employed to evaluate plaque reduction neutralization test results, using non-parametric tests (Kruskall-Wallis test and Spearman's correlation). MedCalc Statistical Software version 19.2.1 (MedCalc Software Ltd, Ostend, Belgium) was used for power analyses. Stata v13.1 (Statacorp, Lakeway Drive, TX, USA) was used to evaluate the assays’ clinical performances, and for multivariate regressions. Bonferroni's adjusted p-value (B-adj) was calculated for multiple comparisons. For ROC analyses, a/the non-parametric empirical method was used to estimate the area under the ROC curve (AUC), while the ‘diagt’ module was used to estimate sensitivity, specificity, and positive and negative predictive values. Cohen's kappa was used to evaluate between methods agreements. Considering a type I error α = 0.05, a power of 0.8 and with 130 positive and 50 negative subjects, AUC above 0.95 can be considered significant with respect to values below 0.89 (null hypothesis).

2.7. Ethics statements

The study protocol (number 23307) was approved by the Ethics Committee of the University-Hospital of Padova. All the patients were informed of the study and voluntarily agreed to participate. All the patients who agreed to participate provided written consent.

2.8. Role of funding source

This study does not receive any specific grant from funding agency in the public, commercial, or not-for-profit sectors.

3. Results

3.1. Patients’ characteristics

Table 1 reports the demographic characteristics of the subjects included in the study. The overall mean age of subjects was 56.9 years, with a standard deviation (±SD) of 19.1 (range 22.7 - 92.2 years). Negative healthy workers (NHW) [Bonferroni's adjusted p-value (B-adj) p < 0.0001], autoimmune/pregnant subjects (AI/Pr) (B-adj p < 0.0001) and asymptomatic/paucisymptomatic (Asympt/Pauci) subjects (B-adj p < 0.0001) were significantly younger than SARS-CoV-2 patients. The percentage of females differed significantly from that of males (p < 0.001), particularly in the AI/Pr group. For SARS-CoV-2 patients, the mean time interval from the onset of symptoms and serological determinations was 24.6 days (SD ±18.6; range 4 - 89 days) (Table 3).

Table 3.

Disease severity, time from symptoms onset, percentage of positive samples to serological determination of SARS-CoV-2 antibodies and PRNT₅₀ titers, subdivided by the studied groups.

Groups	Samples evaluated for SARS-CoV-2 antibodies	Days from symptoms onset and serology (mean±SD)	Percentage of samples with positive assays results					Samples tested for neutralization activity	Percentage of samples with neutralizing antibodies (PRNT50 ≥ 1:10)
			A	B	C	D	E
Negative Healthy Workers (NHW)	33/184 (17.9%)	-	0%	4.7%	3.0%	3.0%	3.0%	-	-
Autoimmune Patients and Pregnant women (AI/Pr)	21/184 (11.4%)	-	0%	0%	0%	0%	0%	-	-
Asymptomatic / Paucisymptomatic SARS-CoV-2 positive Patients (Asympt/Pauci)	8/184 (4.4%)	54.6±22.5	100%	100%	100%	100%	100%	6/8 (0.75%)	6/6 (100%)
Moderate SARS-CoV-2 positive patients (Mod)	56/184 (30.4%)	19.5±13.7	76.8%	75.0%	69.6%	83.6%	73.2%	33/56 (58.93%)	29/33 (87.87%)
Severe SARS-CoV-2 positive patients (Sev)	66/184 (35.9%)	25.6±17.8	89.4%	78.8%	87.9%	89.4%	89.4%	29/66 (43.93%)	27/29 (93.10%)
Overall	184 (100%)	24.6±18.6	59.8%	59.9%	57.6%	62.3%	59.2%	68/184 (36.95%)	62/68 (91.18%)

A = SARS-CoV-2 IgG, Abbott Laboratories; B = Elecsys Anti-SARS-CoV-2, Roche Diagnostic GmbH; C = Anti-SARS-CoV-2 IgG Ortho Clinical Diagnostics; D = Anti-SARS-CoV-2 Total Ortho Clinical Diagnostics; E= ELISA ENZY-WELL SARS-CoV-2 IgG Diesse Diagnostica Senese.

3.2. Repeatability and intermediate precision

Results for precision of all the CLIA assays are reported in Table 4. The ANOVA approach allowed us to estimate repeatability and intermediate precision separately. Only the Architect SARS-CoV-2 IgG insert reported data on precision, claimed at levels of 0.04 and 3.53 S/CO ratio. For this immunoassay, intermediate precision performances statistically deviated from the manufacturer's claims at both levels. All the immunoassays had acceptable analytical imprecision (CV%).

Table 4.

Precision results for the studied immunoassays. Coefficient of variation (CV) are expressed in percentage (%) and were obtained by using pools of samples.

Measurand	Level	Design	Laboratory evaluation of repeatability, CV %	Laboratory Evaluation of Intermediate precision - CV%
Architect SARS-CoV-2 IgG#	0.05 (ratio S/CO)#	5 × 4	10.3*	11.8*
	3.81 (ratio S/CO)#		0.9	1.75*
VITROS Anti-SARS-CoV-2 IgG$	0.01 (ratio S/CO)	5 × 4	< 0.1	< 0.1
	5.35 (ratio S/CO)		4.35	4.35
VITROS Anti-SARS-CoV-2 Total^	0.03 (ratio S/CO)	5 × 4	18.30	30.51
	51.2 (ratio S/CO)		1.97	2.82
Elecsys Anti-SARS-CoV-2°	0.73 (ratio S/CO)	5 × 4	1.21	2.84
	2.57 (ratio S/CO)		0.87	2.63

^#obtained from the Abbott Architect insert CoV-2 IgG 6r86, H07891R02, B6R860 revised April 2020.

^$obtained from the VITROS Anti-SARS-CoV-2 IgG Ortho Clinical Diagnostic insert v4.0 pub. No. GEM1292_US_EN.

^{^}obtained from the VITROS Anti-SARS-CoV-2 Total Ortho Clinical Diagnostic insert v4.0 pub. No. GEM1293_XUS_IT.

^°obtained from the Cobas Anti-SARS-CoV-2 Roche Diagnostic insert 09203095500 v1.0, 2020-05 in Italian.

^⁎indicates that imprecision value was higher than that declared by manufacturers, also after the calculation of UVL as suggested by EP15-A3 (5.4% for Repeatability and 5.9% for Intermediate precision at level 0.04 Index S/C and 1.1 for Repeatability and 1.2% for Intermediate precision at level 3.53 S/CO).

3.3. Linearity assessment

Linearity results for all the CLIA studies are summarized in Fig. 1. All tested mixes of sample pools covered a wide range of values and included the manufacturers’ cut-offs. All immunoassays, except for Elecsys Anti-SARS-CoV-2, deviated from linearity, the coefficients of the second-order polynomial fit attaining high statistical significance.

Fig. 1.

Linearity assessment of the studies immunoassays.

3.4. Evaluation of clinical performances

Sensitivities, specificities, and positive and negative likelihood ratios were estimated using the manufacturers’ cut-offs, while receiver operating characteristic (ROC) curves were used to evaluate overall performance(s). Elecsys Anti-SARS-CoV-2 immunoassay results were available for 172/184 (93.4%) serum samples. Table 5 summarizes estimated clinical performances for all CLIA and the ELISA immunoassays considering the total time frame of 93 days and limiting the analyses to sera collected 12 days after the onset of symptoms. One hundred fifty-eight samples were included and evaluated in this restricted subgroup, while only 146 results were available for Elecsys Anti-SARS-CoV-2. Table 5 shows data on positive and negative likelihood ratios, allowing an easy estimation of positive (PPV) and negative (NPV) predictive values given disease prevalence. Considering two different scenarios of disease prevalence settings: (a) 4%, as found in a Veneto Region (Italy) survey [13]; (b) 10%, as described in a survey conducted in Geneva [14], PPV and NPV were then estimated, using VITROS Anti-SARS-CoV-2 Total and Architect SARS-CoV-2 IgG immunoassays for comparative purposes. Regarding performances calculated 12 days after the onset of symptoms, VITROS Anti-SARS-CoV-2 Total PPV (95%CI) and NPV (95%CI) were 66.3% (22.0–93.2%) and 99.5% (99.2–99.7%) with a prevalence of 4%, 84% (43.0%–97.3%) and 98.6% (97.8%–99.1%) with a prevalence of 10%. Within the total time frame, results for PPV (95%CI) and NPV (95%CI) changed to 68.2% (23.5–93.7%) and 99.8% (99.5–99.9%) with a prevalence of 4%, 85.1% (45.0–97.6%) and 99.5% (98.7–99.8%) with a prevalence of 10%.

Table 5.

Comparison of the clinical performances of all the studies immunoassays, overall and considering only the period ≥ 12 days post symptoms onset.

	Sensitivity (95% CI)		Specificity (95%CI)	Positive Likelihood ratio (95%CI)		Negative Likelihood ratio (95%CI)		ROC analyses (95%CI)
	Overall	≥ 12 days post symptom onset	Overall	Overall	≥ 12 days post symptom onset	Overall	≥ 12 days post symptom onset
VITROS Anti-SARS-CoV-2 IgG	80.8 (72.9-87.2)	93.3 (86.6-97.3)	98.1 (90.1-100.0)	43.6 (6.25-304.6)	50.4 (7.22-351.3)	0.20 (0.14-0.28)	0.07 (0.03-0.14)	94.7 (91.7-97.6)
VITROS Anti-SARS-CoV-2 Total	87.7 (80.6-92.7)	95.2 (89.1-98.4)	98.1 (90.1-100.0)	47.3 (6.8-330.1)	51.4 (7.4-358.5)	0.13 (0.08-0.20)	0.05 (0.02-0.12)	97.6 (95.6-99.6)
Elecsys Anti-SARS-CoV-2	78.5 (70.4-85.2)	89.4 (81.9-94.6)	97.6 (87.4-99.9)	32.9 (4.74-228.9)	37.6 (5.4-260.7)	0.22 (0.16-0.31)	0.11 (0.06-0.19)	96.6 (93.8-99.4)
Architect SARS-CoV-2 IgG	84.6 (77.2-90.3)	95.2 (89.1-98.4)	100.0 (93.4-100.0)	92.8 (5.9-1466.2)	104.2 (6.6-1646.2)	0.16 (0.11-0.24)	0.05 (0.02-0.12)	96.1 (93.3-98.9)
ENZY-WELL SARS-CoV-2 IgG	83.1 (98.1-75.6)	94.2 (87.9-97.9)	98.1 (90.1-100.0)	44.86 (6.43-313.2)	34.4 (7.09-166.84)	0.17 (0.12-0.25)	0.06 (0.03-0.13)	93.8 (90.5-97.2)

Architect SARS-CoV-2 IgG PPV (95%CI) and NPV (95%CI) were 100% (21.6–98.6%) and 99.8% (99.5–99.9%) with a prevalence of 4%, 100% (42.3–99.5%) 99.5% (98.7–99.7%) with a prevalence of 10%. On considering the total time frame, these results changed to 100% (19.7–98.4%) and 99.4% (99–99.6%), and 100.0% (39.5–99.4%) and 98.3% (97.5–98.8%) with prevalence settings of 4% and 10%.

3.5. Comparability of immunoassay results

Since the results of serum samples and the corresponding immunoassays’ cut-offs were used to derive either positive or negative test results, pairwise comparisons of all tests by Cohen's kappa and overall agreements (in percentages) were calculated considering overall data (Table 6).

Table 6.

Agreement and Cohen's kappa of the immunoassays under evaluation.

	Immunoassay agreement results
	VITROS Anti-SARS-CoV-2 IgG	VITROS Anti-SARS-CoV-2 Total	Elecsys Anti-SARS-CoV-2	Architect SARS-CoV-2 IgG
VITROS Anti-SARS-CoV-2 Total	Agreement = 94.5%; Cohen's kappa = 0.886 SE = 0.073	-	-	-
Elecsys Anti-SARS-CoV-2	Agreement = 91.2% Cohen's kappa = 0.817 SE = 0.076	Agreement = 92.4% Cohen's kappa = 0.836 SE = 0.075	-	-
Architect SARS-CoV-2 IgG	Agreement = 93.5% Cohen's kappa = 0.865 SE = 0.073	Agreement = 94.5% Cohen's kappa = 0.885 SE = 0.074	Agreement = 92.4% Cohen's kappa = 0.840 SE = 0.076	-
ENZY-WELL SARS-CoV-2 IgG	Agreement = 94.0% Cohen's kappa = 0.877 SE = 0.073	Agreement = 97.3% Cohen's kappa = 0.943 SE = 0.073	Agreement = 93.0% Cohen's kappa = 0.852 SE = 0.076	Agreement = 96.2% Cohen's kappa = 0.921 SE =0.073

SE = standard error.

3.6. Plaque reduction neutralization test (PRNT)

All the 43 negative control sera sampled in 2018 failed to neutralize virus (titers were < 1:10 for all samples) and, therefore, PRNT assay analytical sensitivity was estimated to be 100%. For this reason, we established that test sera recording titers equal to or above 1:10 should be considered as positive. Subsequently, among the 68 sera from COVID-19 patients, 62 resulted positive, recording values ranging from 1:10 to 1:5120. The signal-to-cut-off (S/CO) ratios of the examined assays, including Liaison SARS-CoV-2 S1/S2 IgG, ENZY-WELL SARS-CoV-2 IgA and IgM and the corresponding PRNT₅₀ titers for the 68 tested SARS-CoV-2 serum samples are shown in Fig. 2. The associations between PRNT₅₀ titers and assays S/CO ratios were evaluated by multivariate analyses, using SARS-CoV-2 antibody levels, age, gender, time from symptom onset and disease severity as additional predictors. At multivariate analyses, the highest association was obtained with ENZY-WELL SARS-CoV-2 IgG (R²adj = 0.569) followed by VITROS Anti-SARS-CoV-2 IgG (R²adj = 0.544), Architect SARS-CoV-2 IgG (R²adj = 0.477), ENZY-WELL SARS-CoV-2 IgM (R²adj = 0.406), Liaison SARS-CoV-2 S1/S2 IgG assay (R²adj = 0.402), and ENZY-WELL SARS-CoV-2 IgA (R²adj = 0.241) (Supplementary Tables 1-8). VITROS Anti-SARS-CoV-2 Total (R²adj = 0.117) and Elecsys Anti-SARS-CoV-2 (R²adj = 0.046) showed a very limited association with PRNT₅₀ titers. Inter-operator reproducibility of the PRNT assay, estimated using a subset of 29 sera from the 68 obtained from SARS-CoV-2 positive subjects, was supported by an intraclass correlation coefficient of 0.933.

Fig. 2.

Comparison of plaque reduction neutralization test (PRNT) and immunoassay results. A) Roche Elecsys Anti-SARS-CoV-2 Total against N antigen, B) Abbott Architect Anti-SARS-CoV-2 IgG against N antigen, C) Liaison Diasorin Anti-SARS-CoV-2 IgG Against S1/S2 protein, D) Ortho Clinical Diagnostics VITROS Anti-SARS-CoV-2 IgG against S protein, E) Ortho Clinical Diagnostics VITROS Anti-SARS-CoV-2 Total against S1 protein, F) Enzy-Well, Anti-SARS-CoV-2 IgG against native antigen; G) ENZY-WELL Anti-SARS-CoV-2 IgA against native antigen, H) ENZY-WELL Anti-SARS-CoV-2 IgM against native antigen. For all the comparisons n = 68 samples were evaluated (except for Liaison Diasorin Anti-SARS-CoV-2 IgG Against S1/S2 protein where n = 52). For Liaison Diasorin, reagents were available for 52 out of the 68 tested neutralization samples.

3.7. Impact on SARS-CoV-2 Ab levels and PRNT₅₀ titers of age, gender, time from symptoms onset and disease severity

The multivariate models, including Ab levels and Age, gender, time from symptoms onset and disease severity, demonstrate higher levels of SARS-CoV-2 Ab with increasing time from symptoms onset. No significant association was found between SARS-CoV-2 Ab levels and age, gender and disease onset (Supplementary Tables 9-13 and Supplementary Figures 1-2). Fig. 3 (left panel) shows the distributions of PRNT₅₀ titers subdivided by disease severity (Kruskal-Wallis test, χ² = 9.70, p = 0.0078) and gender (Kruskal-Wallis test, χ² = 1.11, p = 0.278). Although the number of samples is limited, asymptomatic/paucisymptomatic cases presented a PRNT₅₀ titer not significantly different from moderate/severe cases (Kruskal-Wallis test, χ² = 5.82, p = 0.054). The relationship between Ab neutralization activity and time post symptom onset (right panel) was not statistically significant at multivariate analyses (Supplementary Tables 1-8).

Fig. 3.

Plaque reduction neutralization test (PRNT) results, disease severity and time from symptoms onset (days). Asympt/Pauci = asymptomatic/paucisymptomatic SARS-CoV-2 patients; Moderate and Severe = SARS-CoV-2 positive patients without and with air ventilation support, respectively; M (within bars) = males; F (within bars) = females.

4. Discussion

In the last few months, numerous SARS-CoV-2 serology assays have been developed. The complexity of COVID-19 has called for careful study design to obtain meaningful information, and some of the assays have not yet been extensively validated by independent laboratories. In a recent meta-analysis, it was pointed out that most studies on SARS-CoV-2 serology have assessed sensitivity without considering time from onset of symptoms and/or including COVID-19-positive cases that are rRT-PCR-negative [15]. Moreover, researchers are currently facing other knowledge gaps in SARS-CoV-2 serology. For example, the understanding of the neutralization activity of serum antibodies against viral particles is incomplete, calling for the development of strategies to improve our understanding of their relationship with SARS-CoV-2 Ab detected by conventional immunoassays [16]. Serological tests should be evaluated in parallel to neutralization assays, since not only these represent the gold standard in terms of assay specificity and may provide evidence of the mechanism of development of viral immunity, but also they are the only assays measuring the actual protective immunity of antibodies [17].

In this retrospective study, the analytical and clinical performances of four commercially available CLIA assays and one ELISA assay (Table 2) have been evaluated and compared with neutralization activity using the plaque reduction neutralization test (PRNT). The neutralization activity was evaluated also with respect Liaison SARS-CoV-2 S1/S2 IgG and ENZY-WELL SARS-CoV-2 IgA and IgM. Before conducting the study, precision at two concentration levels and linearity were assessed for CLIA by using a standardized protocol according to the CLSI EP15-A3 and CLSI EP06-A (Table 4 and Fig. 1) [10,11] The results obtained demonstrated that both repeatability and intermediate precisions were comparable with other immunoassays performances for the highest concentration levels [3,5], whilst for the lowest levels, less satisfactory results were obtained for VITROS Anti-SARS-CoV-2 Total and Architect SARS-CoV-2 IgG. However, precision results do not have clinical impact as a small variations of the S/CO will not modify the interpretation of the serology.

Linearity was assessed in a range of values covering manufacturers’ cutoffs (Fig. 1). Linear results were obtained for Elecsys Anti-SARS-CoV-2, whereas the performance of other immunoassays was less effective, demonstrating that a double serum antibodies concentration will not correspond to a double in S/CO ratio. To assess the clinical performances of immunoassays, a total of 184 leftover samples obtained from Negative Healthy workers, Autoimmune patients/pregnant women, asymptomatic/paucisymptomatic and Moderate/Severe SARS-CoV-2 patients were evaluated (Table 1). The 21 autoimmune patients and the eight pregnant women, who were SARS-CoV-2 negative, were included in order to evaluate possible analytical interferences. According to the suggestion on study design for SARS-CoV-2 serology, clinical performances were evaluated by considering the total time frame (overall data), and limiting the analyses to sera collected 12 days after the onset of symptoms as this period greatly impacts on immunoassay sensitivity [5].

Table 3 shows the distribution of sera among patient groups, the delay since symptom onset, the number and the percentages of samples tested positive for the SARS-CoV-2 antibodies assays and the PRNT₅₀ titer. Taking into consideration the time from symptoms onset and sera collection, significant differences were found between Asympt/Pauc and Mod (p < 0.01) and Asympt/Pauc and Sev (p < 0.01). These differences might be at least in part due to the difficulty to obtain serum samples from Asympt patients because they were not hospitalized, but in quarantine at home. Table 3 shows that the percentages of samples with SARS-CoV-2 Ab positive results increased from Moderate to Severe disease. However, multivariate linear regression analyses confirmed that, among the studies variables (age, gender, disease severity and time from symptoms onset), only time from symptoms onset was associated, with an increasing trend, with SARS-CoV-2 Ab serum levels (Supplementary Tables 9-13, Supplementary Figures 1-2).

Results of ROC analyses showed overlapping performances for all immunoassays (Table 5). VITROS Anti-SARS-CoV-2 Total provided the best results, with an AUC over 97% (lower confidence limit > 95%). Considering overall data, Elecsys Anti-SARS-CoV-2 and Architect SARS-CoV-2 IgG sensitivities obtained in our study are similar those reported by Kohmer et al. and Theel et al. [18,19]. Sensitivities differed when only samples collected after 12 days post symptom onset were evaluated. This was expected since, after SARS-CoV-2 infection, antibody levels begin to rise as from the second week of onset of symptoms [20]. Evaluations after 12 days post symptom onset confirmed overall excellent results for all immunoassays, anti-SARS-CoV-2 Total and Architect SARS-CoV-2 IgG performances being the best. However, in contrast with statements in manufacturers’ inserts, our data show that 100% true positive results cannot be obtained, even when the sample collection time is more than 14 days after onset of symptoms. These results are in agreement with findings reported by other Authors on comparing immunoassays results [17,[21], [22], [23]]. Asymptomatic/paucisymptomatic SARS-CoV-2 patients were correctly identified as positive by all immunoassays. These findings are promising, even if the number of evaluated patients is limited to draw robust conclusions. In fact, recent studies have shown that individuals with asymptomatic infections have lower antibody titers and a more rapid decline of those antibody titers, notably neutralizing antibodies [7,24]. Differently, 95% CI of specificities reached 100% for all the assays, and these data are in line with manufacturers’ inserts. Although excellent positive- (PLR) and negative- (NLR) likelihood ratios were obtained for all methods, Architect SARS-CoV-2 IgG PLR outperform all other immunoassays. PPV and NPV were further estimated considering two prevalence settings (4% and 10%). The highest achievable PPV and NPV values were 100% and 99.8% for PPV and NPV, respectively, obtained with Architect SARS-CoV-2 IgG and at a prevalence setting of 10%. These results are partially in agreement with that reported by Pflüger, et al. in settings with a low prevalence [23].

Overall, there was a good agreement between all assays, the better result being obtained for VITROS Anti-SARS-CoV-2 Total and ENZY-WELL SARS-CoV-2 IgG (Cohen's kappa = 0.943). Interestingly, the two methods differ both for the detection specificity (Total Ab vs IgG, respectively) and for viral protein targets (Spike protein S1 vs Native antigen).

To provide insights on neutralization activity compared with immunoassays results, PRNT assay was performed on 68 samples from SARS-CoV-2 positive subjects. With the exception of Elecsys Anti-SARS-CoV-2, other immunoassay results were correlated with PRNT₅₀ titer. These results agree with findings from Tang et al., who found a weak linear correlation between Elecsys Anti-SARS-CoV-2 Ab Tot levels and PRNT₅₀ titers [25].

However, as shown in Fig. 2, the dynamic range of measured antibodies, from the lowest to the highest PRNT₅₀ titer, is very limited, this being in line with the data reported by Jääskeläinen and colleagues [17]. This might be due to several factors, such as: a) antibodies measurements may be specifically developed for detecting positive/negative subjects in order to improve rRT-PCR diagnosis of COVID-19, or b) a non-linear response between the real antibody concentration and instrumental signals. Notably, results of Liaison SARS-CoV-2 S1/S2 IgG assay were similar to those obtained with other immunoassays. At correlation analysis, no differences could be detected on comparing immunoassays developed against Spike or Nucleocapsid proteins. Differently, at multivariate analysis the strongest correlation with PRNT₅₀ titer was found for ENZY-WELL ELISA results, developed with a native antigen of SARS-CoV-2 (R²adj = 0.569, which correspond to a Pearson's rho of 0.753). Recently, Perera et al. found a slightly worse correlation (rho = 0.67) between plaque reduction neutralization results and an in-house developed IgG ELISA with recombinant RBD of the spike protein as coated antigen [26]. In addition, we found a moderate correlation between PRNT and IgM results, thus confirming the data reported by Perera and colleagues [26]. Another report, which compares IgG or total antibodies measurement of three ELISA, two CLIA and two lateral flow tests, in a total of 100 SARS-CoV-2 convalescent plasma donors, found a good correlation (rho > 0.700) between ELISA (Euroimmun IgG and Wantai Total antibodies) and neutralization titer [27].

PRNT₅₀ titer were lower for Asympt/Pauc than Mod or Sev SARS-CoV-2 patients (Fig. 3, left panel), although this difference was not statistically significant. Furthermore, no significant correlation was not found between the PRNT₅₀ titer and the time interval from symptom onset (Fig. 3, right panel), whist a decreasing trend can be observed and could be verified in a more representative sample size.

The present paper has limitations: first, neutralizing antibodies were tested in a limited number of samples and sera were collected at various timepoints and, therefore, should be confirmed in further studies; second, COVID-19 positive patients were selected retrospectively on the basis of available leftover samples, and third cross-reactivity was not assessed with confirmed infection with other virus, namely other coronavirus; therefore NPV and PPV could be overestimated. Another limitation of this study is that no longitudinal sera were analyzed and, therefore, we cannot exclude that some patients might have seroconverted at later time points.

In conclusion, although the performances of SARS-CoV-2 antibody immunoassays are of analytical and clinical value, they could be enhanced by considering the test purposes, emphasizing sensitivity in the screening and specificity in the second-line testing. In addition, a further search should be made for a better dynamic range and a stronger correlation with respect to antibody neutralization activity, in order to, above all, obtain information needed for a better patient management, effective passive antibody therapy and vaccine development against SARS-CoV-2 virus.

Contributors

Study design: MP; AP.

Sample collection and experimental set-up: DN; SZ; CC; FB; MPag; AB.

Data Collection and statistical analyses: DN; DB; AP.

Writing of the manuscript: AP; LS; MP.

Review of the manuscript: FB; AP; LS; MP.

All Authors read and approved the final version of the manuscript.