Assessment of automated high-throughput serological assays for prediction of high-titer SARS-CoV-2 neutralizing antibody

Abstract

COVID19 convalescent patient plasma units with high titer neutralizing antibody can be used to treat patients with severe disease. Therefore, in order to select suitable donors, neutralizing antibody titer against SARS CoV-2 needs to be determined. Because the neutralization assay is highly demanding from several points of view, a pre-selection of sera would be desirable to minimize the number of sera to be tested. In this study, a total of 140 serum samples that had been titrated for SARS-CoV-2 neutralizing antibody by microneutralization assay were also tested for the presence of anti-SARS-CoV2 antibody using 5 different tests: Architect® immunoassay (Abbott Diagnostics), detecting IgG against the nucleocapsid protein, LIAISON XL® (Diasorin) detecting IgG against a recombinant form of the S1/S2 subunits of the spike protein, VITROS® (Ortho Clinical Diagnostics), detecting IgG against a recombinant form of the spike protein, and ELISA (Euroimmun AG), detecting IgA or IgG against a recombinant form of the S1 subunit. To determine which immunoassay had the highest chance to detect sera with neutralizing antibodies above a certain threshold, we compared the results obtained from the five immunoassays with the titers obtained by microneutralization assay by linear regression analysis and by using receiver operating characteristic curve and Youden's index. Our results indicate that the most suitable method to predict sera with high Nab titer is Euroimmun® IgG, followed closely by Ortho VITROS® Anti-SARS-CoV-2 IgG.

Since the beginning of the SARS-CoV-2 pandemic, COVID-19 can be seen as an unusual infection. Although it tends to be more serious in certain categories of patients, there is little correlation between clinical course of infection and antibody levels reached after convalescence [1–4]. Indeed, longitudinal surveys have demonstrated that antibody responses mount earlier in severe COVID patients. However, antibody final titers vary greatly in different patients independently of the clinical course of infection and about 5% patients have undetectable antibody titers despite documented infection [5,6]. In addition, in patients that do have antibodies, titers tend to decline with time to the point that some of them become seronegative [7].

Passive antibody therapy is one of the oldest interventions to control infection and is effective against a vast number of pathogens [8]. Effectiveness of this approach relies on the presence of neutralizing antibodies (NAbs) that prevent viral access into susceptible cells by binding specific domains of the viral entry machinery. Indeed, at the top of the SARS CoV2 attachment and fusion protein, the spike (S) protein, there is an N-terminal domain and the SARS CoV2 receptor-binding domain RBD, the part of S responsible for binding the cellular receptor angiotensin-converting enzyme 2 (ACE2); recently, epitope mapping analysis has shown that these regions contain neutralizing epitopes [9].

Passive antibody therapy has shown promising results promising against other betacoronavirus infections, and there is good possibility that COVID 19 convalescent patient plasma (CCP) is also effective against SARS-CoV-2 [[10], [11], [12]]. In keeping with this hypothesis, infusion of COVID-19 convalescent plasma (CCP) into patients with non-critical disease was found to be beneficial [10, [13], [14], [15], [16], [17], [18]]. European Medicine Agency have only approved remdesivir as a specific antiviral drug, but is associated with a high cost (ema.europa.eu, doi.10.1001/jama.2020.16337). Therefore, clinical trials have been undertaken around the world to evaluate the efficacy of CCP therapy; CCP may ameliorate clinical symptoms when given up to 15–30 days after onset, although many issues including safety remain to be clarified, warranting further studies [[19], [20], [21], [22]].

To date, although no definitive scientific evidence supports the adoption of a defined titer of NAbs in donated CCP, the presence of high levels of anti‐SARS‐CoV‐2 NAbs is recommended. This explains why one of the main criteria to qualify for CCP donation in many studies is the presence of NAbs above a certain threshold [23]. To our knowledge, titration of NAbs can only be achieved by the microneutralization test (NT), a biological assay which is labor-intensive, time-consuming, and biohazardous. On the other hand, CCP therapy for hospitalized COVID-19 patients requires screening of thousands of plasma donors. For these reasons, it would be recommended to find simpler, faster and automatable tests allowing screening of large numbers of plasma samples to preselect sera for NT.

In an effort to find an automated serological test that correlates with NT, we have used five European Community-approved diagnostic automated immunoassays that are claimed by manufacturers to quantify, as arbitrary units (AU/ml) or as a signal/cutoff ratio, anti-SARS-CoV-2 antibodies (Abs) and compared these results with NT titers. We found good correlations between the results obtained by NT and 2 of the immunoassays tested, indicating that, under certain conditions, immunoassays can be used to identify donors likely to have high NAb titers.

1. Materials and methods

This study was carried out with patients adhering to clinical trial “TranSfUsion of coNvalescent plAsma for the early treatment of pneuMonIa due to SARS-CoV2 (TSUNAMI Study): an open label randomized trial” protocol (NCT04393727). TSUNAMI was initially approved by the Ethics Committee of the University Hospital of Pisa and then, to enroll patients from across Italy, further approved by the Agenzia Italiana del Farmaco. All donors and patients who agreed to participate provided written informed consent and, in accordance with the national transfusion laws.

1.1. CCP and criteria used for enrollment

To evaluate the performance of the five immunoassays described below, randomly selected sera were chosen from a cohort of 1200 serum samples obtained from subjects who resolved SARS-CoV-2 infection and voluntarily opted to donate CCP. All patients had a history of mild to severe COVID-19 confirmed by one or more positive SARS-CoV-2-specific molecular assays performed on nasopharyngeal swabs or other respiratory material. Recovery from disease was ascertained by two consecutive negative swabs performed at least 48 h apart and collected two weeks after complete clinical recovery. All patients were living in the Tuscany and Liguria regions and tested negative for hepatitis A and E RNA, and parvovirus B19 DNA, as well as for hepatitis B and C viruses, human immunodeficiency virus and syphilis by molecular tests. Samples were collected between April and September 2020 and titrated for NAbs at the Virology Unit, University of Pisa Hospital. The selective criterion to accept and use CCP for the TSUNAMI study is a NT titer of ≥160; CCP is used for the early treatment of pneumonia in hospitalized COVID patients under a randomized controlled trial. Because only plasma donations positive for anti SARS-CoV-2 antibodies were examined, to minimize the collection of donations, convalescent patients were screened using a diagnostic assay. If positive, donors would be called back for apheresis (plasma donation) as soon as possible, since SARS-CoV-2 NAb titers are thought to decline with time [7,[30], [31], [32], [33], [34]]. At the time of apheresis, a sample of the patient's plasma underwent NT, to determine the NAb titer in the donated plasma. All individuals were free from SARS-CoV-2 infection as determined by two consecutive nasal swabs examined at least two weeks prior to plasma donation.

Evaluation of immunoassays was performed with 140 serum samples from anonymized donors aged 18–70 years (median age 46 years) and 67% male.

1.2. NT assay

NT was carried out using a limiting dilution assay method established by NeuCoV-NET, an Italian network established by selected laboratories [24]. All CCP used under the TSUNAMI study were certified by laboratories belonging to NeuCoV-NET, who generated a set of proficiency sera sent periodically to all laboratories and used to gauge NT assay performances and calculate inter-laboratory variability.

The NT assay was performed with the SARS-CoV-2 strain SARS-CoV-2/Human/ITA/PAVIA10734/2020. This strain was isolated from a 74 years-old symptomatic male patient isolated in February 2020 that has been fully sequenced (sequence available on GISAID, accession no. EPI_ISL_568579, and GeneBank, accession no. M527178.1). It is of clade G and bears D614G mutation in the S protein [25]. SARS-CoV-2/Human/ITA/PAVIA10734/2020 strain was provided at the fourth passage in Vero E6 cells together with Vero E6 by NeuCoV-NET. To prevent cell-culture adapted mutant emergence, the NT assay was performed with viral preparations obtained from the original batch passaged up to three times in Vero E6 cells cultivated in D-MEM supplemented with 10% heat-inactivated fetal bovine serum (FBS), L-glutamine, penicillin and streptomycin. Viral preparations were titrated in Vero E6, checked for the absence of Mycoplasma [26], aliquoted at 1000 tissue culture infectious dose 50% (TCID₅₀)/ml, and stored at −80 °C until use. To titrate serum NAbs, Vero E6 cells (12.000/well) were plated the day before on 96-well plate in MEM with 10% FBS. Serum samples, inactivated at 56 °C for 30 min, were diluted 4-fold 6 times in duplicates from 1:10 to 1: 640 in D-MEM, 10% FBS. One hundred TCID₅₀ SARS-CoV-2 were added to the serum dilutions and incubated for 1 h at 37 °C. The virus-serum mixture was then added to the cells and incubated at 37 °C with 5% CO₂ until the cytopathic effect (CPE) became evident. Cells were then fixed with 1% paraformaldehyde and stained by Gram crystal violet to visualize the CPE by optical microscopy. NT titer was expressed as the last serum dilution that inhibited SARS-CoV-2 CPE by 90%.

1.3. Immunoassays

Serum samples were analyzed using the following five commercial serological assays: LIAISON® XL SARS-CoV-2 S1/S2 IgG CLIA kit S1/S2 based antigen (DiaSorin, Saluggia, Italy) (DIA); Abbott Architect Plus® i2000sr Analyzer, SARS-COV-2 IgG CMIA kit (Abbott Italia, Rome, Italy) (ABB); EURO Labworkstation, SARS-CoV-2 IgG (EUG) and IgA (EUA) ELISA kits® (Euroimmun, Diagnostica Medica, Italia); Ortho Clinical Diagnostics, VITROS® Anti-SARS-COV-2 IgG® (Ortho Clinical Diagnostics, Milan, Italy) (ORT). The antigen target and assay methods are described in Table 1 . Samples were tested unblinded by all assays by senior biomedical technologists within one week from collection. All testing was performed according to manufacturers’ instruction and results were given following the interpretation criteria supplied by the manufacturer. Undetermined results were interpreted as negative for the purpose of the study.

Table 1.

Characteristics of the SARS-CoV-2 antibody immunoassays.

Manufacturer	Abbreviation	Assay name	Isotype	Method	Antigen	Units
Abbott	ABB	CoV2 IgG	IgG	CMIAa	Nucleocapsid	Index (S/C)b
Euroimmun	EUA	Anti SARS CoV-2 ELISA IgA	IgA	ELISAc	Recombinant S1	Ratio (S/C)
Euroimmun	EUG	Anti SARS CoV-2 ELISA IgG	IgG	ELISA	Recombinant S1	Ratio (S/C)
DiaSorin	DIA	LIAISON SARS-CoV-2 IgG	IgG	CLIAd	Recombinant S1 and S2	AU/mle
Ortho Clinical Diagnostics	ORT	VITROS Anti-SARS-CoV-2 IgG	IgG	CLIA	Recombinant S	Ratio (S/C)

^aChemiluminescent microparticle immunoassay.

^bS/C: signal/cutoff.

^cEnzyme-linked immunosorbent assay.

^dChemiluminescent immunoassay.

^eArbitrary Units/ml.

1.4. Statistical analysis

Univariate linear regression was assessed using Pearson's correlation coefficient. Sensitivity and specificity of each test when used to predict NT titers >80 by receiver operating characteristic (ROC) curve analysis was obtained by calculating the area under the curve (AUC), with corresponding 95% Confidence Interval (CI). The AUC allows to determine the percentage of the results that turn positive by a given test for a given threshold. To minimize inter-assay variations, ROC analysis is shown and discussed for the data obtained with 140 sera that were examined with all five immunoassays and NT.

Youden's index, a common summary measure of the ROC curves, was used to identify the best threshold to discriminate plasma samples with NT titers <80 and >80, then we calculated the corresponding sensitivity and specificity, as shown in Table 2 [27]. Specificity and sensitivity were also set at 0.95 to identify the best threshold values (Table 2). Statistics and graphics were performed using R software, version 4.0.3, (https://www.R-project.org/) and Illustrator.

Table 2.

Cut-off values of the serological assays in detecting sera above cut-off NT titers ≥ 80.

	Immunoassay
	Abbott (ABB)	Euroimmun IgA (EUA)	Euroimmun IgG (EUG)	Diasorin (DIA)	Ortho Clinical Diagnostics (ORT)
(1) Youden criteria
Cut-off value	4.78	2.8	7.3	90	16.2
Sensitivity	0.82 (0.63, 0.94)	0.77 (0.63, 0.87)	0.79 (0.65, 0.89)	0.53 (0.39, 0.67)	0.78 (0.63, 0.89)
Specificity	0.62 (0.47, 0.76)	0.67 (0.55, 0.77)	0.88 (0.79, 0.95)	0.92 (0.84, 0.97)	0.81 (0.71, 0.89)
(2) Sensitivity ≥ 0.95
Cut-off point	3.24	1	2.6	12	10.5
Sensitivity	0.96 (0.82, 1.00)	0.96 (0.87, 1.00)	0.98 (0.90, 1.00)	0.98 (0.90, 1.00)	0.96 (0.85, 0.99)
Specificity	0.44 (0.40, 0.60)	0.22 (0.13, 0.33)	0.32 (0.22, 0.44)	0.29 (0.20, 0.39)	0.55 (0.43, 0.66)
(3) Specificity ≥ 0.95
Cut-off point	8.21	9.1	8.5	105	20.08
Sensitivity	0.29 (0.13, 0.49)	0.15 (0.07, 0.28)	0.65 (0.51, 0.78)	0.43 (0.30, 0.58)	0.44 (0.30, 0.60)
Specificity	0.96 (0.85, 0.99)	0.96 (0.89, 0.99)	0.96 (0.89, 0.99)	0.95 (0.89, 0.99)	0.97 (0.91, 1.00)

Cut-off values of the serological assays either as obtained following Youden's criteria (1) or by setting sensitivity (2) or specificity (3) ≥0.95. CI for specificity and sensitivities were 95%.

2. Results

2.1. Experimental procedure

The features of the diagnostic immunoassays used in the present work are outlined in Table 1. The 5 tests in question may be evaluated for their ability to answer three different questions: (1) identifying patients with Covid-19 from patients with no infection. This ability was assessed by the manufacturers and specificity and sensitivity are indicated in each assay instruction sheet. In addition, the immunoassays selected have already been extensively evaluated for their clinical sensitivity and specificity by others [28,29]. For these reasons, we did not evaluate this ability in the present work. (2) Evaluating the association between NT titers with anti SARS-CoV-2 binding Ab titers. (3) Selecting samples eligible for donation from those that were not amongst NT-positive sera. The last 2 properties are addressed in the following paragraphs and results are shown in Figs. 1 and 2 and summarized in Table 2. The NT titer of ≥80 as a cut-off for NT results was selected because it is just one dilution step below 160, and it would increase the possibility of identifying all suitable donors.

Fig. 1.

Scatter plots of NT titers of test samples with values from each of the 5 assays evaluated. A line of best fit (blue) was estimated by univariate linear regression using Pearson's correlation coefficient R. The green line represents the NT cut-off value (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.).

Fig. 2.

ROC analyses of the fine immunological assays to identify anti-SARS-CoV-2 NAb titres ≥ 80. The dot on each graph indicates the pairs of sensitivity and specificity values corresponding to the best threshold, as identified by Youden's index; the gray bar denotes the non-informative curve.

2.2. Assessing correlation between binding and NT Ab titers

We first explored the association between the values of antibody titers found by the 5 different tests and NT titers by univariate regression analysis (Fig. 1). We examined nearly 200 samples from individuals who suffered from COVID-19 at various degrees of severity with NT and one or more immunoassays. The results shown in Fig. 1 revealed that, in order, DIA, EUG and ORT were the best test in terms of correlation with the NT results (R² = 0.34, 0.32 and 0.24, respectively). Values observed for EUA and ABB were poorly correlated with NT result and therefore less able to predict neutralization activity.

To confirm the correlation between the values obtained by each immunoassay and NT, ROC curves were generated by plotting the number of false positive fraction of samples using the given immunoassay, on the x axis, and the true positives on the y axis. NT was the reference assay, scoring values of NT titer <80 and ≥ 80 negative and positive, respectively. This analysis allows to determine the association between clinical sensitivity (defined as the fraction of sera that turned out as true positives (>80) by the test analyzed to all sera with NT titers > 80) and specificity (fraction of true negatives (<80) by the test analyzed to all sera with NT titers < 80) for every possible cut-off for each test. The trade-off between clinical sensitivity and specificity for every possible cut-off for ABB, EUA, EUG, DIA, and ORT assays in identifying NT titer of 80 or greater is illustrated in Fig. 2 [27].

The ROC curves obtained are shown in Fig. 2. A minimum value of 0.95 for sensitivity and specificity and ≥80 NT titer as a threshold were arbitrarily selected to compare assays. To this aim, the AUC under each ROC curve was calculated and are shown for each curve in Fig 2. The AUC allows to determine the percentage of results that turn positive by a given immunoassay at a given NT titer threshold. As can be seen, the two assays ABB and EUA were identified as the worst classifiers, with AUC values of 0.74 and 0.75, respectively. This means that the chance of finding a NT titer ≥80 by ABB and EUA was too close to the non-informative value of 0.50, that corresponds to complete lack of correlation between the values of NT and those of the immune assays tested. In this case, the most suitable tests under these parameters were, in order, EUG, ORT and DIA, with an AUC of 0.89, 0.86, 0.77, respectively.

2.3. Finding the best assay to detect CCP for donations

With the idea of providing an automated method to identify CCP potentially exploitable for the TSUNAMI study, we derived Youden's index from the ROC curves. Youden's index, a convenient value summarizing the performance of each assay, is a value varying from 0, when the assay yields equal positive results for sera with and without Nabs, to 1, when no false positives or false negatives are detected. Youden's criteria indicated DIA as the best test in terms of maximum specificity of 0.92 whereas maximum balance between specificity and sensitivity was, again, EUG: at a cut-off of 7.3, sensitivity was 0.79 and specificity was 0.88.

Moreover, as shown in Table 2, when specificity is fixed at 0.95, EUG again proved to be the best test, with a cut-off of 8.5 (S/C) and a sensitivity of 0.65. Instead, when sensitivity was fixed at 0.95, the best test was ORT, at a cut-off of 10.5 (S/C) and a specificity of 0.55.

3. Discussion

The development of NAbs among patients with COVID 19 exhibits substantial variability after convalescence, in that some patients develop higher antibody responses compared to others [6]. CCP is being tested to treat patients with severe COVID 19, provided that the NAb titer is ≥ 160. There have been contradictory conclusions in terms of clinical benefits offered by CCP therapy, with some early studies showing benefits [10,13,14] while others did not [35]. Because effective treatment options against COVID19 are still few, further investigation on this therapeutic option are warranted.

Because the NT is particularly laborious, a prescreening automated diagnostic method allowing the detection of therapeutic plasma that does not miss high-NT titer samples is urgently needed. Such a reliable immunoassay would allow exclusion of negative samples before the NT test, thereby substantially reducing overall labor and healthcare costs. Furthermore, given that NAbs decline with time, the preselection of sera by an automated assay would facilitate prompt enrolment of convalescent donors [7,25–29].

In this study, we tested five widely used diagnostic assays to assess which would be the most predictive of a NT titer ≥80, in order to obtain therapeutic CCP donations to treat COVID-19 patients. Except for ABB, the remaining four tests target the viral receptor S protein as a whole or as one or both subunits. ABB uses the nucleocapsid protein as its target antigen, Abs against which appear first and in higher amounts compared to S protein [2,3]. This test is used to identify SARS-CoV-2 seropositive donors by the Italian blood bank service. It was not surprising, therefore, that ABB did not correlate well with NT titers and with a very low specificity. Poor performances were also observed with EUA, which detects IgA anti-S1 protein, and showed very low specificity in detecting sera with NT titers >80. Although IgA exhibit neutralizing activity and are known to play a protective role against respiratory viruses at a mucosal level [36,37], it is likely that serum IgA levels are very low and do not significantly contribute to serum neutralizing activity.

DIA, EUG and ORT performed satisfactorily, as they showed sufficient sensitivity and specificity. In agreement with a previous study [38], the assay which correlated best with NT and showed a good balance between sensitivity and specificity was EUG.

As for the TSUNAMI study, when the demand for CCP is limited, it is important to minimize false positives in the pre-screening procedure, even if this implies performing more NT assays. In this scenario, we suggest to fix specificity at 0.95 and, consequently, ORT is considered to be most appropriate. Conversely, in the setting of high demand for CCP (e.g., if CCP becomes an established therapy) elevated sensitivity (0.95) is preferable to minimize donor loss. In these conditions, EUG and ORT would be considered the best assays.

All samples were thawed, analyzed in parallel and in a narrow time interval to minimize inter-assay variability or inconsistent readings due to sample degradation. However, they were collected at different times from recovery, therefore our data cannot be useful in determining the percentage of recovered patients with neutralizing Abs because Nabs tend to decrease rapidly with time from recovery [32].

From these results, we have determined the performance of the assays by maximizing detection of positive samples, e.g., by setting the sensitivity to 0.95 that allows the detection of 95% positive samples, or by detecting only true positive samples, e.g., by setting specificity to 0.95 to reduce spurious positive samples to 5%. By using these conditions, the best assay was ORT followed by ABB. Conversely, by setting specificity to 0.95, the best assay was EUG followed by ORT (Table 2). In addition, this analysis demonstrated that Youden's criteria privileged either sensitivity (i.e., higher ratio of false positive results, ABB and EUG) or specificity (i.e., higher ratio of false negative results, DIA, EUG, and ORT) (Fig. 2, Table 2).

A possible limitation of the study presented is the limited number of samples taken into consideration. However, the results are consistent with those of Franchini et al., who also found EUG suitable to detect high-NT-titer sera [23]. In addition, they confirm those of Padoan et al., who also found ORT as the most reliable immunoassay to detect high NT titers in sera [28,38] (Fig. 3).

Fig. 3.

Schematic of cut-off values calculated from the ROC analyses and determined by Youden's index, and 95% sensitivity and specificity. Sensitivity set at 95% allows the detection of 95% positive samples, the specificity set at 95% limits detection of negative samples to 5%.

Our data were obtained with an Italian isolate of SARS CoV2 used in the NT and sera from patients infected before any viral variant was detected, therefore it is difficult to know whether our results can be extended to the relatively new SARS CoV2 variants B.1.1.7, identified in the UK, and B.1.351 in South Africa [39]. Indeed, both variants have extensive mutations in the S protein and were shown to be at least partially resistant to therapy with plasma from individuals recovering from infection with classical strains. Interesting results performed on pseudotyped viral vectors exposing SARS-CoV-2 S B.1.1.7 variant, found in UK, demonstrate that plasma from patients vaccinated with the BNT162B2 mRNA vaccine retained considerable neutralizing activity against this variant, suggesting that CCP can still be therapeutic even in patients infected with this variant, at least [40]. Therefore, our results might useful for post-vaccination measurement of protective immunity, since the vaccines available at present employ S protein sequences from viral strains isolated in 2019.

To conclude, we believe that this study provides clinicians with important information to aid in the selection of donors for CCP therapy.

CRediT authorship contribution statement

Giovanna Moscato: Data curation. Paola Mazzetti: Data curation, Project administration, Validation. Ersilia Lucenteforte: Formal analysis. Alfredo Rosellini: Formal analysis. Alice Cara: Formal analysis. Paola Quaranta: . Valerio Mainardi: Resources. Pietro Villa: Resources. Daniele Focosi: Resources, Validation. Maria Lanza: Resources. Irene Bianco: Resources. Alessandro Mazzoni: Resources, Validation. Marco Falcone: Resources. Francesco Menichetti: Resources. Fabrizio Maggi: Resources. Michele Lai: Writing - original draft, Writing - review & editing, Software. Giulia Freer: Writing - original draft, Writing - review & editing, Software. Mauro Pistello: Conceptualization, Funding acquisition, Writing - review & editing, Supervision.

Declaration of Competing Interest

The authors declare no competing financial interest.