Evaluation of Cell-Based and Surrogate SARS-CoV-2 Neutralization Assays

ABSTRACT

Determinants of protective immunity against severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) infection require the development of well-standardized, reproducible antibody assays. This need has led to the emergence of a variety of neutralization assays. Head-to-head evaluation of different SARS-CoV-2 neutralization platforms could facilitate comparisons across studies and laboratories. Five neutralization assays were compared using 40 plasma samples from convalescent individuals with mild to moderate coronavirus disease 2019 (COVID-19): four cell-based systems using either live recombinant SARS-CoV-2 or pseudotyped viral particles created with lentivirus (LV) or vesicular stomatitis virus (VSV) packaging and one surrogate enzyme-linked immunosorbent assay (ELISA)-based test that measures inhibition of the spike protein receptor binding domain (RBD) binding its receptor human angiotensin converting enzyme 2 (hACE2). Vero cells, Vero E6 cells, HEK293T cells expressing hACE2, and TZM-bl cells expressing hACE2 and transmembrane serine protease 2 were tested. All cell-based assays showed 50% neutralizing dilution (ND₅₀) geometric mean titers (GMTs) that were highly correlated (Pearson r = 0.81 to 0.89) and ranged within 3.4-fold. The live virus assay and LV pseudovirus assays with HEK293T/hACE2 cells showed very similar mean titers, 141 and 178, respectively. ND₅₀ titers positively correlated with plasma IgG targeting SARS-CoV-2 spike protein and RBD (r = 0.63 to 0.89), but moderately correlated with nucleoprotein IgG (r = 0.46 to 0.73). ND₈₀ GMTs mirrored ND₅₀ data and showed similar correlation between assays and with IgG concentrations. The VSV pseudovirus assay and LV pseudovirus assay with HEK293T/hACE2 cells in low- and high-throughput versions were calibrated against the WHO SARS-CoV-2 IgG standard. High concordance between the outcomes of cell-based assays with live and pseudotyped virions enables valid cross-study comparison using these platforms.

INTRODUCTION

The coronavirus disease 2019 (COVID-19) pandemic, caused by severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2), has caused more than 100 million confirmed infections and over 2.4 million deaths worldwide as of February 15, 2021 (https://www.worldometers.info/coronavirus). Despite governmental regulations designed to minimize virus transmission and reduce mortality, such as mask use and social distancing guidelines, vaccines are required to limit the spread of the virus and the burden of COVID-19.

Most efficacious licensed vaccines would elicit pathogen-neutralizing antibodies (nAb) (1). Humans can mount nAb responses against SARS-CoV-2 during natural infection (2–5). Epidemiologic data suggest that reinfection rates are low, albeit increasing numbers of sporadic reinfections are being reported (6, 7). A crucial unknown at this time is what immune responses are associated with protective immunity. While there is mixed evidence supporting the efficacy of convalescent-phase sera infusion for disease shortening, recent studies suggest that passive infusion of monoclonal antibodies can alter COVID-19 progression (8, 9). In order to determine what constitutes protective immunity, well-standardized, reproducible antibody assays are required to establish correlates of risk and protection. Efficacy data for several SARS-CoV-2 vaccines have been already published, but analyses of correlates of protection are yet to come (10–12). For that, massive serological measurements, including virus neutralization, are under way. In this regard, it is important to understand how results obtained with different virus neutralization platforms can be compared.

The plaque reduction neutralization test (PRNT) is considered a “gold standard” to assess virus neutralizing potency of a serum or antibody sample. However, a variety of live virus neutralization assays that use recombinant SARS-CoV-2 (rSARS-CoV-2) containing a reporter gene at the ORF7 locus of the viral genome have been suggested as alternatives (13, 14). These recombinant viruses replicate similar to SARS-CoV-2 clinical isolates in vitro and successfully infect primary airway epithelial cell cultures. A fluorescence-based rSARS-CoV-2 neutralization assay yielded comparable results to PRNT in nAb detection from convalescent patient plasma (13). With a shorter turnaround time (24 to 48 h for reporter virus versus 3 days for PRNT), rSARS-CoV-2 provides a useful high-throughput (HTS) platform to study nAb responses but unfortunately still requires biosafety level 3 (BSL-3) containment for assay set-up and readout.

Reporter assays with pseudotyped viruses restricted to a single round of replication allow nAb experiments to be performed in BSL-2 laboratories. Pseudotyped viral particles created with lentivirus (LV) and vesicular stomatitis virus (VSV) (15–18) packaging platforms have already been adapted for SARS-CoV-2 (19–21). Several cell lines endogenously or exogenously expressing angiotensin converting enzyme 2 (ACE2), the host receptor for the SARS-CoV-2 spike protein, have been tested, and Vero cells were among the most susceptible to VSV pseudovirus entry (22–24). HEK 293T cells transfected to express ACE2 have also been developed for use in pseudovirus neutralization assays (25). In addition to ACE2, transmembrane serine protease 2 (TMPRSS2) has been shown to prime the spike protein for viral cell entry (24).

Because the receptor binding domain (RBD) of the spike protein is the major target for nAbs (26–28), surrogate enzyme-linked immunosorbent assay (ELISA)-based assays were introduced to evaluate antibodies that compete with ACE2 for RBD binding (20, 29, 30). Major advantages of these assays include low cost, speed, and safety. As opposed to measuring actual virus neutralization, surrogate assays report percent binding inhibition between RBD and ACE2, which is then interpreted as percent neutralization. While they provide inexpensive and rapid detection of RBD-targeting nAbs, surrogate assays cannot measure neutralization via non-RBD spike protein epitopes. The importance of this issue has increased with the increasing prevalence of escape resistant variants of SARS CoV-2 (31–33) (https://www.cdc.gov/coronavirus/2019-ncov/cases-updates/variant-surveillance/variant-info.html).

The global pandemic led to the unprecedented rapid development and implementation of many SARS-CoV-2 neutralization assays. However, interassay comparison and validation is needed to better understand antibody kinetics and longevity of humoral immune responses, correlates of immune protection, and vaccine efficacy (34). In the current study, we aimed to fill this gap by evaluating the same set of plasma samples from convalescent individuals with mild to moderate COVID-19 with five SARS-CoV-2 neutralization assays, including (i) a live rSARS-CoV-2 assay on Vero E6 cells, (ii) VSV pseudotyped with SARS-CoV-2 spike on Vero cells, (iii) LV pseudotyped with SARS-CoV-2 spike on HEK293T cells expressing hACE2 in a regular and HTS format, (iv) LV pseudovirus on TZM-bl cells expressing hACE2 and TMPRSS2, and (v) a surrogate, ELISA-based test that measures inhibition of binding between RBD and ACE2. We also examined the correlation between neutralization and the plasma concentration of SARS-CoV-2 nucleoprotein-, spike-, and RBD-specific IgG.

MATERIALS AND METHODS

Detailed descriptions of reagents and procedures are available in the supplemental material.

Study population and specimen collection.

Plasma samples used for this study were obtained from participants (≥18 years of age) of a seroepidemiology study following a county-wide outbreak of SARS-CoV-2 in Blaine County, ID, in March to April 2020. Study participants were randomly selected after stratification by ZIP code and within ZIP code, age, gender, and race/ethnicity. All volunteers signed electronic consent forms. Demographic information and symptom histories since January 15, 2020, were collected.

Blood was collected in 10-ml vials with acid citrate dextrose and shipped overnight to the laboratory (Fred Hutch, Seattle, WA) where plasma was separated by centrifugation. One aliquot was submitted for the Architect SARS-CoV-2 IgG assay (Abbott, Abbott Park, IL). Other aliquots were heat inactivated for 30 min at 56°C, frozen at −80°C, and distributed to testing laboratories for SARS-CoV-2 neutralization assays. Study participants were informed of the qualitative results of the IgG serology assay via email within 1 week of obtaining test results. This study was approved by the Fred Hutchinson Cancer Research Center Institutional Review Board, and all study materials were provided in both English and Spanish.

Cell lines.

Vero cells (CCL-81; ATCC, Manassas, VA) are kidney epithelial cells of Cercopithecus aethiops; Vero E6 (CRL-1586; ATCC) is a cloned variant of Vero cells. Human embryonic kidney cells (CRL-3216; ATCC), HEK293T, expressing hACE2 (293T/ACE2.MF) were kindly provided by Mike Farzan and Huihui Mu at Scripps (La Jolla, CA). TZM-bl cells (also called JC53BL-13; NIH AIDS Research and Reference Reagent Program, no. 8129) are a HeLa cell derivative engineered by amphotropic retroviral transduction to express CD4, CXCR4, and CCR5 (35) and to contain Tat-responsive reporter genes for firefly luciferase (Luc) and Escherichia coli β-galactosidase (36) and are additionally engineered to express both ACE2 and TMPRSS2 (TZM-bl/ACE2/TMPRSS2 cells); these were kindly provided by Mike Farzan and Huihui Mu at Scripps.

Viruses.

All assays were performed under BSL-2 conditions unless noted differently.

(i) Live SARS-CoV-2. Live recombinant SARS-CoV-2-nanoLuc virus (rSARS-CoV-2-nLuc) was prepared as described elsewhere (14).

(ii) VSV pseudovirus. VSV pseudovirus was prepared using a codon-optimized SARS-CoV-2 spike protein (YP_009724390.1) and a VSV(G*ΔG-luciferase) system purchased from Kerafast (Boston, MA) (18, 37). VSV(G*ΔG-luciferase) pseudotyped with SARS-CoV-2 spike (PsVSV-Luc-D19) was produced in 293T cells and stored at −80°C. Median tissue culture infectious dose (TCID₅₀) was measured using Vero cells (CCL-81; ATCC) with serial 2-fold dilutions of the prepared pseudovirus.

(iii) LV pseudoviruses. An expression plasmid encoding codon-optimized full-length spike of the Wuhan-1 strain (VRC7480) was provided by Barney Graham and Kizzmekia Corbett at the Vaccine Research Center, National Institutes of Health (USA). The D614G mutation was introduced into VRC7480 by site-directed mutagenesis using the QuikChange Lightning site-directed mutagenesis kit (Agilent Technologies, Santa Clara, CA) (LV-pseudo). Pseudovirions were produced in HEK 293T/17 cells (CRL-11268; ATCC). Culture supernatants from transfections were clarified of cells by low-speed centrifugation and filtration (0.45-μm filter) and were stored in 1-ml aliquots at −80°C.

For the HTS format of the LV pseudovirus assay, the pseudovirus was prepared in 293T cells using a five-plasmid system as described in (38). Lentiviral backbone plasmids and SARS-CoV-2 spike (Wuhan-1, D614G) vector were provided by Jessy Bloom at Fred Hutch.

Detection of IgG antibodies to SARS-CoV-2 using a commercial serologic assay.

Plasma samples were tested at the Clinical Laboratory Improvement Amendments (CLIA)-certified University of Washington Virology lab using the Architect SARS-CoV-2 IgG assay (Abbott) under the FDA’s emergency use authorization. The assay is a chemiluminescent microparticle immunoassay that measures IgG antibodies to the SARS-CoV-2 nucleocapsid protein. Qualitative results and index values reported by the instrument were used in analyses. A recommended index value cutoff of 1.40 was used for determining positivity (39).

Luminex SARS-CoV-2 IgG binding antibody assay.

Detailed descriptions can be found in the supplemental material. Two replicate dilutions of plasma were incubated with MagPlex beads conjugated with SARS-CoV-2 spike, RBD, nucleoprotein, and tetanus toxoid followed by incubation with anti-human IgG Fc-phycoerythrin (Fc-PE) (Southern Biotech, Birmingham, AL). Background was established by measuring the mean fluorescence intensity (MFI) of beads conjugated to antigens incubated in assay buffer and was subtracted from all readings. Pooled sera collected in 2015 to 2016 from normal human donors was included as the negative control for SARS-CoV-2 antigens. Convalescent plasma from a subject with PCR-confirmed severe COVID-19 was used as a positive control.

Concentration of antigen-specific IgG was estimated using a standard curve based on the measurement of MFI for serial dilutions of standard IgG (Sigma, St. Louis, MO) captured by MagPlex beads conjugated with goat anti-human Ig Fab-specific antibody (Southern Biotech). MFI readings and associated IgG concentrations were fitted to a four-parameter logistic curve (4PL) using the R packages nCal and drc.

Live SARS-CoV-2 neutralization assay.

All the live virus experiments were performed under BSL-3 conditions at negative pressure by operators in Tyvek suits wearing personal powered air-purifying respirators. Vero E6 cells were seeded at 2 × 10⁴ cells/well in a 96-well plate 24 h before the assay was performed. An 8-point, 3-fold dilution curve was generated for each sample with a starting concentration of 1:50. Seventy-five PFU of rSARS-CoV-2-nLuc (14) was mixed with individual patient plasma at a 1:1 ratio and incubated at 37°C for 1 h; after that, virus was added to cells and incubated at 37°C in 5% CO₂ for 48 h. Luciferase was measured as relative luminescence units (RLU) by a Nano-Glo luciferase assay system (Promega, Madison, WI) following manufacturer protocols using a SpectraMax M3 luminometer (Molecular Devices, San Jose, CA). Percent neutralization was calculated by the following equation: (1 − [RLU with sample/RLU with mock treatment]) × 100. Mouse serum produced by BALB/c mice immunized with SARS-CoV-2 spike was used as a positive control (14).

VSV pseudovirus neutralization assay.

Vero cells were seeded at 2 × 10⁴ cells/well in black-walled 96-well plates 24 h before the assay was performed. A 7-point, 3-fold dilution curve was generated with a starting sample dilution of 1:20. PsVSV-Luc-D19 (3.8 × 10² TCID₅₀) was mixed with the plasma dilutions, incubated at 37°C in 5% CO₂ for 30 min, and then transferred onto Vero cells. Cells were incubated for 18 to 20 h. Luciferase activity was measured by a Bio-Glo luciferase assay system (Promega) using a 2030 Victor X3 multilabel reader (PerkinElmer, Waltham, MA). Percent virus neutralization was calculated as in the live virus assay. Plasma collected from a subject with severe, PCR-confirmed SARS-CoV-2 infection collected after the person was released from the hospital was used as a positive control. Pooled human serum collected in 2015 to 2018 was used as a negative control.

LV pseudovirus neutralization assays.

(i) 293T/ACE2 cells pseudovirus assay. A pretitrated dose of LV-pseudo was incubated with serial 3-fold dilutions of plasma in duplicate for 1 h at 37°C in 96-well plates. Freshly trypsinized 10⁴ 293T/ACE2 cells were added to each well. One set of control wells received cells + virus (virus control) and another set received cells only (background control). After 68 to 72 h of incubation, 100 μl of cell lysate was transferred to a 96-well plate for measurements of luminescence using the Promega luciferase assay system (Promega).

(ii) ACE2/TMPRSS2 TZM-bl cells pseudovirus assay. The assay was carried out similar to the 293T/ACE2 cell pseudovirus assay with the exception that the growth medium used for infection of TZM-bl/ACE2/TMPRSS2 cells contained 75 μg/ml DEAE dextran. After 68 to 72 h of incubation, 100 μl of cell lysate was transferred to a 96-well plate (Costar) for luminescence measurement using the BriteLite luminescence reporter gene assay system (PerkinElmer). Percent virus neutralization was calculated as previously mentioned.

For both LV pseudovirus assays, SARS-CoV-2 neutralizing monoclonal antibody COVA1-18 (40) was used as a positive control, and normal human serum collected in 2016 was used as a negative control.

(iii) HTS version of 293T/ACE2 cells pseudovirus assay. An HTS SARS-CoV-2 neutralization assay was performed in the CLIA-certified University of Washington Virology lab using a Mantis liquid handler (Formulatrix, Bedford, MA) to dispense growth medium, virus, and luciferase substrate. The 293T/ACE2 cells were seeded in 96-well black-walled plates manually at 12,500 cells/well and were incubated for 16 to 18 h. Various amounts of growth medium were dispensed into 96-well plates using Mantis according to the plate map. In the plates with growth medium, patient sera were manually diluted 10-fold followed by six 3-fold serial dilutions for a total of seven dilution points at 60 μl of sample per well. Mantis was then used to dispense 60 μl of diluted pseudovirus at 4 × 10⁵ RLU/well into the 96-well plates with serially diluted serum samples. After incubating at 37°C for 1 h, 100 μl of the pseudovirus and serum mixture was manually added to the 293T/ACE2 cells in 96-well plates. At 52 to 58 h postinfection, 100 μl of medium was manually removed from each well and 30 μl of Bright-Glo luciferase substrate was added by the Mantis. The plates were read with a Victor Nivo multimode microplate reader (PerkinElmer).

For all neutralization assays, neutralization titers are the reciprocal of the plasma dilution at which RLU were reduced by 50% (ND₅₀) and 80% (ND₈₀) compared to virus control wells after subtraction of background RLU.

SARS-CoV-2 surrogate virus neutralization test.

The SARS-CoV-2 surrogate virus neutralization test (sVNT) was performed in a BSL-1 laboratory and was performed according to manufacturer’s (GenScript, Piscataway, NJ) protocol recommendations. Briefly, capture plates were incubated with plasma samples diluted 1:10, washed, and probed with secondary antibody conjugated to horseradish peroxidase. Plates were developed with 3,3′,5,5′-tetramethylbenzidine (ThermoFisher, Waltham, MA), and optical density (OD) at 450 nm was measured using a SpectraMax M2 reader (Molecular Devices). Positive and negative controls were provided in the kit. Binding inhibition was determined using the following formula: inhibition = (1 – [OD of sample/OD of negative control]) × 100. Percent binding inhibition was interpreted as a percent neutralization. In order to determine ND₅₀, plasma samples were serially diluted starting from 1:10, and the assay was performed as described above.

Assay calibration with the WHO anti-SARS-CoV-2 immunoglobulin standard.

The First WHO International Standard for anti-SARS-CoV-2 antibodies developed and distributed by the National Institute for Biological Standards and Control (NIBSC) of the United Kingdom (number 20/136) was used to establish calibrating factors for VSV-pseudo/Vero, LV-pseudo/293T, and HTS-LV-pseudo/293T assays as follows. The lyophilized standard was reconstituted in ultrapure water as per NIBSC instructions. The resulting serum was stored at 4°C for no longer than 1 week and was used in the assays similar to as described above for patient samples via serial dilutions starting at 1:20. The ratio between the assigned neutralization unitage (1,000 IU/ml) and measured ND₅₀ and ND₈₀ for the standard sample was used as a calibrating factor to convert assay-derived ND₅₀ and ND₈₀ readouts into IU/ml.

Statistical analysis and visualization.

Neutralization titers were defined as the plasma dilution that reduced RLU by 50% (ND₅₀) or 80% (ND₈₀) relative to virus control wells (cells + virus only) after subtraction of background RLU in cell-only control wells (see supplemental material for details). Correlations were estimated between pairs of neutralization or binding antibody readouts using Pearson’s correlation coefficient (r), and group means were compared using a paired two-sample t test; measures in units of neutralization and IgG concentration were logged before estimating correlation and comparing group means. Association of neutralization and IgG concentration with age and body mass index (BMI) were conducted using Spearman’s rank correlation. Statistical significance was based on a P value of <0.05.

RESULTS

Cohort characteristics, demographics, survey participation, and serological testing.

To characterize and compare different platforms of SARS-CoV-2 nAb assays, we used plasma samples obtained from a seroepidemiology study conducted May 4 to 19, 2020, following a county-wide outbreak of SARS-CoV-2 in Blaine County, ID. Of 967 participants, 222 (22.8%) had IgG antibodies to SARS-CoV-2 nucleoprotein as measured by the Abbott Architect test (index value of ≥1.40) indicative of prior infection with SARS-CoV-2. From these 222 samples, we randomly selected 40 plasma samples for use in evaluating SARS-CoV-2 neutralization assays. Selected participants had a median age of 51.5 years (range, 23 to 81), and 60% identified as female (Table 1). Only one participant reported being hospitalized, and four participants (10%) were self-described as asymptomatic. Among participants reporting different symptoms, 57.5% had fever, while fatigue (87.5%), cough (72.5%), headache (67.5%), and chills (65%) were more prevalent (Table S1 in the supplemental material). The majority of participants reported COVID-19 symptoms occurring in March of 2020. Based on this, our cohort can be categorized as representing mild to moderate symptomatic COVID-19 infections with samples collected at about 1.5 to 2 months after disease onset.

TABLE 1.

Demographic and exposure/symptom characteristics of study participants

Characteristic	n	%
Age (yrs)
23–40	8	20
41–50	11	27.5
51–60	11	27.5
61–70	6	15
>70	4	10
Median	51.5
Range	23–81
Gender
Female	16	40
Male	24	60
Exposures/symptoms
Tested positive	8	20
Symptomatic contact of known positive	9	22.5
Symptomatic without confirmation	19	47.5
Asymptomatic contact of someone symptomatic	2	5
Asymptomatic, no exposures	2	5
Travel outside United States since 1 December 2019	7	17.5
Other
Essential worker	6	15
Lives with children	14	35

We measured the concentration of IgG in participant sera targeting SARS-CoV-2 spike, RBD, and nucleoprotein via a quantitative, Luminex-based immunoassay. IgG to tetanus toxoid was measured as a proxy for overall IgG level. IgG to spike and RBD were detected in all 40 plasma samples, indicating a seroconversion on all SARS-CoV-2 antigens. The mean plasma concentrations for spike- and RBD-specific IgG were 2.8 μg/ml (95% confidence interval [CI], 1.9 to 4.1) and 2.1 μg/ml (95% CI, 1.4 to 3.3), respectively, which were considerably lower than those to nucleoprotein (7.3 μg/ml [95% CI, 5.3 to 10]) (Fig. 1A). The concentration of tetanus-specific IgG was higher than IgG targeting SARS-CoV-2 antigens for all individuals (mean, 14.5; 95% CI, 11.1 to 18.9). Although the Abbott SARS-CoV-2 IgG CLIA test is designed and used for qualitative detection of IgG against the SARS-CoV-2 nucleoprotein, the instrument reports index values that can be used in quantitative analyses (Fig. 1B) (39).

FIG 1.

SARS-CoV-2 neutralization and binding antibody concentration from COVID-19 convalescent patients. (A) Concentration of IgG against SARS-CoV-2 spike, RBD, nucleoprotein, and tetanus toxoid measured in the Luminex binding antibody assay. (B) Indexes reported by the Abbott Architect nucleoprotein IgG test. (C) ND₅₀ and (D) ND₈₀ neutralization titer measured using five SARS-CoV-2 neutralization assays for 40 plasma samples from 40 participants. Each assay defined its own lower limit of detection (LOD) based on the initial dilution: 50-fold for SARS-CoV-2/VeroE6, 20-fold for the LV and VSV pseudovirus assays, and 10-fold for the sVNT. Data below the LOD (open triangles) are plotted at LOD/2. Number and percentage of samples above the LOD are indicated above each plot. For each assay, the box represents the extent of the interquartile range (IQR) with a line indicating the median; whiskers extend to 1.5 times the IQR.

Cell-based assays provided comparable estimates of neutralization activity.

Forty selected plasma samples were distributed across four laboratories conducting different SARS-CoV-2 neutralization assays (Table 2). Serial plasma dilutions were used in cell-based assays to generate titration curves (Fig. S1) and estimate the 50% and 80% neutralizing dilutions (ND₅₀ and ND₈₀, respectively). In the sVNT, only 22 of 40 samples showed neutralization above 50% when analyzed according to the manufacturer’s protocol in a single 1:10 dilution (Fig. S2A), and 13 samples representing different neutralization capacity were selected for ND₅₀ measurement using serial dilutions (Fig. S2B).

TABLE 2.

SARS-CoV-2 neutralization assay platforms used in the study

	SARS-CoV-2/VeroE6	VSV-pseudo/Vero	LV-pseudo/293T	LV-pseudo/TZM-bl	HTS-LV-pseudo/293T	Surrogate virus neutralization test (sVNT)
Lab	Baric	Corey	Montefiori	Montefiori	Huang/Jerome	Corey
Cell line	Vero E6	Vero	HEK293T	TZM-bl	HEK293T	None
ACE2 expression	Endogenous	Endogenous	Engineered	Engineered	Engineered	Recombinant
TMPRSS2 expression	No	No	No	Engineered	No	NAa
Virus shorthand	rSARS-CoV-2-nLuc	VSV-pseudo	LV-pseudo	LV-pseudo	HTS-LV-pseudo	NA
Virus type	Live recombinant	VSV(G*ΔG-luciferase) pseudotyped	pCMV-ΔR8.2 lentiviral packaging with pHR'-CMV-Luc	pSG3ΔEnv lentiviral packaging	pHDM lentiviral packaging	NA
SARS-CoV-2 strain/isolate	WA-CDC-WA1-A12/2020	Wuhan-Hu-1	Wuhan-Hu-1 (VRC7480) D614G	Wuhan-Hu-1 (VRC7480) D614G	Wuhan-Hu-1 D614G	Unknown
GenBank	MT020880.1	MN908947.3	MN908947.3	MN908947.3	MN908947.3	NA
Amino acid 614	D	D	G	G	G	Unknown
Biosafety level	3	2	2	2	2	1

^aNA, not available.

Overall, the cell-based assays showed comparable estimated ND₅₀ geometric mean titers (GMTs) with considerable overlap in the interquartile range and 95% CI among pairs of assays (Fig. 1C; see also Fig. S3A and Table S2). The SARS-CoV-2/VeroE6 and LV-pseudo/293T assays yielded very similar ND₅₀ GMT values, 141 (95% CI, 93 to 214) versus 178 (95% CI, 112 to 283), respectively. The estimated mean ND₅₀ values for other cell-based assays were also comparable; however, ND₅₀ GMTs for VSV-pseudo/Vero test and HTS-LV-pseudo/293T were the highest among cell-based assays (310 [95% CI, 211 to 454] and 272 [95% CI, 267 to 643], respectively; Table S2). The live virus assay and all three LV pseudovirus assays yielded ND₅₀ values within a 2-fold range, indicating high concordance. Notably, rSARS-CoV-2-nLuc and PsVSV-Luc-D19 contained the spike protein with an aspartate residue at position 614 (Wuhan-1 strain), while the LV pseudoviruses contained spike protein with the D614G mutation. Nevertheless, the difference between outcomes of LV-pseudo/293T assays in regular and HTS formats and the VSV-pseudo/Vero assay were within 2-fold. The lowest ND₅₀ GMT (from the LV-pseudo/TZM-bl assay) was 3.4-fold lower than that of the highest-yielding assay (VSV-pseudo/Vero).

Despite overlapping distributions, it was possible to detect shifts in ND₅₀ for each cell-based assay using a two-sample, paired t test (P < 0.05); the exceptions were SARS-CoV-2/VeroE6 versus LV-pseudo/293T and HTS-LV-pseudo/293T versus VSV-pseudo/Vero. The ND₅₀ GMTs in these two assay pairs were not significantly different (P = 0.112 and 0.856, respectively). Taken together, these data demonstrate that the variability across different cell-based assays is low, and it suggests that results of cell-based assays could be adjusted for head-to-head comparability, as we show more in the next section. In contrast, the sVNT yielded significantly different ND₅₀ values, up to 26-fold lower than cell-based assays.

Differences and similarities among the cell-based assay ND₅₀ values were generally recapitulated using the ND₈₀ values (Fig. 1D; see also Fig. S3B and Table S2). The sVNT and VSV-pseudo/Vero assays yielded the lowest and the highest ND₈₀ GMTs, respectively. However, the overall difference between ND₈₀ values was less dramatic than for ND₅₀. For all cell-based assays, it was within a 3-fold range, and sVNT ND₈₀ was only 6- to 17-fold lower than cell-based assays. As expected, the ND₈₀ titers were consistently lower than the ND₅₀ titers (Fig. S4, Table S3). For other pseudovirus assays, the difference between ND₅₀ and ND₈₀ was greater and ranged between 3-fold and 4.6-fold (Table S3). Interestingly, the live virus assay showed the smallest difference between ND₅₀ and ND₈₀ GMTs, 1.95-fold (Table S3), a direct consequence of the steeper titration curves observed for this assay (Fig. S1A). Indeed, the slope parameter from the live virus neutralization curves was higher than in other assays (slope B = 3.3 versus 0.6, 1.4, and 1.5 for LV-pseudo/293T, LV-pseudo/TZM-bl, and VSV-pseudo/Vero, respectively; all P < 0.001).

The neutralization assay response rate was in agreement with estimated ND₅₀ and ND₈₀ values. The highest response rates came from the VSV-pseudo/Vero assay (100% of ND₅₀ and 97.5% of ND₈₀ titers) and HTS-LV-pseudo/293 assays (100% of ND₅₀ and ND₈₀ titers) (Fig. 1C and D). The lowest response rate among the cell-based assays was measured via the SARS-CoV-2/VeroE6 assay, and the sVNT was the lowest overall. For the SARS-CoV-2/VeroE6 assay, the lower response rate than other cell-based assays could be due to the starting plasma dilution, which was 1:50 versus 1:20 used in the pseudovirus assays.

Strong correlation among neutralization assays.

We conducted a correlation analysis of the ND₅₀ and ND₈₀ values derived from each of the five neutralization assays (Fig. 2; Fig. S5). The live virus and all four pseudovirus neutralization assays generated ND₅₀ values that were highly correlated across samples (Pearson r = 0.78 to 0.89), with the highest correlation observed between the three LV pseudovirus assays (r = 0.89; 95% CI, 0.81 to 0.94; P < 0.001). The readout with the lowest correlation with the cell-based assays was the sVNT ND₅₀ (r = 0.32 to 0.6), although sVNT percent neutralization tended to be more highly correlated (r = 0.73 to 0.8). Similar correlations were observed for ND₈₀ outcomes for cell-based assays (r = 0.69 to 0.88) (Fig. 2B).

FIG 2.

Correlation among assay readouts measuring neutralization or antigen-specific IgG concentration in plasma. Heat map color is determined by the Pearson’s correlation coefficient (r, annotations). Each panel includes either ND₅₀ titers (A) or ND₈₀ titers (B) and their correlation with sVNT percent neutralization, SARS-CoV-2-specific IgG concentration (Luminex bead-based assay), the quantitative index of the Abbott nucleoprotein assay, and tetanus toxoid-specific IgG concentration. ND₅₀ and ND₈₀ values below 50 were truncated at 25.

Plasma neutralization potency correlated with concentration of SARS-CoV-2 binding IgG.

Correlation analyses revealed a strong association between levels of IgG to spike and RBD (r = 0.89; 95% CI, 0.81 to 0.94) (Fig. 2). Luminex immunoassay-measured nucleoprotein-specific IgG highly correlated with the quantitative index of the Abbott SARS-CoV-2 IgG assay (r = 0.95; 95% CI, 0.91 to 0.97), which is based on detection of nucleoprotein-specific IgG, but both parameters only moderately correlated with IgG targeting the other viral antigens (r = 0.58 to 0.68). There was no significant correlation between tetanus-specific IgG and IgG to SARS-CoV-2 antigens (all P > 0.05).

Next, we examined the relationship between virus neutralization and IgG levels to spike, RBD, and nucleoprotein. IgG concentrations to each antigen positively correlated with the ND₅₀ titer measured by each neutralization assay (r = 0.46 to 0.83) (Fig. 2A; Fig. S6). The strongest correlation was observed between sVNT percent neutralization and concentration of RBD IgG (r = 0.89). Among the cell-based assays, the live virus ND₅₀ titer showed the strongest correlation with IgG against spike and RBD (r = 0.83 for both), followed by the VSV-pseudovirus/Vero assay (r = 0.83 and 0.76, respectively). Notably, nucleoprotein-specific IgG only moderately correlated with ND₅₀ titers from the cell-based assays but showed a strong correlation with sVNT percent neutralization. Tetanus-specific IgG did not correlate with any of the SARS-CoV-2-associated IgG concentrations or neutralization titers.

With the caveat that our cohort is rather small for such analyses, we found a moderately positive correlation between age and concentration of spike-specific IgG (Spearman’s rho = 0.37, P = 0.02), RBD-specific IgG (rho = 0.39, P = 0.013), and nucleoprotein-specific IgG (rho = 0.45, P = 0.003) (Table S4). Similarly, there were positive correlations between age and neutralization titer (Table S4, Fig. S7), although the correlations tended to be higher with ND₈₀ titer (rho = 0.51, P = 0.001) than with ND₅₀ titer (rho = 0.28, P = 0.075).

Assay calibration with the WHO anti-SARS-CoV-2 immunoglobulin standard.

To evaluate readout conversion between assays, we calibrated VSV-pseudo/Vero, LV-pseudo/293T, and HTS-LV-pseudo/293T using the First WHO International Standard for anti-SARS-CoV-2 antibodies (Table 3). After conversion, the regular and HTS version of the LV-pseudo assay reported the same ND₅₀ GMT of 58.4 IU/ml. Of note, raw ND₅₀ titers for these assays also showed high concordance and had a less than 2-fold difference. GMT ND₅₀ from the VSV-pseudo/Vero assay was found at 205 IU/ml after calibration. If before calibration the difference in ND₅₀ titers was about 2-fold between VSV and LV pseudovirus assays, after calibration, it increased up to 3.5-fold. Conversion of ND₈₀ GMTs into IU/ml format produced perplexing results. The ND₈₀ value after calibration became greater than ND₅₀ for both LV-pseudo assays and was almost equal to ND₅₀ for the VSV-pseudo assay (Table 3).

TABLE 3.

Calibration of SARS-CoV-2 neutralization assays using first WHO standard for anti-SARS-CoV-2 immunoglobulin

First WHO International Standard for anti-SARS-CoV-2 immunoglobulin	VSV-pseudo/Vero		LV-pseudo/293T		HTS-LV-pseudo/293T
	ND50	ND80	ND50	ND80	ND50	ND80
WHO standard, GMT	1,511	557	3,047	567	4,650	1,396
Calibration factor (1,000 IU/ml ÷ standard GMT)	0.662	1.795	0.328	1.764	0.215	0.716
GMT neutralization titer among participants	309.7	102.8	177.9	41.96	271.7	86.3
Calibrated readout (IU/ml)	205	184.5	58.4	74.0	58.4	61.8

To provide context for our data, we accessed the WHO report that established their reference standard (41) and analyzed the GMTs that were contributed by different research groups using a range of assays and reference samples. We pooled measurements from LV pseudovirus and VSV pseudovirus assays and calculated their respective ND₅₀ GMT values. Of note, most of the LV and VSV pseudovirus assays used for establishing the WHO standard contained the Wuhan-1 D614 spike (41). The calculated GMT for the WHO standard was 1,347 for the VSV pseudovirus assay and 3,406 for the LV pseudovirus assay and were similar to the cognate values from our study (Table 3).

To test possible influence of D614G mutation on the assay readout, we tested VSV pseudovirus carrying D614 versus G614 using the WHO standard and found no difference in ND₅₀ or ND₈₀ titers between virus variants (Fig. S8). Therefore, the difference in readouts between VSV and LV platforms is either due to the target cells or the virus used for pseudotyping.

DISCUSSION

In this study, we conducted a detailed comparison of four cell-based and one ELISA-based SARS-CoV-2 neutralization assays using a set of 40 plasma samples collected from SARS-CoV-2 convalescent individuals with mild to moderate disease. Our data show a high level of congruency among cell-based assays, suggesting that the results obtained with any of the tested pseudovirus platforms accurately reflect the potency of the sample to neutralize the Wuhan-Hu-1 strain of SARS-CoV-2. The 50% and 80% neutralization titers strongly correlate between different assays as well as between the neutralization assays and plasma concentration of RBD and spike-specific IgG, which is consistent with other studies (19, 42–46). Although the correlation was modest in comparison, the ELISA-based sVNT results also positively correlated with the other neutralization assays. The demonstrated differences in ND₅₀ and ND₈₀ GMTs between assays should be considered when conducting SARS-CoV-2 natural history studies and vaccine trials.

Although levels of spike-specific IgG highly correlated with neutralization, our data do not confirm that all IgG targeting the spike protein have neutralization activity. Rather, the results imply that individuals who produce spike-specific binding antibodies are also likely to make neutralizing IgG. The correlation between nucleoprotein-specific IgG and neutralization was consistently lower than the correlations between spike- and RBD-specific IgG with neutralization. This is not surprising, as much of the immunodominant response associated with neutralization involves binding and/or blocking the spike RBD to inhibit viral entry to host cells (26, 28, 47).

The association of age with both spike-specific IgG and neutralization titer suggests that the previously reported association of high neutralization titer among older individuals may be mediated by higher concentrations of spike and RBD-specific IgG (48, 49). However, this is not a result of cross-reactive humoral responses to prior infections with seasonal coronaviruses (50, 51). Whether this is a direct effect of age on the developing immune response to SARS-CoV-2 or a result of cross-reacting T-cell immunity remains unclear (52, 53). Although our cohort was well balanced by sex and age and a positive correlation between age and neutralizing titers was detected, the influence of other demographic and environmental factors cannot be excluded due to a small sample size collected in a limited geographic origin (54, 55). As such, larger, geographically distinct cohorts are required for proper analyses as, for example, a recent publication showing good congruency across geographically distant laboratories with the VSV pseudovirus assay (56).

Our study shows that ELISA-based surrogate assays have two major limitations: (i) inability to account for synergistic action of antibodies targeting different epitopes, and (ii) limit detection to only antibodies that block the RBD/ACE2 interaction, thus missing other antibodies that neutralize via non-RBD sites on the virus glycoprotein (27, 57). In fact, synergistic action of antibodies targeting the RBD and S2 domain has been reported (58). Thus, surrogate assays have a lower sensitivity than cell-based assays and can lead to more false-negative results. These results contradict the use of sVNT as a rapid assay to select positive samples for further screening with cell-based assays, as was recently suggested (59).

TMPRSS2 was shown to be essential for SARS-CoV-2 infectivity of different cell types, although there was no significant difference observed in virus titer at 48 h postinfection between wild-type Vero cells and Vero cells expressing furin (14, 24). Our comparison revealed that the presence of TMPRSS2 is not critical for assay performance, as TZM-bl cells expressing both ACE2 and TMPRSS2 showed no significant difference compared to 293T cells expressing only ACE2.

In addition to lower safety requirements than assays using replication-competent SARS-CoV-2, pseudotyped virus assays are well positioned for HTS testing of antibody responses elicited by natural infection, vaccination, and now, critically, to new viral variants of concern (VOC), as reported for other viruses (60–62). One limitation of the current study was that only two strains, Wuhan and D614G, were tested. The D614G mutation has been shown to be moderately more susceptible to neutralization (63, 64), while in other reports no difference was observed (65, 66). Our results indicate no pattern between D614G and neutralization capability, and thus the marginal differences observed between assays are likely due to assay sensitivity rather than viral sequence. Neutralization of recently emerging VOCs (67) by serum and monoclonal antibodies has been measured using several assay platforms (68–70); however, assay standardization and validation will be required for proper comparisons.

Use of the First WHO International Standard for anti-SARS-CoV-2 immunoglobulin clearly demonstrated that ND₅₀ GMTs measured with the same assay platform in different laboratories and at different throughputs are highly concordant despite pseudotyped virions using different spike proteins. However, comparisons across assay platforms are not straightforward, and the discordance of GMTs between LV pseudovirus and VSV pseudovirus neutralization even after calibration demonstrates that direct conversion from one to the other may require further calibration using multiple samples covering a broad range of neutralization potency.

Compliance with good clinical laboratory practices is required to ensure that assay results are as reliable as possible (71). Therefore, further assay optimization and subsequent validation addressing how a range of test conditions affect assay specificity, precision, linearity, accuracy, limit of detection, limit of quantitation, and robustness will be required before all or one of the methods evaluated in our study will be transferable between laboratories and suitable for even greater throughput in a 384-well format used for clinical trial testing (62, 72, 73).

SARS-CoV-2 is predicted to remain circulating in the global population for many years due to emerging new strains and incomplete vaccine delivery and uptake (74). Therefore, monitoring of acute and convalescent infection and the broad spectrum of immunity against SARS-CoV-2 both in natural infection and after vaccination will become a routine task for clinical microbiology/virology facilities. Selection of a SARS-CoV-2 neutralization assay and the ability to compare results obtained using different assays will remain a crucial issue.