Assay Harmonization Study To Measure Immune Response to SARS-CoV-2 Infection and Vaccines: a Serology Methods Study

ABSTRACT

The Coronavirus disease 2019 (COVID-19) pandemic presented the scientific community with an immediate need for accurate severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) serology assays, resulting in an expansion of assay development, some without following a rigorous quality control and validation, and with a wide range of performance characteristics. Vast amounts of data have been gathered on SARS-CoV-2 antibody response; however, performance and ability to compare the results have been challenging. This study seeks to analyze the reliability, sensitivity, specificity, and reproducibility of a set of widely used commercial, in-house, and neutralization serology assays, as well as provide evidence for the feasibility of using the World Health Organization (WHO) International Standard (IS) as a harmonization tool. This study also seeks to demonstrate that binding immunoassays may serve as a practical alternative for the serological study of large sample sets in lieu of expensive, complex, and less reproducible neutralization assays. In this study, commercial assays demonstrated the highest specificity, while in-house assays excelled in antibody sensitivity. As expected, neutralization assays demonstrated high levels of variability but overall good correlations with binding immunoassays, suggesting that binding may be reasonably accurate as well as practical for the study of SARS-CoV-2 serology. All three assay types performed well after WHO IS standardization. The results of this study demonstrate there are high performing serology assays available to the scientific community to rigorously dissect antibody responses to infection and vaccination.

IMPORTANCE Previous studies have shown significant variability in SARS-CoV-2 antibody serology assays, highlighting the need for evaluation and comparison of these assays using the same set of samples covering a wide range of antibody responses induced by infection or vaccination. This study demonstrated that there are high performing assays that can be used reliably to evaluate immune responses to SARS-CoV-2 in the context of infection and vaccination. This study also demonstrated the feasibility of harmonizing these assays against the International Standard and provided evidence that the binding immunoassays may have high enough correlation with the neutralization assays to serve as a practical proxy. These results represent an important step in standardizing and harmonizing the many different serological assays used to evaluate COVID-19 immune responses in the population.

INTRODUCTION

In 2019, the novel severe acute respiratory syndrome Coronavirus 2 (SARS-CoV-2) began to spread on a global level, triggering the Coronavirus disease 2019 (COVID-19) pandemic and causing 778.9 million confirmed cases and 6.6 million deaths as of December 2022 (1, 2). As the world scrambled to respond, detection of the presence of anti-SARS-CoV-2 antibodies became a critical need, resulting in rapid development of serological assays at an unprecedented rate (3, 4). Meanwhile, knowing that serological data were going to be crucial to getting a handle on the pandemic as quickly as possible, the United States (US) Food and Drug Administration (FDA) modified requirements for new assays to be granted Emergency Use Authorization (EUA) (2, 5). Today, serology assays continue to play an important role as the scientific community works toward effective preventions, treatments, and better understanding of the virus. Roles for serology include exploring which infected individuals develop strong antibody responses against the virus, as well as discerning whether all binding antibodies are associated with immunity, as opposed to only neutralizing antibodies, or whether individuals with high levels of anti-SARS-CoV-2 antibodies are protected against future infection (6). These assays also play crucial roles in all aspects of the battle against the public health crisis, including screening of convalescent donors for plasma donation for treatment, testing the immunogenicity of vaccines, determining immune responses to infection, and serosurveillance of the population (6). Given the crucial role of these assays and the rapid pace of assay development, there was significant need to verify the reliability of these tests (2).

In April of 2020, the FDA collaborated with the National Cancer Institute (NCI), Centers for Disease Control and Prevention (CDC), National Institute of Allergy and Infectious Diseases (NIAID), National Cancer Institute’s Frederick National Laboratory for Cancer Research (NCI-FNLCR), National Institutes of Health (NIH) Clinical Center, and Biomedical Advanced Research and Development Authority (BARDA) to establish an independent evaluation serology program that evaluated over 120 SARS-CoV-2 serology assays and their performance characteristics to inform decisions by FDA for EUA approval (2). Results from 91 of these evaluations were recently published, showing 27 of the 91 tested being authorized by the FDA for emergency use; 2 assays had their EUA revoked, while the remaining assays did not receive EUA authorization (2). These and many other published findings highlight the high variability in serological assay performance, presenting a significant issue when it comes to comparing results across assays or studies.

There are two major hurdles when comparing assay results across independent studies. The first is developing an assay reproducible, sensitive, and specific enough to generate consistent and reliable data across consecutive runs. The second is developing a reliable reagent against which independent assays can be harmonized, resulting in comparable data to support stronger conclusions. The issue of lack of assay standardization has been faced before, such as in Human Papillomavirus (HPV) vaccine serology, and use of an international standard was demonstrated to be critical for harmonization and comparison of results (7–10). Given this history, the World Health Organization (WHO) SARS-CoV-2 International Standard and US Serology Standards were created with the goal of providing a similar key to SARS-CoV-2 serology assay harmonization (11, 12).

The protection against infection afforded by the SARS-CoV-2 vaccines is believed to be mainly mediated by neutralizing antibody responses (13, 14). This belief is attributable theoretically to mechanisms established in previously developed prophylactic antiviral vaccines, experimentally to evidence of protection via passively transferred serum or purified immunoglobulin, and analytically to immune correlates of protection in clinical SARS-CoV-2 efficacy studies (13–15). Options for evaluating antibody responses include determining total antibody quantity via IgG-binding immunoassays (including antibodies with no effective viral neutralization) or measuring antibody neutralization in vitro via neutralization assays (8). Neutralization assays are most relevant to the believed main mechanism of immunity for anti-SARS-CoV-2 vaccines BNT162b2 and mRNA-1273 (16, 17). Also, as new SARS-CoV-2 variants continue to emerge, existing antibodies that bind the SARS-CoV-2 virus may no longer actually neutralize the variant to confer effective immunity, and therefore total antibody measures may not truly represent protection against latest variants (16, 18). Unfortunately, neutralization assays are time-consuming, labor-intensive, expensive, and have much higher coefficients of variability compared to binding immunoassays (8, 12). Therefore, there is a push to determine if total antibody quantity, when used appropriately, can serve as a proxy for antibody neutralization, facilitating faster, less expensive, and more scalable assessment of serological response.

Given the above observations regarding SARS-CoV-2 serological assay variability, and the hundreds of COVID-19 serology tests available, including more than 86 with FDA EUA approval, there are several fundamental questions that comparison studies have sought to address (19). Studies working to explore the reliability of different serology assays have found variable sensitivities and specificities (20–29). Some studies have illustrated variability by comparing measurements of seroprevalence in their sample population or percent positive samples found in confirmed infection or polymerase chain reaction (PCR) verified samples (30, 31). Finally, studies highlighting the current challenges of comparing different serology assay results have proposed a number of solutions, from harmonizing against the WHO International Standard (WHO IS) to comparing to an in-house reference standard (20, 22, 26, 28, 31, 32). This is the first study to directly compare 27 serology assays between laboratories, using the same set of samples with a wide range of antibody levels. In addition, this study seeks to analyze the feasibility of using the WHO IS as a global harmonization tool to facilitate comparison of results between assays.

A number of serology method comparison studies have also sought to answer the question of how closely traditional enzyme-linked immunosorbent assay (ELISA) or binding immunoassays correlate with neutralization results (33). Bonhomme et al. reported a weak correlation between antibody binding and antibody neutralization in convalescent-phase serum samples, but a good correlation between binding and neutralization in sera from vaccinated individuals (21). Other studies have reported variable correlation coefficients between binding and neutralizing antibody titers ranging from 0.127 to 0.916 (22, 27, 34, 35). This study has therefore evaluated the correlation between binding antibody assay and neutralization assay results across different labs, with the goal of addressing this issue.

RESULTS

Participating institutions, sample selection, and assay design.

In total, 17 institutions participated in the study, which accounted for 27 selected assays measuring antibody concentrations consisting of 18 antibody binding assays (including 2 multiplex (3-plex and 5-plex) assays) and 9 (7 live virus, 2 surrogate virus) assays measuring neutralization activity against SARS-CoV-2 (Table 1). To account for the antibody responses measured from the antigens included in the two multiplex assays, each antibody response from the individual antigens was analyzed independently, so the total number of Assay IDs assigned was 33. Participating laboratories were sent 3 frozen aliquots of the US SARS-CoV-2 Serology Standard, in addition to 3 frozen aliquots of a panel of 120 samples from uninfected, infected, vaccinated healthy volunteers, WHO IS, and monoclonal antibodies to SARS-CoV-2 (Table 2). The shipments to the participating institutions occurred between August 2021 and September 2021, and all the testing was completed and data sent back for analysis by April 2022. The vaccinated donors received the primary series of the Moderna, Pfizer, Johnson & Johnson, or AstraZeneca vaccine in December 2020 to April 2021. For each day of testing, the US Serology Standard was to be tested in triplicate, and the panel of 120 samples tested in duplicate. Participating laboratories were to follow in-house standard operating procedures when testing the samples. The 3 aliquots of sample or US Serology Standard were to be tested on 3 separate days to determine inter-day variability.

TABLE 1.

Summary of evaluated assays

Institution	Assay manufacturer	Assay type	Assay name	Antigen	Antibody	Quant/Semi/Qual
1	MSD	ECL based Multiplex	SARS-COV-2 MULTIPLEX (Spike, RBD, Nucleocapsid)	Multiplex Assay-Spike, RBD, Nucleocapsid	IgG	Quant
2	Roche	Double-antigen sandwich ECLIA	Anti-SARS-CoV-2 S	SARS-CoV-2 spike (S) protein receptor binding domain	Pan Ig	Semi
	Abbott	Chemiluminescent MIA	SARS-CoV-2 IgG	SARS-CoV-2 nucleocapsid protein	IgG	Qual
	NA	Surrogate Virus Neutralization	SARS-CoV-2 Neutralization Antibody Detection kit	SARS-CoV-2 RBD	Pan Ig	Qual
3	Siemens	Chemiluminescent assay	SARS-CoV-2 IgG	SARS-CoV-2 S1 RBD	IgG	Quant
4	Ortho Clinical Diagnostics	HRP-CIA	Anti-SARS-CoV-2 IgG Quantitative Test	SARS-CoV-2 spike S1 protein	IgG	Quant
	Ortho Clinical Diagnostics	HRP-CIA	Anti-SARS-CoV-2 Total Test	SARS-CoV-2 spike S1 protein	Pan Ig	Qual
5	NA	In-house Developed Multiplex assay	Luminex Based Multiplex Assay (Spike, RBD, S1, S2, Nucleocapsid)	Multiplex Assay-Spike, RBD, S1, S2, Nucleocapsid	Pan Ig	Semi
	NA	Neutralization Assay	SARS-CoV-2 Plaque reduction neutralization test (Live Virus PRNT)	NA	NA	Semi
6	NA	In-house Developed ELISA	Spike RBD ELISA	Spike RBD	Pan Ig	Quant
	NA	In-house Developed ELISA	Spike NTD ELISA	Spike NTD	Pan Ig	Qual
7	NA	In-house Developed ELISA	Spike RBD ELISA	Spike-RBD	IgG	Semi
8	NA	In-house Developed ELISA	Spike ELISA	Spike	IgG	Quant
	NA	Neutralization Assay	Live Virus Neutralization Assay	NA	NA	Semi
9	NA	In-house Developed ELISA	Spike ELISA	Spike	IgG	Semi
	Kantaro	Spike ELISA	Spike ELISA	Spike	IgG	Quant
	NA	Neutralization Assay	Live Virus Neutralization Assay	NA	NA	Semi
10	NA	In-house Developed Fluorescent Bead Assay	Flow Cytometry Spike Assay	Spike	IgG	Quant
	NA	Surrogate Neutralization Assay	Surrogate Neutralization Assay	Spike, RBD, Nucleocapsid	Total IgG	Semi
11	NA	In-house Developed ELISA	Spike IgG ELISA	SARS-CoV-2 Spike	IgG	Quant
	NA	In-house Developed ELISA	Nucleocapsid IgG ELISA	SARS-CoV-2 Nucleocapsid	IgG	Quant
12	NA	In-house Developed ELISA	RBD Assay	RBD	IgG	Semi
13	Roche	Double-antigen sandwich ECLIA	Anti-SARS-CoV-2	SARS-CoV-2 nucleocapsid protein	Pan Ig	Qual
14	NA	Neutralization Assay	Micro-Neutralization Assay	NA	NA	Semi
15	NA	Neutralization Assay	Live Virus Neutralization Assay (FRNT)	NA	NA	Semi
16	NA	Neutralization Assay	Live Virus Neutralization Assay	NA	NA	Semi
17	NA	Neutralization Assay	Pseudovirus Neutralization Assay	NA	NA	Quant

TABLE 2.

Summary of number of samples selected^a

Groups	N
Negativeb	30
Low positive-infected original strain	12
Intermediate positive-infected original strain	12
High positive-infected original strain	12
Infected variant	5
Low positive-vaccinated	16
Intermediate positive-vaccinated	15
High positive-vaccinated	15
Anti-SAR-CoV-2 RBD mAb	2
Diluted WHO international standard	1
US Serology Standard	1
Total	121

^aSamples were from unique individuals.

^bCollected from subjects who were unvaccinated and uninfected with SARS-CoV-2, and these samples were used for assay cutoff evaluations.

Assay specificity and sensitivity.

Assays were categorized for evaluation as commercial, laboratory developed (LDT) or in-house, or neutralization. Specificity and sensitivity were measured for each assay and graphed for visualization of these performance factors (Fig. 1). Acceptable assay performance cutoffs for specificity and sensitivity were ≥93% and ≥90%, respectively (2). Most commercial assays demonstrated high specificity (≥93%) and sensitivity (≥90%), while in-house assays performed well in sensitivity (≥90%) (Fig. 1). Neutralization assays showed variable results across both measures (Fig. 1).

FIG 1.

Sensitivity and specificity measurements of commercial, in-house, and neutralization serological assays. Graphical representation of the sensitivity and specificity of all tested assays. Specificity ≥93% and sensitivity ≥90% were considered acceptable. Data analysis was performed prior to harmonization.

To determine if variation in sensitivity was affected by sample characteristics, the vaccination results were analyzed against infection according to assay class (Table 3 to 5). In all three assay types, there did not appear to be any significant difference in sensitivity according to whether the sample was collected from an individual who had been infected versus an individual who had been vaccinated against SARS-CoV-2 (Table 3 to 5). However, when results were separated according to known sample antibody levels, lower levels of antibody resulted in higher variability in sensitivity, meaning that some assays struggled to detect at these levels (Table 6, Table S1 and S2). Of note, the vaccinated samples were excluded from analysis with the Nucleocapsid antibody detecting assays.

TABLE 3.

Identification of factors contributing to assay performance, commercial assay^a

		Commercial SARS-CoV-2 serological assay specificity and sensitivity
		Specificity (%)		Sensitivity (%)						Cohen's kappa agreement
Assay ID	Antibody	N	Overall	N	Infected	N	Vaccinated	N	Overall (%)	Overall
1	IgG	34	91.2	119	92.4	138	100.0	257	96.5	0.81
2	IgG	47	93.5	119	97.5	138	100.0	257	98.8	0.92
3	IgG	78	100.0	119	84.9	–	–	119	84.9	0.82
4	Pan Ig	180	100.0	246	98.0	276	100.0	522	99.0	0.98
5	IgG	180	100.0	246	91.9	–	–	246	91.9	0.91
6	IgG	180	100.0	246	82.1	276	73.9	522	77.8	0.64
7	IgG	180	100.0	246	85.8	276	87.0	522	86.4	0.77
8	Pan Ig	165	100.0	201	100.0	237	100.0	438	100.0	1.00
9	IgG	180	100.0	246	97.2	276	98.9	522	98.1	0.96
10	Pan Ig	180	100.0	246	95.9	–	–	246	95.9	0.95

^aFDA recommended performance thresholds (Pan Ig and IgG): Specificity ≥93%; Sensitivity ≥90%. Data highlighted in bold if below FDA recommended performance thresholds. The sensitivity measurements were not evaluated for vaccinated in anti-nucleocapsid assays as indicated by the dashes. Data analysis was performed prior to harmonization.

TABLE 4.

Identification of factors contributing to assay performance, in-house assay^a

		In-House developed SARS-CoV-2 serological assay specificity and sensitivity
		Specificity (%)		Sensitivity (%)						Cohen's kappa agreement
Assay ID	Antibody	N	Overall	N	Infected	N	Vaccinated	N	Overall (%)	Overall
11	Pan Ig	180	75.0	246	100.0	276	100.0	522	100.0	0.82
12	Pan Ig	180	72.8	246	99.2	276	100.0	522	99.6	0.79
13	Pan Ig	180	81.1	246	99.6	276	100.0	522	99.8	0.86
14	Pan Ig	180	39.4	246	95.5	276	89.5	522	92.3	0.36
15	Pan Ig	180	79.4	246	97.6	–	–	246	97.6	0.79
16	Pan Ig	180	90.6	246	99.2	276	98.6	522	98.9	0.91
17	Pan Ig	180	79.4	246	81.3	276	86.2	522	83.9	0.59
18	IgG	180	78.3	246	100.0	276	100.0	522	100.0	0.84
19	IgG	180	98.9	246	79.7	276	73.9	522	76.6	0.62
20	IgG	180	92.2	246	96.7	276	94.2	522	95.4	0.86
21	IgG	180	95.0	246	99.6	276	96.0	522	97.7	0.93
22	IgG	168	96.0	217	97.8	267	100.0	484	99.0	0.95
23	IgG	176	94.4	244	96.0	–	–	244	96.0	0.89
24	IgG	180	46.7	246	98.0	276	98.9	522	98.5	0.53

^aFDA recommended performance thresholds (Pan Ig and IgG): Specificity ≥93%; Sensitivity ≥90%. Data highlighted in bold if below FDA recommended performance thresholds. The sensitivity measurements were not evaluated for vaccinated in anti-nucleocapsid assays as indicated by the dashes. Data analysis was performed prior to harmonization.

TABLE 5.

Identification of factors contributing to assay performance, neutralization assay^a

	Neutralization SARS-CoV-2 serological assay specificity and sensitivity
	Specificity (%)		Sensitivity (%)						Cohen's kappa agreement
Assay ID	N	Overall	N	Infected	N	Vaccinated	N	Overall (%)	Overall
25	90	42.2	123	99.2	138	97.8	261	98.5	0.47
26	180	51.1	246	99.2	276	96.4	522	97.7	0.56
27	182	97.2	232	86.8	270	60.2	502	69.5	0.52
28	60	100.0	123	98.4	127	78.8	250	88.1	0.71
29	90	100.0	123	74.0	138	50.7	261	61.7	0.45
30	89	89.9	120	96.7	137	72.3	257	92.9	0.65
31	180	100.0	246	96.3	276	71.7	522	83.3	0.72
32	180	100.0	246	83.7	276	77.2	522	80.3	0.68
33	–	–	–	–	–	–	–	–	–

^aFDA recommended performance thresholds (Pan Ig and IgG): Specificity ≥93%; Sensitivity ≥90%. Data highlighted in bold if below FDA recommended performance thresholds. Sensitivity and specificity analysis were not evaluated for Assay ID 33 due to no reported cutoff value for the assay, and dashes were added to indicate the data was not analyzed. Data analysis was performed prior to harmonization.

TABLE 6.

Identification of factors contributing to assay performance, antibody levels^a

Neutralization SARS-CoV-2 serological assay specificity and sensitivity by antibody levels
	Specificity (%)		Sensitivity (%)																Cohen's kappa agreement
Assay ID	N	Overall	N	Infected (low)	N	Infected (mid)	N	Infected (high)	N	Infected (variant)	N	Vaccinated (low)	N	Vaccinated (mid)	N	Vaccinated (high)	N	Overall	Overall
25	90	42.2	36	100.0	36	97.2	36	100.0	15	100.0	48	93.8	45	100.0	45	100.0	261	98.5	0.47
26	180	51.1	72	98.6	72	98.6	72	100.0	30	100.0	96	91.7	90	97.8	90	100.0	522	97.7	0.56
27	182	97.2	70	74.1	70	91.6	62	100.0	30	82.8	96	17.8	91	67.5	83	99.3	502	69.5	0.52
28	60	100.0	36	94.4	36	100.0	36	100.0	15	100.0	38	42.4	44	93.3	45	100.0	250	88.1	0.71
29	90	100.0	36	38.9	36	86.1	36	100.0	15	66.7	48	2.1	45	53.3	45	100.0	261	61.7	0.45
30	89	89.9	35	88.6	35	100.0	35	100.0	15	100.0	48	33.3	44	86.3	45	100.0	257	92.9	0.65
31	180	100.0	72	90.3	72	100.0	72	100.0	30	93.3	96	19.8	90	98.9	90	100.0	522	83.3	0.72
32	180	100.0	72	63.9	72	94.4	72	100.0	30	66.7	96	34.4	90	100.0	90	100.0	522	80.3	0.68
33	–	–	–	–	–	–	–	–	–	–	–	–	–	–	–	–	–	–	–

^aFDA recommended performance thresholds (Pan Ig and IgG): Specificity ≥93%; Sensitivity ≥90%. Data highlighted in bold if below FDA recommended performance thresholds. Sensitivity and specificity analysis were not evaluated for Assay ID 33 due to no reported cutoff value for the assay, and dashes were added to indicate the data was not analyzed. Data analysis was performed prior to harmonization.

Assay reproducibility.

Assay reproducibility was measured for within-day replicates and across-day replicates (Table 7 to 9). Commercial assays showed excellent reproducibility both within-day and across-day, with all showing variability below 20% (Table 7). In-house assays had acceptable levels of variability, with about half showing an overall percent variability below 20% (Table 8). Neutralization assays showed high levels of variability in both within-day and across-day replicants (Table 9).

TABLE 7.

Reproducibility and intraclass correlation of commercial assays

Commercial SARS-CoV-2 serological assay reproducibility and intraclass correlation coefficient
Assay ID	Antibody	Within day (%)a	Across day (%)	Overall (%)	Intraclass correlation coefficient (ICC)
1	IgG	–	10.4	10.4	0.997
2	IgG	–	9.4	9.4	0.997
3	IgG	–	9.0	9.0	0.997
4	Pan Ig	17.2	6.4	18.3	0.992
5	IgG	1.5	2.5	2.9	0.998
6	IgG	4.1	6.5	7.7	0.998
7	IgG	2.4	5.4	6.0	0.999
8	Pan Ig	1.9	9.4	9.5	0.998
9	IgG	10.3	8.0	13.0	0.994
10	Pan Ig	1.0	2.7	2.9	1.000

^aDashes represent data that was not evaluated due to no within-day replicate testing. Data analysis was performed prior to harmonization.

Table 8.

Reproducibility and intraclass correlation of in-house assays^a

In-House developed SARS-CoV-2 serological assay reproducibility and intraclass correlation coefficient
Assay ID	Antibody	Within day (%)	Across day (%)	Overall (%)	Intraclass correlation coefficient (ICC)
11	Pan Ig	9.9	12.1	15.6	0.925
12	Pan Ig	8.7	10.8	13.9	0.896
13	Pan Ig	8.3	12.6	15.1	0.936
14	Pan Ig	12.3	15.6	19.9	0.954
15	Pan Ig	8.6	14.6	16.9	0.789
16	Pan Ig	6.6	22.5	23.4	0.964
17	Pan Ig	3.1	9.2	9.7	0.976
18	IgG	12.4	30.2	32.7	0.946
19	IgG	16.3	16.2	23.0	0.980
20	IgG	6.4	25.7	26.5	0.983
21	IgG	8.7	23.1	24.6	0.976
22	IgG	15.4	12.8	20.0	0.986
23	IgG	10.5	10.8	15.0	0.989
24	IgG	8.0	16.4	18.2	0.980

^aVariability above 20% is highlighted in bold. Data analysis was performed prior to harmonization.

TABLE 9.

Reproducibility and intraclass correlation of neutralization assays^a

Neutralization SARS-CoV-2 serological assay reproducibility and intraclass correlation coefficient
Assay ID	Within day (%)	Across day (%)	Overall (%)	Intraclass correlation coefficient (ICC)
25	–	35.3	35.3	0.877
26	41.4	46.2	62.0	0.745
27	53.8	80.7	97.0	0.555
28	–	48.8	48.8	0.901
29	–	27.5	27.5	0.957
30	26.1	36.1	44.5	0.936
31	15.3	24.7	29.1	0.965
32	1.8	4.4	4.7	0.989
33	–	24.2	24.3	0.943

^aVariability above 20% is highlighted in bold. Data analysis was performed prior to harmonization. Dashes represent data that was not evaluated due to no within day replicate testing.

Harmonization with the WHO International Standard.

The three assay classes were harmonized against the WHO IS, and their measures of the US serological standard compared against the calibrated value. The harmonized results of the commercial tests had high correlation with the calibrated antibody level value (Fig. 2A), as did many of the neutralization assays (Fig. 2C). The in-house assays had the highest variability when comparing the results against the US Serology Standard calibrated value (Fig. 2B).

FIG 2.

Commercial, in-house, and neutralization assay measurement of the US Serology Standard after harmonization with the WHO International Standard. Commercial (A), in-house (B), and neutralization (C) assays were harmonized with the WHO International Standard and then used to measure the US Serology Standard. These measurements were then compared to the calibrated value of the US Serological Standard as shown by the red dashed line in each graph.

Correlation between antibody level and neutralization assay results.

Commercial and in-house assays were compared against neutralization using Pearson’s Correlation (Fig. 3). Commercial assays demonstrated high correlation with neutralization (Fig. 3A), while the in-house assays had somewhat lower levels of correlation (Fig. 3B).

FIG 3.

Pearson correlation of commercial and in-house serological assays versus neutralization assays. The commercial (A) and in-house (B) assay measures were compared to the neutralization assay results to calculate the Pearson correlation.

DISCUSSION

The results of this study highlight the variability in performance of currently available anti-SARS-CoV-2 serological assays. Commercial assays were more accurate than in-house, and neutralization-based assays varied greatly in both sensitivity and specificity (Fig. 1). Most of the commercial assays presented with high specificity (≥93%), which is perhaps due to the high standards for validation in the commercial setting. The in-house lab-developed assays, on the other hand, excelled in sensitivity (with the majority reaching ≥90% sensitivity), while some also presented with high specificity (≥93%).

The neutralization assays had the most variability in both sensitivity and specificity, with 5/8 reaching a specificity above the 93% threshold, and 3/8 measuring above the 90% sensitivity threshold (Fig. 1). Note that only 8 of the 9 neutralization assays were evaluated for sensitivity and specificity because one did not have a predefined assay cutoff value, so seropositive and seronegative samples could not be determined. No neutralization assays measured above threshold for both parameters. Such variability is to be expected in neutralization assays, as these protocols are known to vary widely (8, 12, 36, 37). This variability provides further evidence for the potential utility of using binding immunoassays as a proxy for neutralization in high-throughput and/or clinical testing of the population for immune response against the SARS-CoV-2 virus.

To understand the source of the variability in the assays, particularly in the case of the neutralization, the results were separated by vaccination versus infection sample type. There was no significant difference between the two sample types for any of the three assay classes (Table 3 to 5). However, when the samples were separated according to antibody level, a relationship did emerge between the variability of the sensitivity of the assay and antibody level. Specifically, the variability in sensitivity of the assays was higher for samples that were known to have low levels of antibody (Table 6, Table S1 and S2). These results suggest that a number of the assays did not reliably detect low levels of antibody, contributing to the variability in performance in all three classes. These results are in accordance with other studies both within and outside SARS-CoV-2 research, which have reported lower sensitivity in samples with lower antibody levels (24, 30, 38).

The assays were also evaluated for within-day and across-day variability to estimate reproducibility. Reproducibility of commercial assays was excellent, with all having a percent variability less than 20% (Table 7). The in-house assays had greater variability, but still exhibited acceptable reproducibility, with about half having a variability lower than 20% (Table 8). Mirroring the variability seen in sensitivity and specificity, neutralization assays exhibited low reproducibility, with the majority having a variability higher than 20% (Table 9). As stated above, this high variability was expected, as neutralization assays are known to have low reproducibility and high coefficients of variability due to the fact that they are cell-based (8, 12, 36, 37).

An important question in the evaluation of serological assays is if different tests can be compared to each other despite using different parameters, measures, and reagents as well as being developed and used in different laboratories. Using a method previously developed and employed for the evaluation of HPV serological assays, all three classes were harmonized against the WHO IS and used to measure the US Serology Standard. This strategy was utilized with the goal of comparing the results of the different assays for the US Serology Standard against the known value of the standard, which had previously been calibrated against the WHO IS. The commercial assays performed well in harmonization, with only one, which measured pan IgG against the S1 protein, measuring far below the calibrated value of the standard (Fig. 2A). This deviation may be due to the S1 antigen used in the assay not being properly recognized by the antibodies in the US Serology Standard. The in-house developed assays had the highest variability upon harmonization, as there was more scatter compared to commercial assays (Fig. 2B). Several of the neutralization assays after harmonization were able to measure the US Serology Standard around the assigned value (Fig. 2C). These results suggest that harmonization against the WHO IS is a feasible method for the comparison of different serological assays across platforms. These results also provide evidence that such standards are applicable for the evaluation of serological assays across different viruses, as it has been shown to be effective in both anti-HPV and SARS-CoV-2 antigen serology tests.

Finally, the commercial and in-house assays were evaluated for correlation with neutralization. Neutralization assays are considered the best way to measure the actual biological activity of anti-SARS-CoV-2 antibodies, as they measure total immunoglobulins (IgM, IgA, and IgG) that exhibit in vitro neutralizing activity (16, 17). Neutralization is going to be especially important as new variants of interest continue to emerge. As the virus changes, existing antibodies that bind the SARS-CoV-2 virus may no longer actually neutralize the variant, and thus not confer effective immunity. Consequently, total antibody measures, such as those given by binding immunoassays, may not truly represent protection against the latest variants (16, 18). However, neutralization assays are impractical for clinical or population level use, as they do not lend themselves to multiplexing or high-throughput testing, are complex and time-consuming, and have low levels of sensitivity, specificity, and reproducibility (Table 9) (8, 12). Binding immunoassays such as the commercial and in-house developed assays are far more practical as they are fast, high-throughput, amenable to multiplexing, and often reproducible (Table 7 and 8) (8). However, these assays only detect one class of antibody at a time, recognize both conformation and nonconformational epitopes, and measure all antibodies that bind to the antigen in question, regardless of neutralization activity (8).

Given these caveats, it is necessary to determine if binding immunoassays can be reliably used to measure immunogenicity of vaccines in place of neutralization, which is technically a more direct measure of the mechanism at work. In this study, both the commercial and the in-house assays correlated well with neutralization, with commercial assays having slightly higher correlations with neutralization than the in-house assays (Fig. 3). These results are consistent with some previous studies, while other studies have reported much lower levels of correlation (22, 27, 34, 35). The lowest correlations were found in the samples with low antibody levels, suggesting correlation with neutralization was also negatively associated with the inability of the assays to detect lower levels of antibody (data not shown).

The COVID-19 pandemic brought high needs for testing and evaluation of the immunogenicity of viral infection and vaccination on the population level, and with them unprecedented rates of scientific development in the field of serology (2–4). Previous studies have shown significant variability in these assays, both highlighting the significant need for evaluation and validation and the requirement of a method to easily and quickly compare results (2, 8, 12). This study evaluated the performance and reproducibility of commercial and in-house developed immunoassays and neutralization assays. This study demonstrated that there are high performing assays that can be used reliably to evaluate immune responses to SARS-CoV-2 in the context of infection and vaccination. This study also demonstrated the feasibility of harmonizing these assays against the WHO IS and provided evidence that the binding immunoassays may have high enough correlation with neutralization to serve as a practical proxy. This study has potential for far reaching effects in public health and serosurveillance. These results provide key information regarding performance characteristics of a multitude of assays using a common set of samples from infected and vaccinated individuals evaluated side-by-side and represent an important step in standardizing and harmonizing the many different serological assays used to evaluate SARS-CoV-2 antibody responses in the population.

MATERIALS AND METHODS

Study design.

This serology comparison study was conceptualized as part of SeroNet and the Clinical and Translational Serology Task Force (CTTF) activities, a consortium of laboratories and institutions meeting and working together to share resources and ideas to pursue treatment and vaccine development for the amelioration of the COVID-19 pandemic. Laboratories interested in participating in this study were identified through a survey circulated within CTTF and SeroNet.

Human serum samples from naturally infected and vaccinated individuals, as well as the US SARS-CoV-2 Serology Standard were used in this study. Furthermore, two monoclonal antibodies were included in the panel of samples. Part of the samples used in this study were collected from Occupational Health Services (Frederick National Laboratory for Cancer Research, [FNLCR]), and the other samples were obtained from BEI Resources, NIAID, NIH, Arizona State University, and Boca Biolistics. The monoclonal antibodies were provided by National Institute of Standards and Technology through an agreement with Regeneron. All of the donor samples were collected under Institutional Review Board (IRB) approved protocols and informed consent. The FNLCR COVID Serology laboratory aliquoted the samples and stored the samples at −80°C. Frozen samples were provided to each center in single use aliquots. The comparison study involved testing 120 samples, and the US SARS-CoV-2 Serology Standard across 3 days. Participating centers included their internal daily-use standard, controls, or samples, and their own assay standard operating procedures (SOPs) in addition to the samples provided for the study. Since the supply of the WHO International Standard (NIBSC 20/136) (WHO IS) was limited, a diluted WHO IS was provided in the 120-sample set as a suitable alternative, for calibration purposes.

Study samples.

Serum samples from SARS-CoV-2 serology uninfected/unvaccinated individuals (n = 30), SARS-CoV-2 seropositive individuals with evidence of past SARS-CoV-2 infection, confirmed cases of SARS-CoV-2 infection with United Kingdom (UK) variant (B.1.1.7, B.1.351 or 20H/501Y.V2) samples as well as SARS-CoV-2 seropositive recipients of SARS-CoV-2 vaccines (n = 87), WHO IS (n = 1), US SARS-CoV-2 Serology Standard (n = 1), and monoclonal antibodies (n = 2) were selected and aliquoted in a manner to blind the recipient laboratories, so they would not be able to discern seronegative samples from seropositive samples.

All samples were categorized as seronegative, low, intermediate, or high responder as defined by preliminary evaluations at the FNL Serology Lab and antibody test results from CDC. The samples from SARS-CoV-2 uninfected/unvaccinated individuals were collected between July 2020 and January 2021, and these samples were selected based on a negative antibody response at both confirmatory laboratories (FNL Serology Lab and CDC). The categorization of the low, intermediate, and high antibody response groups was evaluated separately for the SARS-CoV-2 infected donors and vaccinated donors. As many of the vaccinated donor samples had strong antibody responses to vaccination, we contrived 34 vaccinated samples by diluting the serum from vaccinated individuals with seronegative serum or sera from other vaccinated donors to generate the three levels of antibody responses. Samples were sent to the participant laboratories for testing using the selected SARS-CoV-2 serology binding and viral neutralization assays (Table 2).

WHO International Standard (IS).

The first WHO International Standard for anti-SARS-CoV-2 (WHO IS) was established and released in December of 2020 and made available under code 20/136. The assigned unitage of the IS is 1,000 International Units (IU) neutralizing activity per milliliter or 1,000 binding antibody units BAU (binding antibody units) per milliliter for binding antibody assays when resuspended in 250 μL of water (11).

US SARS-CoV-2 Serology Standard.

The US SARS-CoV-2 Serology Standard is a pool of plasma samples containing IgM and IgG from four donors targeting SARS-CoV-2 spike and nucleocapsid proteins (4). The US Serology Standard was calibrated against the WHO IS, showing a neutralization value of 813 IU/mL and an antibody concentration of 246 BAU/mL and 764 BAU/mL, respectively, for the anti-SARS-CoV-2 Spike IgM and IgG, and 1037 BAU/mL, and 681 BAU/mL, respectively, for the Nucleocapsid IgM and IgG.

Sample data.

Basic characteristics of each individual and relevant vaccination information or infection status were captured. These data include, when appropriate, vaccine product, number of doses received, timing of additional doses, time between dose and blood draws. For infected patient samples, PCR confirmation or type of viral variant was recorded.

Immunoassays and laboratories.

From 17 institutions, 27 assays measuring antibody concentrations were selected. These consisted of 18 antibody binding assays, including 2 multiplex (3-plex and 5-plex) assays and 9 (7 live virus, 2 surrogate virus; Table 1) assays measuring their neutralization activity against SARS-CoV-2. Despite there only being 27 assays (2 multiplex), the total number of Assay IDs assigned was 33 to account for each antigen assessed individually. Each of these assays have been internally validated and were conducted in laboratories with experience in testing large numbers of samples.

Statistical analysis.

(i) Reproducibility. After log transformation, the intraclass correlation coefficient (ICC) and coefficient of variation (%CV) were calculated. ICC’s were calculated from fitting mixed models. Confidence intervals for all estimates were calculated by a bootstrap procedure. Analyses were performed separately for each assay and each antigen of interest. We also considered subgroup analyses focusing on each of the study groups.

(ii) Sensitivity and specificity. Sensitivity and specificity of each assay was calculated, and 95% confidence intervals were estimated. Analyses were done separately for each assay.

(iii) Correlation. For each binding and neutralization assay, log-transformed measurements were plotted, and Pearson correlation coefficients were determined. Confidence intervals for all estimates were calculated by a bootstrap procedure, and analyses were done separately for each assay. Furthermore, subgroup analyses were performed where appropriate.

(iv) Population differences. Two parameters (sensitivity and correlation) were calculated in each population separately (natural infection [dominant and variant] and vaccinated).

(v) Assay cutoffs. We defined the 95th percentile of the antibody levels in the baseline samples to be the threshold for positivity. Therefore, the specificity of the resulting test based on this threshold should be approximately 0.95.

(vi) Harmonization. The following formula was used to calculate the results calibrated to the WHO IS for each assay: (Geometric Mean of all reported values for a sample/Geometric Mean of all reported values for WHO IS sample) * 100 equals the harmonized result for the sample.