Comparison of 16 Serological SARS-CoV-2 Immunoassays in 16 Clinical Laboratories

Serological assays for severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) are needed to support clinical diagnosis and epidemiological investigations. Recently, assays for large-scale detection of total antibodies (Ab), immunoglobulin G (IgG), and IgM against SARS-CoV-2 antigens have been developed, but there are limited data on the diagnostic accuracy of these assays.

ABSTRACT

Serological assays for severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) are needed to support clinical diagnosis and epidemiological investigations. Recently, assays for large-scale detection of total antibodies (Ab), immunoglobulin G (IgG), and IgM against SARS-CoV-2 antigens have been developed, but there are limited data on the diagnostic accuracy of these assays. This study was a Danish national collaboration and evaluated 15 commercial and one in-house anti-SARS-CoV-2 assays in 16 laboratories. Sensitivity was evaluated using 150 samples from individuals with asymptomatic, mild, or moderate COVID-19, nonhospitalized or hospitalized, confirmed by nucleic acid amplification tests (NAAT); samples were collected 13 to 73 days either from symptom onset or from positive NAAT (patients without symptoms). Specificity and cross-reactivity were evaluated in samples collected prior to the SARS-CoV-2 epidemic from >586 blood donors and patients with autoimmune diseases, cytomegalovirus or Epstein-Barr virus infections, and acute viral infections. A specificity of ≥99% was achieved by all total-Ab and IgG assays except one, DiaSorin Liaison XL IgG (97.2%). Sensitivities in descending order were Wantai ELISA total Ab (96.7%), CUH-NOVO in-house ELISA total Ab (96.0%), Ortho Vitros total Ab (95.3%), YHLO iFlash IgG (94.0%), Ortho Vitros IgG (93.3%), Siemens Atellica total Ab (93.2%), Roche Elecsys total Ab (92.7%), Abbott Architect IgG (90.0%), Abbott Alinity IgG (median 88.0%), DiaSorin Liaison XL IgG (median 84.6%), Siemens Vista total Ab (81.0%), Euroimmun/ELISA IgG (78.0%), and Snibe Maglumi IgG (median 78.0%). However, confidence intervals overlapped for several assays. The IgM results were variable, with the Wantai IgM ELISA showing the highest sensitivity (82.7%) and specificity (99%). The rate of seropositivity increased with time from symptom onset and symptom severity.

INTRODUCTION

In late December 2019, the World Health Organization (WHO) was notified of a cluster of cases of pneumonia in Wuhan City, China. The virus responsible was isolated in the first week of January 2020, and its genome was shared a week later. Phylogenetic analysis showed that it was a novel coronavirus, designated initially as 2019 novel coronavirus (2019-nCoV) and later as severe acute respiratory syndrome coronavirus 2 (SARS‐CoV‐2). SARS‐CoV‐2 quickly spread worldwide, and the WHO declared coronavirus disease 2019 (COVID-19) a pandemic on 11 March 2020 (1).

In the following months, several hundred assays for detecting SARS-CoV-2 emerged. Different versions of nucleic acid amplification tests (NAATs) for naso-/oropharyngeal swabs or washes and lower respiratory tract specimens are essential in diagnosis of COVID-19 (2). However, assays for detecting antibodies produced as part of the humoral immune response to SARS-CoV-2 infection have emerged (3). These assays show that 1 week after the first symptoms, 30% of patients with COVID-19 have seroconverted, increasing to 70% after the second week and to above 90% by the third week (4). Accordingly, serological assays measuring total antibodies (Ab), immunoglobulin G (IgG), or IgM against antigens of SARS‐CoV‐2 have been used for supporting a diagnosis of COVID-19, for monitoring the epidemic, and for screening recovered COVID-19 patients for use in convalescent plasma therapy (5). Currently, the numerous serological assays have been validated on a limited number of samples and have at best been approved for emergency use after only a few days of evaluation. Several serological assays, especially the lateral-flow point-of-care tests, have a suboptimal performance with a low sensitivity and are not recommended for diagnostic use or even for population monitoring (6–8). Recently, several manufacturers of larger platforms have released serological assays useful for mass testing, but few studies have compared these assays directly (9). This comparison is needed for the commutability of the test results and the scientific data. Here, we present a national evaluation of 16 serological SARS-CoV-2 immunoassays across 16 laboratories in Denmark.

MATERIALS AND METHODS

Case panel samples for determination of clinical sensitivity.

The case panel samples tested in all assays (n = 150) were obtained from convalescent patients in the Capital Region of Denmark with a confirmed SARS-CoV-2 NAAT result that were identified in the Danish Microbiology Database from February 2020 to April 2020 (10). A total of 3,692 individuals were contacted via public secure mail and 639 persons responded. Serum samples and EDTA samples were obtained from respondents from 3 to 11 May 2020. For this study, only the first 150 consecutively collected serum samples from 3 May were chosen without any further selection and sent to all participating laboratories. Epidemiologic and clinical data were self-reported in an electronic questionnaire completed on the day of blood sampling.

Archived samples for determination of clinical specificity.

Archived plasma samples from regional pre-COVID-19 blood donations drawn during the influenza seasons of 2017–2018 and 2018–2019 were tested. The numbers of tested samples were >586 for the total-Ab and IgG assays and >400 for the IgM assays. Different sample sets and sample sizes were used across regions, with minor overlap in some cases. The specificities were calculated by combining the data from all sites that validated the same assay.

Archived samples for determination of cross-reactivity.

For all assays, cross-reactivity was investigated by testing samples from patients with unspecified autoimmune diseases (n = 10 to 131). Due to challenges with available amounts of sample material, 10 samples were pooled and tested across all assays. The nonpooled samples were tested in selected assays. Additionally, for all assays, archived local samples from patients with acute infections of cytomegalovirus (CMV) or Epstein-Barr virus (EBV) or other acute viral respiratory infections (respiratory syncytial virus, influenza A and B viruses, and adenovirus) based on positive IgM serology were tested (n = 10 to 37). Different sample sets were used across assays and laboratories. All samples were obtained prior to January 2020, before the first COVID-19 case in Denmark.

Immunoassay platforms.

The diagnostic accuracies of commercial immunoassays for the detection of anti-SARS-CoV-2 total Ab, anti-SARS-CoV-2 IgG, and anti-SARS-CoV-2 IgM were tested on the appropriate platforms by experienced laboratory technicians following the manufacturers’ protocols with the cutoff values suggested by the manufacturers (Table 1).

TABLE 1.

Serological immunoassays tested^a

Manufacturer	Laboratory no.	Analyzer	Catalog no.	Lot no.	Manufacturer’s cutoff	Antigen	Ab type	Method
Roche Diagnostics, Mannheim, Germany	5+6	Elecsys	09175431190	495040, 490258	Negative, <1.0 COI; positive, ≥1.0 COI	rN	Total	Sandwich CLIA
Siemens Healthcare, Tarrytown, NY, USA	4	Atellica IM	11206711	C2T01, C2T02	Negative, <1.0 COI; positive, ≥1.0 COI	rS1 RBD	Total	Sandwich CLIA
7	Dimension Vista 500	K7414	20148BA	Negative, <1000 QU; positive, ≥1,000 QU	rS1 RBD	Total	Sandwich CLIA
Abbott, Abbott Park, IL, USA	10+11+12	Alinity	06R90	17065FN00, 17069FN00	Negative, <1.4 (S/C); positive, ≥1.4 (S/C)	rN	IgG	Sandwich CMIA
9	Architect	R686	16253FN00	Negative, <1.4 (S/C); positive, ≥1.4 (S/C)	rN	IgG	Sandwich CMIA
Snibe, Shenzhen, China	4+13+14	Maglumi 4000+	130219015M	272200501	Negative, <1.0; positive, ≥1.0	r2019-nCoV, nonspecified	IgG	Indirect CLIA
4+13+14	Maglumi 800	130219016M	271200401	Negative, <1.0; positive, ≥1.0	r2019-nCoV, nonspecified	IgM	Capture CLIA
Ortho Clinical Diagnostics, Pencoed, UK	3	Vitros 3600	619 9922	0012, 0021, 0151	Negative, <1.0 (S/CO); positive, ≥1.0 (S/CO)	rS1 RBD	Total	Sandwich CLIA
3	Vitros 3600	619 9919	0100	Negative, <1.0 (S/CO); positive, ≥1.0 (S/CO)	rS1 RBD	IgG	Sandwich CLIA
Wantai, Beijing, China	1	ELISA	WS-1096	NCOA20200301, NCOA20200401	Negative, <1.1 (S/CO); positive, ≥1.1 (S/CO)	rS RBD	Total	Double antigen sandwich ELISA
10	ELISA	WS-1196	NCOM20200301	Negative, <1.1 (S/CO); positive, ≥1.1 (S/CO)	rS RBD	IgM	Two-step solid-phase antibody capture ELISA
DiaSorin, Saluggia, Italy	13+14+ 15+16	Liaison XL	311450	354009, 354010	Negative, <12 AU/ml; equivocal, 12–15 AU/ml; positive, ≥15 AU/ml	rS1, rS2 RBD	IgG	Indirect CLIA
Euroimmun, Lübeck, Germany	9+10	ELISA	EI 2606-9601 G	E200408AO, E200420AW, E200423AF, E200317BP, E200416AE	Negative, <0.8; borderline, ≥0.8 to <1.1; positive, ≥1.1	rS1 RBD	IgG	Solid phase, Enzyme immunoassay
Copenhagen University Hospital and Novo Nordisk A/S, Denmark	2	ELISA	In house	Negative, <9.4 (S/CON); positive ≥9.4 (S/CON)	rS RBD	Total	Double antigen sandwich ELISA
YHLO Biotechnology, Shenzhen, China	8	iFlash 1800	20200301, 20200212	Negative, <10 AU/ml; positive, ≥10 AU/ml	rS	IgG	Indirect CLIA
8	iFlash 1800	20200301, 20200212	Negative, <8 AU/ml; positive, ≥8 AU/ml	rS	IgM	Indirect CLIA

^aCOI, cutoff index; S/C, stored-calibrator index; S/CO, sample-to-cutoff ratio; QU, qualitative units; S/CON, sample-to-control ratio; CLIA, chemiluminescence immunoassay; FIA, fluorescence immunoassay; CMIA, chemiluminescence microparticle immunoassay; r, recombinant; S, spike antigen; RBD, receptor binding domain; N, nucleocapsid antigen; IgG, immunoglobulin G; IgM, immunoglobulin M.

ELISA.

The commercial enzyme-linked immunosorbent assays (ELISAs) for anti-SARS-CoV-2 total-Ab, IgG and IgM detection were performed on open-platform analyzers or manually according to the manufacturers’ instructions. The Euroimmun SARS-CoV-2 IgG assay was performed on an Analyzer I (Euroimmun AG, Lübeck, Germany), Quanta-Lyser 160 (INOVA Diagnostics, San Diego, CA, USA), or Evolis (Bio-Rad, Hercules, CA, USA). The Wantai SARS CoV-2 total-Ab and IgM assays were performed manually and measured using a Tecan Sunrise ELISA reader (Männedorf, Switzerland) at 450 nm with reference at 620 nm.

One in-house ELISA detecting total Ab was tested in this evaluation. Briefly, the CUH-NOVO SARS-CoV-2 total-Ab ELISA (a noncommercial assay produced in a collaboration between Copenhagen University Hospital and Novo Nordisk A/S, Denmark) is based on a recombinant receptor-binding domain (RBD) of the SARS-CoV-2 spike protein used for both coating and detection (11). Briefly, the samples were diluted 1:100 in phosphate-buffered saline with Tween 20 (PBS-T) in 96-well plates coated with RBD. Total Ab was detected using horseradish peroxidase (HRP)-conjugated streptavidin diluted in PBS-T mixed with biotin-labeled RBD. TMB One was used as a substrate. The reaction was stopped with 0.3 M H₂SO₄, and the optical density of the samples was measured at 450 to 620 nm. The CUH-NOVO SARS-CoV-2 total-Ab ELISA used a semiautomated setup, and the results were based on signal-to-noise ratios between the samples of interest and the negative quality control. The cutoff value was calculated based on receiver operating characteristic (ROC) analysis by prioritizing the specificity.

The manufacturers of the Euroimmun ELISA and the DiaSorin Liaison XL assay have defined gray zone/borderline results. As high specificity was prioritized at the cost of some sensitivity, borderline results were interpreted as negative (see Table S3 in the supplemental material).

Some assays were evaluated in more than one laboratory.

Statistics.

Data handling, graphics, and statistics were performed using the R statistical software (12). The parameters of diagnostic accuracy and the plots were determined using the mada package (13). For calculation of the 95% confidence intervals for the sensitivity and specificity, the default “Wilson” option was chosen. For plotting of bivariate confidence regions in the ROC space, a continuity correction of 0.5 was applied, as there were cells with zero counts.

Performance criteria.

We defined acceptance criteria for the diagnostic accuracy of the assays depending on immunoglobulin type and intended use.

Ethics statement.

The study of samples from patients with former SARS-CoV-2 infection for validation of serological SARS-CoV-2 assays was approved by the Regional Committee on Health Research Ethics for the Capital Region of Denmark (H-20028627) and was conducted in accordance with this approval. All blood donors are routinely asked at every blood donation for consent for future use of archived samples in the validation of new methods and assay investigations as quality control projects.

RESULTS

Of 150 patient samples, epidemiologic and clinical data were available for 149 patients; for the patient characteristics, refer to Table 2. Most patients were categorized with clinically mild to moderate symptoms (n = 112), and only 31 patients had been admitted to hospital. Time from symptom onset (TSO) was >21 days for 120 case panel samples, of which 79 were collected >6 weeks after symptom onset. The shortest TSO was 13 days, and the TSO was unknown for 9 patients.

TABLE 2.

Case panel patient characteristics

Characteristica	Value (%)b
Sex
Male	52 (34.9)
Female	97 (65.1)
Median age (IQR) (yr)	54 (43–64)
Age range (yr)	18–83
TSO (days)
0–7	0 (0.0)
>7–14	7 (4.7)
>14–21	13 (8.7)
>21–42	49 (32.7)
>42	71 (47.3)
NA	9 (6.0)
Time from positive SARS–CoV–2 PCR (days)
0–7	1 (0.7)
>7–14	15 (10.0)
>14–21	22 (14.6)
>21–42	90 (60.0)
>42	21 (47.3)
Symptom severity
No symptoms	6 (4.0)
Mild (at home, well)	37 (24.8)
Moderate (home, bedridden)	75 (50.3)
Severe (hospitalized)	2 (1.3)
Critical (assisted ventilation)	29 (19.5)
Total	149 (100)

^aIQR, interquartile range; NA, not available.

^bValues are numbers of patients unless otherwise specified.

Data on the detection of anti-SARS-CoV-2 antibodies by each assay.

The results of the tests, i.e., true positives (TP), false negatives (FN), false positives (FP), and true negatives (TN), as well as the calculated sensitivity and specificity of each assay, are presented in Table 3 in descending order of sensitivity.

TABLE 3.

Anti-SARS-CoV-2 antibody assays with results^a

Manufacturer/platform	Assay	Laboratoryb	No. of:				No. of FP cross-reactive samples/total		% sensitivity (95% CI)	% specificity (95% CI)	% sensitivity in 120 samples with TSO >21 daysc
			TP	FN	FP	TN
							Auto	EBV, CMV
Total-Ab assays
Wantai ELISA	1	1+10	145	5	3	656	0/65	0/37	96.7 (92.4–98.6)	99.5 (98.7–99.8)	97.6
In-house CUH-NOVO ELISA	2	2	144	6	3	617	0/50	0/25	96.0 (91.5–98.2)	99.5 (98.6–99.8)	95.9
Ortho CD Vitros	3	3	143	7	0	605	0/50	0/20	95.3 (90.7–97.7)	100.0 (99.4–100)	95.9
Siemens Atellica	4	4	138	10	3	593	0/50	0/25	93.2 (87.9–96.7)	99.5 (98.5–99.8)	96.7
Roche Elecsys	5	5	139	11	2	216	ND	ND	92.7 (87.3–95.9)	99.1d (96.7–99.7)	95.9
		6	138	12	0	610	0/60	0/25	92.0 (86.5–95.4)	100.0 (99.4–100)	95.9
Siemens Vista	6	7	119	28	0	596	0/10	0/25	81.0 (73.7–87.0)	100.0 (99.4–100)	81
IgG assays
YHLO iFlash	7	8	141	9	4	582	1/50	0/25	94.0 (89.0–96.8)	99.3 (98.3–99.7)	95.9
Ortho CD Vitros	8	3	140	10	0	600	0/50	0/25	93.3 (88.2–96.3)	100.0 (99.4–100)	95.9
Abbott Architect	9	9	135	15	3	600	0/25	1/32	90.0 (84.2–93.8)	99.5 (98.5–99.8)	93.5
Abbott Alinity	10	10	134	16	4	596	0/50	0/25	89.3 (83.3–93.8)	99.3 (98.3–99.7)	93.5
		11	132	18			0/50	ND	88.0 (81.8–92.3)		91.9
		12	132	18			0/53	ND	88.0 (81.8–92.3)		91.9
Euroimmun ELISA	11	9+10	117	33	5	594	0/50	0/35	78.0 (70.7–83.9)	99.2 (98.1–99.6)	82.9
Snibe Maglumi	12	13	116	32	9	1,164	0/50	0/10	78.4 (71.1–84.2)	99.2 (98.5–99.6)	82.8
		14	117	33			0/50	ND	78.0 (70.5–84.4)		82.9
		4	113	37			ND	ND	75.3 (67.6–82.0)		81.0
DiaSorin Liaison XL	13	14	128	22	39	1,349	1/60	0/25	85.3 (78.8–90.1)	97.2 (96.2–97.9)	89.4
		15	127	23			1/60	0/25	84.7 (77.9–90.0)		88.6
		13	125	23			2/50	0/10	84.5 (77.6–89.9)		87.7
		16	123	27			1/50	2/25	82.0 (74.9–87.8)		87.0
IgM assays
Wantai ELISA	14	10	124	26	4	396	0/53	0/25	82.7 (75.8–87.9)	99.0 (97.5–99.6)
YHLO iFlash	15	8	63	87	2	583	0/50	2/25	42.0 (34.4–50.0)	99.7 (98.8–99.9)
Snibe Maglumi	16	14	63	87	44	1,140	1/50	ND	42 (34.4–50.0)	96.3 (95.0–97.3)
		13	45	103			0/50	0/10	30.4 (23.1–38.5)
		4	39	109			ND	ND	26.4 (19.5–34.2)

^aIgG, Immunoglobulin G; IgM, Immunoglobulin M; total-Ab, total antibodies; TP, true positive; FN, false negative; FP, false positive; TN, true negative; Auto, pre-COVID-19 samples from patients with autoimmune diseases; EBV and CMV, preCOVID-19 samples from patients with acute Epstein-Barr virus or cytomegalovirus or other acute viral infections; ND, not done.

^bThe key to each laboratory is presented in Appendix S1 (Table S1) in the supplemental material.

^cThe mean size of the 95% confidence intervals for the sensitivities of the samples collected 21 days or later after symptom onset is 8% for total-Ab assays and 11% for IgG assays.

^dPatient samples from hospitalized patients from before January 2020.

All total-Ab assays performed with high specificities (≥99%). Two total-Ab ELISAs and the Ortho Vitros total-Ab assay performed with sensitivities of ≥95%, while the Siemens Atellica and Roche Elecsys assays performed with sensitivities of ≥92%. One assay, the Siemens Vista assay, performed with a sensitivity of only 81%.

Of the IgG assays, all but the DiaSorin Liaison XL IgG assay performed with specificities of ≥99%. Three assays (the YHLO iFlash IgG, the Ortho Clinical Diagnostics (Ortho CD) Vitros IgG, and the Abbott Architect IgG assays) showed sensitivities of ≥90%.

The sensitivity improved in all total-Ab and IgG assays if the analyses were restricted to samples collected >21 days after symptom onset (Table 3).

Regarding the IgM assays, the Wantai IgM ELISA demonstrated a higher sensitivity than the other IgM assays, with a specificity of 99.0% and no cross-reactivity (Fig. 1 and Table 3). The sensitivity of the YHLO iFlash IgM assay was 42%, and cross-reactivity was detected in two of 25 samples from pre-COVID-19 patients with either acute CMV or EBV infections, whereas the Snibe Maglumi IgM assay performed with a specificity of 96.3% and a sensitivity of 26.4 to 42% (Fig. 1).

FIG 1.

Summary ROC plot of sensitivity and false-positive rate with elliptic 95% bivariate confidence regions corresponding to the data in Table 3 for assays with total Ig, IgG, and IgM, respectively. For the IgG assays where data were available from more than one laboratory, the median result was chosen for the COVID-19 cases, and for the prepandemic blood donors, the total of all the samples was used, as these were from different persons. The vertical broken line at a false-positive rate of 0.01 corresponds to a 99% specificity. The y axis for IgM has a different scale, from 20% sensitivity instead of 70%.

Quantitative ROC analysis is provided in Appendix S2 in the supplemental material. The total-Ab assays all had discriminatory ability, with areas under the curves (AUC) of ≥97%, whereas more variation was seen within the IgG assays, with AUCs ranging from 91.9% to 99.3%. The IgM assays had a larger variation in AUC, ranging from 75.7% to 98%.

Pairwise comparison of identical samples from 150 COVID-19 patients showed a high degree of variation between assays but nearly identical results when the same assay was applied in different laboratories. The details of this pairwise comparison are provided in Appendix S3 in the supplemental material.

Data on the detection of anti-SARS-CoV-2 antibodies according to TSO and disease severity.

The largest variation in true-positive samples between the assays for each immunoglobulin (Ig) type category was shown among samples with a TSO of ≤21 days (Fig. 2). In addition, increasing rates of seropositive results were found with the severity of symptoms in all assays (Fig. 2).

FIG 2.

Antibody development for total-Ab (IgT), IgG, and IgM as a function of days from symptom onset in 3-week periods and severity of symptoms. Assays are color coded. Multiple lines with the same color appear if the same assay was performed in different laboratories, to show the (small) interlaboratory variation.

In four samples SARS-CoV-2 antibodies were not detected by any of the assays. Of these, three patients had mild clinical symptoms and one patient had moderate symptoms, with the TSO varying between 17 and 73 days (data not shown).

DISCUSSION

In the United States, the Food and Drug Administration requires a minimum sensitivity of 90% and a specificity of 95% for emergency use authorization of serologic anti-SARS-CoV-2 assays (14). In the United Kingdom, the Medicines and Healthcare Products Regulatory Agency has defined a “target product profile” for enzyme immunoassays detecting antibodies against SARS-CoV-2 (15), defining an acceptable sensitivity and specificity of ≥98% for anti-SARS-CoV-2-immunoassays among patients with a history of SARS-CoV-2 ≥20 days after symptom onset.

In a low-seroprevalence setting, the specificity of the test is the most important concern and must be high. For this reason, irrespective of specific clinical indication for the use of the anti-SARS-CoV-2 total-Ab and anti-SARS-CoV-2 IgG assays, we defined the specificity acceptance criterion as ≥99%. For the SARS-CoV-2 total-Ab assays, the defined acceptable sensitivity for diagnostic use was ≥92%, with an optimal sensitivity of ≥95%. For the anti-SARS-CoV-2 IgG assays, the defined acceptable sensitivity was ≥90%. For epidemiologic surveys, the acceptable sensitivity was defined as ≥80% for both the anti-SARS-CoV-2 total-Ab and the anti-SARS-CoV-2 IgG assays, as statistical adjustments for low sensitivity can be performed. No acceptance criteria for the diagnostic accuracy of the anti-SARS-CoV-2 IgM assays were defined, as most of the samples in the case panel were collected >21 days after symptom onset.

This study demonstrated that diagnostic accuracy was higher in the group of the SARS-CoV-2 total-Ab assays than the group of the SARS-CoV-2 IgG assays. All total-Ab assays but one exhibited acceptable performance for diagnostic use with regard to both specificity and sensitivity. The SARS-CoV-2 IgG assays demonstrated a larger variation in sensitivities, and only three assays showed acceptable sensitivity for diagnostic use, whereas one assay did not meet the defined acceptance criteria for specificity. All total-Ab and IgG assays showed higher seropositive rates in samples from patients with a TSO of >21 days, and seropositive rates increased with symptom severity in all assays across all Ig types.

Total-Ab assays.

Nominally, the accuracy was superior in the Wantai total-Ab assay, the in-house CUH-NOVO total-Ab ELISA, and the Ortho CD Vitros total-Ab assay, with optimal sensitivity for diagnostic use according to our criteria. The Roche Elecsys total-Ab and the Siemens Atellica total-Ab assays also performed with acceptable sensitivity for diagnostic use, and the confidence intervals for sensitivity and specificity overlapped the predefined acceptance criteria for optimal performance for the mentioned assays. The true values for sensitivity and specificity might therefore fulfill the defined criteria for optimal performance. The poor sensitivity of the Siemens Vista assay was seemingly due to the manufacturer’s setting with a suboptimal assigned cutoff value, as the ROC analysis showed an AUC similar to those of the other assays (see Fig. S2). Thus, adjusting the cutoff used in the Siemens Vista assay appears relevant for improving diagnostic performance. However, for this study, we used the cutoffs specified by the manufacturers.

IgG assays.

Three assays—YHLO iFlash IgG, Ortho CD Vitros IgG, and Abbott Architect IgG—showed acceptable sensitivity for diagnostic use per our predefined acceptance criteria. The confidence intervals for sensitivity in the Abbott Alinity IgG assay included the predefined cutoffs for acceptable performance, and it is possible that the true sensitivity of the assay is acceptable. Regarding specificity, only the DiaSorin Liaison XL IgG assay did not meet the defined criteria.

IgM assays.

More than half of the samples in the case panel had a TSO of >6 weeks and were not optimal for the evaluation of the sensitivity of the IgM assays. However, among the IgM assays, the Wantai IgM assay stood out with a relatively high sensitivity and specificity.

Antigen specificity.

In three of the evaluated assays, a recombinant nucleocapsid antigen (rN) is used in the immunoassay, while in eight assays, a recombinant spike antigen (rS) of the RBD is used; two assays did not specify the protein(s) used as the capturing antigen in the assay, and two assays (the YHLO IgG and IgM) use both rN and rS. Our study does not suggest that the chosen antigen (N versus S RBD) affects the assay performance per se. Instead, the differences in performance seem to be associated with overall assay design rather than the choice of antigen. This parallels the observations made by Haveri et al., who demonstrated the appearance of neutralizing antibodies against both N and S proteins simultaneously (16).

Cross-reactivity.

A potential limitation for the use of immunoassays can be interference due to cross-reactivity in individuals with autoimmune disease, infections with other respiratory viruses, or acute infections with EBV or CMV. We tested this by using samples from patients with autoimmune diseases, samples from pre-COVID-19 patients with viral infections, and pre-COVID-19 samples from blood donors who very likely had been exposed to various respiratory viruses, including non-SARS-CoV-2 coronaviruses. This did not seem to be an issue in most assays, except for the DiaSorin Liaison XL IgG and YHLO iFlash IgM assays, which showed interference in some samples.

In our study, the sensitivities calculated from the case samples with a known TSO of >21 days did not reach 98%, as defined by the UK authorities, in any assay evaluated (Table 3). However, we prioritized high diagnostic specificity (≥99%) as our main criterion, since Denmark has a low anti-SARS-CoV-2 seroprevalence so far. For example, in the Danish population with a SARS-CoV-2 Ab prevalence of only 1.9% (April and May 2020) (17), an assay specificity of 97.2% (DiaSorin Liaison XL) would lead to a low positive predictive value (PPV) of 34%, while a test with a higher specificity of 99.5% (Abbott Architect) would yield a PPV of 76%. The consequences of a low specificity are less pronounced in a higher-prevalence setting; for example, with an antibody prevalence of 22%, as reported in parts of Iran (18), a low-specificity test like the DiaSorin assay will have a PPV of 89% and a more specific test such as the Abbott Architect a PPV of 98%. A recent evaluation of the DiaSorin Liaison XL assay did not include a large specificity panel of donor samples, but it also showed cross-reactivity in some samples from patients with rheumatoid factors or positivity for antinuclear antibodies, substantiating unspecific reactions in this assay (19). Interestingly, many manufacturers use a TSO of ≥14 days, in contrast to >21 days, as a cutoff for optimal sensitivity, indicating a need for international consensus on which TSO to test for optimal sensitivity (20).

A similar national validation study (21), with a large sample size, was performed in the United Kingdom; that study compared five immunoassays, of which four (DiaSorin Liaison XL IgG, Abbott Architect IgG, Roche Elecsys, and Siemens Atellica) were included in the present study. Generally, the study from the UK finds a higher sensitivity and specificity for all assays, including DiaSorin Liaison XL IgG, which (nearly) met the 98% sensitivity and specificity required by the UK authorities (15). This difference may be explained by the fact that the UK study included samples obtained at least 20 days after symptom onset. Based on our results, it is probably not realistic to expect to achieve a 98% sensitivity when a large proportion of milder cases are included in the cohort.

Generally, serological testing had a low sensitivity when carried out less than 3 weeks from symptom onset and in patients who were asymptomatic or had mild disease at home but were not bedridden. If patients had been bedridden or hospitalized, the sensitivity of serological testing of samples from convalescent patients was >90% for most of the total-Ab or IgG assays included in the study (Fig. 2). The large difference in sensitivity according to disease severity was not explained by the difference in time of testing, as two large groups of nonhospitalized symptomatic patients had similar median TSO near 40 days and a range of distribution from symptom onset to blood sampling (see Appendix S1 and Fig. S1).

Nonreactive cases.

We found four cases without detected anti-SARS-CoV-2 antibodies in any assay. This finding could possibly be explained by early-stage infection, mild disease, transient antibody response only, antibodies not produced or produced at nondetectable levels, late or slow antibody development, or false-positive NAATs.

Early-stage infection was not the case in these four patients, who had a TSO of 17 to 73 days, whereas the largest variation in sensitivity performance between the anti-SARS-CoV-2 assays was seen in the samples with a TSO of ≤21 days. The TSO among the COVID-19 case samples indeed determined the absolute sensitivity values obtained, as the median seroconversion time is reportedly 11 days (interquartile range of 7.3 to 14.0 days) after onset of symptoms (22–25).

Of the four patients, three had mild disease and one moderate disease. A combination of mild COVID-19 symptoms and the collection of blood samples in the late convalescent stage might explain the nondetectable antibodies. As we showed, most anti-SARS-CoV-2 total-Ab and IgG assays had a higher rate of seropositivity among hospitalized patients than nonhospitalized patients. Previous studies have also shown that anti-SARS-CoV-2 titers correlate with the severity of COVID-19 among hospitalized patients (26–28) and that there is a time-dependent decline in antibody titers for anti-SARS-CoV-2 Ab in general, including neutralizing antibodies (17, 29).

The false-positive rate for NAATs is estimated to be between 0.8 and 4% in the United Kingdom (30), which could explain the four antibody-negative patients among the 150 NAAT-positive patients in the panel.

Our data provide some interesting preliminary observations regarding the antibody response in general. First, most individuals (approximately 97%) seem to develop some degree of antibody response. Second, this response seems to peak in samples collected approximately 3 weeks after TSO. Third, the response seems positively correlated with disease severity.

It is important to point out that while our study compares the ability of antibody assays to identify individuals who have had NAAT-confirmed COVID-19, it does not compare the ability to identify individuals who are protected against reinfection with SARS-CoV-2. Identification of those who are protected against reinfection, at least for a certain period, would be an important aspect of an assay, but it is too early in the epidemic to make this kind of comparison on a large scale. A recent study from GeurtsvanKessel et al. (31) reported cutoff values in the Wantai total-Ab assay indicating detectable levels of neutralizing antibodies. Those authors suggest this as a tool for the detection of neutralizing antibodies, though the clinical utility of this remains unclear.

Our study has several strengths and limitations. A major strength is that the case panel used for sensitivity across all 16 assays included 150 samples from the same patients, one of the largest panels investigated to date. Additionally, and in contrast to most previous evaluations of serological SARS-CoV-2 assays, this case panel was obtained largely from patients who had milder symptoms of COVID-19, evaluating whether the assays could detect SARS-CoV-2 antibodies among the most common type of patient with SARS-CoV-2 infection. This is valuable knowledge in seroepidemiological investigations. The specificity was evaluated with a significant number of pre-COVID-19 blood donor samples, making this study very solid in terms of clinical accuracy and agreement between the assays investigated. Furthermore, we investigated cross-reactivity using samples from individuals with autoimmune disease or acute infections with EBV or CMV. For these tests of specificity, we could not use samples from the same individuals across all assays due to the small sample volumes. This could potentially introduce heterogeneity between assays in the specificity data. However, using ≥586 samples from healthy donors for each assay makes this a very large sample, and substantial differences in a small homogenous country with same standard operating procedures for Danish blood banks are unlikely. Even though we tried to assess the risk of interference by examining samples from patients with acute viral diseases known to be associated with increased levels of assay-interfering antibodies, these could be present in patients with other diseases, e.g., other common coronavirus infections or cancers. Thus, we could have underestimated the potential for interference.

In conclusion, this comparative study of 15 commercial and one in-house laboratory serological SARS-CoV-2 assays pinpoints differences in accuracy; most total-Ab and IgG assays, including assays with potential for high-throughput production in automated laboratories, reached predefined criteria for acceptable performance, especially in samples from cases with a TSO over 20 days. Additionally, the antibody response seemed to be strongest among patients with more severe disease. It could appear as if the use of emergency authorizations has led to release of suboptimal assays in some cases, and simple measures such as optimization of cutoff values could lead to major improvements in performance. Thus, it is possible that optimized versions of some assays may be released in the near future.