Evaluation of Three Commercial and Two Non-Commercial Immunoassays for the Detection of Prior Infection to SARS-CoV-2

Abstract

Background

Serological testing provides a record of prior infection with SARS-CoV-2, but assay performance requires independent assessment.

Methods

We evaluated 3 commercial (Roche Diagnostics pan-IG, and Epitope Diagnostics IgM and IgG) and 2 non-commercial (Simoa and Ragon/MGH IgG) immunoassays against 1083 unique samples that included 251 PCR-positive and 832 prepandemic samples.

Results

The Roche assay registered the highest specificity 99.6% (3/832 false positives), the Ragon/MGH assay 99.5% (4/832), the primary Simoa assay model 99.0% (8/832), and the Epitope IgG and IgM 99.0% (8/830) and 99.5% (4/830), respectively. Overall sensitivities for the Simoa, Roche pan-IG, Epitope IgG, Ragon/MGH IgG, and Epitope IgM were 92.0%, 82.9%, 82.5%, 64.5% and 47.0%, respectively. The Simoa immunoassay demonstrated the highest sensitivity among samples stratified by days postsymptom onset (PSO), <8 days PSO (57.69%) 8–14 days PSO (93.51%), 15–21 days PSO (100%), and > 21 days PSO (95.18%).

Conclusions

All assays demonstrated high to very high specificities while sensitivities were variable across assays.

Impact Statement

It is important to patients and public health experts to have accurate antibody tests for detection of prior COVID-19 infection, but the tests can have variable results. This paper compares 3 commercial and 2 non-commercial assays for COVID-19 antibodies to determine sensitivity and specificity of each assay among 251 samples known to be PCR test positive and 832 samples collected prior to the COVID-19 pandemic.

Introduction

Many immunoassays have been developed for the detection of prior infection by severe acute respiratory syndrome 2 (SARS-CoV-2) (1–3). Serological assays to detect antibodies to SARS-CoV-2 have received attention due to many assays being used for a range of purposes despite suboptimal validation (4). However, despite enormous potential to guide the global COVID-19 response, confidence in serological tests and consequently the results of seroepidemiological studies have been undermined by poor (or poorly defined) test characteristics (4). Given the importance of vigorous and independent immunoassay cross validation, we report on the performance of 3 commercial and 2 non-commercial assays.

Materials and Methods

Ethical Considerations

The use of study samples and data was approved by the Massachusetts (Mass) General Brigham (MGB) (previously Partners Healthcare System) Institutional Review Board.

Study Design

We conducted a head-to-head test performance study using 3 commercial and 2 non-commercial SARS-CoV-2 immunoassays where laboratories were blinded to sample group.

Study Samples

Here, 251 SARS-CoV-2 polymerase chain reaction (PCR) positive samples from 122 patients (107 hospitalized, 15 ambulatory) treated at the Brigham and Women’s Hospital (BWH) between March 30 and May 29, 2020, were selected from the MGB Biobank, a biorepository that contains serum and other biological samples and linked demographic and clinical data from >121 000 patients enrolled through the MGB network (Table 1) (5). Samples were collected a mean of 14.0 days (SD 13.6 days) post-PCR with reverse transcription (RT–PCR) confirmation and 20.7 days (SD 14.8 days) postsymptom onset (PSO). The median number of samples per individual was 2 (range 1–8) and the median interval between sample collection was 3 days (range 2–47 days). The median age of patient samples was 58 years (range 24–90) and 135 (54%) samples came from females (Table 1).

Table 1.

Demographics and medical history of prepandemic and PCR-positive samples^a.

Variables	Prepandemic (n = 832)	PCR positive
		<8 days (n = 26)	8–14 days (n = 77)	15–21 days (n = 65)	>21 days (n = 83)	All (n = 251)	P value comparing prepandemic vs all PCR positiveb
Demographics
Age in year, median (range)	44 (20–89)	57 (25–79)	57 (25–90)	59 (27–83)	59 (24–84)	58 (24–90)	<0.001
Female sex, N (%)	390 (47%)	16 (62%)	44 (57%)	30 (46%)	45 (54%)	135 (54%)	0.061
Race, N (%)							<0.001
White	513 (62%)	11 (42%)	33 (43%)	26 (40%)	45 (54%)	115 (46%)
Black	282 (34%)	13 (50%)	31 (40%)	28 (43%)	29 (35%)	101 (40%)
Asian or Pacific Islander	17 (2%)	2 (8%)	4 (5%)	3 (5%)	8 (10%)	17 (7%)
American Indian or Alaskan Native	0 (0%)	0 (0%)	2 (3%)	1 (2%)	0 (0%)	3 (1%)
Other or not recorded	20 (2%)	0 (0%)	7 (9%)	7 (11%)	1 (1%)	15 (6%)
Ethnicity, N (%)							<0.001
Non-Hispanic	623 (75%)	17 (65%)	51 (66%)	43 (66%)	52 (63%)	163 (65%)
Hispanic	137 (16%)	6 (23%)	12 (16%)	11 (17%)	12 (14%)	41 (16%)
Other or not recorded	72 (9%)	3 (12%)	14 (18%)	11 (17%)	19 (23%)	47 (19%)
Highest level of care							NA
Ambulatory		0 (0%)	0 (0%)	1 (2%)	14 (17%)	15 (6%)
Hospitalized—non-ICU		19 (73%)	42 (55%)	22 (34%)	19 (23%)	102 (41%)
Hospitalized—ICU		7 (27%)	35 (45%)	42 (65%)	50 (60%)	134 (53%)
Past medical history, N (%)
HTN	259 (31%)	18 (23%)	47 (61%)	40 (62%)	60 (74%)	165 (66%)	<0.001
Obesity	234 (28%)	18 (23%)	43 (56%)	31 (48%)	41 (51%)	133 (53%)	<0.001
CAD	82 (10%)	11 (14%)	27 (35%)	20 (31%)	33 (41%)	91 (37%)	<0.001
Asthma	158 (19%)	7 (9%)	18 (23%)	17 (26%)	30 (37%)	72 (29%)	<0.001
Malignancy	84 (10%)	9 (12%)	16 (21%)	9 (14%)	18 (22%)	52 (21%)	<0.001
DM	77 (9%)	12 (16%)	25 (32%)	26 (40%)	31 (38%)	94 (38%)	<0.001
Liver disease	69 (8%)	6 (8%)	13 (17%)	6 (9%)	14 (17%)	39 (16%)	0.001
COPD	39 (5%)	3 (4%)	7 (9%)	6 (9%)	15 (19%)	31 (12%)	<0.001
Transplant	30 (4%)	0 (0%)	2 (3%)	3 (5%)	4 (5%)	9 (4%)	1.00
Other immune compromised conditions	18 (2%)	2 (3%)	0 (0%)	3 (5%)	2 (2%)	7 (3%)	0.63
Cerebrovascular accident	13 (2%)	3 (4%)	4 (5%)	5 (8%)	8 (10%)	20 (8%)	<0.001

^aEach sample was considered as an independent data point for calculating the values in this table.

^bWilcoxon rank sum test for continuous variables and Fisher’s exact test for categorical variables.

Prepandemic samples included 832 samples from the MGB Biobank collected between August 28, 2017 and September 26, 2019. The median age was 44 years (range 20–89) and 390 (47%) were female. We included a subset of samples with documented recent respiratory infections to assess for cross-reactivity and, we selected prepandemic samples with and without recent respiratory infections. Of the total 832 negative control samples, 600 were from individuals without recent respiratory illness; 31 from individuals with prior laboratory-confirmed respiratory infections; and 101 from individuals with a recent clinical diagnosis of respiratory infections including upper respiratory tract infection (n = 50) or viral (n = 11), bacterial (n = 20), or unspecified (n = 20) pneumonia (Table 2) based on diagnoses recorded in the electronic health record between 1 and 31 days prior to sample collection.

Table 2.

Clinical and confirmed respiratory viral infections among prepandemic samples.

Recent acute illness	Days prior to sample collection	Males	Females	Total
None	NA	370	330	700
URI
	1–14 days	15	10	25
	15–31 days	9	16	25
Bacterial pneumonia
	1–14 days	7	3	10
	15–31 days	5	5	10
Unspecified pneumonia
	1–14 days	7	3	10
	15–31 days	7	3	10
Viral pneumonia
	1–14 days	3	2	5
	15–31 days	4	2	6
Confirmed viral respiratory infectiona	NA	15	16	31
With any recent acute illness		72	60	132
Grand total		442	390	832

^aIncludes Parainfluenza antigen positive (n = 13), Metapneumovirus antigen (9), influenza A/B antigen (8), Influenza A PCR (3), Influenza B PCR (1), RSV antigen (5), RSV PCR (1), Adenovirus antigen (3), Herpes Simplex I (DFA). Total number add up to more than 31 as some individuals recorded >1 positive result.

NA Not available or applicable.

To ensure valid comparison between assays and given differences in plasma/sera requirements according to manufacturer/assay specifications, we only selected samples with both serum and plasma available from the same individual and time point (Table 2). All samples were stored at −80 °C following sample processing and none underwent thaw-refreezing cycles prior to analysis. Except for sample type (i.e., serum or plasma), identical samples were provided to each of the 4 participating laboratories (with 2 fewer samples provided to one due to insufficient volume). Samples were blinded to all laboratory staff and investigators and only unblinded after results were provided to the lead investigators (EJN, EWK, LRB).

Clinical Data

We extracted demographic and clinical data including symptom onset data on PCR-positive samples from the Biobank-linked electronic health records system supplemented by medical record review.

Serological Assays and Protocols

We assessed 5 assays including Elecsys Anti-SARS-CoV-2 (Roche Diagnostics, Indianapolis, USA) intended for the qualitative detection of pan-immunoglobulin antibodies against the nucleocapsid (N) antigen (6); EDI New Coronavirus COVID-19 enzyme-linked immunosorbent assays (ELISA) (Epitope Diagnostics, USA) that detect IgG and IgM against the N antigen (7, 8); Ragon/MGH, an in-house ELISA that detects IgG, IgM, and IgA against the receptor binding domain (RBD); and the single molecule array multiplex assay (Simoa) that detects IgG, IgM, and IgA against the spike protein, S1 subunit, RBD, and NC (9). The Ragon/Massachusetts General Hospital assay was performed at the Ragon Institute of MGH, Massachusetts Institute of Technology, and Harvard; all other assays were performed at the BWH. Commercial assays were performed according to manufacturer specifications. The Simoa and Ragon/MGH assays were performed according to previously described methods (10). All samples were tested for investigatory purposes, not for clinical diagnostic testing. Commercial assays but not non-commercial assays received Emergency Use Authorization from the United States Food and Drug Administration and CE certification from the European Medical Device Safety Service.

Result Classification

Threshold cutoffs for defining positive, negative or indeterminate/borderline test results were defined according to manufacturer specifications for commercial assays. Threshold cutoffs and result determination for the non-commercial assays were established by the respective laboratories prior to the study according to methods previously described (9,10). Given the Simoa multiplex assay includes 12 output measures per sample (IgG, IgM, and IgA against 4 viral epitopes), results were based on 3 prestudy classification models—an “Early Model,” “Late Model,” and full panel “12-Parameter Model.” (9) The Early Model, which previously demonstrated the best performance, includes 4 markers: IgA S1, IgA NC, IgG NC, and IgG Spike (9).

Data Analysis

We performed 5 primary independent analyses: 1 each for the Roche (pan-IG) and Ragon/MGH (IgG) assays; 2 for the Epitope immunoassays (IgG and IgM); and 1 for the primary Simoa assay “Early Model.” Analyses of the Ragon IgA/IgM and Simoa “Late Model” and “12-Parameter Model” are included in the Supplemental Materials. Indeterminate or borderline results were considered negative. Sensitivity was calculated independently for samples collected <8, 8–14, 15–21, and >21 days PSO. Assay agreement was calculated between the Roche, Ragon/MGH IgG, Epitope IgG, and Simoa Early Model using prevalence-adjusted and bias-adjusted Kappas (11). Binomial exact 95% confidence intervals were calculated for all estimates. All analyses were performed using the R software package (v.4.0, www.R-project.org/).

Results

Differences in demographics and medical history between individuals that provided PCR-positive and prepandemic samples are reported in Table 1. PCR-positive samples were from older individuals and more likely to be of non-white race with a higher prevalence of preexisting comorbidities including hypertension, coronary heart disease, stroke, obesity, asthma, diabetes mellitus, malignancy, chronic obstructive pulmonary disease, and liver disease.

Specificity

The Roche assay registered 3/832 false positives for a specificity of 99.64% (95% CI 98.94–99.88%) (Table 3). The Epitope IgM and Ragon/MGH (IgG) assays registered 4/830 and 4/832 false positives for specificities of 99.52% (95% CI 98.77–99.81%). The Epitope IgG and Simoa (Early) assays registered 8/830 and 8/232 false positives for specificities of 99.04% (98.11–99.51%). Data on secondary assays/models are detailed in Tables 1 and 2 in the online Data Supplement. No Epitope false positives overlapped and therefore if combining the 2 assays to provide a single result, the specificity is lower [12/830 false positives; 98.55% (95% CI 97.49%–99.17%)]. Of the 27 false positive results [Roche (3), Ragon/MGH IgG (4), Epitope IgG (8), and IgM (4), and Simoa Early Model (8)], 22 were from 700 prepandemic samples (3.1%) without recent respiratory infection and 5 from 132 prepandemic samples (3.7%) with recent respiratory infection, suggesting cross-reactivity due to recent respiratory infections is unlikely to be an important cause of false positives in these assays. However, no human common coronaviruses (HCoV, e.g., 229E, NL63, OC43, or HKU1) were documented among these samples so these data do not assess for HCoV-specific cross-reactivity.

Table 3.

Assay specificities by isotype.

Immunoassay	No. of prepandemic samplesa	No. testing negative	Percentage	95% CI
Epitope IgG	830	822	99.04	98.11–99.51
Epitope IgM	830	826	99.52	98.77–99.81
Ragon/MGH IgGb	832	828	99.52	98.77–99.81
Rochec	832	829	99.64	98.94–99.88
Simoa (Early)d	832	824	99.04	98.11–99.51

^aGiven limited prepandemic sample aliquots, the Epitope assays were tested against 830 samples versus 832 for the remaining assays.

^bFor specificity of Ragon/MGH IgM and IgA, see Supplemental Materials.

^cThe Roche Elecsys Anti-SARS-CoV-2 immunoassay detects IgG and likely IgM and IgA; details of other isotypes are not provided by the manufacturer.

^dSpecificity of the Simoa multiplex assay Early Model. For specificities of the Late and 12-Parameter Models, see Supplemental Materials.

Sensitivities

The Simoa Early Model registered the highest sensitivity among samples collected <8 days PSO (57.69%), 8–14 days PSO (93.51%), 15–21 days PSO (100%), and >21 days PSO (95.18%) (Table 4). The Epitope IgG registered sensitivities of 42.31%, 82.47%, 92.31%, and 85.54% for respective categories of days since PSO. Sensitivities during the earliest time period, <8 days PSO, was low for all assays (Table 4).

Table 4.

Assay sensitivities by days post symptom onset.

Assay	Days PSO	Total No. of PCR-positive samples	No. testing positive	Percentage	95% CI
Epitope IgG	<8 days	26	11	42.31	25.54–61.05
	8–14 days	77	65	84.42	74.71–90.85
	15–21 days	65	60	92.31	83.22–96.67
	>21 days	83	71	85.54	76.41–91.53
	Overall	251	207	82.47	77.29–86.67
Epitope IgM	<8 days	26	8	30.77	16.50–49.99
	8–14 days	77	43	55.84	44.74–66.39
	15–21 days	65	41	63.08	50.92–73.77
	>21 days	83	26	31.33	22.36–41.94
	Overall	251	118	47.01	40.93–53.18
Ragon/MGH IgGa	<8 days	26	5	19.23	8.51–37.88
	8–14 days	77	44	57.14	46.01–67.60
	15–21 days	65	52	80.00	68.73–87.92
	>21 days	83	61	73.49	63.11–81.80
	Overall	251	162	64.54	58.45–70.20
Rocheb	<8 days	26	13	50.00	32.06–67.94
	8–14 days	77	62	80.52	70.31–87.82
	15–21 days	65	59	90.77	81.29–95.70
	>21 days	83	74	89.16	80.66–94.19
	Overall	251	208	82.87	77.72–87.03
Simoa (Early)c	<8 days	26	15	57.69	38.95–74.46
	8–14 days	77	72	93.51	85.68–97.19
	15–21 days	65	65	100.00	94.42–100.00
	>21 days	83	79	95.18	88.25–98.11
	Overall	251	231	92.03	88.01–94.78

^aFor sensitivity of Ragon/MGH IgM and IgA see Supplemental Materials.

^bThe Roche Elecsys Anti-SARS-CoV-2 immunoassay detects IgG and likely IgM and IgA; details of other isotypes are not provided by the manufacturer.

^cSensitivity of the Simoa multiplex assay Early Model. For sensitivities of the Late and 12-Parameter Models, see Supplementary Materials.

Interassay concordance for prepandemic samples was high for all assay combinations with the highest agreement between Roche and Ragon/MGH IgG assays (Kappa 0.98, 95% CI 0.97–0.99) and the lowest between Epitope IgG and Simoa Early Model (Kappa 0.96, 95% CI 0.94–0.98). Interestingly, of 27 total false positives across the 5 assays, none were overlapping between assays. Interassay agreement for PCR-positive samples was more variable and ranged from Kappa 0.78 (95% CI 0.69–0.86) between the Simoa Early Model and Epitope IgG assays to 0.45 (95% CI 0.33–0.56) between the Ragon/MGH IgG and Simoa Early Model. Lower concordance between PCR-positive samples was largely driven by the higher numbers of false negatives observed in the Ragon/MGH IgG (89/251), Epitope IgG assays (44/251), and Roche assays (43/251). Of the 102 discrete false negative results, 58 overlapped between 2 or more assays.

Discussion

We assessed the performance of 3 widely used commercial and 2 non-commercial SARS-CoV-2 immunoassays using a panel of 251 PCR-positive hospitalized cases, and 1083 well-characterized prepandemic samples. Assays targeted a range of antigens including spike, NC, RBD, and NAD S1. Unlike most head-to-head SARS-CoV-2 immunoasssay performance evaluations (1), common patient-sample combinations were used for all assays. In comparison to 2 large studies comparing the performance of 5 commercial assays (12, 13) that included COVID-19 PCR-positive convalescent samples collected >14 days and >20 days since symptom onset, we included COVID-19 samples collected 1–14 days since symptom onset to assess early sensitivity.

All 5 primary assays demonstrated high specificity with small absolute differences. The Roche specificity of 99.6% (98.9–100) aligned with package insert data of 99.8% (99.7–99.9) (6). The Epitope IgG assay specificity at 99.0% was higher than recent smaller studies that reported specificities of 88.7% from 53 prepandemic samples (14) and 89.8% from 108 prepandemic samples (15). The Ragon/MGH registered high specificity (99.5%) and the Simoa Early Model slightly lower (99.0%). All primary assay specificities registered overlapping confidence intervals.

Sensitivities were reasonable for samples collected ≤21 days: 19.2–57.7% for samples collected <8 days PSO, 57.1–93.5% for samples collected between 8–14 days PSO, and 80.0–100% sensitivity for samples collected 15–21 days PSO; but slightly lower than anticipated for samples collected >21 days PSO (73.5–95.2%). When compared against other available data, the sensitivity of the Roche pan-IG assay and Epitope assays were lower than manufacturer reported data that reported at most time points (6, 7). Similarly, the Roche assay registered lower sensitivity for samples collected ≥21 days PSO (89.2%) in this study than a recent large UK performance study, that reported a sensitivity of 97.2% from 536 PCR-confirmed samples collected ≥20 days PSO (13). The sensitivity of the Epitope IgG assay was lower than package insert data that report 100% for 30 PCR-positive samples in the second week of disease and lower than a German study that reported 100% sensitivity for 22 PCR-positive samples (14) but similar to a US-based study that reported sensitivities of 84% at 6–20 days and 91% at >20 days (15). Surprisingly, assay sensitivities, particularly in the ≥21-day time period when almost all PCR-confirmed cases would be expected to have detectable antibodies (16), were lower than expected with only the Simoa assay maintaining high sensitivity (95.2%) consistent with a prior report. Positive samples in this study were collected prior to COVID vaccine availability or the (known) emergence of major viral variants. Going forward, positive assays post vaccination would be expected on the MGH/Ragon (RBD antigen) and Simoa assays (Spike antigen).

The strengths of this study is the large number of systematically collected and curated control samples and the use of all samples across all assays, which is required to accurately compare assays but is rare among head-to-head assay evaluations (1–3), including COVID-19 samples collected 1–14 days since symptom onset to assess early sensitivity, and using assays targeted to a range of antigens (spike, RBD, nucleocapsid, and S1). We also provide granular details on samples and include a large number of prepandemic samples from individuals recently diagnosed with respiratory infections. We cannot, however, definitively extrapolate findings to other populations. For example, the sensitivity of these assays was largely assessed in samples from RT–PCR-confirmed hospitalized patients expected to have high titers of antibodies (17) with samples collected a mean of 20.7 days (SD 14.8 days) after symptom onset. Given most SARS-CoV-2 infections are mild or asymptomatic and do not require hospitalization, and given that little is known about humoral kinetics >4 months post infection, sensitivities are likely to be lower in these populations (4).

Supplemental Material

Supplemental material is available at The Journal of Applied Laboratory Medicine online.

Nonstandard Abbreviations

MGB

Massachusetts General Brigham

BWH

Brigham and Women’s Hospital

MGH

Massachusetts General Hospital

PCR

polymerase chain reaction

Ig

immunoglobulin

PSO

postsymptom onset

ELISA

enzyme-linked immunosorbent assays

Simoa

single molecule array multiplex assay

NC

nucleocapsid

RBD

receptor binding domain

 

Author Contributions: All authors confirmed they have contributed to the intellectual content of this paper and have met the following 4 requirements: (a) significant contributions to the conception and design, acquisition of data, or analysis and interpretation of data; (b) drafting or revising the article for intellectual content; (c) final approval of the published article; and (d) agreement to be accountable for all aspects of the article thus ensuring that questions related to the accuracy or integrity of any part of the article are appropriately investigated and resolved.

E.W. Karlson, statistical analysis, provision of study material or patients; G. Zhou, statistical analysis; N.V. Tolan, administrative support, provision of study material or patients.

Authors’ Disclosures or Potential Conflicts of Interest: Upon manuscript submission, all authors completed the author disclosure form. Disclosures and/or potential conflicts of interest: Employment or Leadership: N.V. Tolan, The Journal of Applied Laboratory Medicine, AACC; D.R. Walt has a financial interest in Quanterix Corporation, a company that develops an ultra-sensitive digital immunoassay platform. D.R. Walt is an inventor of the Simoa technology, a founder of Quanterix Corporation, and also serves on its Board of Directors. D.R. Walt’s interests were reviewed and are managed by Brigham and Women’s Hospital and Mass General Brigham in accordance with their conflict of interest policies. Consultant or Advisory Role: P. Jarolim, Roche Diagnostics Corporation; A. Woolley, COVAX. Stock Ownership: D.R. Walt has a financial interest in Quanterix Corporation, a company that develops an ultra-sensitive digital immunoassay platform. Honoraria: P. Jarolim, Roche Diagnostics Corporation. Research Funding: National Institutes of Health (UL1TR001102). Funding for this work came from a generous donation from Barbara and Amos Hostetter and the Chleck Foundation. This work was also funded through a grant from the Massachusetts Consortium on Pathogen Readiness. E.J. Nilles is supported by Centers for Disease Control and Prevention (U01 GH002238). E.W. Karlson is supported by National Institutes of Health (U01 HG008685, OT2OD026553, P30 AR070253). D.P. Simmons is supported by NIH K08 AR075850. L.R. Baden is supported by NIH UM1AI069412 and 8. M. Norman, Mass CPR. Expert Testimony: None declared. Patents: M. Norman, patent number not assigned; T. Gilboa, patent number not assigned; D.R. Walt, multiple patents; G. Alter, Systems Serology Platform—pending.

Role of Sponsor: The funding organizations played no role in the design of study, choice of enrolled patients, review and interpretation of data, preparation of manuscript, or final approval of manuscript.

Acknowledgments: We wish to thank the participants in the Mass General Brigham Biobank for their contribution of samples and electronic health record data, the recruitment teams from the Center for Clinical Investigation, and Mass General Brigham Biobank, informatics, and laboratory teams at Mass General Brigham Personalized Medicine who made this study possible.