Analytical evaluation and critical appraisal of early commercial SARS-CoV-2 immunoassays for routine use in a diagnostic laboratory

ABSTRACT

Background: The objective of this study was to evaluate the performance characteristics of early commercial SARS-CoV-2 antibody assays in mild and asymptomatic subjects to enable the selection of suitable immunoassays for routine diagnostic use.

Methods: We used serum samples from a pre-COVID era patient cohort (n = 50, pre-December 2019), designated SARS-CoV-2 negative, and serum samples from a SARS-CoV-2 RT-PCR-positive cohort (n = 90) taken > 14 days post-symptom onset (April–May 2020). Six ELISA assays were evaluated, including one confirmation assay to investigate antibody specificity. We also evaluated one point-of-care lateral flow device (LFIA) and one high throughput electrochemiluminescence immunoassay (CLIA).

Results: The ELISA specificities ranged from 84% to 100%, with sensitivities ranging from 75.3% to 90.0%. The LFIA showed 100% specificity and 80% sensitivity using smaller sample numbers. The Roche CLIA immunoassay showed 100% specificity and 90.7% sensitivity. When used in conjunction, the Euroimmun nucleocapsid (NC) and spike-1 (S1) IgG ELISA assays had a sensitivity of 95.6%. The confirmation Dia.Pro IgG assay showed 92.6% of samples tested contained both NC and S1 antibodies, 32.7% had NC, S1 and S2 and 0% had either S1 or S2 only.

Conclusions: The Roche assay and the Euroimmun NC and S1 assays had the best sensitivity overall. Combining the assays detecting NC and S1/S2 antibody increased diagnostic yield. These first-generation assays were not calibrated against reference material and the results were reported qualitatively. A portfolio of next-generation SARS-CoV-2 immunoassays will be necessary to investigate herd and vaccine-induced immunity.

Introduction

The current worldwide coronavirus pandemic caused by Severe Acute Respiratory Syndrome Coronavirus 2 (SARS-CoV-2) has resulted in millions of confirmed infections, cases of the associated COVID-19 disease and deaths [1,2]. The clinical manifestations of acute infection vary widely from asymptomatic disease to coagulopathy, severe viral pneumonia, and lung failure [3,4] and the extent of chronic organ damage and COVID-19 disease remains to be established. A definitive diagnosis of SARS-CoV-2 infection currently relies on the use of RT-PCR to identify the virus in respiratory samples [5–7]. This only identifies current illness or a viral carriage. The performance of the RT-PCR test is dependent on various factors including time of sampling, viral load and how thoroughly the sample is taken from the nasopharynx[8]. Consequently, a significant proportion of infected individuals may be missed from screening programmes. In contrast, robust serology assays which reliably detect the presence of antibodies against SARS-CoV-2, can determine whether individuals with or without symptoms have previously been infected, thus providing valuable information about prior exposure for epidemiological purposes and the individual patient [9,10].

The introduction of serological testing throughout the UK was rapidly implemented in the spring of 2020 to cover a variety of scenarios. Accordingly, we scanned the first-generation antibody assay horizon for candidate tests we could rollout into routine laboratory diagnostics. During the early months of the pandemic, numerous SARS-CoV-2 immunoassays were rapidly developed and placed on the market. These assays used different antigenic proteins; some used whole virus lysate, recombinant nucleocapsid (NC) or full spike (S) proteins, while some used modified proteins or peptides of the NC or specific domains of the S protein – glycoprotein 1 (S1), spike glycoprotein 2 (S2) or RBD (receptor-binding domain).

Our objective was to evaluate the analytical performance of CE marked IgG ELISAs and a lateral flow device, selected based on the availability in the UK at the time. We later extended this to the analytical performance of the Roche ‘total antibody by electrochemiluminescence (CLIA) immunoassay’, an automated high throughput platform for the Roche assay (the Cobas system).

Materials and methods

In total, 140 patient serum samples were obtained from the Serology and Immunology Departments during March and April 2020. Of these, 50 pre-COVID era samples were used to provide specificity data. Sensitivity was determined with 10 serum samples from adult hospitalised patients (7 male, 3 female, median age 55 years, range 20–81 years) with confirmed SARS-CoV-2 infection, obtained from anonymised excess serum samples. A further 80 serum samples from healthcare workers (HCW) and proven to have had SARS-CoV-2 infection (days post symptoms: mean 21.2 days, range 15–33 days) by viral nucleic acid detection (RT-PCR) from an upper respiratory tract (nasopharyngeal) swab were positive controls. Qualitative RT-PCR was performed in an accredited laboratory using SARS-CoV-2 N1 (2019-nCoV_N1) and CDC-Primers.

Serum was taken from the HCW subjects after their return to work and after two negative swab tests by RT-PCR, having isolated for 2 weeks after their confirmed RT-PCR positive result. Staff were asked to report the date of onset of Covid-19 associated symptoms. In all cases, sensitivity cohort samples were taken > 14 days post-symptom onset to optimise detection of SARS-CoV-2 antibodies. Seventeen (17%) subjects were asymptomatic and sixty-three (63%) had mild symptoms.

The study was an audit of routine sera and was reviewed by the local Audit and Research Committee. All samples were anonymised. Written consent was obtained from the HCW subjects for the audit. Ethical consent is not required for surplus samples used for SARS-CoV-2 new assay in-service validation and verification in the UK as per Department of Health and Social Care statement; local governance rules around standard verification and validation were followed.

Lateral flow immunoassay (LFIA)

The LFIA was only evaluated for specificity using 20 of the pre-COVID era samples due to the limited number of kits to hand. For sensitivity, 80 of the HCW samples were tested by staff at HCA Primary Care facilities.

We used the COVID-19 IgG/IgM Rapid Test (CTK Biotech Inc, Poway, USA), performed in accordance with the manufacturer’s instructions. The test was accepted if the ‘C’ line (control line) showed the appearance of a coloured band, with the result designated as either positive (G and/or M band development) or negative (C line only).

Enzyme-linked immunosorbent assays (ELISA)

We assessed five qualitative ELISAs and one confirmatory ELISA from four different manufacturers as outlined below. All ELISA assays were run using the automated DS2 Analyser (Dynex® Technologies, Inc).

Due to the speed these assays were brought to market, they were not calibrated against any reference material (e.g. IRP 67/86 of human serum immunoglobulins) or international standards. ELISA results were qualitative and based on a calculated ratio (of the relative antibody concentrations) set by the manufacturer or a ‘confidence index’ assigned by the manufacturer.

Epitope diagnostics

The EDI^TM Novel Coronavirus COVID-19 IgG ELISA kit (KT-1032, Epitope Diagnostics, San Diego, USA) utilises a recombinant full length nucleocapsid protein to measure human anti-SARS-CoV-2 IgG antibody in serum. The assay was run as per manufacturer’s instructions and the results were calculated as recommended.

Euroimmun

The Anti-SARS-CoV-2 NCP ELISA (IgG) and Anti-SARS CoV-2 S1 (IgG) ELISAs (Euroimmun Medizinische Labordiagnostika, Lübeck, Germany; EI2606-9620-2 G and EI2606-9601 G, respectively) were performed according to manufacturer’s instructions. The NC assay wells are coated with modified nucleocapsid (NC) of SARS-CoV-2 which according to the manufacturer contains ‘diagnostically relevant epitopes’, whereas the S1 assay wells are coated with an S1 domain of the spike protein of SARS-CoV-2 expressed in the human cell line HEK 293.

Dia.Pro

The Dia.Pro COVID-19 IgG ELISA and the COVID-19 IgG Confirmation ELISA kits (Diagnostic Bioprobes Srl, Milan, Italy) are intended for use in conjunction as an initial screen, and subsequent confirmatory assay, respectively, for antibodies to SARS-CoV-2. The Dia.Pro COVID-19 IgG ELISA contains a mix of recombinant ‘immunodominant antigens’ (nucleocapsid, S1 and S2) whereas the confirmation test is composed of individual wells coated with these antigens and hence enables to determine antibody specificity against NC, S1 and S2 present in the patient’s sample.

NovaLisa®

NovaLisa® SARS-CoV-2 9 (Covid19) IgG ELISA (obtained from NovaTec Immunodiagnostica Gmbh, Dietzenbach, Germany, product code COVG0940) utilises microtiter plates coated with SARS-CoV-2 antigens (antigen names are not provided).

Chemiluminescent assay (CLIA)

The Roche Elecsys total antibody (IgG and IgM) assay was run on the Cobas® e801 analyser. The Elecsys Anti-SARS-CoV-2 assay is an electrochemiluminescence sandwich immunoassay for qualitative detection of both IgG and IgM in human serum and plasma against a recombinant SARS-CoV-2 nucleocapsid antigen.

Statistical analysis

For each method, sensitivity, specificity and 95% confidence intervals were calculated using the R statistical computing environment. A pairwise comparison was performed between each unique combination of assays using a two-sample Z-test for equality of proportions. A p-value of ≤0.01 was used as a threshold for significance to correct for multiple testing. Cohen’s Kappa coefficient with 95% CI was then calculated for each method in comparison to the RT/PCR results.

Results

Four commercial anti-SARS-CoV-2 IgG ELISA kits were evaluated initially, all of which measured antibodies raised against the recombinant nucleocapsid (NC) antigen of the virus. Subsequently, the Euroimmun Anti-SARS-CoV-2 IgG S1 ELISA, which measures antibodies to the S1 (S) domain of the spike protein, became available and was included in the evaluation. Table 1 shows sensitivities and specificities of these 5 ELISA assays and the LFIA and Roche CLIA assay. To maximise specificity and sensitivity for each ELISA, borderline results were classified as negative for the EDI and DIA.PRO kits, and positive for the Euroimmun and NOVALisa kits.

Table 1.

Sensitivities and specificities of ELISA, LFIA and CLiA assays for SARS-CoV-2 antibody detection shown with 95% confidence intervals (CI)

	Number of Serum Samples
	RT-PCR confirmed cases		Pre-COVID control
ASSAY	True Positive	False Negative	True Negative	False Positive	Sensitivity (95% CI)	Specificity (95% CI)
ELISA
EDITM	67	14	49	1	82.7(72.4–89.9)	98.0 (88.0–99.9)
DIA.PRO	83	7	42	8	92.2(84.1–96.5)	84.0 (70.3–92.4)
NOVALisa	67	22	50	0	75.3(64.8–83.5)	100.0 (91.1–100)
Euroimmun NC	81	9	50	0	90.0(81.4–95.0)	100.0 (91.1–100)
Euroimmun S1	80	10	50	0	88.9(80.1–94.3)	100.0 (91.1–100)
LFIA
CTK Biotech	62	16	20	0	79.5(68.5–87.5)	100.0 (80.0–100)
CLIA
Roche	68	7	25	0	90.7(81.1–95.8)	100.0 (83.4–100)

NC = Nucleocapsid. S1 = Spike Glycolipid 1

The Roche assay and the Euroimmun NC and S1 assays had the best sensitivity overall, all with 100% specificity and when combined, the Euroimmun NC and S1 assays had a sensitivity of 95.6%. The Euroimmun ELISAs against 2 viral proteins were therefore chosen as the ELISA(s) most appropriate for deployment.

Euroimmun assays

The sample cohorts were classified as hospital, mild disease and asymptomatic according to details given at time of sampling. The results of the Euroimmun NC and S1 are shown in Table 2 when grouped according to these classifications. These results show that patients who had severe disease all had either S1 or NC antibodies however the mild/asymptomatic cohort showed little difference between the groupings. Of the HCW cohort, 17/80 (21.3%) were asymptomatic.

Table 2.

Percentages of patient cohorts with antibodies to nucleocapsid (NC) and S1 (Spike Glycolipid 1) by Euroimmun ELISA kits

Patient Cohort	Negative (NC and S1)	NC	S1	NC or S1
Hospital (N = 10)	0.0%	90.0%	100.0%	100.0%
Mild (N = 63)	4.8%	88.9%	87.3%	95.2%
Asymptomatic (N = 17)	5.9%	94.1%	88.2%	94.1%

Dia.Pro confirmation assay

The Dia.Pro conformation assay was performed to identify the specific antibodies present in the samples from RT-PCR positive patients that tested positive by the Dia.Pro ELISA screen assay. These results are shown in Table 3. In total, 52 samples were tested on the confirmation assay. All 52 samples contained NC antibodies and the number of samples that had all three antibodies was 17/52 (32.7%). 48/52 (92.6%) contained both NC and S1 antibodies. Of the hospitalised patients, only 1/10 (10%) had anti-NC IgG only and 4/10 (40%) had all three specificities (anti-NC, anti-S1 and anti-S2 IgG). None of the 52 samples contained S2 antibodies only.

Table 3.

Number of samples showing different specificities of antibodies to the major SARS-CoV-2 antigens using the Dia.Pro Confirmation assay

	Number of serum samples
	N	S1	S2	NC only	Both S1 and NC
Positive	52	45	15	4	45
Negative	0	4	35	-	-
Borderline	0	3	2	0	3
Total Sera	52	52	52		48

NC = Nucleocapsid. S1 = Spike Glycolipid 1. S2 = Spike Glycolipid 2.

Comparison between the assays (sensitivity)

Pairwise comparison showed a statistically significant difference between NovaLisa and Dia.Pro ELISA screen assay, p < 0.005.

Comparison between the assays (specificity): pairwise comparison showed a statistically significant difference between NovaLisa and Diapro ELISA screen assay, p < 0.001, Euroimmun S1 and Diapro ELISA screen assay p < 0.001, Euroimmun NC and Diapro ELISA screen, p < 0.001 and NovaLisa and Diapro ELISA screen, p < 0.001.

Agreement between RT-PCR and immunoassays

The agreement was very good (kappa >0.80) for Euroimmun NC (kappa 0.86, 95% CI 0.78–0.95); Euroimmun S1 (kappa 0.85, CI 0.76–0.94) and Roche NC (kappa 0.83, CI 0.71–0.95). The agreement was substantial (kappa 0.61–0.80) for Edi NC (kappa 0.77, CI 0.66–0.88), Dia.pro (kappa 0.77, CI 0.65–0.88), NovaLisa (kappa 0.69, CI 0.57–0.81) and CTK (kappa 0.61, CI 0.44–0.69).

Comparison of discordant samples

Of the confirmed positive RT-PCR samples from the healthcare workers that were run across all seven platforms, contradictory results were found in 23/75 samples. Each of these 23 samples were positive on either of the two Euroimmun assays. It was noted that 3/80 (4%) of the confirmed positive RT-PCR samples from HCWs were negative across all the SARS-CoV-2 antibody tests.

Discussion

We evaluated analytical performance of seven commercial SARS-CoV-2 immunoassays for routine diagnostic use for screening and diagnostic use. Clinical and immunological aspects of the subjects were not considered. The ELISA assays with the highest sensitivity were the Euroimmun NC (90%) and S1 kits (88.9%); when used in combination, sensitivity increased to 95.6%. The Roche Elecsys had 100% specificity and 90.7% sensitivity. These results are in accordance with another evaluation of Roche assay, which found specificity of 100% and sensitivity of 86.1% for samples taken ≥14 days from symptom onset and 86.7% for samples taken ≥21 days since symptom onset[11].

We were not able to establish negative and positive predictive values since the prevalence of SARS-CoV-2 infection is an estimate and was not precisely known. Four per cent of all RT-PCR positive subjects had no detectable antibodies to SARS-CoV-2 suggesting that not all otherwise healthy individuals will mount humoral response to the virus. This has been noted by other groups. [12]

We noted significant differences in assays' sensitivity and specificity (NovaLisa assay was the least sensitive (75%) while Dia.Pro was the least specific (84%) but the most sensitive (92%)). There was significant variability between the assays with regards to the individual sample result (positive/negative). These differences can be explained by analytical variances between the assays, e.g. antigen type, source and immunogenicity. The assays we evaluated used a variety of viral protein antigens, e.g. recombinant modified or full-length protein, mixed antigens or unspecified (NovaLisa). SARS-CoV-2 proteins have complex 3D structure and purification is a hurdle for manufacturers attempting fast development of new immunoassays[13].

Potential assay interferences may lead to false positive immunoassay results. For example, the Dia.Pro kit assay insert states that ‘10% of the reactive normal population collected before the outbreak showed reactivity to nucleocapsid’. It remains to be established whether this positivity in the Dia.Pro assay which uses ‘immunodominant recombinant antigens’ is due to cross-reactivity with other Coronaviridae or it is due to unknown interferences or cross-reactions with the components in the sample. In our hands, Dia.Pro had lowest specificity (84%) of all the ELISA assays evaluated.

The sensitivity and specificity of these assays differed from those reported by the manufacturers. These early assays came on the market following a quick assay evaluation using small numbers of mainly hospitalised individuals with severe disease known to make strong antibody response [14] and therefore not optimised for population-based serological sampling and investigations of asymptomatic or mildly symptomatic individuals with SARS-CoV-2 infection. Hence, the assay performance figures established by the manufacturers were higher than those established independently. The results were reported qualitatively and expressed in ratios and arbitrary units. The positive cut-offs were defined by the patient population chosen by the manufacturer, typically a group of older hospitalised patients with severe disease. These early assays were not calibrated against any reference material, unlike the newer assays, e.g. ELIA Phadia SARS-CoV-2 IgG assays, that use the IgG calibrators traceable to the International Reference Preparation (IRP) 67/86 of Human Immunoglobulins. The quality of the SARS-CoV-2 antibody assays should improve with the 1^st WHO international standard expected in December 2020. External quality assurance schemes have now been established to allow for better quality control.

The major use of SARS-CoV-2 serological assays was to inform seroprevalence in populations with low disease prevalence. The selection of an appropriate antibody assay and the timing of the measurements are of paramount [15]. CDC has recommended independent assays be combined to improve positive predictive values [16]. This is consistent with our findings of improved sensitivity and specificity by combining NC and S1 Euroimmun assays. In the recent systematic review of 491 papers by Huang et al. [17]., the most used assays were binding assays led by ELISA. Median time of detection of SARS-CoV-2 antibody was 11 days and antibody kinetics varied across the severity gradient, with antibodies remaining detectable longer after severe illness. There is a knowledge gap since antibody kinetics data come from symptomatic and not from subclinical or asymptomatic individuals.

Assays based on detection of antibodies to nucleocapsid were attractive for commercial companies since they are detected sooner post-infection. However, they do not last at detectable levels in mild disease and they have the potential for cross-reactivity with other Coronaviridae. Spike proteins are the immunodominant antigens involved in virus receptor binding (S1) and cell membrane fusion (S2) hence anti-spike protein antibodies are better correlates of viral neutralisation. Therefore, anti-SARS-CoV-2 vaccines target spike proteins to elicit immune responses [3,15,17].

Profiling of the acute humoral immune response to SARS-CoV-2 may help predicting disease outcomes. Atyeo et al. [18]. found that spike-specific antibody responses in hospitalised patients with SARS-CoV-2 infection were associated with convalescence and better outcomes while functional antibody responses to the nucleocapsid were elevated in deceased individuals. In that study, a combination of five SARS-CoV-2 specific antibody measurements enabled the authors to distinguish individuals with different disease trajectories. Hence, the diverging immune response and the signature with a higher spike:nucleocapsid ratio appears to be a powerful biomarker of COVID19.

Our study demonstrates that the platforms tested have similar analytical performance (except for the Dia.Pro assay which had low specificity), and that no single assay will be sufficient to dissect antibody response in COVID19. A portfolio of new generation antibody assays should allow the detection of high quality neutralising antibodies produced as a result of the germinal centre response as well as assays specific for the individual viral proteins that distinguish between the immunity due to natural infection, vaccination and recrudescence.

This work represents an advance in biomedical science because it has identified SARS-Cov-2 immunoassays for routine use in the diagnostic laboratory.

Summary table

What is known about this subject? Serology tests can be used to investigate the development of humoral immune response in people exposed to SARS-CoV-2 virus Testing for SARS-Cov-2 antibodies may aid the quantification of asymptomatic and recovered COVID-19 cases These assays are used to estimate the population prevalence of COVID-19 by identifying SARS-CoV-2 seropositive individuals What this work adds:: Early SARS-CoV-2 immunoassays varied significantly in analytical sensitivity and specificity Early assays were not calibrated against reference material and manufacturer’s ratios were used to calculate the results Combining assays that detect antibodies to nucleocapsid and to spike proteins increase diagnostic yield.

Funding Statement

No external funding was required for this study.

Disclosure statement

The authors have no conflicts of interest to disclose.