Clinical performance of four immunoassays for antibodies to SARS-CoV-2, including a prospective analysis for the diagnosis of COVID-19 in a real-life routine care setting

Abstract

Objectives

The aim of the present study was to evaluate the clinical performance of four SARS-CoV-2 immunoassays and their contribution in routine care for the diagnosis of COVID-19, in order to benefit of robust data before their extensive use.

Methods

The clinical performance of Euroimmun ELISA SARS-CoV-2 IgG, Abbott SARS-CoV-2 IgG, Wantai SARS-CoV-2 Ab ELISA, and DiaPro COVID-19 IgG confirmation were evaluated in the context of both a retrospective and a prospective analysis of COVID-19 patients. The retrospective analysis included plasma samples from 63 COVID-19 patients and 89 control (pre-pandemic) patients. The prospective study included 203 patients who tested either negative (n = 181) or positive (n = 22) by RT-PCR before serology sampling.

Results

The specificity was 92.1 %, 98.9 %, 100 % and 98.9 % and the sensitivity 14 days after onset of symptoms was 95.6 %, 95.6 %, 97.8 % and 95.6 % for Euroimmun IgG, Abbott IgG, Wantai Ab, and DiaPro IgG confirmation SARS-CoV-2 immunoassays, respectively. The low specificity of Euroimmun IgG (for ratio <5) was not confirmed in routine care setting (98.5 % negative agreement). Serology was complementary to RT-PCR in routine care and lead to identification of false positive (Ct>38, <2 targets detected) and false negative RT-PCR results (>1 month post onset of symptoms).

Conclusions

Serology was complementary to RT-PCR for the diagnosis of COVID-19 at least 14 days after onset of symptoms. First line serology testing can be performed with Wantai Ab or Abbott IgG assays, while DiaPro IgG confirmation assay can be used as an efficient confirmation assay.

1. Introduction

The novel severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) firstly reported in late 2019 in Wuhan [1] and causing coronavirus disease (COVID-19) has spread across the world and lead to a worldwide sanitary crisis. Detection of viral RNA using reverse-transcriptase polymerase chain reaction (RT-PCR) in respiratory samples is the gold standard for early diagnosis of COVID-19. However, sensitivity of this molecular diagnosis starts to decrease at week 3 after onset of symptoms [2]. Complementary to RT-PCR in respiratory samples, SARS-CoV-2 serology allows identification of COVID-19 cases with a higher sensitivity than RT-PCR several days after onset of symptoms [3]. In addition, it can be used to determine the fraction of the population that has been exposed to the virus [3]. However, results of such serosurveys depend on the performance of immunoassays and on the seroprevalence of SARS-CoV-2 which remains quite low, even in COVID-19 hotspots [4]. Given this low prevalence, it is crucial to have robust data evaluating those assays before clinical or epidemiological use. Four immunoassays were evaluated in our study: Euroimmun ELISA SARS-CoV-2 IgG (Euroimmun, Lübeck, Germany), Abbott SARS-CoV-2 IgG (Abbott Diagnostics, Illinois, USA), Wantai SARS-CoV-2 Ab ELISA (Beijing Wantai Biological Pharmacy Enterprise, Beijing, China), and DiaPro COVID-19 IgG Confirmation (Diagnostic Bioprobes, Milano, Italy). The latter assay was used to determine the specificity of SARS-CoV-2 Ab against S1, S2 and N Ag. The first aim was to evaluate the performance of these SARS-CoV-2 immunoassays on a series of 63 COVID-19 patients and 89 pre-pandemic control patients. The second aim was to evaluate their contribution in routine care to confirm or infirm the diagnosis of COVID-19.

2. Material and methods

2.1. Patients and samples

Two complementary studies were performed. First, clinical performance of immunoassays were evaluated on 63 COVID-19 patients tested positive for SARS-CoV-2 RNA by RT-PCR at Tours University Hospital (Table 1 ). SARS-CoV-2 RT-PCR were performed in respiratory samples using Allplex™ 2019-nCOV assay (Seegene, Seoul, Republic of Korea), Abbott RealTime SARS-CoV-2 assay (Abbott Molecular, Illinois, USA) or Bosphore 2019-nCoV detection kit (Anatolia GeneWorks, Istanbul, Turkey) depending on reagents and systems availability. Among the positive RT-PCR results, inconclusive RT-PCR results were defined as results positive only for one gene (E, ORF1ab or N). All 63 patients required an in-patient hospital stay for COVID-19 and had plasma samples collected between April 8th and May 11th 2020. Retrospective testing for SARS-CoV-2 Ab was performed on these samples, collected between 2–36 days after the onset of symptoms. Mild and critical COVID-19 cases were defined according to WHO [31]. Specificity was evaluated on plasma collected before the end of 2019 in 89 patients from occupational medicine (n = 30), emergency or pneumology departments (n = 26) or from patients tested positive by RT-PCR (Allplex™ RP3, Seegene) for seasonal coronaviruses (n = 33, OC43, 229E or NL63) between 3–82 weeks before serology sampling. Positive and negative predictive values of immunoassays were estimated in a context of low (2.4 %) and high seroprevalence (9.8 %) of SARS-CoV-2 Ab. These estimates were based on prevalence of SARS-CoV-2 Ab in healthcare professionals (2.4 %, 108/4 444 tested from June to August 2020) or from patients (9.8 %, 6/61 tested from May to June 2020).

Table 1.

Clinical presentation of patients.

	First study (clinical performance)		Second study (routine care)
	RT-PCR+	Pre-pandemic control group	RT-PCR-	RT-PCR+
Nb	63	89	181	22
Age (median/IQR)	79/67−90	30/11−54	39/30−50	49/31−58
Sex (F:M)	1.52	1.17	1.51	1.75
Severe outcome	19/63 (30.2 %)	N/A	N/A	0
ICU	18/63 (28.6 %)	NA	N/A	0
Death	3/63 (4.7 %)	NA	N/A	0

SARS-CoV-2 RT-PCR assays: Allplex™ 2019-nCOV (Seegene), Abbott RealTime SARS-CoV-2 or Bosphore 2019-nCoV (Anatolia GeneWorks); NA: not available; ICU: intensive care unit.

Second, the contribution of SARS-CoV-2 serology in routine care for the diagnosis of COVID-19 was evaluated on 203 patients tested for SARS-CoV-2 infection by RT-PCR between April 8th and June 11th 2020 (Table 1). Most of these patients were healthcare professionals (167/203, 82.3 %) who did not require an in-patient hospital stay (125/167, 74.9 %). Other patients required an in-patient hospital stay (31/203) or had ambulatory testing (6/203). These patients were tested for SARS-CoV-2 antibodies at least 14 days after RT-PCR testing (unless otherwise specified) between June 1st and June 25th 2020. The entire study was performed according to French Reference Methodology MR-004, after patient information and anonymization of data. Samples were obtained from the registered biological collection DC-2020−3961.

2.2. SARS-CoV-2 immunoassays

Euroimmun ELISA SARS-CoV-2 IgG, Abbott SARS-CoV-2 IgG (Alinity-I analyzer), Wantai SARS-CoV-2 Ab ELISA, and DiaPro COVID-19 IgG Confirmation assays were performed according to the manufacturer’s recommendations. Euroimmun IgG assay was used as first line immunoassay in routine care setting. Positive or undetermined results and results discordant with RT-PCR were confirmed with other assays. For statistical analysis, Euroimmun IgG and Wantai Ab uninterpretable results were considered negative. DiaPro IgG confirmation assay was considered positive when Ab against at least two targets (S1, S2 or nucleoprotein) were detected.

2.3. Statistical analysis

Statistical analysis was carried out using Graphpad Prism v5. Comparison of sensitivity and specificity were performed using McNemar’s test. All tests were two-sided at the 0.05 significance level.

3. Results

3.1. Clinical performance of SARS-CoV-2 immunoassays in the retrospective study

The sensitivity of Euroimmun IgG, Abbott IgG, Wantai Ab, and DiaPro IgG confirmation SARS-CoV-2 assays seven to thirteen days after the onset of symptoms were 30.8, 46.2, 84.6 and 61.5 % (Fig. 1 and Table 2). In this timeframe, the DiaPro IgG confirmation demonstrated an excellent sensitivity for anti-N Ab (100 %), higher than that for anti-S2 Ab (15.4 %, p = 0.003) and higher, although not significantly, than that for anti-S1 Ab (53.8 %; p = 0.13).

Fig. 1.

Clinical performance of SARS-CoV-2 immunoassays. A) Comparison of SARS-CoV-2 immunoassays results between COVID-19 patients and control patients. Black arrow indicates the heart transplant patient who tested negative with all immunoassays more than 14 days after onset of symptoms. B) ROC curves for evaluation of performances of SARS-CoV-2 immunoassays for samples ≥14 days post onset of symptoms.

The sensitivity of Euroimmun IgG, Abbott IgG, Wantai Ab and DiaPro IgG confirmation SARS-CoV-2 assays 14 days after the onset of symptoms were 95.6 %, 95.6 %, 97.8 % and 95.6 %, respectively (Fig. 1 and Table 2). The DiaPro IgG confirmation assay demonstrated good and similar sensitivities for anti-S1 and anti–N Ab (93.3 % and 97.8 %), both higher than that for anti-S2 Ab (62.2 %, p ≤ 0.002). The single patient who tested negative with all immunoassays was a 61 years old heart transplant patient who experienced fever and myocarditis and had inconclusive RT-PCR result (positive only for the N gene) 26 days post-onset of symptoms (Fig. 1). This patient could have had a false positive RT-PCR result or, less likely false negative serology results because of its immunosuppressive treatments.

The specificity of Euroimmun IgG, Abbott IgG, Wantai Ab and DiaPro IgG confirmation SARS-CoV-2 immunoassay were 92.1 %, 98.9 %, 100 % and 98.9 %, respectively (Fig. 1 and Table 2). Specificity of the DiaPro IgG confirmation assay was lower for anti-N Ab (84.3 %) than for anti-S1 Ab (95.5 %, p = 0.02) and anti-S2 Ab (100.0 %, p = 0.0005). Specificity of Euroimmun IgG assay was lower than other immunoassays, with a significant difference versus Wantai Ab assay (p = 0.02) but not versus Abbott IgG (p = 0.08). False positive Euroimmun IgG results were observed in in an equivalent manner in the different groups of control patients: those with seasonal coronaviruses infections (2/33, both OC43), those from emergency or pneumology departments (3/26) and those from occupational medicine (2/30). False positive Euroimmun IgG results in control patients were associated with a lower ratio (median of 3.13; IQR 1.90–4.40, maximum 5) than those from the COVID-19 patients (median of 7.60, IQR 3.20–11.14, p = 0.02) (Fig. 1). The two different patients with false positive Abbott IgG (1/89) or DiaPro IgG confirmation (1/89) results had a history of infection with Coronavirus 229E.

Positive predictive values based on a 2.4 % and a 9.8 % prevalence rate were 22.9 %/56.8 %; 68.1 %/90.4 %; 100 %/100 %, and 68.1 %/90.4 % for Euroimmun IgG, Abbott IgG, Wantai Ab and DiaPro IgG confirmation SARS-CoV-2 immunoassays. Negative predictive values based on a 2.4 % and a 9.8 % prevalence rate were 99.9 %/99.5 % for Euroimmun IgG, Abbott IgG and DiaPro IgG confirmation, and 99.9 %/99.8 % for Wantai Ab SARS-CoV-2 immunoassays.

Table 2.

Clinical performance of SARS-CoV-2 immunoassays relative to delay with onset of symptoms.

Assay		Euroimmun IgG	Abbott IgG	Wantai Ab	DiaPro IgG confirmation
Platform		ELISA	CLIA (Alinity-i)	ELISA	ELISA
Antigen		S1	N	S (RBD)	S1	S2	N	≥2 Ag
Sensitivity
≥7−13 dps	n/N	4/13	6/13	11/13	7/13	2/13	13/13	8/13
	%	30.8	46.2	84.6	53.8	15.4	100	61.5
	95 % CI	9.1−61.4	19.2−74.9	54.6−98.1	25.1−80.8	1.9−4.5	75.3−100.0	31.6−86.1
≥14 dps	n/N	43/45	43/45	44/45	42/45	28/45	44/45	43/45
	%	95.6	95.6	97.8	93.3	62.2	97.8	95.6
	95 % CI	84.9−99.5	84.9−99.5	88.2−99.9	81.7−98.6	46.5−76.2	88.2−99.9	84.9−99.5
Specificity	n/N	82/89	88/89	89/89	85/89	89/89	75/89	88/89
	%	92.1	98.9	100.0	95.5	100.0	84.3	98.9
	95 % CI	84.5−96.8	93.9−100.0	95.9−100.0	88.9−98.8	95.9−100.0	75.0−91.1	93.9−100.0

dps: days post-onset of symptoms.

3.2. Contribution of SARS-CoV-2 serology in routine care to confirm or infirm the diagnosis of COVID-19

Agreement between RT-PCR and Euroimmun IgG was 68 % (15/22) and 96 % (173/181) for patients who tested positive and negative by RT-PCR, respectively. Results of other immunoassays (Fig. 2 ), the delay between RT-PCR and serology as well as Ct and targets of RT-PCR results were analyzed for these patients (Fig. 3 ). Patients were considered as suffering from COVID-19 if they tested positive with RT-PCR (all targets positive with Ct<38) and/or with at least two out of four SARS-COV-2 immunoassays (Fig. 2). This allowed an accurate definition of 23 COVID-19 cases and 180 non-COVID-19 patients. It lead to the identification of factors associated with false negative and false positive RT-PCR or serology results.

Fig. 2.

Agreement between RT-PCR and serology for 203 patients.

COVID-19 confirmed if positive RT-PCR (>1 gene) and/or Ab detected with two or more immunoassays.

Fig. 3.

Delay between serology and first RT-PCR or onset of symptoms.

Delay between serology and first RT-PCR (plain dots) or onset of symptoms (empty dots). The dashed line represents early serology testing before 14 days after onset of symptoms.

Among the 23 COVID-19 patients, 18/23 were from occupational medicine (8 in-patient hospital stay, non-critical disease). Overall, positive and negative agreement between RT-PCR and COVID-19 diagnosis was 94.6 % (194/203), 78.3 % (18/23), and 97.8 % (176/180), respectively. Overall, positive and negative agreement between Euroimmun IgG and COVID-19 diagnosis was 97.0 % (197/203), 87.0 % (20/23), and 98.3 % (177/180), respectively.

We considered that false positive RT-PCR results occurred in four patients. They were characterized by inconclusive RT-PCR results. Only the N gene was detected, with Ct>38. These patients tested negative with all immunoassays, suggesting the absence of exposure to SARS-CoV-2. Importantly, such inconclusive RT-PCR results were also observed in two COVID-19 patients confirmed positive with all serological assays (Fig. 2), precluding any systematic interpretation of inconclusive RT-PCR results as false positive.

False negative RT-PCR results were observed in two asymptomatic patients, and in three symptomatic patients with late RT-PCR sampling (>1 month after onset of symptoms). These three symptomatic individuals were positive with all immunoassays, confirming the benefit of serology testing for patients with late presentation after onset of symptoms.

False negative serology results were observed in 2/23 COVID-19 patients who were sampled too early for serology testing (<14 days after onset of symptoms, Fig. 2, Fig. 3). Among the 21 remaining COVID-19 patients tested for antibody to SARS-CoV-2 at least 14 days post onset of symptoms, positive agreement with COVID-19 diagnosis was 90.5 % (19/21), 76.2 % (16/21), 90.5 % (19/21) and 95.2 % (20/21) for Euroimmun IgG, Abbott IgG, Wantai Ab and DiaPro IgG confirmation assay, respectively. The DiaPro IgG confirmation assay had the highest positive agreement with COVID-19 diagnosis. It detected anti-S1, -S2 and –N Ab in 20, 12 and 21 of these 21 patients, respectively. In contrast, 5 out of 21 COVID-19 patients were not detected with the Abbott IgG assay. Interestingly, these five COVID-19 patients were also negative for anti-S2 Ab with the DiaPro IgG confirmation assay (Fig. 2). The low positive agreement of Abbott IgG (16/21) with COVID-19 confirmed cases might suggest a lower sensitivity. However, the data could be due to the small sample size and would deserve to be confirmed in larger studies.

4. Discussion

In the first part of our analysis corresponding to the retrospective evaluation, the specificity was 92.1 %, 98.9 %, 100 %, and 98.9 % for Euroimmun IgG, Abbott IgG, Wantai Ab, and DiaPro IgG confirmation SARS-CoV-2 assays, respectively. The sensitivity 14 days after onset of symptoms was 95.6 %, 95.6 %, 97.8 %, and 95.6 % for Euroimmun IgG, Abbott IgG, Wantai Ab, and DiaPro IgG confirmation SARS-CoV-2 assays, respectively. The sensitivity between 7 and 13 days was suboptimal for Wantai Ab (84.6 %) and inadequate (<75 %) for other assays. Our results are in accordance with the available data for which the sensitivity ranges 14 days after onset of symptoms have been described as 61.7–96.0 %, 77.8–100.0 %, 98–100 % and specificity ranges as 86.6–100.0 %, 95.1–100.0 %, 98.0–99.1 % for Euroimmun IgG [[6], [7], [8], [9], [10], [11], [12], [13], [14], [15], [16], [17], [18], [19]], Abbott IgG [9,10,13,[19], [20], [21]] and Wantai Ab immunoassays [15,22,23], respectively. Although we included a collection of serum samples from patients for whom a recent infection by seasonal coronaviruses was documented, we did not notice any specific clustering of false positive results that could be attributed to a particular cross-reactivity.

Clinical performance of the DiaPro IgG confirmation assay (combining S1, S2 and N Ag) were similar to other immunoassays and to the manufacturer’s statement (sensitivity 98 %, specificity 90 %). Interestingly, a higher sensitivity was observed for anti-N Ab and anti-S1 Ab than for anti-S2 Ab (97.8 and 93.3 vs 62.2 %, p ≤ 0.002). This confirmed previous studies based on in-house ELISAs [24]. In contrast, a higher specificity was observed for anti-S2 Ab (100 %) than for anti-N Ab (84.3 %, p = 0.0005). As suggested by previous studies [25,26], combination of S and N Ag probably contributed to the overall good performances of this assay, which had the highest positive agreement of all immunoassays (20/21) with COVID-19 diagnosis in our sub-study performed in real-life routine conditions. Furthermore, this assay has the advantage to allow comparison of the ratio between anti-N IgG and anti-S IgG, which has been associated with the prognosis [26].

We observed a lack of specificity for the Euroimmun IgG assay (92.1 %, CI 95 %: 84.5−96.8 %) in comparison to other assays in our retrospective study. This low specificity was not so problematic in the routine care setting since 98.5 % of the non-COVID-19 patients tested negative with Euroimmun IgG. There are conflicting data in the literature regarding the specificity of this assay. It demonstrated a good specificity (≥ 95 %) in most studies [7,[10], [11], [12],14,18,19], while a minority of studies suggested otherwise [6,9,17]. In this context and given the low prevalence of SARS-CoV-2 Ab, confirmation of positive Euroimmun IgG results seems reasonable. RT-PCR also demonstrated a relative lack of specificity in routine care setting (negative agreement of 97.8 %, 176/180) especially for inconclusive RT-PCR results (only one gene detected, with Ct>38).

Some false negative serology results were associated with early serology sampling (<14 dps), while all false negative RT-PCR results were associated with late RT-PCR sampling (>30dps). This confirms that timing of testing is critical for good sensitivity of SARS-CoV-2 RT-PCR and serology [3]. However, early serology sampling does not explain the relative lack of agreement with COVID-19 diagnosis for Euroimmun IgG (19/21), Abbott IgG (16/21), Wantai Ab (19/21), and DiaPro IgG confirmation assay (20/21) in routine care setting. Similar observations have been made in populations of healthcare professionals, with false negative rates ranging from 1 % [27] to 20 % [[28], [29], [30]] several weeks after disease. This could be due to the high proportion of patients with mild COVID-19, resulting in low rates of seroconversion [28].

Our study confirms that Abbott IgG, Wantai Ab and DiaPro IgG confirmation assays are suitable assays for the diagnosis of COVID-19 at least 14 days after onset of symptoms. Main advantages of these assays are their automation (Abbott IgG), their optimal clinical performance (Wantai Ab) and their ability to differentiate between anti-N, -S1 and –S2 Ab (DiaPro IgG confirmation). DiaPro IgG confirmation assay has a low throughput (four wells per patient) which is adequate for confirmation testing. Euroimmun IgG assay can also be used for the diagnosis of COVID-19 if positive results are confirmed with another assay to compensate for its low specificity.

The huge impact of the SARS-CoV-2 emergence in public health justifies extensive seroepidemiological studies to survey its spread in various populations and numerous settings. There is a burst of serologic assays rolling out in different formats, including simple rapid tests. Our study shows that specificity may be highly variable among available immunoassays for antibody to SARS-CoV-2. Poor specificity of an assay in a population where prevalence and incidence of COVID-19 are low will lead to irrelevant data. Our study, as others, stresses on the absolute necessity to use only carefully validated assays to provide epidemiological data useful to public health decision makers.

Funding

This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

Contribution

ER carried out the experiment. JM, CP, ER and YA analyzed the data. JM, CP, FB, KS and CGG designed the study and wrote the manuscript with support from AL.

AG, SM, AL, LB, GD and HB contributed to the design, patient management and data collection.