Diagnostic accuracy comparison of three fully automated chemiluminescent immunoassay platforms for the detection of SARS-CoV-2 antibodies

Abstract

Background

Serological test is an essential surveillance tool to track down the extensiveness of SARS-CoV-2 transmission and subsequently to move out from the enforced lockdown stage.

Objective

The study measures the diagnostic accuracy of three popular chemiluminescent immunoassay (CLIA) based automated platforms for the detection of anti-SARS-CoV-2 antibodies and compares their agreements.

Study design

Serum samples of 594 COVID-19 positive patients and 100 samples from pre-COVID cases were tested by three CLIA based automated platforms: Abbott architect i2000SR, Roche cobas e411 and Yhlo iFlash 1800 and their diagnostic accuracy were compared by the area under the curves (AUC) value obtained from receiver operator characteristic (ROC) curves. Cohen’s kappa statistic and McNemar’s test were used to interpret the agreement between the platforms.

Results

All three platforms showed high specificity as claimed by the manufacturer. Sensitivity was calculated as 64.48 % (58.67–70.3) for Abbott, 80.48 % (76.62–84.34) for Roche and 76.94 % (72.65–81.23) for Yhlo. AUC was maximum for Roche (0.929). The Cohen’s kappa value was determined in between 0.69−0.89 as the inter-rater agreements.

Conclusion

The overall statistical analysis demonstrated cobas e411 as the diagnostically most accurate platform among the three.

1. Introduction

Accurate and rapid diagnosis of SARS-CoV-2 infection is needed for prompt and effective patient care. Nasopharyngeal swab (NPS) followed by quantitative reverse-transcriptase polymerase chain reaction (qRT-PCR) is the gold standard for molecular diagnosis of SARS-CoV-2 detection. However, sometimes it fails to demonstrate the complete picture of the viral transmission through communities (Winichakoon et al., 2020; Wu et al., 2020). Hence, the serological test or antibody test is believed to be another important diagnostic tool along with the swab test. This test detects anti-SARS-CoV-2 immunoglobulins which are usually formed in the patient’s body at the earliest by 1 week, and on average within 2–3 weeks from the infection onset. Antibody tests are a useful surveillance tool to track seroprevalence, assess the current immune status of a certain community, and may also be useful for decisions on lockdown entry-exit strategies (Fernández-Barat et al., 2020; Randolph and Barreiro, 2020).

Currently, there are several serological assays available in the market for the detection of anti-SARS-CoV-2 antibodies, mainly based on enzyme-linked immunosorbent assay (ELISA), chemiluminescent immunoassay (CLIA), or qualitative point-of-care tests (POCT) (WHO, 2020). The automated machines based on CLIA technology has a high throughput potential for the detection of anti-SARS-CoV-2 antibodies with high sensitivity and specificity (Bastos et al., 2020). In this study, we compared three such CLIA-based analysers: ARCHITECT i2000SR (Abbott Laboratories, Chicago, USA), Cobas e411 (Roche Diagnostics GmbH, Mannheim, Germany) and iFlash 1800 (Shenzhen Yhlo Biotech Co. Ltd., Shenzhen, China) to detect their diagnostic accuracy and to identify the accurate one.

2. Methods

2.1. Collection of serum sample

Serum samples were collected from recovered COVID-19 patients after 4 weeks and not more than 8 weeks from the first qRT-PCR confirmation of SARS-CoV-2 infection. Oropharyngeal and nasopharyngeal swab specimens from all the suspected patients were tested and confirmed only after positive qRT-PCR results. Signed Informed consent was obtained from each individual before their enrollment in the study. A total of 594 positive cases were chosen for this study between 23rd July 2020 and 14th September 2020. A total of 100 serum samples collected during the pre-COVID period (August 2019) and stored at −20 °C in RMRC, Bhubaneswar repository were used as control. Repeated freeze-thawing were avoided. The study was approved by the Institutional Ethics Committee.

2.2. Test method

ARCHITECT i2000SR platform used chemiluminescent microparticle immunoassay (CMIA) technology for the detection of immunoglobulin class G (IgG) antibodies against the nucleocapsid protein of SARS-CoV-2 from human sera. The second automated machine Cobas e411 was based on electrochemiluminescence (ECL) technology to detect total antibodies against nucleocapsid of SARS-CoV-2. The third one, iFlash 1800 was a paramagnetic particle linked chemiluminescent immunoassay (CLIA) technology to determine the IgG antibodies against SARS-CoV-2 nucleocapsid and spike protein. For details, see the supplementary file.

2.3. Statistical analysis

Descriptive statistical analyses were performed by SPSS software (IBM SPSS Statistics for Windows, version 24.0, Armonk, NY). The agreement between the automated platforms was measured by Cohen’s kappa (ĸ) statistic. McNemar’s test was used to determine the significance of difference out of the three automated chemiluminescent immunoassay platforms. Specificity, sensitivity, negative predictive value (NPV) and positive predictive value (PPV) were calculated for each assay. The area under the curves (AUC) was compared from the receiver operator characteristic (ROC) curves of the three different platforms. p-value <0.05 was considered as statistically significant.

3. Results

A total of 594 samples from recovered COVID-19 patients and 100 pre-COVID serum samples were analysed in all three CLIA platforms. In most of the convalescent COVID-19 patients, positive titres of anti- SARS−COV-2 antibodies were observed for all three assay platforms (Fig. 1 ). Among the 594 confirmed COVID cases, 378 sera samples (63.63 %) were tested as reactive, whereas, 109 samples (18.35 %) showed non-reactivity or negative across all three platforms. The calculated specificity and sensitivity of those anti-SARS-CoV-2 detecting automated platforms were described in Table S1. Abbott showed a specificity of 99 % (95 % confidence interval [CI]: 97.09–100.92 %) and a sensitivity of 64.48 % (95 % CI: 58.67–70.3). The PPV and NPV were determined as 99.74 % (99.25–100.24) and 31.94 % (16.1–47.78), respectively. Specificity for Yhlo machine insert iFlash SARS-CoV-2 IgG assay was 100 %, whereas the sensitivity was calculated as 76.94 % (72.65–81.23). This platform had a PPV of 100 % and NPV of 42.2 (27.67–56.72). Roche Elecsys Anti-SARS-CoV-2 insert recorded the highest sensitivity of 80.48 % (76.62–84.34) compared to the other two chemiluminescent platforms, and the specificity (100 %) was the same with Yhlo machine. The PPV and NPV of Roche analyser were 100 % (100.0−100.0) and 46.3 % (32.3–60.3), respectively (Table S1).

Fig. 1.

A total of 594 COVID-19 samples were collected after 4 weeks of SARS−COV-2 detection and 100 pre-COVID samples were tested as control sera. Antibodies titre median at 95 % confidence interval were shown for Architect i200SR (A), Cobas e411 (B) and iFlash 1800 (C). A compiled receiver operating characteristic (ROC) curve was plotted to compare the diagnostic accuracy (D). Dotted lines are indicating respective cut-off values.

The ROC performance curves (Fig. 1) showed that Roche platform had the highest AUC value of 0.929 (95 % CI: 0.910 – 0.948). Architect i2000SR gave AUC value of 0.863 (95 % CI: 0.836−0.889) and for iFlash 1800, it was found to be 0.897 (95 % CI: 0.875 – 0.920). Inter-rater agreements among Roche, Abbott and Yhlo were calculated and described in Table 1 . The best agreement was found between Roche and Yhlo and the least was detected in Roche vs Abbott. McNemar’s test showed significant differences in detection rate among these three similar assay platforms.

Table 1.

Statistical agreements between three automated platforms for SARS-CoV-2 antibodies. Value of ĸ <0.20 poor agreement, 0.21-0.40 fair agreement, 0.41-0.60 moderate agreement, 0.61-0.80 good agreement, and 0.81-1.00 very good agreement. p-value <0.001 was statistically very significant.

Roche vs Abbott		Abbott		Total
		Positive	Negative
Roche	Positive	380 (54.8 %)	98 (14.1 %)	478 (68.9 %)
	Negative	4 (0.6 %)	212 (30.5 %)	216 (31.1 %)
Total		384 (55.3 %)	310 (44.7 %)	694 (100 %)
Kappa Value		0.694 (0.641−0.746)
McNemar’s Test		p-value <0.001
Total Agreement		85.3 %
Positive Agreement		79.4 %
Negative Agreement		98.1 %

Yhlo vs Abbott		Abbott		Total
		Positive	Negative
Yhlo	Positive	380 (54.8 %)	77 (11.1 %)	457 (65.9 %)
	Negative	4 (0.6 %)	233 (33.6 %)	237 (34.1 %)
Total		384 (55.3 %)	310 (44.7 %)	694 (100 %)
Kappa Value		0.758 (0.709−0.807)
McNemar’s Test		p-value <0.001
Total Agreement		88.3 %
Positive Agreement		83.1 %
Negative Agreement		98.3 %

Roche vs Yhlo		Yhlo		Total
		Positive	Negative
Roche	Positive	451 (65.0%)27 (3.9%)478 (68.9 %)	27 (3.9 %)	478 (68.9 %)
	Negative	6 (0.9 %)	210 (30.3 %)	216 (31.1 %)
Total		457 (65.9 %)	237 (34.1 %)	694 (100 %)
Kappa Value		0.892 (0.856−0.927)
McNemar’s Test		p-value <0.001
Total Agreement		95.2 %
Positive Agreement		94.3 %
Negative Agreement		97.2 %

4. Discussion

Routine antibody tests are necessary to identify the community transmission, to determine the herd immunity status and to screen potential convalescent plasma donors (Goudsmit, 2020; Zhao et al., 2020). To our best knowledge, the concordance of these three popular automated chemiluminescent assay platforms is evaluated for the first time in this study. The 18.35 % of non-reactive results revealed a majority of COVID-19 recovered patients were unable to produce detectable titre of antibodies. Both Roche and Yhlo platform were found to have 100 % specificity, which is more than the manufacturer claimed value. In terms of sensitivity, Roche insert showed the highest sensitivity for anti-SARS-CoV-2 antibodies including IgG compared to both Abbott and Yhlo. A study by Perkmann et al. showed a higher sensitivity for Abbott (84.6 %) and Roche (89.2 %) platforms although that might be because of the low number of recruited COVID-19 patients (n = 65) (Perkmann et al., 2020). Similarly, another study with iFlash SARS-CoV-2 IgG measured sensitivity of 76.7 % with 61 positive sera which are the same (76.9 %) as our result and also corroborated our study with a higher sample size (Infantino et al., 2020).

The AUC values represented the diagnostic accuracy of all three platforms, and Roche gave the highest value of 0.929 at 95 % CI. Inter-rater agreement among those platforms was statistically good and found to be the highest between Roche and Yhlo. As per our study, Roche Cobas e411 automated chemiluminescent platform gave the best diagnostic accuracy. The other two platforms weren’t far behind as determined by the percent agreement from ĸ analysis. Elecsys Anti-SARS-CoV-2 assay is based on the determination of total antibody against nucleocapsid of SAR-CoV-2 whereas iFlash SARS-CoV-2 IgG assay detected IgG against both nucleocapsid and spike proteins. Both the inserts had an agreement of 95.2 % even though they targeted different antigens.

In conclusion, the detection of anti-SARS-CoV-2 antibodies is significantly different across the three different automated chemiluminescent assay platforms. Roche Cobas e411 automated chemiluminescent platform gave the best diagnostic accuracy against the other two systems tested. This is the first such demonstration of these three platforms which would be helpful for further development of epidemiological strategies to contain the COVID-19 pandemic and also in the clinical context.

Authors statement

DB & SP designed the study. DP, HRC, GCD, UKR, AP, RRN, were involved in testing and analysis of data. DP, GCD, HRC, JSK, SK, SD and were responsible for data analysis and valuable inputs. DP and GCD did the statistical analysis. DP and DB wrote the manuscript. All authors have read and approved the final manuscript.

Funding source

The study was carried out with intramural funding support from Indian Council of Medical Research, New Delhi.

Ethics approval and consent to participate

The study was approved by the Institutional Ethical Committee of ICMR-Regional Medical Research Centre, Bhubaneswar.

Declaration of Competing Interest

The authors report no declarations of interest.

Appendix A. Supplementary data

The following is Supplementary data to this article: