Abstract
Objectives
This study was aimed at providing some insights into the real-life performance of the commercial, clinically validated anti-SARS-CoV-2 antibody assays.
Methods
The residual, anonymized samples from 97 patients referred for anti-SARS-CoV-2 antibodies testing were included in the study. The initial assessment was performed with the Euroimmun ELISAs, followed by the assays provided by: NovaTec, Snibe, Vircell, Roche, Abbott and DiaSorin. The analyses of the results were performed separately for the antibodies of the early (IgM/IgA) and late (IgG) immune response.
Results
We observed a high variability of the results obtained with the investigated immunoassays. The fully concordant results were reported for only 57 out of 97 samples tested for IgG antibodies and for 34 out of 97 samples for IgM/IgA. The highest percentage of positive results was noted for the Euroimmun and Vircell ELISAs and the lowest for Novatec ELISAs.
We proposed to distinguish true and false positive results based on the sum of positive results obtained with different methods. We arbitrarily considered reference positive samples reactive in at least half of the assays. The assay that proved to correlate the best with those reference results was the Roche electrochemiluminescence immunoassay.
Conclusions
The differences observed between immunoassays targeting the early phase antibodies were much more pronounced than between IgG assays, suggesting their lower value for clinical use. Our study also showed a high percentage of plausibly false (positive or negative) results obtained with ELISAs, which suggests their inferiority to the automated immunoassays.
Keywords: SARS-CoV-2 antibodies, COVID-19 immunoassays, ELISA, CLIA, CMIA
1. Introduction
The SARS-CoV-2 pandemic announced by WHO on March 11, 2020 took 1 million lives worldwide by the end of September 2020 [1]. The long-term complications of the disease and the outcomes of other conditions not being properly treated have been recognized. Therefore, medical devices designed to prevent, treat and properly diagnose SARS-CoV-2 infection are needed.
Although there is no doubt on RT-PCR being the reference method for SARS-CoV-2 infection diagnosis, some limitations of this type of testing, as well as the need for diagnosing the late-phase or past infection, urged the development of serological kits for detecting anti-SARS-CoV-2 antibodies [2]. These may also be useful in the future studies on vaccines efficacy, immunity assessment and in convalescent plasma treatment [3].
Initial approach of the producers was towards the development of the rapid immunochromatography tests, detecting qualitatively anti-SARS-CoV-2 IgM and IgG antibodies. As the feedback from the scientists and the medical community on the accuracy of these tests was not fully optimistic [[4], [5], [6]], and simultaneously the demand for the serological testing on the market grew, the attention was shifted towards better validated, automated, high through-put systems for semi-quantitative or quantitative assessment of the anti-SARS-CoV-2 antibodies. Currently there are dozens of immunoassays available.
As an aid in choosing the appropriate test, the laboratories may compare the results obtained with different methods. Since there is no reference antibody test available for SARS-CoV-2, our study was designed to provide a comparison between seven widely available automated or semi-automated immunoassays, to establish whether there is a relationship between their results and to attempt to indicate the methods that seem to be the most accurate.
2. Material and methods
2.1. Patients and serum samples
This study included residual sera from patients who had been referred to the central laboratory of Poland-wide network of medical laboratories, Diagnostyka for anti-SARS-CoV-2 assessment. The samples were tested with the Euroimmun Anti-SARS-CoV-2 ELISA IgG and IgA assays and based on the results 97 samples were chosen to cover all the possible constellations of antibody classes results (36% IgA-IgG-; 23% IgA + IgG-; 31% IgA + IgG+; 10% IgA-IgG+). The samples were anonymized, aliquoted and stored frozen prior to the further testing with the other investigated methods.
The comparison was performed separately for the results obtained for antibody classes related to the early humoral response (IgA or IgM) and to the late response (IgG). The former included methods provided by: Euroimmun, NovaTec, Snibe, Vircell and Roche, and the latter tests manufactured by Euroimmun, NovaTec, Snibe, Vircell, Abbott, DiaSorin and Roche.
2.2. Serological assays
All the investigated methods were performed strictly to the manufacturers’ instructions.
The Euroimmun’s anti-SARS-CoV-2 IgG and IgA kits (Euroimmun, Germany) are enzyme-linked immunosorbent assays (ELISAs) and were performed on a fully-automated ELISA system EuroLabWorkstation 45. The Euroimmun’s ELISAs provide a semiquantitative determination of IgA and IgG antibodies against the SARS-CoV-2 S1 antigen. The results are reported as a ratio of the extinctions of samples over the calibrator. The ratio interpretation is as follows: <0.8 = negative, ≥0.8 to <1.1 = borderline, ≥1.1 = positive. The borderline Euroimmun’s results were not included in this study.
The NovaTec’s NovaLisa® SARS-CoV-2 IgA, IgM and IgG ELISA kits (NovaTec Immundiagnostica GmbH, Germany) were performed on a fully-automated ELISA system ETI-MAX 3000 (STRATEC Biomedical Systems AG, Germany). These ELISAs are intended for the qualitative determination of IgA, IgM and IgG class antibodies against SARS-CoV-2 N protein [7]. The results are expressed in NTU (NovaTec Units) and are calculated as a ratio of sample and cut-off control absorbances. The ratio interpretation is as follows: <9 – negative, 9–11 – equivocal, >11 – positive.
The Vircell’s COVID-19 IgG and IgM + IgA ELISA kits (Vircell, S.L., Spain) were performed on a fully-automated ELISA system ETI-MAX 3000 (STRATEC Biomedical Systems AG, Germany). These COVID-19 ELISAs are intended for the semiquantitative determination of IgM + IgA and IgG class antibodies against both spike (S) and nucleocapsid (N) SARS-CoV-2 proteins [8]. The results are expressed as indices, calculated as a ratio of sample and cut-off control optical densities. The ratio interpretation for IgM + IgA is as follows: <6 – negative, 6–8 – equivocal, >8 – positive; and for IgG: <4 – negative, 4–6 – equivocal, >6 – positive.
The MAGLUMI 2019-nCoV IgM and IgG assays (Snibe Diagnostic, China) are chemiluminescence immunoassays (CLIAs) and were performed on MAGLUMI 800 analyzer (Snibe Diagnostic, China). The assays provide qualitative assessment of anti-SARS-CoV-2 IgM and IgG. The antibodies detected in these assays are directed against both S and N antigens [9]. The results are expressed as AU/ml and the thresholds of positivity are 1.0 AU/mL for IgM and 1.1 AU/mL for IgG.
The Abbott’s SARS-CoV-2 IgG assay (Abbott, Ireland) is a chemiluminescent microparticle immunoassay (CMIA) for qualitative detection of IgG against the SARS-CoV-2 nucleoprotein. The testing was performed on Abbott Architect i2000sr analyzer. The results are expressed as indices, calculated as a ratio of sample and calibrator signals. The threshold of positivity is 1.4.
The LIAISON®SARS-CoV-2 S1/S2 IgG (DiaSorin S.p.A, Italy) assay is an indirect CLIA and as the only one of those included in this study allows for quantitative determination of S1- and S2- SARS-CoV-2 antigen-specific IgG antibodies. The testing was performed on Liaison XL analyzer. The results are expressed in arbitrary units and interpreted as follows: <12 – negative; ≥12 - <15 – equivocal; ≥15 – positive.
The Elecsys Anti-SARS-CoV-2 (Roche Diagnostics GmbH, Germany) is a electrochemiluminescence immunoassay (ECLIA) and allows for qualitative determination of total (IgM and IgG) mature antibodies against the SARS-CoV-2 nucleoprotein. The testing was performed on Cobas 8000 analyzer. The results are expressed in a form of a cutoff index (COI; signal sample/cutoff) and interpreted as follows: <1.0 - non-reactive; ≥1.0 – reactive.
For the further analyses, all the equivocal results were considered positive.
2.3. Statistical analysis
The statistical analysis was performed with STATISTICA software ver. 12. For the purpose of this study we compared the negative/positive interpretation of the obtained measurements. The significance level was set to 0.05.
Q Cochrane test and Yates’s chi-squared test were used to assess the significance of the differences observed in the positivity rates of the investigated methods. Dunn’s test with the Bonferroni adjustment was used as a post-hoc test for Q Cochrane test. The correlation between the results obtained with different immunoassays was assessed with r Spearman.
Ethical approval
The conducted research is not related to either human or animals experimentational use.
3. Results
3.1. Anti-SARS-CoV-2 IgG antibodies testing
97 samples tested with the Euroimmun kit were further assessed with 6 different anti-SARS-CoV-2 IgG immunoassays. In comparison to 40 samples testing positive with the Euroimmun IgG assay, 41 (including 3 equivocal) were positive in Vircell IgG, 22 (including 2 equivocal) in DiaSorin IgG, 20 in Abbott IgG, 19 in Roche (detecting IgM as well as IgG antibodies), 18 in Snibe IgG and 17 in NovaTec IgG tests (Fig. 1).
Fig. 1.
The number of positive results obtained with different anti-SARS-CoV-2 IgG immunoassays.
The reported numbers of positive samples were significantly (p < 0.0001, Q Cochrane test) different, and the post-hoc Dunn’s test with the Bonferroni adjustment revealed statistically significant differences between the results obtained with the Euroimmun IgG or Vircell IgG and the rest of the methods investigated.
16 samples were sustainably positive in all the methods tested. One sample was negative only in NovaTec IgG ELISA and two samples were negative only in NovaTec IgG and Snibe IgG (Table 1). Hence, 19 samples testing positive in at least 5 different assays were considered reference positive. There were no samples positive in 4 assays. Since there was no equivocal results in the reference positive group, all primarily equivocal results re-interpreted according to the reference results were negative.
Table 1.
The different patterns of anti-SARS-CoV-2 IgG results in the tested cohort.
| Number of samplesa | Euroimmun IgG | Vircell IgG | DiaSorin IgG | NovaTec IgG | Roche IgG + IgM | Abbott IgG | Snibe IgG | Number of positive results | Reference result |
|---|---|---|---|---|---|---|---|---|---|
| 16 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 7 | 1 |
| 1 | 1 | 1 | 1 | 0 | 1 | 1 | 1 | 6 | 1 |
| 2 | 1 | 1 | 1 | 0 | 1 | 1 | 0 | 5 | 1 |
| 1 | 1 | 1 | 1 | 0 | 0 | 0 | 0 | 3 | 0 |
| 1 | 0 | 1 | 0 | 1 | 0 | 1 | 0 | 3 | 0 |
| 1 | 1 | 0 | 1 | 0 | 0 | 0 | 0 | 2 | 0 |
| 5 | 1 | 1 | 0 | 0 | 0 | 0 | 0 | 2 | 0 |
| 1 | 0 | 1 | 0 | 0 | 0 | 0 | 1 | 2 | 0 |
| 1 | 0 | 1 | 1 | 0 | 0 | 0 | 0 | 2 | 0 |
| 14 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 0 |
| 13 | 0 | 1 | 0 | 0 | 0 | 0 | 0 | 1 | 0 |
| 41 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
Each verse represents a group of cases with the same pattern of IgG results. The reference result is based on the sum of positive results obtained for a given sample. Reference positive were considered those results, that were positive in at least half of the immunoassays. “1” – positive result; “0” – negative result.
The remaining 78 samples, which tested negative in at least 4 out of seven immunoassays, were considered reference negative. 41 out of 97 samples were invariably negative in all the assays. 14 samples tested positive only in the Euroimmun IgG ELISA and 13 were positive only in the Vircell IgG ELISA. Out of 8 samples that tested positive in two immunoassays, 5 were positive in the Euroimmun IgG and Vircell IgG and the remaining 3 were positive in the Euroimmun IgG or Vircell IgG and one other method (Table 1). Only 2 samples tested were positive in 3 different methods, and for one of them two equivocal results had been reported. Taking into consideration the proposed interpretation of positivity (half of the tests positive) the numbers of falsely positive results were as follows: 21 for Euroimmun IgG, 22 for Vircell IgG (including 3 equivocal), 3 for Diasorin IgG (including 2 equivocal) and 1 each for Abbott IgG, NovaTec IgG and Snibe IgG. Roche produced no false positive results. The false negative results were obtained only with the NovaTec IgG (3 results) and Snibe IgG (2 results) assays (Table 1). Percent positive agreement (PPA) with our arbitrary reference results for all the other methods tested was 100% (Table 2).
Table 2.
The agreement between the investigated serological assays and the anti-SARS-CoV-2 IgG reference result.
| Reference Result Agreement | Percent Positive Agreement (PPA) | Percent Negative Agreement (PNA) | Percent Overall Agreement (POA) |
|---|---|---|---|
| Euroimmun IgG | 100.0% | 73.1% | 78.4% |
| Vircell IgG | 100.0% | 71.8% | 77.3% |
| Diasorin IgG | 100.0% | 96.2% | 96.9% |
| NovaTec IgG | 84.2% | 98.7% | 95.9% |
| Roche | 100.0% | 100.0% | 100.0% |
| Abbott IgG | 100.0% | 98.7% | 99.0% |
| Snibe IgG | 89.5% | 98.7% | 96.9% |
The percent negative agreement (PNA) between the tested methods and the reference results was 100% only for the Roche method. However, PNA was very high (above 95%) also for Diasorin IgG, NovaTec IgG, Snibe IgG and Abbott IgG. The lowest (below 80%) overall agreement (POA) with the reference result was observed for the Euroimmun IgG and Vircell IgG.
The analysis of anti-SARS-CoV-2 IgG testing results showed the statistically significant associations between the results of each of the immunoassays and the reference results. The strength of these associations was assessed with the Spearman correlation coefficient (r) (Table 3). Perfect correlation was found for the Roche test, almost perfect for Abbott IgG, Diasorin IgG and Snibe IgG, and very high for Novatec IgG. Euroimmun IgG and Vircell IgG correlated with the reference method much weaker, due to the low PNA.
Table 3.
The correlation between the investigated methods and the reference anti-SARS-CoV-2 IgG results, as assessed with r Spearman correlation test.
| Comparison | Significance level p | Correlation r | Strenght of correlation r |
|---|---|---|---|
| Reference result vs IgG Vircell | p < 0.0001 | r = 0.5768 | High |
| Reference result vs IgG Euroimmun | p < 0.0001 | r = 0.5891 | High |
| Reference result vs IgG Novatec | p < 0.0001 | r = 0.8656 | Very high |
| Reference result vs IgG Snibe | p < 0.0001 | r = 0.9003 | Almost perfect |
| Reference result vs IgG Diasorin | p < 0.0001 | r = 0.9112 | Almost perfect |
| Reference result vs IgG Abbott | p < 0.0001 | r = 0.9684 | Almost perfect |
| Reference result vs Roche | p < 0.0001 | r = 1 | Perfect |
Anti-SARS-CoV-2 IgG antibodies testing.
The same 97 samples were tested with the Euroimmun anti-SARS-CoV-2 IgA, followed by 5 different immunoassays. Among these: one test assessing IgA as well (NovaTec), two assays strictly for IgM antibodies (a different NovaTec kit and Snibe), one combining IgA and IgM (Vircell) and one combining IgM and IgG (Roche) (the same test was investigated in the IgG analysis).
Similarly to the IgG assays, the highest positivity rate was observed in the Euroimmun and Vircell assays [52 and 39 (including 6 equivocal) positive cases, respectively]. The lowest number of positive results was obtained with both NovaTec assays: 6 positive (including 2 equivocal) for IgM and 6 positive (including 1 equivocal) for IgA. Snibe IgM test gave 20 and Roche (IgM/IgG) 19 positive results (Fig. 2).
Fig. 2.
The number of positive results obtained with different anti-SARS-CoV-2 IgM/IgA immunoassays.
The differences between the tests were statistically significant (p < 0.0001), as assessed by Q Cochrane test and the post-hoc Dunn’s test with the Bonferroni adjustment revealed statistically significant differences only between the results obtained with the Euroimmun IgA or Vircell IgM/IgA and the rest of the methods investigated (similarly to the IgG tests). No stronger associations were observed between the assays detecting the same antibody class than between the tests analyzing the early response in different classes, which proves that the inclusion of both IgM and IgA into one analysis was correct. The Roche assay, that detects both IgM and IgG antibodies, reported the positive results in the numbers similar to the other methods, with the exception of Euroimmun IgA and Vircell IgM/IgA (Fig. 2).
Out of 97 samples only two were positive in all of the immunoassays (Table 4).
Table 4.
The different patterns of anti-SARS-CoV-2 IgM/IgA results in the tested cohort.
| Number of samplesa | Euroimmun IgA | Vircell IgM + A | NovaTec IgM | NovaTec IgA | Snibe IgM | Roche IgM + IgG | Number of positive samples | Reference Result |
|---|---|---|---|---|---|---|---|---|
| 2 | 1 | 1 | 1 | 1 | 1 | 1 | 6 | 1 |
| 2 | 1 | 1 | 1 | 0 | 1 | 1 | 5 | 1 |
| 1 | 1 | 1 | 0 | 1 | 1 | 1 | 5 | 1 |
| 4 | 1 | 1 | 0 | 0 | 1 | 1 | 4 | 1 |
| 2 | 1 | 1 | 0 | 1 | 0 | 1 | 4 | 1 |
| 1 | 1 | 0 | 0 | 0 | 1 | 1 | 3 | 1 |
| 5 | 1 | 1 | 0 | 0 | 0 | 1 | 3 | 1 |
| 6 | 1 | 1 | 0 | 0 | 1 | 0 | 3 | 1 |
| 1 | 0 | 1 | 1 | 1 | 0 | 0 | 3 | 1 |
| 1 | 0 | 1 | 0 | 0 | 0 | 1 | 2 | 0 |
| 1 | 1 | 0 | 0 | 0 | 1 | 0 | 2 | 0 |
| 1 | 0 | 1 | 1 | 0 | 0 | 0 | 2 | 0 |
| 8 | 1 | 1 | 0 | 0 | 0 | 0 | 2 | 0 |
| 1 | 0 | 0 | 0 | 0 | 0 | 1 | 1 | 0 |
| 3 | 0 | 0 | 0 | 0 | 1 | 0 | 1 | 0 |
| 6 | 0 | 1 | 0 | 0 | 0 | 0 | 1 | 0 |
| 20 | 1 | 0 | 0 | 0 | 0 | 0 | 1 | 0 |
| 32 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
Each verse represents a group of cases with the same pattern of IgM/IgA results. The reference result is based on the sum of positive results obtained for a given sample. Reference positive were considered those results, that were positive in at least half of the immunoassays. “1” – positive result; “0” – negative result.
Another 2 samples were positive in all but NovaTec IgA assay, one sample was only negative in NovaTec IgM, four samples were negative only in both NovaTec tests and two were negative only in the Novatec IgM and Snibe tests. Among the above, there were two samples testing equivocal in NovaTec assays and hence presumably true positive samples. The above 11 samples were positive in more than half of the tests and together with the samples testing positive in half (3 out of 6) of the investigated methods comprised a group of 24 samples arbitrarily considered reference positive (Table 4). Two of the samples included in the reference positive group were equivocal in Vircell.
The remaining 73 samples were considered reference negative. Among them, only 32 samples were negative in all tests investigated and 30 were positive in one test studied. If reference negative samples tested positive in some assays, it was mostly in Euroimmun or Vircell test: 20 samples were positive only in the Euroimmun IgA ELISA, 6 only in Vircell IgM/IgA and 8 tested positive in Euroimmun and Vircell. 3 samples were positive only in the Snibe test and one only in Roche (possibly due to an isolated IgG presence). Among the reference negative samples, there were 4 originally testing equivocal in Vircell and one in NovaTec, presumably negative according to reference results.
Taking into consideration the proposed interpretation of positivity (half of the tests positive), the numbers of falsely positive results were as follows: 29 for Euroimmun IgA, 16 (including 4 equivocal) for Vircell IgM/IgA, 4 for Snibe IgM, 1 for NovaTec IgM and 1 for Roche. Novatec IgA produced no false positive results. The numbers of falsely negative results were as follows: Euroimmun IgA and Vircell IgM/IgA - 1 each, NovaTec IgM 19 and IgA 18, Snibe IgM 8 results and Roche 7 (Table 4). Percent positive agreement (PPA) with the reference results was the highest for the Euroimmun IgA and Vircell IgM/IgA (95.83 for both) but the percent negative agreement (PNA) of these methods with the reference result was very low (60.72 and 78.08 respectively). The contrary was shown for both NovaTec tests: high PNA (100 for IgA and 98.63 for IgM) but low PPA (25.00 for IgA and 20.83 for IgM). Snibe IgG and Roche had the highest overall agreement (87.63 and 90.72% respectively) (Table 5).
Table 5.
The agreement between the investigated serological assays and the reference result of anti-SARS-CoV-2 IgM/IgA.
| Reference Result Agreement | Percent Positive Agreement (PPA) | Percent Negative Agreement (PNA) | Percent Overall Agreeement (POA) |
|---|---|---|---|
| Euroimmun IgA | 95.8% | 60.7% | 69.1% |
| Vircell IgM/A | 95.8% | 78.1% | 82.5% |
| NovaTec IgM | 20.8% | 98.6% | 79.4% |
| NovaTec IgA | 25.0% | 100.0% | 81.4% |
| Snibe IgM | 66.7% | 94.5% | 87.6% |
| Roche | 70.8% | 97.3% | 90.7% |
All of the assays correlated with the reference results, but the correlation was much weaker than in the IgG class testing and none of the methods reached perfect or nearly perfect association. The Roche assay reached a very strong correlation with the reference results (p = 0.0001, r = 0.7403) and strong correlation was observed for Snibe IgM and Vircell IgM/IgA (Table 6). Moderate correlation was noted for the Euroimmun IgA and NovaTec (IgA and IgM) assays.
Table 6.
The correlation between the investigated methods and the reference anti-SARS-CoV-2 IgM/IgA results, as assessed with Spearman correlation test.
| Comparison | Significance level p | Correlation r | Strenght of correlation r |
|---|---|---|---|
| Reference result vs IgM Novatec | p = 0.00323 | r = 0.3486 | Moderate |
| Reference result vs IgA Novatec | p < 0.0001 | r = 0.4478 | Moderate |
| Reference result vs IgA Euroimmun | p < 0.0001 | r = 0.4854 | Moderate |
| Reference result vs IgA/IgM Vircell | p < 0.0001 | r = 0.6505 | High |
| Reference result vs IgM Snibe | p < 0.0001 | r = 0.6526 | High |
| Reference result vs Roche | p < 0.0001 | r = 0.7403 | Very high |
The re-analysis of the results according to the proposed interpretation of positivity both in IgG and IgM/A classes, revealed a considerable switch in the number of negative samples. In comparison to the initial results obtained with the Euroimmun ELISAs, the number of positive results dropped (from 30 to 17 cases positive in both classes; from 22 to 7 positive solely in the early phase antibodies; and from 10 to 2 positive only in the IgG), in favor of the increase in the negative results (from 35 to 71).
4. Discussion
This study shows a high discordance of the results obtained with the different immunoassays used for SARS-CoV-2 antibodies testing. Out of 97 samples, only 57 (58%) tested with the same result in all the investigated methods of IgG assessment. Even higher variability was noted for IgM/IgA immunoassays, with agreeing results being reported in only 34 out of 97 samples (35%).
There are many different reasons for such discordance. One of them might be the different specificity of the antibodies detected by the immunoassays investigated. Some of them include recombinant nucleocapsid (N) SARS-CoV-2 protein (Abbott, Roche, NovaTec), and the others spike (S) SARS-CoV-2 protein. The latter distinguish further to S1 protein (Euroimmun) and S1/S2 protein (DiaSorin). Snibe and Vircell immunoassays are based on both proteins.
The nucleocapsid proteins display higher homology between coronaviruses than spike antigens [10], what may lead to the higher percentage of the false positive results obtained with the immunoassays employing SARS-CoV-2 N protein. The immunoassays based on S protein have been shown to be more specific [10,11] and may last longer [12]. It is expected that the assays based on SARS-CoV-2 N antigen are more sensitive due to the abundance of this antigen [13], although the contrary – higher sensitivity of S-based assays, has also been reported [14]. Some authors are therefore analyzing the results obtained with different immunoassays with distinction to the exact antigenic specificity of the antibodies detected. In our study the Euroimmun and Vircell assays, based on S1 antigen and on both antigens, respectively, had more positive results than the other tests – no matter if using solely N antigen (Roche, Abbott), both antigens (Snibe, Vircell) or S1/S2 antigens (Diasorin). This suggests that the specificity and sensitivity of these immunoassays depend less on the antigen used than on the other features of the tests.
The correlations between reference results and ELISA methods were much weaker than between the reference results and the automated methods in both IgG response (r between 0.57 and 0.86 vs 0.9–1) and in the early phase antibodies (r = 0.35–0.65 vs 0.65–0.74). ELISA methods were available earlier in the pandemic, but currently accessible automatic methods seem to be better validated. We had made similar observations on the Euroimmun’s high rate of potentially false positive results in our previous study. We had compared the results obtained with the Euroimmun ELISA to the Snibe CLIA method and lateral flow immunoassays (LFIAs) [6]. We reported a higher number of the positive results obtained with the Euroimmun’s ELISAs than with the other methods, especially in the IgA class. We must nonetheless mention the study of Kohmer et al. who reported a low cross-reactivity of the Euroimmun and Vircell’s IgG ELISAs, with their respective specificities of 96.2 and 83.3% [15]. Another evaluation study including the Euroimmun ELISAs reported its high specificity of 91.9% for IgG but much lower for IgA (73%) [16]. The disadvantage of the ELISA methods was also a high number of equivocal results.
Having obtained disagreeing results with different methods, we decided to re-interpret the results based on the sum of the positive results reported by the investigated assays. This approach is similar to the orthogonal testing algorithm suggested by the Food and Drug Administration (FDA) [17] and postulated in the literature [18,19]. Since there is no confirmatory test available, it is advised to confirm a positive result of one immunoassay with the second test. Ideally, the tests should differ in the measurement system or the exact antibody specificity.
The distinction between reference positive and negative results was clearer and hence seems more reliable for IgG than for IgM/IgA testing. The samples considered truly positive for IgG were positive in 5, 6 or 7 immunoassays and there were no samples positive in 4 assays.
In the case of IgM/IgA antibodies, the distinction between true positive and false positive results was less obvious, since 13 out of 24 samples considered reference positive tested positive in only half of the assays investigated. What is more, 6 out of these 13 cases were positive in the Roche assay, which detects both IgM and IgG mature antibodies. These samples were among those considered reference positive in the IgG testing, which suggests that these patients were tested in the late phase of SARS-CoV-2 infection. The other 7, positive for IgM/IgA but negative in IgM/IgG Roche assay might have been tested in early stage of the infection when Roche IgM/IgG test is not sensitive enough, as it is designed to detect only mature antibodies.
The challenge that must be faced is how to choose a test or tests appropriate for a given laboratory. On the basis of the producer-declared high sensitivity (>90%) and specificity (95%) the FDA issues Emergency Use Authorization (EUA). The FDA’s EUA list [20] includes (as of October 06, 2020) a few immunoassays included in this study: Abbott IgG, DiaSorin IgG, Euroimmun IgG and Roche. Substantial efforts have also already been put into additional validation studies performed in clinical or research laboratories. Some of the results have already been published [[7], [8], [9],[21], [22], [23], [24]] and seem to prove the performance of the immunoassays as declared by the producers, although sometimes at adjusted cut-offs. However, the performance of the test may be significantly different in the real-life setting of a particular ambulatory laboratory. Proved in our study variable results obtained with the methods already approved on the market and clinically validated, point to the necessity of their cautious implementation. Similar issue was reported by Brochot et al. who indicated the high sensitivity and agreement between different serological methods when used in hospitalized, COVID-19 confirmed cases, but dropping significantly in patients with no proved SARS-CoV-2 infection [25]. Also, Trabaud et al. reported that the rate of positivity varied between different immunoassays, and was influenced by the clinical presentation of the infection [26].
We aware that the biggest limitation of our study is the lack of any clinical information about the patients – if they were diagnosed with COVID-19 or not, and if so, then how long before the blood testing, and what was the clinical presentation. However, the sole agreement, or the lack of thereof, between the assays may provide some guidance for ambulatory laboratories.
Overall, we may conclude, that the agreement between the immunoassays assessing anti-SARS-CoV-2 IgG antibodies is higher than that between the immunoassays assessing anti-SARS-CoV-2 IgM/IgA classes. The issue of high cross-reactivity of IgM antibodies, leading to their compromised specificity and false positive results is well known for other laboratory tests and has also been reported specifically in anti-SARS-CoV-2 antibodies immunoassays [11,13]. In our opinion, the tests provided by Euroimmun and Vircell bear quite a high risk of false positive results and should be verified by a different method. NovaTec IgA and IgM ELISAs seem to not be sensitive enough and therefore producing a risk of false negative results. Roche test seems to be the most accurate with the high overall agreement for both classes of antibodies, followed by Snibe assay for IgM and Abbott, Diasorin and Snibe for IgG tests.
Research funding
None declared.
Author contributions
All authors have accepted responsibility for the entire content of this manuscript and approved its submission.
Declaration of competing interest
Authors state no conflict of interest.
Contributor Information
Jakub Swadźba, Email: jakub.swadzba@diag.pl.
Maciej Bednarczyk, Email: maciej.bednarczyk@diag.pl.
Tomasz Anyszek, Email: Tomasz.anyszek@diag.pl.
Danuta Kozlowska, Email: danuta.kozlowska@diag.pl.
Andrzej Panek, Email: Andrzej.panek@diag.pl.
Emilia Martin, Email: emilia.martin@diag.pl.
References
- 1.WHO Weekly epidemiological update coronavirus disease 2019 (COVID-19) 5 october 2020. 2020. https://www.who.int/docs/default-source/coronaviruse/situation-reports/20201005-weekly-epi-update-8.pdf accessed 06.10.
- 2.WHO Laboratory Testing for Coronavirus Disease (COVID-19) in Suspected Human Cases: Interim Guidance. https://apps.who.int/iris/handle/10665/331501 19 March 2020. World Health Organization.
- 3.Lerner A.M., Eisinger R.W., Lowy D.R., Petersen L.R., Humes R., Hepburn M. The COVID-19 serology studies workshop: recommendations and challenges. Immunity. 2020;53:1–5. doi: 10.1016/j.immuni.2020.06.012. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Lassaunière R., Frische A., Harboe Z.B., Nielsen A.C.Y., Fomsgaard A., Krogfelt K.A. medRxiv preprint; 2020. Evaluation of Nine Commercial SARS-CoV-2 Immunoassays.https://www.medrxiv.org/content/10.1101/2020.04.09.20056325v1 [Epub ahead of print] [Google Scholar]
- 5.Adams E.R., Ainsworth M., Anand R., Andersson M.I., Auckland K., Baillie J.K. medRxiv preprint; 2020. Antibody Testing for COVID-19: A Report from the National COVID Scientific Advisory Panel.https://www.medrxiv.org/content/10.1101/2020.04.15.20066407v2 [Epub ahead of print] [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Swadźba J., Bednarczyk M., Anyszek T., Martin E. A comparison of 7 commercial anti-SARS-CoV-2 antibodies immunoassays. Arch. Med. Sci. 2020 doi: 10.5114/aoms.2020.98361. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Tré-Hardy M., Wilmet A., Beukinga I., Favresse J., Dogné J.M., Douxfils J. Analytical and clinical validation of an ELISA for specific SARS-CoV-2 IgG, IgA, and IgM antibodies. J. Med. Virol. 2020 doi: 10.1002/jmv.26303. 10.1002/jmv.26303 [Epub ahead of print] [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Kohmer N., Westhaus S., Rühl C., Ciesek S., Rabenau H.F. Clinical performance of different SARS-CoV-2 IgG antibody tests. J. Med. Virol. 2020 doi: 10.1002/jmv.26145. 10.1002/jmv.26145 [Epub ahead of print] [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Lippi G., Salvagno G.L., Pegoraro M., Militello V., Caloi C., Peretti A. Assessment of immune response to SARS-CoV-2 with fully automated MAGLUMI 2019-nCoV IgG and IgM chemiluminescence immunoassays. Clin. Chem. Lab. Med. 2020;58:1156–1159. doi: 10.1515/cclm-2020-0473. [DOI] [PubMed] [Google Scholar]
- 10.Okba N., Müller M.A., Li W., Wang C., GeurtsvanKessel C.H., Corman V.M. Severe acute respiratory syndrome coronavirus 2-specific antibody responses in coronavirus disease 2019 patients. Emerg. Infect. Dis. 2020;26:1478–1488. doi: 10.3201/eid2607.200841. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Tang Y.W., Schmitz J.E., Persing D.H., Stratton C.W. The laboratory diagnosis of COVID-19 infection: current issues and challenges. J. Clin. Microbiol. 2020;58:e00512–e00520. doi: 10.1128/JCM.00512-20. ([Epub ahead of print]) [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Seow J., Graham C., Merrick B., Acors S., Steel K.J.A., Hemmings O. Longitudinal evaluation and decline of antibody responses in SARS-CoV-2 infection. medRxiv preprint. 2020 doi: 10.1101/2020.07.09.20148429. ([Epub ahead of print]) [DOI] [Google Scholar]
- 13.To K.K., Tsang O.T., Leung W.S., Tam A.R., Wu T.-C., Lung D.C. Temporal profiles of viral load in posterior oropharyngeal saliva samples and serum antibody responses during infection by SARS-CoV-2: an observational cohort study. Lancet Infect. Dis. 2020;20:565–574. doi: 10.1016/S1473-3099(20)30196-1. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Kontou P.I., Braliou G.G., Dimou N.L., Nikolopoulos G., Bagos P.G. Antibody tests in detecting SARS-CoV-2 infection: a meta-analysis. Diagnostics. 2020;10(5):319. doi: 10.3390/diagnostics10050319. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Kohmer N., Westhaus S., Rühl C., Ciesek S., Rabenau H.F. Brief clinical evaluation of six high-throughput SARS-CoV-2 IgG antibody assays. J. Clin. Virol. 2020;129:104480. doi: 10.1016/j.jcv.2020.104480. [Epub ahead of print] [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Jääskeläinen A.J., Kekäläinen E., Kallio-Kokko H., Mannonen L., Kortela E., Vapalahti O. Evaluation of commercial and automated SARS-CoV-2 IgG and IgA ELISAs using coronavirus disease (COVID-19) patient samples. Euro Surveill. 2020;25:2000603. doi: 10.2807/1560-7917.ES.2020.25.18.2000603. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Centers for Disease Control and Prevention Coronavirus disease 2019 (COVID-19) – information for laboratories. 2020. https://www.cdc.gov/coronavirus/2019-ncov/lab/index.html Available at: accessed 20.05.
- 18.Ramdas K., Darzi A., Jain S. ‘Test, re-test, re-test’: using inaccurate tests to greatly increase the accuracy of COVID-19 testing. Nat. Med. 2020;26:810–811. doi: 10.1038/s41591-020-0891-7. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Galli C., Plebani M. SARS-CoV-2 antibody performances: we need better criteria. Clin. Chem. Lab. Med. 2020 doi: 10.1515/cclm-2020-1358. [Epub ahead of print] [DOI] [PubMed] [Google Scholar]
- 20.2020. https://www.fda.gov/medical-devices/coronavirus-disease-2019-covid-19-emergency-use-authorizations-medical-devices/eua-authorized-serology-test-performance Accessed 06.08.
- 21.Bryan A., Pepper G., Wener M.H., Fink S.L., Morishima C., Chaudhary A. Performance characteristics of the Abbott Architect SARS-CoV-2 IgG assay and seroprevalence in boise, Idaho. J. Clin. Microbiol. 2020:e00941. doi: 10.1128/JCM.00941-20. 20. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Favresse J., Eucher C., Elsen M., Tré-Hardy M., Dogné J.-M., Douxfils J. Clinical performance of the Elecsys electrochemiluminescent immunoassay for the detection of SARS-CoV-2 total antibodies. Clin. Chem. 2020;66:1104–1106. doi: 10.1093/clinchem/hvaa131. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Tré-Hardy M., Wilmet A., Beukinga I., Dogné J., Douxfils J., Blairon L. Validation of a chemiluminescent assay for specific SARS-CoV-2 antibody. Clin. Chem. Lab. Med. 2020;58:1357–1364. doi: 10.1515/cclm-2020-0594. [DOI] [PubMed] [Google Scholar]
- 24.Meyer B., Torriani G., Yerly S., Mazza L., Calame A., Arm-Vernez I. Validation of a commercially available SARS-CoV-2 serological immunoassay. Clin. Microbiol. Infect. 2020;26:1386–1394. doi: 10.1016/j.cmi.2020.06.024. [Epub ahead of print] [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25.Brochot E., Demey B., Handala L., François C., Duverlie G., Castelain S. Comparison of different serological assays for SARS-CoV-2 in real life. J. Clin. Virol. 2020;130:104569. doi: 10.1016/j.jcv.2020.104569. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26.Trabaud M.A., Icard V., Milon M.P., Bal A., Lina B., Escuret V. Comparison of eight commercial, high-throughput, automated or ELISA assays detecting SARS-CoV-2 IgG or total antibody. J. Clin. Virol. 2020;132:104613. doi: 10.1016/j.jcv.2020.104613. [DOI] [PMC free article] [PubMed] [Google Scholar]