Abstract
The aim of this study was to assess the analytical performances, sensitivity and specificity, of two rapid tests (Covid- Presto® test rapid Covid-19 IgG/IgM and NG-Test® IgM-IgG COVID-19) and one automated immunoassay (Abbott SARS-CoV-2 IgG) for detecting anti- SARS-CoV-2 antibodies. This study was performed with: (i) a positive panel constituted of 88 SARS-CoV-2 specimens collected from patients with a positive SARS-CoV-2 RT-PCR, and (ii) a negative panel of 120 serum samples, all collected before November 2019, including 64 samples with a cross-reactivity panel. Sensitivity of Covid-Presto® test for IgM and IgG was 78.4% and 92.0%, respectively. Sensitivity of NG-Test® for IgM and IgG was 96.6% and 94.9%, respectively. Sensitivity of Abbott IgG assay was 96.5% showing an excellent agreement with the two rapid tests (κ = 0.947 and κ = 0.936 for NGTest ® and Covid-Presto® test, respectively). An excellent agreement was also observed between the two rapid tests (κ = 0.937). Specificity for IgM was 100% and 86.5% for Covid-Presto® test and NG-Test®, respectively. Specificity for IgG was 92.0%, 94.9% and 96.5% for Covid-Presto®, NGTest ®, and Abbott, respectively. Most of the false positive results observed with NG-Test® resulted from samples containing malarial antibodies. In conclusion, performances of these 2 rapid tests are very good and comparable to those obtained with automated immunoassay, except for IgM specificity with the NG-Test®. Thus, isolated IgM should be cautiously interpreted due to the possible false-positive reactions with this test. Finally, before their large use, the rapid tests must be reliably evaluated with adequate and large panel including early seroconversion and possible cross-reactive samples.
Keywords: SARS-CoV-2, Serology, Rapid test, Cross-reactivity
1. Introduction
Severe Acute Respiratory Syndrome Coronavirus 2 (SARS-CoV-2) is the etiological agent of the Coronavirus Disease 19 (COVID-19) pandemic. SARS-CoV-2 RNA RT-PCR from a naso-pharyngeal swab is the gold standard test to diagnose COVID-19. Serological tests are also available allowing serological surveys in different populations, in particular patients presenting strong COVID-19 suspicions with negative PCR. Serological tests also make it possible to catch up later with undiagnosed people at time of active infection, since antibodies have been found in almost all people who have been in contact with SARS-CoV-2 within a variable period depending on the severity of the infection [1,2]. Furthermore, studies showed that the kinetics of appearance of IgM and IgG were relatively close [3].
Two types of tests are available to detect anti-SARS-CoV-2 antibodies: rapid lateral flow tests and automated immunoassays. Several studies have assessed analytical performances of the automated immunoassays [[4], [5], [6], [7]]. On the other hand, although a very large number of rapid tests have been developed, few of them have been reliably evaluated with a suitable serum panel. However, this is very important to have data about the efficacy of these rapid tests to reliably detect anti-SARS-CoV-2 antibodies, since their increasing use in the world.
The aim of this study was to assess the analytical performances (sensitivity and specificity) and agreement of two rapid tests and one automated immunoassay for detecting antibodies against SARS-CoV-2.
2. Materials and methods
2.1. Patients and serum samples
This evaluation was performed on 262 serum samples collected in the Virology Laboratory of Bichat-Claude Bernard and Saint-Louis University-Hospitals both in Paris, France.
Eighty-eight serum samples were collected from 54 patients with a confirmed COVID-19 diagnosis by a positive nasopharyngeal sample RT-PCR. Median age was 52 years (range: 27–80), 36 were males. Among them, 29 were hospitalized in intensive care, 11 in infectious diseases.
We constituted a negative panel of 120 sera, all collected before November 2019, to assess the specificity, including samples for testing as part of routine clinical care (n = 56) and serum samples corresponding to a cross-reactivity panel (n = 64). These latter consisted of coronaviruses (HKU1, NL63, 229E and OC43; n = 20), malarial (n = 26), respiratory viruses (Influenza A [n = 2], Influenza B [n = 1] Respiratory Syncytial Virus [n = 2], Metapneumovirus [n = 1], Rhinovirus [n = 1]), sera with acute CMV infection (n = 2), acute EBV infection (n = 1), HIV-HBV co-infection (n = 1), and acute Parvovirus B19 infection (n = 1), Toxoplasma (n = 1). In addition, we assessed five samples containing autoantibodies (four rheumatoid factor and one systemic lupus erythematosus).
We also assessed the serum of 54 health-care workers who presented clinical symptoms during the epidemic period for whom SARS-CoV-2 RT-PCR was negative or not carried out.
2.2. Rapid lateral flow tests
We evaluated two lateral flow tests: Covid-Presto® test rapid Covid-19 IgG/IgM (AAZ, Boulogne-Billancourt, France) and NG-Test® IgM-IgG COVID-19 (NG Biotech, Guipry, France) according to the manufacturer’s instructions. Five and ten microliters of serum for Covid-Presto® test and NG-Test®, respectively, were added and results were read and interpreted 10 min after depositing serum.
2.3. Automated immunoassay
Abbott SARS-CoV-2 IgG kit (chemiluminescent microparticle immunoassay) (Abbott, IL, USA) was performed according to the manufacturer’s instructions. The assay cut-off is an index of 1.40 and the assigned grey zone is comprised between 1.12 and 1.68.
2.4. Statistical analysis
All statistical analyses were performed using Excel. To assess sensitivity, RT-PCR results were chosen as gold standard. Cohen kappa statistics and absolute agreement were calculated to evaluate the agreement between the different tests.
2.5. Ethics
All participants were not opposed to the collection of their data.
3. Results
3.1. Sensitivity assessment
Sensitivity of Covid-Presto® test was assessed on 88 samples collected between day 4 and day 42 after onset of symptoms and sensitivity of the NG-Test® was assessed on a subgroup of 59 samples among the 88 samples tested with Covid-Presto® test, collected between days 7 and 28 after onset of symptoms (Table 1 ).
Table 1.
Performance characteristics of tw o rapid tests and one automated immunoassay.
| Sensitivity |
Specificity |
||||||
|---|---|---|---|---|---|---|---|
| Days after symptoms onset |
Overall | Pre-epidemic samples | Cross-reactivity panel | Overall | |||
| Test | ≤9 | 10−14 | >14 | ||||
| IgM | |||||||
| Covid-Presto® | 67 % (12/18) | 88 % (29/33) | 76 % (28/37) | 78.4 % (69/88) | 100 % (56/56) | 100 % (64/64) | 100 % (120/120) |
| NG-Test® | 83 % (5/6) | 100 % (22/22) | 97 % (30/31) | 96.6 % (57/59) | 91.7 % (22/24) | 82.1 % (23/28) | 86.5 % (45/52) |
| IgG | |||||||
| Covid-Presto® | 72 % (13/18) | 94 % (31/33) | 100 % (37/37) | 92.0 % (81/88) | 98.2 %(55/56) | 98.4 % (63/64) | 98.3 % (118/120) |
| NG-Test® | 83 % (5/6) | 96 % (21/22) | 97 % (30/31) | 94.9 %(56/59) | 91.7 % (22/24) | 100 % (28/28) | 96.2 % (50/52) |
| Abbott | 67 % (4/6) | 100 % (22/22) | 100 % (29/29) | 96.5 %(55/57) | 100 % (24/24) | 92.9 % (26/28) | 96.2 % (50/52) |
| IgM + IgG | |||||||
| Covid-Presto® | 83 % (15/18) | 97 % (32/33) | 100 % (37/37) | 95.5 %(84/88) | NA | NA | NA |
| NG-Test® | 83 % (5/6) | 100 % (22/22) | 97 % (30/31) | 96.6 % (57/59) | NA | NA | NA |
NA: not applicable.
Sensitivity of Covid-Presto® test for IgM was 67 % (n = 12/18), 88 % (n = 29/33) and 76 % (n = 28/37) for samples collected between days 4 and 9, between days 10 and 14, and after 14 days after onset of symptoms, respectively. Sensitivity of Covid-Presto® test for IgG was 72 % (n = 13/18), 94 % (n = 31/33) and 100 % (n = 37/37) for samples collected between days 4 and 9, between days 10 and 14, and after 14 days after onset of symptoms, respectively. When combining IgM and IgG, sensitivity of Covid-Presto® test was 83 % (n = 15/18), 97 % (n = 32/33) and 100 % (n = 37/37) for samples collected between days 4 and 9, between days 10 and 14, and after 14 days after onset of symptoms, respectively.
Sensitivity of NG-Test® for IgM was 83 % (n = 5/6), 100 % (n = 22/22) and 97 % (n = 30/31) for samples collected between days 7 and 9 after, between days 10 and 14, and after 14 days after onset of symptoms, respectively. Sensitivity of NG-Test® test for IgG was 83 % (n = 5/6), 96 % (n = 21/22) and 97 % (n = 30/31) for samples collected between days 7 and 9, between days 10 and 14, and after 14 days after onset of symptoms, respectively. When combining IgM and IgG, sensitivity of NG-Test® test was 83 % (n = 5/6), 100 % (n = 22/22) and 97 % (n = 30/31) for samples collected between days 7 and 9, between days 10 and 14, and after 14 days after onset of symptoms, respectively.
Among the 59 serum samples of this PCR positive panel tested by the two rapid tests, 57 were compared with Abbott SARS-CoV-2 IgG automated immunoassay. Sensitivity of Abbott IgG test was 67 % (n = 4/6), 100 % (n = 22/22) and 100 % (n = 29/29) for samples collected between days 7 and 9, between days 10 and 14, and after 14 days after onset of symptoms, respectively.
Agreement between Abbott assay and rapid tests (IgM/IgG combined) was of 96.5 % (n = 55/57). In one case, the two rapid tests detected IgG that were not detected by Abbott (index = 0.94), this sample was collected between days 7 and 9 after symptoms onset. For the second case, IgG were detected in the greyzone of Abbott (index = 1.45) but not by NG-Test®. This latter sample was collected between days 10 and 14 after symptoms onset and IgM were positive with the two rapid tests.
3.2. Specificity assessment
Specificity of Covid-Presto® test was assessed on 120 samples described in the methods section. Specificity of NG-Test® and Abbott assay was assessed on a subgroup of 52 samples among the 120 samples tested with Covid-Presto® test (Table 1).
Specificity of Covid-Presto® test assessed on 120 samples was 100 % for IgM and 98.3 % for IgG. For IgG one false positive result was observed with one sample containing malarial antibodies, and one false positive result was observed with one pre-epidemic sample.
Specificity of NG-Test® assessed on 52 samples was 86.5 % for IgM and 96.2 % for IgG. Regarding the seven samples false positive for IgM, two were from pre-epidemic panel and five were from samples containing malarial antibodies. Regarding IgG, the two false-positive samples belonged to the pre-epidemic panel.
Specificity of Abbott SARS-CoV-2 IgG kit was 96.2 %. The two false-positive samples had low titers (1.12 and 2.28), both samples contained malarial antibodies.
3.3. Agreement between the two lateral flow tests and the automated immunoassay
Agreement between the assays was performed on 163 samples: (i) 57 samples from the positive panel, (ii) 52 samples from the negative panel for which results were obtained for the two rapid tests and the Abbott SARS-CoV-2 IgG assay, and (iii) 54 samples collected from health workers (Table 2 ).
Table 2.
Agreement between two rapid tests and one automated immunoassay.
 |
NA: not applicable.
The significance of grey highlight corresponded to the no applicability (there is no comparison between the same test).
Absolute agreement between Covid-Presto® and NG-Test® was 82.8 % (n = 135/163, κ = 0.643) and 96.9 % (n = 158/163, κ = 0.937) for IgM and IgG, respectively. Regarding IgM results, 26 samples were negative with Covid-Presto® and positive with NG-Test®, including 8 NG-Test® false-positive results (6 malarial and 2 pre-epidemic sera). Seventeen of the 18 remaining samples were sera of COVID-positive patients collected after day 14 after symptoms onset. In addition, two sera were negative with NG-Test® and positive with Covid-Presto® test, both corresponding to COVID-positive patients (collected at day 8 and day 14 after symptoms onset). Regarding IgG results, three samples were negative with Covid-Presto® and positive with NG-Test® including 2 false-positive pre-epidemic sera and one serum from a COVID-positive patient collected at day 13 after symptoms onset. Furthermore, two sera were negative with NG-Test® and positive with Covid-Presto® test: one serum from a COVID-positive patient collected after day 14 after symptoms onset and one serum from a healthcare worker.
Evaluation between Covid-Presto® IgG test and Abbott SARS-CoV-2 IgG assay was performed on 159 available samples with sufficient volume of serum, showing an absolute agreement of 96.9 % (n = 154/159, κ = 0.936). Three samples were found negative with Covid-Presto® IgG test and positive with Abbott SARS-CoV-2 IgG, all with low Abbott index (1.87, 2.16, 2.57). Two of these three samples were issued from healthcare workers and the remaining one was from a COVID-positive patient (day 13 after symptoms onset) showing positive IgM. Two were positive with Covid-Presto® IgG test and negative with Abbott SARS-CoV-2 IgG (index:0.58 and 0.78), both samples belonged to the group of healthcare workers.
Assessment between NG-Test® IgG and Abbott SARS-CoV-2 IgG was performed on 153 available samples with sufficient volume of serum showing an agreement of 97.4 % (n = 149/153, κ = 0.947). One sample was positive in the greyzone with Abbott SARS-CoV-2 IgG assay (index: 1.45) and negative with NG-Test® IgG corresponding to the serum of a COVID-positive patient (day 11 after symptoms onset). Three samples were negative with Abbott SARS-CoV-2 IgG assay (index: 0.07, 0.12 and 0.94) and positive with NG-Test® corresponding to 2 false-positive pre-epidemic sera and one COVID-positive patient (day 8 after symptoms onset).
Predictive positive and negative values were calculated for 210, 113 and 111 serum samples for Covid-Presto®, NG-Test® and Abbott tests, respectively. For the rapid tests, IgM and IgG results were combined for this analysis. All patients with positive PCR were considered COVID-19 positive and all pre-pandemic samples were considered COVID-19 negative. PPV was 97.7 %, 89.1 % and 96.6 % for Covid-Presto®, NG-Test® and Abbott tests, respectively. NPV was 97.5 %, 95.9 % and 94.3 % for Covid-Presto®, NG-Test® and Abbott, respectively.
4. Discussion
In the present study, we evaluated two different lateral flow tests (Covid-Presto® and NG-Test®) and compared their performances to that of the automated Abbott immunoassay using the same samples panel.
Sensitivity has been assessed using a panel of 88 serum samples of COVID-19-infected patients (confirmed with a positive PCR), serum was collected between day 4 and day 42 after symptoms onset. Sensitivity for IgM, among the samples collected before day 9 after symptoms onset, was 67 % and 83 % for Covid-Presto® test and NG-Test®, respectively. In the recent study of Nicol et al., they found sensitivity of NG-Test for IgM of 43.8 % for the samples collected before day 7 after symptoms onset and of 81.8 % among all samples [5]. The excellent sensitivity of Covid-Presto® test observed in our study confirmed the findings of the Prazuck et al. study showing 100 % of sensitivity in samples collected more than 15 days after symptoms [8].
Among some samples collected before day 10 after symptoms onset, a simultaneous detection of IgM and IgG antibodies has been detected. These findings are in line with the antibodies kinetics described for IgM and IgG also using lateral flow rapids, as previously described with other techniques [3]. In the present study for Covid-Presto® test, it allowed to increase the sensitivity from 67 % when only IgM are taken into account to 83 % when both IgM and IgG are taken into account, highlighting the important added value to interpret the rapid tests by combining IgM and IgG antibodies.
Sensitivity for IgG in samples collected later than 10 days after symptoms onset was excellent with the different tests being equal to 97.1 %, 96.2 % and 100 % for Covid-Presto®, NG-Test®, and Abbott, respectively. Thus, both rapid tests showed an excellent sensitivity for IgG with a very good agreement with Abbott. A previous study assessing Abbott test performance showed sensitivity of 100 % for IgG for samples collected after 15 days after symptoms onset and of 69 % for samples collected between 9 and 14 days after symptoms onset [6]. In this latter study, results sensitivity for IgG were similar using NG-Test® [6]. In another study, IgG sensitivity of Abbott test was 91.8 % for patients hospitalized 15 days after symptoms onset and 95.7 % for patients non-hospitalized 20 days after symptoms onset.
A limitation of our study could be that most of the patients of the positive panel presented severe infections, since 74 % of them were hospitalized in infectious disease unit or in intensive care. Interestingly, among the 14 out-patients, samples were collected for 9 of them 10 days after symptoms onset, showing positive IgM and/or IgG in seven cases with Covid-Presto® test. Insufficient quantity of serum for these patients was available to also test with NG-Test® and Abbott. Previous studies have reported that the kinetics and intensity of immune response could differ depending on the disease severity [1,2], thus it will be needed to also evaluate rapid tests in mild and pauci-symptomatic patients. Another limitation is the difference in the number of tested samples for the early panel (serum samples collected before 9 days after symptoms onset) between the two rapid tests that which can bias the comparison between these tests for this group. A limitation is that we make this evaluation from serum samples and not from capillary blood specimens.
Regarding specificity evaluation, a crucial point for rapid tests, we used a large panel with 120 pre-endemic samples including 64 representatives of different profiles that can generate possible cross-reactivity. In our study, we showed an excellent specificity, above 96 % in all cases and equal to 100 % for IgM with Covid-Presto® test. The excellent specificity of Covid-Presto® test was also observed in the study of Prazuck et al. [8]. In our study, the only issue regarding specificity is for IgM with NG-Test®, since specificity is only of 86.5 %. However, this low specificity is mainly due to cross-reactivity with sera containing reactivity malarial antibodies. In the study of Nicol et al. IgM specificity with NG-Test® was 95.3 % [5], higher than in our study, however their negative panel contained no serum with malaria antibodies. Regarding automated immunoassay, we showed a very good specificity of 96.2 % for IgG with Abbott, confirming previous results of 99.3 %, 99.6 % and 100 % [9]. Serum samples containing malarial antibodies are absent or underrepresented in the negative panel of the other studies, although they are known to generate possible cross reactivity. This is very important to include it in the negative panel, since this is a differential diagnosis in patients returning from malaria endemic region with Flu-like symptoms. Overall, in our study, we observed a very good PPV and NPV for both rapid tests.
In conclusion, analytical performances for detection of anti-SARS-CoV-2 IgG antibodies by two lateral flow rapid tests are very good and quite comparable to those obtained with automated immunoassay. However, serological tests should be used after day 10 following symptoms onset. Before this, RT-PCR is the gold standard test for COVID-19 diagnosis. The interpretation by combining IgM and IgG increased sensitivity of rapid tests. The presence of isolated IgM should be cautiously interpreted due to the possible false-positive reactions. Finally, the rapid tests must be reliably evaluated with adequate and large panels including early seroconversion and possible cross-reactive samples, before their large use and particular interest in low-resource settings.
References
- 1.Long Q.X., Liu B.Z., Deng H.J., Wu G.C., Deng K., Chen Y.K., Liao P., Qiu J.F., Lin Y., Cai X.F., Wang D.Q., Hu Y., Ren J.H., Tang N., Xu Y.Y., Yu LH Mo Z., Gong F., Zhang X.L., Tian W.G., Hu L., Zhang X.L., Xiang J.L., Du HX Liu H.W., Lang C.H., Luo X.H., Wu S.B., Cui X.P., Zhou Z., Zhu M.M., Wang J., Xue C.J., Li X.F., Wang L., Li Z.J., Wang K., Niu C.C., Yang Q.J., Tang X.J., Zhang Y., Liu X.M., Li Z.J., Zhang D.C., Zhang F., Liu P., Yuan J., Li Q., Hu J.L., Chen J., Huang A.L. Antibody responses to SARS-CoV-2 in patients with COVID-19. Nat. Med. 2020;26:845–848. doi: 10.1038/s41591-020-0897-1. [DOI] [PubMed] [Google Scholar]
- 2.Yongchen Z., Shen H., Wang X., Shi X., Li Y., Yan J., Chen Y., Gu B. Different longitudinal patterns of nucleic acid and serology testing results based on disease severity of COVID-19 patients. Emerg. Microbes Infect. 2020;9:833–836. doi: 10.1080/22221751.2020.1756699. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Sethuraman N., Jeremiah S.S., Ryo A. Interpreting diagnostic tests for SARS-CoV-2. JAMA. 2020;(May):6. doi: 10.1001/jama.2020.8259. [DOI] [PubMed] [Google Scholar]
- 4.Meschi S., Colavita F., Bordi L., Matusali G., Lapa D., Amendola A., Vairo F., Ippolito G., Capobianchi M.R., Castilletti C. INMICovid-19 laboratory team, Performance evaluation of Abbott ARCHITECT SARS-CoV-2 IgG immunoassay in comparison with indirect immunofluorescence and virus microneutralization test. J. Clin. Virol. 2020;129:104539. doi: 10.1016/j.jcv.2020.104539. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Nicol T., Lefeuvre C., Serri O., Pivert A., Joubaud F., Dubée V., Kouatchet A., Ducancelle A., Lunel-Fabiani F., Le Guillou-Guillemette H. Assessment of SARS-CoV-2 serological tests for the diagnosis of COVID-19 through the evaluation of three immunoassays: two automated immunoassays (Euroimmun and Abbott) and one rapid lateral flow immunoassay (NG Biotech) J. Clin. Virol. 2020;129:104511. doi: 10.1016/j.jcv.2020.104511. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Jääskeläinen A.J., Kuivanen S., Kekäläinen E., Ahava M.J., Loginov R., Kallio-Kokko H., Vapalahti O., Jarva H., Kurkela S., Lappalainen M. Performance of six SARS-CoV-2 immunoassays in comparison with microneutralisation. J. Clin. Virol. 2020;129:104512. doi: 10.1016/j.jcv.2020.104512. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Chew K.L., Tan S.S., Saw S., Pajarillaga A., Zaine S., Khoo C., Wang W., Tambyah P., Jureen R., Sethi S.K. Clinical evaluation of serological IgG antibody response on the Abbott Architect for established SARS-CoV-2 infection. Clin. Microbiol. Infect. 2020 doi: 10.1016/j.cmi.2020.05.036. S1198-743X(20)30334-30337. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Prazuck T., Colin M., Giache S., Gubavu C., Seve A., Rzepecki V., Chevereau-Choquet M., Kiani C., Rodi V., Lyonnet E., Courtellemont L., Guinard J., Pialoux G., Hocqueloux L. 2020. Evaluation of Performance of Two SARS-CoV-2 Rapid Whole-blood Finger-stick IgM-IgG Combined Antibody Tests.https://www.medrxiv.org/content/10.1101/2020.05.27.20112888v1 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Dellière S., Salmona M., Minier M., Gabassi A., Alanio A., Le Goff J., Delaugerre C. Chaix ML; saint-louis CORE (COvid REsearch) group, evaluation of COVID-19 IgG/IgM rapid test from orient gene biotech. J. Clin. Microbiol. 2020;9(June) doi: 10.1128/JCM.01233-2. [DOI] [PMC free article] [PubMed] [Google Scholar]