Abstract
In an investigation of six anti-SARS-CoV-2 antibody kits with different target antigen and methodology, each kit showed comparable performance. As false-positive reactions occurred independently with different kits, specificity increased to 100% when pairs of kits were used. With three-kit combination, both sensitivity (99.1%) and specificity (100%) increased.
Keywords: Performance, SARS-CoV-2, COVID-19, Antibody, Combination
Introduction
The coronavirus disease 2019 (COVID-19) pandemic due to severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) infection has not been successfully controlled in most countries worldwide.1 Reverse transcriptase polymerase chain reaction (RT-PCR) test is the gold standard method for the diagnosis of SARS-CoV-2 infection, while serologic antibody tests using immunoassay have been utilized for sero-prevalence monitoring, risk assessment of healthcare workers (HCWs), and convalescent plasma therapy.2, 3, 4 Nevertheless, serologic tests have inherent limitations as a diagnostic method: approximately seven days are required for seroconversion, a gold standard method is absent, and the diagnostic performance is not as high as that of RT-PCR.2 , 3 , 5 Various formats of commercial antibody detection kits including enzyme-linked immunosorbent assay (ELISA), fluorescence immunoassay (FIA), and lateral flow immunoassay (LFIA) for rapid diagnostic test (RDT) have been being introduced, but the performance of these kits needs to be further evaluated. Herein, we present the practical strategies to optimize the utilization of anti-SARS-CoV-2 antibody test using multiple kits from different manufacturers.
Methods
Study specimens and serologic test kits for anti-SARS-CoV-2 antibodies
Sera collected after more than seven days of illness from RT-PCR-confirmed asymptomatic to severe COVID-19 patients were used as positive specimens.2 In asymptomatic patients, illness days were counted from the date of diagnosis. Sera from patients who had recovered from conventional respiratory viral infections, including coronavirus 229E/NL63/OC43, influenza, metapneumovirus, parainfluenza virus, rhinovirus, and respiratory syncytial virus, were used as negative controls. In addition, serum specimens from HCWs without risk of COVID-19 were used for negative controls.2 Three lots of intravenous immunoglobulin (IVIG) products (Green Cross Corp., Yongin, Korea, produced on June 28, 2019, October 23, 2019, and February 2, 2018) were also used to evaluated potential cross-reactivity with anti-SARS-CoV-2 antibody test kits.
A total of six different anti-SARS-CoV-2 antibody kits, including one LFIA RDT kits using gold conjugate (SD Biosensor Inc., Suwon, Korea, detecting IgM and IgG antibodies against nucleocapsid protein (NCP)), two FIA kits using europium particles (SD Biosensor Inc., detecting IgM and IgG antibodies against NCP; and Boditech Med Inc., Chuncheon, Korea, detecting IgM and IgG antibodies against NCP), and three ELISA kits (SD Biosensor Inc. detecting total antibody against NCP and the receptor binding domain (RBD) of the spike protein; PCL Inc, Seoul, Korea, detecting total antibody against NCP and RBD; and EUROIMMUN, Lübeck, Germany, detecting IgG antibody against RBD) were used. Details of study specimens and anti-SARS-CoV-2 antibody kits are presented in the Supplementary materials. This study was approved by the Institutional Review Board of Samsung Medical Center.
Results
Tested serum samples
A total of 110 serum specimens from 74 RT-PCR-confirmed COVID-19 patients were subjected to SARS-CoV-2 antibody tests. Sixty-four convalescent sera from asymptomatic and mild patients, 14 serial sera from three mild patients, and 32 serial sera from seven moderate to severe patients were included. For negative controls, 119 serum specimens were collected including 42 convalescent sera from patients infected with conventional respiratory viruses and 77 sera from HCWs.
Performance of individual kits
Test performance of anti-SARS-CoV-2 antibody test kits, including sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) are presented in Table 1 . The RDT kit from SD Biosensor showed a sensitivity/specificity of 78.9/98.3% (IgM alone), 94.5/96.6% (IgG alone), and 99.1/95.0% (IgM or IgG). Although sensitivity of individual IgM and IgG bands were slightly lower, specificity were higher than FIA kits.
Table 1.
Characteristics and performance of six anti-SARS-CoV-2 antibody test kits.
| Test kit |
Sensitivity | Specificity | PPV | NPV | Sensitivity + specificity | |||
|---|---|---|---|---|---|---|---|---|
| Method | Manufacturer | Target protein | Target antibody | |||||
| RDT using gold conjugate | SD Biosensora | NCP | IgM | 78.9 | 98.3 | 97.7 | 83.6 | 177.2 |
| IgG | 94.5 | 96.6 | 96.3 | 95.0 | 191.2 | |||
| IgM or IgG | 99.1 | 95.0 | 94.7 | 99.1 | 194.0 | |||
| FIA using europium particles | SD Biosensor | NCP | IgM | 74.5 | 93.3 | 91.1 | 79.9 | 167.8 |
| IgG | 97.3 | 94.1 | 93.9 | 97.4 | 191.4 | |||
| IgM or IgG | 99.1 | 87.4 | 87.9 | 99.0 | 186.5 | |||
| Boditech Med | NCP | IgM | 14.5 | 99.2 | 94.1 | 55.7 | 113.7 | |
| IgG | 99.1 | 94.1 | 94.0 | 99.1 | 193.2 | |||
| IgM or IgG | 99.1 | 93.3 | 93.2 | 99.1 | 192.4 | |||
| ELISA | SD Biosensorb | RBD & NCP | Total antibody | 96.3 | 100.0 | 100.0 | 96.7 | 196.3 |
| PCL | RBD & NCP | Total antibody | 98.2 | 100.0 | 100.0 | 98.3 | 198.2 | |
| EUROIMMUNc | RBD | IgG, OD ratio ≥0.8d | 95.4 | 96.6 | 96.3 | 95.8 | 192.1 | |
| IgG, OD ratio ≥1.1 | 93.5 | 100.0 | 100.0 | 94.4 | 193.5 | |||
A total of 110 RT-PCR confirmed positive specimens and 119 negative control specimens were used, otherwise indicated.
Abbreviations: SARS-CoV-2, severe acute respiratory syndrome coronavirus 2; PPV, positive predictive value; NPV, negative predictive value; RDT, rapid diagnostic kit; FIA, fluorescence immunoassay; ELISA, enzyme-linked immunosorbent assay; NCP, nucleocapsid protein; RBD, receptor binding domain; COI, cut-off index; OD, optical density.
109 positive specimens were used.
108 positive specimens and 118 negative control specimens were used.
108 positive specimens and 117 negative control specimens were used.
Cut-off value for the borderline result.
The FIA kit from SD Biosensor showed sensitivity/specificity of 74.5/93.3% (IgM alone), 97.3/94.1% (IgG alone), and 99.1/87.4% (IgM or IgG). The FIA kit from Boditech Med showed sensitivity/specificity of 14.5/99.2% (IgM alone), 99.1/94.1% (IgG alone), and 99.1/93.3% (IgM or IgG). Although the sensitivity of the IgM band was low, the sensitivity of the IgG band was highest among the tested kits with similar specificity.
The ELISA kits targeting total antibody showed high sensitivity and specificity. The ELISA kit from SD Biosensor showed sensitivity/specificity of 96.3/100% and that of the PCL Inc kit was 98.2/100%. The ELISA kit targeting IgG (EUROIMMUN) showed sensitivity/specificity of 95.4/96.6 (using a cut-off value of borderline) and 93.5/100% (using a cut-off value of positive).
False-positive results among test kits
A total of 29 negative control specimens produced false-positive test results (Supplementary Table 1). No specimens showed cross-reactive false-positive reactions between antibody test kits from different manufacturers. Eight specimens showed false-positive reactions on the IgM band and seven for the IgG band of the SD Biosensor FIA kit. As the RDT kit and FIA kit from SD Biosensor used the same product materials, false-positive reactions were shared between these kits. One specimen showed a false-positive reaction for the IgM band and seven for the IgG band of the Boditech Med FIA kit. The ELISA kits for total antibodies, both SD Biosensor and the PCL Inc kits, showed no false positive reactions. Four specimens showed borderline values for the ELISA kit for the IgG antibody of EUROIMMUN.
To evaluate the presence of false-positive reactions to immunoglobulins against various antigens, we tested three lots of IVIG products manufactured before the emergence of SARS-CoV-2. Each IVIG product (immunoglobulin concentration of 10,000 mg/dL) was serially diluted two-fold with both serum and phosphate-buffered saline (PBS) buffer, up to 1:32 dilution (312.5 mg/d). None of the tests produced positive results.
Performance of test kits used in combination
Based on the findings of independent false-positive reactions between test kits, we evaluated the performance of test kits in combination (Supplementary Tables 2 and 3). The specimen was interpreted as positive if two or more kits produced a positive result. If all the test kits yielded negative results or only one kit gave a positive result, the specimen was interpreted as negative.
When two different kinds of serologic kits were used in combination, sensitivity was the same or decreased, but specificity increased to 100%. When three different kinds of kits were used, both sensitivity and specificity increased (99.1% and 100%, respectively).
Discussion
Serologic testing for an emerging viral disease has a wide range of clinical implications.2, 3, 4, 5, 6, 7, 8, 9 However, immunoassay methods inevitably have false-positive and false-negative results, and there is no gold standard method to detect the presence of virus-specific antibodies.3 , 5 Although the neutralization test is highly specific and reflects functional activity, low levels of binding antibodies may not be detected by neutralizing tests.2 Several anti-SARS-CoV-2 antibody test kits using immunoassay methods exhibited high sensitivity, but sub-optimal specificity remains a concern. In the present investigation, false-positive reactions occurred independently between kits. Cross-reactive false-positive results were observed only between RDT and FIA kits from the same manufacturer, which shared raw materials. False-positive reactions on anti-SARS-CoV-2 antibody test kits are likely to occur specifically to individual materials, such as recombinant SARS-CoV-2 protein, reagents, or buffers. These findings suggest that false-positivity could be decreased if we use multiple test kits.
As false-positive reactions occurred independently between test kits, the specificity of the kits increased to 100% when two different kits were used together, but the sensitivity inevitably decreased. When three kits were used together, both sensitivity and specificity increased to 99.1% and 100%, respectively. To optimize the performance of serologic testing, whether to use two or more kits in combination should be decided according to the purpose of the test. For the case of sero-prevalence studies in low COVID-19 prevalence areas, specificity and cost are important due to large sample sizes and low PPV. Two-step confirmation using two different kits would be cost-effective in such situation. Meanwhile, for the diagnosis of multisystem inflammatory syndrome in children, both sensitivity and specificity are important and more than two kits could be used to optimize performance.3 , 9
The present analysis has several limitations. First, the number of negative control specimens was limited to 119 samples and there could be additional false positive reactions if more specimens were tested. Second, even if the kits were manufactured by different companies, cross-reactive false positivity may occur if the kits share the same recombinant virus antigen, reagent, or buffer. Third, the use of multiple kits may improve performance but the cost of testing inevitably increases. Selection of tests kits should be decided according to local prevalence, the purpose of testing, and the expected cost. Lastly, the present study data reflects diagnostic performance of sero-converted patients, evaluating serial sera of COVID-19 patients collected after seven days of illness (46 sera from 10 patients, median 24 days of illness) and convalescent sera after recovery (64 sera from 64 patients, median 40.5 days of illness). Diagnostic performance of antibody test kits for acute SARS-CoV-2 infection would be considerably different. It should be further investigated whether measurement of different target proteins and immunoglobulin classes would be useful in diagnosing acute SARS-CoV-2 infection and differentiating patients with acute, recent, past, and re-infections.
In conclusion, in the present analysis of anti-SARS-CoV-2 antibody test kits, false-positive reactions occurred independently between test kits. Using a combination of two different kits increased specificity to 100% and combination of more than two kits further increased performance. The number and type of kits used should be decided based on the outbreak situation and the purpose of testing.
Author contributions
Conceptualization, J.-H.K., Y.-J.K., E.-S.K., K.R.P.; Investigation, J.-H.K., E.-J.J., J.Y.B., K.H., S.Y.C., C.-I.K., D.R.C.; Laboratory work, J.Y.B., H.J.S., E.-S.K.; Data analysis, J.-H.K.; writing–review and editing, J.-H.K., Y.-J.K., E.-S.K., K.R.P. All authors have read and agreed to the manuscript.
Funding
This work was supported by Research Program funded by the Korea Disease Control and Prevention Agency (#2020-ER5328-00).
Declaration of competing interest
There are no potential conflicts of interest relevant to this article to report.
Acknowledgments
We would like to express our sincerest condolences to the patients and families who suffered from the COVID-19 outbreak. We greatly appreciate the patients and HCWs who voluntarily participated in this study. Finally, we would like to thank SD Biosensor Inc., Boditech Med Inc., and PCL Inc. for the donation of their antibody test kits.
Footnotes
Supplementary data to this article can be found online at https://doi.org/10.1016/j.jmii.2021.03.008.
Appendix A. Supplementary data
The following is the Supplementary data to this article:
References
- 1.WHO WHO coronavirus disease (COVID-19) dashboard. 2020. https://covid19.who.int/
- 2.Ko J.H., Joo E.J., Park S.J., Baek J.Y., Kim W.D., Jee J. Neutralizing antibody production in asymptomatic and mild COVID-19 patients, in comparison with pneumonic COVID-19 patients. J Clin Med. 2020;9 doi: 10.3390/jcm9072268. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Hanson K.E., Caliendo A.M., Arias C.A., Englund J.A., Hayden M.K., Lee M.J. Infectious diseases society of America guidelines on the diagnosis of COVID-19:serologic testing. Clin Infect Dis. 2020 doi: 10.1093/cid/ciaa1343. Online ahead of print. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Ko J.H., Lee J.Y., Kim H.A., Kang S.J., Baek J.Y., Park S.J. Serologic evaluation of healthcare workers caring for COVID-19 patients in the Republic of Korea. Front Microbiol. 2020;11:587613. doi: 10.3389/fmicb.2020.587613. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Deeks J.J., Dinnes J., Takwoingi Y., Davenport C., Spijker R., Taylor-Phillips S. Antibody tests for identification of current and past infection with SARS-CoV-2. Cochrane Database Syst Rev. 2020;6:Cd013652. doi: 10.1002/14651858.CD013652. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Ko J.H., Joo E.J., Kim S.H., Kim Y.J., Huh K., Cho S.Y. Clinical application of rapid diagnostic test kit for SARS-CoV-2 antibodies into the field of patient care. J Microbiol Immunol Infect. 2021;54(1):97–100. doi: 10.1016/j.jmii.2020.07.003. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Ko J.H., Seok H., Cho S.Y., Ha Y.E., Baek J.Y., Kim S.H. Challenges of convalescent plasma infusion therapy in Middle East respiratory coronavirus infection: a single centre experience. Antivir Ther. 2018;23:617–622. doi: 10.3851/IMP3243. [DOI] [PubMed] [Google Scholar]
- 8.Ko J.H., Lee J.Y., Baek J.Y., Seok H., Park G.E., Lee J.Y. Serologic evaluation of MERS screening strategy for healthcare personnel during a hospital-associated outbreak. Infect Control Hosp Epidemiol. 2017;38:234–238. doi: 10.1017/ice.2016.251. [DOI] [PubMed] [Google Scholar]
- 9.Kim H., Shim J.Y., Ko J.-H., Yang A., Shim J.W., Kim D.S. Multisystem inflammatory syndrome in children related to COVID-19: the first case in Korea. J Kor Med Sci. 2020;35 doi: 10.3346/jkms.2020.35.e391. [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.