Assessing the performance of commercial serological tests for SARS-CoV-2 diagnosis

Highlights

• •Serological tests offer a promising alternative to polymerase chain reaction for COVID-19 serodiagnosis.
• •The GOLD ELISA COVID-19 IgG + IgM assay had the best sensitivity and specificity.
• •Assay accuracy improves over time, with notable improvements observed for the GOLD enzyme-linked immunosorbent assay after 15-21 days.
• •Discrepancies exist between claimed and actual assay performances.

Abstract

Objectives

The emergence of SARS-CoV-2 has triggered a global pandemic with profound implications for public health. Rapid changes in the pandemic landscape and limitations in in vitro diagnostics led to the introduction of numerous diagnostic devices with variable performance. In this study, we evaluated three commercial serological assays in Brazil for detecting anti-SARS-CoV-2 antibodies.

Methods

We collected 90 serum samples from SARS-CoV-2-negative blood donors and 352 from SARS-CoV-2-positive, unvaccinated patients, categorized by symptom onset. Subsequently, we assessed the diagnostic performance of three commercial enzyme immunoassays: GOLD ELISA (enzyme-linked immunosorbent assay) COVID-19 Ig (immunoglobulin) G + IgM, Anti-SARS-CoV-2 NCP IgM ELISA, and Anti-SARS-CoV-2 NCP IgG ELISA.

Results

Our findings revealed that the GOLD ELISA COVID-19 IgG + IgM exhibited the highest sensitivity (57.7%) and diagnostic odds ratio, surpassing the manufacturer's reported sensitivity in most analyzed time frames while maintaining exceptional specificity (98.9%). Conversely, the Anti-SARS-CoV-2 NCP IgG ELISA demonstrated lower sensitivity but aligned with independent evaluations, boasting a specificity of 100%. However, the Anti-SARS-CoV-2 NCP IgM ELISA exhibited lower sensitivity than claimed, particularly in samples collected shortly after positive reverse transcription polymerase chain reaction results. Performance improved 15-21 days after symptom onset and beyond 22 days, but in the first week, both Anti-SARS-CoV-2 NCP IgM ELISA and Anti-SARS-CoV-2 NCP IgG ELISA struggled to differentiate positive and negative samples.

Conclusions

Our study emphasizes the need for standardized validation protocols to address discrepancies between manufacturer-claimed and actual performance. These insights provide essential information for health care practitioners and policymakers regarding the diagnostic capabilities of these assays in various clinical scenarios.

Graphical Abstract

Introduction

Coronaviruses belong to the Coronaviridae family, a group of single-stranded RNA viruses with a positive-sense RNA genome, classified within the Nidovirales order [1,2]. In December 2019, the emergence of a novel coronavirus, SARS-CoV-2, in Wuhan led to pneumonia, fever, hypoxia, and acute respiratory distress syndrome [3]. This virus swiftly spread beyond Chinese borders, reaching all continents within months, prompting the World Health Organization to declare it a pandemic in March 2020 [4]. As of March 2023, SARS-CoV-2 has infected over 761 million people, resulting in over 6.8 million deaths, with ongoing cases worldwide [5].

Due to the dynamic nature of the pandemic and limited in vitro diagnostic (IVD) capabilities, numerous diagnostic devices have been introduced, displaying significant performance variations. Initially, nucleic acid tests (NATs), specifically reverse transcription polymerase chain reaction (RT-qPCR), served as the primary SARS-CoV-2 IVD method. However, RT-qPCR is intricate, time-consuming, and requires specialized personnel and expensive infrastructure. Its sensitivity decreases over time because of declining viral nucleic acid levels and respiratory epithelium shedding, leading to false negatives, especially with delayed diagnosis. Guo et al. [6] demonstrated that serological assays exhibit higher sensitivity than NATs at 5.5 days after symptom onset. Combining RT-qPCR with immunoglobulin (Ig) M enzyme-linked immunosorbent assay (ELISA) achieved a 98.6% positive detection rate in suspected cases, surpassing NATs alone at 51.9%.

In comparison with RT-qPCR, indirect serological assays may offer advantages in affordability, ease of execution, and convenience, as they can be implemented without complex infrastructure. Other direct detection methods, such as antigen detection, provide practicality similar to that of serological assays and address challenges associated with molecular-based tests, including cost and complexity. However, similarly to RT-qPCR, the sensitivity of direct detection IVD tests decreases over time, resulting in an increasing rate of false negatives [7]. Paradoxically, the sensitivity of indirect serological assays increases over time, making them valuable in cases of delayed diagnosis or suspected false negatives [6].

SARS-CoV-2 expresses four structural proteins: spike (S), nucleocapsid (N), membrane (M), and envelope (E) [2]. Among these, S and N proteins are commonly targeted in serological assays [8]. The S glycoprotein, as a surface antigen, remains continuously exposed to the extracellular environment, making it recognizable by the immune system. Its success in previous MERS [9] and SARS-CoV [10] outbreaks is attributed to its immunogenic nature, eliciting a strong humoral response due to the presence of constitutive immunodominant epitopes that stimulate a significant antibody reaction [11].

The COVID-19 pandemic prompted a global effort to develop diagnostic tools. On February 4, 2020, the United States Secretary of Health and Human Services granted emergency use authorization for SARS-CoV-2 diagnosis. This authorization allowed the commercialization of molecular and immunological IVD tests without formal Food and Drug Administration evaluation, relying solely on data reported by the manufacturers [12]. Numerous companies introduced immunoassays and point-of-care tests, but independent evaluations frequently failed to replicate the diagnostic accuracy claimed by the manufacturers [13,14]. Comprehensive and comparative data on the performance of these tests are imperative for informed decisions in clinical care and public health. In light of this scenario, our study aims to assess the diagnostic performance of three commercially available serological assays designed to detect anti-SARS-CoV-2 antibodies using statistically determined sample panels. Our findings and analysis are presented in this report.

Materials and methods

Sample size and collection of human serum

The required sample size was determined using OpenEpi, an open-source software [15]. Assuming an infinite population, an absolute error of 2.5%, and an expected sensitivity and specificity of 99%, the minimum sample size for this study was estimated to be 61 sera from SARS-CoV-2-infected individuals and 61 sera from SARS-CoV-2-negative individuals. In total, 442 sera were collected from 90 SARS-CoV-2-negative healthy blood donors, and 352 sera were collected from 93 SARS-CoV-2-positive individuals. The positive samples were obtained from unvaccinated patients who tested positive for SARS-CoV-2 RT-PCR and presented clinical symptoms consistent with COVID-19. These samples were collected between March and October 2020 from patients treated at two hospitals in the city of Salvador and its surrounding areas (Aliança and Aeroporto hospitals). Sera of 354 patients were collected between zero and 79 days (median 12 days, standard deviation 12.6 days) after symptom onset. The samples from SARS-CoV-2 patients were categorized according to symptom onset: 0-7 days after symptom onset (week 1), 8-14 days after symptom onset (week 2), 15-21 days after symptom onset (week 3), or 22 or more days after symptom onset (week 4). Negative samples were obtained from HEMOBA (Hematology and Hemotherapy Foundation of Bahia) before the pandemic, collected from healthy individuals who tested negative for Chagas disease, hepatitis B, hepatitis C, HIV-1/2, HTLV-1/2, and syphilis (Figure 1).

Figure 1.

Flowchart depicting study design adhering to the STARD (Standards for Reporting of Diagnostic Accuracy Studies) guidelines.

ELISA, enzyme-linked immunosorbent assay; Ig, immunoglobulin.

Laboratory assays

Three commercial COVID-19-specific enzyme immunoassays were chosen on the basis of their availability and license for use in Brazil. These included the GOLD ELISA COVID-19 IgG + IgM (REM Diagnóstica, São Paulo, SP, Brazil), which detects both IgG and IgM using the S1 and S2 domains of the S protein and the N protein; the Anti-SARS-CoV-2 NCP IgM ELISA (Euroimmun Medizinische Labordiagnostika AG, Lübeck, Germany), which detects IgM antibodies against SARS-CoV-2 using the virus N protein; and the Anti-SARS-CoV-2 NCP IgG ELISA (Euroimmun Medizinische Labordiagnostika AG, Lübeck, Germany), which detects IgG antibodies to SARS-CoV-2 using the N protein. All tests were carried out following the manufacturers’ instructions.

Statistical analysis

Data analysis and presentation were performed using GraphPad Prism version 9.5.1 (San Diego, CA, USA). Descriptive statistics are reported as geometric mean ± standard deviation. We assessed data normality with the Shapiro–Wilk test, followed by Student's t-test. In cases where the assumption of homogeneity was not met, we applied the Wilcoxon signed-ranks test. All analyses were two-tailed, with statistical significance set at P < 0.05. To identify the optimal optical density (OD) cutoff point for distinguishing negative from positive samples, we conducted a cutoff point analysis following the manufacturer's instructions. Results were presented as a reactivity index (RI), representing the ratio of the sample's OD to the corresponding microplate's cutoff OD. Samples with an RI <1.00 were classified as negative. We assessed ELISA performance using a dichotomous approach (2 × 2 contingency table) and compared sensitivity, specificity, accuracy, likelihood ratios (LRs), and the diagnostic odds ratio (DOR) [16]. Accuracy measures the test's ability to provide correct results, while LR, both positive and negative, indicates the likelihood of a positive or negative result in tested individuals compared with non-tested individuals. Positive LR values >10 and negative LR values <0.1 are typically considered diagnostically significant [17]. The DOR, calculated as the ratio of positive to negative LR values, serves as a global performance metric summarizing the test's diagnostic accuracy. It provides a single number describing the probability of obtaining a positive result in a person with the disease compared with someone without the disease [18]. We further evaluated the accuracy of each commercial assay using the area under the receiver operating characteristic curve (AUC), categorized as low (0.51-0.61), moderate (0.62-0.81), elevated (0.82-0.99), or outstanding (1.0) [19]. Confidence intervals (CIs) with a 95% confidence level were applied for all metrics, and non-overlapping 95% CI bars indicating statistical significance [16]. Imprecision assessment relied on Cohen's kappa coefficient (κ) [20]. The strength of agreement was interpreted as poor (κ = 0), slight (0 < κ ≤ 0.20), fair (0.21 < κ ≤ 0.40), moderate (0.41 < κ ≤ 0.60), substantial (0.61 < κ ≤ 0.80), near perfect (0.81 < κ ≤ 1.00), and perfect agreement (κ = 1.00). For adherence to reporting guidelines, we provided a flowchart (Figure 1) in accordance with the STARD (Standards for Reporting of Diagnostic Accuracy Studies) guidelines [21].

Results

In this study, we evaluated three commercial serological assays for detecting specific anti-SARS-CoV-2 antibodies using a data set of 352 serum samples from confirmed COVID-19 cases (individual data points are available in the Supplementary Table 1). Various performance metrics, including sensitivity (SEN), specificity (SPE), accuracy (ACC), positive LR (LR+), negative LR (LR−), and DOR, are summarized for each assay in Figure 2. Among the 352 positive samples, antibody positivity rates ranged from 41.8% for Anti-SARS-CoV-2 NCP IgG ELISA to 57.7% for GOLD ELISA COVID-19 IgG + IgM. Anti-SARS-CoV-2 NCP IgM ELISA exhibited a positivity rate of 44%. Our findings highlight exceptional specificity, exceeding 98.9%, for both GOLD ELISA COVID-19 IgG + IgM and Anti-SARS-CoV-2 NCP IgG ELISA. Anti-SARS-CoV-2 NCP IgM ELISA exhibited a specificity of 95.6%, with no significant difference between these assays, as indicated by overlapping 95% CI. Notably, GOLD ELISA COVID-19 IgG + IgM outperformed the other kits in terms of sensitivity and accuracy. When comparing GOLD ELISA COVID-19 IgG + IgM with Anti-SARS-CoV-2 NCP IgM ELISA, the former exhibited a higher DOR, although no statistically significant difference was observed. Because of its 100% specificity, Anti-SARS-CoV-2 NCP IgG ELISA resulted in incalculable LR+ and DOR values. Despite their lower sensitivity values, both the Anti-SARS-CoV-2 NCP IgM ELISA and Anti-SARS-CoV-2 NCP IgG ELISA yielded AUC values of approximately 70%, whereas GOLD ELISA COVID-19 IgG + IgM achieved an AUC value of 87%. These findings indicate moderate and elevated diagnostic capacity, respectively. Thus, all three assays can distinguish between positive and negative samples despite their lower sensitivity. Additionally, all three kits displayed a relatively low K index, indicating fair agreement with the reference tests.

Figure 2.

RI, AUC values, and diagnostic performance parameters for three commercial assays detecting anti-SARS-CoV-2 antibodies. The established RI cutoff value was 1.0 (dashed line).

Acc, accuracy; AUC, area under the curve; CI, confidence interval; DOR, diagnostic odds ratio; ELISA, enzyme-linked immunosorbent assay; Ig, immunoglobulin; K, Cohen's kappa index; LR, likelihood ratio; RI, reactivity index; Sen, sensitivity; Spe, specificity.

We assessed the performance of three commercial serological assays across various infection stages, categorized by symptom onset. We analyzed 146 samples within 0-7 days, 132 within 8-14 days, 33 within 15-21 days, and 41 after 21 days, as illustrated in Figure 3. Notably, all three assays exhibited improved performance, especially during the 15-21 days after symptom onset and beyond 22 days after symptom onset. The GOLD ELISA COVID-19 IgG + IgM achieved a sensitivity above 93% and an accuracy exceeding 97%, while the Anti-SARS-CoV-2 NCP IgM ELISA and Anti-SARS-CoV-2 NCP IgG ELISA achieved a sensitivity higher than 81% and an accuracy exceeding 91%. However, the Anti-SARS-CoV-2 NCP IgM ELISA showed a significant decrease in sensitivity, dropping from 81.8% at 15-21 days after symptom onset to 39% beyond 21 days after symptom onset. In all four infection stages, all tests effectively differentiated between positive and negative samples, despite lower sensitivity values in the first 2 weeks after symptom onset. However, it is important to note that both the Anti-SARS-CoV-2 NCP IgM ELISA and Anti-SARS-CoV-2 NCP IgG ELISA were unable to differentiate between positive and negative samples in the first 7 days after symptom onset.

Figure 3.

RI, AUC values, and diagnostic performance parameters for three commercial assays detecting anti-SARS-CoV-2 antibodies in samples categorized by symptom onset. The established RI cutoff value was 1.0 (dashed line).

AUC, area under the curve; Acc, accuracy; CI, confidence interval; ELISA, enzyme-linked immunosorbent assay; Ig, immunoglobulin; K, Cohen's kappa index; RI, reactivity index; Sen, sensitivity; Spe, specificity.

Discussion

According to our findings, the GOLD ELISA COVID-19 IgG + IgM identified 203 out of 352 positive samples, resulting in a sensitivity of 57.7%. According to the kit's package insert (version 04/20), when tested with a panel of 98 samples, this system exhibited sensitivities of 37.5%, 43.5%, and 98.5% for samples collected within 9, 10 to 15, and ≥16 days after symptom onset, respectively. Adapting our results to the time frames in the package insert, the GOLD ELISA COVID-19 IgG + IgM correctly identified 71 out of 190 samples (37.4%), 74 out of 98 samples (75.5%), and 58 out of 64 samples (90.6%) collected within ≤9, 10 to 15, and ≥16 days since symptom onset, respectively. It is important to note that the observed sensitivity in the third time frame was slightly lower than the manufacturer's reported sensitivity, accounting for the 95% CI. The GOLD ELISA COVID-19 IgG + IgM exhibited a sensitivity of 57.7%, surpassing the manufacturer's reported sensitivity in two out of the three analyzed time frames. The kit produced a single false-positive result among the 90 negative samples tested, resulting in a specificity of 98.9%, slightly higher than the manufacturer's reported specificity of 97.6%, but still within the 95% CI.

When comparing the GOLD ELISA COVID-19 IgG + IgM with an in-house receptor-binding-domain (RBD) ELISA, the GOLD ELISA COVID-19 IgG + IgM demonstrated a significantly higher sensitivity (57.68%) when assessing anti-RBD IgG (48%; P = 0.0049), but not when compared with anti-RBD IgM (51.1%; P = 0.0472) [22]. Regarding specificity, the GOLD ELISA COVID-19 IgG + IgM exhibited significant superiority when compared with the anti-RBD IgM ELISA (94.6%; P = 0.0043). However, CIs overlapped in RBD IgG ELISA (99.1%; P = 0.7655), rendering this variation statistically insignificant.

Regarding the Anti-SARS-CoV-2 NCP IgG ELISA, it accurately detected anti-N IgG in 147 out of 352 samples (41.8%). According to the package insert (version 26 JAN 2022), this test exhibited sensitivities of 80% and 94.6% for samples collected within ≤10 and >10 days since symptom onset or first positive RT-qPCR result, respectively, using a panel of 74 samples from 69 European individuals. However, our results indicated that the test correctly identified 45 out of 208 samples (21.6%) and 102 out of 144 samples (70.8%) for samples collected within ≤10 and >10 days after symptom onset, respectively. These sensitivity values for both time frames were significantly lower than those reported by the manufacturer. The specificity for the Anti-SARS-CoV-2 NCP IgG ELISA matched the manufacturer's specification. According to the package insert, when assessing 1248 negative samples, the specificity for the kit was 99.8%. In our evaluation, none of the 90 negative samples yielded a false-positive result, resulting in a specificity of 100%. The Anti-SARS-CoV-2 NCP IgG ELISA demonstrated lower sensitivities than reported by the manufacturer, indicating potential variations in performance across different populations and settings, but did match the performance disclosed by independent parties. However, the specificity matched the manufacturer's specifications, reaching 100% in our evaluation.

Our findings for the Anti-SARS-CoV-2 NCP IgM ELISA diverged from the manufacturer's claims, particularly for samples collected shortly after a positive RT-qPCR test result. The kit detected IgM anti-SARS-CoV-2 N antigen in 155 out of 352 samples, with an overall sensitivity of 44%. According to the kit package insert (version 2021-04-22), the assay's sensitivity was reported as 88.2%, 70.6%, 53.6%, 45.5%, 50%, and 11.8% for samples collected at ≤10, 11-15, 16-25, 26-35, 36-45, and ≥46 days since symptom onset or first positive RT-qPCR result. Our observed sensitivity for samples collected within 10 days or less since the first positive RT-qPCR was 30.3% (63/208), notably lower than the manufacturer's claim of 88.2% for this time frame, but in accordance with data reported independently [23]. However, for samples collected at 11-15, 16-25, and ≥26 days since the first positive RT-qPCR result, our observed sensitivities of 72.5% (58/80), 82.1% (23/28), and 30.6% (11/36), respectively, were consistent with the manufacturer's stated performance and with the independently reported performance [23,24]. In terms of specificity, when assessing 90 negative samples collected from blood donors before the ongoing SARS-CoV-2 outbreak, the Anti-SARS-CoV-2 NCP IgM ELISA produced four false-positive results. This indicated a specificity of 95.6%, slightly below the 98.6% specificity reported in the package insert by the manufacturer when evaluating 821 negative samples, but still within the 95% CI.

It is essential to note that test performance may vary according to the population being tested, and manufacturers should use diverse sample panels to properly assess IVD device performance. Obtaining a well-characterized and diverse panel of positive samples during the early months of the pandemic may have been challenging because of the relatively low number of cases and the availability of commercial IVD kits. Standardized validation protocols by regulatory agencies and rigorous independent evaluations of tests are crucial to address discrepancies between manufacturer-claimed diagnostic performance and observed results.

In the era of widespread vaccination against SARS-CoV-2, the utility and specificity of serological assays, especially those targeting the N protein, deserve careful examination. Currently, vaccines granted emergency use authorization primarily stimulate an immune response to the S protein, the mechanism through which the virus enters human cells. Consequently, vaccinated individuals predominantly develop antibodies targeting the S protein rather than the N protein. In this context, serological tests measuring anti-N IgM and IgG offer a unique benefit. These antibodies are not elicited by vaccines focusing on the S protein; thus, the detection of anti-N antibodies in vaccinated individuals likely signifies a natural infection, rather than a vaccine-induced immune response. Therefore, anti-N IgM and IgG assays are particularly valuable for differentiating between responses to vaccination and natural infections among vaccinated individuals.

There are a number of limitations to our study. Firstly, we did not include samples of other respiratory tract infections or unrelated coronavirus infections, such as HCoV 229E, NL63, MERS or SARS-CoV-1. Secondly, while most studies reporting diagnostic performance used convalescent sera, the sera used in this study came from hospitalized individuals with ongoing COVID-19, which resulted in uneven comparisons between reports. Furthermore, a lack of independently evaluated data regarding GOLD ELISA COVID-19 IgG + IgM also affected discussion depth.

Conclusion

In summary, our study evaluated the performance of three commercial serological assays for detecting specific anti-SARS-CoV-2 antibodies. All three kits efficiently distinguished between negative and positive samples, but their performance varied depending on the time of infection. The GOLD ELISA COVID-19 IgG + IgM demonstrated the highest performance values in terms of sensitivity and DOR. This superior performance can be attributed to the antigenic preparation, which captures both IgG and IgM against SARS-CoV-2 S1, S2, and N proteins. In contrast, Euroimmun NCP ELISAs only use N protein as the coating antigen and solely detect one class of Ig (IgG or IgM). These findings underscore the importance of considering the assay design and target antigens when assessing the diagnostic performance of serological tests for SARS-CoV-2. Our study highlights the importance of considering population-specific factors when evaluating diagnostic performance. Additionally, the limitations of our study, such as the lack of samples from other respiratory infections and the use of sera from hospitalized individuals, should be acknowledged. Future studies with more diverse sample panels and standardized validation protocols are crucial to enhancing the reliability of diagnostic tests. Moreover, incorporating the analysis of anti-N IgM and IgG tests in vaccinated populations could substantially broaden the relevance and applicability of our findings. Future investigations should aim to validate these assays across varied demographics, including those vaccinated with different COVID-19 vaccines, to better delineate their utility in both clinical and public health contexts. Addressing these limitations will contribute to a more accurate understanding of the diagnostic capabilities of these assays in different clinical scenarios, providing valuable information for health care practitioners and policymakers.

Declarations of competing interest

The authors have no competing interests to declare.

Appendix. Supplementary materials

Supplementary Table 1: Reactivity Index for diagnostic performance assessment of commercial serological tests for SARS-CoV-2 diagnosis.