Background
Semi-quantitative and quantitative immunoassays are the most commonly used methodology to evaluate immunity post immunization. Objectives: To compare four quantitative SARS-CoV-2 serological assays in COVID-19 patients and immunized healthy individuals, cancer patients, and patients with immunosuppressive therapy. Study design: 210 serological samples from COVID-19 infection and vaccination cohorts were used to create a serological sample repository. Serological methods from four manufacturers, namely Euroimmun, Roche, Abbott, and DiaSorin, were evaluated for quantitative, semi-quantitative, and qualitative antibody measurements. All four methods measure IgG antibodies against the SARS-CoV-2 spike receptor–binding domain and report the results in Binding Antibody Unit/mL (BAU/mL). A Total Error Allowable (TEa) of ±25% was chosen as the criteria to determine whether two methods are clinically equivalent quantitatively. Semi-quantitative results (titers) were derived using numeric antibody concentration divided by the cut-off value for each method. Results: All paired quantitative comparisons demonstrated unacceptable performance. With ±25% as TEa, the best agreement was 74 (35.2% out of 210 samples) between Euroimmun and DiaSorin, whereas the lowest agreement was 11 (5.2% out of 210 samples) between Euroimmun and Roche. Antibody titers amongst all four methods were significantly different (p < 0.001). The highest titer difference from the same sample is between Roche and DiaSorin with a 1392-fold difference. On qualitative comparison, none of the paired comparison showed acceptable comparison (p < 0.001). Conclusions: Poor correlation exists between four evaluated assays, quantitatively, semi-quantitatively, and qualitatively. Further harmonization of assays is required to achieve comparable measurements.
Keywords: SARS-CoV-2 vaccination, Serological assays, Comparability, Standardization
1. Background
It has been more than two years into the global pandemic of SARS-CoV-2 infection, and over twelve billion doses of vaccines have been administered with over 600 million people being infected with COVID-19 worldwide [1]. Health Canada (HC) and the U.S. Food and Drug Administration (FDA) have approved multiple vaccines for Emergency Use Authorization (EUA), such as Moderna SpikeVax (mRNA-1273) and Pfizer-BioNTech Comirnaty (BNT162b2) [2], [3]. Coronaviruses have four structural proteins: the spike protein (S), the nucleocapsid (N), the envelope protein (E), and the membrane protein (M) [4]. The S protein, which protrudes from the virus envelope, is immunodominant and consists of two subunits: the S1 protein, which contains the receptor binding domain (RBD), and the S2 protein, which mediates cell membrane fusion [5]. Multiple platforms of serological testing against different SARS-CoV-2 antigens, such as the spike protein (S) and nucleocapsid protein (N), are available.
Immunoassays, e.g., enzyme-linked immunosorbent assay (ELISA), are the most commonly used methodology to evaluate immunity after immunization [6]. For most other vaccines, a universal cut-off value based on either semi-quantitative or quantitative immunoassay is often chosen to represent protection and immunity [6]. As demonstrated by the Rubella vaccine, the cut-off value should be continuously monitored and adjusted with the aid of large epidemiological studies [7], and may differ among countries [8]. Therefore, it is critical that results across various analytical platforms are comparable and could properly define the immunity post both immunization and infection. The National Committee of Clinical Laboratory Scientists (NCCLS) periodically publishes guidelines for clinical laboratories to properly report testing results with respect to immunity, after a high level of concordance is achieved among various analytical platforms [7], [9].
In healthy individuals, it is known that humoral responses could increase significantly with additional booster doses of SARS-CoV-2 vaccination [10]. Furthermore, humoral response is poor post-immunization in immunocompromised individuals [11]. As a result, the serological methods approved for clinical use must provide accurate results with a broad antibody dynamic range from low to high end.
2. Objectives
To compare four quantitative SARS-CoV-2 serological assays in COVID-19 patients and immunized healthy individuals, cancer patients, and patients with immunosuppressive therapy, using samples collected from the first to the fourth doses of vaccination. We evaluated the four serological assays labeled for clinical use in an extended dynamic (measurable) range.
3. Study design
3.1. Recruitment, sample, and data collection
Institutional ethics committee approval and consent from participants were obtained. From May 2021 to July 2022, we enrolled healthy individuals, cancer patients, and patients with immunosuppressive therapy post one to four doses of COVID-19 vaccination in Kingston, Ontario, Canada. Serum samples from hospitalized COVID-19 patients admitted at the Kingston Health Sciences Centre between February 2021 and April 2022 were obtained as well. There were 210 samples in total, including 82 COVID-19 positive samples and 128 samples from immunized individuals. The latter includes 42 samples from healthy individuals, 44 from cancer patients, and 42 from patients with immunosuppressive therapy (23 from renal transplant patients and 19 from patients on other immunosuppressive therapy). Immunosuppressive therapy for renal transplant patients consisted of triple therapy with a calcineurin inhibitor (tacrolimus or cyclosporine), with an antiproliferative agent (mycophenolate, azathioprine, sirolimus, or everolimus), and corticosteroids. Immunized participants received BNT162b2, AZD1222, mRNA-1273, or a mixture of vaccines from multiple manufacturers.. Among 128 blood samples from immunized participants, 35, 70, 22, and 1 were obtained post the first, second, third, and fourth doses of immunization, respectively. No participants received the bivalent vaccine.
In hospitalized COVID-19 patients, their infection was confirmed by polymerase chain reaction (PCR)/gene mutation analysis performed at Kingston Health Sciences Center following standard protocol. Both positive and negative PCR results were reported to the Public Health Ontario database.
3.2. Quantitative, Semi-quantitative, and qualitative antibody measurement
Four serological methods were evaluated, including Euroimmun Anti-SARS-CoV-2 QuantiVac ELISA (IgG) (product number: EI 2606-9601-10 G), Abbott Architect AdviseDx SARS-CoV-2 IgG II (product number: 6S60), Roche Elecsys Anti-SARS-CoV-2 S (product number: 09289267190), and DiaSorin LIAISON SARS-CoV-2 TrimericS IgG (product number: 311510). Testing of 210 samples was performed at three academic clinical laboratories by medical laboratory technologists. Testing was performed on instruments from each serology test manufacturer, namely, EUROIMMUN Analyzer 1, Abbott ARCHITECT, Roche Cobas e602, and DiaSorin LIAISON. All results were reported in Binding Antibody Unit/mL (BAU/mL). All four methods measure IgG antibodies against the SARS-CoV-2 spike receptor–binding domain (S1).
Semi-quantitative IgG antibody titer against the SARS-CoV-2 spike receptor–binding domain (S1) was derived from the quantitative antibody measurement. Titer was determined by dividing quantitative antibody levels by corresponding cut-off values from each manufacturer. Antibody titers were rounded to a whole number and those greater than 1 were included in the comparison.
Quantitative antibody levels for the SARS-CoV-2 S1 domain were compared against their corresponding positive cut-off value to determine the qualitative test result, either positive or negative. EUROIMMUN testing has a borderline range (≥25.6 to <35.2 BAU/mL); antibody results within the borderline range were excluded in the quantitative comparison.
3.3. Data presentation and statistical analysis
Data presentation and quantitative method comparison was performed using EP Evaluator 12 (Data Innovations LLC, United States). Based on the common regulatory requirements for immunological assays, a Total Error Allowable (TEa) of ±25% was chosen as the criteria to determine whether the two methods are clinically equivalent.
Analysis of semi-quantitative and quantitative comparisons was performed using SPSS Statistics (IBM SPSS Statistics, United States). All groups were determined to not be normally distributed (p < 0.001, Shapiro-Wilk normality test). The results were reported as medians and interquartile range (IQR). Friedman’s and Pearson’s Chi-Square tests were used to determine statistical significance between groups.
4. Results
4.1. Characteristics of the study cohort
The demographics and quantitative antibody levels of the study participants are summarized in Table 1 . Data for the vaccine cohort is further separated into healthy individuals, cancer patients, and patients with immunosuppressive therapy (IST). The dynamic range of the quantitative IgG antibody levels against SARS-CoV-2 are shown.
Table 1.
Characteristics of the Study Population.
| Immunized |
COVID-19 Positive | ||||
|---|---|---|---|---|---|
| Healthy | Cancer | IST | |||
| n = 210 | n = 42 | n = 44 | n = 42 | n = 82 | |
| Age, median (range) | 64 (12–93) |
57 (19–80) | 66 (40–92) | 61 (18–83) | 65 (12–93) |
| Sex | |||||
| Male (%) | 92 (43.8) | 18 (42.9) | 16 (36.4) | 20 (47.6) | 38 (46.3) |
| Female (%) | 118 (56.2) | 24 (57.1) | 28 (63.6) | 22 (52.4) | 44 (53.7) |
| Quantitative Samples, median (range) | |||||
| Euroimmun (BAU/mL) | 390.5 (3.2–67,910) | 505 (14.1–10,448) | 157 (3.2–11,442) | 160 (3.2–7,040) | 1,044 (3.2–67,910) |
| Abbott (BAU/mL) | 172.8 (3.1–35268.1) | 167.8 (3.1–6,128.4) | 129.5 (3.1–6,816.4) | 75.1 (3.1–6,792.4) | 848.0 (3.1–35268.1) |
| Roche (BAU/mL) | 241 (0.4–257,160) | 335 (1.99–50,573) | 240 (0.4–36,052) | 371 (0.4–42,005) | 212 (0.4–257,160) |
| DiaSorin (BAU/mL) | 623 (4.8–86,100) | 865 (6.98–26,600) | 364 (4.8–21,200) | 175 (4.8–14,000) | 1,065 (4.8–86,100) |
4.2. Characteristics of serological testing methods
The main characteristics of the four serological testing methods for IgG against the SARS-CoV-2 S1 protein are summarized in Table 2 . The four methods are all approved for use by HC for clinical use and the FDA for EUA. Three manufacturers set a single cut-off value representing seroconversion, whereas Euroimmun provides a borderline range (≥25.6 to <35.2 BAU/mL).
Table 2.
Main Characteristics of the Four Immunoassays Evaluated.
| Anti-SARS-CoV-2 QuantiVac ELISA (IgG) | Architect AdviseDx SARS-CoV-2 IgG II | Elecsys Anti-SARS-CoV-2 S | LIAISON SARS-CoV-2 TrimericS IgG | |
|---|---|---|---|---|
| Manufacturer | EUROIMMUN Medizinische Labordiagnostika Ag | Abbott Diagnostics Division | Roche Diagnostics | DiaSorin Inc. |
| Clinical using Labeling | HC & FDA EUA | HC & FDA EUA | HC & FDA EUA | HC & FDA EUA |
| Detection of IgG antibodies | Quantitative | Quantitative | Quantitative | Quantitative |
| Measuring interval | 3.2–384.0 | 3.1–3550.0 | 0.40–250.0 | 4.81–2,080.0 |
| Units | BAU/mL | BAU/mL | BAU/mL | BAU/mL |
| Cut off (positive) | ≥35.2Borderline Range (≥25.6 to < 35.2) |
≥7.1 | ≥0.80 | ≥33.8 |
4.3. Comparison of quantitative antibody measurement
Fig. 1 demonstrates the comparisons of the four serological methods amongst 210 samples, which included both immunized and COVID-19 positive samples. All paired comparisons demonstrated unacceptable agreement, using TEa of ±25% as the criterion. The best agreement was 74 (35.2% out of 210 samples) between Euroimmun and DiaSorin, whereas the lowest agreement was 11 (5.2% out of 210 samples) between Euroimmun and Roche. Fig. 1 also shows the linear regression (Deming regression statistics) model for each comparison, with the slope ranging from 1.3 to 7.6. Further analysis based on immunized and COVID-19 positive samples separately demonstrated similar results, which are provided in the supplemental data.
Fig. 1.
Pairwise comparisons of quantitative serological testing.
4.4. Comparison of semi-quantitative antibody measurement
Fig. 2 represents the antibody titers for the four serological methods compared in a pairwise fashion, using 210 samples. The antibody titer of each sample was calculated by numeric antibody concentration divided by the cut-off value for each method. Antibody titers amongst all four methods were significantly different (p < 0.001, Friedman’s test). The highest titer difference from the same sample was between Roche and DiaSorin with a 1392-fold difference. Further analysis based on immunized and COVID-19 positive samples separately demonstrated similar results, which are provided in the supplemental data.
Fig. 2.
Pairwise comparison of semi-quantitative serological testing agreement.
4.5. Comparison of qualitative antibody measurement
The qualitative comparison of the four serological testing methods is summarized in Table 3 . Out of the 173 samples in which there was perfect agreement across all four serologic methods, 148 results were positive and 25 were negative. There were 31 samples in which there was disagreement between at least one of the four testing methods. These disagreement samples were further categorized based on which test provided the distinct result. None of the paired comparisons based on qualitative results showed agreement (p < 0.001, Pearson's Chi-Square test).
Table 3.
Qualitative Antibody Serological Testing Agreement.
| Agreement Type | n | Percentage (%) |
|---|---|---|
| Total Cases | 210 | |
| Perfect Agreement | 173 | 82 |
| Positive | 148 | 70 |
| Negative | 25 | 12 |
| Disagreement | 31 | 15 |
| Euroimmun - | 4 | 2 |
| Abbott + | 2 | 1 |
| Abbott - | 1 | 0.5 |
| Roche + | 12 | 6 |
| Roche - | 3 | 1 |
| DiaSorin + | 3 | 1 |
| DiaSorin - | 1 | 0.5 |
| Euroimmun &DiaSorin – Abbott & Roche + |
4 | 2 |
| Abbott & DiaSorin – Euroimmun & Roche + |
1 | 0.5 |
| Removed Samples | 6 | 3 |
5. Discussion
Based on the knowledge from other vaccination programs, there are multiple surrogate markers to determine immunity. Those include antibody levels determined by immunoassay, viral and bacterial neutralization assay, interferon assay, and hemagglutination assay [6]. The monitoring of cell-mediated immune response to a viral antigen in the laboratory is a labor-intensive procedure. It is performed only in specialized laboratories, e.g., in advanced flow cytometer laboratories, and not used routinely for clinical purposes. In contrast, the detection of circulating antibodies can be performed relatively easily using high-throughput serological assays. Therefore, semi-quantitative (measuring antibody titer) and quantitative (measuring antibody concentration) immunoassay is the most commonly used methodology to evaluate immunity after immunization [6].
Serological testing is essential to evaluate the effectiveness of SARS-CoV-2 vaccination [12], [13]. Since the onset of the COVID-19 pandemic and especially after SARS-CoV-2 vaccination became available globally, extensive research literature has been published to evaluate the humoral immune responses using various immunoassays [12], [13]. Results from various analytical methods should be comparable in order to properly understand the conclusions from various publications. Furthermore, to reliably evaluate SARS-CoV-2 immunization efficacy for clinical purposes, it is critical to develop robust immunoassays and conduct extensive method comparison and standardization. After a high level of agreement among various methods is required, a cut-off based on semi-quantitative or quantitative assays for SARS-CoV-2 could be chosen to represent protection and immunity.
Our study demonstrated poor agreements among four serological methods approved for clinical use by Health Canada and the FDA, manufactured by Euroimmun, Abbott, Roche, and DiaSorin. As the TEa is not known yet for SARS-CoV-2 serological responses, we arbitrarily chose ±25%, which is commonly used for other immunoassays. Using this criterion, poor agreements were observed in all comparisons between any two methods. Although the comparison between Euroimmun and DiaSorin is superior to any other comparisons, only 35.2% of 210 samples were deemed acceptable. Semi-quantitatively, after converting the quantitative results to titers by dividing quantitative antibody levels by corresponding cut-off values from each manufacturer, titers of any two methods differed significantly (p < 0.001). The highest titer difference was between Roche and DiaSorin with a 1392-fold difference. Clearly, current available serological methods differ significantly both quantitatively and semi-quantitatively and cannot provide the performance required for routine clinical use. Furthermore, when we applied the seroconversion cut-off values in the results and determined the positive rates in all four methods in 210 samples, there were significant statistical differences (p < 0.001) in any two methods compared.
In November 2020, the World Health Organization (WHO) established an international standard and reference material for anti-SARS-CoV-2 immunoglobulin (NIBSC code 20/136) [14]. The aim of this reference material is to harmonize humoral immune response assessment after natural infection or vaccination. All fourth methods being evaluated are traceable to this reference material, and their results could be reported in BAU/mL, a unit recommended by the WHO. Euroimmun, in their instruction document, provided the correlation with this reference material with R2 = 0.99. It is unclear why poor agreement was observed in our study. In July 2022, WHO provided the second international standard for anti-SARS-CoV-2 immunoglobulin and reference panel for antibodies to SARS-CoV-2 variants of concern [15]. Likely, the application of this new reference material could improve the concordance of antibody measurements, especially for COVID-19 positive samples.
In conclusion, we found poor correlation between all evaluated assays, quantitatively, semi- quantitatively, and qualitatively. Further harmonization of assays is required to achieve comparable measurements.
Declaration of Competing Interest
The study was funded by the PSI foundation: 2020–1972, and Southeastern Ontario Academic Medical Organization Innovation Fund 2021. The study received in-kind reagent support and limited financial support from EUROIMMUN AG, Abbott, Roche, and DiaSorin.
Acknowledgement
We thank Brie Belle Fraser and Pavla Zabojnikova for their contribution to the research. We especially thank the study participants.
Footnotes
Supplementary data to this article can be found online at https://doi.org/10.1016/j.clinbiochem.2023.02.010.
Appendix A. Supplementary data
The following are the Supplementary data to this article:
References
- 1.Statistics and Research Coronavirus (COVID-19) Vaccinations. https://ourworldindata.org/covid-vaccinations, (accessed on August 03, 2022). .
- 2.Health Canada, Approved COVID-19 Vaccines, (accessed on August 03, 2022). https://www.canada.ca/en/health-canada/services/drugs-health-products/covid19-industry/drugs-vaccines-treatments/vaccines.html.
- 3.The U.S. Food and Drug Administration, “COVID-19 Vaccines,” (accessed on August 03, 2022). https://www.fda.gov/emergency-preparedness-and-response/coronavirus-disease-2019-covid-19/covid-19-vaccines.
- 4.Cui J., Li F., Shi Z.-L. Origin and evolution of pathogenic coronaviruses. Nat. Rev. Microbiol. 2019;17(3):181–192. doi: 10.1038/s41579-018-0118-9. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Bisht H., Roberts A., Vogel L., Bukreyev A., Collins P.L., Murphy B.R., Subbarao K., Moss B. Severe acute respiratory syndrome coronavirus spike protein expressed by attenuated vaccinia virus protectively immunizes mice. Proc. Natl. Acad. Sci. U. S. A. 2004;101(17):6641–6646. doi: 10.1073/pnas.0401939101. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Plotkin S.A. Correlates of protection induced by vaccination. Clin. Vaccine Immunol. 2010;17(7):1055–1065. doi: 10.1128/CVI.00131-10. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.National Committee of Clinical Laboratory Scientists (NCCLS). Detection and Quantitation of Rubella IgG Antibody in the Clinical Laboratory; Approved Guideline. 1997 Wayne, PA.
- 8.Gao Z., Wood J.G., Burgess M.A., Menzies R.I., McIntyre P.B., MacIntyre C.R. Models of strategies for control of rubella and congenital rubella syndrome-A 40 year experience from Australia. Vaccine. 2013;31(4):691–697. doi: 10.1016/j.vaccine.2012.11.043. [DOI] [PubMed] [Google Scholar]
- 9.National Committee of Clinical Laboratory Scientists (NCCLS). Evaluation and Performance Criteria for Multiple Component Test Products Intended for the Detection and Quantification of Rubella IgG Antibody, I/LA6-T, Tentative Guidline. 1985 Wayne, PA.
- 10.Macrae K., Gong C.Y., Sheth P., Martinez-Cajas J., Gong Y. Quantitative analysis of SARS-CoV-2 serological responses post three doses of immunization and prior to breakthrough COVID-19 infections. Vaccines. 2022;10(10):pp. doi: 10.3390/vaccines10101590. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Robinson A., Mazurek A., Xu M., Gong Y. Quantitative analysis of SARS-CoV-2 antibody status between patients with cancer and healthy individuals with extended vaccination dosing intervals in Canada. Curr. Oncol. 2022;29(1):68–76. doi: 10.3390/curroncol29010006. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Folegatti P.M., et al. Safety and immunogenicity of the ChAdOx1 nCoV-19 vaccine against SARS-CoV-2: a preliminary report of a phase 1/2, single-blind, randomised controlled trial. Lancet. 2020;396(10249):467–478. doi: 10.1016/S0140-6736(20)31604-4. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Jackson L.A., Anderson E.J., Rouphael N.G., Roberts P.C., Makhene M., Coler R.N., McCullough M.P., Chappell J.D., Denison M.R., Stevens L.J., Pruijssers A.J., McDermott A., Flach B., Doria-Rose N.A., Corbett K.S., Morabito K.M., O’Dell S., Schmidt S.D., Swanson P.A., Padilla M., Mascola J.R., Neuzil K.M., Bennett H., Sun W., Peters E., Makowski M., Albert J., Cross K., Buchanan W., Pikaart-Tautges R., Ledgerwood J.E., Graham B.S., Beigel J.H. An mRNA vaccine against SARS-CoV-2 - preliminary report. N. Engl. J. Med. 2020;383(20):1920–1931. doi: 10.1056/NEJMoa2022483. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Mattiuzzo. et al. Establishment of the WHO International Standard and Reference Panel for anti-SARS-CoV-2 antibody.2020.(https://www.who.int/publications/m/item/WHO-BS-2020.2403).
- 15.WHO/BS/2022.2427: Establishment of the 2nd WHO International Standard for anti-SARS-CoV-2 immunoglobulin and Reference Panel for antibodies to SARS-CoV-2 variants of concern. https://www.who.int/publications/m/item/who-bs-2022.2427 (accessed Nov. 30, 2022).
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.