Performance of commercial SARS-CoV-2 wild-type and Omicron BA.1 antibody assays compared with pseudovirus neutralization tests

Abstract

Background

Commercially available ELISA-based antibody tests are used to approximate vaccination success against SARS-CoV-2 in at-risk patients, but it is unclear whether they correlate with neutralization of the Omicron variant.

Methods

269 serum samples of a cohort of 44 non-immunosuppressed participants and 65 MTX-treated rheumatic patients taken before and after COVID-19 booster vaccinations were measured using COVID-19 antibody testing systems with wild-type and Omicron BA.1 antigens developed by three different manufacturers (surrogate virus neutralization test cPass, and binding antibody tests QuantiVac and SeraSpot), as well as with a pseudovirus neutralization test (pVNT). The pVNT was considered the gold standard for determining the presence and level of anti-SARS-CoV-2 antibodies.

Results

All three wild-type ELISAs showed excellent test performance compared with wild-type neutralization in pVNT. However, out of 56 samples without Omicron BA.1 neutralization in pVNT, 71.4% showed positive results in at least one and 28.6% in all three wild-type ELISAs at the manufacturer-defined cut-offs. Omicron ELISAs showed either decreased specificity (57.1% and 55.4% for binding ELISAs) or sensitivity (51.2% in cPass) compared to Omicron neutralization in pVNT. The proportion of any false positive results among all samples decreased from 26.5% before to 3.2% after booster vaccination, however binding antibody test specificities remained below 70%.

Conclusions

We found a poorer test performance of new Omicron antibody test systems compared to wild-type tests in detecting neutralizing antibodies against the corresponding SARS-CoV-2 variants. Decisions for booster vaccination or passive immunization of at-risk patients should not be based solely on antibody test results.

1. Introduction

Vaccine- or infection-induced humoral immunogenicity against SARS-CoV-2 can most reliably be measured by virus neutralization, either in plaque reduction (PRNT) or pseudovirus neutralization tests (pVNT) [1]. However, these require special expertise, are cost intensive and not widely available. Therefore, surrogate virus neutralization tests (sVNT) or binding antibody tests without differentiation of neutralizing antibodies are commonly used instead. Despite the differences in functionality, they have been shown to correlate well with virus neutralization [2,3] and vaccine efficacy [4] against wild-type SARS-CoV-2.

The broad use of these tests has increased our understanding of populations who are at risk of attenuated immune responses to vaccinations, in particular organ transplant recipients, cancer patients [5] and patients treated with immunosuppressants [6], such as methotrexate (MTX) [7,8]. Individual authors [9,10] as well as medical committees in Germany and Austria recommend that the success of vaccination in high-risk patients should be verified by antibody testing [11,12], and absence of an antibody response has been recommended as a requirement and indication for pre-exposure prophylaxis [11,13]. Other societies are undecided [14] or critical of antibody testing because of undefined correlates of protection and propose to solely focus on clinical risk factors [15,16].

Importantly, commercially available antibody tests and cut-offs for a positive response were developed for the detection of past infections [17], and not validated for the verification of vaccination success. This is further complicated by the evolution of virus sequelae with structural variations such as the Omicron variant [18], which necessitate booster vaccinations for sufficient immune response [19]. Some manufacturers have produced adapted antibody tests using Omicron BA.1 antigens. However, to our knowledge validations have not yet been published and are generally hindered by the low number of patients who were immune-naive before Omicron infection. It remains unclear how well the wild-type and BA.1 antibody tests correlate with the neutralization of the Omicron variant.

Therefore, we compared the performance of commercially available wild-type and Omicron BA.1 antibody test systems from three manufacturers (Euroimmun, GenScript, Seramun) with neutralization of wild-type and Omicron BA.1 determined by pVNT as the gold standard. We aimed to investigate whether antibody levels measured by ELISAs reliably correspond with virus neutralization and can form a basis for clinical decisions regarding additional vaccinations or passive immunization in times of new virus variants.

2. Materials and methods

2.1. Samples and study population

Serum samples were collected within the VACCIMMUN study (rheumatic patients under MTX treatment) [7,8] as well as the COVIM, EICOV and COVIMMUNIZE studies (non-immunosuppressed persons, NIP, [19]) at Charité Universitätsmedizin Berlin, Germany. Blood samples from May 2021 to March 2022 collected at the following time points were used in this analysis: 2 weeks after the second COVID-19 vaccination and shortly before, 4 and 12 weeks after an mRNA booster against wild-type SARS-CoV-2, as well as shortly before and 4 weeks after a second booster (if applicable). The inclusion criteria for all participants in this analysis were age 18 years or older and three vaccinations against COVID-19 with a vaccine authorized for use in Germany (BNT162b2, mRNA-1273, AZD1222). Further inclusion criteria for the MTX cohort were a rheumatic diagnosis and immunosuppressive therapy including MTX. The NIP cohort consisted of health-care workers and elderly individuals (≥70 years) and were not allowed to have a rheumatic diagnosis or immunosuppressive medication in our study. Samples from individuals who had a COVID-19 infection prior to one of the blood collections were excluded. The patients provided information regarding medical history including COVID-19 vaccination status and/or infection and immunosuppressive therapy directly. The studies were ethically approved by local authorities (see appendix). All participants provided written informed consent.

2.2. Laboratory analysis

Each sample was analyzed with different COVID-19 antibody testing systems developed by three different manufacturers (cPass by GenScript, QuantiVac and Omicron ELISA by Euroimmun, SeraSpot by Seramun) using wild-type (Wu01) and Omicron BA.1 antigens, as well as with a SARS-CoV-2 pVNT (overview in Table 1 ).

Table 1.

Overview of compared test systems.

Abbreviation	Manufacturer	Assay name	Method	Antigen	Isotype	Units
Wt-pVNT	Institute of virology, UHC	In-house pVNT	pVNT	spike protein, wt	All	ID50
Om-pVNT				spike protein, BA.1
Wt-cPass	GenScript	cPass SARS-CoV-2 Neutralization Antibody Detection Kit	sVNT using ELISA technique	RBD, wt	All	Inhibition%
Om-cPass				RBD, BA.1
Wt-QuantiVac	Euroimmun	Anti-SARS-CoV-2 QuantiVac ELISA (IgG)	ELISA	S1, wt	IgG	BAU/ml
Om-QuantiVac		Anti-SARS-CoV-2 Omicron ELISA (IgG)		S1, BA.1		RU/ml
Wt-SeraSpot	Seramun	SeraSpot Anti-SARS-CoV-Z2 IgG	ELISA Spot test	RBD, wt	IgG	BAU/ml
Om-SeraSpot				RBD, BA.1		RU/ml

BAU, binding antibody units; ELISA, Enzyme-linked Immunosorbent Assay; ID50, 50% inhibitory dilution; IgG, immunoglobulin G; om, Omicron BA.1; pVNT, Pseudovirus neutralization assay; RBD, receptor-binding-domain; RU, relative units; S1, subunit 1 of spike protein; sVNT, Surrogate virus neutralization test; UHC, university hospital Cologne; wt, wild-type (refers to Wu01). The abbreviations Wt/Om in the first column refer to the antigens used in the test.

The pVNT was considered the gold standard for determining the presence and level of anti-SARS-CoV-2 neutralizing antibodies. Non-replicating and genetically modified pseudoviruses expressing plasmids encoding wild-type/Wu01 or BA.1 spike protein and a quantifiable luciferase gene were used. The pseudoviruses enter HEK293T host cells through interaction of the viral S protein with the cellular ACE2 receptor. In the infected cell, the viral luciferase gene is expressed and the amount of pseudovirus in the host cell can be determined by measuring a luminescence signal. The background relative luminescence units (RLUs) of non-infected cells were subtracted and the 50% inhibitory serum dilution (ID50) that resulted in a 50% reduction of signal compared to the virus-infected untreated control was determined. The lower limit of quantification (LLOQ) is a 50% inhibitory serum dilution (ID₅₀) of 10, lower values were assigned to half the LLOQ. Detailed nucleotide sequences and transfection procedures have been published previously [7].

The cPass is an sVNT using ELISA technique to measure the capacity of a serum sample to neutralize the interaction between RBD (receptor-binding domain of the spike protein) and the ACE2-receptor after pre-incubation with either wild-type or Omicron BA.1 RBD. The measured optical density (OD) is inversely proportional to the level of neutralizing antibodies present, which is expressed as inhibition percentage and calculated using the OD of the negative control for each individual sample.

Both Euroimmun test systems are conventional quantitative ELISAs, which measure IgG antibodies against the S1 subunit of the spike protein (wild-type or Omicron BA.1). The SeraSpot test is a microarray-based immunoassay allowing simultaneous quantitative measurement of antigens spotted on a microplate, of which wild-type and Omicron BA.1 anti-RBD-IgG spot results were used for this analysis. Only the wild-type tests by Euroimmun and Seramun are calibrated against the WHO international standard 20/136 and expressed in BAU/ml. All ELISA-based tests are commercially available (Seramun tests are currently approved for research use only) and were performed as per manufacturer's instructions.

2.3. Statistical analysis

Descriptive statistics included mean with SD and absolute and relative frequencies. Differences between MTX and NIP were analyzed with Fisher's exact test (sex) and the unpaired t-test with Welch's correction (age, BMI). Spearman's rank correlation was used to test for correlation between the ELISAs and the pVNT. A presence of measurable neutralization in the pVNT was used as a surrogate of vaccination response and as a reference for the calculation of sensitivity and specificity of the ELISAs. Test results that deviated from pVNT results are referred to as “false positive” or “false negative” in this study.

A receiver operating characteristic (ROC) analysis was performed for each comparison and areas under the curve were calculated (AUC). Analyses were performed at manufacturers’ cut-off as well as at cut-offs for positivity determined by maximum Youden index and >99% specificity. The Fisher's exact test was applied to test for characteristics associated with false positive samples. DeLong test was used to compare AUCs. Statistical analyses were performed using GraphPad Prism V.9.4.1.

3. Results

3.1. Samples and pVNT results

A total of 269 serum samples taken before (n = 83) or after (n = 186) a COVID-19 booster vaccination from 44 participants without immunosuppression (NIP, mean age 61.8 years, 65.9% female, mean BMI 25.8 kg/m²) and 65 rheumatic disease patients receiving MTX therapy (mean age 60.8 years, 75.4% female, mean BMI 26.2 kg/m²) were investigated (characteristics in table 2 ). MTX patients provided 185 samples and NIP provided 84 samples. There was no statistically significant difference between patients under MTX therapy and NIP and their respective samples regarding sex, age and BMI.

Table 2.

Participant and sample characteristics.

Characteristics	Participants n = 109	Samples n = 269
Demographics
age, mean (SD; range)	61.2 (16.5; 22.0–86.0)	60.8 (15.8; 22.0–86.0)
female, n (%)	78 (71.6)	191 (71.0)
BMI, mean, (SD; range)	26.0 (4.1; 18.0–37.2)	26.0 (3.9; 18.0–37.2)
NIP	44 (40.4)	84 (31.2)
MTX-treated patients	65 (59.6)	185 (68.8)
Rheumatoid arthritis, n (%)	44 (67.7)	122 (65.9)
Psoriatic arthritis, n (%)	10 (15.4)	30 (16.2)
other diagnosis, n (%)	11 (16.9)	33 (17.8)
MTX monotherapy, n (%)	15 (23.1)	43 (23.2)
MTX in combination therapy, n (%)	50 (76.9)	142 (76.8)
Before booster, n (%)	60 (55.0)	83 (30.9)
After first booster, n (%)	109 (100.0)	186 (69.1)
Vaccines used for basic immunization*
2x BNT162b2, n (%)	94 (86.2)	65 (78.3)
2x mRNA-1273, n (%)	8 (7.3)	11 (13.3)
2x AZD1222, n (%)	4 (3.7)	2 (2.4)
1x AZD1222 + 1x mRNA, n (%)	3 (2.8)	5 (6.0)
Vaccines used for booster**
BNT162b2, n (%)	95 (87.2)	160 (86.0)
mRNA-1273, n (%)	14 (12.8)	26 (14.0)

BMI, body mass index; MTX, methotrexate; NIP, non-immunosuppressed persons; *sample characteristics are given for samples taken before booster; **sample characteristics are given for samples taken after booster.

The majority of samples showed neutralization of SARS-CoV-2 in the pVNTs. Absence of neutralization of wild-type SARS-CoV-2 measured by pVNT was detected in 18 (6.7%) and absence of neutralization of Omicron BA.1 was detected in 56 samples (20.8%). Within samples acquired before booster vaccination, neutralization of Omicron was undetectable in 51.8% of the samples. In contrast, within samples acquired after booster, neutralization of Omicron was undetectable in only 7.0% (distributions of subpopulations at different time points in Fig. 1 ).

Fig. 1.

Wild-type (blue dots) and Omicron (green dots) pVNT results at different time points before and after booster vaccinations. Light blue and green dots indicate results of MTX patients, dark blue and green dots indicate results of NIP. The dotted line marks the lower limit of quantification (LLOQ) of the pVNT (ID50 of 10). ID50s below the LLOQ (ID50=10) were randomly assigned to a value between 4 and 7 to increase visibility of individual results in this graph. MTX, methotrexate; NIP, non-immunosuppressed persons; Om, Omicron; pVNT, Pseudovirus neutralization assay; Wt, wild-type; w, weeks. (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)

3.2. Comparison of ELISA and pVNT results

All wild-type and Omicron ELISAs (Wt-/Om-ELISAs) correlated with neutralization against wild-type and Omicron using pVNT (Wt-/Om-pVNT, Spearman's rank correlation, p<0.001 in all tested correlations, correlation coefficients in Fig. 2 ).

Fig. 2.

Scatter plot and ROC analysis of wild-type and Omicron antibody tests compared with wild-type and Omicron pVNT. A) Wt-ELISAs vs Wt-pVNT B) Wt-ELISAs vs Om-pVNT C) Om-ELISAs vs Om-pVNT. Dotted horizontal and vertical lines mark manufacturer's cut-offs for seropositivity. Om, Omicron BA.1; rS, Spearman's rank correlation coefficient (p<0.001 in all tested correlations); Wt, wild-type; white dots: measurement before first booster; dark dots: measurement after first booster. (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)

To examine test accordance of positive ELISA results with presence of neutralization in the pVNT, we first compared Wt-ELISAs with Wt-pVNT to generate a reference point (Fig. 2A, Table 3 A). All three Wt-ELISAs showed excellent test performance criteria with AUCs >0.98.

Table 3.

ELISA test performances with regard to pVNT results.

Test comparison	At manufacturer's cut-off			At max. Youden index			At >99% specificity
	Cut-off	Sensitivity [%]	Specificity [%]	Cut-off	Sensitivity [%]	Specificity [%]	Cut-off	Sensitivity [%]
A) Compared with Wt-pVNT
Wt-cPass	≥ 30%	90.8	100	> 30.4%	90.8	100	> 30.4%	90.8
Wt-QuantiVac	≥ 35.2 BAU/ml	98.8	72.2	> 109.8 BAU/ml	92.8	100	> 109.8 BAU/ml	92.8
Wt-SeraSpot	≥ 230 BAU/ml	90.4	100	> 180.5 BAU/ml	93.6	100	> 180.5 BAU/ml	93.6
B) Compared with Om-pVNT
Wt-cPass	≥ 30%	99.1	69.6	> 90.7%	82.2	98.2	> 98.5%	63.9
Wt-QuantiVac	≥ 35.2 BAU/ml	100	28.6	> 389.4 BAU/ml	90.6	85.7	> 3062 BAU/ml	29.6
Wt-SeraSpot	≥ 230 BAU/ml	95.8	58.9	> 281.0 BAU/ml	90.6	66.1	> 1278 BAU/ml	16.4
C) Compared with Om-pVNT
Om-cPass	≥ 30%	51.2	100	> 11.9%	65.3	100	> 11.9%	65.3
Om-QuantiVac	≥ 11.0 RU/ml	99.1	57.1	> 32.3 RU/ml	93.0	87.5	> 206.8 RU/ml	40.4
Om-SeraSpot	≥ 247 RU/ml	99.5	55.4	> 1139 RU/ml	93.0	87.5	> 4651 RU/ml	32.9

BAU, binding antibody units; Om, Omicron BA.1; pVNT, Pseudovirus neutralization assay; RU, relative units; Wt, wild-type.

Secondly, we compared Wt-ELISAs with Om-pVNTs (Fig. 2B, Table 3B), acknowledging the fact that only Wt-ELISAs were available when Omicron BA.1 became prevalent and still are in use. In this analysis, all Wt-ELISAs showed lower specificity and AUC values. Out of 56 negative Om-pVNT samples, 71.4% were false positive in at least one Wt-ELISA with values up to 98.5% neutralizing capacity (Wt-cPass) or 3059.2 BAU/ml (Wt-QuantiVac). Wt-QuantiVac showed the highest number of false positive results (Wt-QuantiVac n = 40, Wt-cPass n = 17, Wt-SeraSpot n = 23), while AUC was lowest for Wt-Seraspot (p<0.001 in DeLong test). Test performances increased using the theoretical optimal cut-offs at maximum Youden index. To achieve a specificity of >99% the cutoffs were increased which resulted in a greatly reduced sensitivity for all three test systems (Table 3B).

Thirdly, we compared Om-ELISAs with Om-pVNT results to examine how well the adapted ELISAs performed (Fig. 2C, Table 3C). In this analysis, both Omicron ELISAs by Euroimmun and Seramun showed decreased specificities (57.1% and 55.4% for Om-QuantiVac and Om-Seraspot). The Om-cPass test had no false positive results (specificity 100%) but showed a decreased sensitivity (51.2%) and the lowest overall AUC (p<0.001, Table 3C, Fig. 2C). Notably, specificity of QuantiVac increased by 28.5 percentage points at the manufacturer's cut-off when Omicron instead of wild-type antigens were used in the ELISA (Table 3B and 3C).

3.3. Characteristics of false positive ELISA results

Further evaluation of the false positive ELISA results in relation to Omicron pVNT results showed a considerable overlap between the Wt-ELISAs (out of 56 pVNT-negative samples, 16 were false positive in all three Wt-ELISAs, 8 in two out of three tests, and 16 only in Wt-QuantiVac). There was a similar overlap between the false positive Om-ELISAs (28 samples were false positive in either Om-QuantiVac or Om-SeraSpot, and 21 in both). A subsequent analysis of shared characteristics among false positive test results revealed no significant influence of MTX therapy versus no immunosuppression, but significantly more false positive results before than after booster vaccination (26.5% of all samples before vs 3.2% after booster in either the Omicron QuantiVac or SeraSpot tests, p<0.001 for both tests). However, booster vaccination showed no significant effect on the proportion of false positives when restricting the analysis to pVNT-negative samples (p = 0.358 for Om-QuantiVac and p = 0.754 for Om-SeraSpot), and specificities remained low after the first booster (Wt-QuantiVac 46.2%, Wt-SeraSpot 69.2%, Om-QuantiVac 69.2%, Om-SeraSpot 61.5%).

4. Discussion

To our knowledge, this comparative analysis is the first investigating commercially available Omicron adapted antibody test systems. We found a poorer performance of new Omicron antibody test systems compared to wild-type tests in detecting neutralizing antibodies against the corresponding SARS-CoV-2 variants. This is of clinical importance as various medical societies recommend the use of antibody tests for the verification of vaccination success in immunocompromised patients [11], [12], [13].

Generally, all antibody tests correlated well with pVNT results and the wild-type ELISAs showed good test performance criteria when compared with wild-type neutralization, which is in line with previous research [2,3]. As expected, specificity of the wild-type ELISAs was considerably reduced in comparison with Omicron neutralization in the pVNT at the manufacturer-defined cut-offs. The newly developed Omicron test by Euroimmun showed improved specificity for the detection of Omicron neutralization, but a relevant number of false positive results still occurred in both Omicron tests by Seramun and Euroimmun. Adjusting the cut-offs for positivity alleviated the problem, but only at the expense of reduced sensitivity. In case of a false positive result, humoral immunity against COVID-19 could be wrongly assumed and therefore potentially lead to the non-administration of crucial further booster vaccinations or pre- or post-exposure prophylaxis in vulnerable patients. In contrast, the Omicron cPass showed a specificity of 100%, but a very low sensitivity for the detection of Omicron neutralization in pVNT, which may cause unwarranted anxiety in patients or prompt unnecessary additional vaccinations.

False positive test results were often measured in the same samples across tests and occurred significantly more often before than after first booster vaccination when comparing to all other samples. This may be due to an expected reduction of negative Om-pVNT results after booster [19], since the association was no longer statistically significant when the analysis was restricted to pVNT-negative samples. Specificity of SeraSpot and QuantiVac tests remained low even after the third vaccination, albeit based on low case numbers. Overall, the risk to overestimate the antibody status might persist after the third vaccination, especially in more immunocompromised patients.

It is important to note that tests by Euroimmun and Seramun detect binding IgG antibodies of any functionality, while the pVNT and cPass detect antibodies of all isotypes, but only those with neutralizing capacity. The use of different antigen components with different tertiary structures of the binding antigens in the tests, as well as incomplete affinity maturation of antibodies, especially before booster, may also have contributed to binding without neutralization and thus to false positive ELISA results [19].

Due to the rapid emergence of new variants and immune escape mechanisms, SARS-CoV-2 poses major challenges for the development of new vaccines and test systems. There is little time for thorough validation before the use of new antibody tests in clinical routine. The presented data illustrates that a discrepancy between the antigens used in an antibody testing kit and the predominant virus variant in a population can yield inaccurate results. This may even become more problematic with the occurrence of even newer virus variants with new immune escape mechanisms. This must be kept in mind when interpreting antibody levels as measured by currently available test systems for clinical decisions. Our results call into question the practice of using commercially available ELISA test systems as a surrogate of neutralization.

This work depicts results of a large and diverse cohort, but the reported specificities are still based on relatively small numbers. Although the pVNT is considered the gold standard after the PRNT for the assessment of antibody response and despite of published correlations with better outcomes [1,4], it should be acknowledged that virus neutralization does not necessarily coincide with cellular immunity or translate to protection from severe COVID-19 in individual patients. Furthermore, the inclusion of more MTX-treated patients and unequal numbers of samples from individuals may have contributed to a bias in the results. However, MTX use did not significantly influence the proportion of false positive results. It is also important to note that Omicron strains other than BA.1 may have different resistance features so that our results cannot be generalized to all Omicron subvariants. However, other Omicron subvariants diverge even more in their antigens, which could result in even lower test specificity.

In summary, currently available antibody tests which were primarily developed for the detection of past infections may overestimate neutralization of Omicron variants at the manufacturer's cut-offs and are only of limited use for the evaluation of neutralization after vaccination. Therefore, the decision for additional vaccinations or passive immunization should primarily be based on a patient's overall risk for poor disease outcomes and the probability of a response to past vaccinations as well as time since last immunization, and not solely on antibody testing.

CRediT authorship contribution statement

All authors contributed to the acquisition, analysis or interpretation of data and critical revision of the manuscript for important intellectual content. RB and FNA had full access to all the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis and serve as the guarantors. RB and FNA were responsible for conceptualization. Sample collection was done by EH, AtH, ANAdS, LMF, JZ, FNA, PT-L and the COVIM study group. Experiments and data analysis was performed by EH, LMF, KG, LNM, LG, JK, RB and FNA. Data interpretation was done by all authors. Statistical analyses were done by EH, LMF, JK, RB and FNA. Writing of the manuscript, and creation of tables and figures was performed by LMF, EH, RB and FNA. All authors were involved in critical proof reading of the manuscript.

Ethics statement

The Berlin State Office for Health and Social Affairs (Turmstrasse 21, 10,559 Berlin, Germany) has ethically approved the VACCIMMUN study under file number 21/0098-IV E 13. The EICOV, COVIMMUNIZE and COVIM studies were approved by the local ethics committee of Charité Universitätsmedizin Berlin (COVIMMUNIZE EA4/244/20 and EICOV EA4/245/20), and the EC of the Federal State of Berlin as well as the Paul Ehrlich Institute (COVIM, EudraCT No. 2021–001,512–28). All experiments were performed in accordance with the ethical standards of the Declaration of Helsinki. All participants provided written informed consent.

Data availability statement

Data are available on reasonable request. All data relevant to the study are included in the article.

Declaration of Competing Interest

Euroimmun provided test kits for antibody testing free of charge as part of a research agreement. Otherwise, the authors declare that there are no conflicts of interest.