Evaluation of the performance of SARS‐­CoV­‐2 antibody assays for a longitudinal population­based study of COVID‐­19 spread in St. Petersburg, Russia

Abstract

Geographical variation in severe acute respiratory syndrome coronavirus 2 (SARS­‐CoV­‐2) spread requires seroprevalence studies based on local tests, but robust validation is needed. We summarize an evaluation of antibody tests used in a serological study of SARS‐­CoV­‐2 in Saint Petersburg, Russia. We validated three different antibody assays: chemiluminescent microparticle immunoassay (CMIA) Abbott Architect SARS‐­CoV­‐2 immunoglobulin G (IgG), enzyme­ linked immunosorbent assay (ELISA) CoronaPass total antibodies test, and ELISA SARS­‐CoV‐­2‐­IgG‐­EIA‐­BEST. Clinical sensitivity was estimated with the SARS­‐CoV­‐2 polymerase chain reaction (PCR) test as the gold standard using manufacturer recommended cutoff. Specificity was estimated using pre­pandemic sera samples. The median time between positive PCR test results and antibody tests was 21 weeks. Measures of concordance were calculated against the microneutralization test (MNA).Sensitivity was equal to 91.1% (95% confidence intervbal [CI]: 78.8–97.5), 90% (95% CI: 76.4–96.4), and 63.1% (95% CI [50.2–74.7]) for ELISA Coronapass, ELISA Vector­Best, and CMIA Abbott, respectively. Specificity was equal to 100% for all the tests. Comparison of receiver operating characteristics has shown lower AUC for CMIA Abbott. The cut­off SC/O ratio of 0.28 for CMIA Abbott resulted in a sensitivity of 80% at the same level of specificity. Less than 33% of the participants with positive antibody test results had neutralizing antibodies in titers 1:80 and above. Antibody assays results and MNA correlated moderately. This study encourages the use of local antibody tests and sets the reference for seroprevalence correction. Available tests' sensitivity allows detecting antibodies within the majority of PCR­ positive individuals. The Abbott assay sensitivity can be improved by incorporating a new cut­off. Manufacturers' test characteristics may introduce bias into the study results.

Highlights

• 1)This report is the first diagnostic performance study of antibody assays used in the representative population‐based serological study of SARS‐CoV‐2 in St. Petersburg, Russia.
• 2)The sensitivity for two local assays was equal to 91.1% (95%CI: 78.8‐97.5) and 89.1\% (95%CI: 76.4‐96.4), CMIA Abbott's sensitivity was equal to 63.1\% (95%CI 50.2‐74.7)), with 100% specificity for all the tests.
• 3)Moving the S/CO ratio from Abbott assays from manufacturers recommended 1.4 to 0.28 improved sensitivity from 63% to 80%, without loss in specificity.
• 4)Less than a third of samples positive for binding antibodies were also positive in the virus neutralization test (with 1:80 titer as a threshold) in the population‐based sample.

1. INTRODUCTION

Severe acute respiratory syndrome coronavirus 2 (SARS­‐CoV‐­2) seroprevalence studies have proved to be a valuable tool in the assessment of the COVID‐­19 pandemic dynamics. ¹ , ² , ³ However, researchers have raised several concerns regarding the non­response bias, the non­representative sampling, and the use of non­validated tests for SARS‐­CoV­‐2 antibodies detection. ⁴ The first two issues can be addressed by appropriate study design and population sampling strategies  ⁵ , ⁶ The latter problem requires the assessment of sensitivity and specificity of antibody assays to guide the correction of serological study results. ⁷ , ⁸ , ⁹ , ¹⁰ , ¹¹ Additional requirements for antibody tests are being lodged for longitudinal serologic studies, but manufacturers rarely report the evaluation of the long­term test performance, for example, decreased long­term sensitivity is an issue that could bias the evaluation of immune response durability. ¹² , ¹³

An evident geographical variation in the spread of SARS‐­CoV­‐2 requires local seroprevalence studies. ¹⁴ The use of locally available tests from national manufacturers can be convenient but often comes at the expense of test performance. ³ Local tests require robust validation against the benchmark of already available antibody tests and microneutralization assays using sera samples from cases confirmed by polymerase chain reaction (PCR) and from healthy donors.

The long­term performance of antibody tests is key to understanding the antibody kinetics. Previous validation studies have shown significant changes in the sensitivity of certain antibody tests in the longer follow­up. ¹⁵ This report summarizes the validation of antibody tests used in a representative population‐based serological study of SARS‐­CoV­‐2 in St. Petersburg, Russia, a densely populated city with more than 5.3 million inhabitants making it the fourth largest city in Europe. ⁶ We aimed at establishing test specificity (Sp) and sensitivity (Se) to correct estimates obtained through population­based seroprevalence study.

2. MATERIALS AND METHODS

Information about the corresponding seroprevalence study in St. Petersburg is available in the detailed report. ⁶ In brief, we obtained a representative population sample through phone by using random­digit dialing. The study started in May 2020. The first report was publicly available in August 2020. This study is underway and includes three rounds of blood sample collection with more than 2500 participants. This manuscript represents the evaluation of the performance of SARS­‐CoV‐­2 antibody assays using independent samples of sera taken from PCR positive cases and pre­pandemic sera that are not related to the population­based serological study. We also assess the degree of agreement between the assays and neutralization tests using samples acquired through population­based study.

2.1. Antibody tests

We used three different antibody tests throughout the study: (1) chemiluminescent microparticle immunoassay Abbott Architect SARS­‐CoV‐­2 im­munoglobulin class G (IgG) on the Abbott ARCHITECT i2000sr platform (Abbott Laboratories), detecting IgG antibodies to the nucleocapsid protein of SARS­‐CoV‐­2 with the signal/cut­off (S/CO) ratio of 1.4 for positivity (chemiluminescent microparticle immunoassay [CMIA] Abbot; www.fda.gov/media/137383/download); (2) enzyme­linked immunosorbent assay CoronaPass total antibodies test (Genetico) based on the recombinant receptor binding domain of the spike protein of SARS­‐CoV­‐2 (Department of Microbiology, Icahn School of Medicine), detecting total antibodies with the S/CO ratio of 1.0 for positivity (ELISA Coronapass, pass. genetico. ru); (3) enzyme­linked immunosorbent assay SARS­‐CoV­‐2‐­IgG­‐EIA­‐BEST by Vector­Best, Novosibirsk, Russia also detecting IgG antibodies to the spike protein of SARS­‐CoV‐­2 with the S/CO ratio of 1.1 for positivity. (ELISA Vector­Best, vector‐best. ru/en/prod/index. php?SECTION_ID = 2724).

Test performance for CMIA Abbott is available from the manufacturer materials (Se = 100%, Sp = 99.6%). It was also evaluated in numerous independent studies. ⁹ , ¹⁶ , ¹⁷ , ¹⁸ ELISA Coronapass test manufacturer provides information on its official website (Se = 98.7%, Sp = 100%). ELISA Vector­Best sensitivity and specificity is reported in one study published in Russian (Se = 100%, Sp = 99.8%). ¹⁹

We used CMIA Abbott and ELISA Coronapass for the first publication of a population­based seroprevalence study in May­ June 2020. ⁶ The first round of the study analysis showed that the use of ELISA Coronapass results in a slightly higher seroprevalence estimation. This was the initial reason for the independent validation of the test performance.

For sensitivity assessment, we also used nucleocapsid protein­based CMIA Elecsys Anti­‐SARS­‐CoV­‐2, manufactured by Roche Diagnostics GmbH, Mannheim, Germany (CMIA Roche, S/CO ratio of 1.0 for positivity, Se = 99.5%, Sp = 99.8%; diagnostics. roche. com/ru/ru/products/params/elecsys‐anti‐sars‐cov‐2. html).

2.2. Samples for the sensitivity and specificity assessment

For the sensitivity assessment, we obtained 92 serum samples collected from the personnel of the “Scandinavia” clinic as part of routine antibody monitoring carried out in the clinic. Serum samples were obtained for individuals who reported positive PCR tests for SARS­‐CoV‐­2 RNA. The median time between a positive PCR test result and blood draw for antibody test was equal to 21 weeks (interquartile range: 19.9­–22.5).

For the specificity assessment, pre­pandemic human sera samples were obtained from healthy blood donors collected by the Smorodintsev Research Institute of Influenza through routine influenza surveillance in different cities across Russia. Sera samples are collected twice a year to assess herd immunity against influenza A and B viruses. Written informed consent was obtained in all cases of blood donation and further sample preparation. We used 93 remnant sera samples from herd immunity studies that were kept frozen at ­80°C.

2.3. Neutralization test

A neutralization test was performed for 365 positive samples for binding antibody samples (at least on one of three assays) and for 74 negative samples. TCID₅₀ based microneutralization test (MNA) was used to detect and titrate neutralizing antibodies. ²⁰ First, serum samples were heated for 30 min at 56°C to avoid complement­linked reduction of the viral activity. This was followed by the preparation of serial two­fold dilutions starting from 1:10 in 60 uL volume (tested in triplicate) of each serum specimen in culture medium (Alpha MEM containing antibiotics and 2% heat­inactivated fetal bovine serum). Each dilution was mixed at equal volume with the live SARS­CoV­2 virus (60 µl, containing 25 TCID₅₀/50 µl) and incubated for 60 min at 37°C in plastic microplates. Then 100 μl of the mix was transferred into 96­‐well microplates with monolayer Vero cells. The plates were incubated at 37°C in a 5% CO₂ atmosphere. Readings were evaluated 5–6 days later and neutralization was recorded if 100% of the cells in the well were preserved (no visible plaques or cytopathic effect). Serum neutralizing titer was expressed as the inverse of the higher serum dilution that exhibited neutralizing activity. All experiments were performed in a BSL3 laboratory.

2.4. Statistical analyses

Clinical sensitivity was estimated with the SARS­‐CoV­‐2 PCR test as the gold standard for specificity in pre­pandemic sera samples. We used 46 PCR positive samples and 41 pre­pandemic serum samples to cross­validate all three tests (cross­validation sample). Paired samples were collected from the same individuals tested on all the assays and compared using statistical tests fit

for paired comparison. In the independent test validation (full­validation sample), different sets of non­paired samples (60 PCR positive samples and 65 pre­pandemic samples for CMIA Abbott, 48 and 93 for ELISA Vector­Best, 60 and 92 for ELISA­ Coronapass) were used. Non­paired samples do not necessarily come from the same individuals and are not matched. The difference in the number of paired and non­paired samples is due to technical difficulties, including lack of material, logistics, and labeling issues. Unfortunately, due to the corresponding complexities, it was impossible to use all three tests for processing all available samples. We report the validation study results based on the manufacturers' recommended S/CO ratios, except for CMIA­Abbott, where we use both the S/CO ratio of 1.4 and a more sensitive S/CO ratio of 1.0. Compared with the neutralization test results, the percent agreement (positive, negative and overall), the concordance (Cohen's Kappa coefficient along with the prevalence index (P­index), bias index (B­index), and prevalence and bias­adjusted kappa—PABAK), and Spearman correlation coefficients were calculated. ²¹

The estimates are given with the 95% confidence intervals (CI) where appropriate. R statistical software (v.4.0.2; R Founda­ tion for Statistical Computing) was used for data analysis and presentation. Exact binomial confidence limits were calculated for test sensitivity, specificity, and percent agreement (negative—NPA, positive—PPA, and overall—OPA) applying the epi. tests function from the epiR package. Receiver operating characteristic (ROC) curves were constructed, and the corresponding area under the curve (AUC) was calculated for three tests using the pROC package. CIs for AUCs were calculated with 2000 bootstrap repli­ cates. The bootstrap test was used to compare ROC curves: a test for paired curves was used in the cross­validation sample and non­paired in the full­validation sample. Confidence intervals were calculated for Spearman's correlation with the spearmanCI package.

To report CIs for concordance measures—bias, prevalence, and kappa—we used the bootstrap percentile method (the epi. kappa function from the epiR package). Formal criteria (eg., Landis and Koch) for Cohen's Kappa interpretation were not used.

2.5. Ethical considerations and study registration

The study was approved by the Research Planning Board of the European University at St. Petersburg (on May 20, 2020) and the Ethics Committee of the Clinic “Scandinavia” (on May 26, 2020). All research was performed in accordance with the relevant guidelines and regulations. Informed consent was obtained from all participants of the study. The study was registered with the following identifiers: Clinicaltrials. gov (NCT04406038, submitted on May 26, 2020, date of registration—May 28,2020) and ISRCTN registry (ISRCTN11060415, submitted on May 26, 2020, date of registration—May 28,2020). The amendment, which included a test validation study, was approved by the Ethics Committee of the Clinic “Scandinavia” on October 2, 2020. We used blood samples collected from the clinic “Scandinavia” staff as a part of the routine antibody monitoring for which the hospital obtained a separate informed consent. Additional verbal consent was obtained to use collected samples for the validation study. Personal information was not linked to serum samples in the validation study except for the date of positive PCR for SARS­‐CoV‐­2.

3. RESULTS

Based on cross­validation results, the sensitivity was equal to 91.1% (95% CI: 78.8–97.5) for ELISA Coronapass and 89.1% (95% CI: 76.4–96.4) for ELISA Vector­Best. It was not significantly different from the CMIA Roche—89.1% (95% CI: 76.4– 96.4). However, the sensitivity of CMIA Abbott was equal to 63.1% (95% CI: 50.2–74.7). It was slightly higher with a cutoff SC/O ratio of 1.0—70.7% (95% CI: 57.3–81.9). Specificity was equal to 100% for all the tests (see Table 1).

Table 1.

Sensitivity and specificity of tests based on cross­validatation (paired serum samples)

Test	N	Sensitivity, % (95% CI)	N	Specificity, % (95% CI)
CMIA Abbott 1.0	46	70.7 (57.3–­81.9)	41	100 (88.1­–100)
CMIA Abbott 1.4	46	63.1 (50.2­–74.7)	41	100 (84.6­–100)
CMIA Roche	46	89.1 (76.4­–96.4)	—	—
ELISA Coronapass	46	91.1 (78.8­–97.5)	41	100 (91.6­–100)
ELISA Vector­Best	46	89.1 (76.4­–96.4)	41	100 (91.4­–100)

Abbreviation: CI, confidence interval.

Using non­paired samples in the validation of the tests did not dramatically change the sensitivity and point estimates but narrowed confidence intervals (see Table S1).

Although the comparison of ROCs for three tests has shown that AUC is lower for CMIA­Abbott, the difference was not dramatic—0.96 (95% CI: 0.90–1.00) for ELISA Coronapass and 0.90 (95% CI: 0.82–0.97) for CMIA Abbott, the difference was only significant when paired samples were compared (see Table 2 and Table S2).

Table 2.

Area under the ROC (AUC) of tests based on cross­validation (paired serum samples)

Test	AUC (95% CI)	p ­value
ELISA Coronapass	0.96 (0.90–­1.00)	—
CMIA Abbott	0.90 (0.82­–0.97)	0.041
ELISA Vector­Best	0.96 (0.89­–1.00)	0.476

Abbreviations: CI, confidence interval; ROC, receiver operating characteristic.

The cutoff SC/O ratio of 0.28 for CMIA­Abbott resulted in a sensitivity of 80% at the same full level of specificity. The other test thresholds were optimal based on the ROC analysis of cross­validation and non­paired samples (see Figure 1).

Figure 1.

ROCs for three tests against PCR test results Inverse neutralising MNA titer. ELISA, enzyme‐linked immunosorbent assay; MNA, microneutralization test; PCR, polymerase chain reaction; ROC, receiver operating characteristic

In less than one‐­third of the population­based study participants with positive antibody total IgG test results, we detected neutralizing antibodies in titers 1:80 and above. NPA was between 28.0% (95% CI: 22.0–34.5) for CMIA Abbott (cut­off 1.4) and 20.8% (95% CI: 16.6–25.5) for ELISA Vector­best. NPA between MNA with a cut­off at titers 1:20 was between 74.4% (95% CI: 68.0–80.2) for CMIA Abbott (cut­off 1.4) and 64.2% (95% CI: 58.9–69.3) for ELISA Vector­Best (see Table 3).

Table 3.

Percent agreement between antibody tests and neutralizing antibody tests using two cut­offs (titers 1:20 and 1:80)

Test	MNA100 cut­off	PPA (95% CI)	NPA (95% CI)	OPA (95% CI)
CMIA Abbott 1.0	20	76.5 (70.0­–82.2)	72.0 (65.8­–77.6)	74.0 (69.7­–78.1)
CMIA Abbott 1.4	20	72.8 (66.5­–78.5)	74.4 (68.0­–80.2)	73.6 (69.2­–77.6)
ELISA Coronapass	20	91.1 (84.6­–95.5)	65.8 (60.3­–71.0)	72.9 (68.5­–77.0)
ELISA Vector­Best	20	100 (96.3­–100)	64.2 (58.9­–69.3)	72.2 (67.8­–76.4)
CMIA Abbott 1.0	80	95.5 (91.6­–97.9)	25.9 (20.5­–32.0)	57.6 (52.9­–62.3)
CMIA Abbott 1.4	80	94.7 (91.0­–97.3)	28.0 (22.0­–34.5)	62.6 (57.9­–67.2)
ELISA Coronapass	80	96.7 (91.9­–99.1)	21.2 (16.8­–26.1)	42.4 (37.7­–47.1)
ELISA Vector­Best	80	100 (96.3­–100)	20.8 (16.6­–25.5)	38.5 (33.9­–43.2)

Abbreviation: CI, confidence interval.

The measures of concordance are presented in Table S3. There was a moderate correlation between antibody assays results and MNA: 0.65 (95% CI: 0.59–0.71) for CMIA Abbott, 0.60 (95% CI: 0.54–0.67) for ELISA Coronapass, and 0.76 (95% CI: 0.72–0.81) for ELISA Vector­Best (see Figure 2).

Figure 2.

Correlation between antibody tests results and neutralization test. ELISA, enzyme‐linked immunosorbent assay

4. DISCUSSION

Our validation study encourages the use of local antibody tests for population­based SARS‐­CoV­‐2 surveillance and sets the ref­erence for the seroprevalence correction. However, it discourages the use of Abbott Architect SARS­‐CoV‐­2 IgG for population­ based seroprevalence SARS­CoV­2 surveillance. The decrease in test sensitivity (63% with a cut­off of 1.4) is likely to be related to the long period between infection and blood sampling compared to the manufacturer's validation studies (it was on average 21 weeks after the reported positive PCR test). Previous studies also found similar results for samples taken at a longer follow­ up. ¹³ , ¹⁵  These results may also explain the decline in antibody presence in longitudinal studies that may be incorrectly interpreted by the short­term immune response to SARS‐­CoV­‐2. ²²

Based on our validation study, the Abbott Architect SARS­CoV­2 IgG's sensitivity can be significantly improved by incor­porating a new cut­off. Moving the S/CO ratio from 1.4 or 1.0 for positivity to 0.28 improved sensitivity from 63% or 71% to 80%, respectively, without loss in specificity. In another study ¹¹ the optimized cut­off for Abbott CMIA was 0.91, but the mean cut­off in specificity cohorts ranged between 0.06 and 0.11 (below 0.28). The longer period between infection and blood draw in our study may explain the difference in the optimized cut­off.

Locally available ELISA tests and Roche assays are sensitive enough to detect antibodies in most individuals with previous positive PCR tests. Tests were negative for approximately 10% of participants with positive PCR, with any antibody tests. From the available data, it is impossible to conclude whether test results are false­negative antibody, false­positive PCR, or that infection did not result in the antibody response. Further studies are also needed to look into other mechanisms of the immune response to SARS‐­CoV­‐2. ²³

The results for the assessment of concordance between binding and neutralizing antibodies tests should be interpreted with caution. Less than a third of samples positive for binding antibodies were also positive in the virus neutralization test (with the 1:80 titer as a threshold). However, this does not mean that they are susceptible to SARS‐­CoV‐­2 virus re­infections. Case reports about the second episode of COVID‐­19 infection in the presence of neutralizing antibodies were also published. ²⁴ The follow­up time plays an essential role in neutralization test results, but negative tests do not rule out other protection mechanisms. Recent studies showed that a positive antibody test is a powerful predictor of lowered risk of SARS‐­CoV­‐2 reinfection measures as a positive PCR test. ²⁵ , ²⁶

This finding has an important practical implication. Relying on test characteristics provided by manufacturers for correction of reported prevalence estimates introduces additional bias to the study. If the test reported prevalence is 5%, then with the test sensitivity of 65% and specificity of 100%, the true prevalence would be equal to approximately 8%. ²⁷ If the crude prevalence based on the same test is 40%, then the true prevalence would be around 62%. The magnitude of bias in the population with a significant proportion of individuals with antibodies would limit the ability to detect the herd immunity threshold.

This study has several limitations. PCR test results that were chosen as a golden standard, were carried out in different laboratories. Although official test certificates that are registered in the national database were provided, false­ positive results cannot be ruled out. ²⁸ The sample size for specificity assessment was relatively small, yet the corresponding specificity assess­ments are in line with the other studies that are based on the larger samples. But as mentioned above, PCR false­positivity is likely to underestimate antibody test sensitivity in this study. Neutralization test results should be considered with caution as well. ²⁹ Such tests can be considered as a surrogate marker of protection from reinfection, but this association needs to be explored in population­based epidemiological studies. ¹⁵

In conclusion, this validation study provides a reference that can be used in further seroprevalence reports to correct the results based on test sensitivity and specificity. The Choice of the test for longitudinal surveillance is critical for making conclusions about the spread of SARS­‐CoV­‐2 and the durability of the immune response. Local tests should be rigorously evaluated for seroprevalence studies because the benefits of using appropriately validated tests are not only financial or related to matters of convenience. They may provide more accurate and unbiased assessments for the course of the pandemic.

CONFLICT OF INTERESTS

AB reports personal fees from MSD, Biocad, and AstraZeneca outside the submitted work.

AUTHOR CONTRIBUTIONS

Anton Barchuk, Daniil Shirokov, Lubov Barabanova, Daria Danilenko, and Dmitriy Skougarevskiy conceived the study. Anton Barchuk drafted the first version of the manuscript. Daria Danilenko and Varvara Tychkova performed Vector­ Best ELISA assays, Mariia Sergeeva and Lubov Barabanova performed the neutralization assays. Anton Barchuk, Dmitriy Skougarevskiy, Rustam Tursun­zade, and Olga Dudkina did the analysis. All authors participated in the study design, contributed to the interpretation of data and to drafting sections of the manuscript. All authors read and approved the final manuscript.

Supporting information

Supporting information.

Barchuk A, Shirokov D, Sergeeva M, et al. Evaluation of the performance of SARS‐­CoV­‐2 antibody assays for a longitudinal population­based study of COVID‐­19 spread in St. Petersburg, Russia. J Med Virol. 2021;93:5846‐5852. 10.1002/jmv.27126

DATA AVAILABILITY STATEMENT

The data that support the findings of this study is openly available online at https://github.com/eusporg/spb_covid_ study20/tree/master/validation_of_covid_tests.