Consistency of the results of rapid serological tests for SARS-CoV-2 among healthcare workers in a large national hospital in Tokyo, Japan

Summary

We assessed the consistency of seropositive results of three rapid immunoassays (Kits A, B, and C) for severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) compared to highly accurate serological tests (Abbott and Roche) among healthcare workers in a hospital in Tokyo. The seroprevalence of SARS-CoV-2 immunoglobulin G was 0.41%, 2.36%, and 0.08% using Kits A, B, and C, respectively. Of the 51 samples that were seropositive on any rapid test, all were seronegative on both the Abbott and the Roche assays. Given that the seroprevalence of SARS-CoV-2 immunoglobulin G varied widely according to the choice of rapid test and the rapid test results were inconsistent with the results of highly accurate tests, the diagnostic accuracy of rapid serological tests for SARS-CoV-2 should be assessed before introducing these tests for point-of-care testing or surveillance.

Introduction

Accurate and rapid testing is required for the diagnosis of coronavirus disease (COVID-19) caused by severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2), which has spread rapidly worldwide since December 2019. There are two main categories of diagnostic tests for COVID-19: molecular tests that detect SARS-CoV-2 ribonucleic acid (RNA) and serological tests that detect anti-SARS-CoV-2 immunoglobulins. Reverse transcriptase-polymerase chain reaction (RT-PCR), a molecular test, is widely used as the reference standard for the diagnosis of SARS-CoV-2 infection, while serological tests have generated an interest as an alternative or complement to RT-PCR for diagnosing acute infection.

Several serological tests are available, including laboratory-based (e.g., enzyme-linked immunosorbent assay [ELISA] and chemiluminescent immunoassay [CLIA]) and rapid diagnostic tests (e.g., lateral flow immunoassay [LFIA]). In particular, LFIA-based tests are inexpensive, rapid, and easy to implement at point-of-care. This has stimulated the development and marketing of LFIA commercial kits (1). However, the pace of development has exceeded that of rigorous evaluation, and critical uncertainty about their accuracy remains (1). A systematic review and meta-analysis of the diagnostic accuracy of serological tests for SARS-CoV-2 found evidence that sensitivity and specificity of tests that use the LFIA method may be lower than those of tests that use ELISA or CLIA methods (2).

To date, few studies have examined the consistency of seropositive results from multiple rapid tests. Studies conducted in the United States (3), United Kingdom (4), and Denmark (5) have found that commercial LFIA kits showed varying levels of diagnostic accuracy for SARS-CoV-2. In Japan, however, no studies have assessed the level of agreement of multiple rapid serological tests for detecting SARS-CoV-2 infection. Several studies have found a seroprevalence ranging from 0.03-0.88% on laboratory-based tests and 0.2- 8.53% on rapid tests in the general Japanese population, outpatients, and healthcare workers (HCWs) (6-9). The seroprevalence estimate in different studies has varied widely and appears to be higher with rapid tests than with laboratory-based tests. Differences in the diagnostic accuracy between tests and the repeatability within tests, makes it difficult to compare seroprevalence estimates across studies.

In the present study, we investigated whether the results of three types of rapid LFIA were consistent with those of two types of highly accurate laboratory-based tests for estimating the seroprevalence of SARS-CoV-2 infection among healthcare workers in a large hospital in Tokyo, Japan.

Materials and Methods

Study design and participants

This cross-sectional study was conducted in July 2020 among workers at the National Center for Global Health and Medicine (NCGM), a leading institute working on combating COVID-19 in Japan. The survey targeted mainly those engaged in COVID-19-related work or who worked in a department with a high risk of exposure to SARS-CoV-2 infection. The seroprevalence of SARS-CoV-2 measured using the two laboratory-based tests among the study participants was reported elsewhere (10).

The study protocol was approved by the Ethics Committee of the National Center for Global Health and Medicine, Japan. Written informed consent was obtained from all participants.

Serological tests

We used three LFIA rapid tests from different manufacturers (Kits A, B, and C), performed according to each manufacturer's instructions, to determine whether samples were anti-SARS-CoV-2 immunoglobulin M (IgM) and/or immunoglobulin G (IgG) positive. Serum separated from a blood sample of a brachial vein was used for the SARS-CoV-2 antibody tests with the Abbott and Roche tests, and Kits A and B, while a finger-prick whole blood sample was used for tests with Kit C. Kits A and B were conducted by medical laboratory technicians. Samples that were positive were retested, and the result of the repeat test was adopted. Kit C was performed by trained staff, and used a finger-prick blood sample, so each sample was tested only once. The result was checked by at least two trained staff. Kit A had a reported sensitivity of 87.9% and 97.2%, and specificity of 100% and 100% for measuring IgM and IgG, respectively. Kits B and C had a reported sensitivity of 87.3% and 90.4%, respectively, and 100% specificity. Kits B and C did not differentiate between IgM or IgG.

The Abbott test was run on the Abbott Architect analyzer, using the SARS-CoV-2 IgG assay, which is based on the chemiluminescent microparticle immunoassay method for the qualitative detection of IgG in human serum or plasma against the SARS-CoV-2 nucleoprotein. A value of 1.40 or higher was considered positive. This assay was reported to have 99.9% specificity and 100% sensitivity for detecting the IgG antibody 17 days after symptoms onset (11).

The Roche test was run on the Roche cobas^® e602 analyzer, using the Elecsys^® Anti-SARS-CoV-2, based on the electrochemiluminescence immunoassay for the in vitro qualitative detection of total antibodies (including IgG) to the SARS-CoV-2 nucleoprotein. A value of 1.00 or higher was considered positive. This assay was reported to have 99.8% specificity and 100% sensitivity 14 days after symptoms onset.

We performed post hoc testing of samples with positive results on the Abbott or Roche tests, Kit B, two or more kits, and samples that were close to the index threshold on the Abbott or Roche tests using EUROIMMUN anti-SARS-CoV-2 IgG ELISA. The EUROIMMUN test methods are described in the Supplementary Materials (https://www.globalhealthmedicine.com/site/supplementaldata.html?ID=18).

Statistical analysis

We calculated the seroprevalence of SARS-CoV-2 antibody and its 95% confidence interval for each test. We drew scatterplots to display the index values of the Abbott and Roche tests of the samples that were positive on any of the tests. The data were analyzed using Stata version 16 (Stata Corp., College Station, TX, USA).

Results

Participant characteristics

Of 1,579 workers recruited, 1,228 agreed to participate and completed the questionnaire, and at least one serological test was conducted. The mean (± standard deviation) age of participants was 36.1 ± 11.0 years, and 29% were men. The primary job categories represented were nurses (49%), physicians (19%), and allied healthcare professionals (14%). Only one of the 91 participants who self-reported that they had previously received a PCR test for SARS-CoV-2 (timing unknown) tested positive on the PCR test.

Seroprevalence according to each rapid test

Table 1 shows the seroprevalence of SARS-CoV-2 antibodies using each serological test. All participants received tests using Kits A and B (n = 1,228), while 1,197 had test results available for the Kit C test because ten refused to provide a finger-prick blood sample for testing, and 21 had indeterminate test results. The seroprevalence of IgG on Kits A, B, and C was 2.36% (95% confidence interval [CI]: 2.59-3.37%), 0.41% (95% CI: 0.13-0.95), and 0.08% (95% CI: 0.00-0.46%), respectively, and the seroprevalence of IgM was 2.04% (1.32-3.00%), 0.65% (0.28-1.28%), and 0% (0.00- 0.00%), respectively.

Table 1. Results of multiple serological tests for SARS-CoV-2 among workers in a large national hospital in Tokyo, Japan.

	Abbott	Roche	Kit A		Kit B		Kit C
			IgG	IgM	IgG	IgM	IgG	IgM
Subjects	n = 1,228	n = 1,228	n = 1,228	n = 1,228	n = 1,228	n = 1,228	n = 1,197	n = 1,197
Seropositive, n	1	1	29	25	5	8	1	0
Seroprevalence, %	0.08	0.08	2.36	2.04	0.41	0.65	0.08	0
(95% CI)	(0.02-0.45)	(0.02-0.45)	(2.59-3.37)	(1.32-3.00)	(0.13-0.95)	(0.28-1.28)	(0.00-0.46)	(0.00-0.00)
Pattern A (n = 1)	(+)	(-)	(-)	(-)	(-)	(-)	(-)	(-)
Pattern B (n = 1)	(-)	(+)	(-)	(-)	(-)	(-)	(-)	(-)
Pattern C (n = 2)	(-)	(-)	(+)	(+)	(-)	(+)	(-)	(-)
Pattern D (n = 12)	(-)	(-)	(+)	(+)	(-)	(-)	(-)	(-)
Pattern E (n = 1)	(-)	(-)	(+)	(-)	(-)	(-)	(+)	(-)
Pattern F (n = 14)	(-)	(-)	(+)	(-)	(-)	(-)	(-)	(-)
Pattern G (n = 11)	(-)	(-)	(-)	(+)	(-)	(-)	(-)	(-)
Pattern H (n = 5)	(-)	(-)	(-)	(-)	(+)	(-)	(-)	(-)
Pattern I (n = 6)	(-)	(-)	(-)	(-)	(-)	(+)	(-)	(-)

CI, confidence interval. (+): positive; (-): negative. Abbott: chemiluminescent microparticle immunoassay (CMIA). Roche: electrochemiluminescence immunoassay (ECLIA). Kit A, B, and C: lateral flow immunoassay (LFIA).

Consistency of serological test results

Table 1 shows the consistency of results across the serological tests for participants having at least one positive result. Few participants showed an agreement of seropositive results across the rapid tests: two were IgM seropositive on Kits A and B, and one was IgG seropositive on Kits A and C. One participant who self-reported having tested positive on the PCR test for SARS-CoV-2 was negative on all serological tests.

The two participants who were positive on either the Abbott or the Roche test were negative on all three rapid tests. Figure 1 shows the Abbott and Roche index for each positive sample. None of the samples with positive results on any of the rapid tests had an Abbott index close to the positive threshold (1.40). One individual who was positive on Kit B (IgG) had a Roche index of 0.879, which is close to the positive threshold (1.00).

Figure 1.

Individual index values of the Abbott (left panel) or Roche (right panel) tests in samples that were tested seropositive on any serological test. The Y-axis shows the Abbott or Roche index values. The X-axis shows the results of 53 individuals who were positive on any of the serological tests. (+): positive; (-): negative. The positive threshold value of the Abbott test is 1.40. The positive threshold value of the Roche test is 1.00.

Post hoc testing using the EUROIMMUN test

Samples that were seropositive on the Abbott test, Roche test, Kit B (IgG), or two kits, and those who were close to the threshold of the index of the Abbott or the Roche test were all negative on post hoc testing using the EUROIMMUN test (Supplementary Figure S1, https://www.globalhealthmedicine.com/site/supplementaldata.html?ID=18).

Discussion

In the present study, we assessed the consistency of the seropositivity of three types of rapid tests for SARS-CoV-2 antibodies compared to the results of the Abbott and Roche tests among HCWs in a large hospital designated for the care of COVID-19 patients in Tokyo, Japan. None of the 51 samples that were seropositive on any of the rapid tests were positive on either the Abbott or the Roche tests, and only three cases had consistently seropositive results using two different rapid tests.

In Japan, studies using a rapid SARS-CoV-2 antibody test have found varying seroprevalence (1.79-9.1%) among HCWs (8,9,12). SoftBank Corp conducted a survey using Kit B among 5,850 HCWs at multiple medical institutions across Japan in May and June 2020 and found a seroprevalence of 1.79% (9). Another study measured the antibody among 55 HCWs of two clinics in Tokyo using a different type of kit in April 2020 and found a seroprevalence of 9.1% (8). The difference in seroprevalence found in previous studies could reflect the background of each HCW and timing of measurement (13). Nevertheless, given the marked difference in seropositivity according to each of the three rapid tests in the present population, the use of different rapid test kits may be one of the sources of variation.

The growing body of literature on the accuracy of antibody tests for SARS-CoV-2 shows that the sensitivity and specificity vary widely between rapid tests (3-5). A study that compared the accuracy of ten different rapid tests (IgG) found sensitivities and specificities ranging from 66.7-90.9% and 91.6-100%, respectively (3). In the present study, all samples that were seropositive on any rapid test were negative on the highly specific Abbott and Roche tests, suggesting that all the positive results of the rapid tests were false-positive results. If the prevalence of the outcome is low, a test with a low specificity will produce many false positives and overestimate the positive rate. For example, if 1% of the population is infected with a virus, a test with sensitivity and specificity of 100% and 92% would lead to a prevalence of 9% (89% of all test positives are false positives). The considerably higher seroprevalence (IgG) obtained by using Kit A (2.36%) than those obtained by using Kits B: (0.41%) and C (0.08%), suggests that Kit A has a relatively low specificity. In populations with a very low prevalence of SARS-CoV-2 infection, such as current Japan, it is crucial to adopt a test with a high specificity to estimate seroprevalence accurately.

Rapid test kits may tend to misidentify other viral antibodies as SARS-CoV-2 positive (14). For example, of seven stored serum samples of patients with human common cold coronavirus pneumonia admitted up until January 2019, four were identified to be SARS-CoV-2 positive using a rapid test (15). In a systematic review and meta-analysis on the diagnostic accuracy of SARS-CoV-2 serological tests (2), rapid tests tended to show a higher frequency of false-positive results (i.e., lower specificity) than laboratory tests when blood samples of patients without a history of COVID-19 but with a common cold or another viral infection were used as negative controls. Given these data, it is necessary to evaluate the specificity using blood samples from patients infected with a virus similar to SARS-CoV-2.

A key limitation of the present study is the lack of a "gold standard" for SARS-CoV-2 antibody testing. We confirmed the presence of the antibodies using reliable laboratory-based tests (Abbott and Roche), but these are not perfect measures; for example, Perkmann et al. (16) reported that the sensitivity and specificity were 84.6% and 99.2% for the Abbott test, and 89.2% and 99.7% for the Roche test. In fact, our post hoc analysis revealed that the two participants with seropositive Abbott or Roche tests were negative using the EUROIMMUN quantitative antibody test, suggesting that both results were false-positive results. This result is as expected in a setting of very low seroprevalence, even using highly specific tests, such as the Abbott and Roche tests.

In conclusion, the estimated seroprevalence of SARS-CoV-2 infection varied widely across the three rapid tests, and samples that were seropositive on any rapid test were negative on the highly accurate Abbott and Roche tests. The accuracy of rapid tests should be carefully evaluated before introducing these assays as point-of-care tests or for surveillance.