Evaluation of 32 rapid tests for detection of antibodies against SARS-CoV-2

Abstract

Aims

To evaluate the analytical performance of 32 rapid tests for detection of antibodies against coronavirus SARS-CoV-2.

Materials and methods

We used at total of 262 serum samples (197 pre-pandemic and 65 convalescent COVID-19), and three criteria to evaluate the rapid tests under standardized and optimal conditions: (i) Immunoglobulin G (IgG) specificity “good” if lower limit of the 95% confidence interval was ≥ 97.0%, “acceptable” if point estimate was ≥ 97.0%, otherwise “not acceptable”. (ii) IgG sensitivity “good” if point estimate was ≥ 90.0%, “acceptable” if ≥ 85.0%, otherwise “not acceptable”. (iii) User-friendliness “not acceptable” if complicated to perform or difficult to read result, otherwise “good”. We also included partial evaluations of three automated immunoassay systems.

Results

Sensitivity and specificity varied considerably; IgG specificity between 90.9% (85.9–94.2) and 100% (97.7–100.0), and IgG sensitivity between 53.8% (41.9–65.4) and 98.5% (91.0–100.0). Combining our evaluation criteria, none of the 28 rapid tests that detected IgG had an overall performance considered “good”, seven tests were considered “acceptable”, while 21 tests were considered “not acceptable”. Four tests detected only total antibodies and were not given an overall evaluation. IgG sensitivity and/or specificity of the automated immunoassays did not exceed that of many rapid tests.

Conclusion

When prevalence is low, the most important analytical property is a test’s IgG specificity, which must be high to minimize false positive results. Out of 32 rapid tests, none had a performance classified as “good”, but seven were classified as “acceptable”.

1. Introduction

The current worldwide Coronavirus disease 2019 (COVID-19) pandemic, caused by the severe acute respiratory disease coronavirus 2 (SARS-CoV-2) [1], has led to a surge in research regarding testing, prevention, and treatment. Until safe and efficient vaccines are available for everyone, the World Health Organization maintains that accurate and efficient testing is among the key elements in the strategy to limit virus spread [2]. The current gold standard to detect present infection is real-time revers transcription PCR (RT-PCR), detecting viral RNA directly in a sample from the upper or lower airways, a relatively personnel-, time- and resource demanding procedure.

In late March 2021, the Norwegian Institute of Public Health estimated that approximately 2.5% of the Norwegian population had been infected with SARS-CoV-2 [3]. To what extent SARS-CoV-2 infection leads to transient or long-lasting immunity is debated [4], [5]. Using simple and cheap antibody detecting tests could potentially of value in some situations, for instance to confirm past infection or for epidemiological surveillance [6], [7], [8], [9]. For these purposes, a test’s ability to reliably detect immunoglobulin G (IgG) is considered most important [6], [7], [10].

Many inexpensive antibody detecting tests designed for point-of-care use are currently available for professional use. Although the number of published manufacturer-independent evaluations are growing [9], [11], [12], [13], [14], [15], [16], [17], [18], [19], [20], [21], [22], [23], [24], data on test performance is not always available. The aim of the present study was to evaluate the analytical performance and user friendliness of several rapid tests to aid health care professionals in their choice of antibody-detecting test, particularly in a low prevalence, point-of-care setting. We also included three automated immunoassay systems from hospital laboratories for partial evaluations.

2. Material and methods

The study was a collaboration between the Kristiansand Municipality, Norway, Vestre Viken Hospital Trust, Norway, Lillebælt Hospital, Denmark, and the Norwegian Organization for Quality Improvement of Laboratory Examinations (Noklus).

2.1. Serum samples

To evaluate the analytical specificities of the rapid tests, we used 99 pre-pandemic serum samples from Vejle biobank, primarily set up to study etiologies of diabetes and its comorbidites [25]. We also used 98 pre-pandemic clinical samples left over from routine analyses at Vestre Viken Hospital trust, with no information on indication or suspected diagnosis. To evaluate the analytical sensitivities of the rapid tests and automated immunoassays, we used 65 serum samples from 65 convalescent participants who had previously been confirmed infected with SARS-CoV-2 by RT-PCR. All 65 had been community treated, not requiring hospitalization.

2.2. Rapid tests

The selection of rapid tests (Table 1 and Supplementary table) consisted of the tests that manufacturers or suppliers could send to Noklus by the set deadlines. The first 17 rapid tests were evaluated during June 2020 (65 samples from convalescents and 99 pre-pandemic samples) and September 2020 (98 pre-pandemic samples), and the final 15 rapid tests during October 2020 (all samples). To avoid unnecessary thawing and freezing cycles, the samples were kept in small aliquots at −80 °C and thawed only prior to analyses. All rapid tests were lateral flow immunochromatographic assays, except number 32, which was based on a microfluidic system. Three tests required a reader (tests 8, 31 and 32), while the rest were read visually. Most tests had separate fields for detection of immunoglobulin M (IgM) and IgG on the same cartridge, although four tests detected only total antibodies (number 14, 21, 30, and 32). All rapid tests were performed under standardized and optimal conditions in accordance with the manufacturers’ instructions at Noklus’ headquarters. Faint banding was considered a positive result. Results were read independently by two biomedical laboratory scientists, and in cases of discordant results, a third was used as an arbitrator. Test interpretation was not blinded to reference standard status.

Table 1.

Rapid tests included and user-friendliness.

Test number	Test name	Manufacturer	Antigen target	User-friendliness
1	iCare Covid-19 Rapid Test (Covid-19 IgG/IgM Rapid test Kit)	Nantong Egens Biotechnology Co., Ltd, China	S protein	Easy to perform test. Easy to read result. The fields on the test cassette were suboptimally marked for one lot (white elevations on white background). 14% of test results read by more than two BLS.
2	Healgen COVID-19 IgG/IgM Rapid Test Cassette	Healgen Scientific Limited Liability Company, USA	S protein	Easy to perform test. Easy to read result. The fields on the test cassette were suboptimally marked (white elevations on white background). Did not correspond to picture following the kit. 4% of test results read by more than two BLS.
3	NADAL COVID-19 IgG/IgM Test	nal von minden GmbH, Germany	N + S protein	Easy to perform test. Easy to read result. The fields on the test cassette were suboptimally marked (white elevations on white background). 3% of test results read by more than two BLS.
4	BIOZEK Medical COVID-19 IgG/IgM Rapid Test Cassette	Inzec International Trading, The Netherlands	N + S protein	Easy to perform test. Weak color on control and test lines.3% of test results read by more than two BLS.
5	BIOSYNEX COVID −19 BSS	BIOSYNEX SWISS SA, Switzerland	S protein	Easy to perform test. Weak color on control and test lines.10% of test results read by more than two BLS.
6	Panbio COVID-19 IgG/IgM Rapid Test Device	Abbott Rapid Diagnostics Jena GmbH, Germany	N protein	Easy to perform test. Easy to read result. The fields on the test cassette were suboptimally marked (white elevations on white background). 3% of test results read by more than two BLS.
7	Acro 2019-nCoV IgG/IgM Rapid Teset	Acro Biotech Inc, USA	N + S protein	Easy to perform test. Easy to read result. 3% of test results read by more than two BLS.
8	ichroma COVID-19 Ab+ ichroma II instrument	Boditech Med Incorporated, Republic of Korea	N + S protein	Requires pre analytical mixing of blood and buffer, a pipette for analyses, and an instrument for reading of result.
9	COVID-19 IgG-IgM Rapid test	DIASource ImmunoAssays S.A., Belgium	N protein	Easy to perform test. Easy to read result. 2% of test results read by more than two BLS.
10	Diagnostic Kit for IgM/IgG Antibody to Coronavirus (SARS-CoV-2) (Lateral Flow)	Zhuhai Livzon Diagnostics Inc., China	N + S protein	Two test cassettes (one for IgM and one for IgG). Difficult to open buffer vial without spilling contents. Easy to read result. 5% of test results read by more than two BLS.
11	COVISURE™ COVID-19 IgG-IgM Rapid Test	W.H.P.M. Biosearch & Technology Co.,Ltd., China	Not specifieda	Easy to perform test. Difficult to read result due to pink background. The fields on the test cassette were suboptimally marked (white elevations on white background). 7% of test results read by more than two BLS.
12	STANDARD Q COVID-19 IgM/IgG Combo Test	SD Biosensor, Republic of Korea	N protein	Easy to perform test. Easy to read result. 6% of test results read by more than two BLS.
13	Novel Coronavirus (2019-nCoV) IgG/IgM Test Kit (Colloidal gold)	Genrui Biotech Inc., China	Not specifieda	Difficult to apply serum into the sample well. The test cassette did not correspond to picture following the kit. Easy to read result. 3% of test results read by more than two BLS.
14	WANTAI SARS-CoV-2 Ab Rapid Test	Beijing Wantai Biological Pharmacy Enterprise Co., Ltd., China	S protein	Easy to perform test. Difficult to read result due to pink background. The fields on the test cassette were suboptimally marked for one lot (white indentations on white background). 2% of test results read by more than two BLS.
15	Leccurate SARS-CoV-2 Antibody Test Kit	Beijing Lepu Medical Technology Co., Ltd., China	N protein	Easy to perform test. Particularly easy to read result due to larger test cassette. 4% of test results read by more than two BLS.
16	OnSite Covid-19 IgG/IgM	CTK Biotech, USA	Not specifieda	Easy to perform test. Easy to read result. 4% of test results read by more than two BLS.
17	COVID-19 IgG/IgM Rapid Test Kit	Abbexa Ltd, UK	N + S protein	Easy to perform test. Usually easy to read result, but occasionally white lines would appear in IgG test field (read as negative). 8% of test results read by more than two BLS.
18	Anti-SARS-CoV-2 Rapid Test	AutoBio Diagnostics	S protein	Easy to perform test. Easy to read result but light pink background not optimal for reading weak positive results. 4% of test results read by more than two BLS.
19	Instant-View COVID-19 IgG/IgM Antibody Test	Alfa Scientific Designs, Inc. 13,200 Gregg street, Poway CA 92,064 USA	N + S protein	Easy to perform test. Easy to read results but not optimal for reading IgM results due to the design of the cassette. 8% of test results read by more than two BLS.
20	2019-nCOV IgG/IgM rapid test	Dynamiker Biotechnology (Tianjin) Co., Ltd., China	N + S protein	Easy to perform test. Easy to read result. 3% of test results read by more than two BLS.
21	INgezim COVID 19 CROM (kassett)	Inmunología y Genética Aplicada, S.A. (INGENASA), Spain	N protein	Easy to perform test. Easy to read result. 3% of test results read by more than two BLS.
22	SARS -CoV-2 IgM/IgG Antibody Detection Kit	HONGKONG SENTE INDUSTRIAL INTERNATIONAL TRADE CO., LIMITED, China	Not specifieda	Easy to perform test. The cassette was not optimally marked - “T1” and “T2” instead of “IgG” and IgM”, also with white elevations on white background. 3% of test results read by more than two BLS.
23	COVID19 IgG & IgM Test Kit(colloidal gold method)	Zhejiang Anji Saianfu Biotech Co.,Ltd., China	N protein	Easy to perform test. Easy to read result, but high proportion of invalid tests (3%). 10% of test results read by more than two BLS.
24	COVID-19 IgG/IgM Rapid Test	Hangzhou AllTest Biotech Co., Ltd., China	S protein	Easy to perform test. Easy to read result. 6% of test results read by more than two BLS.
25	2019-nCovid IgG/IgM Rapid Test Cassette	BioMaxima	N protein	Easy to perform test. Easy to read result.3% of test results read by more than two BLS.
26	Diagnostic Kit for SARS-Cov-2 IgM/IgG Antibody (Collodial Gold)	Shanghai Kehua Biological Engineering Co., Ltd.	N protein	Easy to perform test. The cassette was not optimally marked - “T1” and “T2” instead of “IgG” and IgM”. 2% of test results read by more than two BLS.
27	nCOVID-19 IgG & IgM POCT	Technogenetics S.r.l, Italy	N + S + E protein	Easy to perform test. The cassettes were not optimally marked - “T1” and “T2” instead of “IgG” and IgM”. 3% of test results read by more than two BLS.
28	StrongStep® COVID-19 IgG/IgM Combo Test	NanJing Liming Bio-products Co. Ltd.	N + S protein	Easy to perform test. Easy to read result but light pink background not optimal for reading weak positive results, especially in the IgM area.3% of test results read by more than two BLS.
29	COVID-19 IgG/IgM RAPID TEST	ASSUT EUROPE SPA	Not specifieda	Easy to perform test. Easy to read result. 5% of test results read by more than two BLS.
30	EBS Alert SARS-CoV-2 ANTIBODY RAPID TEST	Excelsior Bio-System Incorporation	N protein	Easy to perform test, but hard to avoid air bubbles in buffer vial. The cassette was not optimally marked with white elevations on white background. The positive line often incomplete. 4% of test results read by more than two BLS.
31	Chembio DPP COVID-19 IgM/IgG System 2.0	Chembio Diag. Systems Inc	N protein	Requires pre analytical mixing of blood and buffer, a pipette for analyses, and an instrument for reading of result. Numerous procedure steps.
32	LumiraDx SARS-CoV-2 Ab Test	LumiraDx UK Ltd	S protein	Easy to perform, requires instrument for reading.

^aNot specified in product insert, not available online, and manufacturer/supplier did not respond to our request for information.

2.3. Automated immunoassay systems

Due to the limited volume of some serum samples, they were not all analyzed on all three automated immunoassay systems. Serum samples from the 65 previously RT-PCR positive participants were analyzed on two different platforms for qualitative detection of SARS-CoV-2 IgG; DS2 ® Automated ELISA Processing System (DYNEX Technologies, Inc. 14,340 Sullyfield Circle, Chantilly, VA 20,151 USA) was used with “EDITM Novel Coronavirus COVID-19 IgG Elisa kit” (Epitope Diagnostics, Inc.7110 Carroll Road, San Diego, CA 92121, US), and Alinity i (Abbott, Abbott Park, Illinois, U.S.A.) was used with the kit “SARS-CoV-2 IgG” ref 06R90 (Abbott Ireland, Diagnostics Division, Finisklin Business Park, Sligo, Ireland). 46 of the samples were further analyzed on an iFlash Immunoassay Analyzer 1800 (Shenzen YHLO Biotech Co. ltd. China) with the kit “SARS-CoV-2 IgG” ref C86095G. The 98 pre-SARS-CoV-2 serum samples from Vestre Viken Hospital trust were analyzed on the DS2 platform, and the 99 pre-SARS-CoV-2 serum samples from Vejle biobank were analyzed on the iFlash platform. Only IgG results are reported.

2.4. Statistical methods

Stata IC/16.1 (StataCorp LLC) was used for statistical analyses. IgG and IgM specificities were calculated separately (where possible) from analyses of pre- pandemic sera and defined as the proportion of SARS-CoV-2 antibody negative samples. IgG and IgM sensitivities were defined as the proportion of recovered COVID-19 participants who had detectable IgG and IgM antibodies, respectively. We computed 95% confidence intervals (CIs) for the sensitivities and specificities using the Agresti-Coull Method [26]. Automated immunoassays all report an ambiguous area around the cut-off value, but for the purpose of calculating sensitivity and specificity, we used the laboratory reported cut-offs to classify borderline results as positive or negative (iFlash 1800: 12 AU, DYNEX DS2: 1.0 S/CO, and Alinity i: 1.4 S/C).

2.5. Evaluation criteria

For any test, there is usually a trade-off between sensitivity and specificity, and the most important properties of a test will vary with the clinical situation [27]. When the prevalence of past and present COVID-19 is low, the most important property of a test is a very high specificity in order to minimize the risk of false positive results, and both ≥ 99% [28] and ≥ 97% [10] have been suggested cut-offs. To facilitate the choice of antibody-detecting rapid test in a low prevalence, point-of-care setting, we suggest the following criteria to classify rapid test performance as “good”, “acceptable” or “not acceptable”:

• 1.
  IgG specificity:

  • •
    “good” if the lower limit of the 95% CI of the point estimate is ≥ 97.0%
  • •
    “acceptable” if the point estimate is ≥ 97.0% (while lower limit of the 95% CI less than 97.0%)
  • •
    otherwise “not acceptable”
• 2.
  IgG sensitivity:

  • •
    “good” if the point estimate is ≥ 90.0%
  • •
    “acceptable” if the point estimate is in the interval [85.0% − 90.0%>
  • •
    otherwise “not acceptable”
• 3.
  User-friendliness (for a point-of-care setting):

  • •
    “not acceptable” if complicated to perform or difficult to read result
  • •
    otherwise “good”

To receive an overall evaluation of “good”, all three performance characteristics should be classified as “good”. If one is classified as “not acceptable”, the overall evaluation is “not acceptable”. Otherwise, the performance should be considered “acceptable”. Tests that detected total antibodies and not IgG specifically, were not given an overall evaluation.

2.6. Ethical considerations

The project was considered a method evaluation study and therefore exempt from ethical board approval in Norway. Recovered COVID-19 participants gave written informed consent to participate. In Denmark, use of restmaterial as separated plasma/serum from anonymous healthy persons for technical quality control is not restricted. The project was approved by the Data protection officers in Kristiansand Municipality, at Vestre Viken Hospital Trust, and at Noklus. Suppliers provided their tests free of charge to Noklus and did not pay for the evaluation. In sending the tests, they consented to having the results published.

3. Results

When blood samples were drawn from the 65 recovered COVID-19 participants, the number of days since their onset of symptoms was between 37 and 89 (median 67 days). The participants were between 15 and 75 years old (median age 53), and 38 (58%) were women. Participants reported having had varying degrees of symptoms during COVID-19, though none had required hospitalization.

The analytical performance of the rapid tests varied considerably (Table 2 and Supplementary figure). Twenty-one rapid tests had IgG specificity above 97%. IgM specificity was generally equal to or lower than IgG-specificity (Table 2). We were only able to evaluate the analytical specificity of two out of three automated immunoassays due to small available volumes of pre-pandemic samples. The results, however, suggest analytical specificities of the iFlash and DYNEX DS2 systems were not necessarily superior to several of the rapid tests (Table 3 ).

Table 2.

Antibody detecting rapid tests, results and classification of performance^a.

Rapid test	IgM		IgG		User-friendliness	Overall evaluation
	Sensitivity (95% CI)	Specificity (95% CI)	Sensitivity (95% CI)	Specificity (95% CI)
3	70.7 (58.7–80.5)	98.0 (94.7–99.4)	90.8 (81.0–96.0)	99.5 (96.9–100)	Good	Acceptable
2	67.7 (55.6–77.8)	99.0 (96.1–99.96)	98.5 (91.0–100)	99.0 (96.1–99.96)	Good	Acceptable
4	15.4 (8.4–26.3)	96.4 (92.7–98.4)	92.3 (82.8–97.1)	99.0 (96.1–99.96)	Good	Acceptable
7	15.4 (8.4–26.3)	95.4 (91.4–97.7)	87.7 (77.3–93.9)	99.0 (96.1–99.96)	Good	Acceptable
12	63.1 (50.9–73.8)	96.4 (92.7–98.4)	98.5 (91.0–100)	98.5 (95.4–99.7)	Good	Acceptable
15	81.5 (70.3–89.3)	94.9 (90.8–97.3)	87.7 (77.3–93.9)	98.5 (95.4–99.7)	Good	Acceptable
16	69.2 (57.2–79.2)	98.0 (94.7–99.4)	92.3 (82.8–97.1)	98.0 (94.7–99.4)	Good	Acceptable
5	73.8 (62.0–83.1)	95.9 (91.9–98.0)	84.6 (73.7–91.6)	100 (97.7–100)	Good	Not acceptable
10	55.4 (43.3–66.8)	99.5 (96.9–100)	60.0 (47.8–71.0)	100 (97.7–100)	Good	Not acceptable
6b	9.2 (4.0–19.0)	98.0 (92.5–99.9)	78.5 (66.9–86.8)	100 (95.5–100)	Good	Not acceptable
19	96.9 (88.8–99.8)	82.1 (76.0–86.8)	78.5 (66.9–86.8)	99.5 (96.9–100)	Good	Not acceptable
28	75.4 (63.6–84.3)	96.9 (93.3–98.7)	67.7 (55.6–77.8)	99.5 (96.9–100)	Good	Not acceptable
20	76.9 (65.5–85.6)	92.8 (88.2–95.7)	66.2 (54.0–76.5)	99.0 (96.1–99.96)	Good	Not acceptable
25	20.0 (11.9–31.4)	96.9 (93.3–98.7)	84.6 (73.7–91.6)	98.5 (95.4–99.7)	Good	Not acceptable
9	20.0 (11.9–31.4)	97.0 (93.4–98.7)	81.5 (70.3–89.3)	98.5 (95.4–99.7)	Good	Not acceptable
29	73.8 (62.0–83.1)	94.3 (90.0–96.9)	83.1 (72.0–90.5)	98.5 (95.3–99.7)	Good	Not acceptable
24	55.4 (43.3–66.8)	96.4 (92.6–98.4)	96.9 (88.8–99.8)	96.9 (93.2–98.7)	Good	Not acceptable
18	16.9 (9.5–28.0)	99.5 (96.9–100)	90.8 (81.0–96.0)	92.9 (88.3–95.8)	Good	Not acceptable
17b	78.5 (66.9–86.8)	80.8 (71.9–87.4)	96.9 (88.8–99.8)	91.9 (84.6–96.1)	Good	Not acceptable
1	72.3 (60.4–81.8)	89.3 (84.2–93.0)	84.6 (73.7–91.6)	90.9 (85.9–94.2)	Good	Not acceptable
13	67.7 (55.6–77.8)	95.4 (91.4–97.7)	75.4 (63.6–84.3)	99.5 (96.9–100)	Not acceptable	Not acceptable
31	56.9 (44.8–68.2)	98.9 (96.0–99.96)	95.4 (86.8–98.9)	98.9 (96.0–99.96)	Not acceptable	Not acceptable
26	24.6 (15.7–36.4)	98.5 (95.4–99.7)	78.5 (66.9–86.8)	98.5 (95.4–99.7)	Not acceptable	Not acceptable
22	60.0 (47.8–71.0)	97.9 (94.6–99.4)	53.8 (41.9–65.4)	98.0 (94.7–99.4)	Not acceptable	Not acceptable
27	27.7 (18.2–39.6)	98.5 (95.4–99.7)	86.2 (77.5–92.8)	97.0 (93.4–98.7)	Not acceptable	Not acceptable
8	4.6 (1.1–13.2)	99.5 (96.9–100)	92.3 (82.8–97.1)	95.9 (92.1–98.1)	Not acceptable	Not acceptable
11b	46.2 (34.6–58.1)	95.9 (89.6–98.7)	58.5 (46.3–69.6)	95.9 (89.6–98.7)	Not acceptable	Not acceptable
23	47.6 (35.8–59.7)	95.2 (91.0–97.6)	92.1 (82.3–96.9)	95.2 (91.0–97.6)	Not acceptable	Not acceptable
			Total antibodies,lateral flow assays		User-friendliness
21			80.0 (68.6–88.1)	99.5 (96.9–100)	Good
14			83.1 (72.0–90.5)	98.0 (94.7–99.4)	Good
30			64.6 (52.4–75.2)	91.7 (86.9–94.9)	Not acceptable
			Total antibodies, microfluidic system		User-friendliness
32			100 (93.3–100)	99.5 (96.8–100)	Good

^aRapid tests were classified according to three performance characteristics: (i) IgG specificity, (ii) IgG sensitivity, and (iii) user-friendliness, see text for details. No test was classified as good in all three areas; hence no test received an overall evaluation of “good”. When at least one characteristic was classified as “not acceptable”, so was the overall evaluation. Otherwise, performance was considered “acceptable”. Tests are sorted according to overall evaluation, user-friendliness, and IgG specificity.

^bSpecificities calculated from 99 serum samples from Vejle biobank.

Table 3.

Automated immunoassays, results.

Test name	Manufacturer	Antigen target	IgG
			Sensitivity (95% CI)	Specificity (95% CI)
iFlash 1800a	SHENZHEN YHLO BIOTECH CO., LTD., China	N-protein	84.8 (71.5–92.7)	99.0 (93.9–100)
DYNEX DS2b	DYNEX Technologies, USA	N-protein	86.2 (75.5–92.8)	96.0 (89.7–98.7)
Alinity i	Abbott, USA	N-protein	87.7 (77.3–93.9)

^aSpecificites calculated from 99 serum samples from Vejle biobank.

^bSpecificites calculated from 98 serum samples from Vestre Viken Hospital Trust.

Analytical sensitivity also varied considerably. Eleven rapid tests had point estimates of IgG sensitivity above 90%, while 14 tests had point estimates of IgG sensitivity below 85%. IgG sensitivities of the included automated immunoassays did not exceed that of several of the rapid tests (Table 3).

We calculated predictive values at various prevalences for three rapid tests at different ends of the performance spectrum (Table 4 ). While high specificity increases positive predictive value (PPV) at lower prevalences, even a test with a specificity of 99% (test 2) would have less than 70% positive predictive value at 2% prevalence.

Table 4.

Positive and negative predictive values (PPV and NPV) for three rapid tests at various prevalences.

	Test 3a		Test 2b		Test 11c
Prevalence	PPV	NPV	PPV	NPV	PPV	NPV
2%	78.8	99.8	66.8	100	22.6	99.1
5%	90.5	99.5	83.8	99.9	42.9	97.8
10%	95.3	99.0	91.6	99.8	61.3	95.5
20%	97.8	97.7	96.1	99.6	78.1	90.2

^aIgG sensitivity: 90.8%, IgG specificity: 99.5%.

^bIgG sensitivity: 98.5%, IgG specificity: 99.0%.

^cIgG sensitivity: 58.5%, IgG specificity: 95.9%.

There was great variability in the number of rapid tests that were positive in samples from each of the 65 recovered COVID-19 participants (Supplementary figure 1, panel B). For instance, 18 of the 65 participants tested positive for IgG antibodies on all 28 IgG detecting rapid tests. While none of the tests had 100% sensitivity, no participant tested negative on all rapid tests either.

The majority of the tests were considered easy to perform and interpret. For three rapid tests, more than 10% of test results had to be interpreted by more than two BLS to reach consensus (Table 1). Nine tests were judged not acceptable by the user-friendliness evaluation criterion for a point-of-care setting.

Combining our evaluation criteria of IgG specificity, IgG sensitivity and user-friendliness, no test’s overall performance was considered “good”, but tests 2, 3, 4, 7, 12, 15, and 16, were considered “acceptable” for use in a low prevalence, point-of-care setting.

4. Discussion

We evaluated 32 antibody-detecting rapid tests for SARS-CoV-2 using criteria of IgG specificity, IgG sensitivity, and user-friendliness. We found great variability in analytical performance. Emphasizing test properties considered most important in a low prevalence, point-of-care setting, no test was considered “good”, but seven tests were given an overall evaluation of “acceptable”.

Strengths of our study include the large number of rapid tests evaluated under identical and optimal conditions, allowing direct comparisons of test properties. The use of serum samples predating the emergence of SARS-CoV-2 allowed us to evaluate analytical specificities. Also, since previous studies have shown that the amount of antibodies produced is associated with the severity of COVID-19 [14], [29], analytical properties of the tests will depend on the population it is used in [28]. To evaluate the rapid tests in a community setting, we used sera from recovered, community-treated COVID-19 patients (all confirmed RT-PCR positive), who had not been hospitalized for COVID-19. In addition, the community treated recovered participants all had more than a month between onset of symptoms and blood sampling, allowing everyone ample time to develop antibodies [30], [31].

Our study also has a number of limitations. We did use a reasonably large number of serum samples for the evaluation, both from recovered COVID-19 participants (n = 65) and from before the pandemic (n = 197). However, a larger number of serum samples would have made the evaluation even more robust. We did not have access to sera with known antibodies, or sera from patients with a known previous non-SARS coronavirus or other infection, to further challenge the tests for cross-reactivity. Also, while we did not have sufficient volumes of the pre-pandemic sera to allow full evaluations of the automated immunoassays, we included the results mainly to give an indication of their performance compared to that of the rapid tests. Further, by performing the evaluation under optimal conditions, not by intended users, and not blinded to reference standard status, both pre-analytical and analytical errors were minimized, thus performance could be poorer in real life. In addition, even if all manufacturers state that full blood (capillary or venous), serum or plasma are equally suitable test materials, evaluating rapid tests using whole blood rather than serum would have closer mimicked real-life use. It is reassuring that studies have reported that the performance of most rapid antibody tests using whole blood was comparable to serum or plasma [22], [32]. Finally, we were only able to investigate analytical sensitivity and specificity, and not real-life diagnostic accuracy of the rapid tests. Test performance could therefore be poorer in a diagnostic setting. An ideal, prospectively designed study would involve repeated testing with RT-PCR and antibody tests for a large group of people over a prolonged period of time. Still, we believe this study provides valuable and relevant information for health care professionals faced with a choice between many antibody-detecting rapid tests.

Several previous evaluations of rapid tests have now been published. Some are of limited quality because they use sera from a small number of pre-pandemic or COVID-19 participants [11], [12], [17], [18], [20], [21], [23], [24]. Many use sera from hospitalized COVID-19 patients [11], [13], [15], [16], [17], [18], [20], [21], or do not state whether participants had been hospitalized [12], [23], which limits knowledge about the tests’ usefulness in a community setting. Some studies report results for IgM and IgG combined [11], [16], [17], [20], which does not allow conclusions about past infection. There seems to be a very large number of antibody-detecting rapid tests available, and we found only a few studies including some of the tests we evaluated. In these cases, the study designs were not considered similar enough to allow meaningful comparisons of test performances across studies. The Cochrane review published in June 2020, also noted the lack of high-quality diagnostic accuracy studies evaluating SARS-CoV-2 antibody tests in general, and point-of-care rapid tests in particular [9].

Tests 14, 21, 30, and 32 detected total antibodies and not IgG antibodies specifically. In our study, IgM-specificity was generally equal to or lower than IgG-specificity, implying a higher risk of false positive results if using IgM results. Thus, in a patient with a positive total-antibody test, the possibility of an isolated IgM response, which could be due to unspecific cross-reactivity, cannot be ruled out without supplementary testing. Furthermore, it may be considered a disadvantage that IgM and IgG tests often come in the same cartridge. Past infection is confirmed with IgG alone [10], but the IgM result may be misinterpreted and cause confusion [15]. At worst, unspecific cross-reactivity may give the wrongful impression that the patient has some protection against future infection with SARS-CoV-2, which may affect behavior and increase the risk of future COVID-19 and spread of SARS-CoV-2. It is noteworthy, however, that test 32, despite the risk of unspecific interference, demonstrated high specificity in combination with a very high sensitivity in our study population

Our user-friendliness criterion was primarily designed for a point-of-care setting (i.e., primary health care, health center, nursing home, etc.). Test 31 is intended for use in laboratories of moderate complexity and has for that reason been judged as not acceptable under the user-friendliness criterion. However, that does not imply that this test is not suitable for use in a laboratory facility of moderate complexity.

It has been reported that not everyone infected with SARS-CoV-2 will develop detectable antibodies [30], [33]. The various antibody-detecting tests target antibodies against different SARS-CoV-2 antigens, and for the rapid tests, which antigen it detects is rarely declared. It is an interesting observation that while none of our rapid tests detected antibodies in all the recovered COVID-19 participants, none of the participants tested negative on all the rapid tests either. The differences probably reflect that people infected with SARS-CoV-2 produce both different amounts and different types of antibodies. Our results further suggest that by combining several antigens, or tests targeting several antigens, sensitivity may be increased. However, this could possibly increase the risk of false positive results, thereby lowering specificity.

Clinical use of antibody detecting rapid tests is currently debated. In a hospital setting where RT-PCR is negative, antibody status may be helpful for the clinician [9]. Confirming past infection in a community setting has also been suggested, as well as seroprevalence studies for epidemiological surveillance [9]. For tests using the Spike-protein of SARS-CoV-2 as the antigen, vaccination status confirmation is an upcoming possible indication that we have not investigated. Since we evaluated rapid tests in a population that had had varying degrees of symptoms during COVID-19, we do not know how the tests would perform in those who have had very little or no symptoms, which would be relevant in a seroprevalence study. Currently in Norway, where only an estimated 2.5% of the population have ever been infected with SARS-CoV-2, we suggest confirmation of past COVID-19 infection in a community setting may be the most appropriate area of use for an antibody-detecting rapid tests. What is considered the most important property of a rapid test will vary with the clinical setting, but in this situation, avoiding false positive results is important. For this reason, we have emphasized IgG specificity as the most important test property. However, as a larger proportion of the population becomes vaccinated, this type of use will probably become gradually less relevant. The appropriate future use of antibody-detecting rapid tests in the clinical pathway of SARS-CoV-2 infection is currently uncertain.

5. Conclusion

When an antibody-detecting rapid test is used in a low prevalence setting, the most important consideration should be the test’s IgG specificity, which must be very high to minimize the risk of false positive results. Taking also IgG sensitivity and user friendliness into consideration, none of the 32 rapid tests evaluated had a performance classified as “good”, but seven tests were classified as “acceptable” for use in a low prevalence, point-of-care setting.

Funding

This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

CRediT authorship contribution statement

Mette C. Tollånes: Conceptualization, Methodology, Formal analysis, Investigation, Writing - original draft, Project administration. Pål A. Jenum: Conceptualization, Writing - review & editing. Helene Kierkegaard: Methodology, Investigation, Writing - review & editing. Eirik Abildsnes: Conceptualization, Writing - review & editing. Roar Magne Bævre-Jensen: Methodology, Investigation, Writing - review & editing. Anne C. Breivik: Conceptualization, Methodology, Investigation, Writing - review & editing. Sverre Sandberg: Conceptualization, Writing - review & editing, Supervision.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Appendix A. Supplementary material

The following are the Supplementary data to this article: