Diagnostic performance of commercially available COVID-19 serology tests in Brazil

Abstract

Timely and accurate laboratory testing is essential for managing the global COVID-19 pandemic. Reverse transcription polymerase chain reaction remains the gold-standard for SARS-CoV-2 diagnosis, but several practical issues limit the test’s use. Immunoassays have been indicated as an alternative for individual and mass testing. Objectives: To access the performance of 12 serological tests for COVID-19 diagnosis.

Methods

We conducted a blind evaluation of six lateral-flow immunoassays (LFIAs) and six enzyme-linked immunosorbent assays (ELISAs) commercially available in Brazil for detecting anti-SARS-CoV-2 antibodies.

Results

Considering patients with seven or more days of symptoms, the sensitivity ranged from 59.5% to 83.1% for LFIAs and from 50.7% to 92.6% for ELISAs. For both methods, the sensitivity increased with clinical severity and days of symptoms. The agreement among LFIAs performed with digital blood and serum was moderate. Specificity was, in general, higher for LFIAs than for ELISAs. Infectious diseases prevalent in the tropics, such as HIV, leishmaniasis, arboviruses, and malaria, represent conditions with the potential to cause false-positive results with these tests, which significantly compromises their specificity.

Conclusion

The performance of immunoassays was only moderate, affected by the duration and clinical severity of the disease. Absence of discriminatory power between IgM/IgA and IgG has also been demonstrated, which prevents the use of acute-phase antibodies for decisions on social isolation.

Introduction

A novel coronavirus (SARS-CoV-2) disease (COVID-19) was first identified in Wuhan City, Hubei Province, China in December 2019, followed by an outbreak across the world. On March 12, the World Health Organization (WHO) declared COVID-19 a global pandemic (World Health Organization (WHO), 2020a). Four months later, more than 12 million cases and 550 000 deaths had already been reported worldwide. In July 2020, notified cases in Brazil exceeded 1 800 000 (BRASIL, 2020), with associated deaths exceeding 70 000, placing the country at the epicenter of the pandemic in Latin America, with the second-highest case rate in the Americas after the USA (Max Roser HR, Esteban Ortiz-Ospina and, 2020).

Preventing transmission to control the spread of SARS-CoV-2 — from symptomatic and asymptomatic individuals (Lee et al., 2020a) — is the main objective of any containment strategy. The approach of testing, tracking, and tracing has become a central tool for achieving this objective (World Health Organization (WHO), 2020b). However, the response to the COVID-19 pandemic has been hampered by a lack of aggressive testing for the infection in several regions of world. To date, assays based on the reverse transcription polymerase chain reaction (RT-PCR) in respiratory samples are the gold standard for COVID-19 diagnosis (Tang et al., 2020). This diagnostic strategy has been limited by significant logistic and capacity constraints, ranging from the short period of high viral excretion levels in respiratory secretions to poor availability of well-equipped laboratories, trained personnel, reagents, swabs used for the collection of nasopharyngeal specimens, and personal protective equipment for healthcare providers collecting samples. Thus, RT-PCR is particularly challenging in resource-limited settings. Additionally, RT-PCR execution is relatively time consuming and highly dependent on the pre-analytical phase.

In this context, numerous immunological tests, based on detection of antigens or antibodies and including point-of-care or conventional platforms, have recently become available and approved for use worldwide. The tests developed to detect SARS-CoV-2 antibodies are typically based on lateral-flow immunoassays (LFIAs), enzyme-linked immunosorbent assays (ELISAs), or chemiluminescent immunoassays (CLIAs). Unlike tests based on viral detection, whose diagnostic window is short and related to the period of viral excretion, serological tests have the advantage of being longer-lasting markers of infection, and have classically been used as a tool in assessing the dissemination of infections in populations.

Currently, the available tests predominantly target antibodies to the main surface proteins of the novel coronavirus (Lu et al., 2020). In theory, a serological strategy, using a point-of-care approach based on rapid tests, would have the potential to significantly improve the current testing capacity for COVID-19. Serology is easier to perform, requiring less technical expertise and equipment, and has a much lower unitary cost than RT-PCR assays. The samples are blood collected in tubes or taken from digital pulp, which pose a lower potential risk to the healthcare staff. Serology can be performed in a basic clinical laboratory or in community settings, thereby allowing wider application. These potential advantages have been sufficient to encourage governments from several countries, especially those with limited resources, and employers in the private sector, to acquire and use serological tests on a large scale during the COVID-19 pandemic, both as a diagnostic tool and as a marker of previous infection and guarantee of immunity. However, the serodiagnostic power of the specific IgM, IgA, and IgG antibodies against SARS-CoV-2 remains largely uncertain, as does the relationship between the presence of antibodies and the level of immunity against reinfection (Wölfel, 2020). Although serological tests contribute little to urgent decisions on social withdrawal and quarantine, from a public health perspective, serological analysis could be useful for estimating epidemiological variables, such as attack rates and case–fatality rates, which are necessary to assess the virus’s community transmission and burden (Lee et al., 2020b).

Regardless of the intended use, the first stage of any decision on the implementation of serological tests in the context of the COVID-19 pandemic is the careful analysis of their performances over the various stages of the infection in different specimens, and their specificity, challenged in the face of other clinical conditions. In this study, we aimed to describe the accuracy of serological assays for COVID-19 registered in Brazil up to May 2020, and the comparative performances of rapid tests performed in digital whole blood and serum, and between patients with severe and mild clinical manifestations. We also assessed the positivity of the different antibodies among patients less than 7 days, between 7 and 14 days, and more than 14 days after the onset of symptoms.

Materials and methods

Study design

This panel-based study comprised 289 serum samples from 173 symptomatic patients with confirmed SARS-CoV-2 infection and 116 negative controls. All the cases (SARS-CoV-2 positive) were confirmed by RT-PCR testing of nasopharyngeal or oropharyngeal swabs, and had their clinical conditions and demographic data compiled. The RT-PCR tests used for case confirmation were performed according to the protocols proposed by the US Center for Disease Control (Centers for Disease Control and Prevention, 2019) or the Charité Hospital in Germany (Corman et al., 2020), both accepted by World Health Organization (World Health Organization (2020)). The negative control sera were all obtained before January 2020, which marked the introduction of the new coronavirus in Brazil, from patients with serological markers for other infectious or non-infectious diseases. Only one sample per individual was included in this panel.

Source of samples and ethical approval

Sera from confirmed cases of SARS-CoV-2 infection were provided by the Minas Gerais State Department of Health, which has been responsible for collecting and storing a biorepository since the beginning of the COVID-19 pandemic. Additionally, the performance of LFIAs performed on blood and serum was compared in a group of 32 patients who consented to be double tested. The SARS-CoV-2 RT-PCR-confirmed cases included hospital patients and outpatients. For each case, information about the presence of acute respiratory distress syndrome (ARDS), according to the definition adopted in Brazil (≥30 breaths/min or an oxygen saturation ≤93% at rest), was registered. Negative control sera were collected in previous studies before the emergence of the novel coronavirus, and were kindly provided by their legal guardians, with authorization from the Ethical Board of Instituto René Rachou, according to Brazilian legislation for research on humans. Only serum samples were available for the control group; consequently, no control was submitted to SARS-CoV-2 RT-PCR. Due to the small volume of serum available from patients with malaria, samples from ten patients infected with Plasmodium vivax and nine patients with Plasmodium falciparum were pooled and included in the control. The main demographic and clinical characteristics of cases, and the serological or parasitological markers that determined the diagnosis of the controls, are available as supplementary material.

Ethical approval

Ethical approval was obtained from the institutional review board of Instituto René Rachou, Fundação Oswaldo Cruz, CAAE: 30960120.0.0000.5091, approval numbers 4.001.133 and 4.128.388.

Sample calculation

The minimum sample required for this validation was estimated through a one-sample proportion test, using Statistics/Data Analysis software (Stata), version 11.0. A power of 80% and significance of 5% were chosen to reproduce the sensitivity and specificity of tests with expected exact binomial 95% confidence limits, based on the lowest performance rates reported by the manufacturers to ANVISA up to May 18, 2020 (86% for sensitivity and 98% for specificity). On this basis, a minimum of 149 cases was chosen, with seven or more days of symptoms and 116 controls. Additionally, a minimum number of 20 tests performed on digital blood and serum from SARS-CoV-2-confirmed patients was estimated as sufficient to identify a minimum difference in sensitivity of 20% between the two clinical specimens. Two of the manufacturers involved in this validation provided fewer tests than requested, so the evaluation was carried out with a proportional sub-group of case samples and negative controls, chosen randomly.

Research and selection of tests registered in Brazil

A search for diagnostics for COVID-19 was carried out in records available at the Brazilian National Health Surveillance Agency (ANVISA) via the Agency’s website (https://consultas.anvisa.gov.br/#/saude/). The search strategy was based on the terms “COVID 19”, “SARS”, “nCOV”, “COV”, and “coronavirus”, and was carried out on May 18, 2020. Sixty-seven serological tests registered in Brazil to diagnose SARS-CoV-2 were identified: 55 lateral-flow immunoassays (LFIAs), six ELISA assays, four chemiluminescence tests, and two immunofluorescence tests. Five manufacturers did not present any commercial contact information, 38 manufacturers did not respond to contact, and another three refused to participate. All the companies responsible for producing the tests, identified using the available commercial contact information, were invited to participate in this validation. By the end of June, nine companies had sent kits for validation and nine others were committed to donating the tests, albeit not yet received. Subsequently, 12 registered and commercially available serological tests for SARS-CoV-2 diagnosis were included in this analysis; their main characteristics are shown in Table 1 . They included six LFIA and six ELISA tests.

Table 1.

List of tests evaluated and their main characteristics.

Test	Registration number	Manufacturer (country)	Product batch	Storage temperature (°C)	Method	Sample volume	Specimen	Biomarker (antibody)
One Step COVID-2019 Test	80,537,410,048	Guangzhou Wondfo Biotech Co., Ltd. (China)	W195004112 W19500458	2–30 °C	LFIA	10 μL	Blood, serum, plasma	Total IgG/IgM
TR DPP® COVID-19 IGM/IGG — Bio-Manguinhos	80,142,170,039	Fundação Oswaldo Cruz (Brazil)	204EXVD01Z	2–30 °C	LFIA	10 μL	Blood, serum, plasma	IgM and IgG
COVID-19 IgG/IgM ECO Teste	80,954,880,132	Eco Diagnostica Ltda (Brazil)	202,005,043	2–30 °C	LFIA	20 μL (blood), 10 μL (serum or plasma)	Blood, serum, plasma	IgM and IgG
COVID-19 IgG/IgM	80,258,020,106	Qingdao Hightop Biotech Co., Ltd. (China)	COV1252004C	4–30 °C	LFIA	20 μL (blood) 10 μL (serum or plasma)	Blood, serum, plasma	IgM and IgG
Imuno-Rápido COVID-19 IgG/IgM	10,310,030,208	Wama Produtos Para Laboratorio Ltda (Brazil)	20E017	2–30 °C	LFIA	20 μL (blood), 10 μL (serum or plasma)	Blood, serum, plasma	IgM and IgG
COVID-19 IgG IgM Gold Analisa Diagnóstica LTDA	80,022,230,214	Gold Analisa Diagnóstica Ltda (Brazil)	200,653	2–30 °C	LFIA	20 μL (blood), 10 μL (serum or plasma)	Blood, serum, plasma	IgM and IgG
COVID-19 ELISA IgM + IgA	80,263,710,088	Vircell Microbiologists (Spain)	20ECOVMA107 20ECOVMA108	2–8 °C	ELISA	5 μL	Serum or plasma	IgM/IgA
COVID-19 ELISA IgG	80,263,710,087	Vircell Microbiologists (Spain)	20ECOVG107	2–8 °C	ELISA	5 μL	Serum or plasma	IgG
Allserum EIA COVID19 IgM	80,047,580,200	Mbiolog Diagnosticos Ltda (Brazil)	4A20 and 5A20A	2 to 8 °C	ELISA	10 μL	Serum or plasma	IgM
Allserum EIA COVID19 IgG	80,047,580,201	Mbiolog Diagnosticos Ltda (Brazil)	1B20	2–8 °C	ELISA	10 μL	Serum or plasma	IgG
Anti-SARS-CoV-2 ELISA (IgA)	10,338,930,226	Euroimmun AG (Germany)	E200507BE	2–8 °C	ELISA	10 μL	Serum or plasma	IgA
Anti-SARS-CoV-2 ELISA (IgA)	10,338,930,227	Euroimmun AG (Germany)	E200507AE	2–8 °C	ELISA	10 μL	Serum or plasma	IgA

Among the LFIA tests, only one exhibited a total antibody detection line, while in the other five the cassette displayed two test lines (M and G) and a quality control line (C). All the kits used capture reaction to detect SARS-CoV-2 antibodies, and were based on the colloidal gold-labeled immunochromatography principle and one-step method, with results obtained within 10–30 min, using whole blood, serum, or plasma samples. Briefly, the sample is absorbed by capillary action and mixes with the SARS-CoV-2 antigen-dye conjugate. The conjugate binds to the antibodies present in the sample and, after adding the buffer, the antibody–conjugate complex migrates chromatographically across the membrane and finds the test region, in which the anti-human IgG and anti-IgM antibodies are immobilized, forming a colored line. The presence of this line indicates a positive result and its absence indicates a negative result. Among the ELISA tests, three were based on IgG detection, one on IgM, one on IgA, and one on IgA and IgM without distinction.

Sample preparation and test execution

The serum samples were randomly coded and kept frozen at ―70 °C until needed, and then thawed for 10 min at room temperature and homogenized before testing. The tests were carried out following each manufacturer’s instructions strictly. All the samples were submitted to a particular test before moving on to the next test. The reproducibility of the LFIA kits was assessed through interpretation of the results by three independent observers using the Kappa index. The final result was defined as that indicated by at least two of the three readers. For one of the LFIA kits, the result was obtained using a micro reader provided by the manufacturer. For LFIA blood/serum comparison, approximately 10% of the SARS-CoV-2-confirmed patients represented in this panel were consecutively recruited to donate capillary blood for testing, until reaching the minimum of 20 tests performed for each brand test.

Data analysis

The performance parameters of interest were sensitivity, specificity, and accuracy, defined as follows: i. sensitivity (S): the proportion of positive tests among diseased individuals; ii. specificity (E): the proportion of negative tests among non-diseased individuals; iii. accuracy: the sum of true positives and true negatives among the total number of tests performed. Exact binomial 95% confidence intervals (95% CI) were calculated individually for each performance parameter by test, and sensitivity was also stratified by time since symptom onset. The Kappa index was interpreted following the criteria of Landis and Koch (1977) (Landis, 1977) (13) and interpreted as follows: <0, no agreement; 0–0.2, slight agreement; 0.2–0.4, fair agreement, 0.4–0.6, moderate agreement; 0.6–0.8, substantial agreement; 0.8–1, almost perfect agreement. McNemar’s test was used to determine the statistical differences between tests (all diagnostic tests were applied to the same set of samples). χ² tests at a significance level of 0.05 were used to determine the statistical differences. Analyses were performed using SPSS version 23 and MedCalc statistical software version 19.4.

Results

The patients whose serum samples were tested in this study were diagnosed with COVID-19 between April 21 and June 10, 2020, in Minas Gerais, Brazil. The age ranged from 22 to 96 years (median 47.5 years), and 52.6% were female. Regarding the time since the symptoms onset until testing, 25 patients (15%) had up to 6 days of symptoms, 74 patients (43%) had between 7 and 14 days from the onset of symptoms, and 74 patients had 15 days or more from the onset of symptoms. Among this latter group, 19 patients (26%) had between 31 and 60 days from the onset of symptoms and 13 (17.5%) had more than 2 months from the onset of symptoms. Fifty-nine percent of patients met the criteria for ARDS.

Negative control serum was collected from adult patients with a serological or parasitological marker for the following diseases: dengue, Zika, Chagas disease, syphilis, toxoplasmosis, viral hepatitis, malaria, visceral leishmaniasis, cytomegalovirus, Epstein Barr virus infection, and HIV infection. Besides these, the control panel also included 26 sera (22%) from patients under investigation for acute febrile illness or metabolic disease, without confirmation of an infectious condition.

Table 2, Table 3 summarize the sensitivity by serological test and immunoglobulin class detected. For LFIA tests, the sensitivity for IgM ranged from 13.3% to 72.3%, while that for IgG ranged from 51.4% to 65.9%. For all except one LFIA test, the sensitivity of IgG detection alone was numerically higher than that observed for IgM, and for all of them, the highest detection rates were observed by combining the IgM and IgG results, ranging from 52.6% to 75.1% (Table 1). Among the ELISAs, a test based on IgA/IgM detection exhibited the highest sensitivity (90.2%, 95% CI 84.9–93.8%). For IgG, the sensitivity for ELISAs ranged from 58.7% to 76.8% (Table 3). Considering only patients with 7 or more days of symptoms, the sensitivity ranged from 59.5% to 83.1% for LFIAs and from 50.7% to 92.6% for ELISAs (Table 4 ). As expected, the sensitivity for patients with fewer than 7 days of symptoms was generally poor — no more than 40% for all except one ELISA test. The sensitivity tended to increase with the number of days from symptom onset. However, there was substantial overlap between the sensitivity 95% confidence intervals for the groups of confirmed cases with 7–14 days and more than 14 days of symptoms (Table 5 ).

Table 2.

Sensitivity by lateral flow test and immunoglobulin class detected.

Test	IgM			IgG			IgM or IgG
	n	Positive	Sensitivity (95% CI)	n	Positive	Sensitivity (95% CI)	n	Positive	Sensitivity (95% CI)
One Step COVID-2019 Test (Guangzhou Wondfo Biotech)	–	–	–	–	–	–	173	124	71.7 (64.3–78.2)
TR DPP® COVID-19 IGM/IGG Bio-Manguinhos (Fundação Oswaldo Cruz)	173	91	52.6 (45.2–59.9)	173	109	63 (55.6–69.8)	173	119	68.8 (61.5–75.2)
COVID-19 ECO IGM/IGG (Eco Diagnostica)	173	125	72.3 (65.2–78.4)	173	114	65.9 (58.6–72.6)	173	130	75.1 (68.2–80.9)
COVID-19 IgG/IgM (Qingdao Hightop Biotech)	173	39	22.5 (16.9–29.3)	173	89	51.4 (44.0–58.7)	173	91	52.6 (45.2–59.9)
Imuno-Rápido COVID-19 IgG/IgM (Wama Produtos Para Laboratorio)	173	70	40.5 (33.5–47.9)	173	108	62.4 (54.9–69.3)	173	119	68.8 (61.6–75.2)
COVID-19 IgG IgM (Gold Analisa Diagnóstica)	173	23	13.3 (9.1–19.2)	173	98	56.6 (49.2–63.8)	173	100	57.8 (50.4–64.9)

Table 3.

Sensitivity by ELISA test and immunoglobulin class detected.

Test	Antibody detected	n	Positive	Sensitivity (95% CI)
Covid-19 ELISA IgA + gM (Vircell Microbiologists)	IgA/IgM	173	156	90.2 (84.9–93.8)
Covid-19 ELISA IgG (Vircell Microbiologists)	IgG	173	133	76.8 (70.0–82.5)
Anti-SARS-CoV-2 ELISA IgA Euroimmun	IgA	109	83	76.1 (67.1–83.1)
Anti-SARS-CoV-2 ELISA IgG Euroimmun	IgG	109	64	58.7 (49.3–67.5)
Allserum EIA COVID-19 IgM (Mbiolog)	IgM	166	76	45.8 (38.4–53.4)
Allserum EIA COVID-19 IgG (Mbiolog)	IgG	166	100	60.2 (52.6–67.3)

Table 4.

Sensitivity 7 days after symptom onset, specificity, and overall accuracy by test.

Test	Method	Antibody detected	Sensitivity (95% CI)	Specificity (95% CI)	Accuracy (95% CI)
One Step COVID-2019 Test (Guangzhou Wondfo Biotech)	LFIA	Total IgM/IgG	79.7 (72.3–85.9)	100 (96.8–100)	88.6 (84.2–92.2)
TR DPP® COVID-19 IGM/IGG Bio-Manguinhos (Fundação Oswaldo Cruz)	LFIA	IgG + IgM	73.6 (65.9–80.0)	81.0 (72.9–87.1)	76.9 (71.5–81.6)
COVID-19 ECO IGM/IGG (Eco Diagnostica)	LFIA	IgG + IgM	83.1 (76.2–88.3)	99.1 (95.2–99.8)	90.1 (85.9–93.1)
COVID-19 IgG/IgM (Qingdao Hightop Biotech)	LFIA	IgG + IgM	59.5 (51.4–67.1)	100 (96.8–100)	77.3 (71.9–81.9)
Imuno-Rápido COVID-19 IgG/IgM (Wama Produtos Para Laboratorio)	LFIA	IgG + IgM	75.0 (67.5–81.3)	97.4 (92.6–99.1)	84.8 (79.9–88.7)
COVID-19 IgG IgM (Gold Analisa Diagnóstica)	LFIA	IgG + IgM	64.9 (56.9–72.1)	98.3 (93.9–99.6)	79.5 (74.2–84.2)
Covid-19 ELISA IgA/IgM (Vircell Microbiologists)	ELISA	IgA	92.6 (87.2–95.8)	23.3 (16.5–31.8)	62.1 (56.1–67.70
Covid-19 ELISA IgG (Vircell Microbiologists)	ELISA	IgG	83.8 (82.7–92.9)	53.4 (47.4–59.3)	70.4 (64.6–75.6)
Anti-SARS-CoV-2 ELISA IgA (Euroimmun)	ELISA	IgA	82.9 (74.0–89.2)	82.2 (72.7–89.8)	82.6 (76.2–87.6)
Anti-SARS-CoV-2 ELISA IgG (Euroimmun)	ELISA	IgG	67.0 (56.9–75.7)	95.8 (88.5–98.5)	79.6 (72.8–85.0)
Allserum EIA COVID-19 IgM (Mbiolog)	ELISA	IgM	50.7 (42.6–58.8)	70.4 (61.2–78.2)	59.2 (53.0–65.2)
Allserum EIA COVID-19 IgG (Mbiolog)	ELISA	IgG	66.9 (58.8–74.1)	98.1 (93.4–99.5)	80.4 (75.0–84.8)

Table 5.

Sensitivity by serology test according to time from onset of symptoms.

	< 7 days Positive/total	7–14 days Positive/total	> 14 days Positive/total	Total Positive/total
	% (95% CI)	% (95% CI)	% (95% CI)	% (95% CI)
One Step COVID-2019 Test (Guangzhou Wondfo Biotech)
IgM + IgG	6/25	56/74	62/74	124/173
	24.0 (11.5–43.4)	75.7 (64.8–84.0)	83.7 (73.7–90.4)	71.7 (64.3–78.2)
TR DPP® COVID-19 IGM/IGG Bio-Manguinhos (Fundação Oswaldo Cruz)
IgM	7/25	38/74	46/74	91/173
	28 (14.3–47.6)	51.4 (40.2–62.4)	62.2 (50.8–72.4)	52.6 (45.2–59.9)
IgG	6/25	45/74	58/74	109/173
	24 (11.5–43.4)	60.8(49.4–71.1)	78.4 (67.7–86.2)	63.0 (55.6–69.8)
IgM + IgG	10/25	48/74	61/74	119/173
	40 (21.9–61.3)	64.9 (53.5–74.9)	82.4 (72.2–89.4)	68.8(61.5–75.2)
COVID-19 ECO IGM/IGG Teste (Eco Diagnostica)
IgM	7/25	57/74	61/74	125/173
	28 (14.3–47.6)	77 (63.0–82.6)	82.4 (72.2–89.4)	72.3 (65.2–78.4)
IgG	5/25	44/74	65/74	114/173
	20 (8.9–39.1)	59.5 (48.1–69.9)	87.8 (78.4–93.4)	65.9 (58.6–72.6)
IgM + IgG	7/25	57/74	66/74	130/173
	28 (14.3–47.6)	77 (66.2–85.1)	89.2 (80.1–94.4)	75.1 (68.2–80.9)
COVID-19 IgG/IgM (Qingdao Hightop Biotech)
IgM	2/25	16/74	21/74	39/173
	8 (2.2–24.9)	21.6 (13.7–32.2)	28.4 (19.4–39.5)	22.5 (16.9–29.3)
IgG	2/25	36/74	51/74	89/173
	8 (2.2–24.9)	48.6 (37.6–59.8)	68.9 (57.6–78.3)	51.4 (44.0–58.7)
IgM + IgG	3/25	37/74	51/74	91/173
	12 (4.2–29.9)	50 (38.9–61.1)	68.9 (57.6–78.3)	52.6 (45.2–59.9)
Imuno-Rápido COVID-19 IgG/IgM (Wama Produtos Para Laboratorio)
IgM	5/25	32/74	33/74	70/173
	20 (8.7–39.1)	43.2 (32.5–54.5)	44.6 (33.8–55.9)	40.5 (33.5–47.9)
IgG	6/25	40/74	62/74	108/173
	24 (11.5–43.4)	54.1 (42.9–64.9)	83.8 (73.8–90.5)	62.4 (54.9–69.3)
IgM + IgG	8/25	47/74	64/74	119/173
	32 (17.2–51.6)	63.5 (52.1–73.6)	86.5 (76.9–92.5)	68.8 (61.6–75.2)
COVID-19 IgG IgM (Gold Analisa Diagnóstica)
IgM	0/25	14/74	9/74	23/173
	0 (0–13.3)	18.9 (11.6–29.3)	12.2 (6.6–21.6)	13.3 (9.1–19.2)
IgG	4/25	39/74	55/74	98/173
	16.0 (6.4–34.6)	52.7 (42.9–64.9)	74.3 (62.9–83.8)	56.6 (49.2–63.8)
IgM + IgG	4/25	41/74	55/74	100/173
	16.0 (6.4–34.7)	55.4 (44.1–66.2)	74.3 (63.3–82.9)	57.8 (50.4–64.9)
Covid-19 ELISA IgA/IgM (Vircell Microbiologists)
IgA + IgM	19/25	66/74	71/74	156/173
	76.0 (56.6–88.5)	89.2 (80.1–94.4)	95.9 (88.7–98.6)	90.2 (84.9–93.8)
Covid-19 ELISA IgG (Vircell Microbiologists)
IgG	9/25	55/74	69/74	133/173
	36.0 (20.3–55.5)	74.3 (63.3–82.9)	93.2 (85.1–97.1)	76.8 (70.0–82.5)
Anti-SARS-CoV-2 ELISA IgA (Euroimmun)
IgA	5/15	36/47	42/47	83/109
	33.3 (15.1–58.3)	76.6 (62.8–86.4)	89.4 (77.4–95.4)	76.1 (67.3–83.1)
Anti-SARS-CoV-2 ELISA IgG (Euroimmun)
IgG	1/15	22/47	41/47	64/109
	6.7 (1.2–29.8)	46.8 (33.3–60.8)	87.2 (74.8–93.9)	58.7 (49.3–67.5)
Allserum EIA COVID-19 IgM (Mbiolog)
IgM	4/24	36/71	36/71	76/166
	16.7 (6.7–35.9)	50.7 (39.4–62.0)	50.7 (39.4–62.0)	45.8 (38.4–53.4)
Allserum EIA COVID-19 IgG (Mbiolog)
IgG	5/24	39/71	56/71	100/166
	20.8 (9.2–40.4)	54.9 (43.4–65.9)	78.9 (68.1–86.8)	60.2 (52.6–67.3)

For all LFIA tests and three of the six ELISAs, the sensitivity was higher among patients presenting with ARDS than among those presenting with mild symptoms (Table 6 ).

Table 6.

Sensitivity by serology test according to clinical severity.

	ARDS present	ARDS absent	p-value
LFIA
One Step COVID-2019 Test (Guangzhou Wondfo Biotech)	78.6 (69.5–86.1)	61.4 (49.0–72.3)	0.02
TR DPP® COVID-19 IGM/IGG Bio-Manguinhos (Fundação Oswaldo Cruz)	75.7 (66.6–82.9)	58.6 (46.9–69.4)	0.03
COVID-19 ECO IGM/IGG (Eco Diagnostica)	82.5 (73.8–89.3)	64.3 (52.6–74.5)	0.01
COVID-19 IgG/IgM (Qingdao Hightop Biotech)	63.1 (53.5–71.8)	37.1 (26.7–48.8)	0.00
Imuno-Rápido COVID-19 IgG/IgM (Wama Produtos Para Laboratorio)	77.7 (68.7–84.7)	55.7 (44.1–66.7)	0.00
COVID-19 IgG IgM (Gold Analisa Diagnóstica)	66.0 (56.4–74.4)	45.7 (34.6–57.3)	0.01
ELISA
Covid-19 ELISA IgA/IgM (Vircell Microbiologists)	90.3 (83.1–94.6)	90.0 (80.7–95.1)	0.95
Covid-19 ELISA IgG (Vircell Microbiologists)	83.5 (75.2–89.4)	67.1 (55.5–76.9)	0.02
Anti-SARS-CoV-2 ELISA IgA (Euroimmun)	81.8 (69.6–89.9)	70.4 (57.2–80.9)	0.24
Anti-SARS-CoV-2 ELISA IgG (Euroimmun)	67.3 (54.1–78.2)	50.0 (37.1–62.9)	0.10
Allserum EIA COVID-19 IgM (MBiolog)	55.1 (44.7–65.2)	32.4 (21.5–44.8)	0.00
Allserum EIA COVID-19 IgG (Mbiolog)	68.4 (58.2–77.4)	48.5 (36.2–61.0)	0.01

ARDS: acute respiratory distress syndrome.

Except for one LFIA, which presented an exceptional low specificity (81%, 95% CI 72.9–87.1%), the specificity of all other LFIAs was generally high, varying between 97.4% and 100% (Table 4). Overall, sensitivity for ELISAs was higher than for LFIAs. Excluding one IgG-based ELISA test, which presented a sensitivity rate of 58.7%, the rates varied from 66.9% to 92.6%. In contrast, the specificity for ELISA tests was in general lower than that for LFIA, except for the same test presenting the lowest sensitivity referred to above, which exhibited the highest specificity among the ELISA tests (95.8%). Agreement between the results of LFIAs performed on digital blood and serum varied markedly among different commercial kits — from perfect to only slight agreement (Table 7 ). Among the 116 control sera, only 21 did not show reactivity to any of the 12 tests evaluated, 53 were positive in one test, 27 in two tests, 10 reacted positively in three tests and five controls showed a positive reaction in four different tests. The patients in the control group whose samples reacted falsely in more than three different SARS-Cov-2 serological tests had serum markers to HIV, dengue, zica, Chagas disease, syphilis, or toxoplasmosis, or parasitological confirmation of visceral leishmaniasis or malaria.

Table 7.

Agreement between results of tests performed on serum and capillary blood among confirmed SARS-Cov-2 cases.

Results of tests performed on digital blood		Results of test performed on serum		Total	Kappa
		Negative	Positive
One Step COVID- 2019 Test	Negative	2	3	5	0.50
	Positive	0	15	15
Total		2	18	20
TR DPP® COVID-19 IGM/IGG Bio-Manguinhos	Negative	1	1	2	0.24
	Positive	3	16	19
Total		4	17	21
COVID-19 ECO IGM/IGG Teste	Negative	2	0	2	1.0
	Positive	0	26	26
Total		2	26	28

Discussion

This is the first study to assess comparatively the clinical performance of serological tests available in Brazil for diagnosing SARS-CoV-2. Although some systematic reviews have already been published on the subject (Deeks et al., 2020, Lisboa Bastos et al., 2020), none has included data from Brazil, the current epicenter of the COVID-19 pandemic in Latin America. Local accuracy data based on real scenarios are essential given the marked regional differences in the performances of the tests. In the case of SARS-CoV-2 serological tests, this information is especially relevant to the current reality of Brazil, a developing country that faces serious budgetary constraints and that has been performing sub-optimally in relation to its mass testing capacity. In contrast, several successful strategies implemented worldwide, such as aggressive testing and isolation, have promoted transmission control (Lee et al., 2020c, Song et al., 2020). In this sense, the inverse association between testing capacity and mortality from COVID-19 has provided firm evidence of the impact of isolating those infected, and the tracing and quarantining of their contacts (Max Roser HR, Esteban Ortiz-Ospina and, 2020). The role of diagnostic testing and its impact on community transmission are dependent on the types of test available and on the logistical arrangements. RT-PCR-based assays performed on respiratory specimens remain the gold standard for COVID-19 diagnosis. However, it is a time-consuming method limited by several practical issues, including relatively invasive sampling and the need for specialist operators and certified laboratories, making its use particularly challenging in resource-limited settings. Additionally, the test offers a narrow window of diagnostic opportunity, typically between the 4th and 6th day of symptom onset (To et al., 2020), coinciding with the peak viral load in the upper respiratory tract (Kucirka et al., 2020), further restricting its suitability for mass use. In this context, serological immunoassays have been proposed as an alternative diagnostic tool for use during the acute and symptomatic phases. To date, many commercial companies have developed serological assays for detecting SARS-CoV-2. These assays are mainly directed at two immunogenic targets: S protein, which is the most exposed viral protein, or its receptor-binding domain (RBD), or N protein, which is abundantly expressed (Lee et al., 2020d). Despite the large number of tests approved for commercialization after a quick evaluation process in many countries, some health authorities have recommended caution in the implementation of serological tests, which should be done after a national validation of their performanc (Food and Drug Administration (FDA), 2020, United Kingdom (UK), 2020).

In general, when there is a second confirmatory test, sensitivity is the most desired parameter for screening. In the Covid-19 pandemic scenario, both sensitivity and specificity are important parameters for screening. Contrary to this rationale, our results confirm that, serological tests should not be used up to the 6th day of symptoms because of their extremely low sensitivity. This aspect limits the application of serology testing at the stage with the greatest expected viral excretion and, consequently, with the greatest risk of disease transmission. However, later serology testing has also been considered to diagnose symptomatic cases not detected by RT-PCR, or for those who did not have access to this test ​(Liu et al., 2020). False-negative RT-PCR results would be expected — to an extent difficult to estimate (Kucirka et al., 2020) — relating to flaws in the swab collection process, conditions of storage/transport of the sample, variations in the viral load and excretion, and the time of infection when the collection was performed. For this application — as a complementary test several days after the onset of symptoms — it is important to know the performance of serological tests in different phases of the disease. Although the sensitivities of all serological tests tend to increase with the number of days of symptoms and with the clinical severity, based on the results presented here, the highest sensitivity rate observed after 14 days of symptoms reaches only a moderate level — just over 90% for a few tests in the best scenario. Thus, in regions with SARS-CoV-2 prevalence below 10%, a reality in many regions of the world, the positive predictive value (PPV) of these tests remain below 80%, that is, these tests will produce around 20% false-positive results if they have a very high specificity. Additionally, for tests with specificities lower than 95%, this moderate sensitivity will generate an even lower PPV, even with a disease prevalence above 20%. However, considering that a set of clinical manifestations can be used as disease suspicion criteria, increasing the pre-testing diagnosis probability, in theory, immunoassays could play a complementary function to RT-PCR, enhancing COVID-19 detection sensitivity and accuracy (Food and Drug Administration (FDA), 2020) to the required levels.

An important observation presented here is a test performance that was in general lower than that described by others (Nicol et al., 2020, Khomer et al., 2020). The main difference between this and those studies lies in the studied population. In or study, approximately 40% of the cases did not meet the Covid-19 severity criterion adopted in Brazil, while in most of the earlier studies only hospitalized patients were tested. Furthermore, the performances of tests in the two groups, stratified according to the clinical criteria, were significantly different, reinforcing the link between clinical severity and positivity in serological tests. Other factors, such as sample size issues and genetic specificities, could also explain that difference. On the other hand, immunoassays will still produce delayed information in terms of the critical period of viral transmission — a common result among validation studies (Xiao et al., 2020). For a more accurate performance, serological tests should be used beyond 2 weeks after symptom onset to achieve a high diagnosis probability. Unlike PCR-based tests, serology cannot be used to confirm the presence of the SARS-CoV-2 virus, limiting its use in clinical decision making, or as reinforcement in the recommendation of social isolation.

There is even less evidence to support serological testing of asymptomatic individuals, as proposed for the screening of contacts of COVID-19 confirmed cases or in the supposed assessment of protective immunity. The antibody presence and circulating titers may exhibit different features from those observed in symptomatic infection. Assuming a similar performance of serological tests after a silent infection, and considering the performance reported here and the still low prevalence of SARS-CoV-2 in general, except in few hotspots, we can expect many more false positive than true positive results.

With regard to the choice between LFIA and ELISA tests, in addition to performance, logistical issues and total cost involved should be considered. To assist in this decision, cost-effectiveness analysis needs to be conducted, which should guide more appropriate decisions applied to different scenarios in which pre-test probability or disease prevalence are estimated.

The inadequacy of the use of IgA and IgM antibodies as markers of contagiousness needs to be highlighted. Our results revealed that both increase directly with the number of days of symptoms, reaching the highest rate among samples from patients with more than 14 days of symptoms — a period in which SARS-CoV-2 infectivity is considered low (Bullard et al., 2020).

Specificity was homogeneously high for all LFIA tests, except for one test, assembled and packaged in Brazil but imported from a manufacturer based in the USA, where the kit had its FDA (Food and Drug Administration) license revoked in June 2020, along with more than 70 other tests, because of poor performances detected by independent analyses (Food and Drug Administration (FDA News), 2020).

Although low specificity was not a problem for most LFIAs, data for ELISAs differed significantly from those described in the package inserts of the tests and in relation to other accuracy studies carried out in China and in countries in the northern hemisphere. Infectious diseases prevalent in the tropics, such as leishmaniasis, arboviruses, and malaria, were for the first time described as causes of false-positive tests for SARS-CoV-2, which raises concern about the use of ELISAs in these regions. On the other hand, there were few samples from patients with acute respiratory symptoms enrolled in the control group, which would be the true control for such validation. Nevertheless, other studies testing samples from acute respiratory patients, including other confirmed endemic coronavirus cases, have revealed similar specificity rates (Deeks et al., 2020, Premkumar et al., 2020, Whitman et al., 2020). Another group of patients possibly prone to cross-reactivity are those with chronic autoimmune diseases, as indicated by some previous observations with SARS-CoV — an association still not confirmed for SARS-CoV-2 (Wang et al., 2004).

High specificity is one of the most important properties required for a test used in epidemiological surveys. Estimation of the extent of the population that has already been infected in the community is essential for understanding the spread of the epidemic and the main characteristics of the virus — its attack rate and lethality — and the impact of the various prevention and control interventions. An understanding of these parameters would also be useful in monitoring the resumption of social and economic activities. A limiting factor, however, is the lack of knowledge of the longevity of these antibodies — an issue that will require studies with longer observation periods. In this panel, only 19 patients were between 31 and 60 days since the onset of symptoms, and another 13 had exceeded 60 days. This small sample of evaluated patients does not allow us to confirm the lack of differences in positivity rates of the tests over time.

Another limiting factor for the use of LFIAs as a point-of-care test is the heterogeneity observed between the results of tests performed using fingerstick whole blood and serum, with the kappa value varying from 0.2 to 1.0. This observation suggests caution regarding the possibility of lower performance using blood, which is variable among kits, and commitment to LFIAs, with their significant potential advantages of agility and decentralization in mass testing. To date, few studies have addressed this issue because most have presented the results of tests performed solely on serum. In addition, the sample sizes evaluated were generally small and the results conflicting (Whitman et al., 2020, Li et al., 2020).

Finally, the most controversial point regarding the use of immunoassays is the lack of robust evidence of a correlation between circulating antibodies and acquired immunity (GeurtsvanKessel et al., 2020), that is, whether the antibodies detected are protective for a significant period of time. Thus far, no definitive study has been conducted on the titers of neutralizing antibodies necessary for protection against SARS-CoV-2. However, there are several reports of new detection of the virus through molecular tests among individuals who have again presented signs and symptoms of the disease (Kang et al., 2020), and reports of the occurrence of new outbreaks in regions where the infection would have already reached high levels of exposure in the population. This casts doubt on the ability of viral exposure to produce protective immunity, and, consequently, on the role of immunoassays in determining the immunity of healthcare workers and in supporting the resumption of social and economic activities.

The accumulated experience with SARS-CoV, another coronavirus with strong genetic similarity to the current SARS-CoV-2, and involved in an outbreak in 2002, provides some insights on immunity. Specifically, the presence of antibodies has been shown to extend for at least 3 years, being more intense among patients with the most severe forms, with progressive and significant reduction in neutralizing antibodies over time (Manners et al., 2020). Thus, at this point, the use of serology as a marker of immunity and as a criterion for allowing or preventing the resumption of social life is only speculative, and should not be recommended.

In summary, our observations revealed marked differences among the serological tests registered in Brazil. Generally, sensitivity was only moderate, with insufficient performance when used up to 7 days after symptom onset, as expected for a method based on detecting antibodies. The sensitivity rates reached around 80–90% for LFIAs and ELISAs after 14 days from the onset of symptoms, confirming that immunoassays are not suitable tools for screening SARS-CoV-2 virus infection in the general population, except for regions presenting high prevalence rates of over 20%. Specificity was better for LFIA than ELISA.

These results also confirm the inadequacy of using immunoassays as references in the validation of point-of-care tests. The increase in positivity with the duration of symptoms, even for the acute-phase antibodies IgM and IgA, confirms that immunoassay results cannot be used as an indication of infectivity or as evidence to support quarantine recommendations. Studies addressing the local performance of immunoassay tests should contribute to the rational use of this diagnostic tool in the COVID-19 pandemic context. A lack of information about the antigens used prevents a deeper discussion about the reasons for the performance differences among tests. This information, if it were required by the health regulatory agencies, could contribute to the understanding of the role of various antigens and help accelerate the development of new tests.

Of the three uses recommended so far — as a diagnostic method for acute cases, as a marker of immunity to allow the resumption of social life, and as an instrument for measuring viral dissemination in epidemiological studies — only the latter seems to be justified. Seroprevalence can play an important role in the understanding of COVID-19 spread. However, to estimate the proportion of the population that has already been infected in the community, the estimated prevalence rates need to be adjusted according to test sensitivity and specificity. As an epidemiological tool, seroprevalence could still be helpful in retrospectively assessing the impact of different collective interventions on different demographics. As a diagnostic tool for symptomatic patients, serology represents delayed information, greatly limiting its role in decision making. The inaccuracy in detecting IgM or IgA, as markers of active infectivity, was also confirmed by our results. Further data need to be gathered correlating antibody detection and protective immunity, in addition to the duration of protection.

Conclusion

Many questions remain unanswered regarding the value of serological testing in COVID-19 diagnosis and monitoring. Our findings confirm the poor value of immunoassays for individual diagnosis in the first 7 days from symptom onset, and the relationship between sensitivity and time. Although attractive due to their lower cost and ease of execution, serological tests have the main disadvantage of late positivity during the disease course. The suboptimal performance of available serological tests for COVID-19 and the serum/blood reproducibility inconsistencies for LFIAs raise questions about the usefulness of such methods for medical decision making. Additionally, the detection of SARS-CoV-2 antibodies does not guarantee protection against COVID-19 infection, because there is no confirmation that anti-SARS-CoV-2 antibodies are neutralizing antibodies. More studies addressing the cross-reactivity of SARS-CoV-2 antigens with other infectious diseases should be carried out. Nonetheless, serology assays remain a potential tool for studying the seroepidemiology of COVID-19, and clinically validated serological tests with good performance will be able to provide a more accurate picture of the overall spread of COVID-19.

Funding

This research received no external funding. GC and AR are currently receiving grants [301,384/2019 and 311,641/2009−1] from CNPq (National Counsel of Technological and Scientific Development). The tests used in this validation were kindly provided by the manufactures.

CRediT authorship contribution statement

Gláucia Cota: Conceptualization, Methodology, Data curation, Formal analysis, Writing - original draft, Writing - review & editing. Mariana Lourenço Freire: Conceptualization, Formal analysis, Writing - review & editing, Project administration. Carolina Senra de Souza: Conceptualization, Writing - review & editing, Project administration. Mariana Junqueira Pedras: Writing - review & editing. Juliana Wilke Saliba: Writing - review & editing, Project administration. Verônica Faria: Writing - review & editing, Project administration. Líndicy Leidicy Alves: Formal analysis, Writing - review & editing. Ana Rabello: Writing - review & editing, Supervision. Daniel Moreira Avelar: Conceptualization, Methodology, Formal analysis, Writing - review & editing.

Appendix A. Supplementary data

The following IS Supplementary data to this article: