Validation and performance comparison of two SARS-CoV-2 IgG/IgM rapid tests

Abstract

Severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) causes a disease called COVID-19. COVID-19 is primarily diagnosed using molecular techniques mainly real-time reverse transcriptase PCR. Reliable and accurate serologic assays for COVID-19, are an important tool for surveillance and epidemiologic studies. In this study, the IgG/IgM Rapid Test Cassette and the Prima COVID-19 IgG/IgM Rapid Test for the detection of SARS-CoV-2 antibodies in blood, serum and plasma samples collected from patients up to 48 days after symptom onset in Saudi Arabia were validated. Overall, both tests showed poor performance and cannot be utilised for COVID-19 diagnosis as a point of care test or to determine seroprevalence.

1. Introduction

Several novel viruses have emerged and caused serious diseases in the last two decades – including, severe acute respiratory syndrome coronavirus (SARS-CoV), H1N1 influenza and the Middle East respiratory syndrome coronavirus. Coronavirus disease 2019 (COVID-19) was first recorded in Wuhan, China, on 12th December 2019. On 30th January 2020, the World Health Organization officially announced viral pneumonia as a public health emergency of international concern (Guo et al., 2020a, Guo et al., 2020b). On 11th March 2020, the World Health Organization declared COVID-19 as a pandemic illness caused by SARS-CoV-2.

SARS-CoV-2 is a highly contagious virus and has the same transmission routes as other respiratory infections – mainly by inhalation or contact with saliva and discharges or droplets. Its incubation period is estimated to be 2–14 days. However, further investigation is needed to understand the mode of transmission of the infection, as most of the information about the transmission is derived from similar respiratory coronaviruses (Rothe et al., 2020). COVID-19′s common symptoms are fever, cough, sore throat, breathlessness and fatigue, which are mainly mild. However, in more vulnerable individuals (the elderly with underlining diseases), the infection may advance to pneumonia, acute respiratory distress syndrome and cytokine storm – leading to multiorgan dysfunction and, subsequently, death (Wang et al., 2020).

At the time of writing this article, the latest figures indicate that there are more than 108 million confirmed cases of COVID-19 around the globe and more than two million deaths (World Health Organization, 2020a). In Saudi Arabia, there are more than 372,000 confirmed cases and more than 6000 deaths (World Health Organization, 2020b).

A proper diagnosis of the infection plays a critical role in the treatment and control of the disease, and real-time reverse transcription-polymerase chain reaction (RT-PCR) assays remain the preferred molecular test for the diagnosis of SARS-CoV-2 infection (Chu et al., 2020, Corman et al., 2020, Loeffelholz and Tang, 2020). However, serology‐ based assays are being used as an additional method to RT-PCR.

The development of accurate serologic assays that can profile the antibodies against SARS-CoV–2 to detect the previous infection and acquired immunity will be essential for epidemiologic seroprevalence studies and vaccine assessments and has potential for risk assessment of health-care workers. Several immunoassays are commercially available in some countries – although the diagnostic accuracy and specificity of these kits, which are needed to define the optimal use of these kits, are undefined (World Health Organization, 2020c).

Up to date, COVID-19 infection rate data mostly cover acute clinical infections with respiratory symptoms, while the number of subclinical infections and atypical patients remains undetermined (Lerner et al., 2020). Therefore, serological COVID-19 detection will help in the profiling of the COVID-19 infection spectrum. Seroprevalence surveys would help in estimating the entire infection number in the population, improve estimation of the number of SARS-CoV-2 infected individuals and describe the entire disease spectrum of COVID‐19.

This study aimed to compare and evaluate the performance of two commercially available serological assays for the detection of SARS-Cov-2 antibodies: Prima COVID-19 IgG/IgM Rapid Test (PRIMA Lab SA, Balerna, Switzerland) and Innovita 2019-nCoV Ab Test (Innovita Biological Technology, China). Both assays were tested against the same panel of serology samples from patients with confirmed COVID-19 and a group of negative control samples.

2. Material and methods

2.1. Patients and sample collection

To investigate the sensitivity and specificity of the COVID-19 IgG/IgM Rapid Test Cassette, blood samples from patients with known RT-PCR COVID-19 positive samples and COVID–19 negative samples result from King Abdullah Bin Abdulaziz University Hospital were used to investigate the validity of the kit. In total, 82 patients who tested for COVID-19 by RT-PCR were recruited based on the diagnostic criteria of a suspected case of COVID-19 and according to the diagnostic and treatment protocol for COVID-19, including epidemiological history and clinical manifestations. Information about age, sex, vital signs, laboratory, coexisting disorder and clinical data on admission were extracted from medical records. Whole blood, plasma, or serum were used to test the IgM and IgG antibodies. The test was performed following the operation method described in the product inserts.

2.2. IgG/IgM rapid test cassette

A qualitative detection and differentiation test of IgM and IgG antibodies against SARS-CoV–2 in whole blood, serum or plasma from individuals suspected of COVID-19 who had COVID-19 positive and negative samples, which was confirmed with RT-PCR method, was done. The RT-PCR was performed in the diagnostic lab at King Abdullah bin Abdulaziz University Hospital upon the admission of the patient. Serum (10 μL), plasma sample or 20 μL whole blood sample were added to the sample well(S) on the cassette. Then we added two drops of buffer (approximately 80 L), started the timer, and then read the results at 10 min, as the results cannot be interrupted after 20 min.

2.3. Data analysis

Statistical analysis was performed with the Statistical Analysis System software (SPSS). The specificity, sensitivity, positive and negative predictive values and accuracy of the rapid test kits were calculated according to the following formulas:

$$\mathrm{S}\mathrm{p}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{f}\mathrm{i}\mathrm{c}\mathrm{i}\mathrm{t}\mathrm{y}(\%)=100\times [\mathrm{t}\mathrm{r}\mathrm{u}\mathrm{e}\mathrm{n}\mathrm{e}\mathrm{g}\mathrm{a}\mathrm{t}\mathrm{i}\mathrm{v}\mathrm{e}/(\mathrm{t}\mathrm{r}\mathrm{u}\mathrm{e}\mathrm{n}\mathrm{e}\mathrm{g}\mathrm{a}\mathrm{t}\mathrm{i}\mathrm{v}\mathrm{e}+\mathrm{f}\mathrm{a}\mathrm{l}\mathrm{s}\mathrm{e}\mathrm{p}\mathrm{o}\mathrm{s}\mathrm{i}\mathrm{t}\mathrm{i}\mathrm{v}\mathrm{e})]$$

$$\mathrm{S}\mathrm{e}\mathrm{n}\mathrm{s}\mathrm{i}\mathrm{t}\mathrm{i}\mathrm{v}\mathrm{i}\mathrm{t}\mathrm{y}(\%)=100\times [\mathrm{t}\mathrm{r}\mathrm{u}\mathrm{e}\mathrm{p}\mathrm{o}\mathrm{s}\mathrm{i}\mathrm{t}\mathrm{i}\mathrm{v}\mathrm{e}/(\mathrm{t}\mathrm{r}\mathrm{u}\mathrm{e}\mathrm{p}\mathrm{o}\mathrm{s}\mathrm{i}\mathrm{t}\mathrm{i}\mathrm{v}\mathrm{e}+\mathrm{f}\mathrm{a}\mathrm{l}\mathrm{s}\mathrm{e}\mathrm{n}\mathrm{e}\mathrm{g}\mathrm{a}\mathrm{t}\mathrm{i}\mathrm{v}\mathrm{e})]$$

$$\mathrm{P}\mathrm{o}\mathrm{s}\mathrm{i}\mathrm{t}\mathrm{i}\mathrm{v}\mathrm{e}\mathrm{p}\mathrm{r}\mathrm{e}\mathrm{d}\mathrm{i}\mathrm{c}\mathrm{t}\mathrm{i}\mathrm{v}\mathrm{e}\mathrm{v}\mathrm{a}\mathrm{l}\mathrm{u}\mathrm{e}\mathrm{s}(\%)=100\times [\mathrm{t}\mathrm{r}\mathrm{u}\mathrm{e}\mathrm{p}\mathrm{o}\mathrm{s}\mathrm{i}\mathrm{t}\mathrm{i}\mathrm{v}\mathrm{e}/(\mathrm{t}\mathrm{r}\mathrm{u}\mathrm{e}\mathrm{p}\mathrm{o}\mathrm{s}\mathrm{i}\mathrm{t}\mathrm{i}\mathrm{v}\mathrm{e}+\mathrm{f}\mathrm{a}\mathrm{l}\mathrm{s}\mathrm{e}\mathrm{p}\mathrm{o}\mathrm{s}\mathrm{i}\mathrm{t}\mathrm{i}\mathrm{v}\mathrm{e})]$$

$$\mathrm{N}\mathrm{e}\mathrm{g}\mathrm{a}\mathrm{t}\mathrm{i}\mathrm{v}\mathrm{e}\mathrm{p}\mathrm{r}\mathrm{e}\mathrm{d}\mathrm{i}\mathrm{c}\mathrm{t}\mathrm{i}\mathrm{v}\mathrm{e}\mathrm{v}\mathrm{a}\mathrm{l}\mathrm{u}\mathrm{e}\mathrm{s}(\%)=100\times [\mathrm{t}\mathrm{r}\mathrm{u}\mathrm{e}\mathrm{n}\mathrm{e}\mathrm{g}\mathrm{a}\mathrm{t}\mathrm{i}\mathrm{v}\mathrm{e}/(\mathrm{t}\mathrm{r}\mathrm{u}\mathrm{e}\mathrm{n}\mathrm{e}\mathrm{g}\mathrm{a}\mathrm{t}\mathrm{i}\mathrm{v}\mathrm{e}+\mathrm{f}\mathrm{a}\mathrm{l}\mathrm{s}\mathrm{e}\mathrm{n}\mathrm{e}\mathrm{g}\mathrm{a}\mathrm{t}\mathrm{i}\mathrm{v}\mathrm{e})]$$

$$\mathrm{A}\mathrm{c}\mathrm{c}\mathrm{u}\mathrm{r}\mathrm{a}\mathrm{c}\mathrm{y}(\%)=100\times [(\mathrm{t}\mathrm{r}\mathrm{u}\mathrm{e}\mathrm{p}\mathrm{o}\mathrm{s}\mathrm{i}\mathrm{t}\mathrm{i}\mathrm{v}\mathrm{e}+\mathrm{t}\mathrm{r}\mathrm{u}\mathrm{e}\mathrm{n}\mathrm{e}\mathrm{g}\mathrm{a}\mathrm{t}\mathrm{i}\mathrm{v}\mathrm{e})/\mathrm{t}\mathrm{o}\mathrm{t}\mathrm{a}\mathrm{l}\mathrm{s}\mathrm{a}\mathrm{m}\mathrm{p}\mathrm{l}\mathrm{e}\mathrm{s}]$$

3. Results

This study aimed to investigate the performance of two commercially available kits that are used to detect COVID-19 antibodies in Saudi Arabia – the Prima kit and the Innovita kit. Both kits are lateral flow rapid test strips that can be used for a point-of-care test (POCT). To calculate the statistical measures of performance, we tested these two kits against the standard method in the current time: RT-PCR for the virus RNA.

A total of 82 serum samples were collected from May to July 2020. The samples were collected from 46 patients with COVID-19, which was confirmed by RT-PCR tests. Thirty-nine samples were collected from patients with respiratory symptoms who tested negative for SARS-CoV-2 after a RT-PCR test. The mean age of the patients was 45 years (range: 21–85 years), and 53% were females (Table 1). All the patients with RT-PCR positive tests were hospitalised. The dates of symptom onset to date of hospitalisation ranged from 0 to 19 days with a mean of 7 days post symptom onset. The mean duration of hospitalisation was 8 days (range 1–90 days).

Table 1.

Demographic data.

Variable	Value
Age (Years)
mean	45
median	40
range	21–85
Sex
Female (n (%))	40(53%)
Male (n (%))	35(47%)
Duration of hospitalisation (Days)
mean	12
median	8
range	1–90

3.1. Performance characteristics of the point-of-care tests

Of the 82 samples, 46 tested positive for SARS-CoV-2, while 36 tested negative following the RT-PCR test. To evaluate the kit performance, the results were identified as true positive, true negative, false positive and false negative based on the duration between onset of the symptoms and day the sample was taken from the patient and the PCR result.

The performance of the Prima kit POCTs was poor with sensitivities ranging from 37.5% to 77.7%. Predicted negative values and accuracy were also poor. The specificity predicted positive values were good but below the acceptable value (95%). The Innovita kit also had poor sensitivity (41–69%), while the predicted negative values, accuracy, specificity, and predicted positive values were all sufficient but below the acceptable value (95%). Comparing the two kits, the Innovita kit had a higher sensitivity, whereas the Prima kit had a better performance in predicting negative values, specificity and positive values. Both kits had a similar accuracy result (Table 2). Overall, both kits have poor performance and are not recommended for use in clinical or epidemiological studies.

Table 2.

Performance of two SARS-CoV-2 lateral flow assays (POCTs) compared with RT-PCR.

		(−ve) (TN)	(+ve) (FP)	(+ve) (TP)	(−ve) (FN)	Specificity%	Sensitivity%	PPV%	NPV%	Accuracy %
Prima	IgM	32	3	21	6	91.42857	77.77778	87.5	84.21053	85.48387
	IgG	30	5	19	19	85.71429	50	79.16667	61.22449	67.12329
Overall	IgM/IgG	34	1	15	25	97.14286	37.5	93.75	57.62712	65.33333
Innovita	IgM	30	5	18	8	85.71429	69.23077	78.26087	78.94737	78.68852
	IgG	30	5	24	14	85.71429	63.15789	82.75862	68.18182	73.9726
Overall	IgM/IgG	30	5	16	23	85.71429	41.02564	76.19048	56.60377	62.16216

(−ve); negative, (+ve); positive, (POCTs); Point of care test, (RT-PCR); Reverse transcription polymerase chain reaction, (TN); True negative, (FP); False positive, (TP); True Positive, (FN); False negative, (PPV); Positive predictive value, (NPV) Negative Predictive value.

By looking at the performance of the kit at different duration after the onset of the symptoms, the results indicate that the performance of the kit improved after 16 days post-onset of the symptoms. The sensitivity of the kits was 12–44% for samples collected 7–16 days after the onset of the symptoms and 64–93% for the samples collected between 16 and 48 days after the onset of the symptoms (Table 3). However, the performance of the two kits was below the acceptable range (<95%). In terms of the duration of more than 48 days, there was only one available sample; therefore, this result is not representative.

Table 3.

The performance of the two kits during different durations (days) after onset of the symptoms.

Number of days		7–16			16–48
Number of samples		(n = 31)			(n = 14)
		(+ve) (TP)	(−ve) (FN)	Sensitivity	(+ve) (TP)	(−ve) (FN)	Sensitivity
prima	IgM	10	21	32.25	9	5	64.3
	IgG	8	10	44.44	13	1	93
	IgM/IgG	6	25	19.35	9	5	64.2
Innovita	IgM	12	19	38.70	12	2	85.7
	IgG	5	12	29.41	13	1	93
	IgM/IgG	4	27	12.90	12	2	85.71

4. Discussion

Few weeks after the outbreak of COVID-19, several manufacturers began promoting POCT for detecting SARS-CoV-2 antibodies. But, to date, only a few studies have been conducted to check the viability and validity of these commercial kits in the management of the COVID-19 outbreak. Therefore, this study aimed to evaluate the performance of the two diagnostic kits by comparing their result to RT-PCR. The results showed that both kits have poor sensitivity. Prima kit showed good specificity for the detection of IgM but the overall specificity was below the required level. Similarly, the Innovita kit showed poor specificity. Thus, both kits are not appropriate for diagnostic or research use.

The accuracy and significance of the serological test is dependent on the detection day, to ensure the detection of IgM and IgG after their onset. The SARS-CoV host antibodies are detectable after several days to weeks. A study investigated the median time from symptom onset to antibody detection by using an enzyme-linked immunosorbent assay that reacts with antibodies against the receptor-binding domain of the spike protein. The result showed that IgM was detected by day 12, while IgG was detected by day 14 after the onset of the symptoms. By day 15, the detection rate of IgM and IgG were 94% and 80%, respectively (Zhao et al., 2020). The results of the current study are consistent with the findings of other studies (Guo et al., 2020a, Guo et al., 2020b, Qu et al., 2020).

Therefore, serologic tests may be able to identify some patients with current infection (particularly those who present late in the course of illness), but they are less likely to be reactive in the first several days to weeks of infection. Thus, it may have less utility for diagnosis in the acute setting (Guo et al., 2020a, Guo et al., 2020b, Zhao et al., 2020).

On the other hand, a systemic meta-analysis that assessed the performance of serological diagnosis of COVID-19 revealed that using an enzyme-linked immunosorbent assay had better performance – that is, higher sensitivity and specificity than POCT (Lisboa Bastos et al., 2020). In addition, automated chemiluminescent immunoassay has shown greater sensitivity and specificity and is now used as a golden standard for COVID-19 serological detection (Noce et al., 2020). Also, the accuracy and the rate of cross-reactivity with other respiratory viruses, especially other coronaviruses, is a potential concern, as cross-reactivity with these viruses has been reported, although the overall specify was in the acceptable range (Charlton et al., 2020).

The sensitivity and the specificity of POCT kits vary between different manufacturers (Lassaunière et al., 2020, Charlton et al., 2020). Some studies have found that POCT were not significantly accurate – for example, the UK and Spain authorities have rejected the use of these POCT for seroprevalence. The American Society for Microbiology, the Public Health Agency of Canada and the WHO have also published recommendations against using serology testing for the diagnosis of acute infection (Patel et al., 2020). The Saudi center of disease prevention and control (SCDC) that laboratories in order to provide diagnostic testing for COVI-19, the laboratory should use nucleic acid based detection system. Furthermore, Cassaniti et al. (2020) found that the sensitivity of one of these rapid tests was <20%, and these results align with findings reported elsewhere (Döhla et al., 2020), indicating that some rapid tests have been commercialised without significant clinical and analytical validation. Therefore, it is recommended that all commercial serological rapid tests be appropriately validated clinically before routine clinical usage. However, these kits can be used to support PCR results but should not replace them (Lippi et al., 2020).

Our results showed an increase in the sensitivity of the rapid test kits after 16 days. This finding has been observed in some studies (Lassaunière et al., 2020, Pan et al., 2020, Riccò et al., 2020). Charlton et al. described the performance of serological assays as unreliable before 14 days; however, improves over time, and the antibodies can be detected up to day 45 (Charlton et al., 2020).

However, this delay in detecting antibodies, especially IgM, limits the use of the kit for diagnosis purposes, as the majority of patients with SARS-CoV-2 are asymptomatic. Thus, the symptoms of patients with mild infections may be undiagnosed and their onset cannot be estimated, or patients may be tested in an early stage of the SARS-CoV-2 infection.

5. Limitations of the study

The limitation of this study was the inability to obtain the samples during the IgG and IgM onset, although this did not apply to some samples. Patients that tested positive for COVID-19 were hospitalised; therefore, patients with mild symptoms or asymptotic were not included in this analysis.

Studies have shown that the timeline for the detection of immune-response antibodies (serological diagnosis) is very critical to give accurate serological test results. The antibodies can be detected eight days after the symptoms appear (Lerner et al., 2020). In this study, the timing of the diagnosis was one of the most important limiting factors because it was difficult to get the samples at the appropriate time due to the hospital restricted protocol. The blood samples were collected at the time of visiting, which ranged from 5 to 8 days after the symptoms appear. In some complicated cases, the patients were admitted and another blood samples were collected before they were discharged from the hospital. Inappropriate diagnostic timing may lead to false-negative results, especially at an early stage of infection. In our study the number of samples available for follow up after 16 days were decreased to <15 samples. In covid 19 a significant point is how long antibodies persist in the blood, therefore bigger number of follow up sample will be helpful to clarify this point. Another limitation of our study is that all the patients that were positive for COVID-19 were hospitalised; therefore, patients with mild symptoms or asymptotic were not included in this analysis. As shown in other studies, patients with severe clinical symptoms had higher IgG antibody levels than patients with mild symptoms (Okba et al., 2020, Klein et al., 2020, Yongchen et al., 2020).

6. Conclusion

Here, we report a serology panel consisting of serum from patients tested positive for COVID-19 and those tested negative. We compared the performance of two different commercial POCT with the same serum panel to give an accurate comparison across all platforms. Our results showed that POCT antibody detection for COVID-19 should not be used for the diagnosis of acute infections nor for serosurveys to facilitate estimation of seroprevalence in a population and identify previous exposure to the virus. Also, it is not recommended for research use.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.