Performance of five rapid serological tests in mild-diseased subjects using finger prick blood for exposure assessment to SARS-CoV-2

Abstract

Objectives

Assess the performance of five SARS-CoV-2 rapid serological tests (RST) using finger prick (FP) blood on-site to evaluate their usability for exposure assessment in population-based seroprevalence studies.

Study design

Since cross-reactivity with common cold human coronaviruses occurs, serological testing includes a risk of false-positive results. Therefore, the selected cohort for RST-validation was based on combined immunoassay (presence of specific antibodies) and RT-qPCR (presence of SARS-CoV-2) data. RST-performance for FP blood and serum was assessed by performing each RST in two groups, namely SARSCoV- 2 positive (n=108) and negative healthcare workers (n=89). Differences in accuracy and positive and negative predictive values (PPV, NPV) were calculated for a range (1-50%) of SARS-CoV-2 prevalence estimates.

Results

The OrientGene showed overall acceptable performance, with sensitivities of 94.4% and 100%, and specificities of 96.6% and 94.4%, using FP blood and serum, respectively. Although three RST reach optimal specificities (100%), the OrientGene clearly outperforms in sensitivity. At a SARS-CoV-2 prevalence rate of 40%, this RST outperforms the other tests in NPV (96.3%) and reaches comparable PPV (94.9%). Although the specificity of the Covid-Presto is excellent when using FP blood or serum (100% and 97.8%, respectively), its sensitivity decreases when using FP blood (76.9%) compared to serum (98.1%).

Conclusions

Performances of the evaluated RST differ largely. Only one out of five RST (OrientGene) had acceptable sensitivity and specificity using FP blood. Therefore, the latter could be used for seroprevalence studies in a high-prevalence situation. The OrientGene, which measures anti-RBD antibodies, can be valuable after vaccination as well.

1. Background

Since the beginning of the COVID-19 pandemic, caused by SARS-CoV-2, many RST became available on the market, though often poorly validated [1]. These easy-to-use devices are designed for fast (10–15 min) non-quantitative detection of anti-SARS-CoV-2 antibodies and are mainly based on lateral flow chromatography. By targeting IgM and IgG antibodies, these tests are intended to support diagnosis of a SARS-CoV-2 infection in case of late presentation. Depending on the manufacturer, RST target antibodies against different SARS-CoV-2 antigens or combinations thereof, including the nucleocapsid protein, full spike protein or its S1 subunit, S2 subunit or RBD - receptor binding domain [2].

By detecting SARS-CoV-2-specific antibodies on-site using FP blood, these RST can also be valuable in large-scale population-based seroprevalence studies [3,4]. While the intended use of RST is with FP blood, available evaluations were mainly performed using serum, obtained after intravenous sampling. Since performance characteristics of RST are largely sample type dependent, the validation of RST using FP as matrix is mandatory [5]. Up-to-date, reports comparing the performances of RST using FP blood with serum are limited.

2. Objectives

The objective of the present study was to evaluate the performance of five SARS-CoV-2 RST using FP blood on-site in order to reveal their usability for population-based seroprevalence studies. To this end, RST-performance was assessed using specimens from SARS-CoV-2 positive and negative cases, confirmed by combined RT-qPCR and immunoassay positivity/negativity.

3. Study design

3.1. Study population and design

This study was performed on a previously described cohort of healthcare workers of the Jessa General Hospital Hasselt (Belgium) [6]. This cohort participated in a prospective longitudinal SARS-CoV-2 seroprevalence study that started in May 2020 and was approved by the EC of the Jessa General Hospital Hasselt. For the present study, performed in August 2020, an amendment was EC-approved (B2432020000012). Additional written informed consents were obtained from 252 subjects and their biological material is registered in the Sciensano Biobank BB200019. All participants reported mild COVID-19 symptoms between March 3rd and April 17th 2020, i.e. at the onset of the epidemic. At that time, they were tested for the presence of SARS-CoV-2 by RT-qPCR (standard protocol). Between August 2nd and August 18th 2020, a single occupational healthcare team of the Mensura Occupational Health Service performed five RST on-site (Mensura non-laboratory site, Hasselt) using FP blood. Simultaneously, an intravenous blood sample was taken from all participants for later serum-analyses (ELISA and RST) in the Sciensano laboratories.

3.2. RST

Five RST were selected and their performance characteristics compared (Table 1 ). Selection of RST was based on: (i) performance data reported by manufacturers or obtained from independent studies; (ii) their availability on the market; and (iii) the different (combinations of) SARS-CoV-2 antigens used in the RST. All selected RST were intended for use with serum, plasma and whole blood (intravenous and FP blood). Apart from the Wantai Rapid test, all selected RST differentiate between IgM and IgG antibodies to SARS-CoV-2. In the Wantai Rapid test, one colored line indicates the presence of IgM and/or IgG. Since the objective of the present study was to evaluate the usefulness of these RST for population-based seroprevalence studies, not for individual diagnosis, differentiation between IgM and IgG was not considered. A RST was therefore scored positive if one or both lines (IgM alone, IgG alone or IgM and IgG together) were visible in the correct test zones. Sub-analysis looking at detection of IgG antibodies alone can be found in the supplementary material. Prior to testing on-site, a verification of the selected RST was performed to confirm performance data claimed by the manufacturers (supplementary Table 1). For this, RST were performed in the laboratory, according to manufacturer's instructions, on a set of 8 serum samples containing different levels of anti-SARS-CoV-2 antibodies (none, low, medium, high). For the validation study, five RST were performed on-site on all participants using FP blood, according to manufacturer's instructions. Subsequently, RST were performed using the corresponding serum samples in a laboratory setting. On-site interpretation of test results was done by a single trained co-worker and professional photos were taken of each RST for later confirmation analysis by an independent scientist. This photo-based analysis was implemented to confirm validity of the on-site interpretations. Results of the photo-based analysis are available in supplementary Table 2.

Table 1.

Selected rapid serological tests.

Rapid serological test (RST)		Manufacturer	Batch used	Target antigen(s)
full name	abbreviated name	distributer
		reference
QuickZen COVID-19 IgM/IgG	QuickZen	ZenTech (Belgium)	SLW25-006B SLW25-007A	RBD
		Intermed
		SLW-25
COVID-19 IgG/IgM Rapid Test	OrientGene	Healgen (USA), subs. of OrientGene (China)	2003309	RBD
		Analis
		GCCOV-402a
Wantai SARS-CoV-2 Ab Rapid Test	Wantai Rapid	Wantai (China)	JNB20200304	RBD + S1
		Sanbio
		WJ-2750
COVID-PRESTO® TROD IgG/IgM	Covid-Presto	AAZ (France)	2004227-FR20005 2004227-FR20007	N + S1
		AAZ
		TR-COV-002
COVID-19 IgM/IgG Antibody Rapid Test	Multi-G	Multi-G (Belgium)	COV1452003C	N + S
		Multi-G
		MGS/COV

RBD, receptor binding domain (part of S1); S, full spike protein; S1, subunit 1 of the spike protein; N, nucleocapsid protein.

3.3. ELISA testing

To determine the presence of SARS-CoV-2-specific antibodies in serum samples, two different ELISA's were used according to manufacturer's instructions. The Wantai SARS-CoV-2 Ab ELISA (WS-1096; Beijing Wantai Biological Pharmacy Enterprice Co. Ltd., China), detecting all antibodies (IgG, IgA and IgM) targeting the RBD antigen, and the Euroimmun Anti-SARS-CoV-2 IgG ELISA (EI 2606-9601G; Medizinische Labordiagostika AG, Germany), detecting IgG antibodies targeting the S1 antigen. Based on in house and reported serum-validations, the estimated sensitivity, measured >14 days post onset of symptoms, and specificity for the anti-RBD Ig ELISA (Wantai) are 100% (155/155; 95%CI 97.6-100) and 99.6% (772/775; 95%CI 98.9-99.9), respectively. The estimated sensitivity and specificity of the anti-S1 IgG ELISA (Euroimmun) are 96.0% (71/74; 95%CI 88.8-98.9) and 98.6% (494/501; 95%CI 97.1-99.3), respectively [7], [8], [9], [10], [11], [12], [13].

3.4. Defining positive and negative SARS-CoV-2 cases for validation of RST

Validation of RST was done using well-defined SARS-CoV-2 positive and negative subjects. Since the presence of SARS-CoV-2 was assessed by RT-qPCR at the onset of the epidemic and antibody levels could have waned by the moment of rapid testing (several months after RT-qPCR testing), the presence of SARS-CoV-2 specific antibodies was evaluated at the moment of validation. SARS-CoV-2 positive cases were defined as subjects with RT-qPCR-positivity and ELISA-positivity in both ELISA's. SARS-CoV-2 negative cases (controls) were defined as RT-qPCR negative subjects lacking antibodies in both ELISA's. Performance of RST was thus assessed using positive and negative cases based on combined RT-qPCR and ELISA data. Subjects only positive for RT-qPCR or positive in only one of both ELISA's were not considered.

Of the 252 participants initially included, 15 subjects were excluded from the analyses because (i) photos taken from their RST, for confirmation analysis, showed irregularities, (ii) a RST failed (control line not positive) or (iii) borderline ELISA results. Based on combined RT-qPCR and ELISA positivity/negativity, 197 participants were considered in the final analysis of which were classified as SARS-CoV-2 positive (n=108) or SARS-CoV-2 negative (n=89). Validation of the RST, using FP blood or serum, was assessed by comparing results obtained in both subject groups.

3.5. Data-analysis

For each RST, FP whole blood and serum results were compared between confirmed SARS-CoV-2 positive (n=108) and negative (n=89) cases, defined by RT-qPCR and ELISA. Agreement of interpretation (%), i.e. concordance between measurements using either FP blood and serum was assessed. Performance characteristics that were calculated included: sensitivity, specificity and accuracy. Confidence intervals around these performance characteristics were estimated based upon the Exact binomial confidence limits method [14]. Differences in sensitivity and specificity between RST using FP blood were evaluated with the Exact binomial adapted test for paired proportions. Confidence intervals around these estimated differences in paired proportions were calculated by the Agresti-Min method [15]. Statistical significance was determined at a level of 0.05 (α).

Since post-test probability largely depends on the prevalence of a disease within the population [16], accuracies as well as PPV and NPV were calculated for different prevalence estimates. Prevalence estimates ranging from 1% to 50% were considered, thus covering the most plausible values for the SARS-CoV-2 prevalence in humans.

All analyses were performed with R version 4.0.2 (2020-06-22), using RStudio (version 1.3.1056) and the R-packages epiR and DTComPair.

4. Results

4.1. Performance characteristics of RST using FP blood

Test performances for RST using FP blood versus serum are shown in Table 2 . Only the OrientGene showed a performance exceeding 90% for all parameters, with sensitivities of 94.4% and 100% and specificities of 96.6% and 94.4% when using FP blood and serum, respectively. Although the specificity of the Covid-Presto is excellent when using FP blood or serum (100% and 97.8%, respectively), its sensitivity largely drops when using FP blood (76.9%) instead of serum (98.1%). The Wantai Rapid and Multi-G tests have optimal specificities (100%) using both sample types, but their sensitivities are poor (8.3% and 23.1% using FP blood; 38.9% and 49.1% using serum). The QuickZen shows a good specificity when using serum (91.0%), but performs weaker when used with FP blood (sensitivity 75,0%; specificity 73,0%). Although most RST have good specificities, independent of the sample type used, only the OrientGene performs good in terms of specificity as well as sensitivity when using FP blood. Using serum, both the OrientGene and Covid-Presto tests clearly perform better than the other RST. In terms of accuracy, all RST performed better using serum versus FP blood. Results of the sub-analysis for the RST, looking at detection of IgG antibodies alone, can be found in Supplementary Table 3 and show comparable findings.

Table 2.

Performance of RST assessed in SARS-CoV-2 positive (n=108) and negative (n=89) cases, using FP blood versus serum and IgM/IgG detection.

Test performance >90% (green), 70-90% (orange), <70% (red); 95% CI, 95% confidence interval; RST, rapid serological test; FP, finger prick.

Between-test differences in sensitivity and specificity for RST performed on FP blood and significance levels of these differences are shown in Supplementary Table 4. The percentage agreement between the results obtained with FP blood and serum are shown in Table 3 . The highest overall concordance between results (n=197) was obtained for the OrientGene (95.9%). Considering only SARS-CoV-2 positive cases (n=108), the best agreement (94.4%) was observed for the OrientGene. Considering only true negative cases (n=89), high agreement (97.8%-100%) was observed for all RST, except for the QuickZen.

Table 3.

Outcome agreement for IgM/IgG detection between rapid serological test (RST) results using finger prick (FP) blood versus serum.

Outcome agreement in confirmed positive and negative casesa(N = 197)
	True positive and negative results
RST	Using FP blood (n/N)	Using serum (n/N)	Agreement (%)
QuickZen	146/197	157/197	78.2
OrientGene	188/197	192/197	95.9
Wantai Rapid	98/197	131/197	82.2
Covid-Presto	172/197	193/197	87.3
Multi-G	114/197	142/197	82.7
Outcome agreement in confirmed positive casesa(N = 108)
	True positive results
RST	Using FP blood (n/N)	Using serum (n/N)	Agreement (%)
QuickZen	81/108	76/108	82.4
OrientGene	102/108	108/108	94.4
Wantai Rapid	9/108	42/108	67.6
Covid-Presto	83/108	106/108	78.7
Multi-G	25/108	53/108	68.5
Outcome agreement in confirmed negative casesa(N = 89)
	True negative results
RST	Using FP blood (n/N)	Using serum (n/N)	Agreement (%)
QuickZen	65/89	81/89	73.0
OrientGene	86/89	84/89	97.8
Wantai Rapid	89/89	89/89	100
Covid-Presto	89/89	87/89	97.8
Multi-G	89/89	89/89	100

^aCategory confirmed by combined RT-qPCR and ELISA

4.2. Comparison of PPV, NPV and accuracies over different prevalence rates

Since accuracy, PPV and NPV strongly depend on the prevalence of a disease in the population, these parameters were calculated for prevalence rates ranging from 1% to 50% (Fig. 1 ) for the different RST using FP blood. As can be seen from the graphs, at prevalence rates of 40%, the PPV estimates of the OrientGene (94.9%), Covid-Presto (100%), Wantai Rapid (100%) and Multi-G (100%) are largely similar, but the NPV (96.3%) and accuracy (95.7%) of the OrientGene exceed those of all other RST. In terms of PPV at prevalence rates of 10%, three out of five RST performed optimal (100%) and exceed the performances of the OrientGene (PPV = 75.5%) and the QuickZen (PPV = 23.6%). Apart from the QuickZen, all RST reached NPV between 90.8% and 99.4%, and accuracies between 90.8% and 97.7%.

Fig. 1.

Comparison of PPV, NPV and accuracy over different prevalence rates using finger prick (FP) blood. Positive predictive value (PPV, panel A), negative predictive value (NPA, panel B) and accuracy (panel C) were calculated assuming that the estimated sensitivity and specificity of the five rapid serological tests (RST) are fixed for different prevalence rates. Result for the RST are depicted over different prevalence rates (1-50%) using FP blood. Dashed red line, low prevalence (10%); Dashed blue line, high prevalence (40%).

5. Discussion

The goal of the present study was to assess the applicability of RST using FP blood for SARS-CoV-2 seroprevalence studies. To this end, performance characteristics of five SARS-CoV-2 RST using FP blood were assessed and compared to serum-performances. Using FP blood, a sensitivity and specificity exceeding 90% was only observed for the OrientGene. Although three other RST reach optimal specificities (100%), the OrientGene clearly outperforms in sensitivity (94.4%). At high estimated SARS-CoV-2 prevalence rates (40%), the OrientGene outperforms the other tests in NPV (96.3%) and reaches comparable PPV (94.9%). Using serum, the OrientGene and Covid-Presto reach over 90% for sensitivity and specificity. The best overall agreement between results obtained with serum and FP blood are found for the OrientGene.

Today, most studies reporting validations of RST were performed in laboratory conditions using serum as sample type. Here, results obtained with serum are comparable to these findings [1,[17], [18], [19]]. Discordant results, as reported by several authors [1,[20], [21], [22], [23], [24]], can probably be attributed to differences in study populations used for validation (size, time of sampling since onset of symptoms, disease-characteristics). Whereas most studies use severe-diseased patients [1,[17], [18], [19],22], we used a cohort of SARS-CoV-2 patients with mild disease characteristics. Furthermore, some studies consider IgG or IgM seropositivity separately. Here we considered IgG and/or IgM seropositivity, without discriminating between both.

Only few studies report on the performance of RST using FP blood. Pollán et al. [5] published results of a Spanish nationwide survey (n=51.958) using the OrientGene with FP blood testing. These authors report a sensitivity of 79.6% and specificity of 98.3% for IgG detection, compared to immunoassay results. We report a comparable specificity (96.6%), but higher sensitivity (94.4%). This can probably be attributed to differences in selected methods (IgG- vs. IgG/IgM-positivity; immunoassay vs. RT-qPCR/immunoassay) or study populations (disease-status unknown vs. mild-diseased) used for comparison. Two smaller studies using the OrientGene report similar results as found in the present study [25,26]. Using leftover whole blood samples (n=91), Andrey et al. [25] reported a sensitivity of 92% and a specificity of 100% for IgG detection, compared to immunoassay results. Hoffman et al. [26] found a sensitivity of 93.1% and a specificity of 99.2% for IgG detection (n=153), using capillary blood as well as serum and comparing to RT-qPCR only. Prazuck et al. [27] reported on their findings using the Covid-Presto with FP blood (n=381), comparing to RT-qPCR results. These authors observed a sensitivity of 68% and a specificity of 100%, comparable to our results. Of note, when considering only symptomatic patients, sampled more than 15 days since onset of symptoms (n=48), sensitivity of the Covid-Presto increased to 100% [27]. To the best of our knowledge, no studies are available reporting the performance of the QuickZen, Wantai Rapid and Multi-G tests using FP blood. The lower performances for these tests, observed in the present study, can thus not be compared with previous findings. Till now, only Flower et al. [4] reported on FP blood versus serum using a population of SARS-CoV-2 diagnosed non-hospitalized individuals (n=276). The authors reported that concordance between results using FP blood and corresponding serum samples varied depending on the RST used [4], as is demonstrated in here.

The strength of the present study is the use of well-defined SARS-CoV-2 positive and negative subjects. For validation, only subjects positive by RT-qPCR and having antibodies, and subjects negative by RT-qPCR and lacking antibodies were included. All subjects were tested for the presence of SARS-CoV-2 by RT-qPCR in the beginning of the epidemic. Since the present study was conducted several months later and antibodies could have waned by then, anti-RBD Ig and anti-S1 IgG immunoassays were performed at the onset of the study to confirm seropositivity. A second strength of the study is the use of mild-diseased COVID-19 subjects for validation. Whereas in most studies validations are performed on severely diseased hospitalized patients only, the present study was performed in a population more closely resembling the majority of subjects in seroprevalence studies (mild–diseased and asymptomatic). A third strength of the study is the interpretation in terms of relevance for epidemiological studies. Since PPV and NPV depend on the prevalence of a disease in the population, these parameters were calculated for a range of SARS-CoV-2 prevalence estimates.

Currently, the world market is flooded with RST for SARS-CoV-2. Good quality RST can be valuable to support individual late SARS-CoV-2 diagnosis or for monitoring seroprevalence at the population level [28,29]. Although many RST are CE-approved, conform European standards, this conformity label is self-declared by manufacturers and only refers to general quality standards [30]. Manufacturers often claim FP blood performs well in their RST, but concerns have been raised on their reliability and independent performance validations are often lacking [1]. Here, we focused on FP blood testing since easy-to-use devices on-site provide an important advantage for population-based seroprevalence studies. We can conclude that the OrientGene meets all requirements to be valuable in higher prevalence settings. However, in low prevalence settings, a RST with higher specificity is often preferred, accepting a lower sensitivity. For the near future, we assume that RST may become valuable for the monitoring of herd immunity (infection- or vaccine-induced) in the population. In that perspective and according to our sub-analysis, looking at IgG detection alone, use of the OrientGene, which measures anti-RBD antibodies, can be valuable after vaccination as well.

Authors’ contributions

Concept and methodology: ID, LG, KD, AV, MG. Performance using FP blood on-site: NVL, MV, KM, DR. Performance using serum: DT, SDC. Photo-validation: LLR. Project administration and organization of the study: MG, LG, ID, KD, SDC, NVL. Analysis and interpretation of the results: ID, DT, RDP, LG, PP, AV. Visualization: ID, DT, RDP. Writing original draft: DT, ID. Intellectual input and review/editing of the draft: all authors. Funding acquisition: KD. All authors consent with the final manuscript.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.