Comparative evaluation of SARS-CoV-2 serological tests shows significant variability in performance across different years of infection and between the tests

Abstract

Introduction

While the global COVID-19 pandemic is slowly coming under control, current efforts are focused on understanding the epidemiology of endemic SARS-CoV-2. The tool of choice for doing so remains serological tests that detect SARS-CoV-2 induced antibodies. However, the performance of these tests should be evaluated to ensure they comply with the specific performance criteria desired by each country that they are used in.

Methods

Here, we use pre-COVID-19 plasma and plasma from SARS-CoV-2-infected individuals collected in 2020, 2021 and 2022 to evaluate the performance of two commercial Rapid Lateral Flow (RLF) tests (the PANBIO™ COVID-19 IgG/IgM rapid test and the LABNOVATION™ COVID-19 (SARS-CoV-2) IgG/IgM rapid test) and one commercial ELISA test (the PLATELIA™ SARS-CoV-2 total Ab).

Results

We find that whereas the specificity of the two RLF tests is ≥ 95%, it was 91% for the ELISA tests. However, at 14 days post-COVID-19 date of diagnosis (DoD), only the ELISA test constantly achieved a sensitivity of ≥80% over all the three years. In addition, the rate of detection of the two RLF tests varied across the years with a sensitivity ranging from <80% in 2021 to >80% in 2022. More importantly the capacity of these two RLF tests to detect IgG antibodies decreased with time. On the contrary, the sensitivity of the ELISA test was still above 80% more than six months post DoD.

Conclusion

We recommend that sero-epidemiological surveys focused on testing antibodies should not rely on performances reported by the assay manufacturers. They should include a formal evaluation of the selected assays to ensure its limitations and strengths conform with the data-accuracy requirements of the surveys.

1. Introduction

In late December 2019, pneumonia cases of unknown origin were reported in Wuhan, the Hubei province of china. As a response to COVID-19 pandemic that ensued, various techniques were used to diagnose SARS-CoV-2 infections including RT-PCR based tests, rapid antigen tests, and a range of serological tests [1]. Unlike RT-PCR and rapid antigen tests which respectively detect the presence of viral RNA and proteins, serological tests detect anti-SARS-CoV-2 antibodies in the serum/plasma of individuals who have been previously infected with the virus. In addition to being able to detect possible ongoing SARS-CoV-2 infections, such tests may also detect previous SARS-CoV-2 infections or be used to test the immunological responses of individuals who have been vaccinated against the virus. Given the early hopes that “herd immunity” to SARS-CoV-2 might be attainable and would herald the end of the pandemic, numerous serological tests were developed in the early phases of the pandemic specifically to monitor growing levels of population-wide immunity to SARS-CoV-2. In a review published in 2020, Deeks and collaborators [2] identified 57 publications focusing on the evaluation of both commercial and in-house serological tests, most of which were derived in Africa. They found that beyond three weeks post-symptom onset, the average sensitivity for tests to detect anti-SARS-COV-2 IgG/IgM was 96.0% [2].

Several studies have reported on the cross-reactivity of pre-pandemic African patient sera (i.e. sera containing antibodies elicited by non-SARS-CoV-2 antigens) with SARS-CoV-2 in Africa [3–5]. The presence in patient sera of antibodies that cross react with SARS-CoV-2 proteins could substantially impact the performance of the various serological tests used across the continent. In fact, evaluations of many of these tests in African settings have revealed a degree of sensitivity that was lower than that reported by the test manufacturers [6–8].

Across Africa, various protocols have been used to evaluate serological tests. For example while Ouedraogo and collaborators [8] from Burkina Faso, evaluated the sensitivity of ten Rapid Lateral Flow (RLF)-based tests against a serological enzyme-linked immunosorbent assays (ELISA) test, Ige and collaborators [6] compared the sensitivity of five serological tests (four ELISA and one chemiluminescent microparticle immuno-assay) against the RT-qPCR test; and finally, Jugwanth and collaborators [7] used two chemiluminescent microparticle immunoassays to perform the SARS-CoV-2 IgG assay on serial samples collected up to 45 day post-symptom onset. Results from these diverse studies are therefore difficult to compare and require further studies, particularly in a context where efforts are focused on tracking the epidemiology of the disease in Africa: a region of the world which has had both limited molecular diagnostic testing capacity and low degrees of vaccine-based immunization against SARS-CoV-2. Here, using a unique cohort of patient plasma samples from SARS-CoV-2 infected individuals that were collected during three years of the pandemic, both at the time of diagnosis, and up to six months later, we evaluated the specificity and sensitivity of three commercial serological tests in Cameroon.

2. Material and methods

2.1. Study design and sample collection

This study was performed in three consecutive years (2020, 2021 and 2022) in Yaoundé, Cameroon. Individuals seeking COVID-19 testing were invited to participate in the study. Each study participant signed a concern form and responded to a questionnaire to provide socio-demographic and clinical data. Ten ml of blood was then collected for laboratory testing. Only samples from participants who tested positive by quantitative reverse transcription PCR (RT-qPCR) were used for the study. All the study participants were diagnosed by RT-qPCR using the DAAN GENE system (SUN YAT-SEN UNIVERSITY; Guangzhou, China). Results were expressed in Ct (Cycle threshold) and a Ct value for ORF1ab and N genes (the two targeted genes) more or equal to 37 was considered negative and less than 37 was considered positive. SARS-CoV-2-infected participants were quarantined and asked to perform another RT-qPCR test 14 days later according to the national protocol guidance and provide another 10 ml of blood. Following this, a subset of study participants in 2020 and 2021 agreed to donate 10 ml again every month for up to six months to evaluate the persistence of the anti-SASR-CoV-2 antibodies that they were producing. The collected bloods were centrifuged at 2000 rpm for 10 min and plasma were separated and used for laboratory experiments. In addition to these SARS-CoV-2-infected samples, a set of 149 pre-COVID-19 plasma samples collected between 2008 and 2013 was used as a negative control group to test the specificity of each test. Some of these samples originated from HIV-, HBV- and HCV-infected or co-infected individuals. The ethics approval to perform this study was granted by the Cameroon national ethics committee (Nº2020/05/1218/CE/CNERSH/SP).

2.2. Serological testing

Two commercial Rapid Lateral Flow (RLF) tests for detecting anti-SARS-CoV-2 antibodies were used in this study, both of which were run in parallel on the day of sample collection: These tests were (1) the PANBIO™ COVID-19 IgG/IgM rapid test (Abbott Diagnostics, Abbott Park, Illinois, USA); and (2) the LABNOVATION COVID-19 (SARS-CoV-2) IgG/IgM rapid test one (Labnovation Technologies Inc, CHINA). In addition, we also evaluated a commercial ELISA test: the PLATELIA SARS-CoV-2 total Ab (Bio-Rad, FRANCE). The ELISA test was performed on all samples once they had all been collected. Whereas all three tests detect antibodies against the SARS-COV-2 nucleocapsid (N) protein, the two RLF tests qualitatively detect both IgG and IgM in the plasma, the ELISA test semi-quantitatively detects total antibodies (IgM/IgG/IgA). All the tests were performed as recommended by the various manufactures. The RLF tests were adjudged as being either reactive for IgM or IgG and when a sample was reactive to both IgM and IgG, it was counted as unique reactivity.

2.3. Statistical analyses

The R software package (Version 4.2.2; R Core Team, USA) was used for measuring sensitivity and specificity against true positives (defined against RT-qPCR) and negatives (defined as archived samples from before the COVID-19 pandemic), and expressed as a percentage. The data were categorized according to year of sampling, time points and tests used. Differences between data were analyzed using the chi square test. The specificity and sensitivity were calculated and presented with 95% confidence intervals. Graphs were generated using the Graphpad prism (Version 9.5.0; Graphpad software, USA). p-values <0.05 were considered statistically significant. Specificity was calculated as the percentage of true negatives (in this case, archived pre-COVID samples that were non-reactive to the serological tests) over the sum of the true negatives and false positives (archived pre-COVID samples that were reactive to the serological tests). The sensitivity was calculated as a percentage of true positives (in this case, samples that were reactive to the serological tests) over the sum of the true positives and false negatives (samples that were non-reactive to the serological tests) compare to the RT-qPCR test.

3. Results

3.1. Participant characteristics

A total of two hundred and thirty-five (235) participants were enrolled in this study; including 130 in 2020, 88 in 2021 and 17 in 2022. Of these, 123 were male and 112 were female; the mean age was 39±12.6 years (5–72) (Table 1). The vast majority of the study participants were asymptomatic (77%) irrespective of patient sex (Table 1). Overall, seven major clinical symptoms were recorded among our study participants with cough being the most commonly reported symptom, followed by headache, sore throat, fatigue, myalgia, stomach-ache and fever. However, the distribution of these symptoms varied across sampling years (Fig. 1). The median cycle threshold was similar across the years, averaging at 30.6 in 2020, 30.4 in 2021 and 29.6 in 2022. No participant was vaccinated based on a self-reported questionnaire.

Table 1. Baseline characteristics and demographic data of study participants.

		Gender
Parameter	Total	Female	Male
Number of participants (%)	235	112 (48%)	123 (52%)
Age mean ± SD (years)	39yrs ± 12.6	40yrs ± 13.5	38yrs ± 11.9
Symptomatic (%)	55 (23%)	28 (25%)	27 (22%)
Asymptomatic (%)	180 (77%)	84 (75%)	96 (78%)

^*SD; Standard deviation.

Fig. 1. Clinical symptoms of the study participants at the time of diagnosis.

3.2. Assay performance against negative control (archive samples)

The overall specificity of all the tests was first determined using archived plasmas collected prior to the COVID-19 pandemic between 2008 and 2013. For the two commercial RLF tests, PANBIO™ COVID-19 IgG/IgM and LABNOVATION COVID-19 IgG/IgM, the specificity was 85% (95% CI: 78.4%−90.1%) and 91% (95% CI: 85.4%−94.4%) respectively for the IgM and 95% (95% CI: 91.1%−98.2%) and 97% (95% CI: 93.3%−99%) respectively for the IgG and the difference was not significant (Fig. 2a). The ELISA PLATELIA SARS-CoV-2 total Ab test which detects IgA antibodies in addition to IgM and IgG antibodies, has a specificity of 91% (95% CI: 86.4%−95%): similar to that of the LAB-NOVATION COVID-19 IgG/IgM total antibody test (Fig. 2a). Analyses of the reactive plasmas revealed that only one sample out of 14 that were reactive with the ELISA test, also reacted in the two RLF tests (Fig. 2b). Cross reactivity was observed in almost all the categorized plasma (HIV negative and positive, HBV and HCV positive) and in all the tests used (Table 2).

Fig. 2. Specificity of the three commercial tests using pre-COVID samples.

a) Comparison of the specificity of the different tests for IgM, IgG and total antibodies (IgM/IgG for the rapid lateral flow or RLF tests and IgM/IgG/IgA for the ELISA test). b) Venn diagram showing the distribution of false positives detected between the two RLF and ELISA tests. The specificity was calculated as the percentage of true negatives (in this case, archived pre-COVID samples that were non-reactive to the serological tests) over the sum of the true negatives and false positives (archived pre-COVID samples that were reactive to the serological tests). ns: not significant

Table 2. Distribution of false positives across different tests.

	PANBIO RLF (Reactive/Total)	LABNOVATION RLF (Reactive/Total)	PLATELIA ELISA (Reactive/Total)
HIV+	1/25	2/25	2/25
HBV+	3/10	2/10	1/10
HCV+	1/10	1/10	0/10
HIV+/HBV+	7/10	5/10	0/10
HIV+/HCV+	1/3	0/3	0/3
HIV-	14/91	8/91	11/91
Total	27/149	18/149	14/149

3.3. Assay performance against RT-qPCR (sensitivity)

At day 14 post day of diagnosis (DoD), as expected, we observed an increased sensitivity in all the three tests used, albeit with significant differences observed between the tests (Fig. 3b). While the IgG detection was at 80% (95% CI: 72–86%) with the PANBIO™ COVID-19 IgG/IgM, it was at 58% (95% CI: 49.3–66.4%) with the LABNOVATION COVID-19 IgG/IgM (Fig. 3b). When considering total antibodies, the ELISA test had the best performance with a sensitivity of 88% (95% CI: 81–93%), followed by the PANBIO™ COVID-19 IgG/IgM test with a sensitivity of 83% (95% CI: 76–89%) and the LABNOVATION COVID-19 IgG/IgM test with a sensitivity of 69% (95% CI: 60%−76.1%) (Fig. 3b).

Fig. 3. Sensitivity of the two RLF (Panbio and Labnovation) and the ELISA Platelia test.

At the time of diagnosis or DoD (a, c, e and g) and 14 days post DoD (b, d, f and h). Globally (a and b) and with cohort sampled in 2020 (c and d), 2021 (e and f) and 2022 (g and h). The sensitivity was calculated as a percentage of true positives (in this case, samples that were reactive to the serological tests) over the sum of the true positives and false negatives (samples that were non-reactive to the serological tests) compare to the RT-qPCR.

ns: not significant; *: significant p<0.05; **: significant p<0.01; ***: significant p<0.001.

Separate analyses of the sensitivity attained with samples collected in 2020, 2021 and 2022 revealed a marked differences between the tests when applied to plasma samples collected at day 14 post DoD. Specifically the kinetics of humoral responses suggests broad production of IgG (Fig. 3d, f and h). For the 2020 plasma samples, the sensitivity of the PANBIO™ COVID-19 IgG/IgM test (84% with 95% CI of 73–91%) was significantly higher than that of the LABNOVATION COVID-19 IgG/IgM test (48% with 5% CI of 36.4–61%) (Fig. 3d). However, the sensitivity of the LABNOVATION COVID-19 IgG/IgM test increased to 62% (95% CI: 48%−75%) with the 2021 plasma samples, while that of the PANBIO™ COVID-19 IgG/IgM decreased to 71% (95% CI: 57%−82.2%) for these samples (Fig. 3f). For the 2022 plasma samples, both tests had a sensitivity of more than 80% (Fig. 3h). The sensitivity of the ELISA test was consistently >80% across all the three years although not significantly higher than that observed for the total antibodies (IgM and IgG) detected by the two RLF tests (Fig. 3d, f and h).

We further compared the sensitivity across the three years for each individual test. At day 14 post DoD, the level of IgG detection by the PANBIO™ COVID-19 IgG/IgM test was 80% in 2020 and 2022 and below 80% in 2021; but the differences across the year were not significant (Fig. 4b). Similarly, observed fluctuations in the detection sensitivity of the LABNOVATION COVID-19 IgG/IgM test were also not significant across the three years; although only for the 2022 plasma samples, was the sensitivity above 80% (Fig. 4d). The ELISA test achieved a sensitivity of more than 80% in all the three years with no significant differences in sensitivity being observed (Fig. 4f).

Fig. 4. Sensitivity of each test across the three years of infection.

a) Sensitivity of the Panbio RLF test at the time of diagnosis or DoD and b) 14 days post DoD. c) Sensitivity of the Labnovation RLF test at DoD and d) 14 days post DoD. e) Sensitivity of the Platelia ELISA test at DoD and f) 14 days post DoD. The sensitivity was calculated as a percentage of true positives (in this case, samples that were reactive to the serological tests) over the sum of the true positives and false negatives (samples that were non-reactive to the serological tests) compare to the RT-qPCR. ns: not significant; *: significant p<0.05; **: significant p<0.01.

Data relating to the utility of the serological tests as potential diagnostic tools at DoD, to complement the real-time quantitative PCR (RT-qPCR), are summarized in Fig. 3 (a, c, e and g) and Table 3.

Table 3. Summary of Sensitivities of the commercial tests across the years of infection and at different time points.

	SENSITIVITY (95% Confidence Interval)
	PANBIO RLF			LABNOVATION RLF			PLATELIA ELISA
TIMEPOINTS	Y2020 N = 130	Y2021 N = 88	Y2022 N = 17	Y2020 N = 130	Y2021 N = 88	Y2022 N = 17	Y2020 N = 130	Y2021 N = 88	Y2022 N = 17
Day of diagnostic (DoD)	N = 127 46% (37.3%–54.3%)	N = 83 23% (15.1%–33%)	N = 17 70% (47%–87%)	N = 127 41% (33%–50%)	N = 83 19% (12.2%–29%)	N = 17 67% (41.3%–83%)	N = 123 47% (38%–55.1%)	N = 80 39% (29%–49.3%)	N = 17 71% (47%–87%)
14 days Post-DoD	N = 62 87% (77%–93.3%)	N = 45 76% (61.3%–86%)	N = 17 88% (66%–98%)	N = 62 65% (52.1%–54.3%)	N = 45 69% (54.3%–81%)	N = 17 82% (59%–94%)	N = 60 89% (78.4%–94.4%)	N = 44 84% (71.2%–92.2%)	N = 16 94% (73%–100%)
44 days Post-DoD	N = 45 89% (76%–95.1%)	N = 39 77% (62%–87.4%)	-	N = 45 89% (76%–95.1%)	N = 39 62% (50%–75.1%)	–	N = 42 93% (82%–98%)	N = 39 72% (56.2%–84%)	–
74 days Post-DoD	N = 31 81% (64.2%–91.1%)	N = 31 60% (42.3%–75.4%)	–	N = 31 81% (64.2%–91.1%)	N = 31 67% (49%–81%)	–	N = 30 90% (75.1%–97%)	N = 30 73% (56%–86%)	–
104 days Post-DoD	N = 18 83% (61%–94.2%)	N = 28 54% (365–70.4%)	–	N = 18 50% (29%–71%)	N = 28 46% (30%–64.2%)	–	N = 18 89% (67.2%–985)	N = 25 71% (53%–85%)	–
134 days Post-DoD	N = 17 77% (53%–90.4%)	N = 26 23% (11%-42%)	–	N = 17 52% (31%–74%)	N = 26 35% (19.4%–54%)	–	N = 17 88% (66%–98%)	N = 26 69% (50%–84%)	–
164 days Post-DoD	N = 14 64% (39%–84%)	N = 24 25% (12%–45%)	–	N = 14 71% (45.4%–88.3%)	N = 24 29% (15%–49.1%)	–	N = 14 86% (60.1–98%)	N = 24 88% (69%–96%)	–
194 days Post-DoD	N = 9 56% (27%–81.1%)	N = 17 29% (13.2%–53.1%)	–	N = 9 44% (19%–733.3%)	N = 17 29% (13.2%–53.1%)	–	N = 9 89% (57%–99.4%)	N = 17 77% (53%–90.4%)	–

3.4. Assay performance and durability of antibody produced

Finally, we assessed the ability of the different tests to detect SARS-CoV-2 antibodies several months after the apparent clearance of infections. As shown in Fig. 5a, the sensitivity of the commercial ELISA test rose rapidly from 47% (95% CI: 38–55.1%) at DoD to 89% (95% CI: 78.4%−94.4%) at 14 days post DoD. The rate of detection was still above 80% up to six months post DoD (Fig. 5a). The same trend was observed in 2021 at DoD and 14 days post DoD; then the detection decreased to less than 80% by day 134 post DoD where it started to rise to more than 90% at day 164 post DoD before starting to decrease again (Fig. 5b and Table 3).

Fig. 5. Longitudinal profile of the antibody detection by the three commercial tests after 6 months of post diagnostic.

a) Year 2020 and b) Year 2021.

The two RLF tests followed a different pattern in the detection of IgG antibodies. In 2020, after a peak of detection sensitivity at 44 days post DoD, there was a steady decrease of IgG detection up to six months post DoD for the PANBIO™ COVID-19 IgG/IgM test. For the LABNOVATION COVID-19 IgG/IgM test, a second peak in detection sensitivity was achieved at 164 days post DoD before decreasing again (Fig. 5a and Table 3). For both RLF tests and for the 2021 plasma samples, the decrease in IgG detection sensitivity was rapid, from day 44 post DoD to six months post DoD (Fig. 5b and Table 3).

4. Discussion

In this study, we used plasma samples collected before the COVID-19 pandemic and plasma samples collected during the three consecutive years of the pandemic (2020, 2021 and 2022) to evaluate the performance of two commercial Rapid Lateral Flow or RLF tests, the PAN-BIO™ COVID-19 IgG/IgM rapid test and the LABNOVATION COVID-19 (SARS-CoV-2) IgG/IgM rapid test one; and a commercial ELISA test, the PLATELIA SARS-CoV-2 total Ab. We found that the IgG specificity of all tests was high (≥ 95% for the RLF tests and 91% for the ELISA test. However, At day 14 post DoD, the sensitivity of the ELISA test was significantly higher than that of either of the RLF tests, remaining >80% across plasmas collected from all years of the pandemic and up to six months post DoD. As expected, the capacity of the two RLF tests to detect IgG antibodies waned with time with declines being more rapid in plasma sampled in 2021. (summarized in Table 3).

Over the course of the pandemic, several studies had also reported good specificity (>95%) and sensitivity (>80%) for the ELISA tests and reduced performances for the RLF tests when comparing either in-house or commercial ELISA-based serological tests [9]. While most of these studies used cohorts recruited at only one time-point, our study enrolled participants in three different years. This therefore represented a unique opportunity to account for the influence on the sensitivity of the tests of the SARS-CoV-2 variants that were in play during the three years. In one of the rare studies that also used serial sampling, Jugwanth and collaborators [7] found that, the sensitivity of two chemiluminescent microparticle immunoassay tests increased from day 14 to day 50 post DoD; although this increase was not linear. They subsequently recommended the used of these two tests at day 14 post DoD as a diagnostic tool.

In this study, we focused on the capacity of the different serology tests to detect IgG. As the pandemic is coming slowly under control and endemicity is being reached, most attention is now being paid to monitoring the levels of population-scale immunity to SARS-CoV-2: an endeavor predominantly focused on continued testing for SARS-CoV-2 reactive IgG antibodies in the plasma of randomly sampled people.

The different patterns that we observed in the sensitivities of the tests across the years could be attributed to the SARS-CoV-2 variants that were circulating in these different years. It was shown that in 2020 in Africa, the pandemic was largely driven by the initial Wuhan-Hu-1 PANGO B-lineages of the virus (generally referred to as the wild type); this lineage was replaced in 2021 when several variants such as Alpha, Beta and Delta successively circulated with varying amounts of overlap; and in 2022 when the Omicron variant displaced all others [10]. As most diagnostic tests were made using the wild-type variant, it is possible that cross-reactivity with antibodies against other variants could vary across different tests.

During the peak period of COVID-19 pandemic, reinfection was likely common. In fact, in a systematic review published in 2021, Townsend and collaborators [11] found that reinfection by SARS-CoV-2 under endemic conditions would likely occur between 3 months and 5 years after peak antibody response, with a median of 16 months. Although no participant enrolled in the present study mentioned having been reinfected during follow-up or before enrollment, it cannot be excluded that cases of reinfections could have occurred especially among participants enrolled in 2012 and 2022 and during follow up period. This could perhaps explain the performances observed in 2022 which were higher than those of previous years. In addition, serological assay performance sometime involves a complexity of antibody detection in real settings. In a recent meta-analyses published by Xiaomeng and collaborators in 2022 [12], results showed that manufacturer evaluations of assay were overestimations compared to independent validations. In a setting where parameters like population background and past or current pathogen exposures are identical, except for viral mutations that can influenced assay performance [13], the other main factor that also contributes for this, is the assay intrinsic performance, like its ability for example to detect low titre of antibody.

One of the limitations of this study is that we were not able to get access to the infecting viruses to assess their genotypes and test for associations between infecting virus genotypes and antibody detection sensitivity. In addition, the sample size of the third year (2022) and those collected after three months post DoD in 2020 and 2021 were low and this might have potentially influenced the power of our analyses. Despite these limitations, to our knowledge, this is the first study that has demonstrated the sensitivity performance of these commercial assays both during different years of the COVID-19 pandemic and several timepoints post DoD.

As pointed out by Bastos and collaborators [14], seroprevalence estimates can vary considerably based on the assay used, even in the same population and based on the same samples; our study therefore favours independent evaluations on representative populations and not rely on performances reported by the assay manufacturers.