ABSTRACT
We assessed the performance of the CoronaCHEK lateral flow assay on samples from Uganda and Baltimore to determine the impact of geographic origin on assay performance. Plasma samples from severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) PCR-positive individuals (Uganda, 78 samples from 78 individuals, and Baltimore, 266 samples from 38 individuals) and from prepandemic individuals (Uganda, 1,077, and Baltimore, 532) were evaluated. Prevalence ratios (PR) were calculated to identify factors associated with a false-positive test. After the first positive PCR in Ugandan samples, the sensitivity was 45% (95% confidence interval [CI], 24,68) at 0 to 7 days, 79% (95% CI, 64 to 91) at 8 to 14 days, and 76% (95% CI, 50 to 93) at >15 days. In samples from Baltimore, sensitivity was 39% (95% CI, 30 to 49) at 0 to 7 days, 86% (95% CI, 79 to 92) at 8 to 14 days, and 100% (95% CI, 89 to 100) at 15 days after positive PCR. The specificity of 96.5% (95% CI, 97.5 to 95.2) in Ugandan samples was significantly lower than that in samples from Baltimore, 99.3% (95% CI, 98.1 to 99.8; P < 0.01). In Ugandan samples, individuals with a false-positive result were more likely to be male (PR, 2.04; 95% CI, 1.03,3.69) or individuals who had had a fever more than a month prior to sample acquisition (PR, 2.87; 95% CI, 1.12 to 7.35). Sensitivity of the CoronaCHEK was similar in samples from Uganda and Baltimore. The specificity was significantly lower in Ugandan samples than in Baltimore samples. False-positive results in Ugandan samples appear to correlate with a recent history of a febrile illness, potentially indicative of a cross-reactive immune response in individuals from East Africa.
KEYWORDS: lateral flow antibody assay, SARS-CoV-2, Uganda, assay performance
INTRODUCTION
Severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) infection causes coronavirus disease 2019 (COVID-19) (1), which has been detected on all continents and continues to be a public health emergency globally (2). Critical to public health efforts to combat the pandemic are accurate serologic assays to differentiate exposed from unexposed individuals (3). Many studies investigate the performance of these assays on samples from Asia (4), Western Europe (5), and the United States (6). However, little information is available on the performance of these assays in an African setting, though initial studies provide evidence of potential problems (7), particularly among febrile patients infected by other infectious pathogens (8).
Serologic assays used for the detection of antibodies to different viral infections can vary in performance based on the origin of the samples being tested, as has been seen with HIV (9), hepatitis C virus (HCV) (10), and herpes simplex virus 2 (HSV-2) (11). It is thought that these differences in specificity result from host genetics of the source population and the frequency and distribution of the infectious agents the population is exposed to (12). We sought to compare the performance of the CoronaCHEK lateral flow assay (LFA) on samples from Uganda and the United States to assess the impact of geographic origin on the performance of this assay. Samples from known SARS-CoV-2-infected individuals with known duration of infection and prepandemic samples were tested to evaluate the sensitivity and specificity of the assay and to identify factors associated with a false-positive result.
MATERIALS AND METHODS
Ethics statement.
The use of samples from Baltimore was approved by The Johns Hopkins University School of Medicine Institutional Review Board (IRB00247886, IRB00250798, and IRB00091667). The use of samples from Uganda was approved by the Uganda Virus Research Institute’s Research Ethics Committee (GC/127/20/04/773 and GC/127/13/01/16), Western Institutional Review Board, protocol 200313317, and the Uganda National Council for Science and Technology (HS637ES). The parent studies were conducted according to the ethical standards of the Helsinki Declaration of the World Medical Association, where all subjects provided written informed consent. All samples were deidentified prior to testing.
Sample sets.
To assess sensitivity, samples from subjects known to be SARS-CoV-2 PCR+ in Uganda and the United States with known duration from first PCR+ date were evaluated. Samples from 78 PCR+ individuals at different time intervals were identified at the Uganda Virus Research Institute in Entebbe and at Makerere University in Kampala, Uganda. None of the Ugandan individuals were hospitalized, and all had mild disease. Samples (n = 266) from the United States were from 38 hospitalized COVID-19 patients attending the Johns Hopkins Hospital in Baltimore, MD, in the United States (13).
To assess the specificity of the assay, prepandemic samples were tested. This included 1,077 stored samples from the Rakai Community Cohort Study, collected between 2011 and 2013 (14). The Ugandan samples included 543 individuals who reported having been febrile within the month prior to sample acquisition and 534 individuals who did not report a febrile illness, matched by age and gender. The 532 prepandemic samples from the United States were remnant plasma samples collected in the Johns Hopkins Hospital Emergency Department (JHH ED) between December 2015 and January 2016 (15).
Laboratory testing and statistical analysis.
All samples were analyzed with the CoronaCHEK LFA (Hangzhou Biotest Biotech Co. Ltd.) according to the manufacturer’s protocol. CoronaCHEK contains separate IgM and IgG bands and uses the spike receptor binding domain (RBD) as the target antigen. The manufacturer reports a combined sensitivity of 100% (confidence interval [CI], 88.7 to 100) and a combined specificity of 100% (CI, 95.4 to 100). Sensitivity by duration of infection and specificity among prepandemic samples were assessed for the presence of either IgM or IgG bands for any reactivity. Statistical analysis was performed with STATA 14.2 (StataCorp, College Station, TX, USA), and 95% CI for sensitivity and specificity were calculated with the Clopper-Pearson exact method. Bivariate Poisson regression models were used to calculate prevalence ratios (PR) for factors associated with a false-positive test among prepandemic samples.
RESULTS
There were significant differences in the performance for the CoronaCHEK LFA between samples from Uganda and Baltimore (Table 1). When any reactivity was compared (IgM or IgG), there was no significant difference in reactivity by duration of infection. Though 100% of samples from Baltimore were seropositive by 14 days after their first time point, this was not the case for the Ugandan samples. Specificity, when any reactive band was considered a false-positive result, was significantly lower in Ugandan samples at 96.9% (CI, 95.2 to 97.5) than in those from Baltimore, at 99.3% (CI, 98.1 to 99.8; P < 0.01).
TABLE 1.
Sensitivity and specificity of the CoronaCHEK lateral flow point-of-care assay for the detection of IgM and IgG antibodies to SARS-CoV-2
| Parameter (n) | % (95% CI) for: |
||
|---|---|---|---|
| IgM | IgG | IgM or IgG | |
| Sensitivity | |||
| Uganda | |||
| ≤7 days (22) | 41 (21–64) | 41 (21–64) | 45 (24–68) |
| >7 to 14 days (39) | 74 (58–87) | 49 (32–87) | 79 (64–91) |
| >14–28 days (17) | 41 (18–67) | 65 (38–86) | 76 (50–93) |
| Baltimore | |||
| ≤7 days (102) | 34 (25–44) | 21 (13–30) | 39 (30–49) |
| >7 to 14 days (132) | 82 (74–88) | 75 (67–82) | 86 (79–92) |
| >14–28 days (32) | 100 (89–100) | 100 (89–100) | 100 (89–100) |
| Specificity | |||
| Uganda (1,077) | 96.9 (95.7–97.9) | 99.4 (98.7–99.7) | 96.5 (95.2–97.5) |
| Baltimore (532) | 99.3 (98.1–99.8) | 100 (99.3–100) | 99.3 (98.1–99.8) |
There were 4 and 38 false-positive results in Baltimore prepandemic samples and Ugandan samples, respectively. All four from Baltimore were all faint IgM bands, while 82% (31/38) of the false-positive samples from Uganda had only reactive IgM bands. Of the seven prepandemic Ugandan samples that were IgG reactive, two were also reactive for IgM. Ugandan samples were significantly more likely to be misclassified if they came from men (PR, 2.04; 95% CI, 1.03 to 3.69; P = 0.04) or if the individual had reported fever more than a month prior to sample collection (PR, 2.87; 95% CI, 1.12 to 7.35; P = 0.028). There was a trend to test positive if they had reported pneumonia-like symptoms (PR, 2.34; 95% CI, 0.98 to 5.59; P = 0.056). Other factors not associated with a false-positive result included age, community type, and HIV status (Table 2). There were too few misclassified samples from Baltimore to assess factors associated with misclassification within this population.
TABLE 2.
Factors associated with a false-positive SARS-CoV-2 antibody response in samples from Uganda
| Defining category | SARS-CoV-2 antibody-positive outcome |
|
|---|---|---|
| % (no./total) | PR (95% CI) | |
| Sex | ||
| Female | 2.7 (20/737) | Reference |
| Male | 5.3 (18/340) | 2.06 (1.03 to 3.69) |
| Age | ||
| 18–24 | 3.1 (10/327) | Reference |
| 25–34 | 4.3 (19/439) | 1.42 (0.66 to 3.04) |
| 35–44 | 2.7 (7/260) | 0.88 (0.34 to 2.31) |
| 45–54 | 3.9 (2/61) | 1.28 (0.28 to 5.85) |
| Community type | ||
| Agrarian | 3.2 (14/436) | Reference |
| Fishing | 5.1 (19/372) | 1.59 (0.80 to 3.17) |
| Trading | 1.9 (5/269) | 0.58 (0.21 to 1.61) |
| Pregnancy (no males in analysis) | ||
| Not pregnant | 2.5 (8/318) | Reference |
| Pregnant | 2.9 (12/419) | 1.14 (0.47 to 2.78) |
| Fever < 1 mo | ||
| No | 3.2 (17/534) | Reference |
| Yes | 3.9 (21/543) | 1.21 (0.64 to 2.30) |
| Fever > 1 mo | ||
| No | 3.2 (33/1,023) | Reference |
| Yes | 9.3 (5/54) | 2.87 (1.12 to 7.35) |
| Cough | ||
| No | 3.3 (27/825) | Reference |
| Yes | 4.4 (11/252) | 1.33 (0.66 to 2.69) |
| Pneumonia | ||
| No | 3.2 (32/997) | Reference |
| Yes | 7.5 (6/80) | 2.34 (0.98 to 5.59) |
| HIV status | ||
| Negative | 3.4 (21/618) | Reference |
| Positive | 3.7 (17/459) | 1.09 (0.58 to 2.07) |
DISCUSSION
This study demonstrates differential performance of the CoronaCHEK LFA on samples collected from Uganda compared to those collected from Baltimore. Though sensitivity for both IgG and IgM in samples from Baltimore was 100% by 14 days after the subjects’ first PCR+ date, unlike samples from Uganda, this difference was not significantly different. Though not significant, there was a substantial difference in the point estimates, with the Ugandan samples having a sensitivity of only 76% at >14 days after the subjects’ first PCR+ date. These differences in sensitivity could be due to Ugandan samples coming from individuals with mild disease while the samples from Baltimore were from hospitalized individuals. Specificity was significantly lower in the Ugandan prepandemic samples than those from Baltimore, though this difference was all associated with the IgM band. False-positive results in Ugandan samples were higher among men and those who had reported a febrile episode more than a month prior to sample acquisition. Of the false-positive results detected, the vast majority were IgM reactive.
These results demonstrate that the performance characteristics of serological assays for SARS-CoV-2 antibody detection cannot be extrapolated to different populations without adequate validation studies. This study supports the need for validation studies on SARS-CoV-2 serologic assays in Africa, an area for which few data exist (16). Though a lower specificity was found in Ugandan samples than those from Baltimore, the specificity of 96.5% was much greater than the 85% found for the Euroimmun IgG S1 enzyme-linked immunosorbent assay (ELISA) in prepandemic samples from Benin (8). As shown in the study by Mboumba Bouassa (7), our study demonstrated that the main cause for false-positive results was a reactive IgM test. If one ignores the presence of an IgM band, the specificity of the CoronaCHEK increased to 99.4% (95% CI, 98.7 to 99.7) for Ugandan samples and 100% (95% CI, 99.3 for 100) for Baltimore samples, with no loss of sensitivity at 14 days after the first positive PCR for SARS-CoV-2.
There are a number of limitations of our study. First, the samples from Uganda of SARS-CoV-2-infected patients were limited, with only six samples within the first week after the first PCR-positive test and no serial samples for a given individual. Additionally, these samples came from known infected Ugandan individuals with limited symptoms, while the Baltimore samples all came from known SARS-CoV-2-positive individuals who were hospitalized. The prepandemic samples from Baltimore were not matched to those from Uganda based on symptomology, though historically, individuals attending the ED in the United States have a high prevalence of fever and viral infections (17). Samples from the JHH ED do have a high burden of chronic viral infections, as demonstrated by seroprevalences of 6%, 12%, and 50% for HIV, HCV, and HSV-2, respectively (18).
In summary, the geographical origin of the samples appeared to impact the performance of the CoronaCHEK LFA. IgM reactivity was the main cause for the false-positive results. Since IgM responses generally appear a couple of days before IgG responses, it may be useful not to measure IgM at all in serological studies, given the improvement in specificity. Further evaluations of serologic assays are needed to find appropriate tools for serosurveillance in an African setting.
ACKNOWLEDGMENTS
We acknowledge all of the participants who contributed specimens to this study and the study staff, without whom this study would not have been possible.
We have no conflicts of interest to declare.
This study was supported by the Division of Intramural Research, National Institute of Allergy and Infectious Diseases (NIAID), National Institutes of Health (NIH), as well as by extramural support from NIAID UM1-AI068613 to R.E.F. and NIH Center of Excellence in Influenza Research and Surveillance HHSN272201400007C to R.E.R.
Contributor Information
Oliver Laeyendecker, Email: olaeyen1@jhmi.edu.
Yi-Wei Tang, Cepheid.
REFERENCES
- 1.Zhu N, Zhang D, Wang W, Li X, Yang B, Song J, Zhao X, Huang B, Shi W, Lu R, Niu P, Zhan F, Ma X, Wang D, Xu W, Wu G, Gao GF, Tan W, China Novel Coronavirus Investigating and Research Team . 2020. A novel coronavirus from patients with pneumonia in China, 2019. N Engl J Med 382:727–733. doi: 10.1056/NEJMoa2001017. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Eurosurveillance Editorial Team. 2020. Note from the editors: World Health Organization declares novel coronavirus (2019-nCoV) sixth public health emergency of international concern. Euro Surveill 25:200131e. doi: 10.2807/1560-7917.ES.2020.25.5.200131e. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Bermingham WH, Wilding T, Beck S, Huissoon A. 2020. SARS-CoV-2 serology: test, test, test, but interpret with caution! Clin Med J R Coll Physicians London 20:365–368. doi: 10.7861/clinmed.2020-0170. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Zhao J, Yuan Q, Wang H, Liu W, Liao X, Su Y, Wang X, Yuan J, Li T, Li J, Qian S, Hong C, Wang F, Liu Y, Wang Z, He Q, Li Z, He B, Zhang T, Fu Y, Ge S, Liu L, Zhang J, Xia N, Zhang Z. 2020. Antibody responses to SARS-CoV-2 in patients with novel coronavirus disease 2019. Clin Infect Dis 71:2027–2034. doi: 10.1093/cid/ciaa344. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Sacristan MS, Collazos-Blanco A, Cintas MIZ, García AS, de Villavicencio CY, Maestre MM. 2020. Comparison of various serological assays for novel SARS-COV-2. Eur J Clin Microbiol Infect Dis 2020:1–6. doi: 10.1007/s10096-020-04091-4. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Patel EU, Bloch EM, Clarke W, Hsieh YH, Boon D, Eby Y, Fernandez RE, Baker OR, Keruly M, Kirby CS, Klock E, Littlefield K, Miller J, Schmidt HA, Sullivan P, Piwowar-Manning E, Shrestha R, Redd AD, Rothman RE, Sullivan D, Shoham S, Casadevall A, Quinn TC, Pekosz A, Tobian AAR, Laeyendecker O. 2020. Comparative performance of five commercially available serologic assays to detect antibodies to SARS-CoV-2 and identify individuals with high neutralizing titers. J Clin Microbiol 59:e02257-20. doi: 10.1128/JCM.02257-20. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Mboumba Bouassa RS, Péré H, Tonen-Wolyec S, Longo JDD, Moussa S, Mbopi-Keou FX, Mossoro-Kpinde CD, Grésenguet G, Veyer D, Bélec L. 2021. Unexpected high frequency of unspecific reactivities by testing pre-epidemic blood specimens from Europe and Africa with SARS-CoV-2 IgG–IgM antibody rapid tests points to IgM as the Achilles heel. J Med Virol 93:2196–2203. doi: 10.1002/jmv.26628. [DOI] [PubMed] [Google Scholar]
- 8.Yadouleton A, Sander AL, Moreira-Soto A, Tchibozo C, Hounkanrin G, Badou Y, Fischer C, Krause N, Akogbeto P, De Oliveira Filho EF, Dossou A, Brünink S, Aïssi MAJ, Djingarey MH, Hounkpatin B, Nagel M, Drexler JF. 2021. Limited specificity of serologic tests for SARS-CoV-2 antibody detection. Emerg Infect Dis 27:233–237. doi: 10.3201/eid2701.203281. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Van Kerckhoven I, Vercauteren G, Piot P, Van Der Groen G. 1991. Comparative evaluation of 36 commercial assays for detecting antibodies to HIV. Bull World Health Organ 69:753–760. [PMC free article] [PubMed] [Google Scholar]
- 10.Mullis CE, Laeyendecker O, Reynolds SJ, Ocama P, Quinn J, Boaz I, Gray RH, Kirk GD, Thomas DL, Quinn TC, Stabinski L. 2013. High frequency of false-positive hepatitis C virus enzyme-linked immunosorbent assay in Rakai, Uganda. Clin Infect Dis 57:1747–1750. doi: 10.1093/cid/cit602. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Biraro S, Mayaud P, Morrow RA, Grosskurth H, Weiss HA. 2011. Performance of commercial herpes simplex virus type-2 antibody tests using serum samples from sub-Saharan Africa: a systematic review and meta-analysis. Sex Transm Dis 38:140–147. doi: 10.1097/OLQ.0b013e3181f0bafb. [DOI] [PubMed] [Google Scholar]
- 12.Ng KW, Faulkner N, Cornish GH, Rosa A, Harvey R, Hussain S, Ulferts R, Earl C, Wrobel AG, Benton DJ, Roustan C, Bolland W, Thompson R, Agua-Doce A, Hobson P, Heaney J, Rickman H, Paraskevopoulou S, Houlihan CF, Thomson K, Sanchez E, Shin GY, Spyer MJ, Joshi D, O'Reilly N, Walker PA, Kjaer S, Riddell A, Moore C, Jebson BR, Wilkinson M, Marshall LR, Rosser EC, Radziszewska A, Peckham H, Ciurtin C, Wedderburn LR, Beale R, Swanton C, Gandhi S, Stockinger B, McCauley J, Gamblin SJ, McCoy LE, Cherepanov P, Nastouli E, Kassiotis G. 2020. Preexisting and de novo humoral immunity to SARS-CoV-2 in humans. Science 370:1339–1343. doi: 10.1126/science.abe1107. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Conklin SE, Martin K, Manabe YC, Schmidt HA, Miller J, Keruly M, Klock E, Kirby CS, Baker OR, Fernandez RE, Eby YJ, Hardick J, Shaw-Saliba K, Rothman RE, Caturegli PP, Redd AD, Tobian AAR, Bloch EM, Benjamin Larman H, Quinn TC, Clarke W, Laeyendecker O. 2020. Evaluation of serological SARS-CoV-2 lateral flow assays for rapid point-of-care testing. J Clin Microbiol 59:e02020-20. doi: 10.1128/JCM.02020-20. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Grabowski MK, Serwadda DM, Gray RH, Nakigozi G, Kigozi G, Kagaayi J, Ssekubugu R, Nalugoda F, Lessler J, Lutalo T, Galiwango RM, Makumbi F, Kong X, Kabatesi D, Alamo ST, Wiersma S, Sewankambo NK, Tobian AAR, Laeyendecker O, Quinn TC, Reynolds SJ, Wawer MJ, Chang LW. 2017. HIV prevention efforts and incidence of HIV in Uganda. N Engl J Med 377:2154–2166. doi: 10.1056/NEJMoa1702150. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Mohareb AM, Patel AV, Laeyendecker OB, Toerper MF, Signer D, Clarke WA, Kelen GD, Quinn TC, Haukoos JS, Rothman RE, Hsieh Y-H. 2021. The HIV screening cascade: current Emergency Department-based screening strategies leave many patients with HIV undiagnosed. J Acquir Immune Defic Syndr 87:e167–e169. doi: 10.1097/QAI.0000000000002609. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Jacobs J, Kühne V, Lunguya O, Affolabi D, Hardy L, Vandenberg O. 2020. Implementing COVID-19 (SARS-CoV-2) rapid diagnostic tests in sub-Saharan Africa: a review. Front Med (Lausanne) 30:557797. doi: 10.3389/fmed.2020.557797. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Weiss AJ, Wier LM, Stocks C, Blanchard J. 2006. Overview of emergency department visits in the United States, 2011: Statistical Brief #174, Healthcare Cost and Utilization Project (HCUP) statistical briefs. Agency for Healthcare Research and Quality, Rockville, MD. [PubMed] [Google Scholar]
- 18.Patel EU, Laeyendecker O, Hsieh YH, Rothman RE, Kelen GD, Quinn TC. 2016. Parallel declines in HIV and hepatitis C virus prevalence, but not in herpes simplex virus type 2 infection: a 10-year, serial cross-sectional study in an inner-city emergency department. J Clin Virol 80:93–97. doi: 10.1016/j.jcv.2016.05.003. [DOI] [PMC free article] [PubMed] [Google Scholar]