Abstract
Introduction
Brazil is the second largest country with COVID-19 positive cases worldwide. Due to the potent spread of the virus and the scarcity of kits and supplies, the Brazilian Ministry of Health has granted authorization for the use of kits available during this emergency, without an accurate evaluation of their performance. This study compared the performance and cost-effectiveness of seven molecular assays/kits available in São Paulo, Brazil, for SARS-CoV-2 diagnosis
Materials and methods
A total of 205 nasopharyngeal/oropharyngeal samples from suspected cases of COVID-19, were tested using the following assays: (i) GeneFinder COVID-19 plus RealAmp kit; (ii) 2019-nCoV RNA PCR-Fluorescence Probing, Da An Gene Co.; (iii) in-house RT-qPCR SARS-CoV-2 IAL; (iv) 2019-nCoV kit, IDT; (v) molecular SARS-CoV-2 (E) kit, Bio-Manguinhos; (vi) Allplex 2019-nCoV modified Assay, Seegene Inc, and (vii) Biomol one-step COVID-19 kit, IBMP. The criteria for determining a SARS-CoV-2 true positive result included the cycle threshold cut-off values, the characteristics of exponential/linear curves, the gene target diversity, and a positive result in at least two assays
Results
The overall sensitivity of the assays listed were GeneFinder 83.6%, Da An Gene 100.0%, IAL 90.4%, IDT 94.6%, Bio-Manguinhos 87.7%, Allplex 97.3%, and IBMP 87.7%. The minor sensitive gene target was RdRP. Although all assays had a Cohen's Kappa index ≥0.893, the best tests used multiplex assays identifying N-gene and/or E-gene targets
Conclusion
All assays tested accurate for diagnosis, but considering cost-effectiveness (cost, time consumption, number of samples tested, and performance), the in-house IAL assay was ideal for COVID-19 diagnosis in São Paulo, Brazil.
Keywords: Severe acute respiratory syndrome (SARS), SARS-CoV-2, Coronavirus disease 2019 (COVID-19), Reverse transcription-quantitative polymerase chain reaction (RT-qPCR), Diagnosis, Assay performances, Cost-effectiveness
1. Introduction
On 23 March 2021, the World Health Organization (WHO) reported more than 123.4 million confirmed cases of COVID-19 and more than 2.7 million deaths worldwide. Brazil alone, accounts for more than 12.1 million cases and 298 thousand deaths, making it the second country with the highest number of confirmed cases and deaths, according to worldwide rankings [1,2]. The Brazilian pandemic has raised serious concerns since the end of 2020, when the second wave of COVID-19 emerged in the country [2]. In Brazil, the state of São Paulo accounts for the highest number of COVID-19 cases, with 2,332,043 cases confirmed and 68,623 deaths [2], and São Paulo city reported 691,379 cases and 20,574 deaths [3].
The Instituto Adolfo Lutz (IAL) is a public health laboratory located in São Paulo city and a reference laboratory for the diagnosis and surveillance of respiratory viruses. In February 2020, the IAL started to diagnose COVID-19, searching for SARS-CoV-2 in samples from suspected patients. However, given the exponential spread of the virus and the scarcity of kits and supplies available in the market, the Ministry of Health of Brazil has granted authorization for the use of various commercially available kits, without an accurate evaluation of their performance.
To select the most appropriate kits/assays for SARS-CoV-2 molecular diagnosis in São Paulo, Brazil, that would help to reduce costs and provide quick, accurate results, especially during this pandemic, when health care expenditure is rising and resources are limited, we carried out a comparative study on the performance and cost-effectiveness of seven RT-qPCR kits/assays, including one kit modified for the laboratory conditions (Allplex kit) and one standardised in-house assay (duplex RT-qPCR), using SARS-CoV-2 samples obtained from IAL in São Paulo, Brazil.
2. Materials and methods
2.1. Study population
Two hundred and five nasopharyngeal/oropharyngeal samples collected between June and August 2020 and sent to IAL for COVID-19 diagnosis were obtained from a biorepository. These samples were randomly selected for RT-qPCR performance analyses, and the use of samples containing volumes larger than 2000 µL, was the only inclusion criteria. Three new RNA extractions (200 µL each) were conducted using the Biogene Extração de DNA/RNA viral kit (Bioclin Quibasa, MG, Brazil) according to the manufacturer's instructions. The pool of RNA extractions that allowed testing of all RT-qPCR assays was aliquoted, maintained at -70 °C, and used within four days after the extraction.
2.2. Laboratory methods
The kits/assays employed in the RT-qPCR COVID-19 comparative performance analyses are described in Table 1 .
Table 1.
Diagnostic kits/assays tested for the molecular diagnosis of SARS-CoV-2 infection in São Paulo, Brazil.
| Name of the Kit | Developer/Manufacturer (country) | Assay/kit study name | Gene target (labelled probe) | Methods | Ct values considered for positive results |
|---|---|---|---|---|---|
| GeneFinder COVID-19 plus RealAmp kit | Osang Healthcare Co. (Korea) | GeneFinder | RdRP (FAM), N (VIC), E (Texas Red), IC (Cy5) | Multiplex RT-qPCR | Ct up to 40 |
| 2019-nCoV RNA PCR-Fluorescent Probing | Da An Gene Co. (China) | Da An Gene | N (FAM), ORF 1ab (Yellow), IC (Cy5) | Multiplex RT-qPCR | Ct up to 40 |
| In-house RT-qPCR SARS-CoV-2, IAL | IAL (Brazil) | In-house IAL | E (FAM), RP (HEX) RdRP (FAM) | Duplex RT-qPCR Single RT-qPCR | Ct up to 38 |
| 2019-nCoV kit | IDT (USA) | IDT | N1 (FAM), N2 (FAM), RP (FAM) | Single RT-qPCR | Ct up to 39 |
| Molecular SARS-CoV-2 (E) Bio-Manguinhos | Bio-Manguinhos (Brazil) | Bio-Manguinhos | E (FAM), RP (FAM) | Single RT-qPCR | Ct up to 40 |
| Allplex 2019-nCoV Assay (modified) | Seegene (Korea) | Allplex | E (FAM), N (Quasar 670), RdRP (Cal Red 610), IC (HEX) | Multiplex RT-qPCR | Ct up to 40 |
| Biomol OneStep/COVID-19 IBMP | IBMP (Brazil) | IBMP | N (HEX/VIC), ORF 1ab (FAM), IC (ROX) | Multiplex RT-qPCR | Ct up to 40 |
Ct, cycle threshold ; RdRP, RNA-dependent RNA polymerase; ORF1ab, open-reading frame ORF1ab region ; N, nucleocapsid ; E, envelope gene; RP, human ribonuclease p; IC, internal control; FAM, fluorescein amidite; VIC, 2′‑chloro-7′-phenyl-1,4-dichloro-6-carboxyfluorescein; Cy5, Cyanine 5; HEX, hexachloro fluorescein; ROX, carboxy-X-rhodamine.
These kits differ in terms of composition, gene targets, labelled probes, methods, and interpretation of results, and were used according to the respective manufacturer's instructions, except for the Allplex kit, Seegene, which was modified. For the Allplex kit, 5 µL of RNA were used instead of the recommended 8 µL, and no internal control (IC) was added to the RNA extraction reaction, once it was standardized for use with the Seegene NIMBUS/STARlet equipment. These modifications were aimed at comparing the performance across all assays using the same RNA extraction product (quantity and quality) and the equipment available in our laboratory. Thus, we considered the IC of all other assays to assure the quality of the RNA to be amplified by the Allplex kit.
The results were interpreted according to the manufacturer's instructions. As the Ct limit of positivity (cut-off) value varies according to each developer/manufacturer's criteria across all the assays/kits (Ct varying from 38 to 40), we also analyzed the characteristics of the exponential/linear curves (slope and linear regression analysis).
The RT-qPCR reactions were conducted with 5 μL of RNA and in duplicate, except for the one with the Da An Gene kit, which was conducted individually due to its limited availability. In samples showing Ct values around the cut-off (half Ct above the cut-off value), the reactions were repeated in duplicate (depending on the availability of RNA samples and assays/kits), resulting in four results of several samples for analysis. According to the IDT kit manufacturer samples tested positive for only one N-gene target were considered inconclusive. The highest Ct value obtained in duplicate was considered for data interpretation.
The in-house SARS-CoV-2 IAL assay comprised two steps. The first employed a duplex RT-qPCR assay using oligonucleotides for the E-gene target and the human RNase P (RP-gene) as a control [4]. Thermocycler conditions used were as described by Corman and coworkers (Charité/Berlin Protocol) [5]. The probe for target E was labelled with fluorescein amidite (FAM) and the internal control RP was labelled with hexachloro fluorescein (HEX). Samples that were positive for target E were further tested for SARS-CoV-2 (second step) by RT-qPCR analysis using the RdRP oligonucleotides. Ct values ≤38 were considered positive.
All quantitative PCR assays were performed using the Bio-Rad CFX96 equipment (Bio-Rad, Hercules, CA, USA).
2.3. Statistical analysis
GraphPad Prism software version 5.03 (GraphPad, San Diego, CA, USA) was used for RT-qPCR analysis using Kruskal-Wallis method of variance (ANOVA), complemented with the Dunn's multiple comparison test, and the Mann-Whitney U test for the comparison of two assays. Any p-values ≤0.05 were considered statistically significant. In addition to the Ct values of the median, minimum and maximum, values at 25 and 75% of the mean and standard deviation (SD), and the coefficient of variation (CV) were calculated.
The Epi Info software, version 3.5.4 (Atlanta, GA, USA) was used for the comparative analyses of the final results of samples across the kits. The sensitivity, specificity, and Cohen's Kappa values were determined. To identify a SARS-CoV-2 true positive sample, the characteristics of the exponential/linear curves, and the genetic diversity among SARS-CoV-2 strains in Brazil were carefully analyzed, as they may affect the performance of molecular assays [6], [7], [8]. Our criteria required a positive result in at least two of the seven assays independently, based on its interaction with one or more gene targets.
Differences in sensitivity and specificity of the assays were statistically evaluated using the Chi-square test or Fisher's exact test, as appropriate. Statistical significance was set at p ≤ 0.05.
For a health economic evaluation, cost-effectiveness was calculated based on factors such as a correlation between both cost and outcome (in USD) of each assay/kit acquired in May 2020, number of samples analyzed using each assay/kit, and consequently the number of results released by plate, time consumption, ease of use, and assay sensitivity and specificity [9].
2.4. Ethical review
The study was approved by the IAL Technical and Scientific Council (CTC#21M-2020) and the Ethics Committee for Research under the Ministry of Health protocol number CAAE – 33282820.8.0000.0059. The data was analyzed while strictly adhering to patient confidentiality guidelines.
3. Results
The RT-qPCR SARS-CoV-2 assay performances of different kits were compared and summarized in Table 2 . The Da An Gene was highly sensitive, whereas GeneFinder was the least sensitive assay.
Table 2.
Results of the kits tested for the molecular diagnosis of SARS-CoV-2.
| Assay/Kit | Positive | Negative | Inconclusive | Total |
|---|---|---|---|---|
| GeneFinder (Korea) | 61 | 144 | 0 | 205 |
| Da An Gene Co. (China) | 76 | 129 | 0 | 205 |
| In-house IAL (Brazil) | 67a/49b | 138a/156b | 0 | 205 |
| IDT (USA) | 71 | 128 | 6 | 205 |
| Bio-Manguinhos (Brazil) | 64 | 141 | 0 | 205 |
| Allplex (Korea) | 71 | 134 | 0 | 205 |
| IBMP (Brazil) | 64 | 141 | 0 | 205 |
a, considering the first step; b, considering the second step.
Subsequent analyses were conducted to understand the discrepancy in results among assays and establish the criteria to consider the sample as a true positive.
When the Ct values of the positive samples were analyzed, the median was similar and independent of the gene target across all the SARS-CoV-2 molecular assays tested (Fig. 1 ). Statistically significant differences were detected when comparing the Ct values of RdRP-gene target in GeneFinder and in-house IAL assay, in-house IAL assay and Allplex, and for E-gene target when comparing Bio-Manguinhos and Allplex (Fig. 1).
Fig. 1.
Cycle threshold (Ct) values obtained from positive samples for different SARS-CoV-2 genes using seven RT-qPCR assays obtained from different developers/manufacturers. The number of samples positive for each gene corresponds to the number inside the box. Statistical significance is depicted by values * p < 0.05 and ** p ≤ 0.01 using the Kruskal Wallis test complemented with Dunn's Multiple Comparison Test.
The number of positive samples in each assay and their Ct values for each target are listed in Table 3 . Overall, the median Ct values varied from 27 to 31 and the mean Ct values from 27.34 to 30.17. Interestingly, the lower coefficient of variation was detected for the RdRP-gene target (Table 3, in bold), although this target showed minor positive results, while the N-gene target showed more positive results.
Table 3.
Analysis of the Cycle threshold (Ct) values obtained from each gene target using seven RT-qPCR assays employed for SARS-CoV-2 molecular diagnosis in São Paulo, Brazil.
| GeneFinder | Da An Gene | In-house IAL | IDT | Bio-Man guinhos | Allplex | IBMP | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| RdRP | N | E | N | ORF1ab | E | RdRP | N1 | N2 | E | E | RdRP | N | ORF1ab | N | |
| Number of positive samples | 47 | 61 | 56 | 76 | 73 | 67 | 49 | 71 | 71 | 64 | 69 | 65 | 70 | 64 | 63 |
| Minimum | 21,00 | 18,00 | 17,00 | 17,00 | 18,00 | 17,00 | 20,00 | 16,00 | 16,00 | 19,00 | 15,00 | 19,00 | 15,00 | 17,00 | 20,00 |
| 25% Percentile | 27,00 | 26,00 | 24,00 | 24,00 | 26,00 | 24,00 | 24,00 | 23,00 | 23,00 | 26,00 | 22,50 | 25,50 | 25,75 | 25,00 | 26,00 |
| Median | 31,00 | 29,00 | 27,00 | 28,00 | 30,00 | 28,00 | 28,00 | 27,00 | 28,00 | 31,00 | 27,00 | 30,00 | 29,50 | 28,50 | 30,00 |
| 75% Percentile | 33,00 | 33,50 | 31,00 | 34,00 | 35,50 | 32,00 | 31,00 | 33,00 | 33,00 | 35,00 | 32,00 | 34,50 | 35,00 | 33,00 | 33,00 |
| Maximum | 36,00 | 40,00 | 37,00 | 40,00 | 40,00 | 38,00 | 33,00 | 39,00 | 39,00 | 40,00 | 37,00 | 39,00 | 40,00 | 40,00 | 38,00 |
| Mean | 29,87 | 29,52 | 27,34 | 28,50 | 30,16 | 27,97 | 27,61 | 27,75 | 27,86 | 30,17 | 26,41 | 29,91 | 29,60 | 28,64 | 29,43 |
| Std. Deviation | 3848 | 5458 | 4933 | 6551 | 6392 | 5792 | 3931 | 6124 | 6220 | 5924 | 5939 | 5648 | 6289 | 5610 | 4613 |
| Std. Error | 0,5613 | 0,6988 | 0,6592 | 0,7515 | 0,7482 | 0,7076 | 0,5615 | 0,7268 | 0,7382 | 0,7405 | 0,7150 | 0,7005 | 0,7516 | 0,7012 | 0,5812 |
| Lower 95% CI of mean | 28,74 | 28,13 | 26,02 | 27,00 | 28,67 | 26,56 | 26,48 | 26,30 | 26,39 | 28,69 | 24,98 | 28,51 | 28,10 | 27,24 | 28,27 |
| Upper 95% CI of mean | 31,00 | 30,92 | 28,66 | 30,00 | 31,66 | 29,38 | 28,74 | 29,20 | 29,33 | 31,65 | 27,83 | 31,31 | 31,10 | 30,04 | 30,59 |
| Coefficient of variation | 12.88% | 18.49% | 18.04% | 22.99% | 21.19% | 20.71% | 14.24% | 22.07% | 22.33% | 19.64% | 22.49% | 18.88% | 21.25% | 19.59% | 15.68% |
RdRP, RNA-dependent RNA polymerase; N, nucleocapsid; E, envelope gene; ORF1ab, open-reading frame ORF1ab region; Std., standard; CI, confidence interval.
Meticulous analysis of results using the seven assays/kits revealed that they were negative in 125 samples and positive in 49 samples. In 31 samples, inconclusive results were detected, which are presented in Table 4 , confirming that RdRP was the minor sensitive target.
Table 4.
Cycle threshold (Ct) values of samples tested for SARS-CoV-2.
| GeneFinder |
Da An Gene |
In-house IAL 1st step |
In-house IAL 2nd step |
|||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| RdRp | N | E | Result | N | ORF1ab | Result | E | Result | RdRP | Result | ||
| 1 | 0/0/0/0 | 35/35/35/34 | 0/35/0/0 | Positive | 33 | 35 | Positive | 34/33 | Positive | 0/0/0/0 | Negative | |
| 2 | 0/0/0/0 | 36/36/39/36 | 0/0/0/0 | Positive | 34 | 36 | Positive | 37/37 | Positive | 0/0/0/0 | Negative | |
| 3 | 0/0/0/0 | 0/40/41/0 | 0/0/0/0 | Negative | 37 | 39 | Positive | 0/0/36/36 | Positive | 0/0/0/0 | Negative | |
| 4 | 0/0/0/0 | 34/34/35/35 | 0/0/37/35 | Positive | 32 | 35 | Positive | 0/0/33/33 | Positive | 0/0/0/0 | Negative | |
| 5 | 0/0 | 0/39 | 0/0 | Negative | 39 | 40 | Positive | 0/0 | Negative | 0/0 | Negative | |
| 6 | 0/0 | 34/34 | 36/34 | Positive | 33 | 36 | Positive | 33/33 | Positive | 0/0/0/0 | Negative | |
| 7 | 0/0 | 0/0 | 0/0 | Negative | 40/39/38 | 41/41/39 | Positive | 39/0/0/38 | Negative | 0/0 | Negative | |
| 8 | 0/0/0/0 | 0/38/0/0 | 0/0/0/0 | Negative | 38/36 | 42/40 | Positive | 37/37 | Positive | 0/0/0/0 | Negative | |
| 9 | 0/0 | 0/39 | 0/0 | Negative | 39/40/40 | 0/41/0 | Positive | 0/39/38/0 | Negative | 0/0 | Negative | |
| 10 | 0/0/0/0 | 37/36/38/36 | 37/0/0/0 | Positive | 34 | 28 | Positive | 36/35 | Positive | 0/0/0/0 | Negative | |
| 11 | 0/0/0/0 | 37/36/38/36 | 0/0/0/0 | Positive | 35 | 38 | Positive | 36/35 | Positive | 0/0/0/0 | Negative | |
| 12 | 0/0 | 35/34 | 34/35 | Positive | 34 | 38 | Positive | 34/34 | Positive | 0/0/0/0 | Negative | |
| 13 | 0/0/0/0 | 37/36/38/0 | 0/0/0/0 | Negative | 34 | 39 | Positive | 36/35 | Positive | 0/0/0/0 | Negative | |
| 14 | 0/0/0/0 | 37/40/36/38 | 41/0/0/0 | Positive | 36 | 39 | Positive | 36/36 | Positive | 0/0/0/0 | Negative | |
| 15 | 0/0 | 34/34 | 34/33 | Positive | 32 | 35 | Positive | 34/34 | Positive | 0/0/0/0 | Negative | |
| 16 | 0/0 | 0/0 | 0/0 | Negative | 39/0 | 41/0 | Negative | 0/39/40/40 | Negative | 0/39/40/40 | Negative | |
| 17 | 34/34 | 33/33 | 33/32 | Positive | 31 | 33 | Positive | 32/31 | Positive | 0/0/0/0 | Negative | |
| 18 | 0/0 | 35/35 | 33/33 | Positive | 34 | 36 | Positive | 34/34 | Positive | 0/0/0/0 | Negative | |
| 19 | 0/0/0/0 | 38/37/38/0 | 0/0/0/0 | Negative | 37 | 40 | Positive | 37/37 | Positive | 0/0/0/0 | Negative | |
| 20 | 0/0/0/0 | 37/38/0/37 | 36/0/0/0 | Negative | 38 | 39 | Positive | 38/38 | Positive | 0/0/0/0 | Negative | |
| 21 | 0/0 | 0/0 | 0/0 | Negative | 39/0 | 41/0 | Negative | 39/0/39/0 | Negative | 0/0 | Negative | |
| 22 | 0/0 | 40/0 | 0/0 | Negative | 39 | 40 | Positive | 39/0/0/40 | Negative | 0/0 | Negative | |
| 23 | 0/0 | 0/0 | 0/0 | Negative | 39 | 39 | Positive | 0/39/40/0 | Negative | 0/0 | Negative | |
| 24 | 0/0/0/0 | 38/0/40/0 | 0/0/0/0 | Negative | 36 | 38 | Positive | 39/38/38/39 | Negative | 0/0 | Negative | |
| 25 | 35/36 | 32/33 | 32/32 | Positive | 31 | 32 | Positive | 31/31 | Positive | 0/0/0/0 | Negative | |
| 26 | 0/0 | 0/0 | 0/0 | Negative | 0 | 0 | Negative | 38/38 | Positive | 0/0/0/0 | Negative | |
| 27 | 0/0 | 0/0 | 0/0 | Negative | 0/40/40 | 40/40/0 | Positive | 0/0 | Negative | 0/0 | Negative | |
| 28 | 0/0 | 39/0 | 0/0 | Negative | 37 | 39 | Positive | 0/38 | Negative | 0/0 | Negative | |
| 29 | 0/0 | 0/40 | 0/0 | Negative | 37 | 39 | Positive | 0/38/39/40 | Negative | 0/0 | Negative | |
| 30 | 0/0 | 0/0 | 0/0 | Negative | 0 | 41 | Negative | 0/0 | Negative | 0/0 | Negative | |
| 31 | 0/0 | 39/0 | 0/0 | Negative | 38/39/39/39 | 41/41/41/0 | Positive | 38/0/40/40 | Negative | 0/0 | Negative | |
| IDT |
Bio-Manguinhos |
Allplex |
IBMP |
|||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| N1 | N2 | Result | E | Result | E | RdRP | N | Result | ORF1ab | N | Result | |
| 1 | 33/33 | 33/32 | Positive | 38/36 | Positive | 32/32 | 37/39 | 36/36 | Positive | 35/35 | 35/35 | Positive |
| 2 | 34/34 | 34/34 | Positive | 36/38 | Positive | 34/34 | 38/38 | 36/36 | Positive | 35/36 | 36/37 | Positive |
| 3 | 35/35 | 36/36 | Positive | 39/0/37/37 | Positive | 34/34 | 38/38 | 38/37 | Positive | 39/38 | 38/38 | Positive |
| 4 | 32/32 | 33/32 | Positive | 36/36 | Positive | 32/31 | 36/36 | 33/33 | Positive | 35/35 | 33/33 | Positive |
| 5 | 38/0/0/0 | 38/39/38/39 | Inconclusive | 0/39/0/0 | Negative | 37/40/37/0 | 0/40/0/39 | 0/0/0/0 | Negative | 0/0 | 0/38 | Negative |
| 6 | 33/32 | 33/33 | Positive | 0/0/35/36 | Positive | 32/32 | 36/36 | 33/34 | Positive | 34/33 | 34/33 | Positive |
| 7 | 38/39/0/37 | 39/0/38/37 | Inconclusive | 38/0 | Negative | 36/36 | 0/40 | 0/0 | Negative | 0/0 | 0/0 | Negative |
| 8 | 38/36 | 37/37 | Positive | 0/0/38/38 | Positive | 0/35/37/36 | 40/0/0/0 | 37/37/38/40 | Positive | 39/37 | 37/36 | Positive |
| 9 | 0/37/37/38 | 39/36/0/39 | Inconclusive | 0/0 | Negative | 36/37 | 0/37 | 39/37 | Positive | 0/0/0/0 | 37/37/0/0 | Negative |
| 10 | 33/34 | 34/34 | Positive | 40/39 | Positive | 33/33 | 35/39 | 35/34 | Positive | 35/34 | 35/35 | Positive |
| 11 | 34/35 | 35/35 | Positive | 0/0/37/38 | Positive | 33/33 | 36/36 | 34/34 | Positive | 37/36 | 34/35 | Positive |
| 12 | 34/33 | 33/33 | Positive | 34/34 | Positive | 32/32 | 35/36 | 33/34 | Positive | 34/34 | 34/34 | Positive |
| 13 | 34/34 | 34/35 | Positive | 0/0/0/38 | Negative | 33/33 | 37/38 | 34/35 | Positive | 39/38 | 34/35 | Positive |
| 14 | 35/34 | 35/36 | Positive | 36/35 | Positive | 34/34 | 36/38 | 34/35 | Positive | 0/39/0/38 | 35/36/0/36 | Negative |
| 15 | 32/33 | 33/33 | Positive | 33/34 | Positive | 32/31 | 35/35 | 32/32 | Positive | 34/34 | 34/34 | Positive |
| 16 | 39/38 | 39/39 | Positive | 0/0 | Negative | 0/38 | 0/0 | 38/0 | Negative | 39/0 | 0/0 | Negative |
| 17 | 31/31 | 31/31 | Positive | 33/33 | Positive | 30/30 | 34/34 | 32/32 | Positive | 33/33 | 34/34 | Positive |
| 18 | 32/32 | 33/32 | Positive | 35/36 | Positive | 33/33 | 36/36 | 33/34 | Positive | 0/37/0/0 | 35/36/37/37 | Negative |
| 19 | 36/36 | 36/36 | Positive | 0/0/37/39 | Positive | 35/35 | 36/37 | 36/37 | Positive | 36/36 | 36/38 | Positive |
| 20 | 37/38 | 36/37 | Positive | 0/0/0/38 | Negative | 36/36 | 39/37 | 38/38 | Positive | 0/0/39/40 | 36/38/0/38 | Positive |
| 21 | 38/0/37/0 | 39/37/39/38 | Inconclusive | 0/0 | Negative | 37/0/0/39 | 0/38/0/39 | 0/39/39/41 | Negative | 38/0 | 0/0 | Negative |
| 22 | 38/38/0/0 | 0/39/0/38 | Negative | 0/0 | Negative | 38/0/0/0 | 0/0/0/39 | 38/38/40/40 | Positive | 39/0/0/0 | 0/38/38/0 | Negative |
| 23 | 38/39/39 | 0/38/39 | Positive | 0/0 | Negative | 0/37 | 0/0 | 35/0 | Negative | 0/0 | 0/0 | Negative |
| 24 | 36/36 | 36/37 | Positive | 38/0/0/0 | Negative | 37/35 | 0/0 | 38/37 | Positive | 0/0/0/0 | 0/0/37/38 | Negative |
| 25 | 31/31 | 30/30 | Positive | 33/34 | Positive | 29/29 | 34/34 | 32/31 | Positive | 33/34 | 34/34 | Positive |
| 26 | 39/0/0 | 39/39/40 | Negative | 0/0 | Negative | 0/0/0/0 | 0/0/0/0 | 37/38/0/0 | Negative | 0/0 | 0/0 | Negative |
| 27 | 0/0 | 0/0 | Negative | 0/0 | Negative | 0/0 | 0/0 | 37/0 | Negative | 0/0 | 0/0 | Negative |
| 28 | 37/0/38 | 37/36/39 | Positive | 40/0 | Negative | 37/37/36/0 | 39/0/0/0 | 36/0/38/39 | Positive | 0/0/0/0 | 41/38/38/0 | Negative |
| 29 | 38/37/0 | 39/40/38 | Inconclusive | 39/0/0/38 | Negative | 36/36 | 0/0 | 36/36 | Positive | 0/0 | 0/0 | Negative |
| 30 | 39/0/38/38 | 39/0/0/38 | Inconclusive | 0/0 | Negative | 0/37/40/0 | 0/0/0/0 | 38/38/40/0 | Negative | 38/38/0/0 | 0/0/0/38 | Negative |
| 31 | 39/38 | 39/37 | Positive | 0/0 | Negative | 0/0 | 0/0 | 39/0 | Negative | 0/38/0/0 | 38/0/0/0 | Negative |
The in-house IAL assay showed 18 of the 31 RdRP negative samples as positive in the first step and/or in other assays/kits (samples code 1 to 4, 6, 8, 10 to 15, 17 to 20, 25, and 26, Table 4). Of these 18 samples, 10 were positive in 7 assays, 5 in 6 assays, 2 in 5 assays, and one was positive only in the first step of this in-house assay. These results corroborate the minor sensitivity of RdRP-gene target when employed in the second step of the in-house IAL assay. The results obtained by the GeneFinder kit, show RdRP negative results and positive results for other targets in samples with codes 1, 2, 4, 6, 10, 11, 12, 14, 15, and 18 (Table 4). Therefore for the in-house IAL assay, the results of the first step are considered as final, in accordance with the revised recommendation of the Pan American Health Organization (PAHO/WHO) using the Charité protocol [10].
The inconclusive RT-qPCR results were detected in six samples using the IDT assay (samples code 5, 7, 9, 21, 29, and 30, Table 4), which employs only the N-gene target.
Taking into consideration the deficiencies in detecting some targets and the differences in analytical sensitivity of the RT-qPCR assays [11,12], to calculate the sensitivity and specificity of assays, we established the following criteria to identify true positive and negative samples.
First, we re-analyzed the number of negative samples listed in Table 4 that were positive in other assays/kits, including the inconclusive results of the IDT kit. The result of these analyses are shown in Fig. 2 .
Fig. 2.
Samples negative for SARS-CoV-2 in each molecular assay that resulted in positive and/or inconclusive outcomes in other assays. The number inside the spheres denotes the number of samples categorised as: Pos 6, positive in 6 assays; Pos 5, positive in 5 assays; Pos 4, positive in 4 assays; Pos 3, positive in 3 assays; Pos 2, positive in 2 assays; Pos 1, positive in one assay; Pos 2 + Inc, positive in 2 assays plus one inconclusive result; Pos 1 + Inc, positive in one assay plus one inconclusive; Inc, one inconclusive.
We identified characteristics of the exponential/linear curves in samples that resulted in Ct values close to the cut-off value, and in samples presenting inconclusive results. Using both analyses, we identified the samples that were negative in all assays (n = 125) as true negative, Pos 1 (sample 16, IDT; sample 26, in-house IAL; sample 27, Da An Gene); Pos 1 + Inc. (samples 5 and 7, Da An Gene, and IDT) and Inc. (samples 21 and 30, IDT) (Table 4) and, the sample that was positive in at least two assays was considered as a true positive (n = 73).
Thus, the final results of SARS-CoV-2 obtained from 205 samples tested by each assay/kit, along with its sensitivity, specificity, and Cohen's kappa indices, are presented in Table 5 .
Table 5.
Sensitivity and specificity of kit/assays used to test SARS-CoV-2 samples.
| Assay/Kit | True Positive | Sensitivity | Specificity | Kappa index (95% CI) |
|---|---|---|---|---|
| GeneFinder (Korea) | 61 | 83.56% | 100.00% | 0.893 (0.834–0.952) |
| Da An Gene Co. (China) | 73 | 100.0% | 97.78% | 0.975 (0.948–1.003) |
| In-house IAL (Brazil) | 66 | 90.41% | 99.25% | 0.931 (0.884–0.978) |
| IDT (USA) | 70 | 94.59% | 99.25% | 0.958 (0.922–0.994) |
| Bio-Manguinhos (Brazil) | 64 | 87.67% | 100.00% | 0.921 (0.871–0.972) |
| Allplex modified (Korea) | 71 | 97.26% | 100.00% | 0.983 (0.960–1.006) |
| IBMP (Brazil) | 64 | 87.67% | 100.00% | 0.921 (0.871–0.972) |
CI, confidence interval.
The sensitivity of the assays varied from 83.6% to 100.0%. Differences in assay sensitivity were detected between the following, GeneFinder and Da An Gene (p < 0.001), GeneFinder and IDT (p = 0.034), GeneFinder and Allplex (p = 0.005), Da An Gene and in-house IAL (p = 0.020), Da An Gene and Bio-Manguinhos (p = 0.006), and Allplex and IBMP (p = 0.028) assays/kits. No significant differences in specificity were observed. All assays performed well as per the Cohen's Kappa index analysis (all above 0.893). Considering both sensitivity and specificity, the best assay kits were the Allplex, followed by the Da An Gene.
Data summarizing the cost-effectiveness of the assays/kits are shown in Table 6 . According to the purchase value of kits/assays in May 2020, and other parameters (presented in Table 6), the most economic kit was the in-house IAL first-round assay.
Table 6.
Cost-effectiveness of seven SARS-CoV-2 molecular diagnostic assays/kits available in São Paulo, Brazil.
| Parameters | GeneFinder | Da An Gene | In-house IAL 1st step | IDT | Bio-Manguinhos | Allplex | IBMP |
|---|---|---|---|---|---|---|---|
| Number of targets | 4 | 3 | 2 | 3 | 2 | 4 | 3 |
| Number of reactions | 1 | 1 | 1 | 3 | 2 | 1 | 1 |
| Amount of RNA (µL) for final result | 5 | 5 | 5 | 15 | 10 | 5 | 5 |
| Cost of the assay in American dollar ($) | 1112.93 | 1550.40 | 1820.00a | 2461.85a | 1681.70 | 701.36 | 1191.49 |
| Number of tests in each assay/kit | 100 | 96 | 500 | 500 | 96 | 100 | 96 |
| Cost of one test in American dollar ($) | 11.13 | 16.15 | 3.64 | 4.92 | 17.52 | 7.01 | 12.41 |
| Number of samples tested per plateb | 96 | 96 | 96 | 32 | 48 | 96 | 96 |
| Limitations | 16.4% False-negative results | 2.2% False-positive results | 9.6% False-negative results | 5.4% False- negative including inconclusive results | 12.3% False- negative results | 2.7% False-negative results | 12.3% False-negative results |
aIncluding SuperScript III Platinum One-Step qRT-PCR System, 500 rxs (Invitrogen, Thermo Fisher Scientific); b including positive and negative controls.
4. Discussion
Brazil has a population of 212,821,952 inhabitants, of which 46,289,333 belong to the state of São Paulo [13], and 12,325,232 inhabit São Paulo city [14]. The state and municipality are responsible for the major number of cases and deaths due to COVID-19 in Brazil [2,3]. Hence, a large number of samples are sent daily to the IAL for analysis, emphasizing the need for optimum testing within minimal time and complete accuracy of results. At the beginning of the pandemic, only the IDT and Bio-Manguinhos assays/kits (both using single RT-qPCR reactions) were available, therefore, we decided to standardize one duplex RT-qPCR using the E- and RP-gene targets. We tested RdRP levels to confirm SARS-CoV-2; however, the second strategy was inefficient. The possible success and performance of the in-house IAL first step assay and the poor performance of the second step assay have also been confirmed by recently published studies [[6], [7], [8],15].
Briefly, Ramirez et al. [6] highlighted the role of E-gene as a screening target for the diagnosis of SARS-CoV-2. Using in silico analysis of 373 genomes from South America (95 from Brazil), they identified mutations in the primer-probe binding sites to RdRP, N, and E genes. The results revealed abundant genomic diversity in the RdRP and N genes, in contrast to the E gene. This highlights a probable effect on the false-negative results when employing the RdRP gene as the target, and supports the use of the E gene for SARS-CoV-2 screening in South America, thereby corroborating the results obtained from our study conducted in São Paulo, Brazil.
Vogels et al. [7] compared the analytical efficiencies and sensitivities of the primer-probe sets of four most common SARS-CoV-2 RT-qPCR assays developed by the China Center for Disease Control (China CDC), United States CDC (US CDC), Charité Institute of Virology, Universitätsmedizin Berlin (Charité), and Hong Kong University (HKU). They concluded that the E-gene Sarbeco (Charité) was highly sensitive, and the RdRP-SARS gene (Charité) was the least sensitive. The authors attributed the deficiency of the RdRP-gene target to a mismatch in the reverse primer, when compared to the majority of RdRP-genes of SARS-CoV-2 circulating in the USA. Nevertheless, they detected similar analytical sensitivities of the four assays, as observed in the present study. The minor sensitivity of the RdRP primer-probe was further confirmed by Nalla et al. [15] which compared the N-, RdRP-, and E-gene primer-probe sets described by Corman et al. [5], and N1, N2, and N3 primer-probe sets developed by the CDC-USA to diagnose SARS-CoV-2 clinical samples, and detected the E and N2 targets as the most sensitive [15]. In addition, based on in silico alignments, Pillonel et al. showed the use of an incorrect degenerate base (S instead of R) in the design of the reverse RdRP primer described by Corman et al. [8]. These studies reinforce the need to confirm the low sensitivity of such RdRP primers/probe sets for SARS-CoV-2 molecular diagnosis.
In the present study, we were unable to sequence the SARS-CoV-2 false-negative samples. However, a recent study at IAL showed the complete genomic sequencing of 29 SARS-CoV-2 samples that resulted in RdRP-negative outcomes using the Charité-Berlin protocol and SARS-CoV-2 positivity using the Allplex kit. These sequences were deposited in the Global Initiative on Sharing Avian Influenza Data - GISAID (accession numbers EPI_ISL_693215 to EPI_ISL_693243), confirming that the RdRP-gene target employed in the Charité-Berlin protocol is inaccurate for SARS-CoV-2 detection. Studies identifying mutations in regions harboring primer-probe binding sites are currently in progress.
On 19 March 2020, the WHO recommended considering a COVID-19 positive result when at least two specific gene targets were positive on nucleic acid amplification tests (NAAT), or when one target was positive and the COVID-19 virus was identified by sequencing (partial or whole genomic sequence) [16]. On 30 March 2020, PAHO guidelines recommended the use of E and RdRP genes (Charité-Berlin protocol) for diagnosis, prioritizing the E gene for single-target testing [17]. On 8 July 2020, these guidelines were revised and an algorithm that employs only the E-gene for confirming COVID-19 virus infection was recommended, as the only Sarbecovirus that currently circulates in humans is the SARS-CoV-2 [10]. These guidelines and prior studies altogether support the use of in-house IAL first-step RT-qPCR assays for COVID-19 molecular diagnosis.
In conclusion, although the GeneFinder and Allplex detected the major number of gene targets (four) in a single multiplex assay, the GeneFinder resulted in low sensitivity and increased costs. In contrast, the Da An Gene kit presented the highest sensitivity, lacked specificity, and is also expensive. The Bio-Manguinhos and IBMP assays/kits, of Brazilian origin had the same performance, in addition to being expensive however; the IBMP has an advantage as a multiplex assay.
When compared to the in-house IAL and IDT assays, although both were inexpensive, the IDT kit takes longer as it employs three single RT-qPCR reactions in contrast to the in-house IAL assay that employs one reaction (32 tests versus 96 tests per plate, respectively). Interestingly, although the IAL assay showed low sensitivity when compared to other assays, (except for the GeneFinder assay) a Ct value cut-off of 40 would improve assay sensitivity to 94.8%. In retrospect, the stringent criteria for a true-positive adopted by us (Ct up to 38, the minor of all assays) needs to be revised considering the characteristics of the exponential/linear curves, and the new kit launched in the national market by Bio-Manguinhos-FIOCRUZ-RJ, Brazil (Kit Molecular SARS-CoV-2 Bio-Manguinhos E/RP) which employs the E/RP RT-qPCR duplex assay, and a cut-off value of ≤40 for the E-gene target.
Concerning the Allplex kit, despite the problems pointed by several laboratories in Brazil (unpublished data), and also detected by us regarding its internal control (Ct values more than 35, and 20% of false-negative results when employed in the Loccus Extracta RNA equipment and using 10 µL of IC input), we have to consider that this assay was standardized for the conditions and the equipment of the Seegene manufacturer, and more recently for four kits/equipments of RNA extractions: two automatized (SEEPREP 32, Seegene, and NucliSENS easyMAG, BioMérieux), and two manual RNA extractions kits (Ribospin vRD kit, GeneAII, and QIAamp DSP Virus Spin Kit, Qiagen). None of these kits/equipments were available in our laboratory. However, this kit performed well against other genetic targets, showing the best performance among all assay kits tested (Cohen's Kappa index= 0.983). Therefore, this kit could be used without an IC. This assay is cost-effective and is currently in use at IAL and other public health laboratories in Brazil, using an RNA control, in a single or multiplex RT-qPCR assay (unpublished data). Recently, this kit was used to test the community spread of COVID-19 among laboratory staff and other employees of IAL, identifying differences in Ct values based on the clinical status of SARS-CoV-2 infection: symptomatic, presymptomatic, and asymptomatic [18].
Another aspect to consider is the limit of detection (LoD) of the RT-qPCR assays using RNA SARS-CoV-2 standard samples and clinical samples [11,12], which could explain our findings. We did not exclude the lack of SARS-CoV-2 positivity when the viral loads of the samples were under the LoD of the assays.
Besides the limitations of assays reported in the literature [[6], [7], [8],[10], [11], [12]] and in this study, the results showed a good performance of all assays/kits (all Cohen's Kappa index above 0.893), enabling their use in routine diagnosis. Moreover, the results obtained proved that RdRP is not the best gene target by which patients suspected with COVID-19, can be identified in São Paulo, Brazil.
In conclusion, multiplex RT-qPCR assays are the choice for COVID-19 diagnosis and can cater to the current demand for testing in São Paulo, Brazil. The in-house IAL duplex assay proved to be the best method, with respect to cost, time, sensitivity, reagent consumption, ease of performance, immediate results, and accuracy and can be effectively used in the current SARS-CoV-2 pandemic plaguing São Paulo, Brazil.
Declaration of Competing Interest
The authors declare no conflicts of interest.
Acknowledgments
This study was supported by grants from Coordenação Geral de Laboratórios de Saúde Pública (CGLAB), Secretaria de Vigilância em Saúde (SVS), Ministério da Saúde; Instituto Butantan (IB), and Instituto Adolfo Lutz (IAL).
The authors are indebted to the COVID-19 IAL-study group, in special to Adriana Bugno (director of IAL) and Adriano Abbud (director of Centro de Respostas Rápidas, IAL) for providing funds and conditions to carry out the present study.
The funders had no role in study design, data collection, and interpretation, or the decision to submit the work for publication.
References
- 1.World Health Organization, WHO coronavirus disease (COVID-19) dashboard. Data last updated 2021/3/23. Available: https://covid19.who.int/(accessed 24 March 2021).
- 2.Brasil - Ministério da Saúde, 23/03/2021; Painel Coronavirus. Available: https://covid.saude.gov.br/ (accessed 24 March 2021).
- 3.Cidade de São Paulo Saúde - Boletim Diário COVID-19, 23/03/2021, No. 362. Available: https://www.prefeitura.sp.gov.br/cidade/secretarias/upload/saude/20210323_boletim_covid19_diario_v2.pdf (accessed 24 March 2021).
- 4.Emery S.L., Erdman D.D., Bowen M.D., Newton B.R., Winchell J.M., Meyer R.F., Tong S., Cook B.T., Holloway B.P., McCaustland K.A., Rota P.A., Bankamp B., Lowe L.E., Ksiazek T.G., Bellini W.J., Anderson L.J. Real-time reverse transcription-polymerase chain reaction assay for SARS-associated coronavirus. Emerg. Infect. Dis. 2004;10(2):311–316. doi: 10.3201/eid1002.030759. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Corman V.M., Landt O., Kaiser M., Molenkamp R., Meijer A., Chu D.K.W., Bleicker T., Brünink S., Schneider J., Schmidt M.L., Mulders D.G.J.C., Haagmans B.L., van der Veer B., van den Brink S., Wijsman L., Goderski G., Romette J.L., Ellis J., Zambon M., Peiris M., Goossens H., Reusken C., Koopmans M.P.G., Drosten C. Detection of 2019 novel coronavirus (2019-nCoV) by real-time RT-PCR. Eurosurveillance. 2020;25 doi: 10.2807/1560-7917.ES.2020.25.3.2000045. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Ramírez J.D., Muñoz M., Hernández C., Flórez C., Gomez S., Rico A., Pardo L., Barros E.C., Paniz-Mondolfi A.E. Genetic diversity among SARS-CoV2 strains in South America may impact performance of molecular detection. Pathogens. 2020;9(7):580. doi: 10.3390/pathogens9070580. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Vogels C.B.F., Brito A.F., Wyllie A.L., Fauver J.R., Ott I.M., Kalinich C.C., Petrone M.E., Casanovas-Massana A., Catherine Muenker M., Moore A.J., Klein J., Lu P., Lu-Culligan A., Jiang X., Kim D.J., Kudo E., Mao T., Moriyama M., Oh J.E., Park A., Silva J., Song E., Takahashi T., Taura M., Tokuyama M., Venkataraman A., Weizman O.E., Wong P., Yang Y., Cheemarla N.R., White E.B., Lapidus S., Earnest R., Geng B., Vijayakumar P., Odio C., Fournier J., Bermejo S., Farhadian S., Dela Cruz C.S., Iwasaki A., Ko A.I., Landry M.L., Foxman E.F., Grubaugh N.D. Analytical sensitivity and efficiency comparisons of SARS-CoV-2 RT-qPCR primer-probe sets. Nat. Microbiol. 2020;5(10):1299–1305. doi: 10.1038/s41564-020-0761-6. Oct. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Pillonel T., Scherz V., Jaton K., Greub G., Bertelli C. Letter to the editor: SARS-CoV-2 detection by real-time RT-PCR. Eurosurveillance. 2020;25(21) doi: 10.2807/1560-7917.ES.2020.25.21.2000880. pii=2000880. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Rudmik L., Drummond M. Health economic evaluation: important principles and methodology. Laryngoscope. 2013;123:1341–1347. doi: 10.1002/lary.23943. [DOI] [PubMed] [Google Scholar]
- 10.Pan American Health Organization. PAHO. Laboratory guidelines for the detection and diagnosis of COVID-19 virus infection, 8 July 2020 [Internet]. Pan American Health Organization: Washington, DC, USA; 2020 [cited 2020 December 7]. Available from: https://iris.paho.org/handle/10665.2/52458
- 11.Kubina R., Dziedzic A. Molecular and serological tests for COVID-19. A comparative review of SARS-CoV-2 Coronavirus laboratory and point-of-care diagnostics. Diagnostics. 2020;10:434. doi: 10.3390/diagnostics10060434v. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Loeffelholz M.J., Tang Y.W. Laboratory diagnosis of emerging human coronavirus infections – the state of the art. Emerg. Microbes Infect. 2020;9(1):747–756. doi: 10.1080/22221751.2020.1745095. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Instituto Brasileiro de Geografia e Estatística - IBGE. Cidades e Estados: São Paulo 2020a. Available: https://www.ibge.gov.br/cidades-e-estados/sp.html. (accessed 3 March 2021).
- 14.Instituto Brasileiro de Geografia e Estatística - IBGE. Cidades e Estados: São Paulo, SP 2020b. Available: https://cidades.ibge.gov.br/brasil/sp/sao-paulo/panorama. (accessed 3 March 2021).
- 15.Nalla A.K., Casto A.M., Huang M.L.W., Perchetti G.A., Sampoleo R., Shrestha L., Wei Y., Zhu H., Jerome K.R., Greninger A.L. Comparative performance of SARS-CoV-2 detection assays using seven different primer-probe sets and one assay kit. J. Clin. Microbiol. 2020;58 doi: 10.1128/JCM.00557-20. e00557-20. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.World Health Organization. WHO. Laboratory testing for 2019 novel coronavirus (2019-nCoV) in suspected human cases: interim guidance, 19 march 2020 [Internet]. WHO; 2020. Available: https://apps.who.int/iris/bitstream/handle/10665/331501/WHO-COVID-19-laboratory-2020.5-eng.pdf?sequence=1&isAllowed=y (accessed 3 March 2021).
- 17.Pan American Health Organization . Pan American Health Organization; Washington, DC, USA: 2020. Laboratory Guidelines For the Detection and Diagnosis of COVID-19 Virus Infection. PAHO. 30 March 2020. [Google Scholar]
- 18.Lopez-Lopes G.I.S., Ahagon C.M., Bonega M.A., Santos F.P., Santos K.C.O., Cilli A., Prado L.S., Borges da Silva D.B., Borges da Luz N., Saraceni C.P., Afonso A.M.S., Timenetsky M.C., Brigido L.F.M. medRxiv; 2020. Throat Wash as a Source of SARS-CoV-2 RNA to Monitor Community Spread of COVID-19. Posted August 01. [DOI] [Google Scholar]