Low clinical performance of the Isopollo COVID-19 detection kit (M Monitor, South Korea) for RT-LAMP SARS-CoV-2 diagnosis: A call for action against low quality products for developing countries

Abstract

Background

Multiple molecular kits are available for the diagnosis of severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) worldwide, with many lacking proper clinical evaluation due to the emergency caused by the coronavirus disease 2019 (COVID-19) pandemic, particularly in developing countries.

Methods

This study was conducted to evaluate the clinical performance of the Isopollo COVID-19 detection kit (M Monitor, South Korea) for reverse transcription loop-mediated isothermal amplification (RT-LAMP) SARS-CoV-2 diagnosis, using the SARS-CoV-2 reverse transcription polymerase chain reaction (RT-PCR) protocol as the gold standard.

Results

A total of 220 clinical samples were included in the study; 168 samples were SARS-CoV-2-positive and 52 samples were SARS-CoV-2-negative according to the SARS-CoV-2 RT-PCR protocol. For the Isopollo COVID-19 detection kit, only 104 out of 168 samples were SARS-CoV-2-positive. This result shows a low clinical performance, with sensitivity of 61.9% for the evaluated RT-LAMP assay.

Conclusions

Proper clinical performance evaluation studies by regulatory agencies in developing countries such as Ecuador should be mandatory prior to clinical use authorization of SARS-CoV-2 diagnosis kits, particularly when those kits lack either US Food and Drug Administration or country of origin clinical use authorization.

Introduction

The coronavirus disease 2019 (COVID-19) pandemic has challenged public health systems worldwide, not only in terms of patient care and pandemic surveillance and control, but also in guaranteeing the quality of diagnostic tools for severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2). For instance, multiple SARS-CoV-2 molecular diagnosis kits are available on the market, with most of them based on quantitative reverse transcription polymerase chain reaction (qRT-PCR); however, some are based on reverse transcription loop-mediated isothermal amplification (RT-LAMP). Some of these kits have received emergency use authorization (EUA) from the US Food and Drug Administration (FDA) (FDA, 2020), or at least the authorization of the regulatory agencies in their country of production, while others are only supported by clinical evaluation studies conducted by the manufacturer.

Among the kits available on the market, the US Centers for Disease Control and Prevention (CDC) designed 2019-nCoV CDC EUA kit (IDT, USA), which is based on N1 and N2 gene targets to detect SARS-CoV-2, has received positive evaluations in recent reports. This kit uses the RNaseP gene target as an RNA extraction quality control and is considered a gold standard for clinical evaluation (Lu et al., 2020, Center for Diseases Control and Prevention, 2021, Rhoads et al., 2020, Nallaa et al., 2020, Freire-Paspuel et al., 2020a). The Isopollo COVID-19 detection kit (M Monitor, South Korea) is a fluorescence-based RT-LAMP kit that includes two gene targets for SARS-CoV-2 detection, ‘RdRp’ and ‘N’, but has no target for RNA extraction quality control. This RT-LAMP kit does not have EUA approval either from the FDA or from the Korea Disease Control and Prevention Agency (FDA, 2020, Hong et al., 2020), but it is currently available in Ecuador for SARS-CoV-2 clinical diagnosis.

The aim of this study was to evaluate the clinical performance and analytical sensitivity of the Isopollo COVID-19 detection kit using the SARS-CoV-2 RT-PCR CDC protocol as the gold standard.

Materials and methods

Study design

A total of 220 clinical specimens (nasopharyngeal swabs collected in 0.5 ml Tris–ethylenediaminetetraacetic acid (TE) pH 8 buffer) were included in this study. In addition, 10 negative controls (TE pH 8 buffer) were included as controls for carryover contamination, one for each set of RNA extractions.

RNA extraction and RT-qPCR for SARS-CoV-2 diagnosis using the CDC protocol

All of the samples included in the study were tested following an adapted version of the CDC protocol reported previously by our laboratory (Freire-Paspuel et al., 2020b, Freire-Paspuel et al., 2020c, Freire-Paspuel and Garcia-Bereguiain, 2020a, Freire-Paspuel et al., 2020, Freire-Paspuel and Garcia-Bereguiain, 2020b, Freire-Paspuel et al., 2020d).

RT-LAMP for SARS-CoV-2 diagnosis using the Isopollo COVID-19 detection kit

The same RNA extractions from all of the samples included in the study were tested using the Isopollo COVID-19 detection kit. The quality of the RNA was tested by running the RT-qPCR for the RNaseP probe. Initially, 128 samples were processed using a final reaction volume of 15 μl (7.5 μl of reaction buffer 2×; 0.6 μl of enzyme mix; 1.2 μl of primers mix; 0.6 μl of RNase free water; 5 μl of RNA extraction). Subsequently, 92 samples were processed using a final reaction volume of 25 μl, following the manufacturer’s manual.

Analytical sensitivity

The limit of detection (LoD) was assessed using the 2019-nCoV N positive control provided (IDT, USA) at 200 000 genome equivalents/ml for the SARS-CoV-2 RT-PCR CDC protocol. As 40 μl of elution buffer and 0.2 ml of sample are used in the RNA extraction protocol, a 200 conversion factor is applied to change LoD units from copies/μl of RNA extraction solution to copies/ml of nasopharyngeal sample. Regarding the Isopollo COVID-19 detection kit, a positive control is included in the kit but the concentration is not detailed, so it was not possible to calculate the LoD.

Ethics statement

All samples were submitted for routine patient care and diagnostics. Ethical approval was not sought because the study involved laboratory validation of test methods and the secondary use of anonymous pathological specimens falls under the category ‘exempted’ according to the Comité de Etica para Investigación en Seres Humanos of the Universidad de Las Américas.

Results

Clinical performance of the Isopollo COVID-19 detection kit compared to the SARS-CoV-2 RT-PCR CDC gold standard protocol

A total of 220 samples were tested for SARS-CoV-2 using the two protocols described in the Methods section. The first set of 128 samples was processed with the CDC protocol and Isopollo COVID-19 detection kit at a final reaction volume of 15 μl. With the CDC protocol, 97 samples were SARS-CoV-2-positive and 31 samples were SARS-CoV-2-negative, while only 59 out of the 97 samples were also positive with the Isopollo COVID-19 diagnosis kit (Supplementary Material Table S1), yielding a positive percentage of agreement of 60.82% between the two methods. The second set of 92 samples was also processed with both SARS-CoV-2 diagnosis methods, and in these cases a final reaction volume of 25 μl was used for the Isopollo COVID-19 detection kit, as indicated in the manufacturer’s manual. Of these 92 samples, 71 were SARS-CoV-2-positive and 21 were SARS-CoV-2-negative with the CDC protocol, while only 45 out of the 71 samples were also positive with the Isopollo COVID-19 diagnosis kit (Supplementary Material Table S2), yielding a positive percentage of agreement of 63.4% between the two methods.

In summary, while the overall specificity of the Isopollo COVID-19 detection kit was 100%, the overall sensitivity compared to the CDC protocol was 61.9%, as 104 out of 168 SARS-CoV-2-positives samples were detected (Table 1 ).

Table 1.

Clinical performance of the Isopollo COVID-19 detection kit compared to the CDC protocol (the values of 100% and 61.9% correspond to specificity and sensitivity, respectively).

	Isopollo COVID-19
	Positive	Negative
CDC protocol
Positive	104	64 (61.9%)
Negative	0	52 (100%)

CDC, US Centers for Disease Control and Prevention.

Estimation of the limit of detection (LoD) for the Isopollo COVID-19 detection kit

The viral loads detailed in Supplementary Material Tables S1 and S2 were calculated running a calibration curve with the 2019-nCoV N positive control (IDT, USA). The LoD for the CDC protocol was set at 1000 viral RNA copies per milliliter of sample (or 5 RNA copies/μl of RNA extraction solution) in previous studies (Lu et al., 2020, Freire-Paspuel et al., 2020a, Freire-Paspuel et al., 2020b, Freire-Paspuel et al., 2020c, Freire-Paspuel and Garcia-Bereguiain, 2020a, Freire-Paspuel et al., 2020, Freire-Paspuel and Garcia-Bereguiain, 2020b). Although the LoD could not be calculated for the Isopollo COVID-19 detection kit, as described in the Methods section, even for samples with viral loads above 100 000 RNA copies/ml (500 RNA copies/μl of RNA extraction solution), only 81 out of 88 (92.04%) samples were also positive with the Isopollo COVID-19 detection kit. As the LoD is defined as the lowest viral load in which all replicates are detected (100% sensitivity), the study data indicate that the LoD for the Isopollo COVID-19 detection kit would be higher than 100 000 RNA copies/ml of sample.

Discussion

The results of this study indicate that the Isopollo COVID-19 diagnosis kit has poor clinical performance, with a reduction in sensitivity of up to 38.1% compared to the CDC protocol. Moreover, these findings are particularly worrying considering that the amplification of a single viral target is enough to consider a sample as SARS-CoV-2-positive with the Isopollo COVID-19 detection kit, while the amplification of two viral targets is required by the CDC protocol (Lu et al., 2020, Center for Diseases Control and Prevention, 2021, Freire-Paspuel et al., 2020a). Also, the lack of any gene target for RNA extraction quality control like RNaseP, and the unreported concentration of the positive controls provided in the Isopollo COVID-19 kit that does not allow viral load calculations, are also limitations to be considered when using this kit.

As detailed in the Results section, the LoD of the Isopollo COVID-19 detection kit was estimated to be higher than 100 000 viral copies/ml of sample, as only 81 of 88 samples included in the study with viral loads above that LoD were actually detected as positive. Considering the viral load frequency distribution for SARS-CoV-2, this high LoD would potentially affect more than 30% of true-positive cases if the Isopollo COVID-19 detection kit was used for surveillance programs (Lavezzo et al., 2020, Kleiboeker et al., 2020).

The Isopollo COVID-19 detection kit has neither FDA EUA nor Korea Disease Control and Prevention Agency EUA (FDA, 2020, Hong et al., 2020), so it is not actually used for clinical diagnosis in its country of production. However, it is available in Ecuador, where no evaluation studies are conducted by the governmental regulatory agency responsible for clinical use authorization for SARS-CoV-2 diagnosis. Under this scenario, the municipal government of Quito (the capital city of Ecuador, with a population of over two million people) purchased Isopollo COVID-19 kits for around 100 000 RT-LAMP tests (https://www.diarioque.ec/comunidad/mas-de-100-000-pruebas-pcr-para-coronavirus-ya-estan-en-quito/). Although we reported the results of the study presented here to the Quito authorities, they decided to request an extra evaluation study, to be performed by a private diagnosis laboratory. In that study, in which neither viral loads nor cycle threshold (Ct) values of the SARS-CoV-2 samples were detailed, there was also a reduction in sensitivity of up to 14%. Moreover, the study included a larger number of negative samples than ours, and a reduction in specificity of up to 5% was reported (Municipio de Quito, 2020). Unfortunately, the Isopollo COVID-19 diagnosis kit is still in use by the government of the city of Quito, despite our warnings of the risk of the high rate of false-positive and false-negative diagnoses in a massive surveillance program.

Considering the worldwide high demand for reagents for SARS-CoV-2 RT-qPCR diagnosis, a supplies shortage is a fact, and multiple companies are marketing recently developed diagnosis kits. Under this scenario, clinical performance studies should be mandatory to guarantee the quality of the supplies on the market for every country in the world.

This study aims to be a call for action to prevent the use of low quality SARS-CoV-2 diagnosis kits in Ecuador and other developing countries.

Author contributions

Byron Freire-Paspuel and Miguel Angel García Bereguiain analyzed the data and wrote the manuscript.

Funding

This study was funded by Universidad de Las Américas (Quito, Ecuador).

Conflict of interest

All authors have no conflicts of interest to declare.

Appendix A. Supplementary data

The following is Supplementary data to this article: