Abstract
Introduction
The Tokyo Metropolitan Government (TMG) conducted an external quality assessment (EQA) survey of pathogen nucleic acid amplification tests (NAATs) as a TMG EQA program for SARS-CoV-2 for clinical laboratories in Tokyo.
Methods
We diluted and prepared a standard product manufactured by Company A to about 2,500 copies/mL to make a positive control and distribute it with a negative control. The participants reported the use of the NAATs methods for SARS-CoV-2, the name of the real-time RT-PCR kit, the name of the detection device, the target gene(s), nucleic acid extraction kit, Threshold Cycle value in the case of RT-PCR and the Threshold time value and Differential calculation value in the case of Loop-Mediated Isothermal Amplification (LAMP) method.
Results
As a result, 17 laboratories using fully automated equipment and 34 laboratories using the RT-PCR method reported generally appropriate results in this EQA survey. On the other hand, among the laboratories that adopted the LAMP method, there were a plurality of laboratories that judged positive samples to be negative.
Conclusion
The false negative result is considered to be due to the fact that the amount of virus genome contained in the quality control reagent used this time was below the detection limit of the LAMP method combined with the rapid extraction reagent for influenza virus. On the other hand, false positive results are considered to be due to the non-specific reaction of the NAATs. The EQA program must be continued for the proper implementation of the pathogen NAATs.
Keywords: COVID-19, External quality control, Nucleic acid amplification testing, SARS-CoV-2, RT-PCR, Loop-mediated isothermal amplification (LAMP) method
1. Introduction
The Tokyo Metropolitan Government (TMG) implemented the 39th external quality assessment (EQA) program for clinical laboratories on July 30, 2020. An overview on this EQA program was reported by Kumasaka et al. in 2001 [1]. The nucleic acid amplification tests (NAATs) has been added to the program items of EQA from 2019. In 2019, hepatitis B virus was targeted, but in 2020, with the spread of Coronavirus disease 2019 (COVID-19), an EQA of NAATs for the Severe Acute Respiratory Syndrome Coronavirus 2 (SARS-CoV-2) was conducted.
SARS-CoV-2 belongs to the family Coronaviridae, which has a single-strand plus-chain RNA [2]. Therefore, the NAATs for this virus were performed through a plurality of steps such as extracting the RNA genome, reverse transcription reaction from RNA to DNA, amplifying DNA, and detecting the amplified product [3]. For EQA, clinical laboratories must be required to conduct a comprehensive evaluation of all these processes and must be able to confirm them.
In this EQA program, positive control containing the SARS-CoV-2 virus genome in alphavirus (pseudo-virus) particles was distributed to participating laboratories. Participating laboratories included a clinical laboratory and a clinical laboratory division in a hospital that desires to implement the TMG EQA program.
Some EQA program for SARS-CoV-2 has already been implemented in Germany [4,5], Austria [6,7], China [8], South Korea [9] and internationally [10,11]. From the results of an EQA survey of SARS-CoV-2 NAATs conducted as the first EQA by a local government in Japan, we shall describe problems with the participating laboratory and points to be noted in processes of NAATs.
2. Materials and methods
-
i)
Recruitment of participating laboratories
The target laboratories were 28 registered commercial clinical laboratories, 7 special commercial clinical laboratories for only SARS-CoV-2 NAATs and 76 clinical laboratory divisions in hospital with the capacity to conduct NAATs in Tokyo. The recruitment was conducted from June 5 to June 22, 2020.
-
ii)
Survey contents
-
a.
a) Gene amplification methods
We surveyed which methods were used as NAATs for SARS-CoV-2: real-time RT-PCR method, Loop-Mediated Isothermal Amplification (LAMP) method or other methods.
In real-time RT-PCR kit, when fully automatic amplification equipment including nucleic acid extraction is used, we request its name to be reported. The name of the detection device, if it is not a fully automatic device, was requested to be reported. The target gene to be detected was also requested to be reported.
-
b)
Nucleic acid extraction
The name of the nucleic acid extraction kit used or device name when using an automatic nucleic acid extractor were requested to be reported.
-
a.
c) Sample test results
In addition to the positive and negative results, the survey requested to be reported the Threshold Cycle (Ct) value for RT-PCR, and the Threshold time (Tt) value and Differential calculation (Df) value for the LAMP method.
-
iii)
Sample preparation
-
a.
a) Reference laboratory
The reference laboratory was the Tokyo Metropolitan Institute of Public Health and the Department of Microbiology and Infectious Diseases, Toho University School of Medicine, and the same sample was inspected for multiple days to confirm the difference between days.
-
a.
b) Preparation of the quality control standard
We diluted and prepared a standard product manufactured by Company A to about 2,500 copies/ mL to make a positive control (constructed by pseudo-virus) and distribute it as a nasopharyngeal swab suspension together with a negative control, using 1% fetal bovine serum-added cell culture medium as a transport medium. The sample contains virus particles assuming RT-PCR, and was expected to have a nucleic acid concentration near the limit of detection (LOD) by the LAMP method using column extraction technique. The samples were sent refrigerated to arrive on July 30, 2020.
-
a.
c) Reagents and measuring instruments
The nucleic acid extraction kit used was the QIAamp Viral RNA Mini Kit (QIAGEN). In addition, SARS-CoV-2 Direct Detection RT-qPCR kit (Takara Bio) and 2019 new coronavirus detection reagent kit (Shimadzu Corporation) were used as simple detection kits that do not require nucleic acid extraction. These simple extraction/ detection kits of Takara Bio Inc. and Shimadzu Corporation target the N1/ N2 and N1 regions and the N2 region, respectively [3]. BD Max™ (Becton Dickinson, Japan) was used as a fully automatic device, with an open reagent to detect the N2 region described in the manual of the National Institute of Infectious Diseases, Japan. The real-time RT-PCR device used was QuantStudio™ 12K Flex (Thermo Fisher Scientific). The real-time turbidity measuring device LoopampEXIA® (Eiken Chemical) was used for the LAMP method.
3. Results
-
i)
Participating laboratories
A total of 53 laboratories which account for 12 registered commercial clinical laboratories, 4 special commercial clinical laboratories for only SARS-CoV-2 NAATs and 37 clinical laboratory divisions in hospitals that were located in Tokyo participated in an EQA survey of SARS-CoV-2 NAATs in the TMG EQA program in 2020.
-
ii)
Reagents and measuring instruments used by participating laboratories
The survey was attended by 53 institutions, with including a total of 73 used methods. In this manuscript, all the aggregated values will be described for the total number (number of methods), but the unit will be "clinical laboratory". It was reported that 35 clinical laboratories were conventional real-time RT-PCR, and one of them was unmeasurable. Twenty-one clinical laboratories used the LAMP method and one of them reported to be unable to obtain a result. These laboratories are all classified as hospital clinical laboratory divisions. Four laboratories used both real-time RT-PCR system and the LAMP method. Two facilities that reported "unmeasurable" or "undecidable" were excluded from the tabulation in this study.
The device, nucleic acid extraction reagents and nucleic acid amplification reagents used in the participating laboratories are shown in Table 1, Table 2, Table 3 . The fully automatic device was used by the 17 laboratories participating in the survey which account for 16 clinical laboratory divisions in hospitals and 1 registered commercial clinical laboratory. In the RT-PCR method, BD Max™ (Japan Becton Dickinson) was used in 5 laboratories, 3 of which used the BD MAX™ SARS-CoV-2 as an in vitro diagnostic (IVD) reagent and 2 laboratories used the BD MAX™ open system reagents for laboratory developed tests (LDT). GeneXpert (Beckman Coulter) at 3 laboratories, cobas 6800 (Roche Diagnostics) at 2 laboratories and cobas 8800 (Roche Diagnostics) at 1 laboratory were used. Other than the RT-PCR method, the Film Array Torch system (bioMérieux Japan) was used at 4 laboratories. Each remaining laboratory used TRC Ready-80 (Tosoh) and μ TAS Wako g1 (FujiFilm, Wako Pure Chemical Industries) respectively.
Table 1.
List of devices used for detection of SARS-CoV-2 by laboratories participated in external quality assessment survey for SARS-CoV-2 pathogen nucleic acid amplification tests in Tokyo, 2020.
| Detection system | Principle of amplification | Device | Manufacturer | Commercial laboratory | Temporary laboratory | Hospital |
|---|---|---|---|---|---|---|
| Fully automatic system | RT-PCR | BD MAX™ | Becton, Dickinson | 0 | 0 | 5 |
| GeneXpert® system | Beckman Coulter | 0 | 0 | 3 | ||
| cobas6800 | Roche Diagnostics | 0 | 0 | 2 | ||
| cobas8800 | 1 | 0 | 0 | |||
| Others | FilmArray® Torch system | bioMérieux Japan | 0 | 0 | 4 | |
| TRC Ready®-80 | Tosoh | 0 | 0 | 1 | ||
| μTAS Wako g1 | FUJIFILM Wako Chemical | 0 | 0 | 1 | ||
| Manual or semiautomatic system | RT-PCR | cobasZ480 | Roche Diagnostics | 4 | 0 | 5a |
| LightCycler 480 II | 1 | 0 | 4a | |||
| LightCycler 96 | 0 | 0 | 3a | |||
| LightCycler 480 | 1 | 0 | 0 | |||
| QuantStudio 5 | Thermo Fisher Scientific | 2 | 1 | 0 | ||
| QuantStudio 3 | 0 | 1 | 0 | |||
| QuantStudio 1 | 1 | 0 | 0 | |||
| QuantStudio 5Dx | 0 | 0 | 1 | |||
| StepOne Plus | 0 | 1 | 2 | |||
| StepOne | 0 | 0 | 1 | |||
| Applied Biosystems 7500 Real-Time PCR System | 2a | 0 | 1 | |||
| Applied Biosystems 7500 Fast Dx Real-Time PCR instrument | 0 | 0 | 1 | |||
| CFX96 Touch Deep Well Real-Time PCR Detection System | Bio-Rad Laboratories | 1 | 0 | 1 | ||
| Loop-Mediated Isothermal Amplification | LoopampEXIA | Eiken Chemical | 0 | 0 | 20a |
Includes laboratories that also used other reagents.
Table 2.
List of nucleic acid extraction reagent used by laboratories participated in external quality assessment survey for SARS-CoV-2 pathogen nucleic acid amplification tests in Tokyo, 2020.
| Nucleic acid amplification method | Reagents | Manufacturer | No. of laboratories |
|---|---|---|---|
| RT-PCR | QIAamp viral RNA Mini Kit | Qiagen | 7a |
| QIAsymphony DSP Virus/Pathogen Mini Kit | 1 | ||
| Maxwell® RSC Viral TNA | Promega | 4 | |
| MagDEA® Dx SV | Precision System Science | 1 | |
| High pure viral RNA purification kit | Roche Diagnostics | 1 | |
| MagNA Pure96 DNA&Viral NA SV Kit | 1 | ||
| MGIEasy Nucleic Acid Extraction Kit | Sysmex | 1 | |
| NIPPONGENE ISOSPIN RNA Virus | Nippon Gene | 1 | |
| NucleoSpin® RNA Virus | Takara Bio | 1 | |
| Loop-Mediated Isothermal Amplification | Loopamp® RNA extraction reagent for influenza virus | Eiken Chemical | 15a |
| QIAamp Viral RNA Mini Kit | Qiagen | 5a |
Includes laboratories that also used other reagents.
Table 3.
List of nucleic acid amplification reagents used by laboratories participated in external quality assessment survey for SARS-CoV-2 pathogen nucleic acid amplification tests in Tokyo, 2020 (exception of unmeasurable and undecidable).
| Nucleic acid amplification method | Principle of amplification | Used reagents |
No. of laboratories | |
|---|---|---|---|---|
| Manufacturer | Name of reagents | |||
| Fully automatic system | RT-PCR | Becton, Dickinson | BD MAX™ SARS-CoV-2 | 3a |
| BD MAX™ ExK TNA-3、BD MAX™ cartridge | 2a | |||
| Beckman Coulter | Xpert Xpress SARS-CoV-2 | 3a | ||
| Roche Diagnostics | cobas® SARS-CoV-2 | 3a | ||
| Others | bioMérieux Japan | BIOFIRE® Respiratory 2.1 | 4a | |
| Tosoh | 2019 novel coronavirus RNA detection reagent "TRCReady SARS CoV-2" | 1 | ||
| FUJIFILM Wako Chemical | μTAS Wako COVID-19 | 1 | ||
| Manual or semiautomatic system | RT-PCR | Shimadzu | 2019 novel coronavirus detection reagents kit | 9a |
| Takara Bio | SARS-CoV-2 Direct Detection RT-qPCR Kit | 4a | ||
| One Step PrimeScript III RT-qPCR Mix | 3a | |||
| Roche Diagnostics | LightMix Modular SARS-CoV (COVID19) E-gene、N-gene | 3a | ||
| LightMix Modular SARS-CoV (COVID19) E-gene、RdRP-gene | 1a | |||
| LightMix Modular SARS-CoV (COVID19) E-gene | 2 | |||
| LightCycler Multiplex RNA Virus Master | 2a | |||
| Toyobo | SARS-CoV-2 Detection Kit N1/N2 set | 2a | ||
| SARS-CoV-2 Detection Kit -N2 set- | 2 | |||
| THUNDERBIRD Probe One-step qRT-PCR Kit | 1a | |||
| Thermo Fisher Scientific | TaqMan Fast Virus 1-Step Master Mix | 2 | ||
| TaqMan Fast Advanced Master Mix | 1 | |||
| Qiagen | QuantiTect Probe RT-PCR Kit | 1a | ||
| Sysmex | 2019-nCoV fluoresceinated real-time PCR Kit | 1 | ||
| Promega | Primers & probes mix for detection of novel coronavirus（2019-nCoV） | 1 | ||
| LAMPb | Eiken Chemical | Loopamp® novel coronavirus2019 (SARS-CoV-2) detection reagents kit | 20a | |
There is duplication in the data.
Loop-Mediated Isothermal Amplification.
All 20 laboratories reported results by the LAMP method using LoopampEXIA® as a NAAT reagent. Of 20 laboratories, 15 used Loopamp influenza extraction reagent (Eiken Chemical), 2 of which also used the QIAamp Viral RNA Mini Kit (QIAGEN). Only the Mini Kit (QIAGEN) was used at 3 laboratories.
-
iv)
Results of reference laboratories
Table 4 shows the measurement results of the positive control at the reference laboratories. Although the measurement result of the negative control is not shown, the negative result was obtained correctly. In the positive control, positive results were obtained with any set of reagents and measuring instruments. As for the Ct value, the Ct value of the Takara Bio kit tended to be slightly higher than that of other kits. In addition, the diurnal variation of the kit for detecting N2 of Shimadzu Corporation tended to be larger than that of other kits. Other than that, there is no big problem in the detection by RT-PCR, and it is confirmed that the measurement error is within the allowable range, and the reference value (Ct value 30 to 35 in the N2 region of the National Institute of Infectious Diseases) is set based on this result.
-
v)
Results of sample test in the participating laboratories
Table 4.
Results of Ct value or Tt value of reference samples tested in reference laboratories in external quality assessment survey for SARS-CoV-2 pathogen nucleic acid amplification tests in Tokyo, 2020.
| Nucleic acid amplification method | Nucleic acid extraction reagent | Nucleic acid amplification reagent | Detection target gene(s) | Used equipment | Measurement date | Ct value or Tt value |
|---|---|---|---|---|---|---|
| RT-PCR | Included | BD MAX™ ExK™ TNA-3, BD MAX™ Cartridge (Becton, Dickinson) | N2 | BD MAX™ System (Becton, Dickinson) | 30-Jul-20 | 30.70 ± 0.26 |
| 31-Jul-20 | 31.07 ± 0.50 | |||||
| 2-Aug-20 | 31.40 ± 0.53 | |||||
| QIAGEN QIAamp Viral RNA mini kit (QIAGEN)a | QIAamp Viral RNA mini kit (QIAGEN) | N2 | QuantStudio 12K Flex (Thermo Fisher Scientific) | 31-Jul-20 | 34.57 ± 0.32 | |
| 3-Aug-20 | 34.87 ± 0.75 | |||||
| 4-Aug-20 | 35.20 ± 0.30 | |||||
| Not needed | SARS-CoV-2 Direct Detection RT-qPCR K (Takara Bio) | N1/N2 (CDC) | QuantStudio 12K Flex (Thermo Fisher Scientific) | 31-Jul-20 | 37.23 ± 0.23 | |
| 3-Aug-20 | 37.43 ± 0.12 | |||||
| 4-Aug-20 | 36.93 ± 0.25 | |||||
| No needs | 2019 Novel Coronavirus Detection Kit (Shimadzu) | N1/N2 (CDC) | QuantStudio 12K Flex (Thermo Fisher Scientific) | 31-Jul-20 | 34.60 ± 0.36/33.30 ± 2.33 | |
| 3-Aug-20 | 34.87 ± 0.15/35.43 ± 0.49 | |||||
| 4-Aug-20 | 35.07 ± 0.15/34.77 ± 1.36 | |||||
| Loop-Mediated Isothermal Amplification | QIAGEN QIAamp Viral RNA mini kit (QIAGEN)a | Loopamp™ SARS-CoV-2 Detection Kit (Eiken Chemical) | N/RdRp | LoopampEXIA® (Eiken Chemical) | 30-Jul-20 | 20: 56 ± 1: 12 |
| 31-Jul-20 | 22: 14 ± 0: 54 | |||||
| 2-Aug-20 | 20: 44 ± 0: 18 |
QIAcube (QIAGEN) was also used for nucleic acid extraction.
Table 5 shows the percentage of correct answers by measurement method for the results of the participating laboratories. As mentioned above, the total number of facilities that participated in the survey was 73. From Table 5, it was excluded 2 facilities that reported "undecidable" or "unmeasurable" and 15 facilities that used the LAMP method combined with the Loopamp® RNA extraction reagent for influenza virus together were excluded. Fully automatic devices except ⎧TAS Wako g1 correctly judged positive control as positive.
Table 5.
Results of SARS-CoV-2 detection by due to a combination of nucleic acid extraction method and amplification method adopted by laboratoriesc participated in external quality assessment survey for SARS-CoV-2 pathogen nucleic acid amplification tests in Tokyo, 2020.
| Detection system | Nucleic acid extraction method | Principle of nucleic acid amplification | No. of laboratories (Correct answer rate) |
|
|---|---|---|---|---|
| Negative | Positive | |||
| Fully automatic system | Included | RT-PCR | 11(100％)a | 11(100％)a |
| Included | Others | 6 | 5 | |
| Manual or semiautomatic system | Column method/ magnetic bead method | RT-PCR | 18(100％)a | 18(100％)a |
| direct PCR | 16(100％) | 16(100％) | ||
| Column method | Loop-Mediated Isothermal Amplification | 5 (100％)ab | 5 (100％)ab | |
| Total No. | 56(100％) | 55(98.2％) | ||
including duplication.
Excluded because the amount of virus in the distributed positive sample is below the detection limit of the combination of the simple influenza extraction kit and the Loop-Mediated Isothermal Amplification method.
Since the results of laboratories that used multiple measurement methods are included in the total, the number of laboratories in the table and the number of laboratories participating in the survey are different. The total does not include laboratories that have reported that measurement is not possible or judgment is not possible.
The results of the laboratories that adopted the real-time RT-PCR method were generally correct. Negative controls were judged correctly except for those that reported "undecidable" or "unmeasurable" at 2 laboratories.
The reported Ct values (RT-PCR method), Tt values and Df values (both are LAMP methods) are shown in Table 6 a), b) , c). Although the N region of the Ct value includes N1 and N2, only 47.4% (27 laboratories) of the laboratories reported that the Ct value was 30–35, which was considered to be the correct value. Seven laboratories reported Ct values of 25 or less, 40.1 or more, or negative (Table 6 a)).
Table 6.
The range of Ct, Tt and Dt values reported by laboratories. participated in external quality assessment survey for SARS-CoV-2 pathogen nucleic acid amplification tests in Tokyo, 2020.
| a) Ct value for detection target genes reported by laboratories using RT-PCR method | ||||
|---|---|---|---|---|
| Ct value | No. of laboratories |
|||
| Detection gene | ||||
| E | Na | ORF1ab | RdRP | |
| ≦25.0 | 3 | |||
| 25.1–27.5 | ||||
| 27.6–30.0 | ||||
| 30.1–32.5 | 4 | 7 | 3 | |
| 32.6–35.0 | 9 | 20 | 1 | |
| 35.1–37.5 | 2 | 13 | ||
| 37.6–40.0 | 9 | 1 | ||
| ≧40.1 | 1 | |||
| Negative | 3 | |||
| Total No. | 15 | 56 | 4 | 1 |
| b) Tt values reported by laboratories using Loop-Mediated Isothermal Amplification method | |
|---|---|
| Tt value | No. of laboratories |
| ≦20.0 | |
| 20:01–22:00 | 2 |
| 22:01–24:00 | 3 |
| 24:01–26:00 | 1 |
| 26:01–28:00 | |
| 28:01–30:00 | |
| ≧30:01 | 2 |
| Undescribed | 1 |
| Negative | 11 |
| Total No. | 20 |
| c) Dt values reported by laboratories using Loop-Mediated Isothermal Amplification method | |
|---|---|
| Df value | No. of laboratories |
| ≦0.010 | 7 |
| 0.011–0.100 | 1 |
| 0.101–0.130 | 3 |
| 0.131–0.150 | 2 |
| 0.151–0.170 | 2 |
| 0.171–0.190 | 1 |
| ≧0.191 | 0 |
| Undescribed | 4 |
| Total No. | 20 |
Laboratories using different detection genes and methods were duplicated.
Includes N1 set, N2 set, N gene, N1 (CDC), N2 (CDC), N1/ N2 (CDC).
Table 7 shows the results by the LAMP method including the laboratory using the Loopamp® RNA extraction reagent for influenza virus. 20 laboratories adopted the LAMP method, and all five laboratories that extracted nucleic acids by the column method reported appropriate results. Among the laboratories that adopted the LAMP method combined with the Loopamp® RNA extraction reagent, there were a plurality of laboratories that judged positive samples to be negative. On the other hand, of the 14 laboratories that used the the Loopamp® RNA extraction reagent in the LAMP method, only 4 (26.7%) reported positive. Two laboratories used both the simplified extraction reagent and the QIAmp Viral RNA Mini Kit.
Table 7.
Results of SARS-CoV-2 detection due to Loop-Mediated Isothermal Amplification method by extraction method excluding undecidable in laboratories participated in external quality assessment survey for SARS-CoV-2 pathogen nucleic acid amplification tests in Tokyo, 2020.
| Nucleic acid extraction method | No. of laboratories | No. of correct judged |
|
|---|---|---|---|
| Positive | Negative | ||
| QIAamp Viral RNA Mini Kit | 5 | 5 | 5 |
| Loopamp influenza extraction reagent | 14 | 4 | 14 |
| Uncertain | 1 | 0 | 1 |
| Total No. | 20 | 9 | 20 |
4. Discussion
In Japan, many reagents and kits based on IVD, research use only (RUO) reagents, or LDT shown by the National Institute of Infectious Diseases are commercially available. IVDs are validated by the manufacturer, but RUO reagents and LDTs must confirm their validity and LOD at their own laboratories [12]. In other words, it is required to introduce a combination of reagents and kits suitable for the purpose of the laboratory after evaluation and verification. However, in SARS-CoV-2 NAATs, there was a situation in which each laboratory had to be introduced without knowledge and skills for validating reagents and kits. Compared to the commercial clinical laboratory that participated in this EQA survey, the results of the special commercial clinical laboratories for only SARS-CoV-2 NAATs and clinical laboratory divisions in hospitals were not appropriate. While there were many commercial clinical laboratories that had already used NAATs, it is thought that one of the reasons was that the introduction of NAATs was not sufficiently carried out in hospitals. In the future, it is considered necessary to provide education and enlightenment for laboratory personnel in laboratories that carry out pathogen NAATs.
The quality control reagent distributed this time contains virus particles assuming RT-PCR, and was expected to have a nucleic acid concentration near LOD by the LAMP method. All the laboratories that adopted the nucleic acid extraction method by the column method gave a positive test, and this result was fully satisfactory. In clinical laboratories such as hospitals where quick results are required, many laboratories used the Loopamp® influenza virus extraction reagent in combination with the LAMP method for SARS-CoV-2. The viral load in the distributed positive control is below the LOD for the LAMP method combined with the Loopamp® influenza virus extraction reagent (data not shown). It is probable that the results of the laboratory that judged these positive samples as negative were correct in consideration of the LOD, therefore, this combination was excluded from the aggregation (Table 5). However, looking at the report, 4 institutions that adopted this combination achieved a positive result. The reports of these laboratories are "false positives" and these laboratories were individually instructed on the need for improvement. It is necessary to confirm the standard operating procedure at each laboratory. This problem is not solely due to the laboratories; the manufacturers are responsible for correctly communicating product information to the user. Tokyo Metropolitan Government provided information on the results of this EQA program to the Ministry of Health, Labour and Welfare, which led to the manufacturer providing appropriate information to laboratories.
On the other hand, in the laboratories that answered with a negative result for this combination, although they were not informed that the concentration was below the detection limit, provided comments such as "Loopamp® influenza virus extraction reagent is not suitable as a test material for the sample in which the nasopharyngeal swab is suspended in the transport medium" or "the concentration of extracted nucleic acid was deceased due to suspension in the transport medium". We thought that these laboratories understood well the characteristics of the reagents correctly and could operate them properly.
A total of 8 laboratories (2 commercial clinical laboratories, 2 special commercial clinical laboratories for only SARS-CoV-2), including 1 laboratory that answered that "it cannot be measured" by RT-PCR, and 7 laboratories that reported widely deferential Ct values compared with reference value.
These 8 laboratories including 4 clinical laboratories in hospitals were informed of points to be improved by the Tokyo Metropolitan Government. And of these, commercial clinical laboratories and special commercial clinical laboratories for only SARS-CoV-2 NAATs were requested to report improvement measures. One special commercial clinical laboratories for only SARS-CoV-2 NAAT using RT-PCR that answered "unmeasured", despite mentioning the possibility of reduced sensitivity due to differences in lots and versions of nucleic acid extraction kits, continues to use kits that were considered insensitive.
A laboratory reported a large different Ct value from the reference value. This laboratory mentioned the necessity of improvement such as full process control for internal quality assessment, temperature control in all testing processes, and preparation of primers and probe sets for nucleic acid amplification reagent kits in their improvement report. In addition, a laboratory voluntarily reported the following improvement measures: as a corrective measure, it changed the threshold to improve the Ct value. This laboratory should understand that changing the threshold does not lead to substantial improvement.
Buchta et al. reported that some of the participating laboratories reported Ct values that deviated significantly [7]. This EQA program was performed by cooperation between the Center for Virology of the Medical University of Vienna and the Austrian Association for Quality Assurance and Standardization of Medical and Diagnostic Tests. EQA of NAATs requires verification of raw data such as Ct value as well as positive prediction value and negative prediction value.
For commercial clinical laboratories and special commercial clinical laboratories for only SARS-CoV-2 that could not report appropriate improvements, the Tokyo Metropolitan Government EQA committee visits directly several laboratories that could not report appropriate results, and provide guidance to improve inspection accuracy.
This EQS survey of SARS-CoV-2 NAATs conducted by the TMG led to the EQS survey of the Ministry of Health, Labour and Welfare in the fall of 2020. In addition, the results of this EQS survey were public on the website. Furthermore, the seminar was held to participants of TMG EQA program. Participants have question and answer to the person in charge of the TMG, which leaded to the maintenance and improvement of inspection accuracy at participating laboratories.
As of 2021, the COVID-19 outbreak continues in Tokyo. Securing and maintaining highly reliable testing is required to monitor the occurrence of COVID-19 patients based on accurate diagnosis. It is important to implement continuous EQA for laboratories that perform SARS-CoV-2 NAATs.
Icmje statement
Contributors YI, KS and YS were responsible for the organization and coordination of the study. KA, MO, MI, RM, MN and HK were responsible for the data analysis. YI, MO, RM, KS and YS prepared this study design. All authors contributed to the writing of the manuscript.
Declaration of competing interest
All authors have nothing to declare.
Acknowledgments
We acknowledge the assistance provided by all members of the Tokyo Metropolitan EQA Committee. We also thank Ms. Keiko Murata for her critical reading of the manuscript.
References
- 1.Kumasaka K., Kawano K., Yamaguchi K., Igari J., Minowa K., Oishi H., et al. A study of quality assessment in clinical microbiology performance of independent laboratories in Tokyo: 18-year participation in the Tokyo Metropolitan Government External Quality Assessment Program. J Infect Chemother. 2001;7:102–109. doi: 10.1007/s101560100016. [DOI] [PubMed] [Google Scholar]
- 2.V’kovski P., Kratzel A., Steiner S., Stalder H., Thiel V. Coronavirus biology and replication: implications for SARS-CoV-2. Nat Rev Microbiol. 2021;19:155–170. doi: 10.1038/s41579-020-00468-6. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Corman V.M., Landt O., Kaiser M., Molenkamp R., Meijer A., Chu D.K., et al. Detection of 2019 novel coronavirus (2019-nCoV) by real-time RT-PCR. Euro Surveill. 2020;25 doi: 10.2807/1560-7917.ES.2020.25.3.2000045. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Haselmann V., Özçürümez M.K., Klawonn F., Ast V., Gerhards C., Eichner R., et al. Results of the first pilot external quality assessment (EQA) scheme for anti-SARS-CoV2-antibody testing. Clin Chem Lab Med. 2020;58:2121–2130. doi: 10.1515/cclm-2020-1183. [DOI] [PubMed] [Google Scholar]
- 5.Malecki M., Luesebrink J., Wendel A.F., Mattner F. Analysis of external quality assessment samples revealed crucial performance differences between commercial RT-PCR assays for SARS-CoV-2 detection when taking extraction methods and real-time-PCR instruments into account. J Virol Methods. 2021;295 doi: 10.1016/j.jviromet.2021.114202. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Görzer I., Buchta C., Chiba P., Benka B., Camp J.V., Holzmann H., et al. First results of a national external quality assessment scheme for the detection of SARS-CoV-2 genome sequences. J Clin Virol. 2020;129 doi: 10.1016/j.jcv.2020.104537. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Buchta C., Görzer I., Chiba P., Camp J.V., Holzmann H., Puchhammer-Stöckl E., et al. Variability of cycle threshold values in an external quality assessment scheme for detection of the SARS-CoV-2 virus genome by RT-PCR. Clin Chem Lab Med. 2020;59:987–994. doi: 10.1515/cclm-2020-1602. [DOI] [PubMed] [Google Scholar]
- 8.Wang Z., Chen Y., Yang J., Han Y., Shi J., Zhan S., et al. External quality assessment for molecular detection of Severe Acute Respiratory Syndrome Coronavirus 2 (SARS-CoV-2) in clinical laboratories. J Mol Diagn. 2021;23:19–28. doi: 10.1016/j.jmoldx.2020.10.008. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.NSung H., Han M.G., Yoo C.K., Lee S.W., Chung Y.S., Park J.S., et al. Nationwide external quality assessment of SARS-CoV-2 molecular testing, South Korea. Emerg Infect Dis. 2020;26:2353–2360. doi: 10.3201/eid2610.202551. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Matheeussen V., Corman V.M., Donoso Mantke O., McCulloch E., Lammens C., Goossens H., et al. RECOVER project and collaborating networks. International external quality assessment for SARS-CoV-2 molecular detection and survey on clinical laboratory preparedness during the COVID-19 pandemic, April/May 2020. Euro Surveill. 2020;25 doi: 10.2807/1560-7917.ES.2020.25.27.2001223. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Fischer C., Mögling R., Melidou A., Kühne A., Oliveira-Filho E.F., Wolff T., et al. Variable sensitivity of SARS-CoV-2 molecular detection in European expert laboratories: external quality assessment, June and July 2020. J Clin Microbiol. 2021;59 doi: 10.1128/JCM.02676-20. e02676-20. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Bank P.C.D., Jacobs L.H.J., van den Berg S.A.A., van Deutekom H.W.M., Hamann D., Molenkamp R., et al. The end of the laboratory developed test as we know it? Recommendations from a national multidisciplinary taskforce of laboratory specialists on the interpretation of the IVDR and its complications. Clin Chem Lab Med. 2020 doi: 10.1515/cclm-2020-1384. [DOI] [PubMed] [Google Scholar]