Performance of Different Mono- and Multiplex Nucleic Acid Amplification Tests on a Multipathogen External Quality Assessment Panel

Abstract

An external quality assessment (EQA) panel consisting of a total of 48 samples in bronchoalveolar lavage (BAL) fluid or transport medium was prepared in collaboration with Quality Control for Molecular Diagnostics (QCMD) (www.qcmd.org). The panel was used to assess the proficiency of the three laboratories that would be responsible for examining the 6,000 samples to be collected in the GRACE Network of Excellence (www.grace-lrti.org). The main objective was to decide on the best-performing testing approach for the detection of influenza viruses A and B, parainfluenza virus types 1 to 3, respiratory syncytial virus (RSV), human metapneumovirus, coronavirus, rhinovirus, adenovirus, Chlamydophila pneumoniae, Mycoplasma pneumoniae, and Legionella pneumophila by nucleic acid amplification techniques (NAATs). Two approaches were chosen: (i) laboratories testing samples using their in-house procedures for extraction and amplification and (ii) laboratories using their in-house amplification procedures on centrally extracted samples. Furthermore, three commercially available multiplex NAAT tests—the ResPlex (Qiagen GmbH, Hilden, Germany), RespiFinder plus (PathoFinder, Maastricht, The Netherlands), and RespiFinder Smart 21 (PathoFinder) tests—were evaluated by examination of the same EQA panel by the manufacturer. No large differences among the 3 laboratories were noticed when the performances of the assays developed in-house in combination with the in-house extraction procedures were compared. Also, the extraction procedure (central versus local) had little effect on performance. However, large differences in amplification efficacy were found between the commercially available tests; acceptable results were obtained by using the PathoFinder assays.

INTRODUCTION

GRACE (www.grace-lrti.org) is a Network of Excellence focusing on the complex and controversial field of community-acquired lower respiratory tract infections (CA-LRTI), which are among the leading reasons for seeking medical care. The promiscuous use of antibiotics for the treatment of CA-LRTI accounts for a major part of the community burden of antibiotic use and contributes dramatically to the rising prevalence of resistance among major human pathogens. The overall objective of GRACE is to combat antimicrobial resistance by integrating centers of research excellence and exploiting genomics in the investigation of CA-LRTI.

A multitude of nucleic acid amplification techniques (NAATs) for the detection of pathogenic organisms in respiratory specimens have been described (5, 8, 10). Currently, a few commercial assays are available, but the majority of assays applied in clinical diagnostic laboratories have been developed in-house. Therefore, there is a need for interlaboratory exchange of clinical samples in order to compare results and evaluate individual assays, particularly when collaboration takes place in a multicenter network.

Part of the GRACE project is dedicated to the evaluation and validation of rapid diagnostic tests such as NAATs. One of the objectives is to select the best-performing strategy for nucleic acid (NA) extraction, amplification, and detection of pathogenic organisms involved in lower respiratory tract infections. The procedure selected will then be applied to specimens obtained from 3,000 adult patients presenting with lower respiratory tract infections at their general practitioners' offices and 3,000 matched controls.

In the present study, the complete coded external quality assessment (EQA) panel, consisting of 48 samples, was analyzed by PCR in two out of three diagnostic laboratories participating in the GRACE network. The third laboratory analyzed only the subpanel 3 samples. The three laboratories applied their own ‘‘in-house’’ PCR protocols for extraction, amplification, and detection. Moreover, laboratory 3 also extracted the nucleic acids by using a NucliSens EasyMag extraction protocol, after which the extracted nucleic acids were sent to the other two laboratories for analysis with their in-house amplification and detection protocols. Thus, in total, two different DNA extraction methods, as well as different amplification and detection protocols, were evaluated. In addition, the GRACE EQA panel was also analyzed by three commercially available tests.

MATERIALS AND METHODS

Panel preparation and panel composition.

The EQA panel consisted of a total of 48 samples that had been included in previous Quality Control for Molecular Diagnostics (QCMD) EQA panels (2, 9, 11–14, 19, 20) and was divided into three subpanels (see Tables 4, 5, and 6). The 21 samples in respiratory virus subpanel 1 contained a virus transport medium spiked with the following viruses in various concentrations: human metapneumovirus (hMPV) (n = 4), influenza A virus (INF A) (n = 5), influenza B virus (INF B) (n = 1), respiratory syncytial virus (RSV) (n = 3), parainfluenza virus type 1 (PIV-1) (n = 3), PIV-2 (n = 1), and PIV-3 (n = 1). Three samples were negative for all viruses. The 13 samples in EQA subpanel 2, prepared in Dulbecco's modified Eagle's medium and fetal calf serum, were spiked with the following viruses in various concentrations: human coronaviruses (HCOV) (n = 3), human rhinoviruses (HRV) (n = 5), and human adenoviruses (HADV) (n = 4). One sample was negative for all viruses. EQA subpanel 3 consisted of 14 samples spiked with the following bacteria in various concentrations: Mycoplasma pneumoniae (n = 4), Chlamydophila pneumoniae (n = 4), and Legionella pneumophila (n = 4). Two samples were negative. The following EQA subpanel 3 samples were prepared in bronchoalveolar lavage (BAL) fluid: GRACE-37, GRACE-38, GRACE-39, GRACE-40, GRACE-44, GRACE-45, GRACE-47, and GRACE-48.

Table 4.

EQA subpanel 1 results

Sample no.	Sample contenta	Concn/dilution	CT					Resultb by:
			Reference lab (real-time PCR)	Lab 1		Lab 2		ResPlex 1	ResPlex 2	RespiFinder
				With in-house extraction	With EasyMag extraction	With in-house extraction	With EasyMag extraction			Plus	Smart 21
GRACE-14	hMPV-I	1.0 × 10−4	31	31.38	31.19	32.2; INF A, 40.3	32.3	hMPV	hMPV	hMPV	hMPV
GRACE-01	hMPV-I	1.0 × 10−6	36	37.93	39	45.1	40.5	Negative	Negative	Negative	Negative
GRACE-09	hMPV-II	1.0 × 10−4	33	34.02	34.08	35.3	37.4	hMPV	hMPV	hMPV	hMPV
GRACE-07	hMPV-II	1.0 × 10−5	35	37.85	36.52	41.5	Negative	Negative	hMPV	hMPV	hMPV
GRACE-02	INF A type H3	1.0 × 10−7		PIV-2/4, 44.17	Negative	Negative	Negative	Negative	Negative	Negative	Negative
GRACE-08	INF A type H3	1.0 × 10−7		Negative	37.9	39.5	Negative	Negative	Negative	Negative	Negative
GRACE-15	INF A type H3	1.0 × 10−6		37.75	35.97	36.3	37.2	Negative	Negative	INF A	INF A
GRACE-17	INF A type H1	1.0 × 10−5		35.38	33.75	34.4	35	Negative	INF A	INF A	INF A
GRACE-12	INF A type H1	1.0 × 10−6		40.39	35.05	Negative	36.7	Negative	HCOV, EV	INF A	Negative
GRACE-06	INF B	1.0 × 10−6		38.9	36.43	39.3	Negative	Negative	Negative	INF B	INF B
GRACE-21	RSV A	1.0 × 10−3	24	24.24	27.75	29.5	29.8	RSV, HADV	RSV A, HADV	RSV A, HADV	RSV A, HADV
GRACE-19	RSV A	1.0 × 10−5	30	32.26	30.07	40.7	35.3	HRV	HCOV, EV	RSV A	RSV A
GRACE-16	RSV B	5.0 × 10−4	32	28.88	30.36; PIV-2/4, 42.4	33.4	34.4	Negative	INF B, HRV	RSV B	RSV B
GRACE-20	PIV-1		27	27.85	27.19	31.1	30.9	PIV-1	PIV-1	PIV-1	PIV-1
GRACE-10	PIV-1		30	30.53	29.87	34	33.8	PIV-1	PIV-1	PIV-1	PIV-1
GRACE-13	PIV-1		34	33.71	34.08	38.1	38	PIV-1	PIV-1	PIV-1	Negative
GRACE-03	PIV-2		34	32.7	31.57	33.1	30.9	PIV-2	PIV-2, HCOV, EV	PIV-2	PIV-2
GRACE-18	PIV-3		32	34.7	32.85; PIV-2/4, 40.87	31.3	32.5	PIV-3	PIV-3	PIV-3	PIV-3
GRACE-04	NTM	0		Negative	Negative	Negative	Negative	Negative	Negative	Negative	Negative
GRACE-05	NTM	0		Negative	Negative	Negative	Negative	Negative	Negative	Negative	Negative
GRACE-11	NTM	0		Negative	Negative	Negative	Negative	HRV	Negative	Negative	Negative

^ahMPV, human metapneumovirus; INF, influenza virus; RSV, respiratory syncytial virus; PIV, parainfluenza virus; NTM, negative transport medium.

^bHCOV, human coronavirus; EV, enterovirus; HADV, human adenovirus; HRV, human rhinovirus.

Table 5.

EQA subpanel 2 results

Sample no.	Sample contenta	Concn/dilution	CT					Resultb by:
			Reference lab (real-time PCR)	Lab 1		Lab 2		ResPlex 1	ResPlex 2	RespiFinder
				With in-house extraction	With EasyMag extraction	With in-house extraction	With EasyMag extraction			Plus	Smart 21
GRACE-31	HCOV 229E	2.0 × 10−4	30–32	31.75	33.39	37.0; HADV, 45.0	43.7	HCOV 229E	HCOV 229E	HCOV 229E	HCOV 229E
GRACE-30	HCOV OC43	1.0 × 10−4	34–36	35.35	31.86	42.4	Negative	Negative	HCOV OC43, EV	HCOV OC43	HCOV OC43
GRACE-22	HCOV NL63	2.0 × 10−6	35–37	32.79	34.62	Negative	Negative	Negative	Negative	HCOV NL63	HCOV NL63
GRACE-28	HRV 16	1.0 × 10−4	29–30	40.25	29.91	34.6; HADV, 40.0	33.8	HRV	HRV	HRV	HRV
GRACE-25	HRV 16	1.0 × 10−6	38–40	39.7	37.36	42.3	39.7	Negative	Negative	Negative	Negative
GRACE-34	HRV 72	1.0 × 10−5	33–34	35.43	32.97	39.1	36.7	Negative	Negative	HRV	HRV
GRACE-23	HRV 90	1.0 × 10−3	29–30	29.58	28.15	31.4; HADV, 44.6	30.3	HRV	HRV	HRV	HRV
GRACE-33	HRV 90	1.0 × 10−6	37–38	Negative	Negative	Negative	Negative	Negative	Negative	Negative	Negative
GRACE-27	HADV 3	5.0 × 102		34.06	31.34	35.4	33.5	Negative	HADV	HADV	HADV
GRACE-29	HADV 4	2.5 × 102		31.04	31.75	37.1	35.8	HADV	HADV	HADV	HADV
GRACE-24	HADV 31	1.0 × 102		38.89	41.2	39.8	37	Negative	HADV	Negative	Negative
GRACE-32	HADV 1	1.0 × 102		31.83	36.69	38.4	38.7	Negative	Negative	HADV	HADV
GRACE-26	NTM	0		Negative	Negative	Negative	Negative	Negative	Negative	Negative	Negative

^aHCOV, human coronavirus; HRV, human rhinovirus; HADV, human adenovirus; NTM, negative transport medium.

^bEV, enterovirus.

Table 6.

EQA subpanel 3 results

Sample no.	Sample contenta	Concn/dilution (/ml)b	CTc			Resultd by:
			Lab 1	Lab 2	Lab 3	ResPlex 1	ResPlex 2	RespiFinder
								Plus	Smart 21
GRACE-35	M. pneumoniae	5,000 CCU	29.15	36.3	28.44	M. pneumoniae	Negative	M. pneumoniae	M. pneumoniae
GRACE-42	M. pneumoniae	500 CCU	32.6	39.1	31.78	Negative	Negative	M. pneumoniae	M. pneumoniae
GRACE-48	M. pneumoniae	500 CCU	33.34	39.5	32.6	Negative	Negative	M. pneumoniae	M. pneumoniae
GRACE-38	M. pneumoniae	50 CCU	34.49	Negative	36.85	Negative	Negative	M. pneumoniae	M. pneumoniae
GRACE-37	L. pneumophila	1,800 CFU	33.25; M. pneumoniae, 32.7	41.2; M. pneumoniae, 34.5	33.61; M. pneumoniae, 32.85	Negative	Negative	L. pneumophila, M. pneumoniae	M. pneumoniae
GRACE-39	L. pneumophila	180 CFU	36.73	36.4	37.62; M. pneumoniae, 35.18	Negative	Negative	L. pneumophila	Negative
GRACE-47	L. pneumophila	60 CFU	38.07; M. pneumoniae, 35.9	Negative	39.26	Negative	Negative	M. pneumoniae	Negative
GRACE-45	L. pneumophila	18 CFU	40.08	Negative	40.85	Negative	Negative	Negative	Negative
GRACE-43	C. pneumoniae	490 IFU	22.88	28.5	21.04	C. pneumoniae	C. pneumoniae	C. pneumoniae	C. pneumoniae
GRACE-40	C. pneumoniae	49 IFU	26.27	31.2	24.17	C. pneumoniae	C. pneumoniae	C. pneumoniae	C. pneumoniae
GRACE-44	C. pneumoniae	4.9 IFU	30.86	33.1	28.15	Negative	Negative	C. pneumoniae	C. pneumoniae
GRACE-46	C. pneumoniae	4.9 IFU	29.91	34.8	27.69	Negative	Negative	C. pneumoniae	C. pneumoniae
GRACE-36	NTM	0	Negative	Negative	Negative	Negative	Negative	Negative	Negative
GRACE-41	NTM	0	Negative	Negative	Negative	Negative	Negative	Negative	Negative

^aNTM, negative transport medium.

^bCCU, color-changing units; IFU: inclusion-forming units.

^cItalicized C_T values indicate cycle threshold.

^dResPlex 1, original ResPlex format; ResPlex 2, primers and probes adapted by the manufacturer after obtaining the results of the ResPlex 1 assay. The RespiFinder plus and RespiFinder Smart 21 assays were from Pathofinder, Maastricht, Netherlands; the ResPlex assays were from Qiagen GmbH, Hilden, Germany.

All BAL fluid pools spiked with a respiratory virus or with M. pneumoniae, C. pneumoniae, or L. pneumophila were tested in triplicate by mono-PCRs for the presence of that specific organism but not for the presence of the other organisms, unless they were part of the same EQA panel.

Laboratories 1 and 2, as well as Qiagen GmbH (Hilden, Germany) and PathoFinder (Maastricht, The Netherlands), analyzed the complete panel. In laboratory 3, only samples from EQA subpanel 3 were analyzed.

Distribution of the proficiency panels.

The panel samples were randomized by QCMD, freeze-dried, labeled, packed, and distributed at ambient temperature to participants along with a panel receipt form and an instruction manual. Results were reported back to QCMD.

External quality assessment process.

The laboratories were given 5 weeks to test the panel samples using their routine molecular diagnostic tests and to return their results to QCMD. Participants were asked to return qualitative data (presence/absence) separately for each pathogen and, if available, (semi)quantitative data, e.g., cycle threshold (C_T) values.

RNA and DNA extractions. (i) RNA and DNA extractions at laboratory 1.

Before the extraction of nucleic acid (NA), QCMD samples were reconstituted in 1 ml NA-free water and were spiked with internal controls—a known amount of phocine herpesvirus (DNA) and a known amount of encephalomyocarditis virus (RNA)—to monitor the efficient extraction of DNA and RNA, respectively, as described previously (4). All 48 samples were tested separately for the pathogens. RNA and DNA extraction was performed by using a MagnaPure LC total nucleic acid kit (Roche Diagnostics, Mannheim, Germany) as described by Houben et al. (4). Briefly, 200 μl of a clinical specimen was mixed with lysis buffer and proteinase K and was subsequently incubated with magnetic particles to allow binding of the nucleic acid. Unbound material was removed by several washing steps. The nucleic acid was then eluted in 200 μl of elution buffer and was directly used for cDNA synthesis (pathogens carrying an RNA genome) and real-time TaqMan PCR (RNA plus DNA pathogens).

(ii) RNA and DNA extractions at laboratory 2.

The freeze-dried samples were resuspended in 1 ml of NA-free water. Subsequently, 200 μl of this suspension was subjected to nucleic acid extraction using the MagNApure LC total nucleic acid kit, by following the same procedure as that used by laboratory 1. In laboratory 2, however, equine arteritis virus (EAV) was used as an internal control for RNA extractions.

(iii) RNA and DNA extractions at laboratory 3.

Nucleic acids were extracted with the NucliSens EasyMag system (bioMérieux, Grenoble, France) as described previously (7). After extraction, three aliquots were prepared and were frozen at −70°C until frozen shipment to laboratories 1 and 2.

Amplification methods. (i) Amplification methods at laboratory 1.

The isolated viral RNA was reverse transcribed using a MultiScribe reverse transcriptase (RT) kit and random hexamers (Applied Biosystems, Foster City, CA), according to the manufacturer's guidelines, followed by RT inactivation for 5 min at 95°C.

Primers and probes were selected using Primer Express software (Perkin-Elmer Applied Biosystems) and were based on highly conserved genomic regions. To cover subgroups, type-specific primers and probes were chosen for INF A and B, as well as for PIV-1 to -3. The forward and reverse primers, as well as the probes used, are given in Tables 1 through 3.

Table 1.

Primers and probes used at laboratory 1

Pathogen identifieda and primer/probe	Primer/probe sequence (5′ → 3′)	Concn (nM)	Comments	Target gene,b position (comment)
INF-A			Pan-INF-A PCR
Primer 1	AAG ACC AAT CCT GTC ACC TCT GA	900		M1/2, 169–191
Primer 2	CAA AGC GTC TAC GCT GCA GTC C	900		M1/2, 263–242
Probe	TTT GTG TTC ACG CTC ACC GTG CC	150		M1/2, 209–231
INF-B
Primer 1	AAA TAC GGT GGA TTA AAC AAA AGC AA	300		HA, 970–995
Primer 2	CCA GCA ATA GCT CCG AAG AAA	300		HA, 1119–1139
Probe	CAC CCA TAT TGG GCA ATT TCC TAT GGC	100		HA, 1024–1050
PIV-1
Primer 1	TGA TTT AAA CCC GGT AAT TTC TCA T	900		HN, 375–399
Primer 2	CCT TGT TCC TGC AGC TAT TAC AGA	900		HN, 456–433
Probe	ACG ACA ACA GGA AAT C	100		HN, 413–428
PIV-2
Primer 1	AGG ACT ATG AAA ACC ATT TAC CTA AGT GA	300		F and HN, 2852–2880
Primer 2	AAG CAA GTC TCA GTT CAG CTA GAT CA	900		F and HN, 3006–2983
Probe	ATC AAT CGC AAA AGC TGT TCA GTC ACT GCT ATA C	75		F and HN, 2885–2918
PIV-3
Primer 1	TGA TGA AAG ATC AGA TTA TGC AT	900		HN, 840–862
Primer 2	CCG GGA CAC CCA GTT GTG	300		HN, 1088–1071
Probe	TGG ACC AGG GAT ATA CTA CAA AGG CAA AAT AAT ATT TCT C	75		HN, 984–1023
RSV			RSV A+B PCRc
Primer A1	AGA TCA ACT TCT GTC ATC CAG CAA	900		N, 53–76 (serotype A)
Primer A2	TTC TGC ACA TCA TAA TTA GGA GTA TCA AT	900		N, 136–108 (serotype A)
Primer B1	AAG ATG CAA ATC ATA AAT TCA CAG GA	300		N, 164–189 (serotype B)
Primer B2	TGA TAT CCA GCA TCT TTA AGT ATC TTT ATA GTG	300		N, 266–234 (serotype B)
Probe A	CAC CAT CCA ACG GAG CAC AGG AGA T	58.3		N, 80–104 (serotype A)
Probe B	TTC CCT TCC TAA CCT GGA CAT AGC ATA TAA CAT ACC T	66.7		N, 231–195 (serotype B)
HRV			Pan-Rhino PCR
Primer 1	GCC TGC GTG GCT GCC	300		Polyprotein, 88–102 (A strain)
Primer 2	CCT GCG TGG CGG CC	300		Polyprotein, 122–135 (B strain)
Primer 3	ACG GAC ACC CAA AGT AGT TGG T	300		Polyprotein, 286–265 (A strain)
Primer 4	ACG GAC ACC CAA AGT AGT CGG T	300		Polyprotein, 318–297 (B strain)
Probe A	TCC GGC CCC TGA ATG TGG CTA A	100		Polyprotein, 175–196 (A strain)
Probe B	TCC GGC CCC TGA ATG CGG CTA A	100		Polyprotein, 207–228 (B strain)
HCOV			229E, NL63, OC43
Primer 1	CAG TCA AAT GGG CTG ATG CA	300		N, 154–173 (229E)
Primer 2	CAA AGG GCT ATA AAG AGA ATA AGG TAT TCT	300		N, 231–201 (229E)
Primer 3	GCG TGT TCC TAC CAG AGA GGA	50		N, 157–177 (NL63)
Primer 4	GCT GTG GAA AAC CTT TGG CA	300		N, 275–256 (NL63)
Primer 5	CGA TGA GGC TAT TCC GAC TAG GT	900		N, 577–599 (OC43)
Primer 6	CCT TCC TGA GCC TTC AAT ATA GTA ACC	900		N, 652–626 (OC43)
Probe A	CCC TGA CGA CCA CGT TGT GGT TCA	100		N, 199–176 (229E)
Probe B	ATG TTA TTC AGT GCT TTG GTC CTC GTG AT	100		N, 180–208 (NL63)
Probe C	TCC GCC TGG CAC GGT ACT CCC T	125		N, 601–622 (OC43)
HADV			Pan-Adeno PCR
Primer 1	TTT GAG GTG GAY CCM ATG GA	225		Hexon, xd
Primer 2	TTT GAG GTY GAY CCC ATG GA	225		Hexon, yd
Primer 4	AGA ASG GSG TRC GCA GGT A	225		Hexon, x + 105d
Primer 5	AGA ASG GTG TRC GCA GAT A	225		Hexon, y + 105d
Probe A	ACC ACG TCG AAA ACT TCG AA	100		Hexon, x + 45d
Probe B	ACC ACG TCG AAA ACT TCA AA	100		Hexon, y + 45d
Probe C	ACA CCG CGG CGT CA	100		Hexon, x/y + 80d
M. pneumoniae
Primer 1	GGT CAA TCT GGC GTG GAT CT	50		P1, 3967–3986
Primer 2	TGG TAA CTG CCC CAC AAG C	300		P1, 4032–4014
Probe	TCC CCC GTT GAA AAA GTG AGT GGG T	125		P1, 3988–4012
L. pneumophila
Primer 1	GCA ATG TCA ACA GCA ATG GC	300		MIP, 13–32
Primer 2	CGG CAC CAA TGC TAT AAG ACA A	300		MIP, 94–73
Probe	CAA CCG ATG CCA CAT CAT TAG CTA CAG ACA	100		MIP, 35–64
C. pneumoniae
Primer 1	AAA CAA TTT GCA TGA AGT CTG AGA A	900		MOMP, 756–732
Primer 2	TCC GCA TTG CTC AGC C	300		MOMP, 631–646
Probe	TAA ACT TAA CTG CAT GGA ACC CTT CTT TAC TAG G	75		MOMP, 667–700

^aINF, influenza virus; PIV, parainfluenza virus; RSV, respiratory syncytial virus; HRV, human rhinovirus; HCOV, human coronavirus; HADV, human adenovirus.

^bM1/2, matrix 1 and 2; HA, hemagglutinin; HN, hemagglutinin-neuraminidase; F, fusion; N, nucleoprotein; P1, cytadhesin; MIP, macrophage infectivity potentiator; MOMP, major outer membrane protein.

^cSee reference 18a.

^dStart position differs with serotype.

Table 3.

Primers and probes used at laboratory 3

Pathogen identified and primer/probe	Sequencea	Concn (pmol/ml)
M. pneumoniae
Primer 1	5′ CGG GAT TCC CCG CGG AGG 3′	10
Primer 2	5′ CAC CCT CGG GGG CAG TCAG 3′	10
Probe	5′ GCC TTA TCA TTC CTT CAC CCC GCC CC FLU 3′	4
Probe	5′ LC Red 640 TTC AGA GCT GGA GGT TGG CTT GGT CGA Gp 3′	4
L. pneumophila
Primer 1	5′ CAACCGATGCCACATCATTA 3′	10
Primer 2	5′ TAGCCATTGCTTCCGGATTA 3′	10
Probe	5′ GCCTTGATTTTTAAAATTCTTCCCAA FLU 3′	4
Probe	5′ LC Red 640 TCGGCACCAATGCTATAAGACAACTp 3′	4
C. pneumoniae
Primer 1	5′ TCCGCATTGCTCAGCC 3′	10
Primer 2	5′ AAACAATTTGCATGAAGTCTGAGAA 3′	10
Probe	5′ CTGCATGGAACCCTTCTTTACTAGGAA FLU 3′	4
Probe	5′ LC Red 640 TGCCACAGCATTGTCTACTACTGATTC p 3′	4

^aFLU, fluorescein.

Samples were assayed in duplicate in a 25-μl reaction mixture containing 5 μl of cDNA, 12.5 μl of 2× TaqMan universal PCR master mix (Perkin-Elmer Applied Biosystems), and the concentrations of the forward primers, reverse primers, and probes indicated in Table 1. The fluorogenic probes were both labeled with the 5′ reporter dye 6-carboxyfluorescein (FAM) and the 3′ quencher dye 6-carboxytetramethylrhodamine (TAMRA). Amplification and detection were performed with an ABI Prism 7700 sequence detection system under the following conditions: 2 min of incubation at 50°C to attain optimal AmpErase uracil-N-glycosylase activity, 10 min at 95°C to activate the AmpliTaq Gold DNA polymerase, and 45 cycles of 15 s at 95°C and 1 min at 60°C.

The viral load was determined by the number of amplification cycles needed for a positive PCR test (C_T). A C_T value of 45 was chosen as a cutoff for sample positivity. Samples were controlled for the presence of possible inhibitors of the extraction or amplification reaction by the indicated internal controls: C_T values had to range within clear-cut intervals. Positive results were confirmed by a second analysis of the same sample. In case of discrepant results, a third analysis was performed. C_T values are mean values for duplicate reactions.

(ii) Amplification methods at laboratory 2.

Primers and probes were designed using Beacon Designer software (Premier Biosoft International, Palo Alto, CA) and have been described previously (15, 16), except that molecular beacons were replaced by TaqMan hydrolysis probes, using the same target sequences. The assays were performed as four multiplex real-time PCR assays, combining INF A, INF B, and RSV; PIV-1 to -4; HRV, hMPV, and the EAV internal control; and finally the four HCOV 229E, OC43, NL63, and HKU1.

Real-time PCR was performed in 50 μl of a reaction mixture consisting of 10 μl of 5× one-step RT-PCR buffer (One-Step RT-PCR kit; Qiagen, Hilden, Germany), 10 mM deoxynucleoside triphosphates (dNTPs), 4.5 mM MgCl₂, 0.6 μM each primer (Table 2), and 0.34 μM TaqMan probes, with 5 μl of the template. The PCR thermal profile consisted of an initial cDNA step of 30 min at 50°C, followed by 15 min at 95°C and 45 cycles of 30 s at 95°C, 30 s at 55°C, and 30 s at 72°C.

Table 2.

Primers and probes used at laboratory 2^b

Pathogen identified and primer/probe namea	Primer or probe	Sequence	Fragment length (bp)	Probe typec	Labeld
					5′	3′
PIV-1
609PIV1-TQ-YAK	Probe	CAAACGATGGCTGAAAAAGGGA	164	TQ	YAK	BHQ-1
1101PIV1s	Sense primer	AAAAACTTAGGGTTAAAGACAATCCA
1102PIV1as	Antisense primer	GCCAGATGTRTGTCYTTCCTGCTGGT
PIV-2
621PIV2-TQ-TEXAS RED	Probe	AATCGCAAAAGCTGTTCAGTCAC	113	TQ	TXR	BHQ-2
231PIV2s	Sense primer	CCATTTACCTAAGTGATGGAA
232PIV2as	Antisense primer	CGTGGCATAATCTTCTTTTT
1065PIV2as	Antisense primer	TGTGGCATAATCTTCTTTCT
PIV-3
566PIV3-TQ-FAM	Probe	ACCCAGTCATAACTTACTCAACAGCAAC	154	TQ	FAM	BHQ-1
1106PIV3s	Sense primer	CAGGAAGCATTGTRTCATCTGT
1107PIV3as	Antisense primer	ATAGTGTGTAATGCAGCTYGT
PIV-4
675PIV4-TQ-CY5	Probe	GTCTCAAAATTTGTTGATCAAGAYAATACAATT	200	TQ	Cy5	BHQ-2
264PIV4	Sense primer	CCTGGAGTCCCATCAAAAGT
1071PIV4as	Antisense primer	GCATCTATACGAACRCCTGCT
INF A
1815FLUA-TQ-FAM	Probe	CCTCGCTCACTGGGCACGGT	119	TQ	FAM	BHQ-1
1233FluAs	Sense primer	CATGGARTGGCTAAAGACAAG
1234FluAas	Antisense primer	TYTGGACAAAGCGTCTACG
INF B
681FLUB-TQ-TEXAS-RED	Probe	GCAAACACTGGGCTGCARCT	145	TQ	TXR	BHQ-2
220FluBs	Sense primer	GTCCATCAAGCTCCAGTTTT
221FluBas	Antisense primer	TCTTCTTACAGCTTGCTTGC
RSV
654RSV-TQ-HEX	Probe	CCATGTGAATTCCCTGCATCAAT	155	TQ	HEX	BHQ-1
674RSV-TQ-HEX	Probe	CCTGcGAATTCCCTGCcTCAAT	155	TQ	HEX	BHQ-1
1070RSVs	Sense primer	TTTCCACAATATTTAAGTGTtAA
412RSVs	Sense primer	TTTCCACAATATYTAAGTGTCAA
413RSVas	Antisense primer	TCATCWCCATACTTTTCTGTTA
hMPV
567MPV-TQ-YAK	Probe	GCATGYCAYTGGTGTGGGATATT	170	TQ	YAK	BHQ-1
342MPVs	Sense primer	CATGCCCACTATAAAAGGTCAG
343MPVas	Antisense primer	CACCCCAGTCTTTCTTGAAA
1068MPVs	Sense primer	TATGCCTACCATAAAAGGTCAA
1069MPVas	Antisense primer	CACCCCAGTCTTTCCTAAAG
EAV
615EAV-TQ-CY5	Probe	CGCTGTCAGAACAACATTATTGCCCAC	134	TQ	Cy5	BHQ-2
417EAVs	Sense primer	CATCTCTTGCTTTGCTCCTTAG
418EAVas	Antisense primer	AGCCGCACCTTCACATTG
HRV
606HRV-TQ-FAM	Probe	TCCTCCGGCCCCTGAATGYGGCTAA	142	TQ	FAM	BHQ-1
777HRV_1s	Sense primer	GACAGGGTGTGAAGAGCC
778HRV_2s	Sense primer	GACATGGTGTGAAGACCC
779HRV_3S	Sense primer	GACAAGGTGTGAAGAGCC
780HRV_4s	Sense primer	GACATGGTGTGAAGACTC
1039HRVs	Sense primer	GACATGGTGTGAAGATCT
1037HRVas	Antisense primer	ACACGGACACCCAAAGTAGT
HCOV-NL63
599HCOV-NL63-TQ-TXR	Probe	CGCATACGCCAACGCTCTTGAACA	143	TQ	TXR	BHQ-2
750HCOV-NL63s	Sense primer	GTTCTGATAAGGCACCATATAGG
751HCOV-NL63as	Antisense primer	TTTAGGAGGCAAATCAACACG
HCOV-229E
598HCOV-229E-TQ-FAM	Probe	ATGAACCTGAACACCTGAAGCCAATCTATG	137	TQ	FAM	BHQ-1
741HCOV-229Es	Sense primer	CATACTATCAACCCATTCAACAAG
742HCOV-229Eas	Antisense primer	CACGGCAACTGTCATGTATT
HCOV-OC43
587HCOV-43-TQ-YAK	Probe	TGCCCAAGAATAGCCAGTACCTAGT	110	TQ	YAK	BHQ1
484HCOV43s	Sense primer	CATACTCTGACGGTCACAATAATA
485HCOV43as	Antisense primer	ACCTTAGCAACAGTCATATAAGC
HCOV-HKU1
677HCOV-HKU1-TQ-CY5	Probe	TYCGCCTGGTACGATTTTGCCTCA	147	TQ	Cy5	BHQ-2
864HCOV-HKU1s	Sense primer	TCCTACTAYTCAAGAAGCTATCC
865HCOV-HKU1as	Antisense primer	AATGAACGATTATTGGGTCCAC
HADV
692ADV-XS-FAM	Probe	AGCCCACCCTKCTTTAT	139	TQ	FAM	BHQ-1
658ADV4-TQ-YAK	Probe	GAGTCYACCCTTCTCTATGT			YAK	BHQ-1
372ADVs	Sense primer	CATGACTTTTGAGGTGGATC
346ADVas	Antisense primer	CCGGCCGAGAAGGGTGTGCGCAGGTA
423ADV4s	Sense primer	CATGAATTTCGAAGTCGACC
424ADV31s	Sense primer	TATGACATTTGAAGTTGACC
M. pneumoniae
612MYCPN-TQ-YAK	Probe	CAAAGCCACCCTGATCACCC	151	TQ	YAK	BHQ-1
224MYCPNs	Sense primer	ATTCCCGAACAAAATAATG
225MYCPNas	Antisense primer	GTTTGACAAAGTCCGTGAAG
C. pneumoniae
611CPN-TQ-FAM	Probe	GGGATCTTCGGACCTTTCGG	154	TQ	FAM	BHQ-1
214CPN16Ss	Sense primer	GCGGAAGGGTTAGTAGTACA
215CPN16Sas	Antisense primer	ATCGCATAAACTCTTCCTCA
Legionella spp.
539LEGSP-MB-FAM	Probe	CCGAGCGGTGAGTAACGCGTAGGAATATGGCTCGG	212	MB	FAM	Dabcyl
156LegSPs	Sense primer	AGGCTAATCTTAAAGCGCC
157LegSPas	Antisense primer	CCTGGCTCAGATTGAACG
L. pneumophila
593LEGPN-TQ-YAK	Probe	GCATTGGTGCCGATTTGGGA	124	TQ	YAK	BHQ-1
269LGPNs	Sense primer	TGGTGACTGCAGCTGTTATG
270LGPNas	Antisense primer	CATTGCTTCCGGATTAACAT

^aPIV, parainfluenza virus; INF, influenza virus; RSV, respiratory syncytial virus; hMPV human metapneumovirus; EAV, equine arteritis virus; HRV, human rhinovirus;; HCOV, human coronavirus; HADV, human adenovirus.

^bYAK, Yakima Yellow; TXR, Texas Red; HEX, hexachlorofluorescein; R = (AG); Y = (CT); K = (GT); W = (AT).

^cMB, molecular beacon; TQ, TaqMan.

^dBHQ, black hole quencher.

For DNA targets, published assays (17) were used; these were performed in HotStar Taq master mix (Qiagen). Amplification, detection, and data analysis were performed using the iCycler IQ real-time detection system (Bio-Rad, Veenendaal, The Netherlands).

When samples were tested in duplicate reactions, one positive reaction was considered a positive result, since in proficiency testing, samples with concentrations around the limit of detection (LOD) can be detected. In case both reactions were positive, the value in the table is the mean value.

(iii) Amplification methods at laboratory 3.

Real-time in-house mono-PCRs were applied as described previously for M. pneumoniae, C. pneumoniae, and L. pneumophila detection (6, 18). Primers and probes are presented in Table 3. Positive results were confirmed by a second analysis of the same sample. In case of discrepant results, a third analysis was performed.

Commercially available PCRs.

The panels were sent to Qiagen GmbH to be analyzed by the ResPlex 1 assay and to PathoFinder to be analyzed by the RespiFinder plus and RespiFinder Smart 21 assays according to the manufacturer's instructions. After receiving the first ResPlex results, the manufacturer made modifications to the kit, producing the ResPlex 2 assay.

The RespiFinder Smart 21 assay is a real-time variant (under development) of the MultiFinder PCR technology (13) that enables the detection and differentiation of 21 respiratory pathogens: INF A, INF A H1N1, INF B, RSV-A, RSV-B, HADV, HRV, PIV-1 to -4, HCOV 229E, NL63, OC43, and HKU1, hMPV, bocavirus, M. pneumoniae, C. pneumoniae, L. pneumophila, and Bordetella pertussis.

RESULTS

Results of in-house PCRs for the detection of respiratory viruses.

The performances of the NAATs for the detection of the respiratory viruses (subpanels 1 and 2) at laboratories 1 and 2 on extracts obtained with the routine MagNA pure LC and NucliSens EasyMag nucleic acid extraction procedures were comparable; no major differences in sensitivity and specificity were observed. Using its own protocol for extraction and amplification, laboratory 1 obtained one false-positive result (sample GRACE-02; PIV-2/4) and three false-negative results, all on samples with very low viral loads (samples GRACE-02, GRACE-08, and GRACE-33) (Tables 4 and 5). When examining NucliSens EasyMag-extracted samples, laboratory 1 reported the correct virus in samples GRACE-09, GRACE-16, and GRACE-18, but each time in combination with PIV-2/4.

Laboratory 2 reported two and three false-negative results after applying its own nucleic acid extraction procedure and the NucliSens EasyMag procedure, respectively, for the 21 samples of subpanel 1 and one and three false-negative results for the 13 samples of subpanel 2 (Tables 4 and 5). No false-positive results were reported after NucliSens EasyMag extraction, in contrast to four false-positive results obtained after laboratory 2 used its own nucleic acid extraction (Tables 4 and 5).

In general, after NucliSens EasyMag extraction, both laboratories obtained C_T values equal to or lower than those obtained with their in-house extraction procedures (Tables 1 to 4).

Results of in-house PCRs for the detection of atypical pathogens.

In Table 6, the results of the M. pneumoniae, C. pneumoniae, and L. pneumophila EQA subpanel are presented. No false-negative results were reported by laboratories 1 and 3. Laboratory 2 failed to detect 3 positive samples: GRACE-38, containing the lowest concentration of M. pneumoniae, 50 color-changing units (CCU)/ml, and GRACE-45 and GRACE-47, spiked with 18 and 60 CFU/ml of L. pneumophila, respectively. None of the negative-control samples (negative transport medium [NTM]) were found positive by the three GRACE laboratories. In addition to the correct pathogens, laboratories 1 and 3 also detected M. pneumoniae in two samples. Laboratory 2 also found M. pneumoniae in one of these samples, indicating that contamination had occurred before the start of analysis of these samples.

Results by commercially available PCRs: the ResPlex, RespiFinder Plus, and RespiFinder Smart 21 assays.

After examination of the EQA panel for the presence of hMPV, INF A/B, PIV-1 to -3, and RSV in subpanel 1 (Tables 4 and 7), correct results were reported for 9/21, 11/21, 17/21, and 15/21 samples by use of the ResPlex 1, ResPlex 2, RespiFinder plus, and RespiFinder Smart 21 assays, respectively. Sample GRACE-11, containing transport medium only, was reported to be positive for HRV by the ResPlex assay. GRACE-19 was spiked with RSV-A but was found to be positive for HRV by the ResPlex 1 assay and positive for HCOV and enterovirus by the ResPlex 2 assay. GRACE-16 was spiked with RSV-B but was found to be positive for INF B and HRV by the ResPlex 2 assay, whereas GRACE-12 contained INF A but was reported to be positive for HCOV and enterovirus by the same assay. Additionally, the ResPlex 2 assay identified 2 extra viruses in GRACE-03. Sample GRACE-21 was found to be positive for both HADV and RSV by all 4 commercial tests.

Table 7.

Summary of results

No. of results	Lab 1		Lab 2		Lab 3	ResPlex		RespiFinder
	In-house extraction	EasyMag extraction	In-house extraction	EasyMag extraction		1	2	Plus	Smart 21
Subpanel 1 (n = 21)
Correct	20/21	17/21	18/21	17/21	NAa	10/21	10/21	18/21	16/21
False positive	1/21	3/21	1/21	0/21	NA	2/21	4/21	0/21	0/21
False negative	1/21	1/21	2/21	4/21	NA	10/21	8/21	3/21	5/21
Subpanel 2 (n = 13)
Correct	12/13	12/13	8/13	10/13	NA	5/13	7/13	10/13	10/13
False positive	0/13	0/13	3/13	0/13	NA	0/13	1/13	0/13	0/13
False negative	1/13	1/13	2/13	3/13	NA	8/13	5/13	3/13	3/13
Subpanel 3 (n = 14)
Correct	14/14	NA	11/14	NA	14/14	5/14	4/14	12/14	10/14
False positive	0/14	NA	0/14	NA	0/14	0/14	0/14	0/14	0/14
False negative	0/14	NA	3/14	NA	0/14	9/14	10/14	2/14	4/14

^aNA, not applicable.

No difference in sensitivity was observed between the RespiFinder plus and the RespiFinder Smart 21 assay in examination of subpanel 2 for HADV, HCOV, and HRV (Tables 5 and 7): both assays failed to detect HADV 31, HRV 16, and HRV 90 in samples GRACE-24, GRACE-25, and GRACE-33, respectively. The latter sample was also reported as negative by all of the in-house tests. The ResPlex assay (ResPlex 1) failed to detect the same three samples. Additionally, five other samples were also reported as negative by the ResPlex 1 assay. After modification of the assay to the ResPlex 2 format, sensitivity improved slightly, with five false-negative results. No false-positive results were obtained with any of the commercially available assays.

Upon examination of subpanel 3, the RespiFinder plus assay failed to detect the sample spiked with 18 CFU of L. pneumophila (Table 6). All other samples were correctly identified. The RespiFinder Smart 21 assay did not detect any of the four samples spiked with L. pneumophila. The manufacturer was contacted on this issue and improved the assay. GRACE-37 and GRACE-39 were correctly identified as L. pneumophila-positive samples after retesting with a newer version of the RespiFinder Smart 21 assay. No other false-negative or false-positive results were reported by use of the RespiFinder assays.

The sensitivities of both ResPlex assays were very low. The original format (ResPlex 1) yielded only three positive samples, all with the highest loads of M. pneumoniae or C. pneumoniae. The assay failed to detect any L. pneumophila-positive samples. After the assay was adapted (ResPlex 2 results), no improvement was seen.

DISCUSSION

One of the objectives of this study was to check whether in-house nucleic acid extraction procedures could be replaced by a central nucleic acid extraction method with subsequent transport of extracts to other centers for nucleic acid amplification purposes in the context of a large study. The RNA and DNA sent to the participating laboratories were extracted from the EQA panel at laboratory 3 with the NucliSens EasyMag system, producing nucleic acid extracts of high quality, as reflected by the results obtained. After comparison of the results, the different extraction methods did not reveal significant differences: comparable sensitivities and specificities were obtained with both in-house nucleic acid extraction methods and the NucliSens EasyMag extraction procedure. Considering the overall workload and the results obtained, the method of choice for extraction of nucleic acids from respiratory samples collected in the GRACE network is the NucliSens EasyMag procedure performed in laboratory 3.

For comparison of the sensitivities and specificities of the different nucleic acid amplification methods, it was decided that laboratory 1 would apply its in-house PCRs for the detection of RSV, INF A/B, HCOV OC43, NL63, and 229E, and the polyomaviruses WUPyV and KIPyV. Laboratory 2 would examine samples for PIV-1 to -4, HRV, hMPV, HAdV, and bocavirus by using its in-house PCRs, and the in-house PCRs of laboratory 3 would be used for the detection of M. pneumoniae, C. pneumoniae, and L. pneumophila. This decision was not based on C_T values, since the Bio-Rad iCycler, used in laboratory 2, usually gives higher C_T values than the real-time equipment used in the other two laboratories.

The RespiFinder Smart 21 assay (PathoFinder) is a real-time multiplex PCR assay under development and is not yet commercially available. It is a further development of the MultiFinder technology as applied in the RespiFinder plus assay. According to the manufacturer (personal communication), the analytical sensitivity of the RespiFinder ranges from 5 to 50 copies per reaction for most targets when commercially available quantitated DNA/RNA PCR controls (Vircell) are used. Seven samples positive for a respiratory virus were missed by the assay, two more than with the RespiFinder plus assay. All these samples contained low viral loads. All L. pneumophila-spiked samples were classified as negative based on the RespiFinder Smart 21 results, whereas two had been positive with the RespiFinder plus assay. The manufacturer was contacted on this issue and improved the assay. The commercially available ResPlex assay (Qiagen GmbH), a multiplex PCR, was also evaluated in this study but did not perform well. Even when the company had made some modifications to the kit after their first results (ResPlex 1), the performance of the assay improved only slightly (ResPlex 2), and it was considered too insensitive for further evaluations. The manufacturer was contacted and is aware of the sensitivity problems of the ResPlex assay. It intends to improve the sensitivity of the test. According to the literature, the analytical sensitivity reported by the supplier of the ResPlex II assay is about 500 viral genomes per reaction (21). Serial dilutions of titrated strains were prepared by Wang et al., and sensitivities on the order of 3.0 · 10⁻² 50% tissue culture infective dose (TCID₅₀)/reaction for INF A, 1.0 · 10⁻³ TCID₅₀/reaction for INF B, 1.4 · 10⁻¹ TCID₅₀/reaction for RSV, and 7.0 TCID₅₀s/reaction for human enterovirus were found (21). Lower sensitivities for the ResPlex II assay than for multiplex NAATs are also reported in the literature (1, 3).

All samples used in this GRACE quality control (QC) panel originated from previous EQA distributions. All pools spiked with a respiratory virus or with M. pneumoniae, C. pneumoniae, or L. pneumophila were tested for the presence of that specific organism but not for the presence of the other organisms, unless they were part of the same EQA panel. When the commercially available multiplex assays were applied to the GRACE QC samples, more than 1 target organism was detected in some GRACE samples. If the result was confirmed by at least one of the other commercially available multiplex tests, e.g., GRACE-21 and GRACE-37, the additional organism was probably already present in the original pool, and the result should be considered correct. On the other hand, the detection of PIV-2/4 in GRACE-09, GRACE-16, and GRACE-18 (Table 4) and of HADV in GRACE-23, GRACE-28, and GRACE-31 (Table 5) was probably due to contamination that occurred during the extraction/amplification procedure. This conclusion is supported by the fact that the reported C_T values are similar.

In conclusion, this study demonstrated the importance of including a sufficient number of weakly positive samples and negative controls in amplification runs to detect possible false-positive and false-negative results when the best-performing test must be selected and when a new assay is to be validated.