Multicenter Proficiency Testing of Nucleic Acid Amplification Methods for the Detection of Enteroviruses

Abstract

A multicenter study of molecular detection of enteroviruses was conducted using a proficiency panel. Of 70 data sets, 46 (66%) reported correct results for samples containing at least 1 50% infective dose per ml and for negative samples. Variation in performance between laboratories demonstrates the need for ongoing quality control.

Nucleic acid amplification methods are widely used for enterovirus (EV) detection because they are sensitive, specific, and rapid (1, 3, 6, 10, 12, 17). However, reliable molecular diagnosis requires assay standardization and continuous monitoring of sensitivity and specificity. A quality assessment program for EV detection was therefore established within the framework of the European Union Concerted Action on Quality Control of Nucleic Acid Amplification in Diagnostic Virology (QCCA) to assess the proficiency of laboratories using molecular EV detection methods. Here we describe results of 70 data sets reported by 59 laboratories upon testing an EV proficiency panel. To our knowledge, this represents the largest such study reported to date.

The proficiency panel consisted of 12 coded samples, including a coxsackievirus A9 (CVA9) dilution series, other enterovirus serotypes representing different genetic clusters of human enteroviruses (5, 11), human parechovirus type 1 (HPEV1; formerly classified as echovirus 22 but now known to be genetically distinct from enteroviruses), and negative controls (Table 1). The sources, production, and characterization of the viruses used are described elsewhere (7, 15). Titration of viral infectivity was performed on the original virus stocks (50% tissue culture infective doses [TCID₅₀] per milliliter). Virus stocks were inactivated at 56°C for 30 min, freeze-dried in 1-ml volumes, and stored at 4°C. Freeze drying resulted in an approximately 10-fold reduction in levels of PCR-detectable viral RNA. Full details of the production and evaluation of the proficiency panel are available on the QCCA website (www.qcca.org.uk). Prior to release, external quality control testing was performed by two reference laboratories to assess sample quality and homogeneity. Four vials of each sample were analyzed in duplicate using the Amplicor EV-PCR assay (Roche Molecular Systems, Branchburg, N.J.) and an in-house PCR. A stochastic distribution of positive results was found in the highly diluted samples close to the detection limit of the assay in use (data not shown). To assess the effects of storage and of transportation on sample stability, samples were tested before and after 1, 2, 4, and 7 days of storage at room temperature. Identical results were obtained, with the exception of one borderline sample, which produced a 2.5-fold decrease in optical density values in the Amplicor EV-PCR assay over a period of 7 days (data not shown). Storage of the samples for 1 year at 4°C demonstrated no detectable loss of activity.

TABLE 1.

Composition of enterovirus proficiency panel and participant results

Sample code	EV serotype	Virus titer (TCID50/ml)a	PCR result in reference laboratoryb		No. of data sets (n = 70)
			1	2	Positive (correct dilution series)	Equivocal	Negative
EV-A12	CVA9	360	Pos	Pos	69		1
EV-A05	CVA9 (1% blood)	360	Pos	Pos	69		1
EV-A06	CVA9	36	Pos	Pos	63 (62)	1	6
EV-A09	CVA9	3.6	Pos	Pos	55 (54)	1	14
EV-A10	CVA9	0.36	Neg	Pos	30 (28)		40
EV-A11	CVA9	0.036	Neg	Neg	10 (8)		60
EV-A01	Echovirus 9	11,000	Pos	Pos	69		1
EV-A02	Echovirus 9	110	Pos	Pos	60	1	9
EV-A08	Poliovirus 2	1,600	Pos	Pos	64		6
EV-A04	HPEV 1	32,000	Neg	Neg	3		67
EV-A07	No virus		Neg	Neg	3		67
EV-A03	No virus		Neg	Neg	2		68

^aBefore inactivation and freeze-drying. 

^b1, Roche Amplicor assay; 2, qualitative in-house PCR. Pos, positive; Neg, negative. 

The proficiency panel was distributed at ambient temperatures to 63 European laboratories, which were asked to reconstitute each sample with 1 ml of water prior to testing using a molecular assay and to report their results to a neutral office (Manchester, United Kingdom) within 6 weeks. Additionally, a questionnaire was sent to obtain information on the various aspects of amplification procedures. Most participants (83%) received the panel within 3 days. Anonymized data sets were analyzed at the Department of Virology, University Medical Center, Utrecht, The Netherlands.

A total of 71 data sets were reported by 59 laboratories. One laboratory reported a data set with the results of a PCR specific for HPEV1. This was excluded, leaving 70 data sets for final analysis. The response to the questionnaires was high (70 of 71). Results are summarized in Table 1. The detection limits of the various assays as determined using the CVA9 dilution series varied by a factor 10³ to 10⁴. Correct dilution series results for the CVA9 samples (i.e., without inconsistent results in the sequence of the dilution series) were reported in 66 data sets. Six and seven of these 66 data sets tested positive for only the lower-dilution CVA9 samples EV-A12 and EV-A06, respectively. In 25 data sets, correct results were reported for the dilution series up to sample EV-A09; the higher-dilution samples EV-A10 and EV-A11 were identified as positive by 20 and 8 data sets, respectively. The participant that failed to detect virus in the sample containing 1% blood also failed to identify the corresponding sample without blood as positive. The five false-positive results (3.6%) were reported by four laboratories (6.8%). Of the three participants that identified sample EV-A04 as positive, only one identified this sample as a parechovirus. The remaining two positive results may represent false-positive results, since there is limited sequence homology between parechoviruses and enteroviruses (4, 14).

To assess proficiency, the test samples were classified as (strongly) positive (>1 TCID₅₀/ml of original virus stock), weakly positive (<1 TCID₅₀/ml), or negative. Sample EV-A04 was excluded from this analysis. A data set reporting correct results for all (strongly) positive and negative samples was considered to indicate adequate proficiency, since this corresponds in sensitivity and specificity to cell-culture based detection. In 46 data sets (66%), this level of proficiency was achieved.

To allow further comparison of performance and its relationship to PCR methodology in this and future proficiency panel distributions, a numerical performance score was assigned to each data set as follows: one point was given for every correct result, and one point was deducted for every false-positive or false-negative result, with the exception of negative results on the weakly positive samples. Equivocal results were not scored. Details of the assays employed were provided by 68 participating laboratories. Nested PCRs were associated with better performance than other PCR methods (Table 2). (P = 0.011, χ² analysis, Yates' correction). However, a maximum score was obtained at least once with each type of assay, indicating that additional factors, including staff proficiency and laboratory facilities, are likely to influence performance. This is further illustrated by the variation in performance observed among participants using the commercially available Roche Amplicor assay. No other aspects of assay design, such as RNA extraction, primer sequences, cDNA amplification, or PCR product detection, showed any statistical association with performance.

TABLE 2.

Comparison of performances and the type of nucleic acid amplification assay employed in the study

Score (points)	No. (%) of data sets	No. obtained with (n):
		Amplicor (16)	Single PCR (15)	Seminested PCR (11)	Nested PCR (26)
11	8 (11)a	1	1	1	4
10	18 (26)	2	3	2	11
9	21 (30)a	9	4	2	5
8	3 (4)	0	0	3	0
7	10 (14)	3	3	1	3
≤6	10 (14)	1	4	2	3

^aOne questionnaire was not available or suitable for analysis. 

Our study has thus shown that the majority of the participating laboratories have succeeded in introducing a reliable EV PCR assay, with several participants demonstrating an increased sensitivity relative to that of viral culture. However, a large group of participants (34%) performed inadequately. Performance was better among nested PCR users, and other aspects of assay design may also influence performance, although such effects could not be verified statistically in this study. Even when using a standardized assay, performance among laboratories varied, as has been observed in other studies of commercially produced PCR assays for enteroviruses (8, 9), human immunodeficiency virus (13, 16), or hepatitis C virus (2). This emphasizes the importance of ongoing quality assessments. For this reason the distribution of EV proficiency panels by the QCCA will continue. This multicenter quality assurance scheme can thus contribute to the optimization and standardization of molecular diagnosis of EV infections.