First Experience of a Multicenter External Quality Assessment of Molecular 16S rRNA Gene Detection in Bone and Joint Infections

Abstract

The objective of this study was to assess the performance of seven French laboratories for 16S rRNA gene detection by real-time PCR in the diagnosis of bone and joint infection (BJI) to validate a large multicenter study. External quality control (QC) was required owing to the differences in extraction procedures and the molecular equipment used in the different laboratories. Three proficiency sets were organized, including four bacterial DNA extracts and four bead mill-pretreated osteoarticular specimens. Extraction volumes, 16S rRNA gene primers, and sequencing interpretation rules were standardized. In order to assess each laboratory's ability to achieve the best results, scores were assigned, and each QC series was classified as optimal, acceptable, or to be improved. A total of 168 QCs were sent, and 160 responses were analyzed. The expected results were obtained for 93.8%, with the same proportion for extracts (75/80) and clinical specimens (75/80). For the specimens, there was no significant difference between manual and automated extraction. This QC demonstrated the ability to achieve good and homogeneous results using the same 16S rRNA gene PCR with different equipment and validates the possibility of high-quality multicenter studies using molecular diagnosis for BJI.

INTRODUCTION

Diagnosis and effective treatment of bone and joint infections (BJI) are largely based on bacterial documentation (1). Microbiological diagnosis remains difficult, mostly for chronic or low-grade infections, and requires prolonged culture conditions after pretreatments such as bead mill processing of perioperative samples (2) or implant sonication (3). Molecular biology provided a diagnostic tool that showed its effectiveness, especially when antibiotics were administered before surgery or when fastidious bacteria were involved, particularly for the diagnosis of endocarditis (4). For BJI diagnosis, broad-range 16S rRNA gene PCR has shown higher, lower, or equivalent sensitivity compared with conventional culture methods, but sometimes to the detriment of specificity (5–15).

These contradictory results may be due to different pretreatment procedures and a lack of standardization, which makes it difficult to compare the results of studies performed in different laboratories. The wide range of PCR performances in various monocenter studies underlies the interest in multicenter protocols designed to evaluate 16S rRNA gene PCR.

In 2011, within the Centre de Référence des Infections Ostéo-articulaires du Grand Ouest (CRIOGO) network, we carried out a multicenter study to assess the contribution of 16S rRNA gene PCR to BJI diagnosis (15). To qualify the centers participating in the study (Angers, Brest, Nantes, Orléans, Poitiers, Rennes, and Tours University Hospitals), we developed the first multicenter external quality control (QC) assessment for 16S rRNA gene PCR. The objective was to validate the consistency of the 16S rRNA gene PCR results submitted by the seven participating laboratories.

MATERIALS AND METHODS

Participants.

Seven French university hospital microbiology laboratories, which were involved in the multicenter study, participated to assess their proficiency in the use of 16S rRNA gene PCR for osteoarticular infection.

Specimen preparation and dispatch.

Four bacterial DNA extracts and four bead-milled suspensions of osteoarticular samples were sent three times (November 2010, June 2011, and March 2012) to the seven laboratories, representing a total of 168 samples. These QCs were prepared blind by the routine lab team at the Poitiers University Hospital microbiology laboratory. For DNA extracts (E), a 0.5 McFarland (McF) bacterial suspension was mixed with half bacterial lysis buffer (BLB) (Roche Molecular Biochemicals, Mannheim, Germany). Extraction was carried out with a MagNA Pure DNA isolation kit on a MagNA Pure compact instrument (Roche Molecular Biochemicals) according to the manufacturer's recommendations. Unusual and common osteoarticular agents (clinical strains) were selected (QC 1: Staphylococcus epidermidis, Propionibacterium acnes, and Staphylococcus aureus; QC 2: Staphylococcus lugdunensis, Enterococcus faecalis, and Haemophilus influenzae; QC 3: Corynebacterium striatum, Staphylococcus haemolyticus, and Morganella morganii). Twenty-microliter samples of these nine extracts or three negative samples (molecular biology-quality water) were frozen at −20°C until dispatch. For clinical specimens (S), nine samples from bone and joint infections with monomicrobial positive culture and three negative samples from patients undergoing primary total arthroplasties were selected. Infected tissues or bone biopsy specimens were sampled from two hip prosthesis (S2: Streptococcus agalactiae; S8: Proteus mirabilis), three knee prosthesis (S4: Streptococcus dysgalactiae; S6: Enterobacter cloacae; S10: S. agalactiae), and two vertebral column (S7: S. aureus; S11: Streptococcus oralis) samples and two cases of tibia or ankle osteitis (S3: Pseudomonas aeruginosa; S9: S. epidermidis). Four samples were sent to each laboratory three times during the study (see Table 3). Approval was obtained from the institutional review board, and informed consent was obtained from each patient before inclusion.

TABLE 3.

Laboratory scores and classification for each 16S rRNA gene PCR quality control series

Laboratory no.	Result for bacterial extract, expected responsea				Result for clinical specimen, expected response				Scoreb	Classc
	E1, S. epidermidis	E2, P. acnes	E3, negative	E4, S. aureus	S1, primary arthroplasty, negative	S2, hip prosthesis, S. agalactiae	S3, ankle osteitis, P. aeruginosa	S4, knee prosthesis, S. dysgalactiae
QC 1d
L1	+	+	−	+	−	+	+	+	8/8	Optimal
L2	+	+	−	+	−	+	FN	+	7/8	Acceptable
L3	+	+	FP	+	−	+	+	+	7/8	To be improved
L4	+	+	−	+	−	+	+	+	8/8	Optimal
L5	+	+	Unint.	+	−	+	+	+	7/8	Acceptable
L6	+	+	−	+	−	FN	+	+	7/8	Acceptable
	E5, S. lugdunensis	E6, negative	E7, E. faecalis	E8, H. influenzae	S5, primary arthroplasty, negative	S6, knee prosthesis, E. cloacae	S7, spondylodiscitis, S. aureus	S8, hip prosthesis, P. mirabilis
QC 2
L1	+	FP	+	+	−	+	+	+	7/8	To be improved
L2	+	−	+	+	−	+	+	+	8/8	Optimal
L3	+	Unint.	+	+	−	+	+	+	7/8	Acceptable
L4	+	−	+	+	−	+	FN	+	7/8	Acceptable
L5	+	−	+	+	−	+	+	+	8/8	Optimal
L6	+	−	+	+	−	+	+	+	8/8	Optimal
L7	+	−	+	+	−	+	+	+	8/8	Optimal
	E 9, C. striatum	E10, S. haemolyticus	E11, negative	E12, M. morganii	S9, tibia osteitis, S. epidermidis	S10, knee prosthesis, S. agalactiae	S11, spondylodiscitis, S. oralis	S12, primary arthroplasty, negative
QC 3
L1	+	+	−	+	+	+	+	−	8/8	Optimal
L2	+	+	−	+	FN	+	+	−	7/8	Acceptable
L3	+	+	−	+	+	+	+	−	8/8	Optimal
L4	+	+	−	+	+	+	+	Unint.	7/8	Acceptable
L5	+	+	−	+	+	+	+	−	8/8	Optimal
L6	+	+	−	+	+	+	+	−	8/8	Optimal
L7	+	+	Unint.	+	+	+	+	−	7/8	Acceptable

^a+, true positive, −, true negative; FN, false negative; FP, false positive; Unint., uninterpretable result.

^b(Number of true positives + number of true negatives)/number of samples.

^cOptimal, all results were correct; acceptable, 1 FN; to be improved, ≥2 FN or 1 FP.

^dLaboratory 7 did not participate in the first QC.

After the addition of 10 ml sterile water and 10 sterile 4-mm steel beads, the specimens were agitated on a Retsch MM401 bead mill for 2 min 30 s at 30 Hz/min (2, 15). Two hundred-microliter samples of each bead-milled suspension were stored frozen at −20°C until dispatch. All participants were informed by email about sending samples.

16S PCR and sequencing.

Before starting the study, the seven laboratories harmonized the main 16S PCR and sequencing steps. Specimens were extracted from 200-μl samples of each bead-milled suspension after pretreatment with proteinase K (concentration, 2 g/liter) for 3 h at 65°C. The participants used their own extraction reagent (Table 1). Real-time PCR was performed with Sybr green to target the 5′ part of the 16S rRNA gene (forward primer 27F, 5′-AGA GTT TGA TCM TGG CTC AG3′, and reverse primer 685R3, 5′-TCT RCG CAT TYC ACC GCT AC-3′; 658-bp amplification product; GenBank accession number NR024570). A negative control and a positive control (Roseomonas sp. DNA from a clinical strain of our own collection prepared and sent with QC) were run with each series. For the validation of a negative PCR, a human beta-globin gene fragment was amplified to control DNA extraction and the absence of a PCR inhibitor. Real-time PCR amplicons were systematically sequenced in both directions, and the resulting sequences were compared to those in the BIBI (Bioinformatics bacteria identification, http://umr5558-sud-str1.univ-lyon1.fr/lebibi/lebibi.cgi) and GenBank (BLAST, http://www.ncbi.nlm.nih.gov) databases. A sequence similarity of ≥98% was used to define the species level, and a similarity of 96% to <98% was used to define the genus level (16, 17). Participants were free to analyze samples according to their local molecular equipment and reagents (Table 1) and to optimize the most stringent PCR conditions (primer concentrations) that did not compromise sensitivity.

TABLE 1.

Description of molecular platform and reagents

Laboratory no.	Extraction method, manufacturer	Premix, manufacturer	Thermocycler, manufacturer
1	Manual, Qiagen	Sybr Ex Taq, TaKaRa	MX 3000, Stratagene
2	Manual, Qiagen	qPCR master mix, Promega	LightCycler 2.0, Roche
3	Automated, iPrep, Invitrogen	Sybr Ex Taq, TaKaRa	MX 3000, Stratagene
4	Automated, Easy Mag, bioMérieux	Sybr green master mix, Applied Biosystems	ABI 7900, Applied Biosystems
5	Automated, MagNA Pure Compact, Roche	Sybr Ex Taq, TaKaRa	SmartCycler, Cepheid
6	Manual, Qiagen	Sybr green master mix, Applied Biosystems	StepOne plus, Applied Biosystems
7	Manual, Qiagen	iQ Sybr green Supermix, Bio-Rad	Chromo4, Bio-Rad

Data gathering and statistical analysis.

Participants were asked to submit their results by return by email. Possible answers were negative, positive (genus and species according to the criteria defined above), and uninterpretable. Uninterpretable PCR results were positive amplifications with incorrect sequencing (mixed chromatogram, poor-quality sequences, or sequences without the correct percentage of similarity compared with the databases). The coordinating center classified the responses as true negative, true positive, false negative, or false positive (including identified bacteria and uninterpretable answers for a negative sample or extract). The numbers of correct species or genus identifications were noted. As reported by Escadafal et al. (18), we assigned one point for each correct answer, and false-positive or false-negative results were not scored. For each QC series, the participant laboratories were classified as optimal when all the results were correct, acceptable when only one response was a false negative, and to be improved when one or more false-positive or several false-negative results were reported.

A mean comparison of laboratory scores with or without automated extraction was performed via an unpaired Student's t test, with a P value of <0.05 considered statistically significant.

RESULTS

The different reagents and equipment for extraction and 16S rRNA gene real-time PCR used by laboratories are shown in Table 1.

The response rate was 95% (160/168), with one laboratory not taking part in the first QC series. The global results are shown in Table 2. The overall rate of correct answers is 93.8% (150/160), with the same proportion for bacterial DNA extracts (75/80) and specimens (75/80).

TABLE 2.

16S rRNA gene PCR quality control results for bacterial DNA extracts and bone and joint infection specimens

Proficiency set	Extract or specimen no.a	Expected results	No. of accurate results/total no. (%)	No. of accurate results at genus level only
First	E1	S. epidermidis	6/6
	E2	P. acnes	6/6
	E3	Negative (water)	4/6
	E4	S. aureus	6/6	1
	S1	Negative	6/6
	S2	S. agalactiae	5/6
	S3	P. aeruginosa	5/6
	S4	S. dysgalactiae	6/6
Second	E5	S. lugdunensis	7/7
	E6	Negative (water)	5/7
	E7	E. faecalis	7/7
	E8	H. influenzae	7/7
	S5	Negative	7/7
	S6	E. cloacae	7/7	7
	S7	S. aureus	7/7
	S8	P. mirabilis	7/7
Third	E9	C. striatum	7/7	1
	E10	S. haemolyticus	7/7	1
	E11	Negative (water)	6/7
	E12	M. morganii	7/7
	S9	S. epidermidis	6/7	5
	S10	S. agalactiae	7/7
	S11	S. oralis	6/7	1
	S12	Negative	6/7
Total	Extract		75/80
	Sample		75/80
	Tests		150/160 (93.8)

^aE1 to E12, bacterial DNA extracts; S1 to S12, BJI specimens.

Out of 60 positive DNA extract samples, 95% (57/60) were matched at the species level and 5% (3/60) only at the genus level, with no false negatives (Table 3). Out of 20 negative water extracts, 3 were uninterpretable (with a positive PCR without a correct sequence analysis) and two were false positives (a Eubacterium sp. and Escherichia coli). Out of 60 positive clinical specimens, 71.7% (43/60) were matched at the species level and 21.7% (13/60) only at the genus level, and 6.6% (4/60) of the responses were false negatives (four different microorganisms in three different laboratories). Out of 20 negative clinical specimens, there was only one false-positive result that was uninterpretable owing to incorrect sequencing. In the particular context of this quality control, overall sensitivity and specificity in comparison with the expected results were 96.7% and 86.4%, respectively, with 100% and 80% for DNA extracts and 93.3% and 95% for clinical specimens.

To assess each laboratory's ability to achieve the best results, scores were assigned, and each QC series was classified as optimal, acceptable, or to be improved (Table 3). All laboratory scores were 7/8 or 8/8 for each QC. In the first QC set, involving six laboratories, one was classified as to be improved (for one false-positive response), three were classified as acceptable (two false-negative responses and one uninterpretable response), and two were classified as optimal. In the second QC set, involving seven participants, one laboratory was classified as to be improved (one false-positive response from a different laboratory than the first QC), two were classified as acceptable (one false-negative and one uninterpretable response), and four were classified as optimal. Finally, in the third QC, three laboratories were classified as acceptable and four as optimal; no laboratories were classified as to be improved.

There were no significant differences between the QC scores of laboratories using automated extraction (laboratories 3, 4, and 5) and the QC scores of laboratories using manual extraction (laboratories 1, 2, 6, and 7) (P = 0.43).

DISCUSSION

This QC aimed to harmonize and assess the performance of seven French university hospital laboratories for 16S rRNA gene real-time PCR in BJI diagnosis. The results validated the laboratories' participation in a prospective multicenter study to assess the contribution of 16S rRNA gene PCR in the diagnosis of BJI (15). To the best of our knowledge, this is the first multicenter quality control for 16S rRNA gene real-time PCR with bacterial DNA or clinical samples.

In this quality control, two types of samples were used. DNA extracts of bacterial strains facilitated monitoring of the technique (PCR, sequencing, and database use) independently of extraction difficulties. Using these samples, some laboratories were able to optimize their PCR conditions after the first QC (laboratories had access to the results, including threshold cycles, of all other participants). We also wished to assess the laboratories' abilities for the pretreatment and DNA extraction of bone and joint specimens, which are known to be difficult to extract. The advantage of using bead mill processing for the specimens was that we split a highly homogeneous clinical sample between the participating laboratories. In this study, the 16S rRNA gene PCR quality control did not assess variability in the bead-beating procedure. Nevertheless, all participating laboratories used the same equipment (Rescht MM401 and 4-mm steel beads) and the same procedure (2 min 30 s at 30 Hz/min). Finally, our results showed that manual and automated extraction techniques recorded similar performances for osteoarticular specimens.

Results concordance (95%) was high, with few discrepancies observed. Sensitivity and specificity in comparison with the expected results (95% and 93%, respectively, for specimens) were high compared with other studies (5, 7, 11). These results are related to the choice of samples for quality control. DNA extracts derived from rich bacterial suspensions and clinical samples were chosen because they were positive in culture (10 to 50 CFU per plate). The objective of this study was not to evaluate the performance of 16S rRNA gene PCR for the diagnosis of BJI but rather to assess the performances of laboratories using this technique while taking into account the pitfalls and difficulties encountered. Therefore, this QC does not include certain bacteria which are difficult to extract and detect with 16S PCR. Nevertheless, during our study, using the same primers, poor detection of P. acnes was noted (15).

Analysis of the few failures found that they highlight the major difficulties involved in using broad-range PCR. First, for some organisms, identification cannot be accurate to the species level. Indeed, E. cloacae was identified as E. cloacae complex, and S. epidermidis was poorly separated from other negative Staphylococci. In fact, there is a large difference between the rates of level identification in DNA extracts versus clinical specimens (95% versus 71%, respectively). This worst rate of species-level identification is due to the bacterial species and does not indicate a defect in 16S PCR performance in specimens.

False-positive results represent the major pitfall. Despite the essential precautions taken by a molecular biology laboratory, contamination related to the laboratory environment may occur. In this case, two false-positive results with E. coli and Eubacterium sp. were noted. Unfortunately, with 16S rRNA gene PCR, some false-positive results can also arise directly from contaminants in the molecular reagents (19–21). During our study, several problems of this type occurred; contamination of the premix by Pseudomonas orientalis affected two laboratories using the same lot of this reagent or Acinetobacter spp. in the lysis buffer used for automated extraction. Fortunately, negative controls in each series meant that the problem was quickly detected, and it was subsequently possible to repeat the analysis with another lot of reagent. DNA-free reagents are now available in commercial special kits for 16S rRNA gene PCR and should help to improve test specificity (22).

As regards false-negative results, several authors have extensively documented the poor sensitivity of broad-range amplification and sequencing from complex specimens. They recommended that this type of PCR be used only in exceptional circumstances, such as for patients treated with antibiotics, or when a strong suspicion of infection in a negative culture justifies a search for fastidious bacteria (14, 23).

Clinical bone and joint specimens contain high levels of human DNA and variable levels of bacterial DNA. This factor may be responsible for the cross-reactivity of 16S primers with human DNA. Kommendal et al. (24) described mixed chromatograms or amplification of fragments corresponding to part of human chromosome 9. Our primers can also lead to nonspecific amplification of 150 bp of human chromosome 1, so these cases were classified as negative (data not shown). Some authors have proposed extraction techniques, including a smaller amount of human DNA to eliminate PCR false-positive results, but this can also reduce the amount of bacterial target DNA, thereby leading to reduced sensitivity (25, 26). Moreover, to improve performance, it would be interesting to increase the volume of samples analyzed by molecular tools. The contribution of large-volume extraction kits should be evaluated.

Finally, difficulty lies in the quality of DNA sequences. The length of the fragment selected in this study (658 bp) discriminates quite closely among different bacterial species but remains more difficult for sequencing. In BJI molecular diagnosis, polymicrobial infections are also a cause of uninterpretable DNA sequences. Only a cloning step can provide a reliable result, but it is not feasible as part of a laboratory routine (6).

Despite the pitfalls associated with 16S rRNA gene PCR, the results are highly satisfactory. An improvement in laboratory scores was observed throughout the study. After each QC series, meetings were organized to present and evaluate the results, and they led to discussion of possible improvements for each center. In our experience, external quality control allowed our microbiologist group to improve the daily use of broad-range PCR. The success rate was very high despite the heterogeneity of the equipment used in each laboratory. It was impossible to compare performances between the laboratories based on the different PCR platforms because of the small numbers of each platform involved. Nevertheless, general observation showed that the threshold cycles were closest for laboratories using the same PCR reagent-PCR platform association (data not shown). For clinical osteoarticular specimens, all of the laboratories used the same pretreatment conditions (bead mill and protein K lysis), and the study showed that there was no influence from the extraction technique that was chosen.

In conclusion, to the best of our knowledge, the results of this multicenter external quality control assessment are the first to reveal excellent concordance in terms of performance among seven laboratories routinely using the same type of 16S rRNA gene PCR amplification method. PCR and sequencing results demonstrated good overall performance, with scores ranging from acceptable to optimal for both bacterial extracts and clinical samples. This study demonstrates the ability to achieve good and homogeneous results using the same 16S PCR in laboratories with different equipment and confirms the possibility of conducting high-quality multicenter studies using molecular diagnosis for bone and joint infections.