A multi-site, international laboratory study to assess the performance of penicillin susceptibility testing of Staphylococcus aureus

Abstract

Objectives

There is clinical uncertainty over the optimal treatment for penicillin-susceptible Staphylococcus aureus (PSSA) infections. Furthermore, there is concern that phenotypic penicillin susceptibility testing methods are not reliably able to detect some blaZ-positive S. aureus.

Methods

Nine S. aureus isolates, including six genetically diverse strains harbouring blaZ, were sent in triplicate to 34 participating laboratories from Australia (n = 14), New Zealand (n = 6), Canada (n = 12), Singapore (n = 1) and Israel (n = 1). We used blaZ PCR as the gold standard to assess susceptibility testing performance of CLSI (P10 disc) and EUCAST (P1 disc) methods. Very major errors (VMEs), major error (MEs) and categorical agreement were calculated.

Results

Twenty-two laboratories reported 593 results according to CLSI methodology (P10 disc). Nineteen laboratories reported 513 results according to the EUCAST (P1 disc) method. For CLSI laboratories, the categorical agreement and calculated VME and ME rates were 85% (508/593), 21% (84/396) and 1.5% (3/198), respectively. For EUCAST laboratories, the categorical agreement and calculated VME and ME rates were 93% (475/513), 11% (84/396) and 1% (3/198), respectively. Seven laboratories reported results for both methods, with VME rates of 24% for CLSI and 12% for EUCAST.

Conclusions

The EUCAST method with a P1 disc resulted in a lower VME rate compared with the CLSI methods with a P10 disc. These results should be considered in the context that among collections of PSSA isolates, as determined by automated MIC testing, less than 10% harbour blaZ. Furthermore, the clinical relevance of phenotypically susceptible, but blaZ-positive S. aureus, remains unclear.

Introduction

The proportion of Staphylococcus aureus invasive infections reported as penicillin susceptible has increased worldwide. Recent reports have determined that penicillin-susceptible S. aureus (PSSA) make up 13%–57% of invasive S. aureus.^1–4 The optimal treatment for PSSA infections remains unknown, and clinical practice guidelines that assume high levels of penicillin resistance amongst invasive infections due to S. aureus have not been updated since the increased prevalence of S. aureus penicillin susceptibility.^5,6 Clinical outcomes may be better with benzylpenicillin than with isoxazolyl penicillins (e.g. oxacillin, cloxacillin) due to the ability to achieve higher free drug concentrations with penicillin in conjunction with lower rates of toxicity compared with cloxacillin.^6–8 However, due to concern over the accuracy of β-lactamase detection methods in S. aureus, guidelines do not recommend the use of penicillin for treatment.⁵ Consequently, in many laboratories, routine testing for penicillin susceptibility has been discontinued.

Phenotypic susceptibility methods to detect penicillin resistance are reliant upon the complex inducible pathway of penicillinase production in S. aureus, which is encoded by blaZ and regulated by blaZ/blaI/blaR1, whereby some isolates that may test as penicillin susceptible by disc or MIC testing produce relatively low levels of penicillinase in vitro.⁹ At present, CLSI and EUCAST recommend phenotypic testing with a P10 (penicillin 10 U) or P1 (penicillin 1 U) disc, respectively, rather than relying solely on MIC breakpoints.^10,11 By using the penicillin disc, isolates harbouring blaZ may still appear susceptible according to the zone of inhibition size, but low-level blaZ expression may be identified by inspection of the zone edge, and is indicated by the appearance of a sharp edge (no reduction of growth towards the zone edge, like a ‘cliff’). If the edge is not sharp (reduction of growth towards zone edge, like a ‘beach’) isolates are then reported as penicillin susceptible. Phenotypic disc testing that included assessment of both zone of inhibition size and the zone edge at centralized reference laboratories has been determined to be sensitive in detection of β-lactamase production.¹²

The S. aureus Network Adaptive Platform (SNAP) trial aims to investigate the optimal therapy for S. aureus bloodstream infections (BSIs) through the comparison of different therapies according to the antimicrobial resistance profile.¹³ As such, an arm of SNAP will investigate the optimal therapy for PSSA BSIs through an evaluation of clinical outcome difference between patients treated with benzylpenicillin versus flucloxacillin or cloxacillin. As a pragmatic study, patients will be allocated to the PSSA silo based on standard laboratory practice at the recruiting hospital. Similar to previous investigator-led studies, the laboratory testing methodology for antimicrobial susceptibility testing (AST) may vary between CLSI and EUCAST and between automated platform-based testing or manual testing with disc diffusion or gradient MIC strips. To mitigate against the risk of falsely inferring penicillin susceptibility, SNAP trial sites wishing to participate in the PSSA silo are required to perform confirmatory testing with a P1 or P10 disc, interpreted according to EUCAST or CLSI, respectively, on all penicillin-susceptible isolates. The aim of this study was to assess penicillin testing performance using the P1 and P10 disc according to EUCAST and CLSI methodology across trial sites in Australia, Canada, Singapore, Israel and New Zealand.

Methods

Isolate selection

Nine isolates (as shown in Table 1) were selected from a broader collection of 472 previously well-characterized S. aureus isolates. Selected isolates tested penicillin susceptible according to VITEK 2 (bioMérieux, France) MIC testing (penicillin MIC ≤ 0.12 mg/L), were cefoxitin disc screen-negative for MRSA according to EUCAST breakpoints and had been categorized by their blaZ PCR result.¹⁴ Of note, penicillin testing using disc diffusion was performed using benzylpenicillin; from which other penicillin results can be inferred. Four isolates were chosen that had zone diameter inhibition less than the susceptible EUCAST breakpoint when tested by the investigators. One isolate (strain F) was included that consistently had zone diameter breakpoints greater than the susceptible EUCAST breakpoint but also had a sharp edge indicating the presence of a functional β-lactamase. This isolate is important to assess performance as the method described by EUCAST and CLSI relies not only on the zone of inhibition, but also the laboratory correctly recording the presence or absence of a sharp edge to correctly attribute a resistant or susceptible category. The nine isolates were sent in triplicate and tested by participating laboratories in a blinded manner. Although not reflective of the general penicillin-susceptible S. aureus blaZ prevalence (∼7% in the aforementioned collection) a high proportion of strains harbouring blaZ were chosen to provide a conservative upper-bound estimate of major errors (MEs) and very major errors (VMEs). Five of the six blaZ-harbouring strains had disc diffusion zone diameters less than the EUCAST breakpoint for penicillin (susceptible ≥26 mm) when centrally tested. The other blaZ-harbouring isolate had a zone diameter greater than the susceptible breakpoint and therefore relied on interpretation of the zone edge to correctly classify the isolate as penicillin resistant. At participating laboratories, results were reported based on EUCAST (v12.0) and CLSI (M100-ED32:2021) breakpoint tables.^10,11 Further details regarding EUCAST and CLSI testing are found in Table S1, available as Supplementary data at JAC Online.

Table 1.

Nine isolates (isolate groups A to I) were provided in triplicate (labelled as ‘SN identifier’)^a

Isolate group	SN identifiers	VITEK 2 benzylpenicillin categorization	blaZ PCR result	P1 zone-of-inhibition diameter (mm)
A	SN-1, SN-28, SN-30	Susceptible	Detected	22
B	SN-2, SN-27, SN-29	Susceptible	Not detected	29
C	SN-3; SN-9; SN-18	Susceptible	Detected	22
D	SN-4; SN-12; SN-15	Susceptible	Not detected	32
E	SN-5; SN-7; SN-17	Susceptible	Not detected	35
F	SN-6; SN-10; SN-11	Susceptible	Detected	28
G	SN-8; SN-14; SN-19	Susceptible	Detected	23
H	SN-16; SN-23; SN-25	Susceptible	Detected	22
I	SN-21; SN-22; SN-26	Susceptible	Detected	23

Additional testing details including PCR testing conditions can be found in Henderson et al.⁷

Data entry and collection

Participating laboratories were allocated a unique access group in a REDCap database for the purpose of data entry. Laboratories were required to enter laboratory testing method (EUCAST versus CLSI), zone of inhibition diameter, zone edge interpretation (cliff or edge) and final AST interpretation. Participating laboratories could enter results according to both EUCAST and CLSI methods in separate data collection forms. To better reflect ‘real-world’ practice the database was locked following data entry with no subsequent data verification prior to analysis.

Analysis

Data were analysed using R. VMEs were calculated as the proportion of blaZ-positive isolates reported as penicillin susceptible. MEs were calculated as the proportion of blaZ-negative isolates reported as penicillin resistant. Categorical agreement between laboratory-reported results and reference testing was assessed by the interpretive susceptibility category of susceptible or resistant against the known blaZ PCR result. The effect of the zone of inhibition diameter size on interpretation of the zone edge was performed for blaZ-positive isolates with zone diameter size greater than the susceptible EUCAST and CLSI breakpoint using an unpaired two-sample t-test.

Results

The 34 participating laboratories were from Australia (n = 14), New Zealand (n = 6), Canada (n = 12), Singapore (n = 1) and Israel (n = 1). Twenty-two laboratories reported results for CLSI and 19 for EUCAST methods, with 7 reporting for both methods. Twenty-five laboratories provided details of current routine reporting of antimicrobials for isolates determined to be PSSA from blood culture specimens, with 18 out of 25 reporting isolates as penicillin susceptible to clinicians based on standard operating practice in their laboratory. The other seven laboratories did not routinely report penicillin-susceptibility results.

Results according to laboratories using EUCAST method

Of the 19 laboratories reporting penicillin susceptibility according to EUCAST methods and interpretation, 10 were in Australia, 6 in Canada, 2 in New Zealand, and 1 in Singapore. The distribution of 513 individual tests is shown in Figure 1(a), with 55 of the 342 blaZ-positive results associated with a zone diameter size greater than the EUCAST susceptible cut-off of ≥26 mm. For isolates appearing susceptible based on zone size, 18/55 (33%) were correctly interpreted as resistant because of a sharp zone edge. Examining the aggregate zone size distribution by isolate triplicate, strain F consistently resulted in a zone diameter(s) greater than the EUCAST breakpoint, resulting in only 45% of tests for this triplicate being correctly reported (Figure 2a). The difference between zone diameter size for isolates incorrectly reported as susceptible was not significantly different from the resistant isolates (mean 27 versus 27 mm, P value 0.14). Using the blaZ PCR result as the gold standard, Table 2 demonstrates the overall performance of the EUCAST method with a calculated VME rate of 11% (37/342) and ME rate of 1% (1/171). The categorical agreement between the reported results according to EUCAST methods compared with blaZ PCR was 93% (475/513). One isolate that did not harbour blaZ was noted to have a zone size reported as 21 mm, despite the laboratory correctly assigning the categorical interpretation as susceptible. Further correspondence confirmed this to be a transcription error in zone of inhibition measurement.

Figure 1.

Penicillin zone size (mm) distribution by method. The number of laboratory tests by diameter of zone size split and coloured by the isolates blaZ PCR result (detected—red; not detected—grey). (a) and (b) depict results using the EUCAST (P1 disc) and the CLSI (P10 disc) methods, respectively, with method breakpoints indicated by the dashed lines. This figure appears in colour in the online version of JAC and in black and white in the print version of JAC.

Figure 2.

Overall laboratory penicillin confirmatory testing results showing aggregated zone diameter by isolate. Aggregated penicillin zone diameter (mm) box and whisker plot (median with 95% CI) for the nine isolates in triplicate labelled A to I on the x-axis. Boxes filled in red represent isolates that are blaZ PCR positive. (a) and (b) depict results using the EUCAST (P1 disc) and the CLSI (P10 disc) methods, respectively. The numbers above the triplicate groups indicate the percentages of correct results. This figure appears in colour in the online version of JAC and in black and white in the print version of JAC.

Table 2.

Overall confirmatory penicillin testing performance by disc phenotype compared with blaZ PCR detection

	Susceptible n (%)	Resistant n (%)	Total n
EUCAST methodology (n = 19 laboratories)
ȃblaZ PCR detected	37 (11)	305 (89)	342
ȃblaZ PCR not detected	170 (99.4)	1 (0.6)	171
ȃTotal	207	306	513
CLSI methodology (n = 22 laboratories)
ȃblaZ PCR detected	83 (21)	312 (79)	395
ȃblaZ PCR not detected	195 (98)	3 (1.5)	198
ȃTotal	278	315	593

Results according to laboratories using CLSI method

Of the 22 laboratories reporting penicillin susceptibility testing results according to CLSI method and interpretation, 11 were in Canada, 9 in Australia and 1 each in New Zealand and Israel. Overall, 593 individual tests across 27 isolates were reported, with one isolate not reported for a single laboratory. The zone size distribution was shifted to the right relative to the EUCAST method, with a greater proportion of blaZ-positive isolates [242/395 (61%)] obtaining a zone diameter size greater than the susceptible cut-off of ≥29 mm (Figure 1b). Of the 242 blaZ-positive isolates with zone diameter in the susceptible range, 159 (66%) were interpreted correctly through identification of a sharp zone edge. In contrast to the EUCAST method, a zone size difference was found between correctly and incorrectly categorized isolates (mean 32 versus 35 mm, P value <0.001). Inter-laboratory results showed strains A, C and F consistently above the breakpoint (Figure 2b). Like the EUCAST method, the percentage of tests correct by triplicate was linked to zone size relative to the breakpoint. Using the blaZ PCR result as the gold standard, Table 2 demonstrates the aggregated results, with a calculated VME rate of 21% (84/395) and ME rate of 1.5% (3/198). The categorical agreement between results reported by laboratories using CLSI methods compared with blaZ PCR was 85% (506/593).

Seven laboratories provided data for both methods. The VME rate was similar to the entire dataset with VME rates of 12% and 24% for the EUCAST and CLSI methods, respectively. Laboratories that reported penicillin susceptibility routinely were compared with those who did not report penicillin susceptibility. No differences in results were observed when laboratories were grouped by reporting of penicillin susceptibility (data not shown).

Discussion

We assessed the clinical laboratory performance of penicillin susceptibility testing according to CLSI and EUCAST methodology against nine S. aureus strains tested in triplicate by 34 laboratories in five different countries. The isolates chosen for this study represented six blaZ-positive and three blaZ-negative strains previously testing as penicillin susceptible by VITEK 2 (bioMérieux, France).

An important finding of the study was the difference in the VME rate for laboratories that used EUCAST versus CLSI methodology, at 11% versus 21%, respectively. This finding was consistent when assessed by the seven laboratories that performed and reported both CLSI and EUCAST methods. To our knowledge, this is the first large-scale multi-laboratory assessment of penicillin susceptibility testing on isolates classified as PSSA by VITEK 2 comparing CLSI and EUCAST methods. Although a previous study by Papanicolas et al.¹⁵ compared both methods against 38 blaZ-positive isolates, two differences should be noted. The study was performed at a single centre with all isolates having zone diameter sizes less than the EUCAST breakpoint (i.e. the correct result should have been able to be made based on the zone size alone). Regardless, like our study, the EUCAST P1 disc performed better than the CLSI P10 disc (sensitivity 100% versus 89%).¹⁵ In contrast, a Danish and USA study testing 45 blaZ-positive isolates found both methods performed well with a sensitivity of >95% provided the zone edge was interpreted correctly.¹²

One explanation for the high VME rate and low rate of MEs observed in our study was the inclusion of one triplicate (strain F) with a zone diameter size greater than the susceptible EUCAST breakpoint. Of the laboratories reporting susceptibility testing according to the EUCAST method, 45% of tests were reported as penicillin resistant for this strain (strain F). This pattern was more marked using the CLSI method for strain F. The correct result in these cases is reliant on the interpretation of the zone edge, which our study suggests remains difficult and ‘subjective’, leading to numerous errors. The reason for the poorer performance of the CLSI method is linked to both the antibiotic concentration used in the test disc and/or the breakpoint leading to a greater reliance on the zone edge interpretation. This is best illustrated by the zone size distribution, which failed to split the resistant/susceptible populations with the zone of inhibition diameters greater than the CLSI breakpoint in 61% of the blaZ-positive isolates and incorrect results for strains A and C. Our study would therefore suggest reviewing the calibration of the CLSI method to improve its performance.

Although the VME rate of 11% (EUCAST) and 21% (CLSI) is much higher than acceptable from a laboratory-based perspective, the clinical impact is far less clear. First, the study was enriched for blaZ-positive PSSA isolates with a penicillin MIC of ≤0.12 mg/L. These isolates represented 7% (34/472) of the original collection, with 5 of 34 (1%; 5/472) mimicking strain F.¹⁴ A similar overall prevalence (6.9%) was detected by Richter et al.¹⁶ among 448 isolates in a national surveillance collection from the USA while a recent multicentre Spanish study found only 3% of PSSA isolates harboured blaZ.⁸ Therefore, we estimate that only 0.7% (EUCAST) to 1.4% (CLSI) of isolates with penicillin MICs ≤0.12 mg/L, as determined by automated testing, could be miscalled as susceptible when in fact harbouring blaZ. Second, the clinical implication of treatment of a patient with benzylpenicillin for a BSI due to S. aureus that harbours blaZ but which is susceptible according to MIC and zone diameter breakpoint is unknown especially as non-functional blaZ isolates occur.¹² Third, observational data have not demonstrated that treating patients with BSIs due to PSSA with benzylpenicillin rather than flucloxacillin was associated with poorer outcomes. Using current testing methods for PSSA, retrospective cohort studies have shown similar and lower 30 day mortality rates in patients receiving benzylpenicillin.^7,17

Taken together, the small number of isolates likely impacted by phenotypic miscalling of the presence of blaZ and the unclear clinical relevance of such a miscall when the phenotypic MIC is well below achievable levels of penicillin with recommended dosing, means that the comparative effectiveness of benzylpenicillin versus (flu)cloxacillin remains an important clinical question. With regard to the SNAP Trial, the SNAP Microbiology Working Group has recommended that the EUCAST method is preferred over the CLSI method for penicillin testing against S. aureus. Additional trial safety procedures will occur including ongoing monitoring of trial site laboratory testing results compared with centralized reference testing of PSSA isolates, and the data and safety monitoring committee will review primary and secondary outcomes within the PSSA silo and antibiotic backbone domain as the trial proceeds.

Conclusions

This study demonstrates the real-world performance of P1 and P10 disc testing against S. aureus strains. Although the EUCAST method provided a lower VME rate, this study showed that the error rate is still high at 11%. These results should be interpreted in the real-world context where the overall prevalence of blaZ amongst PSSA strains is less than 10%.

Funding

The Staphylococcus aureus Network Adaptive Platform trial receives funding from the Australian National Health and Medical Research Council, the New Zealand Health Research Council, the Canadian Institutes of Health Research, the Singapore National Medical Research Council and the United Kingdom National Institutes of Health Research.

Transparency declarations

All authors have no conflicts of interest to declare.

Supplementary data

Table S1 is available as Supplementary data at JAC Online.