Reply to Humphries and Simner, “Verification Is an Integral Part of Antimicrobial Susceptibility Test Quality Assurance,” and Wojewoda et al., “College of American Pathologists (CAP) Microbiology Committee Perspective: the Need for Verification Studies”

REPLY

We thank Drs. Humphries and Simner and the CAP Microbiology Resource Committee for their interest in our commentary (1). In their letters to the editor, they bring up points directly addressed in our article as well as related issues. Both deserve further discussion.

As a general observation, the existence of a regulation described by the CAP Microbiology Resource Committee does not mean that it is appropriate and helpful. The reality is that an ill-defined and ill-conceived hurdle prevents clinical laboratories from implementing AST for new antibiotics. Surely some patients with resistant infections will have adverse outcomes in the absence of AST to guide the appropriate use of newly available agents. Such AST-directed therapy for the latest antimicrobials should not remain the province of a few well-funded and well-staffed clinical laboratories, when antimicrobial-resistant pathogens know no such boundaries.

In their letters, Drs. Humphries and Simner and the CAP Microbiology Resource Committee support the requirement for accuracy and precision verification studies prior to implementation of testing for new antimicrobials on systems and/or using methods previously verified for use in a clinical laboratory. Implicit in their comment letter and in their interpretation and endorsement of federal regulations, respectively, is the assumption that verification studies of the type proposed are adequately powered to evaluate AST method performance and therefore contribute meaningfully to the goal of providing safe and useful data for patient care. However, these studies do not provide adequately powered accuracy and precision data.

For example, the performance of an AST verification study with 30 isolates (containing 15 isolates resistant to the antibiotic being studied) with 1 very major error (VME) would be considered unsatisfactory based on a >3% VME rate. Notably, the 95% confidence interval for this VME rate is ≈0% to 32% (2; modified Wald method, GraphPad QuickCalcs, https://www.graphpad.com/quickcalcs/ConfInterval1.cfm). However, if no VME were identified in the same study, the performance would be considered adequate, despite similar confidence intervals of 0% to 24%. We should not delude ourselves that such small studies, whose interpretation can be inappropriately swayed by a single aberrant result, can ensure the accuracy and precision of AST methods.

To alleviate the burden of these verification studies, a burden that discourages laboratories from adopting AST for new antibiotics (1), Drs. Humphries and Simner propose a risk-based stratification model in which, with medical director discretion, even fewer than the 30 isolates recommended by some authors (see Cumitech 31A [3]) can be tested. Such small studies for adding new antibiotics to existing systems have also been advocated in the CLSI M52 document cited by the CAP Microbiology Resource Committee in their letter (4). Such studies would suffer to an even greater extent from the statistical limitations noted above and would not establish whether the method is performing according to FDA accuracy and precision metrics (5).

FDA clearance of AST methods is established by testing hundreds of isolates, including those with resistance mechanisms that may rarely be encountered and could not be evaluated in any statistically meaningful way by an individual clinical laboratory. We therefore argue that quality has to be assured at the time of FDA clearance/approval and perpetuated through federally mandated manufacturer quality system regulations (6). This is analogous to the quality systems that ensure that the antibiotics being used to treat patients are what they claim to be without the need for additional verification at each site of use.

Drs. Humphries and Simner further argue that a quality control (QC)-based approach, which we advocate, would be insufficient to ensure the adequate performance of AST methods. They list as an example potential varying results depending on the supplier of Mueller-Hinton agar or lot-to-lot variability. QC ranges promulgated by the Clinical Laboratory and Standards Institute span three to four doubling dilutions or a disk diffusion zone range that takes into account the biological variability inherent in AST (7). Notably, if the method was not performing appropriately and the AST method was capable of testing within the QC range, then initial 20-day QC testing centered inappropriately at either end of the range would presumably fail statistically and therefore provide appropriate feedback to the laboratory initially and thereafter on an ongoing basis. Therefore, the cited variability in agar medium should be detected by QC testing.

Drs. Humphries and Simner note that the QC range for meropenem-vaborbactam is substantially lower than their breakpoints (8). We thank them for emphasizing the point that QC ranges for many antibiotics lie several doubling dilutions below those tested on breakpoint panels and therefore are not by themselves adequate to ensure the ongoing performance of several commercial methods other than by detecting catastrophic method failure. We have made note of this QC issue previously in this journal (9), and we argue that this is yet an additional reason that reliability needs to be established during the initial AST clearance process and maintained through the existing quality system regulatory framework that ensures ongoing, reliable manufacturing of AST devices (6).

Drs. Humphries and Simner indicate that verification is needed for off-label testing, for example, for use with new breakpoints not addressed in the U.S. Food and Drug Administration (FDA) clearance. We agree. In our commentary (1), we specifically advocate the elimination of supplementary verification testing for new antibiotics and new panels only when they are used with methods and systems already established and verified by the clinical laboratory and only when they are performed according to the instructions in the manufacturer’s FDA-cleared/approved package insert.

Finally, to address the burden of verification studies, Drs. Humphries and Simner propose initiatives to increase reimbursement for AST so that all laboratories would have the resources to devote to verification studies. The United States already spends twice as much on health care as other industrialized nations, with less favorable outcomes (10). We believe that supplemental AST verification studies are emblematic of such nonproductive health care expenditure. In this vein, it is time for CLIA regulations for verification studies related to AST referenced by the CAP Microbiology Resource Committee to be rewritten and/or for their vague wording to be clarified and restated along the lines that we suggest. A limited initial verification study when AST methods are first introduced in a specific laboratory should be performed to establish that operator-dependent and -independent variables do not affect method performance, as we elaborate more fully in our original commentary (1). These studies cannot robustly establish the accuracy and precision of the method, and likely these terms should be changed to accurately describe and reflect the purpose of the initial verification activity. Thereafter, by adopting the commonsense, QC-centered approach advocated in our commentary, it will be possible for labs of all sizes and resource levels to add testing of new antibiotics to existing systems and methods, without additional expenditure, and thereby to improve patient outcomes.