The Value of Outcomes Data in the Practice of Clinical Microbiology

EDITORIAL

“I don't see much sense in that,” said Rabbit. “No,” said Pooh humbly, “there isn't. But there was going to be when I began it. It's just that something happened to it along the way.”

—A. A. Milne, Winnie-the-Pooh, 1926

In the beginning, there were triple sugar iron (TSI) slants. These were followed by API strips; then came the Vitek machine and, after that, nucleic acid amplification. Today, we have matrix-assisted laser desorption ionization–time of flight mass spectrometry (MALDI-TOF MS), and tomorrow, we will have next-generation sequencing. All of this developed over a 40-year period of time. Technology in clinical microbiology has evolved at a dizzying pace. And yet, when last I checked, save for the accumulation of various and sundry determinants of virulence and antimicrobial resistance, Escherichia coli in 2014 is little different from E. coli in 1974. The same is true of Klebsiella pneumoniae, Pseudomonas aeruginosa, and Acinetobacter baumannii. The difference is that we get to a definitive identification using bells and whistles rather than simple technologist insight. Oh, and yes, in at least some instances, at wildly greater cost. I wonder what Rabbit would say.

Rabbit would have asked the question, “Does there exist objective evidence that new technologies in clinical microbiology, irrespective of their focus, have, in any clearly definable way, actually contributed positively to the care of patients with infection?” He might have wondered if there exist selected circumstances in which new technologies have actually impacted negatively on patient care. Rabbit would certainly have acknowledged that the foundation of any new laboratory procedure is analytical reproducibility. The damned thing had better do what it is intended to do. And for the most part, the discipline of clinical microbiology has done a very good job in ensuring that new technologies are adequately precise. But Rabbit would have opined that analytical reproducibility is merely the beginning, a starting point. Of ultimate importance is the ability of a new technology to impact favorably on infectious disease outcomes. Presumably, that is why clinical laboratories exist in the first place. Sadly, this is an area in which needed information is almost nonexistent.

The concepts of evidence-based clinical practice are relatively new, having first been articulated in the early 1990s by Montori and Guyatt (1). Today, 20 years later, the notion that clinical practice in medicine is best predicated on data derived from objective investigations is incontrovertible and has percolated into every corner of the health care enterprise (2). Were a spaceship from Mars to land in a cornfield in Iowa tomorrow and its captain to ask, “What today are the most defining paradigms in the practice of medicine in the developed world?”, we might be inclined to reply, “The technology of medicine” or perhaps “The business of medicine” (especially in the United States). We might be tempted to cite our overarching drive to keep people alive, often well beyond their intended years. But certainly somewhere on our list, perhaps at the very top, we would include evidence-based clinical practice.

In its simplest form, evidence-based clinical practice means that it is no longer acceptable to function out of habit, because it sounds reasonable or because it has always been done that way. We require data, data derived from robust, objective studies, to either support, refute, or refine clinical practice. This is as true of laboratory medicine as it is of any other discipline in health care.

In clinical microbiology, the most meaningful form of evidence in support of laboratory practice is derived from clinical outcomes studies. Simply put, clinical microbiologists should base laboratory practice algorithms on data that assess the true impact of individual practices on the outcome of patients with infection. Stated another way, on the other end of every clinical microbiology laboratory procedure, no matter how big or how small, is an ill patient with infection. Method precision and accuracy, cost, and test complexity vis-à-vis staff training level and skill set are all important determinants in establishing test algorithms in clinical microbiology. But of greatest importance is the direct proven benefit, or lack thereof, of a given procedure on the outcome of patients with infection. And when more than one procedure or algorithm seems appropriate, it is essential to understand their comparative levels of effectiveness as assessed ideally in carefully controlled, prospective, randomized clinical investigations which objectively measure outcomes. This is true both with respect to individual patients and with patient populations in general.

Unfortunately, however, objective, systematic outcomes studies in clinical microbiology are almost nonexistent. A reasonably comprehensive review of the peer-reviewed literature reveals only a few such studies in clinical microbiology (3–17).

We have been content to accept analytical precision as a justification for adopting new technologies.

With this editorial, I would make a plea that going forward, clinical microbiologists begin to embrace the importance of data which establish the true clinical impact of what we do in the laboratory as the ultimate measure of the utility of new technologies. This will require the performance of objective, systematic, controlled clinical outcomes studies, the results of which must then be published in the peer-reviewed literature.

Such studies should be conducted by clinical microbiologists in care settings that reflect the circumstances in which a new procedure will be used. A multicenter study format is always preferred. To wit, a two-center study is always more than twice as good as a one-center study. Both a control group and a study group are essential, i.e., outcomes studies need to be comparative. In a perfect world, the control and study groups are assessed prospectively and simultaneously. In such cases, objective randomization of subjects into control and study groups is essential. However, given the difficulty in crafting prospective outcomes studies with simultaneous control and study groups, especially with technologies that are likely to have a positive impact on patient outcomes, use of an historical control group may suffice.

Carefully thought-out patient inclusion criteria are extremely important. In this regard, it is essential that an attempt be made to eliminate as many confounding variables as possible in selecting patients for inclusion in the study. Of equal importance are the outcomes measures that will be tracked in the study. These must be objectively definable, relevant to the procedure being assessed, and, whenever possible, not subject to extraneous variables. An extensive list of patient outcome parameters that are often relevant to the assessment of new clinical microbiology procedures can be found in references 3, 5, 6, 11, and 12.

Numbers are a huge consideration. Outcomes studies must be large enough to stand up under rigorous statistical analyses which compare the study group to the control group. The manner in which data are collected and stored is also important. Especially as pertains to clinical assessments that require subjective judgments, when more than one individual is engaged in making assessments, it is essential that this process be standardized to the extent possible. After data have been collected, the entry of information onto a database system with adequate functionality is most important. A robust database provides a secure repository for the information gathered in the study and further expedites data analysis.

Given the nature of outcomes studies, authorization by institutional research review panels is always a requisite. Clinical microbiologists often seem to be intimidated by the process of seeking institutional review board (IRB) approval of studies that have a clinical component. This may be understandable, given the current climate that exists regarding patient-related research initiatives. In our thirst to ensure patient rights, oversight groups such as IRBs seemingly often lose sight of the central importance and value of clinical research. This problem is amplified by the composition of IRBs. Today, IRBs are invariably littered with individuals who know little or nothing about the realities of clinical research, e.g., the lay public, attorneys, the clergy, etc. As a consequence, the process of seeking and gaining IRB approval for a given study is often time-consuming and onerous. This process can also be costly, as IRBs typically charge investigators for review of protocols. It remains, however, that IRB approval is required for outcomes studies in clinical microbiology. As a result, those conducting such studies must put aside their natural and understandable disinclination to work with IRBs and become a participant in the process.

Another salient consideration in conducting outcomes studies is the cost of such studies. Simply put, outcomes studies are invariably expensive studies. So who should pay for them? This is an easy one. Whatever manufacturer has provided the technology that is to be assessed in the outcomes study should provide all of the necessary financial support for the study. After all, they will be the ones who ultimately derive financial benefit from adoption of their technology. My own strongly held view is that a company that is reluctant to provide all of the necessary monetary support required to perform a well-designed outcomes study that will generate new and useful information about a product that they sell is not a company worth working with. Use their competitor's product.

And finally, after an instructive outcomes study has been conducted, it is essential that the information derived from the study be used as the basis for a report to be published in the peer-reviewed literature. In this way, everyone can derive benefit from the investigation.

However pie-in-the-sky, consider the good that would come from 100 well-designed multicenter outcomes studies conducted during the next year, each assessing the true clinical value or lack thereof of a different technology, test algorithm, or procedure in clinical microbiology and, in turn, published in a peer-reviewed clinical microbiology journal. If we do that, perhaps Rabbit will say instead, “Oh, now I see the sense in that. Well done, Pooh!”