Machine Learning Takes Laboratory Automation to the Next Level

Clinical microbiology laboratories face challenges with workload and understaffing that other clinical laboratory sections have addressed with automation. In this issue of the Journal of Clinical Microbiology, M. L. Faron, B. W. Buchan, R. F. Relich, J. Clark, and N. A. Ledeboer (J Clin Microbiol 58:e01683-19, 2020, https://doi.org/10.1128/JCM.01683-19) evaluate the performance of automated image analysis software to screen urine cultures for further workup according to their total number of CFU.

ABSTRACT

Clinical microbiology laboratories face challenges with workload and understaffing that other clinical laboratory sections have addressed with automation. In this issue of the Journal of Clinical Microbiology, M. L. Faron, B. W. Buchan, R. F. Relich, J. Clark, and N. A. Ledeboer (J Clin Microbiol 58:e01683-19, 2020, https://doi.org/10.1128/JCM.01683-19) evaluate the performance of automated image analysis software to screen urine cultures for further workup according to their total number of CFU. Urine cultures are the highest volume specimen type for most laboratories, so this software has the potential for tremendous gains in laboratory efficiency and quality due to the consistency of colony quantification.

TEXT

According to the Bureau of Labor Statistics website, the number of laboratory technologists and technicians in the United States declined by several thousand from 2016 to 2018 in the face of more than double the growth rate in positions relative to other occupations. The vacancy rate in this time period was over 7% (1). The reasons for this shortage are complex, but laboratory automation offers a solution for portions of the laboratory workflow that are routine, repetitive, and high volume. This leaves lower volume, high-skill work to the technologist staff, a subset of whom are themselves tasked with tending to the total laboratory automation system.

Against this background of painfully short staffing, concern about the impact of machine learning and “artificial intelligence” on technologists’ continued employment and job satisfaction has crept into watercooler conversation (2). As Faron and colleagues illustrate in this issue, the field is only slowly approaching “total” laboratory automation as artificial intelligence stands to screen out some of the least interesting (yet highest volume) work of the laboratory (3–5).

There are two commercially available Food and Drug Administration (FDA)-approved microbiology laboratory automation platforms in the United States, namely, WASPLab (Copan Diagnostics Inc.) and Kiestra (Becton Dickinson) (6). Each system is highly customizable and consists of front-end processing, “smart” incubation according to laboratory protocol, and plate imaging. The processing unit performs medium selection, application of patient information and barcodes for tracking, medium inoculation, and plate streaking. Automation of these processes cuts down on and improves the consistency of repetitive tasks previously performed by technologists. In addition, automation decreases preanalytical laboratory errors, such as selection of incorrect medium, mislabeled specimens, failure to inoculate or streak plates, and lost plates (7). Manual plate streaking is a highly variable process, and the use of automated plate streaking improves the consistency of spreading specimen over agar medium and produces more isolated colonies (8). In mixed cultures, the presence of isolated colonies allows for testing to be performed from the initial agar plate without the need for subculture and overnight incubation, which delay reporting of patient results.

Smart incubators have a high capacity and contain small openings for agar plates entering and exiting. Plate imaging allows technologists to read and interpret culture plates using enlarged images viewed on a computer screen. Agar plates are only removed from the incubator if they are required for hands-on testing of an isolate for purposes of identification of susceptibility testing. Bacteria and yeast from clinical specimens grow much faster than when using traditional culture methods due to the constant temperature and CO₂ environment in which culture plates are housed (9). Notably, however, FDA approval of laboratory automation systems is only for the processor, incubator, and imaging software. Procedures guiding culture interpretation and reporting vary widely among microbiology laboratories and require in-house validation when implementing laboratory automation.

Digital imaging and machine learning provide an opportunity to improve upon the efficiency and quality of culture interpretation and reporting. Machine learning has already made headway in other areas of medicine, where large amounts of complex data, especially images, must similarly be distilled into a clinical-grade judgment (for a review of these applications and of machine learning in general, see reference 10). For urine cultures with no growth, which may constitute the majority of a laboratory’s workload, there is an opportunity to autoverify these results without human intervention using appropriate image analysis software. For positive growth plates, an artificial intelligence screen for “significant growth” versus “insignificant growth” would allow certain classes of positive plates to be autoresulted as well. Triage of culture specimens would allow technologists to dedicate their increasingly scarce time to high-complexity tasks for which they have been extensively trained, such as Gram stain and culture reading and interpretation, microorganism identification, and susceptibility testing and interpretation.

In this issue, Faron et al. evaluate WASPLab automated digital image analysis software for urine cultures at three microbiology laboratories. Results were compared to institution-specific laboratory protocols. On top of the identification of culture plates with growth or no growth of microorganisms, they added a second layer of analysis via machine learning to quantify microorganisms and categorize cultures as significant growth or insignificant growth based on laboratory-specific thresholds. Put into practice, this additional analysis furthers efficiency by focusing technologist time on working up positive urine cultures and eliminating time spent reading urine cultures with no growth or insignificant growth. Urine culture is a wonderful test model because it is high volume, dominated by negative cultures, and potentially performed cheaply with standard agars that are modestly selective and differential if read by a trained eye. Also, as highlighted in this paper, there is surprisingly little agreement over an absolute definition of negative (which may or may not have colonies of bacteria or yeast) and nonnegative cultures. Any imaging algorithm tasked with making a negative/nonnegative call on a standard urine culture must, therefore, be ready to learn what these entities look like on a per-lab basis and be adaptable to each laboratory’s protocol. This element, facilitated by machine learning, is the major advance presented in this paper.

Copan, collaborating with the University of Brescia (11), previously published an approach to colony counting on WASPLab. The repeatability of the WASPLab photographs allows a comparison of a plate’s photo to an archived time-zero photo; anything new is presumed to be bacterial colonies, which are nominally round to elliptical when isolated and less so when confluent. These colonies are of an expected size, and the lighting produces reflections that emphasize lightness and color. Clumps of up to six colonies are split to count individually, but past that the algorithm in this original publication surrenders. The algorithm has a training set to learn from and is told to use features of what it sees to best match the counts that a human operator has made on the same image. After learning from a human-curated “training set,” the algorithm is fixed, and a “test set” is used to assess the accuracy of its training. Much like a human learning to read plates, adjustments can be made to the approach, additional training can be performed, and the machine can try to do better in the next iteration, but unlike humans who may experience “protocol drift” over time, an image analysis algorithm is locked down and, absent changes that would impact the images, it will always perform the same.

The image analysis program described in this paper has the flexibility to adjust to different protocols, allowing laboratories to set individual thresholds for significant and insignificant growth. Previous studies used chromogenic agars (3, 4), where colony color is a much more discriminatory feature from which to judge colony growth and quantitation. This study shows that image analysis can also be used for 5% sheep blood and MacConkey agars which are available at a much lower cost and are the mainstay of most laboratory urine culture protocols.

Much as procedures vary from lab to lab, reading and interpreting cultures also varies from technologist to technologist. For this study WASPLab digital images and machine learning of urine cultures was compared with plate reading by laboratory technologists using the same digital images. Discrepant analysis showed that for cultures that were negative by manual reading and positive by automation, over 90% of the discrepancies were caused by mixed cultures where the total colony count pushed them over the threshold into the “significance” category, but the laboratory interpretative reporting protocol deemed them “insignificant.” Although intraobserver variability was not tested in this study, less than 10% of discrepant results (approximately 1.5% of total cultures observed) were due to differences in colony counts between technologists and the machine learning software.

This introduces the question of how laboratories can implement this technology in practice. Image analysis software is not currently FDA approved, so the algorithm it deploys qualifies as a high-complexity laboratory-developed test when used to make definitive calls about microorganism presence/absence or culture significance. In this context, the end user need not understand the internal workings any more than they understand the inner workings of most computers. Additionally, as with most laboratory software, manufacturer assistance is provided in training the algorithm. Labs may, therefore, validate performance according to familiar sensitivity and specificity (for significant growth), precision and accuracy (for quantification), and procedural variation (coefficients of variation, Kappa statistics). As with any test, revalidation must be performed if components of the test change. The number of samples needed to train the algorithm (hundreds to thousands) will be algorithm dependent but easily available due to their common nature, facilitating both initial and revalidation using new plate images. Validation of machine learning image analysis for laboratory automation may, overall, be comparable to that performed for whole-slide imaging as used in histopathology, where the object of validation is a process as much as a machine (12) and where modest interobserver agreement may set a similarly modest benchmark for machine learning performance (13).

All of this benchmarking is itself an advance for the field; it is unlikely that labs currently know the magnitude of their intra- or interreader variability in plate reading. A well-designed validation would prove definitively whether machine learning is more accurate than standard practice and, if so, by how much. Previous reports on interobserver variability in plate reading are rare but are generally not encouraging, and as Glasson and colleagues previously reported in the Journal of Clinical Microbiology on their experiences during a similar machine learning effort on the BD Kiestra system (14), humans do not necessarily set an ideal example for machines to learn from. This means that great care must be taken in defining a training set for training and validating local laboratory machine learning algorithms lest they learn from bad practice (10) and in reading literature where poor curation of training sets for machine learning may not be obvious (or worse, where the training and test sets are not carefully segregated). Examples from radiology (15) and whole-slide imaging in histopathology (13) suggest that the highest accuracy is obtained when a multireviewer expert consensus standard is applied to gold standard training set labels, such as colony count and positive or negative interpretations.

In conclusion, the automation monolith is evolving laboratory practice to where we can imagine the microbiology lab of the future. Urine specimens arrive in the lab and are immediately placed into the laboratory automation system, which understands how to set up a culture according to the sample’s barcode. Cultures are read around the clock using machine learning image analysis that is trained according to local laboratory practice. Negative cultures are autoverified in the laboratory information system, and agar plates are discarded from the laboratory automation system without technologist intervention. This system should be familiar to any clinical chemistry laboratory, where nearly 100% of samples might be autoverified (16), or to automated blood culture systems, where blind subculture and manual manipulation have long been abandoned in favor of black-box autoresulting algorithms. In fact, with sufficient standardization, the lab of the future may see a pretrained, FDA-approved image recognition system that is a turnkey solution, similar to modern blood culture systems.

A second quantitative layer of digital imaging would allow positive urine cultures to be sorted by colony count so that when a technologist sits down to read several hundred cultures in their queue, they can focus on those most likely to be causing clinical infections (>100,000 CFU/ml) and work their way down to cultures with lower bacterial burdens. The majority of antimicrobial susceptibility testing from urine cultures can be set up earlier in the day and can be read earlier the next day. Coupled with the laboratory automation’s ability to generate more isolated colonies and do it faster, this approach is an indirect way to make phenotypic susceptibility testing faster by extension. The culmination of these activities is that significant results are pushed out to clinicians earlier in the day, when they are most likely to be acted upon. Now imagine this process expanded to other specimen types, such as respiratory specimens and sterile-source cultures. Image analysis and machine learning could reduce repetitive tasks, use technologist skills to their fullest, and provide clinicians more timely results to improve patient care for a majority of a typical laboratory’s culture volume. This will happen while maintaining a need for the skilled, hands-on, satisfying elements that attract newcomers to clinical microbiology and laboratory medicine. Laboratory automation is now aspiring to its own self-improvement, and to paraphrase HAL 9000 in “2001: A Space Odyssey,” “to put itself to the fullest possible use, which is all that any laboratory automation system can ever hope to do.” Based on the manuscript by Faron et al., the future of clinical microbiology is almost here.

The views expressed in this article do not necessarily reflect the views of the journal or of ASM.