Evaluation of the WASPLab Segregation Software To Automatically Analyze Urine Cultures Using Routine Blood and MacConkey Agars

Automation of the clinical microbiology laboratory has become more prominent as laboratories face higher specimen volumes and understaffing and are becoming more consolidated. One recent advancement is the use of digital image analysis to rapidly distinguish between chromogenic growth for screening bacterial cultures. In this study, colony segregation software developed by Copan (Brescia, Italy) was evaluated to distinguish between significant growth and no growth of urine cultures plated onto standard blood and MacConkey agars.

ABSTRACT

Automation of the clinical microbiology laboratory has become more prominent as laboratories face higher specimen volumes and understaffing and are becoming more consolidated. One recent advancement is the use of digital image analysis to rapidly distinguish between chromogenic growth for screening bacterial cultures. In this study, colony segregation software developed by Copan (Brescia, Italy) was evaluated to distinguish between significant growth and no growth of urine cultures plated onto standard blood and MacConkey agars. Specimens from 3 sites were processed on a WASP instrument (Copan) and incubated on the WASPLab platform (Copan), and plates were imaged at 0 and 24 hours postinoculation. Images were read by technologists following validated laboratory protocols (VLPs), and results were recorded in the laboratory information systems (LIS). Image analysis performed colony counts on the 24-hour images, and results were compared with the VLP. A total of 12,931 urine cultures were tested and analyzed with an overall sensitivity and specificity of 99.8% and 72.0%, respectively. After secondary review, 91.1% of manual-positive/automation-negative specimens were due to expert rules that reported the plate as contaminated or growing only normal flora and not due to threshold counts. Nine specimens were found to be manual-positive/automation-negative; a secondary review demonstrated that the results of 8 of these specimens were due to growth of microcolonies that were programmed to be ignored by the software and 1 were due to a colony count near the limit of significance. Overall, the image analysis software proved to be highly sensitive and can be utilized by laboratories to batch-review negative cultures to improve laboratory workflow.

INTRODUCTION

Traditionally, the clinical microbiology laboratory has been staffed by highly trained technologists who perform complex tasks, including bacterial culture interpretation, isolate identification, antimicrobial susceptibility testing, and molecular assays from various specimen sources. Many health care centers are consolidating microbiology services, which increases specimen load and demand on the laboratory labor. However, due to an aging workforce and reduction in clinical laboratory training programs, laboratories are seeing a shortage of workers to properly staff the microbiology laboratory (1). One solution to these demands on the laboratory is the adoption of automation, which has been used very successfully in clinical chemistry and core laboratories. More recently, advances in automation have demonstrated improved workflow and increased productivity for microbiology laboratories (2).

Automation in the clinical microbiology has been improving and becoming more widely implemented since 2006 (3, 4). Early platforms automated urine culture processing, as these specimens typically arrive in a standardized set of containers, are easy to manipulate due to their liquid state, have low diversity of viscosity, and are among the most common specimens tested in the laboratory (5, 6). Studies evaluating automated processing instruments found improved isolation and recovery of bacterial colonies, which allowed identification (ID) and antimicrobial susceptibility testing (AST) to be performed more often from primary plates than manual processing (7, 8). Further improvements to automation occurred by adding incubators and imaging stations to reduce the hands-on time of technologists. Overall, evaluation of these fully automated systems has been found to improve workflow and reduce turnaround time for reporting results (5, 9).

Software has also been developed to allow for automation of image analysis. Initial versions detected color changes using chromogenic agar to distinguish negative and nonnegative plates. Evaluations of this software showed high sensitivity and did not miss a single positive in over 160,000 specimens tested (10, 11). Recent developments have expanded on this software, which allows for interpretation of nonchromogenic agar and the ability to count and distinguish colony morphologies.

This study evaluated the software for its use in segregating negative for potential pathogens (defined as <10 CFU or <50 at 1 site) and nonnegative (defined as >10 or >50 CFU based on site protocols) urine cultures based on the standard protocols used in 3 laboratories across the United States (12). Urine cultures collected at 3 laboratories were processed on the WASP platform and incubated on the WASPLab platform, and plates were imaged at time of inoculation (T0) and after 24 hours of incubation (T24). The segregation software used these images to count and record the number of colonies on blood agar plates (BAPs) (Remel Inc, San Diego, CA; BD, Franklin, NJ) or MacConkey agar (MAC) (Remel Inc.; BD) as negative or nonnegative. In this study, the software was limited to count colonies and did not differentiate morphology, so a total colony count was used for automation interpretation. Results were then compared with those manually read by a technologist who was blind to the software results working with the same image from a workstation.

MATERIALS AND METHODS

Specimen processing.

Testing was conducted at 3 laboratories, including Wisconsin Diagnostic Laboratories (Milwaukee, WI), Indiana University Health Pathology Laboratory (Indianapolis, IN), and Alverno Clinical Laboratories (Hammond, IN). Study protocols were identical, with the exception that individual laboratory policies were used to define a positive culture. Testing was performed using 1-μl aliquots of clean catch urines that were collected with or without boric acid preservation. Urine cultures were processed on the WASP platform, incubated in the WASPLab platform, and imaged T0 and again at T24. Each specimen was plated on BAP and MAC.

Manual reading of urine cultures.

Manual analysis of cultures was performed by laboratory staff following each laboratory’s validated procedure for reading and reporting using a 1-μl loop. Two sites defined 10 or more CFUs as significant, whereas the other site used a threshold of 50 or more CFUs. In addition, manual reporting of negative urine cultures was also defined as those that contained multiple pathogens (3 or more) designated “no more workup” (NMW), or those that contained only normal skin flora designated “no significant growth” (NSG) regardless of quantity. One of the laboratories did not differentiate between NMW and NSG but instead resulted these cultures together as “mixed urogenital flora” (MUF). All reading was performed on high-definition (HD) monitors using digital images, and the laboratory results were recorded in the LIS. All technologists were blind to the software’s results.

Automated digital image analysis of BAP and MAC.

The segregation software works by measuring pixel changes detected between T0 and T24 images. The algorithm detects these changes and distinguishes individual colonies based on gaps of no change between the time points. After reading the entire plate, the software then reports the total CFUs for the culture. The software can also distinguish overlapping colonies based on the expected shapes of a colony to perform counting similar to that of a technologist. However, the software did not distinguish between colony morphologies, meaning automated reporting was based on total numbers of organisms present on the plate. These organisms include Gram-positive, Gram-negative, and yeast colonies. The software evaluated each culture and reported automation-negative (AN) if no colonies were detected or if there were less than the laboratory’s threshold of 10 or 50 CFUs on both plates; alternatively, it reported automation-positive (AP) if there were greater than the laboratory’s threshold detected on either BAP or MAC. The software was programmed to disregard detection of microcolonies, as the 3 laboratories included in the study generally instructed technologists to only perform minimal identification for these organisms.

Discrepant analysis.

Data capturing was done in real-time, but data analysis was performed after testing was completed. By the time comparisons were performed, the original culture plates had been discarded by the laboratory; thus, discordant analysis was performed by review of the images. Images from discordant results were reviewed by a second technologist, and the reason for the discrepant results was recorded.

Statistical analysis.

Analysis of the data was performed by comparing the results of the manual read to the software read, with manual reading considered to be the gold standard. Performance characteristics were calculated using standard methods for sensitivity and specificity. Confidence intervals of 95% were determined using a binomial expansion.

RESULTS

Specimen characteristics.

Between the 3 sites, a total of 12,931 urine specimens were enrolled and tested. Images were analyzed from blood and MacConkey agar, so interpretation of significant growth was performed on 25,862 plates. The percentage of urine cultures containing a significant pathogen was 43.9%. These percentages ranged from 29.3% to 56.9% between the 3 sites. Using an increased cutoff at 50 CFUs rather than 10 CFUs may have reduced percent positivity at site 2, as they observed a prevalence of only 29.3% compared with the other sites at 50.0% and 56.9%.

Comparison of the segregation software to manual reading.

Overall, the software was highly sensitive, with an average sensitivity of 99.8% that ranged between 99.7% and 99.9% among the 3 sites (Table 1). These data included 5,678 specimens that were positive by both manual reading (manual positive [MP]) and automated reading (automation positive [AP]) and 9 specimens that were MP, automation-negative (AN). Specificity showed an overall rate of 72.0%, which included 5,598 manual-negative (MN)/AN specimens and 2,180 MN/AP specimens (Table 1). There were significant differences based on 95% confidence intervals between the three sites. The lowest site had a specificity of 48.7% followed by 75.0% and 84.0% at the other sites. The site with the lowest specificity used a mixture of preserved and unpreserved urine samples, which may have allowed growth of low burden organisms that could have then been ruled out as contamination.

TABLE 1.

Performance of WASPLab digital imaging compared with manual reading

Clinical test site	No. of specimens tested	Results (no.)a				Performance (% [95% CI])b
		MP/AP	MN/AN	MN/AP	MP/AN	Sensitivity	Specificity
1	5,201	2,958	1,092	1,149	2	99.9 (99–100)	48.7 (47–51)
2	5,513	1,613	3,274	621	5	99.7 (99–100)	84.0 (83–85)
3	2,217	1,107	1,232	410	2	99.8 (99–100)	75.0 (73–77)
Total	12,931	5,678	5,598	2,180	9	99.8 (99–100)	72.0 (71–73)
Total postdiscrepancy	12,931	7,665c	5,606	193	1	99.9 (99–100)	96.7 (96–97)

^aMP/AP, manual positive/automation positive; MN/AN, manual negative/automation negative; MN/AP, manual negative/automation positive; MP/AN, manual positive/automation negative.

^bCI, confidence interval.

^cRule based reporting as negative that contained 10 or more colonies were counted as MP in discrepant analysis.

Discrepant analysis.

The segregation software defined negative and nonnegative plates based on the presence of 10 (2 sites) or 50 CFUs (1 site) indiscriminate of colony morphology, whereas manual results were based on laboratory rules to limit overworking contaminated or poorly collected specimens. Images of MN/AP specimens were further reviewed along with a detailed culture report. Two laboratories differentiated specimens as NMW indicating likely contamination by fecal flora, and NSG which indicated poor collection and contamination with skin and/or genital flora. The third site did not distinguish between the two and reported such results as MUF. Separation of MN/AP specimens based on these reporting structures is found in Table 2.

TABLE 2.

Discrepant analysis of MN/AP specimens

Site	No. of specimens according to MN/AP resulta				Total no. of specimens
	MNb	NMWc	NSGc	MUFc
1	56	363	730	N/A	1,149
2	120	100	401	N/A	621
3	17	N/A	N/A	393	410
Total	193	463	1131	393	2,180

^aMN, manual negative; NMW, no more workup; NSG, no significant growth; MFU, mixed urogenital flora. N/A, not applicable.

^bManual negative plates were reported as negative for pathogens and had <10 CFUs.

^cPlates had 10 or more CFU.

Reevaluation of discrepant specimens found that 91.1% (1,987/2,180) of MN/AP specimens were caused by plates defined as negative due to rules and not based on colony count, meaning that although the laboratory called the culture negative, there were colonies present on the plate(s) (Table 2). Two examples are shown in Fig. 1. In the first example, a mixture of various coagulase-negative Staphylococcus (CoNS) are growing at >10 CFUs (Fig. 1A), and in the second example, CoNS is present with a Streptococcus mitis group growing at >10 CFUs (Fig. 1B). The remaining 8.9% (193/2,180) were caused by differences in colony counts between the software and technologist. The software has the capability to count colonies based on colony morphology, which was not used in this study. It could be possible to use software rules to match technologist interpretation for urine culture rules; a total of 1,987 specimens would have been called negative by the software, and the specificity would increase to 97.5%.

FIG 1.

(A and B) two examples of MN/AP specimens. Although the cultures were >10 CFUs, they were called negative due to growth of normal genital flora. (C) Example of an image that was called automation-negative and manual-positive based on the presence of 39 colonies and ∼30 microcolonies. (D) Specimen that was called MP/AN; the technologist counted 55 colonies (50 threshold) and the software count was <50.

The 9 MP/AN discrepant results were found to fall into two categories (Table 3). The most common cause for discrepancy (8 of 9 cultures) was the presence of microcolonies that were counted as positive by the technologist. These microcolonies were also found with normal-sized colonies, which, when counted together, were above the threshold set by the laboratory for calling positive. An example of this can be found in Fig. 1C. Review of each of these reports indicated minimal ID on the microcolonies. Of the 8 discordant specimens, 4 were reported as containing only normal flora, and the rest contained Gram-positive organisms, with 1 of these identified as Lactobacillus sp. and 1 as a Corynebacterium sp. All microcolonies were identified only on BAP, suggesting that they contained Gram-positive organisms. Turning on the microcolony counts on the software resulted in all 8 cultures correctly being called positive for growth. The remaining MP/AN specimen was due to a difference in bacterial count near the threshold of reporting (Fig. 1D). In this specimen, the laboratory report had a bacterial count of 55 (site 2) and automation was just under the threshold of 50 CFUs (49 colonies counted by the algorithm). A comparison of the discrepant results did not detect any other trends outside microcolonies, and no specific bacterial isolates were observed to have issues being counted by the software. These data suggest that the software is robust at counting various colony morphologies detected in urine cultures.

TABLE 3.

Discrepant Analysis of MP/AN specimens

Cause of MP/AN	No. of specimens	Description
Microcolonies	8a	Software ignored microcolonies at <50 CFUs that the technologist counted as significant
Difference in counts near the significance limit	1a	Technologist counted 55 CFUs, but software called <50 CFUs

^aWhen analyzed with microcolonies counted, these specimens are reported as AP.

DISCUSSION

Past publications about digital image analysis has focused on using chromogenic agar for specimen screening. Although those evaluations suggest that automation can reduce the need for technologist time and potentially lower overall laboratory expenses, many laboratories do not use chromogenic medium primarily due to its cost. To our knowledge, this is the first study that evaluates software that is capable of reading routine, nonselective media. Urine was evaluated, as it is one of the largest specimens collected and received in the clinical microbiology laboratory, but similar approaches could be used for any specimen source that is normally sterile.

The implementation of this segregation software is similar to that of the chromogenic detection module (CDM) where laboratories would use the analysis to quickly filter out negative from nonnegative specimens. The segregation software performed similarly to that of the CDM, which was found to have very high sensitivity (100% in methicillin-resistant Staphylococcus aureus [MRSA] and vancomycin-resistant enterococci [VRE]) but had reduced specificity that ranged from 88.8% to 96.0% (10, 11). The segregation software in this study was also observed to have a very high overall sensitivity of 99.8%. Concerning the 9 MP/AN specimens, 8 of these discrepancies were due to the presence of microcolonies that the technologist counted toward “reportable.” The software used in this study has the capability to count microcolonies, but this option was turned off for this study, as microcolonies are only minimally worked up with a Gram stain. When the software revaluated the images with the microcolony counts being reported, the 8 discrepancies were correctly reported as AP. However, by counting microcolonies, this reduced the specificity from 72.0% to 66.5%, as more plates had counts above the threshold. The remaining MP specimen that was called negative by the software was due to counts very near the reporting threshold. Manual counting of CFUs can vary between technologists, so this small difference would be expected in any evaluation. Interestingly, turning on the microcolony counts resulted in this plate being called AP and resulted in 100% sensitivity of the software. To reduce MP/AN specimens in the clinical laboratory, the software could be used to identify urine cultures with no growth (as a subset of the negative cultures) and technologists could review these specimens for confirmation and batch reporting/resulting.

The study found that the software specificity ranged from 48% to 84% between the sites. Up to 91% of the MN/AP specimens were a result of the complexity of urine cultures and determining significance of the colonies present. These errors could be corrected if the algorithm could distinguish colony morphology and assign individual counts. For example, if there were 13 colonies identified on the plate and 7 were a Staphylococcus sp. and 5 were a Streptococcus sp., the algorithm could report this as negative even though the threshold of >10 colonies was achieved. Furthermore, similar rules could be created that if ≥3 Gram-negative bacilli morphologies are detected, the software could report the culture as possible contamination.

The high sensitivity of the software means that the negative predictive value is highly reliable to batch or autoreport negative specimens. In this study, 43.3% of all specimens resulted as MN/AN. In our clinical laboratory, up to 350 urine specimens are processed each day, and using this software to read culture images would remove or reduce the workload by 151 cultures a day. Over a year, this would be a reduction of 55,000 urine cultures that would not need individual technologist review. Urine cultures are a large workload for clinical laboratories, so this reduction in analysis could improve overall workflow. If a technologist takes approximately 15 seconds to read and report a negative plate, the addition of this software could save 229 technologist hours. With further software refinements to distinguishing colony morphology and the creation of expert rules, this annual reduction could be increased to 75,000 urine specimens a year.

The use of segregation algorithms could also improve workflow by developing standardized counting for urine cultures. Colony counting can be variable between technologists, which was demonstrated by one of the MP/AN specimens in this study that was initially read as 55 but could be interpreted to be from 45 to 60 CFU depending on interpretation. There was only one of these cultures found in this study, but instances of count difference between technologists were observed in previous algorithm studies (10, 11). Most cultures are not affected by small differences in colony counts, but in cases where the count is near a cutoff threshold, small differences could potentially result in different treatment options for the patient. Segregation using a software algorithm based on colony counts would remove this human variability. In this workflow, technologists would receive results for the work-up of nonnegative specimens and then determine if the growth seen is contamination or a possible pathogen(s).