Evaluation of MetaSystems Automated Fluorescent Microscopy System for the Machine-Assisted Detection of Acid-Fast Bacilli in Clinical Samples

ABSTRACT

Manual reading of fluorescent acid-fast bacilli (AFB) microscopy slides is time-intensive and technically demanding. The aim of this study was to evaluate the accuracy of MetaSystems’ automated fluorescent AFB slide scanner and analyzer. Auramine O-stained slides corresponding to 133 culture-positive and 363 culture-negative respiratory (n = 284), tissue (n = 120), body fluid (n = 81), and other (n = 11) sources were evaluated with the MetaSystems Mycobacteria Scanner running the NEON Metafer AFB Module. The sensitivity and specificity of the MetaSystems platform was measured as a standalone diagnostic and as an assistant to technologists to review positive images. Culture results were used as the reference method. The MetaSystems platform failed to scan 57 (11.5%) slides. The MetaSystems platform used as a standalone had a sensitivity of 97.0% (129/133; 95% CI 92.5 to 99.2) and specificity of 12.7% (46/363; 95% CI 9.4 to 16.5). When positive scans were used to assist technologists, the MetaSystems platform had a sensitivity of 70.7% (94/133; 95% CI 62.2 to 78.3) and specificity of 89.0% (323/363; 95% CI 85.3 to 92.0). The manual microscopy method had a sensitivity of 79.7% (106/133; 95% CI 71.9 to 86.2) and specificity of 98.6% (358/363; 95% CI 96.8 to 99.6). The sensitivity of the MetaSystems platform was not impacted by smear grade or mycobacterial species. The majority (70.3%) of false positive smears had ≥2+ smear results with the MetaSystems platform. Further performance improvements are needed before the MetaSystems’ automated fluorescent AFB slide reader can be used to assist microscopist in the clinical laboratory.

INTRODUCTION

Clinical laboratories worldwide are embracing automation in order to improve efficiency and quality, while also addressing technologist shortages (1, 2). Digital pathology is an emerging field that takes digitized microscopy slides and evaluates them by an automated computer-based algorithm (3). Manual fluorescent acid-fast bacilli (AFB) microscopy is currently used to detect mycobacteria in different types of clinical samples. The manual reading of AFB smears is labor-intensive and requires a trained and licensed clinical laboratory scientist (CLS). Automation of fluorescent AFB microscopy has been reported by several groups and shown to have moderate to high sensitivity but low to high specificity (4–8). Thus far, digital microscopy has been shown to have high sensitivity or high specificity but not both (4–7). The specificity can be improved if positive images are confirmed by a microscopist (5) or if corresponding samples are tested with a nucleic acid amplification test (4, 7). Recently, MetaSystems (Medford) has commercialized an automated fluorescent AFB microscopy scanner and reader to assist the technologists in the clinical laboratory. The aim of this study was to evaluate the accuracy of the MetaSystems’ automated fluorescent slide scanner and analyzer for AFB smears.

MATERIALS AND METHODS

Ethics.

This study was conducted as a quality improvement project at Stanford Health Care clinical microbiology laboratory.

Manual microscopy.

A total of 553 AFB smears were scanned by the MetaSystems platform. Slides that passed scanning quality check were included in the analysis and consisted of 496 AFB smears from 284 respiratory samples, 120 tissue, 81 body fluid, and 11 other sources were included. Samples were selected non-consecutively to include more positive smears. Samples from non-sterile sources, such as sputa and bronchoalveolar lavage (BAL) were decontaminated and concentrated per standard laboratory procedure (9). Sterile tissues were homogenized, and sterile fluids were concentrated. Slides were stained with auramine O and counterstained with potassium permanganate (AlphaTec) according to the manufacturer’s instructions. Prepared slides were read manually by a trained CLS using a 20x and 40x objective lens and 10x ocular lens and quantified as ± or 1+ to 4+ per the CDC guidelines, and reported in the electronic medical records for patient care (10). Slides with ± results were confirmed by a second CLS before reporting. Samples with positive smear but negative AFB culture were incubated for an additional 2 weeks for a total of 8 weeks to see if mycobacteria can be recovered in culture. Additionally, Mycobacterium tuberculosis PCR and or bacterial 16S rRNA sequencing may have been ordered by the provider to resolve smear-positive culture-negative discrepancies. Slides were stored at room temperature in a light-proof box for automated analysis at a later time.

Study design.

The sensitivity and specificity of the MetaSystems Mycobacteria Scanner running the NEON Metafer AFB Module was measured using culture and manual AFB microscopy as the reference methods. The MetaSystems platform was evaluated as a standalone system with no CLS intervention, and also as an assistant to technologist where positive images were reviewed by 3 CLSs and their consensus or majority result was accepted.

MetaSystems platform.

The system is comprised of a fluorescence microscope with an automated stage and camera, which scans a designated area of a slide and captures images of each field with a 20x objective lens. Each camera field image is broken up into 560 smaller image tiles, which are used as input to be analyzed by the deep neural network (DNN) classifier in the MetaSystems NEON Metafer AFB Module. The MetaSystems image gallery is generated based on the DNN probability threshold which was set to 50% probability for this study. Slides with DNN probability threshold ≥50% were considered AFB positive and made available in the user interface for technologist review. Positive slides were further quantitated on a scale of ± and 1+ to 4+. Slides with DNN probability threshold <50% were classified as negative and reported to the user with no image tiles for review. When the system is used as an assistant to the CLS, positive slide images are reviewed, and the result is determined based on CLS interpretation of the presented images.

Statistical analysis.

Sensitivity and specificity were calculated using culture and manual AFB microscopy as the reference standard. Confidence interval (CI) was calculated using Prism 9 software (GraphPad Software). Chi-Square test was used to analyze differences between proportions.

RESULTS

With AFB culture used as the reference standard, manual AFB microscopy was 79.7% (106/133; 95% CI 71.9 to 86.2) sensitive and 98.6% (358/363; 95% CI 96.8 to 99.6) specific (Table 1). Out of the 553 slides that were scanned by the MetaSystems platform, 57 (11.5%) failed scanning due to the system's inability to find a focal point. Since there were no results available for these 57 slides, they were excluded from all subsequent analysis. The MetaSystems platform used as a standalone diagnostic had a sensitivity of 97.0% (129/133; 95% CI 92.5 to 99.2) and specificity of 12.7% (46/363; 95% CI 9.4 to 16.5) (Table 1). When MetaSystems platform was used to assist CLSs, it was 70.7% (94/133; 95% CI 62.2 to 78.3) sensitive and 89.0% (323/363; 95% CI 85.3 to 92.0) specific (Table 1). With respect to mycobacterial species recovered in culture (Table 2), the sensitivity of the MetaSystems platform unassisted was 95.5% (64/67; 95% CI 87.5 to 99.1) for Mycobacterium avium complex compared with 100% (34/34; 95% CI 89.7 to 100) for M. tuberculosis complex (P > 0.05), and 96.8% (30/31; 95% CI 83.3 to 99.9) for other non-tuberculous mycobacterial species (P > 0.05). The sensitivity and specificity of the unassisted MetaSystems platform by the specimen type was 97.3% (107/110; 95% CI 92.2 to 99.4) and 17.2% (30/174; 95% CI 12.0 to 23.7) for respiratory samples, 100% (7/7; 95% CI 59.0 to 100) and 14.9% (11/74 95% CI 7.7 to 25.0) for body fluids, and 100% (12/12; 95% CI 73.5 to 100) and 2.8% (3/108; 95% CI 0.6 to 7.9) for tissue, respectively. The specificity for tissue samples was significantly lower compared to respiratory (P < 0.001) and body fluids (P < 0.01) specimens. The smear grade for true positive and false positive results are shown for manual and automated methods in Table 3. The majority (70.3%) of false positive smears with the MetaSystems platform had ≥2+ smear results. For true positive smear results, 51.9% of manual smears had ≥2+ smear results compared with 89.8% with the automated method (P < 0.001).

TABLE 1.

Performance of MetaSystems automated AFB microscopy

Microscopy method evaluated	Reference standard
	Culture		Manual microscopy
	Sensitivity (na/Nb; 95% CI)	Specificity (n/N; 95% CI)	Sensitivity (n/N; 95% CI)	Specificity (n/N; 95% CI)
Manual	79.7% (106/133; 71.9–86.2)	98.6% (358/363; 96.8–99.6)	NAc	NA
MetaSystems automated	97.0% (129/133; 92.5–99.2)	12.7% (46/363; 9.4–16.5)	97.3% (108/111; 92.3–99.4)	12.0% (46/385; 8.9–15.6)
MetaSystems automated assisting CLS	70.7% (94/133; 62.2–78.3)	89.0% (323/363; 85.3–92.0)	85.6% (95/111; 77.7–91.5	75.3% (290/385; 70.7–79.6)

^an, number of true results.

^bN, number of true results plus false results; 95% CI, 95% confidence interval.

^cNA, not applicable.

TABLE 2.

Mycobacterial species identified in corresponding cultures in this study

Species	No. (total = 133)
M. avium complex	67
M. tuberculosis	32
M. abscessus group	17
M. xenopi	5
M. chelonae	3
M. bovis	2
M. fortuitum group	1
M. gordonae	1
M. kansasii	1
M. lentiflavum	1
M. marinum	1
M. mucogenicum group	1
Tsukamurella spp.a	1

^aAlthough not a Mycobacterium, the corresponding AFB smear slide was positive with both manual and automated methods.

TABLE 3.

Smear grade reported by manual readings and by MetaSystems automated AFB microscopy in true and false positives using culture as the reference method

	Manual		MetaSystems automated
Grade	True positives % (Na)	False positives % (N)	True positives % (N)	False positives % (N)
+/−b	9.4% (10)	20% (1)	10.2% (13)	24.6% (78)
1+	38.7% (41)	60% (3)	0	5.0% (16)
2+	18.9% (20)	20% (1)	35.9% (46)	33.4% (106)
3+	22.6% (24)	0	39.1% (50)	32.8% (104)
4+	10.4% (11)	0	14.8% (19)	4.1% (13)

^aN, number.

^bFor the automated method +/- refers to (+/−).

With manual microscopy used as the reference standard, the MetaSystems platform as a standalone diagnostic had a sensitivity of 97.3% (108/111; 95% CI 92.3 to 99.4) and specificity of 12.0% (46/385; 95% CI 8.9 to 15.6) (Table 1). When used to assist CLSs, the MetaSystems sensitivity was 85.6% (95/111; 95% CI 77.7 to 91.5) and specificity was 75.3% (290/385; 95% CI 70.7 to 79.6) (Table 1). The sensitivity of the MetaSystems platform as standalone diagnostic was 100% (10/10) for ± smears, 95.0% (38/40) for 1+ smears, 95.0% (19/20) for 2+ smears, 100% (24/24) for 3+ smears, and 100% (12/12) for 4+ smears.

Representative images of true positive and false positive results with the MetaSystems platform are shown in Fig. 1. Discrepancies between the MetaSystems and manual microscopy were due to images captured out-of-focus and images with low resolution which compromised the visible morphology of AFB and made it difficult to distinguish “out-of-focus” AFB from artifacts.

FIG 1.

Examples of MetaSystems microscopy images classified correctly and incorrectly as AFB. (A) Images from a sample that was positive for Mycobacterium tuberculosis and called “positive” by MetaSystems and confirmed as positive by CLS based on visible characteristic “banding pattern”. (B) Images of true positive AFB from three different samples called “positive” by MetaSystems but interpreted as negative by the CLSs due to low image resolution or out-of-focus images. (C) Images of artifacts from three different samples called “positive” by MetaSystems and interpreted as positive by the CLSs due to low image resolution or out-of-focus images. (D) Images of artifacts from three different samples called “positive” by MetaSystems but interpreted as negative by the CLSs. The resolution of images shown here is what was available to CLSs during image review.

Per the recommendation of MetaSystems, to determine if a different counterstain could improve focusing and scanning, an additional 8 positive and 8 negative slides were stained with auramine O alone, and auramine O counterstained with thiazine red (AlphaTec). This counterstain gives a slight stain to the background that does not affect the signal from the mycobacteria but is visible for focusing purposes. The sensitivity and specificity of the MetaSystems platform used to assist CLSs in the counterstained subgroup were 62.5% (5/8) and 37.5% (3/8), respectively, which was identical to the subgroup stained with auramine alone.

DISCUSSION

Evaluation of MetaSystems AFB microscopy with generic classification of AFB (DNN probability threshold set to 50%) showed that as a standalone diagnostic the automated platform was highly sensitive but very nonspecific. When used to assist CLSs to identify potentially positive slides, the specificity drastically increased from 12.7% to 89.0%, however, sensitivity decreased from 97.0% to 70.7%, which was 9% lower than sensitivity obtained with manual microscopy. Specificity of assisted microscopy was nearly 10% lower compared to manual microscopy. The main driver of errors with assisted microscopy was image resolution provided by the MetaSystems platform to the CLS for review. Because slides were scanned and captured using a 20x objective lens, the digital images did not reach the ideal resolution for discerning morphology upon review. Furthermore, out-of-focus images frequently occurred because images were captured on a single focal plane, but not every field on the slide is on the same focal plane. The findings are unlikely to be related to the brand of the stain as per the manufacturer the stain formulas are based on the standard WHO guidelines (Auramine O: 0.1% with 0.3% phenol; Decolorizer: 0.5% Acid-Ethanol; Potassium Permanganate counterstain: 0.5%). The staining times were also comparable between AlphaTech and WHO guidelines (auramine O 15 versus 20 min; decolorizer 2 to 3 versus 1 to 2 min; potassium permanganate 2 to 4 versus 1 min). Using thiazine red counterstain did not improve assay performance, however, only 16 smears were evaluated which may not have been sufficient to adequately assess the impact of thiazine red counterstain. Going forward, availability of digital images using a higher objective lens (i.e., 40x) may provide higher resolution digital images and thus allow CLSs to more accurately confirm the morphology of the suspected bacilli and discern between artifact and true AFB. Additionally, rigorous training of technologists performing assisted digital microscopy, customized classification of AFB (customized DNN probability threshold) based on local smear preparation, AFB stain, and filter settings need to be investigated to determine if they can help improve assay accuracy. Lastly, assessment of accuracy in treatment-naive versus antibiotic-treated patients may also be informative.

The performance of the MetaSystems platform is consistent with the current status of digital pathology where it is proposed to assist operators rather than a replacement for them (2, 4–6, 11). Our findings are consistent with a prior study using a pre-market version of the MetaSystems platform using culture as the reference method and showing 96.4% (54/56) sensitivity and 60.0% (285/475) specificity when used as standalone diagnostic and 71.4% (40/56) sensitivity (compared to 60.7% with manual microscopy) and 93.3% (443/475) specificity when used to assist technologists (12). Thus, as in our study, there was a need to confirm positive slides to improve specificity, which came at the cost to sensitivity. Both studies showed that the MetaSystems platform reports an inflated smear grade in true positives which is consistent with its nonspecificity. However, Hovath and colleagues reported that the 190 false positive proposals by the MetaSystems consisted of the lowest possible reportable grades (171 of “[+]” and second lowest (19 “+”) (12) which is consistent with other studies showing improvements in specificity with slight change in sensitivity when automated microscopy results reported as scanty were reviewed by a microscopist for confirmation (5, 6). In contrast, in our study the majority (70.3%) of false positive results had reportable grades of 2+ or higher. Studies evaluating another unassisted commercial digital microscopy software (TBDx TBDx by Applied Visual Sciences) have shown high specificity but with low sensitivity (4, 7). The trade-off between sensitivity and specificity is a function of assay cutoff based on the number of AFB detected. Thus, it remains to be seen if high sensitivity and specificity can be achieved in the same digital microscopy assay after assay improvements.

In summary, the MetaSystems automated AFB microscopy platform can assist CLS with sensitivity and specificity that is inferior to manual AFB microscopy. Although Horvath and colleagues concluded that AI-assisted microscopy improves efficiency by reducing the average time needed to review a slide to 10 s (12), the sensitivity and specificity achieved in our study suggest there is a need for further improvement of MetaSystems performance before it can be used to assist technologists in a clinical setting.