The state of the art for artificial intelligence in lung digital pathology

Abstract

Lung diseases carry a significant burden of morbidity and mortality worldwide. The advent of digital pathology (DP) and an increase in computational power have led to the development of artificial intelligence (AI)‐based tools that can assist pathologists and pulmonologists in improving clinical workflow and patient management. While previous works have explored the advances in computational approaches for breast, prostate, and head and neck cancers, there has been a growing interest in applying these technologies to lung diseases as well. The application of AI tools on radiology images for better characterization of indeterminate lung nodules, fibrotic lung disease, and lung cancer risk stratification has been well documented. In this article, we discuss methodologies used to build AI tools in lung DP, describing the various hand‐crafted and deep learning‐based unsupervised feature approaches. Next, we review AI tools across a wide spectrum of lung diseases including cancer, tuberculosis, idiopathic pulmonary fibrosis, and COVID‐19. We discuss the utility of novel imaging biomarkers for different types of clinical problems including quantification of biomarkers like PD‐L1, lung disease diagnosis, risk stratification, and prediction of response to treatments such as immune checkpoint inhibitors. We also look briefly at some emerging applications of AI tools in lung DP such as multimodal data analysis, 3D pathology, and transplant rejection. Lastly, we discuss the future of DP‐based AI tools, describing the challenges with regulatory approval, developing reimbursement models, planning clinical deployment, and addressing AI biases. © 2022 The Authors. The Journal of Pathology published by John Wiley & Sons Ltd on behalf of The Pathological Society of Great Britain and Ireland.

Introduction

Lung diseases are among the leading causes of mortality worldwide, accounting for nearly 1/6th of death [1]. Over the last 2 years, the COVID‐19 pandemic has had a profound impact across the globe, with nearly 5.6 million deaths to date [2]. Tuberculosis is one of the most common causes of death in developing countries and the immunocompromised population [3]. Meanwhile, lung cancer is the leading cause of cancer‐related mortality worldwide, with nearly 1.8 million deaths in 2020 [4]. Non‐small cell lung cancer (NSCLC) is the predominant subtype, accounting for ~85% of the cases, and includes squamous cell and adenocarcinoma, which are the most common histological subtypes [5]. Small cell lung cancer, an aggressive form with high mortality, meanwhile accounts for ~15% of the cases [6].

Histopathologic tissue analysis forms the backbone of lung disease characterization and management [7, 8, 9], and histopathologic confirmation remains the gold standard in clinical workflow for diagnosis [10]. Tissue samples obtained from lung biopsies and resections are analyzed by pathologists to detect and classify cells, identify tumor morphology and subtype, and evaluate features that predict response to treatment and prognosis [7]. Lung pathology is unique, with several distinct biomarkers. For instance, lung adenocarcinoma (LUAD) is driven by accumulated genetic alterations (‘driver mutations’) such as EGFR, KRAS mutations, and ALK rearrangements [11, 12, 13]. In this era of immunotherapy, studies have focused on tumor‐infiltrating lymphocytes (TILs) and the tumor microenvironment to better characterize the immune response to tumors. Similarly, fibroblast foci have been investigated as a morphologic marker in interstitial lung diseases, with prognostic implications [14, 15]. Studies from COVID‐19 lung autopsies have shown a distinct immune architecture with neutrophilic and lymphocytic predominance [16, 17, 18]. The vast number of biomarkers visually discernible on histopathology images creates an opportunity to apply artificial intelligence (AI) tools to assist clinical workflow.

The advent of whole slide scanning along with improvements in AI‐based tools has paved the way for advances in computational pathology (Table 1). Mukhopadhyay et al [29] compared the diagnostic performance of digital pathology (DP) and traditional microscopy‐based methods in a study with specimens from 1,992 patients, evaluated by 16 surgical pathologists from four institutions. The study found that DP was non‐inferior to conventional microscopy for primary diagnosis in surgical pathology. The results paved the way for the Food and Drug Administration (FDA) [30] approval of whole slide imaging through the de novo premarket review pathway [31] for primary diagnosis in surgical pathology in the United States and its widespread implementation. Meanwhile, the FDA granted a 510(k) clearance to the first DP‐based AI tool, Paige Prostate, for detecting prostate cancer on whole slide images (WSIs) [32]. While many applications of AI in pathology have focused on breast [33, 34, 35, 36], prostate [37, 38, 39, 40], and head and neck cancers [41, 42, 43, 44], there has been a steady interest in applying these tools in the context of lung diseases. The introduction of AI‐based tools, with their power to unlock pathological diagnostic, prognostic, and predictive features, could assist pathologists, pulmonologists, and thoracic oncologists to guide patient management [45] (Figure 1). In diagnostics, these tools can assist in detecting and quantifying morphological features, acting as a decision support system for pathologists. Prognosis involves stratifying patients based on their risk or likelihood of disease recurrence or progression and would aid pulmonologists and oncologists to plan the course of treatment. Additionally, AI tools could identify sub‐visual patterns which may predict response to treatments such as radiotherapy and immunotherapy, thereby assisting clinicians in selecting the appropriate treatment.

Table 1.

Description of common computational pathology terms

Artificial intelligence	Umbrella term to indicate technologies with the ability to simulate intelligent behavior, allowing it to function appropriately and with foresight in its environment [19]
Digital pathology	Also known as whole slide imaging, it is the dynamic, image‐based environment that enables the acquisition, management, and interpretation of pathology information generated from a digitized glass slide [20]. Digital pathology has a growing interest in diagnostic medicine and lays the ground for AI tool deployment
Pathomics	The high throughput extraction of quantitative features from digitized histopathology images, thereby converting images into data [21]
Biomarker	“A biological marker, or biomarker, is a characteristic that is objectively measured and evaluated as an indicator of normal biological processes, pathogenic processes or a response to a therapeutic intervention” [22]. In digital pathology, it is an image characteristic that can be quantitatively measured and analyzed, and is typically associated with a disease or therapeutic outcome
Computational pathology	The use of artificial intelligence tools to extract information from whole slide images and associated patient metadata for specific clinical indications [23]
Machine learning	A subtype of artificial intelligence in which the system uses a provided set of data (training data) to learn and structure it, and subsequently uses a representative function to best describe the data and provide a prediction when presented with previously unseen data [24]
Deep learning	A type of machine learning technique with a representation‐learning method. Deep learning attempts to learn by example using neural networks [25]
Artificial neural network	An architecture inspired by the neuronal networks found within the brain and capable of constructing non‐linear relationships. Each layer in an artificial neural network has multiple perceptrons, which function like neurons, capable of receiving and sending signals [26]
Convolutional neural network (CNN)	A subtype of artificial neural network commonly applied to medical images; its extensive use derives from the architecture upholding the integrity of spatial relationships in image data. CNN is a feedforward network that specializes in filtering spatial data to create feature maps [27]
Generative adversarial network (GAN)	An example of generative AI, GAN is a deep learning approach that aims to ‘generate’ artificial data that resemble the real data using a combination of two neural networks – a generator, which creates synthetic data, and a discriminator, which scrutinizes the authenticity of the data [28]

Figure 1.

Digital pathology‐based AI tools add value to the workflow of the pulmonologist, oncologist, and pathologist. Created with Biorender.com.

Previous works in lung cancer have focused on the use of radiomics, which is a high throughput analysis of radiologic images [46]. In this article, we look at DP‐based AI tools used in lung diseases. First, we discuss various AI representations applied, elaborating on hand‐crafted and deep learning (DL)‐based methods. Next, we review the applications of AI approaches both in the field of lung cancers [47, 48, 49, 50] and in non‐oncologic diseases such as interstitial lung disease [51, 52, 53], tuberculosis [54, 55, 56], and COVID‐19 [57]. Further, we look at emerging applications, which we anticipate will be explored in the context of lung diseases. Lastly, we discuss the opportunities and challenges while developing AI algorithms regarding biases, regulatory approvals, reimbursements, and the clinical deployment of these tools.

AI approaches in lung pathology

AI is a broad discipline with multiple approaches to construct a model for a particular task (Table 1) [45]. There are two common ways to extract feature representations for building an AI model: DL‐based unsupervised feature learning and the hand‐crafted approach. While the former allows the system to identify features suitable for automatic classification, the latter uses existing domain expertise in lung histopathology to select the features. DL‐based unsupervised feature learning is favored in low‐level tasks such as cell and lung tumor detection and classification. This is useful since visual confirmation of the result is sufficient and does not require interpretation of the selected features. This contrasts with many high‐level tasks such as prognosis or treatment response prediction which require a certain level of interpretability and hence hand‐crafted domain‐inspired features may be favored by the medical community to construct these models [58].

Deep learning (DL)‐based unsupervised feature learning

Most WSIs are unlabeled, and annotating architecturally complex regions is laborious for pathologists, who are already facing increasing clinical workload [59, 60]. This hinders the development of machine learning models, which require well‐annotated datasets. Unsupervised feature learning is an approach in which the system automatically learns and selects appropriate features from the image to maximize class separability.

Convolutional neural network (CNN) [27] is a type of DL approach that has been used extensively in computational pathology models for a variety of tasks, including segmentation, object detection, and image classification [52, 55, 56, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70]. CNNs are composed of multiple layers of networks and are designed to learn spatial hierarchies of features automatically and adaptively through a backpropagation algorithm. They do so by deconvolution of the image content into thousands of salient features, followed by selection and aggregation of the most meaningful features and recognition of these patterns in yet‐unseen images. CNN is well suited for detecting, identifying, and classification of images. For example, upon providing a CNN with a set of images with the corresponding annotations, the network automatically learns the features and generates a probability map on a separate set of images. Inception v3 and Resnet50 are two common types of CNNs, representing supervised DL, that have been used in lung research for various tasks such as classification of histologic subtypes [65, 70, 71] or cell counting [72]. U‐Net is another CNN, where the ‘U architecture’ of the network combines the structural detail with its spatial context [73]. Hence, such models are used for quick and accurate segmentation of WSIs [74, 75, 76, 77] (Figure 2). With a particular interest in TILs as a prognostic biomarker in NSCLC, AlexNet [56, 75] has been employed in classifying TIL shape and density. Trained DL models are often not interpretable and hence are regarded as being ‘black‐box’ in nature [78]. This means that it is often challenging to mechanistically link the individual layers of a neural network function to a corresponding output, or which features from an image helped the DL model attain corresponding results. For some tasks, such as training a DL algorithm to identify nuclei or other structures, a visual confirmation of the output of the model by a domain expert is sufficient to test its accuracy and subsequently build imaging biomarkers that might be interpretable [79]. On the other hand, if the DL model generates a direct prognosis, the lack of interpretability might undermine trust among the physicians [80].

Figure 2.

Example of a neural network architecture for nuclei segmentation.

Hand‐crafted approach

In the hand‐crafted approach, relevant features from the data are manually selected. Domain knowledge is employed for feature engineering, which is subsequently used to construct machine learning models. This requires close collaboration between clinicians and machine learning engineers to construct appropriate AI models. In contrast to DL‐based unsupervised feature learning, through incorporating the expertise of subject matter experts, hand‐crafted‐based models lend themselves to some interpretability. Hand‐crafted features can be either domain‐agnostic or domain‐inspired (Table 2). The former may include morphometric features like shape, texture, and gland features of nuclei, which are common to multiple organs and tissue, while domain‐inspired features may have relevance to a particular disease site, like collagen fiber orientation, which was recently implicated in determining the aggressiveness of breast cancer [91]. In lung cancer, TILs [77] and cancer cell nuclei [74] have been explored as biomarkers to predict recurrence and risk of recurrence or death in NSCLC (Figure 3). For instance, the knowledge that hyperchromatic nuclei in malignant cells lead to a folded nuclear membrane [92] can be captured as pixel intensity variation of cancer nuclei using AI tools. Similarly, there has been a growing understanding that tumors behave as a cluster of cells [93], allowing graph‐based measurements of the tumor nuclei to analyze the spatial relationship within the tumor to assess its aggressiveness. Hand‐crafted features also facilitate the construction of AI models rooted in the understanding of cellular architecture and morphology. Studies have demonstrated that the density of TILs has a prognostic value [94]. An automated approach to quantify TIL density can overcome the challenge with inter‐reader variability in manual assessment. Similarly, the relationship of cells in the tumor microenvironment holds relevance, and mapping them could add prognostic value. For example, a low PD‐1 to CD‐8 ratio has been associated with a better prognosis in NSCLC [95]. Hand‐crafted features quantitatively analyzing the spatial architecture of the cells (immune, cancer, stroma) have been performed by constructing Delaunay triangulation (Figure 4). Likewise, textural‐based features have been analyzed using a gray level co‐occurrence matrix. For example, textural features extracted from the peri‐nuclear region in lung cancer may represent the consistency of cytoplasmic staining, with heterogeneous cytoplasm indicative of aggressive tumors. Lu et al [75] demonstrated a novel method of constructing feature‐driven local cell graphs (FLocK) that take into account nuclear intensity in addition to nuclear proximity, thereby adding granularity to the algorithm. They demonstrated that FLocK features extracted from WSIs of early‐stage NSCLC patients were associated with overall survival. While such hand‐crafted approaches alleviate some of the apprehension associated with the black‐box nature of DL models and inspire confidence among clinicians, they require extensive involvement of domain experts in defining regions of interest and interpreting results.

Table 2.

Description of hand‐crafted features used in lung pathology applications

Feature class	Description	Mathematical expression	Lung pathology studies
Spatial architecture	Describes the spatial organization of primitives (e.g. nuclei or glands) in the tissue. Facilitates identification of areas with a high disorder such as cancerous regions	Voronoi diagram, Delaunay triangulation, minimum spanning tree, local graphs	Yao et al [81], Wang et al [82], Vaidya et al [83]
Spatial interplay	Extracts measures from the points where two or more structures in the tissue come together and affect each other	Intersection, neighborhood diversity	Lu et al [75], Saltz et al [84], Barrera et al [85], Corredor et al [57], Corredor et al [77]
Texture	Quantifies the spatial arrangement of color or intensities in an image region. It is useful to differentiate among different structures in the tissue, for example, lymphocytes or cancerous cells	Haralick (gray level co‐occurrence matrix), Gabor, Laplacian, Laws features	Yao et al [81], Wang et al [82], Vaidya et al [83], Sandino et al [86], Corredor et al [87], Wang et al [88], Alvarez‐Jimenez et al [89]
Shape	Provides information on the physical appearance (e.g. size or silhouette) of some structures in the tissue such as cells, cartilages, vessels, nodules, among others	Area, length of axes, eccentricity, equivalent diameter, Zernike moments	Wang et al [82], Vaidya et al [83], Corredor et al [87], Lu et al [75]
Color	Extracts measures associated with the perception derived from the spectrum of light interaction with the human eye	Intensity, RGB channels, YUV channels, CMYK channels, HSV channels	Wang et al [82], Lu et al [90], Corredor et al [86]
Orientation	Provides metrics of the position of an element (e.g. nucleus) in relation to its surroundings	Angle between axes	Wang et al [82], Vaidya et al [83]

Figure 3.

Illustration of the use of hand‐crafted features for risk stratification of lung cancer. Tissue microarray from low‐risk and high‐risk lung cancer patients showing: (A, F) digitized H&E images; (B, G) features with automated nuclei detection; (C, H) features with Delaunay triangulation – a type of global graph; (D, I) local graph based on the distance between closest nuclei; and (E, J) feature showing the spatial architecture of tumor‐infiltrating lymphocytes with lymphocytes (blue) and non‐lymphocytes (green). Clusters are built based on distance thresholds. If cells of the same type are closer than a threshold, they form a cluster.

Figure 4.

Illustrative examples of hand‐crafted features used in lung pathology applications. Left to right: Delaunay triangulations capturing the spatial architecture; shape features: equivalent diameter, length of axes, and area; and color features demonstrating the RGB channels.

AI applications in lung pathology

Biomarker quantification and histogenomics

In recent years, the need for an increased number of ancillary studies such as immunohistochemistry, driver mutations, and programmed death‐ligand 1 (PD‐L1) has placed growing demands on pathologists, a scenario where they have to ‘do more with less’ [96, 97, 98]. Computational approaches have been developed to automate the estimation of biomarkers, as well as for the discovery of newer ones by integrating complementary modalities of data. For instance, computational pathology approaches which provide spatial and morphological information of the disease from WSIs have been combined with genomics, which provides the molecular profile of disease to aid in discovery of newer biomarkers [99, 100].

Immunohistochemical determination of tumor PD‐L1 status has become routine in identifying patients who may benefit from programmed death‐1 (PD‐1)/PD‐L1 inhibitors in NSCLC [101]. Currently, pathologists visually estimate the fraction of tumor cells expressing PD‐L1 via a tumor proportion score (TPS) system, which is constrained by ill‐defined cutoffs, differences in positivity estimation, and inter‐observer variability, making it a tedious and error‐prone task. It is also challenged by inadequate tissue samples or a lack of resources in peripheral settings [102]. Sha et al [103] developed a ResNet‐based DL model on WSIs to predict PD‐L1 status, thereby developing a ‘virtual stain’ using H&E WSIs alone. They used H&E and immunohistochemistry WSIs from 130 patients with pathologist‐annotated PD‐L1 positive and negative tumor regions on training samples as the ground truth. The model remained effective over a range of PD‐L1 cutoff thresholds [area under the receiver operating characteristics curve (AUC) = 0.67–0.81, p ≤ 0.01], even when different proportions of the labels were randomly shuffled to simulate inter‐pathologist disagreement (AUC = 0.63–0.77, p ≤ 0.03). Generative adversarial networks (GAN) (Table 1) have been explored as an approach for virtual H&E staining [104, 105]. Bayramoglu et al [106] used a conditional GAN model for virtual H&E staining of unstained lung tissue, which can allow automating DP workflow. Meanwhile, Kapil et al [107] developed a semi‐supervised GAN‐based approach to automate an objective PD‐L1 expression scoring method for patients with late‐stage NSCLC, reducing the need for manual annotations. Wu et al [108] developed a U‐Net‐based end‐to‐end DL model to automatically detect tumors, segment cells, and calculate the TPS of PD‐L1 by highlighting PD‐L1‐positive tumor cells. In a study with 437 NSCLC patients, Xia et al [109] demonstrated the utility of computational pathology approaches in the discovery of newer biomarkers. Using tissue microarrays, they discovered a new histologic feature, the stroma inflammation score, which was highly correlated with patient overall survival in NSCLC. This biomarker was subsequently validated by two blinded observers for prognostic value.

The World Health Organization classification of lung tumors has been recently modified [110] to emphasize the role of genetic mutations, given our increasing understanding of molecular profiles [110, 111]. Identifying the driver mutation is crucial for NSCLC management [112]. For example, LUADs with EGFR, ALK, ROS1, or RET mutations are associated with specific targeted therapies, and the KRAS G12C mutation is targeted by KRAS inhibitors [113, 114]. Coudray et al [71] trained a CNN model for a multi‐task classification, predicting the ten most common genetic mutations from H&E‐stained WSIs. The six commonly found mutations including STK11, EGFR, FAT1, SETBP1, KRAS, and TP53 were predicted by the model with AUCs between 0.733 and 0.856.

Lung adenocarcinoma is unique for having driver mutations which are potential targets for therapy. The recent approval of sotorasib for KRAS G12C‐mutated NSCLC [115] demonstrates the growing interest in developing novel therapies that target driver mutations. This often requires a companion diagnostic test that checks for the presence of specific mutations. These genomic assays often have longer turnaround times [116], and the costs could be prohibitively expensive, especially in lower‐middle‐income countries [117]. However, AI tools could be developed to detect the presence of point mutations from H&E slides alone, along with being non‐disruptive [118].

The diagnosis of idiopathic pulmonary fibrosis (IPF) requires a pathologic or radiologic diagnosis of usual interstitial pneumonia (UIP) [119]. In more than half of the patients, the diagnosis of UIP cannot be made by imaging alone, requiring a surgical biopsy, associated with a risk of morbidity and mortality [120]. Although transbronchial lung biopsy is a low‐yield substitute, it poses challenges for pathologists to confirm a diagnosis. Raghu et al [51] trained a machine learning model using histopathology and RNA sequence data from 90 patients to identify UIP. With the addition of molecular data, obtained along with biopsy samples, they demonstrated a significant improvement in the performance of the model. In addition, fibroblast foci are a known biomarker in IPF and increasing numbers of fibroblast foci have been associated with a poorer prognosis in some studies [15]. Mäkelä et al [52] trained a CNN to quantify fibroblast foci, interstitial mononuclear inflammation, and intra‐alveolar macrophages, and analyzed the association of these parameters with survival. Interstitial mononuclear inflammation and intra‐alveolar macrophages were associated with prolonged survival, suggesting their role as novel prognostic histologic biomarkers in IPF. In addition, the CNN‐based DL model also confirmed the association between high numbers of fibroblast foci and poor prognosis for patients with IPF.

Computer‐aided diagnosis (CAD)

Traditional pathology involves the use of a microscope by pathologists to analyze tissue samples for architecture, cellular patterns, and structures, and interpretation of these findings in the context of the clinical picture. This process is inherently subjective and prone to inter‐observer variability [121, 122, 123]. In some instances, pathologic analysis requires repetitive tasks such as counting mitoses in malignant cells or manual enumeration of cells positive for certain immunohistochemical markers (e.g. Ki‐67). Since computational approaches automate these processes, they have the potential to improve diagnostic accuracy and workflow efficiency, cutting down the time that pathologists spend on time‐consuming tasks, allowing more time for intellectually demanding aspects of interpretation [10, 124].

An issue that has become a focus of interest over the last 12 years due to advances in targeted therapies has been accurate histologic subtyping of non‐small cell lung carcinomas (NSCLCs). Despite substantial advances in immunohistochemical subtyping of NSCLC that have facilitated tumor subtyping [125], the accurate interpretation of subtypes of NSCLC can occasionally be challenging, particularly in poorly differentiated examples with overlapping immunophenotypes [126]. In one of the first studies to demonstrate the utility of CNNs in distinguishing between lung cancer subtypes, Coudray et al [71] trained a de novo Inception v3 network using transfer learning to automate the histologic type classification of each slide. Notably, in contrast to real‐life practice, where pathologists need to pick between dozens of choices when faced with a lung biopsy, the task in this study was limited to the separation of LUAD and squamous cell carcinoma of the lung. Another noteworthy point is that pathologists were allowed to use H&E morphology only, in contrast to real‐life practice, where immunohistochemistry is used for subtyping of cases that cannot be subtyped on H&E. Having acknowledged these limitations, the tumor subtype agreement between the DL model and the classification provided by the gold standard for the study (The Cancer Genome Atlas; TCGA) was higher than the agreement between each pathologist and the TCGA. In analyzing the slides that posed challenges to pathologists but were correctly classified by the model, it performed well even with poorly differentiated tumors, demonstrating the advantage of CNN approaches in ambiguous cases.

In LUAD, in addition to the stage at the time of diagnosis, which has been the most important prognostic parameter for decades, the predominant tumor growth pattern also impacts the outcome. The lepidic growth pattern is the least aggressive; papillary and acinar growth patterns are considered intermediate; and micropapillary and solid growth patterns are categorized as the most aggressive, associated with an increased likelihood of recurrence and poor survival [127]. Recently, the cribriform pattern has also been identified as a marker of unfavorable prognosis in early‐stage LUAD [128].

In another application, Gertych et al [64] constructed a lightweight CNN using an AlexNet model. The goal was to assess the accuracy of the CNN in identifying four LUAD growth patterns (acinar, micropapillary, solid, and cribriform) in surgically resected cases. Compared with the gold standard (pathologist interpretation of tumor versus non‐tumor and LUAD growth patterns), they also observed that de novo trained models performed significantly better than pre‐trained models and that the model performed better in some validation sets than in others. Wei et al [129] used a ResNet model for a multi‐label classification problem. They trained the model on 422 WSIs of LUAD and compared the output of the model in predicting the probability of six histological subtypes against three pathologists. The model achieved a Kappa score of 0.525 and showed an agreement of 66.6% with three pathologists, which was higher than the inter‐pathologist Kappa score (0.485). Yu et al [130] recently constructed a fully automated framework to identify tumor regions and their histologic subtype, and correlate them with individual transcriptomic profiles, allowing the tool to function as an end‐to‐end decision support system for thoracic pathologists in lung cancer evaluation.

AI tools have also been applied to cytopathology samples obtained via fine needle aspiration, typically in the form of individual cells and smaller tissue fragments. They are less invasive than a biopsy and are increasingly performed for the evaluation and staging of lung cancer [131]. The individual cells seen in these samples can be computationally analyzed for cellular and nuclear morphology. In 2020, Gonzalez et al [132] studied cytology and biopsy specimens from 40 patients and trained a CNN‐based DL algorithm to distinguish small cell lung carcinoma (SCLC) and large cell neuroendocrine carcinoma based on morphologic features. The number of cases used was very small, although in this limited sample both the cytology algorithm and the biopsy algorithm were able to correctly identify most cases of both tumors, demonstrating the feasibility of using cytology specimens for CAD systems.

One challenge for pathologists is to identify metastatic tumor deposits in lymph node biopsies and excisions, which can be difficult in a background of lymphocytes, histiocytes, endothelial cells, and other distractors. To resolve this, Pham et al [61] developed a two‐step DL model with a CNN approach, in which the first algorithm eliminated all lymphoid follicles from the WSI and subsequently detected ‘true positive’ nodal metastasis in the second step. Using this DL method, they reduced errors by 36.4% on average and up to 89% in slides with reactive lymphoid follicles. Although developed on lung cancer patients, the detection of metastatic tumor cells in a background of reactive immune cells can be expanded to other disease indications as well.

Corredor et al [57] performed one of the first studies applying computational pathology approaches in COVID‐19 to quantitatively characterize the spatial architecture of lymphocytes and non‐lymphocytic cells on digitized autopsy images from patients who died of COVID‐19 and compared it with those who died of H1N1, a comparable viral infection. While visual assessment of lymphocytes in COVID‐19 samples has been performed [133, 134], it was qualitative with inter‐reader variability. In contrast, in this study, the authors used a watershed approach to segment the nuclei, a machine learning classifier to identify lymphocytes, and subsequently constructed clusters of cells to analyze spatial patterns.

Tuberculosis screening is a laborious task that involves the detection of acid‐fast bacilli on a pathology slide stained with acid‐fast stains such as the Ziehl–Neelsen stain [135, 136]. Experts have described best practices for the detection of acid‐fast bacteria in pathology specimens, but these remain labor‐intensive and time‐consuming [137]. In a complicated study that involved errors both in the pathologist‐based ground truth and in the AI model, Xiong et al [54] developed a CNN model, named tuberculosis AI (TB‐AI) by feeding patches from positive and negative regions of the WSI to train the model. Although the TB‐AI system initially misdiagnosed several cases of dye residue and contaminants as acid‐fast bacteria, the algorithm was improved in a subsequent run, achieving 97.94% sensitivity and 83.65% specificity. Lo et al [56] developed a CNN classification model using 1,815 image blocks to identify acid‐fast bacilli with an accuracy of 95.3%. In addition, they repeated the same experiment without color and saw a drop of accuracy to 73.8%, thereby demonstrating the role of color information in detections (Figure 5).

Figure 5.

Applications of computational pathology in lung diseases.

Prognostic and predictive applications

Pleural mesothelioma is a rare malignancy of the pleura, with a poor prognosis [138]. Courtiol et al [67] developed MesoNet, a fully unsupervised deep CNN using WSIs to predict the overall survival of mesothelioma patients without any pathologist‐provided locally annotated regions. They validated the model on an internal validation cohort and an independent cohort from the TCGA. Subsequently, they demonstrated the model to be more accurate in predicting patient survival than current pathology practices. However, unlike classical black‐box DL methods, MesoNet identified regions contributing to patient outcome prediction, primarily in the stroma.

The classification of LUAD developed by the International Association for the Study of Lung Cancer (IASLC), the American Thoracic Society (ATS), and the European Respiratory Society (ERS) was an effort to develop an integrated approach to classify NSCLC and identify prognostic and predictive biomarkers [139]. While it demonstrated that growth patterns in LUAD are prognostic, they are not currently used for clinical decision making, and there remains a lack of robust and objective biomarkers to determine the risk of recurrence and metastasis in patients with early‐stage LUAD. In early‐stage NSCLC treated with surgery, the added benefit of adjuvant chemotherapy is unknown, and administration depends on each patient's risk of recurrence. However, there are no validated biomarkers to predict the risk of recurrence accurately and consistently.

Yu et al [140] developed a prognostic model for early‐stage NSCLC using quantitative histomorphometric analysis by combining an automated feature extraction method with a machine learning classifier. They analyzed local anatomical structures (nuclei and cytoplasmic shape) as well as global features (nuclear and cellular texture) and found them to be predictive of patient survival. In a similar study, Luo et al [141] demonstrated that nuclear and cell texture features were independently prognostic of survival across all stages of NSCLC. In a study with 305 patients, Wang et al [82] focused on graph features extracted from tumor nuclei features of tumor cells on tissue microarray core images to develop a supervised classification model for predicting recurrence in early‐stage NSCLC. Using a multivariable Cox proportional hazard model, including additional markers such as gender and disease stage, the computationally derived risk score was found to be independently prognostic. A unique prospect of computational pathology techniques is the ability to perform multiple analyses stepwise. In a subsequent study with over 1,000 patients, Wang et al [74] developed an image risk score from nuclear and perinuclear histomorphometric features which was prognostic in early‐stage NSCLC. In addition, it was one of first models in lung cancers to additionally predict the added benefit of adjuvant chemotherapy by demonstrating a subset of high‐risk patients treated with only surgery who might have had a better survival if treated with adjuvant chemotherapy. Saltz et al [84] used WSIs from the TCGA to study the association of TIL patterns with molecular features and overall survival. Corredor et al [77] analyzed the spatial architecture of TILs and non‐lymphocytes as well as the interplay and co‐localization of these cells, and demonstrated features associated with recurrence in early‐stage NSCLC patients.

Lu et al [76] developed a feature classifier named CellDiv, which analyzes the local cellular morphological diversity displayed by lung cancer owing to its heterogeneity. CellDiv features showed a strong correlation with overall survival in early‐stage NSCLC in addition to being able to distinguish KRAS mutation status and being associated with biologic pathways of cellular differentiation, apoptosis, and signaling. Vaidya et al [83] took an approach combining the top predictive features from lung CT scans and H&E WSIs to construct an integrated ‘RaPtomic’ classifier to predict 5‐year recurrence free‐survival in NSCLC. The integrated model had a better performance (AUC 0.78; p < 0.005) than radiomic (AUC 0.74; p < 0.01) and pathomic (AUC 0.67; p < 0.05) features alone.

In SCLC, Jain et al [142] studied the association of quantitative features of tumor nuclei extracted from platinum‐based chemotherapy‐treated patients with prognosis and observed that the top features capturing nuclei shape, size, and texture were able to stratify patients based on their overall survival.

In the last decade, several novel therapeutic agents have been discovered to treat previously incurable lung diseases [142, 143, 144, 145, 146, 147, 148]. While targeted agents such as immunotherapy have shown high efficacy, they are expensive and have low response rates [149, 150, 151]. Hence, there is a need to study the therapeutic response and identify patients who are likely to benefit from treatment or are at a higher risk for adverse effects. With the approval of the first immune checkpoint inhibitor (ICI) pembrolizumab as a first‐line drug for metastatic NSCLC [152], there has been much interest in exploring the tumor microenvironment and tumor‐infiltrating lymphocytes in the hope of finding reliable biomarkers to predict immunotherapy response [94, 153, 154].

In a study of 221 patients treated with nivolumab, Gataa et al [155] analyzed the association of TILs with treatment response to ICI. They employed a semi‐quantitative method by pathologists using H&E‐stained sections from archival pretreatment tumor tissue sample to assess the TIL density and observed that a high TIL density (>10%) was associated with response to ICI therapy in advanced NSCLC, thereby raising the possibility of TIL density as a potential biomarker for predicting response to ICI.

Wang et al [156] quantitatively extracted features related to the spatial arrangement of TILs and nuclei from 82 NSCLC patients treated with the PD‐1 inhibitor nivolumab to construct a random forest classifier, which yielded an AUC of 0.68 on the validation set. Ofek et al [157] used a DL approach to analyze the spatial arrangement of TILs in the tumor microenvironment. This imaging biomarker was predictive of response to pembrolizumab in patients with advanced NSCLC. Barrera et al [85] took a hand‐crafted approach by having two expert pathologists manually delineate tumor regions which were used to identify the TILs. Subsequently, TIL arrangement features were computed to create a machine learning classifier that accurately distinguished patients who responded to nivolumab versus those who did not. The detection and subsequent analysis of TILs also pave the way for studying other biomarkers such as tumor‐associated macrophages in lung cancer [158].

Jain et al [142], in a study of 106 SCLC patients, trained a naïve Bayes classifier analyzing morphological and functional features (shape, size, intensity, cellular texture) of the cancer nuclei to predict response to platinum‐based chemotherapy. Barrera et al [159] looked at a combination of textural and shape features from WSIs with features from CT scans to construct a machine learning classifier for predicting response to chemotherapy in SCLC.

Emerging applications

Multi‐omics

Intratumoral heterogeneity in NSCLC has been well studied [160]. Recent studies have mapped spatial transcriptomics onto histopathology feature maps to analyze the role of different cell populations in cancers [161, 162]. The underlying molecular mechanisms for several morphologic features in NSCLC have largely remained unexplained [163]. Meanwhile, there are more imaging features currently not readily evaluable by clinicians, yet associated with patient outcomes [164]. For example, lung cancer can present either as ground‐glass opacities or as solid nodules on chest CT scans [165, 166]. Despite being the same clinical entity, their appearance, and the clinical course, can differ, indicating distinct tumor biology. With 538 LUAD cases from the TCGA, Yu et al [167] used WSIs, RNA sequencing, and proteomics data, and constructed a machine learning model to identify molecular pathways associated with morphologic features. Interestingly, the authors chose a random forest [168], a machine learning algorithm which works well with high‐dimensional data such as multi‐omics, to correlate with outcomes. AbdulJabbar et al [169] developed a DL tool integrating spatial histology with multi‐region exome sequencing data to map regions of immune ‘hot spots’ and their association with intratumoral heterogeneity. The tool, validated on squamous cell carcinoma and LUAD patients from the TraceRx study and LATTICe‐A study, demonstrated that the overall number of immune cold regions within a tumor determined the prognosis of LUAD.

A radiologic examination is often the first modality of investigation in lung disease due to its non‐invasive nature. However, one of the major drawbacks of radiographic imaging is the inability to accurately define the disease extent, which still requires histopathologic analysis. Recent computational approaches have allowed co‐registering or ‘mapping’ the histologic volume from biopsies onto the CT scan volume to allow facilitating early characterization of lung nodules' extent on non‐invasive imaging modalities by avoiding biopsy [170]. Rusu et al [170] performed one of the first known studies in co‐registering lung ex vivo pathology specimens with CT scans. They demonstrated a method of first constructing a 3D histological volume using group‐wise registration and subsequently co‐registering the CT scan with the reconstructed histology. Their approach showed a Kappa value of 0.68, indicating substantial agreement.

Microbiome

The lung microbiota has been shown to play a role in lung cancer development as well as in prognosis and response to treatments [171]. It is also associated with the histological subtypes of lung cancer [172]. Given its role in disease progression, there have been large‐scale efforts to collect microbiome and metabolome data [173]. With the demand to process and analyze this ‘big data’, machine learning tools in combination with image‐based biomarkers captured from WSIs offer a bright prospect.

3D pathology

Novel imaging techniques such as 3D microscopy [174] have recently been utilized in DP workflows. The non‐destructive nature of the technology along with its ability to analyze larger samples of tissue could help to overcome challenges faced by spatial and temporal heterogeneity. Currently, it has been applied in the context of prostate cancer in enhancing risk stratification [175]. With its ability to interrogate spatial relationships of various cell types by reconstructing entire tissue blocks, techniques such as 3D microscopy combined with DP would be an invaluable tool in a pathologist's armamentarium.

Transplant rejection prediction

A major cause of mortality in lung transplant recipients is graft failure, and predicting the risk of rejection poses a challenge for clinicians as clinical criteria alone are inaccurate [176]. Studies developing machine learning models using WSIs of tissues from heart [177, 178] and kidney [179, 180] transplant patients have demonstrated unique morphological patterns related to lymphocytes and stroma to predict the risk of transplant rejection. Similar techniques could potentially be applied to lung transplant recipients in the future.

Drug development and discovery

Recently, there has been renewed interest in drug development for pulmonary diseases such as IPF and pulmonary artery hypertension to address unmet medical needs [181, 182, 183]. A crucial step is the estimation of response to novel agents using scoring systems that may involve manual assessment of histopathology slides [184, 185]. Computational approaches can accelerate the process by automated analysis of digitized image data [186, 187, 188, 189].

Challenges and opportunities

Challenges pertaining to both the technical development of AI tools and their clinical translation need to be addressed before adoption by the clinical community. Past works have explored the various technical hurdles such as batch effects, slide artifacts, and staining variability, and their solutions [190, 191, 192]. One of the issues with the clinical translation of DP‐based AI tools is the appropriate regulatory approval pathways from agencies such as the FDA, in the USA and the Conformité Européenne (CE) mark in Europe [193]. Currently, no specific regulatory pathways exist for the approval of AI‐based tools in the United States. The recent FDA approval of Paige Prostate [32] for prostate cancer detection was classified as a Class II device, indicating a moderate to high risk to patients [45]. It underwent the de novo pathway [194] and was categorized as a ‘software algorithm device to assist users in digital pathology’. This creation of a new product code is expected to allow more DP‐based solutions to follow suit in obtaining approvals for clinical translation [195]. While the tools approved so far aid in diagnostics, regulation for predictive tools is expected to be rigorous, with careful evaluation of their performance owing to the downstream impact they may have in patient management. Even after approvals, diagnostic decision support systems will have to be monitored via prospective trials to avoid misuse as primary diagnostic tools [196]. While there are no DP‐based AI tools for lung cancer at the time of writing, Optellum [197] received a 510(k) approval for ‘Virtual Nodule Clinic’ as the first radiomics‐based AI tool for lung cancer detection, which demonstrates the interest in these tools in pulmonary diseases.

Additionally, AI tools need to address issues with reimbursement, which may be complicated since the existing fee‐for‐service model cannot be applied [198, 199]. The American Medical Association (AMA) current procedural terminology (CPT) codes and Centers for Medicare & Medicaid Services (CMS) currently do not have a payment pathway for DP‐based AI tools. The recent decision by the CMS to grant New Technology Add‐on Payment reimbursement to a stroke detection model by Viz.AI [200] performed under a unique ICD‐10 Procedure Coding System (ICD‐10‐PCS) procedure code [201] is a step forward. One approach would be to create a new CPT code for CAD tools [202], whereas another might be to analyze and modify existing ones to incorporate DP services [203].

The clinical deployment of DP‐based AI tools requires collaboration between stakeholders, including machine learning scientists, software developers, and clinicians who are the end‐users. While diagnostic tools such as Paige Prostate have been deployed as decision support tools integrated with image viewers, such a process is yet to be developed for prognostic and predictive tools [204]. DP‐based AI tools can be deployed either as Storage as a Service (SaaS) cloud business model or as on‐premise solutions [205]. While cloud‐based programs may facilitate remote access of these tools, the larger size of high‐resolution pathology images might favor an on‐premise tool. The mode of deployment may vary according to institutions, and financial implications need to be taken into consideration as well [206].

Given that results from AI tools may have a far‐reaching impact on the life of a patient, there are important legal challenges to be considered for their safer translation into clinical practice [207]. An equitable liability scheme needs to be designed to define the roles of AI tool developers, physicians, and healthcare institutions. This discussion would revolve around what is considered as the gold standard adopted by the clinical community, and whether AI would continue to remain as assistive technology, or become the gold standard against which the performance of a physician is evaluated [208].

Finally, an important consideration before the integration of AI tools into clinical practice is to make sure that these tools have been ethically trained and are not subject to bias [199]. While human errors in diagnostics have been largely attributed to cognitive biases [209], AI errors tend to be systematic and often introduced during the training phase [210]. Efforts need to be taken to ensure that future AI tools are trained and validated on a plurality of populations. There have been attempts to incorporate molecular and phenotype differences among populations which may manifest as tissue morphology differences into AI models. Studies in prostate [211], breast [212], and endometrial [213] cancers suggest that population‐specific AI models could be more accurate in stratifying patients based on their risk.

Conclusion

The use of AI tools in lung DP is expanding rapidly [214]. While CNN‐based DL models performing detection and pattern recognition are intuitively suited for low‐level tasks such as automated lung tissue segmentation, disease diagnosis, and quantifying biomarkers such as PD‐L1, hand‐crafted models are being integrated with clinical data and outcomes for risk stratification and prediction of treatment response, which can be further made interpretable by using domain‐inspired hand‐crafted features [215]. Integration of DP in the clinical workflow across major institutions remains a challenge, due to organizational structures, the cost of initial set‐up, requirements of advanced security systems in hospitals, and demands for storage of ‘big data’ [214, 215, 216, 217]. Currently, most existing AI tools have been validated on retrospective data [196], which does not represent real‐world scenarios. Validating the algorithms in randomized controlled trials and prospective studies will be a crucial step towards clinical adoption [199]. There is still hesitancy within the clinical community regarding the black‐box nature of some AI‐based tools, which may determine regulatory approvals as well as their acceptance for clinical deployment. Hence, there is a growing interest in interpretable AI [218]. DP‐based AI tools hold immense potential for improving workflow issues for pathologists by reducing labor‐intensive, time‐consuming tasks, and for pulmonologists and thoracic oncologists by providing risk stratification and companion diagnostics tools for lung diseases. While there are significant challenges in translating these technologies into clinical practice, recent regulatory approvals and deployment of these tools [219] signal a changing trend.

Author contributions statement

VSV and AM researched data for the manuscript. GC and VSV created the illustrations for the manuscript. All the authors were responsible for reviewing and editing the manuscript.