Deep Learning–based Classification of Fibrotic Lung Disease: Can Computer Vision See the Future?

Despite existing diagnostic criteria and guidelines for identifying and classifying fibrotic lung disease such as idiopathic pulmonary fibrosis (IPF) (1, 2), their diagnosis can be challenging. Current guidelines emphasize the role of high-resolution computed tomography (HRCT), and place particular importance on identifying the presence of underling usual interstitial pneumonia (UIP), which suggests a diagnosis of IPF (1). However, this approach is binary: it requires patients be classified based on the predominant pattern on HRCT, while in practice patients may have some evidence of UIP features but a different predominant disease pattern (1, 2). Clinically, current UIP diagnosis also relies on subjective readings of the HRCT that may vary from radiologist to radiologist (3). These issues are of particular concern because of the importance of UIP in identifying patients likely to have faster disease progression and worse prognosis (4–7). Thus, missed or inaccurate diagnosis has the potential to have significant clinical impact, and there has been great interest in developing tools to improve the detection and classification of UIP on HRCT (3).

Deep learning is a form of machine learning that utilizes multi-layered, or deep, neural networks to learn from complex data such as imaging. Deep learning based tools have proliferated in radiologic research over the past decade and have shown great promise in the analysis of HRCTs for diseases ranging from lung cancer to IPF (8–10). One of the strengths of deep learning algorithms is that they may capture patterns not seen or ignored by the human eye, potentially improving disease classification and quantification (11).

In this issue of the Journal, Walsh and colleagues (pp. 883–891) report the results of their study using the Systematic Objective Fibrotic Imaging Analysis Algorithm (SOFIA), a previously developed and validated deep convolutional neural network tool, to identify UIP in 516 patients from the Australian IPF registry (12, 13). For each participant’s HRCT, SOFIA generates 500 unique, 4-slice image montages. Then, for each montage, the device calculates the probabilities for each of 4 categories: definite UIP, probability UIP, indeterminate UIP, and not UIP. These probabilities (which sum to 1.0) are then averaged, and the device provides the patient level probability for each category (13). The investigators then used Cox proportional hazards and logistic regression models to determine the relationships between SOFIA UIP probabilities with transplant-free survival and with 12-month disease progression, respectively. Multiple forms of analysis were performed including using predicted probabilities as continuous measures as well as grouped into categories based on the Prospective Investigation of Pulmonary Embolism Diagnosis (PIOPED) approach. Univariate analyses, bivariable analyses that included both SOFIA based information and expert radiologist assessment, and multivariable models that were additionally adjusted for age, gender, radiologist-defined computed tomography disease severity, and other clinical measures of disease severity, were all performed as well.

In general, the authors found that not only did SOFIA UIP probabilities, both as a continuous variable and into grouped into PIOPED categories, predict transplant free survival and disease progression in univariate analyses, but also when evaluated using bivariable analyses that included radiologist defined disease extent. In fact, in those bivariate analyses only the SOFIA based measures were associated with adverse outcomes, not the radiologist defined disease extent. Similar findings were present in multivariable analyses adjusted for total disease extent and clinical variable associated with adverse outcomes in fibrotic lung disease like age and lung function. Importantly, the predictive utility of SOFIA UIP was maintained in subgroup analyses of patients with UIP on HRCT or histopathology versus and in those with other fibrotic patterns.

A particularly interesting finding of this study was the predictive probability of SOFIA in the subset of patients with indeterminate UIP on HRCT. Using the PIOPED based binning strategy, SOFIA reclassified over a quarter of HRCTs classified as indeterminate UIP by expert radiologists as intermediate, high, or pathognomonic probability of UIP, and this reclassification predicted transplant-free survival in the multivariable model, with a hazard ratio of 1.73 (95% confidence interval, 1.40–2.14) in the indeterminate UIP subgroup. The reclassification of a large proportion of indeterminate UIP HRCT cases is perhaps unsurprising when considering that the interobserver agreement between two radiologists for the ATS/ERS/JRS/ALAT (American Thoracic Society, European Respiratory Society, Japanese Respiratory Society, and Latin American Thoracic Society) criteria is only moderate, even among expert thoracic radiologists with over ten years of experience (3). Still, these results highlight the potential utility of a deep learning model to identify subtle fibrosis patterns and improve clinical diagnosis, especially in indeterminate cases that are more challenging to determine.

Despite this study’s many strengths, it has several limitations as well. The number of analyses and comparisons raises the question of multiple testing, and 34.6% of the cohort was receiving antifibrotic therapy, potentially affecting the analysis of transplant-free survival, as patients with more severe disease are more likely to be receiving therapy. Still, the biggest limitations of this work are not unique to this specific study but related to the field of artificial intelligence in medicine more generally. For example, when there are multiple possible algorithms for diagnosing or classifying a disease such as pulmonary fibrosis available, how do we know which to choose? And once one is chosen, who will fund its certification as software as a medical device with regulators, especially when it is unclear who would pay for its clinical use? Finally, who is responsible if the algorithm is wrong and misdiagnoses a patient as having a disease when they do not, or vice versa? Before a deep learning model like SOFIA is brought to the clinical setting, these questions and others need answering. In the meantime, work such as this by Walsh and colleagues demonstrates the potential power of deep learning in medicine and the need to answer these difficult questions so that patients can benefit from the insight artificial intelligence can provide.