Improving Prognostication in Pulmonary Hypertension Using AI-quantified Fibrosis and Radiologic Severity Scoring at Baseline CT

Abstract

Background

There is clinical need to better quantify lung disease severity in pulmonary hypertension (PH), particularly in idiopathic pulmonary arterial hypertension (IPAH) and PH associated with lung disease (PH-LD).

Purpose

To quantify fibrosis on CT pulmonary angiograms using an artificial intelligence (AI) model and to assess whether this approach can be used in combination with radiologic scoring to predict survival.

Materials and Methods

This retrospective multicenter study included adult patients with IPAH or PH-LD who underwent incidental CT imaging between February 2007 and January 2019. Patients were divided into training and test cohorts based on the institution of imaging. The test cohort included imaging examinations performed in 37 external hospitals. Fibrosis was quantified using an established AI model and radiologically scored by radiologists. Multivariable Cox regression adjusted for age, sex, World Health Organization functional class, pulmonary vascular resistance, and diffusing capacity of the lungs for carbon monoxide was performed. The performance of predictive models with or without AI-quantified fibrosis was assessed using the concordance index (C index).

Results

The training and test cohorts included 275 (median age, 68 years [IQR, 60–75 years]; 128 women) and 246 (median age, 65 years [IQR, 51–72 years]; 142 women) patients, respectively. Multivariable analysis showed that AI-quantified percentage of fibrosis was associated with an increased risk of patient mortality in the training cohort (hazard ratio, 1.01 [95% CI: 1.00, 1.02]; P = .04). This finding was validated in the external test cohort (C index, 0.76). The model combining AI-quantified fibrosis and radiologic scoring showed improved performance for predicting patient mortality compared with a model including radiologic scoring alone (C index, 0.67 vs 0.61; P < .001).

Conclusion

Percentage of lung fibrosis quantified on CT pulmonary angiograms by an AI model was associated with increased risk of mortality and showed improved performance for predicting patient survival when used in combination with radiologic severity scoring compared with radiologic scoring alone.

© RSNA, 2024

Supplemental material is available for this article.

Summary

Combining artificial intelligence (AI)–quantified lung fibrosis and radiologic severity scoring at baseline CT improves prognostication in pulmonary hypertension; AI detects minor disease not identified by radiologists.

Key Results

• ■ In this retrospective study of 521 patients with pulmonary hypertension, artificial intelligence (AI)–quantified lung fibrosis at baseline CT was associated with an increased risk of mortality (hazard ratio range, 1.01–1.03; P value range, <.001 to .04).

• ■ Combining AI and radiologic severity scoring improved predictions of patient survival compared with radiologic scoring alone (concordance index, 0.67 vs 0.61; P < .001).

Introduction

Lung disease severity is an important distinguishing factor between two pulmonary hypertension (PH) phenotypes—pulmonary arterial hypertension (group 1) and PH associated with lung disease (PH-LD) (group 3). In practice, distinguishing between these two groups is challenging in patients with radiologically scored mild or no fibrosis due to overlapping clinical characteristics, particularly in the most common form of pulmonary arterial hypertension, idiopathic pulmonary arterial hypertension (IPAH) (1–3). Only patients diagnosed with IPAH are eligible for novel targeted therapies that improve survival (4). Chest CT imaging is routinely performed for diagnosis but is not currently used for prognostication (5). Lung disease is typically semiquantitatively scored as none, mild, moderate, or severe, but this scoring has poor reproducibility even among specialist radiologists (6,7).

There is a clinical need to better quantify lung disease in these patients. A recent seminal study by Hoeper et al (3) characterized this problem and identified a new phenotype—IPAH with a lung phenotype—which lies between IPAH and PH-LD and overlaps between radiologically scored no and mild lung disease. In the study by Hoeper et al (3), 71% of patients with IPAH with a lung phenotype were scored as having no fibrosis, and 25% were scored as having mild fibrosis. Currently, patients with mild lung disease are treated but show poorer response to treatment and lower survival (3,8). Advances in artificial intelligence (AI) deep learning approaches have made quantitative analysis of imaging features possible (9). The motivation of this work was to use AI with CT pulmonary angiography (CTPA) imaging performed routinely in these patients to help solve this important clinical problem. The AI model developed for this study provides the respective percentage of lung volume involvement of common radiologic patterns and an output that can be interpreted by the reading radiologist. The transparent patch-based approach used (10) and the model implemented (11) have been described in detail. The purpose of this study was to assess the value of AI-quantified fibrosis in predicting survival, independent of and in combination with radiologic scoring.

Materials and Methods

Study Patients

Adult patients with incidental IPAH and PH-LD diagnosed between February 2007 and January 2019 were prospectively recorded in Sheffield Pulmonary Vascular Disease Unit databases as part of the ASPIRE (Assessing the Spectrum of Pulmonary Hypertension Identified at a Referral Centre) registry, as previously described (3,12). The ASPIRE registry was established as a large clinicoradiologic imaging data set in PH and has been used in multiple studies (3,12). There may be overlap in patients with other unrelated studies given the large registry, but the analysis in this study is unique. For all patients, diagnosis was made at the host tertiary center after multidisciplinary review, and this approach was uniform throughout the study period. The current study is a secondary analysis of this prospectively collected data set. Ethical approval was granted by the institutional review board of Sheffield Teaching Hospitals NHS Foundation Trust and approved by the National Research Ethics Service (16/YH/0352). The need for written informed consent was waived due to the retrospective nature of this study.

The inclusion criterion was baseline CT imaging performed at the time of diagnosis. The exclusion criteria were as follows: patients with no baseline imaging or imaging inappropriate for AI analysis (section thickness > 1.25 mm), incomplete lung imaging or segmentation, non-CTPA imaging, severe breathing artifact, or lobectomy. These eligibility criteria were chosen in an attempt to capture a spectrum of lung disease and PH etiology, regardless of the assigned diagnosis of IPAH or PH-LD, and to minimize bias. Mortality data were obtained from systems linked to the National Health Service Personal Demographics Service, which is updated when a death is registered in the United Kingdom. Patients undergoing lung transplantation were censored at the time of surgery, and mortality data were collected using a census date of May 2019.

CT Image Acquisition, Reporting, and Analysis

CTPA protocol scans were performed in multiple centers on scanners from all major manufacturers with varying technical parameters (Table S1). The AI model was applied to all scans; reports for all scans were written by experienced subspecialty chest radiologists, who radiologically scored fibrosis as none, mild, moderate, or severe as previously described (3,13). All analyses were performed at the patient level with one single corresponding incident CTPA examination. Lung segmentation was performed using an externally validated nnU-net (no new U-Net) model (14). The AI method uses a patch-based five-class DenseNet121 deep learning architecture that classifies parenchyma into regions of ground glass, ground glass with reticulation, emphysema, honeycombing, and normal lung as per the Fleischner Society glossary of terms (15). Fibrosis for this patient cohort was defined as a composite variable summing regions of ground glass with reticulation and honeycombing. The method is described in more detail in Appendix S1, and the model code is available at https://github.com/mrsharkleton/lung_texture_segment.

Statistical Analysis

Survival analysis.—Patients were divided into a training set and a test set based on the institution where imaging was performed. Imaging examinations performed at the host institution composed the training set; all external imaging examinations composed the test set. Multivariable Cox proportional hazards regression was used to determine the association between AI-identified CT parenchymal patterns and survival. Hazard ratios (HRs) correspond to the ratio of two hazard rates (of death) when a continuous variable is increased by one. P < .05 was considered to indicate a statistically significant difference. The multivariable Cox proportional hazards model was adjusted for age, sex, World Health Organization functional class, pulmonary vascular resistance (PVR) (a hemodynamic marker of PH severity), and the diffusing capacity of the lungs for carbon monoxide (Dlco) (a spirometry marker and the current gold standard of lung disease severity assessment in these patients). Age, PVR, and Dlco were stratified using thresholds of 50 years, 5 Wood units (WU), and 45%, respectively, as established in prior studies (3,16–18). Model assumptions of linearity and proportional hazards were assessed using scaled Schoenfeld residuals and Martingale residuals. In the training set, locally estimated scatterplot smoothing analysis, with estimated probability of death at 1 year on the y-axis and percentage of fibrosis on the x-axis, was used to derive the threshold for fibrosis corresponding to 20% 1-year mortality. Patients in the test set were grouped by the derived thresholds. Survival estimates from the time of registry were calculated using Kaplan-Meier analyses, and survival curves were compared using the log-rank test.

Added value of AI over visual radiologic scoring alone.—The strength of Cox proportional hazards regression models was used to determine the added predictive value of AI quantification. Model A included visual fibrosis scoring only; model B included visual scoring and AI-quantified percentage of fibrosis. The model concordance index (C-index) was used to assess the prognostic strength of the model: A better model fit is indicated by a higher C index. Both models included the same patients, and a likelihood ratio test was used to assess whether there was a statistically significant difference. The analysis was repeated stratified by assigned diagnosis of IPAH or PH-LD. Prespecified subgroup analysis was performed in all patients scored as having no fibrosis in visual scoring by radiologists. Cox proportional hazards regression was used to determine the association between AI-quantified disease severity and survival.

Analysis was performed with R version 4.1.2 (R Foundation for Statistical Computing). The study sample size and number of events are appropriately powered to detect statistically significant differences. Categorical data are presented as numbers and percentages, and continuous data are presented as medians and IQRs. Two-sample Wilcoxon rank sum tests were used to compare continuous data, and Pearson χ² tests or Fisher exact tests were used to compare categorical data, between the training and test cohorts.

Results

Patient and Cohort Characteristics

Of 5643 patients in the ASPIRE registry, 521 patients (median age, 67 years [IQR, 25–74 years]; 270 women [52%]) met the eligibility criteria and were included for analysis. Of 1143 patients with a diagnosis of IPAH or PH-LD, 243 were excluded because there was no baseline imaging, 202 because CT section thickness was greater than 1.25 mm, 121 for non-CTPA imaging, 30 for severe breathing artifact, 24 for incomplete or incorrect lung segmentation, and two for lobectomy (Fig 1). There were no failures in AI analysis (Fig 2), and the model gave the same result for scans when repeated.

Figure 1:

Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) diagram shows patient selection. ASPIRE = Assessing the Spectrum of Pulmonary Hypertension Identified at a Referral Centre, CTPA = CT pulmonary angiography, IPAH = idiopathic pulmonary arterial hypertension, PH-LD = pulmonary hypertension associated with lung disease.

Figure 2:

Example outputs from the artificial intelligence (AI) model. The lung parenchyma is automatically segmented and then classified into five lung parenchymal patterns: normal (red), emphysema (teal), honeycombing (yellow), ground glass (green), and ground glass with additional reticulation (blue). This is done for every section of the CT pulmonary angiogram. Example input (A) axial, (B) sagittal, and (C) coronal CT sections (top row) from one patient are presented with the corresponding AI-quantified outputs (bottom row). The AI model output is provided as a Digital Imaging and Communications in Medicine, or DICOM, scan series that can be interrogated and reviewed alongside the original CT scan series.

The cohort had a wide range of lung disease severity as assessed using pulmonary function testing (median Dlco percent predicted, 31% [IQR, 22%–49%]; median forced vital capacity percent predicted, 90% [IQR, 77%–107%]; median forced expiratory volume in 1 second percent predicted, 75% [IQR, 55%–90%]) and hemodynamic PH disease severity (median PVR, 8.0 WU [IQR, 5.0–11.6 WU]). There were 275 patients in the training cohort and 246 in the external test cohort. Patients in the test cohort were younger (median age, 65 years [IQR, 51–72 years] vs 68 years [IQR, 60–75 years]; P < .001), were more likely to be female (58% vs 46%; P = .01), and had more severe PH (median PVR, 9.4 WU [IQR, 6.4–13.2 WU] vs 7.2 WU [IQR, 4.2–10.8 WU]; P < .001). A greater proportion of patients in the external cohort were diagnosed with IPAH (61% vs 40%; P < .001). There was no significant difference in Dlco percent predicted between groups (median, 32% [IQR, 23%–53%] in training set vs 30% [IQR, 22%–47%] in test set). These data are presented in Table 1, and data availability for each variable is shown in Table S2. The external test group therefore formed an ideal heterogeneous test cohort to assess model generalizability, with differences in demographic and hemodynamic characteristics. There were no demographic differences between the excluded patients and the study cohort (Table S3). CT visual scoring was available in 166 of 246 (67%) patients in the external test cohort. The test set included CTPA imaging examinations from 37 centers on 33 different scanners, as described in Table S1.

Table 1:

Patient Demographic and Clinical Characteristics for All Cohorts

Survival Analysis

The model assumptions of linearity and proportional hazards were validated and are described in Table S4 and Figure S1. Univariable and multivariable associations between each AI-identified pathologic parenchymal pattern and mortality in the training cohort are presented in Table 2. The median follow-up duration was 1.96 years (IQR, 0.91–4.35 years). The outcome of mortality occurred in 183 of 275 (66%) patients. Two of 275 (0.7%) patients underwent lung transplantation. All AI-quantified pathologic CT patterns were found to be associated with increased risk of patient mortality in univariable analysis—ground glass (HR, 1.02 [95% CI: 1.01, 1.03]; P = .004), ground glass with reticulation (HR, 1.04 [95% CI: 1.02, 1.05]; P < .001), honeycombing (HR, 1.05 [95% CI: 1.03, 1.07]; P < .001), emphysema (HR, 1.01 [95% CI: 1.01, 1.02]; P < .001), and the composite variable fibrosis (HR, 1.03 [95% CI: 1.02, 1.04]; P < .001). Only fibrosis was associated with increased risk of mortality in the multivariable analysis (HR, 1.01 [95% CI: 1.00, 1.02]; P = .04). When this prognostic model was applied to the external test set, it demonstrated good discrimination in predicting mortality (C index, 0.76).

Table 2:

Cox Proportional Hazards Regression Assessing the Association of AI–quantified CT Features and Radiologic Scoring with Mortality Outcomes in the Training Cohort (n = 275)

Locally estimated scatterplot smoothing, or LOESS, analysis (Fig S2) identified a percentage of fibrosis of 3.43% as the threshold corresponding to 20% 1-year mortality. Kaplan-Meier survival analysis demonstrated that patients with fibrosis above the identified threshold of 3.43% in the test cohort had poorer survival (log-rank test; P < .001) (Fig 3). The estimated survival rates for patients with and without fibrosis above the threshold were 69% versus 88% for 1-year survival, 36% versus 66% for 3-year survival, and 24% versus 56% for 5-year survival, respectively. This 3.43% threshold was a significant predictor of poor survival in IPAH and PH-LD cohorts when analyzed separately (P < .001 for both; Fig S3).

Figure 3:

Graph shows Kaplan-Meier survival curves for patients in the external test cohort (n = 246) stratified into groups by the threshold for significant fibrosis (3.43%) derived in the training cohort (n = 275). Log-rank test P value is shown. Shaded regions represent the CIs.

Additional Prognostic Value of AI Metrics Over Radiologic Scoring Alone

Model B (AI quantification and visual scoring combined) had better predictive strength than Model A (visual scoring only) in the full cohort (C index, 0.67 vs 0.61; P < .001) and when patients were stratified by diagnosis (IPAH: C index, 0.63 vs 0.59, P < .001; PH-LD: C index, 0.65 vs 0.59, P < .001) (Table 3). A total of 300 of 421 patients (71%) were radiologically scored as having no fibrosis. The AI model reported minor levels of fibrosis (1.5%), ground glass (1.8%), ground glass with additional reticulation (0.6%), and honeycombing (0.7%) in these patients (Table 4). Fibrosis (HR, 1.03; P = .004) and honeycombing (HR, 1.08; P < .001) were associated with increased risk of mortality in this subgroup. Example outputs of mild, moderate, and severe fibrosis are shown in Figure 4.

Table 3:

Additional Predictive Value of AI Quantification in Combination with Visual Scoring

Table 4:

Significance and Extent of AI-quantified Disease in Subgroup of Patients Radiologically Scored as Having No Fibrosis

Figure 4:

Examples of degrees of fibrosis quantified and radiologically scored. CT pulmonary angiograms (top row) were radiologically scored as having mild (left), moderate (middle), and severe (right) disease; the corresponding output and fibrosis percentage as quantified by the artificial intelligence (AI) model are shown in the bottom row.

Discussion

There is clinical need to better quantify lung disease in pulmonary hypertension (PH) due to an overlap in radiologically scored severity between different phenotypes. The first key finding of our study is that artificial intelligence (AI)–quantified fibrosis at CT was prognostic (hazard ratio, 1.01; P = .04) for survival in patients with PH and lung disease. This was independent of demographic characteristics, hemodynamic disease severity (pulmonary vascular resistance), and spirometry (diffusing capacity of the lungs for carbon monoxide), and was externally validated (concordance index [C index], 0.76) in images from 37 centers. The second key finding of our study is that the AI model was sensitive to minor parenchymal differences and, when used in combination with radiologic reporting, provided additional predictive prognostic value, with a C index of 0.67 for AI-quantified fibrosis and radiologic scoring versus 0.61 for radiologic scoring alone (P < .001).

This is the first study to our knowledge to clinically evaluate an AI quantitative CT model for lung disease assessment in PH. Existing work in this domain uses pulmonary function testing (Dlco) or radiologic scoring for lung disease assessment. Pulmonary function tests show variability and low reproducibility and can be insensitive to changes in disease progression (19–22). Radiologic scoring is subjective and has inter- and intraobserver variability even among specialist radiologists (6,7). It is also relatively imprecise, being typically scored categorically as none, mild, moderate, or severe. A few continuous visual scoring scales exist, in which radiologists quantify lobar disease to the nearest 5%, but these scales have been limited to the research domain (22). Guidelines for interpretation of fibrosis focus on identification of parenchymal features and not grading or severity assessment (23). Carefully quantifying five different lung parenchymal features is tedious, time-consuming, and impractical in routine clinical reporting, thereby making it an ideal assistive task for AI.

Percentage of fibrosis was found to be an independent prognostic marker (HR, 1.01; P = .04) for mortality even after correcting for hemodynamic disease severity (PVR) and Dlco. The PVR threshold of 5 WU was chosen based on recent studies that demonstrated its utility in predicting mortality in both IPAH and PH-LD (16,18,24). The Dlco threshold of 45% was derived from previous studies that determined the strong prognostic value of this threshold (8,17,25–27). Dlco is often the only biomarker used for lung disease severity across multiple studies and is part of the REVEAL 2.0 risk score for pulmonary arterial hypertension risk stratification (3,8,28,29). We derived 3.43% as a clinically applicable threshold for fibrosis severity, which corresponded to 20% 1-year mortality. This threshold was validated in an external test cohort, where between-group differences in survival were found between patients stratified by this threshold. This was also the case for IPAH and PH-LD analyzed separately. We postulate that the additional prognostic value of AI-quantified CT parenchymal patterns compared to Dlco may be because Dlco is a functional marker of gas exchange, whereas CT findings are an anatomic marker of architectural distortion.

Combining AI-quantified fibrosis and radiologic scoring improved prognostication. In patients radiologically scored as having no fibrosis, the AI model identified minor levels of lung disease that were associated with increased risk for mortality. This could be due to a combination of (a) lack of detectability and (b) variability in subjective radiologic opinion regarding whether minor parenchymal changes represent true fibrosis. In our experience, it is not uncommon for different radiologists to differ in their opinions among none, minor, and mild fibrosis. It has been established that there is interobserver variability in the detection of fibrosis, which is magnified at low levels of disease and worse for individual parenchymal patterns than for overall diagnosis (30,31). Our repeatable and interpretable quantitative AI model outputs would likely aid in these cases to improve consensus. The C index of 0.67 for this model in external testing demonstrates good discrimination, particularly considering that this C index value is for disease severity scoring alone. In clinical practice this value will be higher as it will also account for patient demographic characteristics, clinical features, and other imaging.

There is substantial overlap between PH phenotypes, with 71% of patients with IPAH with a lung phenotype being scored as having no fibrosis, and 7% of those with classic IPAH being scored as having mild or moderate fibrosis (3). Mild lung disease can be treated with pulmonary arterial hypertension therapy in the current guidance, but these patients have poorer survival and treatment response compared with those with no lung disease (3,8). However, the prognostic clinical significance of lung disease has not been precisely quantified and has relied on semiquantitative radiologic scoring. The combined approach in our study could enable better phenotyping and identification of patients likely to respond to treatment. Limitations of deep learning approaches include their lack of transparency and their susceptibility to unknown, difficult-to-quantify biases. Our combined approach was designed to put the radiologist at the center of the decision-making process. The model provides an interpretable image mask of the analyzed scan, which can be evaluated and assessed by the radiologist. This enables recognition of errors and verification of model outputs. The proposed AI model use case is as an adjunct to reporting, providing a continuous percentage of lung involvement for individual lung parenchymal patterns.

We acknowledge limitations inherent to the retrospective setting, CTPA imaging, and AI models. First, while all patients were assigned a diagnosis in the specialist tertiary center, there may be some bias due to patients not being appropriately identified in nonspecialist settings. Second, there are data availability and demographic differences between the training and external test cohorts, which could be a source of algorithmic bias. The main reason for exclusion was lack of thin-section CTPA imaging, which is performed less frequently in nonspecialist centers. Third, variations in breath holding and contrast agent administration timing during CT scan acquisition can cause errors in the AI classification. We excluded only patients with severe breathing artifact (30 patients) in this study, and the AI use case proposed is as an adjunct to clinical reporting. The output is interpretable and transparent to the reporting radiologist. Fourth, this AI approach was applied in a single large tertiary center, but we preferentially chose an external test cohort of patients with a range of demographic characteristics, lung disease, and hemodynamic disease severity to validate and generalize our findings. Fifth, our thresholds, although validated externally, might not be generalizable to all institutions and may need to be adjusted for different prevalence rates and patient demographic characteristics.

In conclusion, our study showed that the percentage of lung fibrosis quantified on CT pulmonary angiograms using an artificial intelligence (AI) model was associated with increased risk of mortality in patients with pulmonary hypertension (PH). Furthermore, a prognostic model combining AI-quantified fibrosis and radiologic severity scoring showed improved performance for predicting patient survival compared with a model including radiologic severity scoring alone. Further larger multiregistry prospective studies that investigate the use of such AI models in better characterizing and phenotyping patients with PH and lung disease are warranted.