Translation of adapting quantitative CT data from research to local clinical practice: validation evaluation of fully automated procedures to provide lung volumes and percent emphysema

Abstract.

Current clinical chest CT reporting includes limited qualitative assessment of emphysema with rare mention of lung volumes and limited reporting of emphysema, based upon retrospective review of CT reports. Quantitative CT analysis performed in COPDGene and other research cohorts utilize semiautomated segmentation procedures and well-established research method (Thirona). We compared this reference QCT data with fully automated QCT analysis that can be obtained at the time of CT scan and sent to PACS along with standard chest CT images. 164 COPDGene^® cohort study subjects enrolled at Brigham and Women’s Hospital had baseline and 5-year follow-up CT scans. Subjects included 17 nonsmoking controls, 92 smokers with normal spirometry, 15 preserved ratio impaired spirometry (PRISm) patients, 12 GOLD 1, 20 GOLD 2, and 8 GOLD 3–4. 97% ($n=319$) of clinical reports did not mention lung volumes, and 14% ($n=46$) made no mention of emphysema. Total lung volumes determined by the fully automated algorithm were consistently 47 milliliters (ml) less than the Thirona reference value for all subjects (95% confidence interval $-62$ to $-32\mathrm{ml}$). Percent emphysema values were equivalent to the Thirona reference values. Well-established research reference data can be used to evaluate and validate automated QCT software. Validation can be repeated as software is updated.

1. Introduction

Qualitative CT imaging of diffuse lung disease, such as emphysema, allows for detection of signs of early lung disease before impairments are noted on pulmonary function testing.^1,2 Visual analysis allows for distinction between subtypes of emphysema based on the extent and location of disease. However, qualitative reports are prone to intraobserver and interobserver variability, and the human eye struggles to detect potentially emphysematous holes $<5\mathrm{mm}$ in size. Some scoring methods visually quantify emphysema by lung region and percent emphysema.^2,3 The Fleischner Society recently published a 6-tier (5-point) visual classification system to standardize terminology for emphysema severity.⁴ However, appropriate use of this system requires visual training for each observer. Clinical reporting of emphysema on CT scans may under-report subtle disease and over-report extensive disease, hampering the detection of disease progression over time.

Quantitative data from CT imaging is less subjective and can detect subtle changes in lungs over time. Image analysis software procedures have been developed to process the large numerical datasets generated from each scan. Over time, the requirements for manual preparation of data have diminished, with semiautomated procedures used to evaluate the Genetic Epidemiology of Chronic Obstructive Pulmonary Disease (COPDGene^®) cohort study⁵ CT scans providing a valuable reference standard. The purpose of this investigation is to compare a fully automated procedure that can readily be presented to the radiologist at the time of clinical CT interpretation with the reference standard. This would allow the addition of quantitative CT (QCT) data into clinical chest CT reports. The detection of subtle disease and changes over time not apparent on clinical inspection of images would increase identification of early disease and assessment of treatments, thereby improving patient care.

Through labor-intensive procedures, COPDGene^® has established reference data for QCT parameters that correlate with clinical reference data, including pulmonary function testing.⁵ If fully automated procedures provide an adequate substitute, these measurements can complement qualitative CT reports and potentially more precisely track change over time without added burden on clinical reporting of chest CT scans. Syngo.via PulmoCT (Siemens) is a fully automated image-processing procedure that results in a display of volume and density data via tabulated numbers and color-coded emphysema maps; this can be added to CT scans as part of the data reconstruction process and provided to the radiologist in PACS. Syngo.via is also directly available within our institutional PACS.

Little is known about how QCT measurements from fully automated software correlate with the research reference standard. In the current retrospective study, we examine the clinical reporting of COPDGene^® CT scans at a single institution [Brigham and Women’s Hospital (BWH)] and the validity of adding fully automated QCT lung volume and lung density measurements to improve clinical chest CT reporting. Hypothesizing that clinical reports inconsistently report emphysema, we first examined historical clinical CT reports for BWH COPDGene^® scans to assess qualitative reporting of emphysema and description of change over time. We then quantified lung density and volume using Syngo.via and benchmarked these measurements to the reference standard measurements in COPDGene^® obtained through semiautomated processing⁶ (Thirona); then we compared Syngo.via-derived measurements with qualitative data from the clinical reports to assess concordance between qualitative and quantitative assessments of emphysema.

2. Materials and Methods

2.1. Subjects

COPDGene^® is an ongoing, multicenter longitudinal study that investigates the genetic and epidemiologic characteristics of COPD. The protocols for subject recruitment and data collection for the COPDGene^® study have been previously described.⁵ For this study, we identified 181 COPDGene^® subjects who were enrolled at the BWH site and had baseline (phase 1, 2008 to 2010) and 5-year follow-up (phase 2, 2013 to 2015) noncontrast thoracic CT scans obtained on a Siemens 64 slice multidetector CT scanner. Only subjects who had no prior or interim thoracic CT scans were included in this study, in order to examine qualitative reports of change over time from the baseline to the follow-up scan. Patients who had additional imaging for clinical care at BWH were excluded because the clinical reports were not based specifically on the COPDGene CT scans obtained at the 5-year interval. 17 subjects did not have quantitative data from Thirona present in the COPDGene^® database and were thus excluded from analysis, resulting in a total of 164 subjects. This population was 51% male and 87% nonHispanic white, with a mean age of 63 years (standard deviation of 7.4) at the time of enrollment. Retrospective review of the electronic health records was approved by the Institutional Review Board.

2.2. CT Image Analysis

Inspiratory CT scans from phase 1 and phase 2 of COPDGene^® were reconstructed using subcentimeter inspiratory images reconstructed with the standard b31f kernel and analyzed using the Thirona Lung Quantification software.^5,6 The corresponding 328 CT scans, reconstructed at the same kernel, were then reassessed using the Syngo.via PulmoCT software. QCT data of interest were total lung volume (TLV) and percent emphysema (% low attenuation area or %LAA, defined as the percentage of lung voxels with attenuation lower than $-950\mathrm{HU}$ at maximal inspiration). Unlike the Thirona data set, Syngo.via scans were not visually checked for appropriate lung segmentation quality, although the option for manual re-segmentation is present in Syngo.via and can be accessed through thin client application in PACS as well as at the CT scanner. Clinical radiology reports created shortly after each scan was obtained were reviewed in the electronic medical records.

2.3. Statistical Analysis

Linear regression analysis for two independent variables was used to estimate agreement between the COPDGene^® Thirona reference standard and the Syngo.via PulmoCT fully automated measurement of TLV and percent emphysema for each of the 328 reconstructed scans. Bland–Altman analyses were also performed to display limits of agreement (LoA) for these comparisons. Similar regression and Bland–Altman analyses were used to assess the variability in lung volumes on serial scans obtained 5 years apart. Qualitative emphysema grading via the clinical CT reports was treated as an ordinal variable, and correlation between this and percent emphysema was obtained using Spearman’s correlation.

3. Results

3.1. Retrospective Review of Reports

The 164 subjects represented a spread of COPD disease severity and were classified as previously described using spirometry.⁵ Review of clinical CT scan reports showed that 97% ($n=319$) of clinical reports did not mention lung volumes and 14% ($n=46$) made no mention of the presence or absence of emphysema (Table 1). Emphysema was reported inconsistently. 43 out of the 73 phase 1 scans reported to have some degree of emphysema had a listed emphysema score; emphysema score was not included in the phase 2 scan reports. Clinical reports used a 7-tier subjective scale (0 to 6: no emphysema, minimal, mild, mild to moderate, moderate, moderate to severe, severe). All 17 phase 1 scans for nonsmokers were designated as having no emphysema according to the clinical report, while emphysema was detected variably in smokers. Change over time was also documented inconsistently for the 164 follow-up scans. Only 45% ($n=73$) of follow-up reports made any mention of the presence or absence of change (including emphysema, fibrosis, bronchiectasis, or inflammation) relative to the baseline scan.

Table 1.

Patient demographics and clinical report at baseline (phase 1) and 5-year follow-up (phase 2) including qualitative assessment of lung volumes, emphysema, and change over time.

	Never smoker	GOLD 0	PRISm	GOLD 1	GOLD 2	GOLD 3/4
Number	17	92	15	12	20	8
Sex (%male)	10	27	4	5	8	2
Age (mean)	64	52	61	66	64	68
Race (%NHW)	10	27	4	5	8	2
Phase 1 report emphysema (%)	14	88	14	11	18	8
Phase 1 report no emphysema (%)	13	47	7	2	8	0
Phase 1 report lung volumes (%)	1	3	0	0	4	0
Phase 2 report emphysema (%)	13	84	13	11	16	2
Phase 2 report no emphysema (%)	14	4	7	1	8	0
Phase 2 lung volumes (%)	0	0	4	0	0	0
Comparison of emphysema phases 1 and 2	2	11	7	7	9	7

3.2. Quantitative CT

Reconstructed submillimeter inspiratory b31f scans were successfully processed by the commercial analysis software (Syngo.via PulmoCT) using fully automated segmentation to generate QCT measurements. The mean TLV and % emphysema generated for each group are in Table 2. Automated QCT measures of supine TLV and percent emphysema for the whole lung were linearly related to those from Thirona. Corresponding linear regression analyses are shown at left in Fig. 1, with Bland–Altman plots at right. TLVs determined by the fully automated algorithm were consistently 47 milliliters (ml) less than the Thirona reference TLV for all subjects (95% confidence interval $-62$ to $-32\mathrm{ml}$), with a clear 1 to 1 relationship between the two measurements ($\text{slope}=1.001$, with 95% confidence interval 0.998 to 1.004). The mean difference was $-41\mathrm{ml}$, with 95% LoA of $-98$ to 16 ml. Percent emphysema values were equivalent to the Thirona reference (intercept of 0.02% with 95% confidence interval $-0.01\%$ to 0.04%), with a near 1 to 1 slope (0.985, with 95% confidence interval 0.982 to 0.988). The mean difference was $-0.04\%$, with 95% LoA of $-0.45\%$ to $+0.37\%$.

Table 2.

TLVs and percent emphysema for all subjects at baseline and follow-up.

	$N$	Thirona				Syngo.via
		Baseline TLV (liters)	Follow-up TLV	Baseline % emphysema	Follow-up % emphysema	Baseline TLV	Follow-up TLV	Baseline % emphysema	Follow-up % emphysema
Nonsmokers	17	$5.196\pm 1.193$	$4.901\pm 1.333$	$1.70\%\pm 1.67\%$	$1.21\%\pm 1.26\%$	$5.167\pm 1.186$	$4.865\pm 1.337$	$1.75\%\pm 1.53\%$	$1.26\%\pm 1.15\%$
Smokers (GOLD 0)	92	$5.239\pm 1.213$	$5.192\pm 1.205$	$1.66\%\pm 1.89\%$	$1.76\%\pm 2.39\%$	$5.196\pm 1.221$	$6.155\pm 1.206$	$1.69\%\pm 1.80\%$	$1.74\%\pm 2.25\%$
PRISm	15	$4.676\pm 1.280$	$4.533\pm 1.034$	$0.86\%\pm 1.20\%$	$0.86\%\pm 1.34\%$	$4.613\pm 1.282$	$4.475\pm 1.015$	$0.88\%\pm 1.18\%$	$0.86\%\pm 1.31\%$
GOLD 1	12	$6.571\pm 1.362$	$6.387\pm 1.446$	$8.32\%\pm 10.39\%$	$9.86\%\pm 10.91\%$	$6.531\pm 1.362$	$6.342\pm 1.454$	$8.18\%\pm 10.26\%$	$9.73\%\pm 10.86\%$
GOLD 2	20	$5.900\pm 1.549$	$6.015\pm 1.743$	$7.66\%\pm 9.29\%$	$9.95\%\pm 11.37\%$	$5.844\pm 1.563$	$5.970\pm 1.745$	$7.51\%\pm 9.22\%$	$9.75\%\pm 11.3\%$
GOLD 3 to 4	8	$6.098\pm 1.610$	$6.018\pm 1.641$	$19.49\%\pm 14.30\%$	$21.29\%\pm 14.00\%$	$6.037\pm 1.600$	$5.962\pm 1.638$	$19.10\%\pm 14.23\%$	$20.89\%\pm 14.02\%$

^aTLV, total lung volume in liters; %, percent; and values shown are mean ± standard deviation.

Fig. 1.

(a) Linear regression between Thirona and Syngo.via TLVs and (c) percent emphysema. (b), (d) Corresponding Bland–Altman plots.

3.3. Five-Year Intrasubject Variability in Lung Volumes and Density

Fully automated QCT-derived (Syngo.via) TLVs from scans taken 5 years apart (baseline versus follow-up scans) correlated positively [$r=0.890$, Fig. 2(a)]. The differences were normally distributed, with a mean difference of $-69\mathrm{cc}$ and 95% LoA of $-1325$ to $+1188\mathrm{ml}$ [Fig. 2(b)], reflecting an up to 30% change in volumes for 95% of the data points. Distribution of the absolute percent change relative to the baseline scans is shown in Fig. 2(c). Mean percent variation was 9%, with a median of 6%. Percent emphysema values for scans taken 5 years apart were linearly related with minimal change over time; this mean change over time was $-0.5\%$, with 95% LoA of $-4.7\%$ to 3.8%.

Fig. 2.

Relationship of lung volumes at baseline versus 5-year follow-up: (a) corresponding Bland–Altman plot, (b) absolute percent change in volumes over the 5-year interval, and (c) absolute percent change in volumes over the 5-year interval relative to baseline.

3.4. Percent Emphysema

Correlation between the language used in clinical reports and the percent emphysema data is shown in Fig. 2. The subjective 7-tier qualitative report terminology was treated as ordinal data (0 to 7 from none to severe) and showed a weak positive correlation (Spearman’s rho, $r_s=0.36$ for $n=328$, respectively) between qualitative and quantitative measures of emphysema severity. The clinical reports did notice visual patterns of emphysema even in patients with low percent emphysema (defined as % low attenuation area below $-950\mathrm{HU}$). Scans with more than 15% emphysema were typically recognized as having worse disease on the CT report, although three clinical reports described lungs with over 35% emphysema as having no disease.

There were 8 follow-up scans for which the clinical reports denoted increased emphysema, 10 with increased fibrosis, and 4 with mixed change. On review of these individual subjects, these changes did not clearly correspond to changes in TLV or percent emphysema (data not shown). Likewise, scans with $>1\mathrm{l}$ change in TLV or $>5\%$ change in percent emphysema frequently did not have a qualitative correlate in the reports.

4. Discussion

Our study identified near universal failure to report lung volumes in clinical reporting of COPDGene^® subject chest CT scans at our institution. The presence or absence of emphysema was infrequently included in clinical reports. Radiologists under-report emphysema on chest CT scans in heavy smokers without a prior diagnosis of COPD.⁷ Under-reporting of emphysema may decrease the identification of early lung disease that could benefit from medical intervention. Longitudinal comparisons of emphysema, when included in clinical reports, revealed intra- and interobserver variability, providing confusion rather than clarity regarding both stability and progression of disease. This study confirmed a fundamental difference between chest radiography evaluations, where assessment of lung volumes is the first step in the interpretation process while all but completely ignored in evaluation of chest CT scans. The large number of CT images and more detailed imaging features undoubtedly contributed to this disparity. In the nine reports in which lung volumes were mentioned, the context was clearly for quality assurance purposes and not as a consideration in anatomy, physiology, or pathology within lung parenchyma.

Inclusion of percent emphysema could improve the consistency of qualitative reporting. Three CT scans with confluent to advanced emphysema per Fleischner grading guidelines (as done by the COPDGene^® imaging core⁸) were mistakenly described as having no emphysema, and these scans in fact had more than 35% emphysema. Had QCT data been available to the radiologist at the time of interpretation, it may have prompted more careful image review. Beyond reducing reader error, increased percent emphysema correlates with worse pulmonary function⁹ and mortality risk,⁸ and its inclusion on a CT report may provide a gestalt for disease severity.

The automated analysis software also creates simplified color-coded images that correlate with percent emphysema thresholds and could be used to improve communication with primary care providers and as a patient-education tool to improve patient understanding of disease severity and progression. Artificial intelligence can be expected to incorporate QCT and complex clinical data to potentially predict impending exacerbations as well as important turning points in subtle disease progression.

TLV is not currently thought to be a good marker of COPD progression. TLV of COPD patients in the complete COPDGene^® dataset did not differ from healthy nonsmoking controls.⁶ However, CT-calculated and plethysmographic TLV do correlate well,^10–12 and unexpectedly low TLVs might raise concern for other pathology such as fibrosis or attenuated effort, which should be commented on and could prompt further clinical evaluation. TLV provides a good first step for incorporation of QCT measurements into clinical CT interpretation because it is frequently used in the adjustment of other QCT measurements such as lung density and airway thickness that indicate parenchymal destruction⁶ and airway inflammation.¹³

Our comparison of fully automated QCT data with the best reference standard (Thirona) has identified systematic variance that is quite consistent for both volumes and percent low attenuation area. The difference of $\sim 50\mathrm{ml}$ is quite small relative to not only TLVs but also the variations in an individual’s TLV from their baseline to follow-up scans, which we found to be anywhere from 0 to 1000 ml; this individual variation is consistent with other studies.^12,14 Consistently lower lung volume is undoubtedly related to the automated segmentation algorithm that places the segmentation line within peripheral lung parenchyma. Although the segmentation can be adjusted, the automated algorithm provides a level of consistency that would be difficult for the radiologist to match. This also underlies the systematic over-estimation of change on serial CT scans.

The validation process used in our study can be applied to any other fully-automated software. Software updates may warrant repeating the process with a smaller validation set of CT data to re-evaluate the variance due to small differences in software algorithms. Indeed, we reprocessed a small subset of our data 1 year in Syngo.via after the initial process date and found deviations of no more than 5 ml in TLV and 0.1% emphysema.

Although QCT data are promising, the subjects in this study redemonstrate^6,8 a discordance between clinical and radiologic assessment of emphysema that would not be miraculously eliminated by incorporation of QCT data in clinical reporting. Some scans described as having anywhere from moderate to severe emphysema (Fig. 3)—and confirmed to have moderate to advanced disease per Fleischner grading guidelines⁸—had low percent emphysema per QCT, suggesting that qualitative observation can be more sensitive than QCT in some respects. In fact, increased Fleischner severity confers higher mortality risk⁸ independent of percent emphysema, underscoring the complementary role of quantitative and qualitative image assessment. In focusing on lung volumes and lung density, we have not addressed important additional potential for incorporating QCT into clinical reporting with the large population of COPD patients without emphysema for whom identifying more airway-oriented imaging markers are of great interest.¹⁵

Fig. 3.

Correlation plots of percent emphysema versus qualitative clinical reports.

We have not studied restrictive lung disease, although lung volumes can be obvious determinants of disease progression in fibrotic lung diseases and the smaller interscan variations in TLV for such patients may allow for observation of change over time.¹⁴ Routine utilization of QCT data in clinical chest CT reporting may point to additional uses for the data. Adding the dimension of QCT to clinical CT reporting can add valuable data for patient care without burdening the radiologist. Adopting QCT for clinical reporting will identify additional opportunities for QCT to inform the application of artificial intelligence to chest CT scans for patients with COPD and other diffuse lung diseases.

Limitations in this study include selection bias due to exclusion criteria for subjects with additional chest CT scans for comparison in clinical reports. This decreased the number of subjects with active disease, particularly GOLD 3 and GOLD 4 patients included in this study due to their more frequent need for clinical CT scans. This study only validated the software that can automatically provide the QCT data with the CT images obtained in our department. From our experience in adding CT scans that had initially been missed, we foresee the need to periodically revalidate software at intervals not addressed in this study. Comparison of chest CT scans over long periods of time adds to the variability of qualitative assessment of subtle lung disease and confounds precision in QCT for which simple measurements with repeatable validation sets may add a robust dimension.

5. Conclusions

QCT derived by fully automated commercial analysis software can be understood through comparison with reference standards and sent with clinical chest CT images to PACS. Lung volumes and emphysema are both inconsistently reported and under-reported in current clinical CT reports. Inclusion of QCT data using fully automated commercial software offers the potential to improve detection and progression of lung disease on serial chest CT imaging.

Biographies

Francine L. Jacobson is director of lung cancer screening for Brigham Health and assistant professor of radiology at HMS. Her radiology research interest in perception has led to her involvement in many studies of lung nodule detection and detection of lung parenchymal features from CT scan images. She served as the BWH site primary investigator for the National Lung Screening Trial, and currently serves as site radiologist for the COPDGene cohort study.

Biographies of the other authors are not available.

Disclosures

No conflicts of interest, financial or otherwise, are declared by the authors.