A New Simple to Use Chest CT Scoring System: Validation Study

ABSTRACT

Background

Current chest computed tomography (CT) based scoring systems are complex.

Objective

Validity of new simple and easy to use chest CT scoring system was determined in pwCF with varying severity of lung disease.

Study Design

Three pulmonary physicians scored 60 chest CT scans in 30 pwCF, ages 12–60 years, done at initiation and at 1‐year on ETI therapy and scores were correlated with ppFEV₁.

Results

Interrater agreement among three raters, determined by intraclass correlation coefficient (ICC) (95% confidence interval [CI]) scores in total cohort, was 0.83 (0.72, 0.91), for bronchiectasis (BEC), 0.83 (0.72, 0.91) for mucus plugging (MP), and for bronchial wall thickness (BWT) and parenchymal lung disease (PLD) was 0.56 (0.35, 0.74) and 0.7 (0.54, 0.83) respectively (all in the moderate to good range). Intrarater reliability (4–6 weeks apart) on 10 pwCF (20 CT scans) in total cohort was excellent, ranging from 0.92 to 0.98. For individual components it was excellent for BEC, good to excellent for PLD and MP, and moderate for BWT. ppFEV₁ correlated well with total chest CT scores, r = −0.75 (p < 0.001). For individual components, the strongest correlations were noted with BEC (−0.71), PLD (−0.73) and MP (−0.68). With ETI therapy, median (interquartile range [IQR]) total scores improved from 49 (42, 54) to 23.5 (18, 31.7) (p < 0.001) in total cohort and similar patterns were also seen in individual components (BEC, BWT, MP, AT, and PLD).

Conclusion

Our scoring system was validated with acceptable interrater and intrarater agreement and was able to quantify expected changes with therapy.

Abbreviations

AT

air trapping

BEC

Bronchiectasis

BWT

bronchial wall thickening

CFTR

Cystic fibrosis transmembrane conductance regulator

CT

Computed tomography

ETI

Elexacaftor‐tezacaftor‐ivacaftor

ICC

intraclass correlation coefficient

MP

mucous plugging

PLD

parenchymal lung disease

ppFEV₁

Percent predicted forced expiratory volume in 1 s

pwCF

people with cystic fibrosis

1. Introduction

Several scoring systems have evolved to define disease severity and extent in a numerical reproducible manner to quantify structural lung disease [1]. These scoring systems were developed by radiologists and are complicated to understand, apply, and reproduce within and outside the domain of thoracic radiology; thus, they have limited clinical usefulness and are used primarily as research tools measuring changes in structural lung disease.

Initial chest computed tomography (CT) based scoring systems, such as the one proposed by Bhalla et al. in 1994 [2], had nine features with scores for multiple degrees of severity. With time, increasing technological improvements in CT techniques, lower radiation exposures and high‐resolution imaging for both inspiratory and expiratory films were able to define smaller changes with more clarity [3]. The Brody scoring system [4, 5] accounted for these changes, but is even more complex with total possible score of > 200. Thoracic radiologists need training and practice before scoring chest CT scans and there are still marked variations in intrarater and interrater agreement while scoring both individual components and total score [5, 6, 7, 8, 9, 10, 11]. Recently, the Perth‐Rotterdam Annotated Grid Morphometric Analysis (PRAGMA) method was introduced which uses algorithms to analyze making it even more difficult for clinicians to interpret on their own [12, 13, 14].

Current scoring systems were developed to accurately identify and quantify common abnormalities in chest CT scans such as bronchiectasis, mucous plugging, airway wall thickness, etc. They are successfully fulfilling their objective but in search of perfection they have become so complex that they are very difficult to interpret, which has increased variability in interpretations even by experienced radiologists. These techniques and complexities were used to make these scoring systems more sensitive, but currently these scoring systems are rarely used in clinical care or investigator driven clinical research because they require thoracic radiologists to interpret [11].

There is a need to develop a simple to use chest CT scoring system which can be used in day‐to‐day monitoring of lung disease in both children and adults and can quantify change in lung disease with time. With such abilities, this new system can be used for clinical research purposes to quantify impact of therapies over time, as a surrogate outcome measure and for patient care.

The objective of this study is to develop a user friendly simple but effective CT based scoring system (Sheikh scoring system) to fulfill these needs.

2. Experimental Design/Methods

2.1. Study Population

This study obtained ethics approval from the local institutional review board (IRB) of Nationwide Children′s Hospital, Columbus Ohio [#00003633]. Informed consent was waived by IRB as risk level was no more than minimal risk. Our study group was 30 people with cystic fibrosis (pwCF) between 12 and 60 years of age who had completed 1 year of elexacaftor/tezacaftor/ivacaftor (ETI) therapy for the treatment of cystic fibrosis (CF) at a tertiary care children′s hospital between October 2019 and September 2022. These pwCF had a diagnosis of chronic lung disease and had chest CT done before initiation of ETI and at 1‐year of ETI therapy (total 60 chest CT scans). These were regular chest CT scans with only inspiratory films without breath holding and without expiratory films. These patients were divided into three groups of ten based on severity of underlying lung disease at baseline as determined by their ppFEV₁ (10 in each group). Group 1 had milder disease (ppFEV₁ ≥ 80%), group 2 had moderate disease (ppFEV₁ 50%–79%) and group 3 had severe disease (ppFEV₁ < 50%). This subdivision was done for subgroup analysis and to correlate different severity of lung disease as measured by ppFEV₁ with chest CT findings. Chest CTs were scored using our new scoring system (Sheikh scoring system) by three pulmonary clinicians independently and results were compared for interrater and intrarater agreements. Interpreters varied in their clinical experience ranging from 2 to 25 years. All three interpreters re‐scored CT scans after 4–6 weeks on 10 of the randomly selected patients for intrarater agreement. These physicians were blinded to clinical history except that these were pwCF receiving ETI therapy. In each pwCF, chest CT before and at 1‐year of ETI therapy were labelled as a and b. At first, baseline CT scan (a) (done at initiation of ETI) was scored separately and then follow‐up CT (b) (done after 1‐year of ETI therapy) was scored comparing with baseline CT to quantify changes.

Demographic data were collected from the electronic medical record (EMR) at baseline (at or before the time of initiation of ETI/chest CT) and included age, gender, race, and CF mutations. Clinical data and pulmonary function tests (ppFEV₁) were collected at initiation of ETI therapy and at 1‐year of CFTR modulator therapy. Bronchiectasis was defined as the diameter of the bronchus greater than the accompanying pulmonary artery, and airways were abnormally widened compared to other airways [15, 16]. Mucus plugging was defined as the presence of bronchi with opacification of the lumen. Bronchial wall thickening was subjectively assessed based on comparison with other bronchi throughout the lung and defined as a wall thickness greater than 20% of the airway diameter [16]. Air trapping was defined as low attenuation areas of increased air content compared to other areas of the lung.

2.2. Imaging Protocol

Chest CT exams were done using a 320‐detector row CT scanner (Aquilion ONE; Canon Medical Systems, Otawara, Japan). CT technique was determined using a weight based low dose protocol ranging from 80 to 100 Kvp with CT dose index levels ranging between 0.5 and 4.8. Images were done during a single breath hold from the thoracic inlet through the bottom of the diaphragm. A helical volume acquisition was obtained with a single breath hold and images reconstructed to 1 mm using a bone algorithm. No sedation or contrast was used. Chest CTs were analyzed with a visual scoring system. No automated tools were used.

2.3. Chest CT Sections and Components

Lungs were divided into six sections (three sections on each side corresponding to each lobe, namely right upper lobe, right middle lobe, right lower lobe, left upper lobe, lingula and left lower lobe. Five components were measured in each section: bronchiectasis (BEC), bronchial wall thickening (BWT), mucous plugging (MP), air trapping (AT) and parenchymal lung disease (PLD) (Table 1). Chest CT at baseline and after 1‐year of ETI therapy were compared to quantify change with therapy, if any, over time. Progression and regression of disease was measured by number of lobes involved and change in severity with time.

Table 1.

Sheikh scoring system.

		Right lung			Left lung
	Components	Upper lobe	Middle lobe	Lower lobe	Upper lobe	Lingula	Lower lobe
1	Bronchiectasis (BEC)
2	Bronchial wall thickening (BWT)
3	Mucous plugging (MP)
4	Air trapping (AT)
5	Parenchymal lung disease (PLD)

Note: Baseline CT scans: maximum score/cell is 2 and maximum total score is 60. Follow‐up CT scans: maximum score/cell is 3 with maximum total score of 90.

2.4. Severity and Extent

Each lobe was evaluated separately. Absence=0 and presence=2 points for BEC in each lobe, with 12 total points. Same criteria for BWT, MP, AT and PLD, with highest score of 12 × 5 = 60.

2.5. Follow‐up CT Scan/Comparison

For comparison, both initial (a) and follow‐up CT scan (b) were read side by side to define change in any of the five components in every lobe:

In follow‐up CT, each component in each lobe was scored as follows:

• a.If any lobe/component in initial CT was normal: if follow‐up CT was also normal, score was 0; if any lobe/component in follow‐up CT was abnormal, then score for each abnormal lobe/component was 2.
• b.If any lobe/component in initial CT was abnormal: if follow‐up CT was also normal (resolution), then score was 0; if in follow‐up CT there was no change, then score was 2; if in follow‐up CT any lobe/component worsened, then score was 3; and if in follow‐up CT abnormal lobe/component was improving, then score was 1.

Thus, each lobe with new finding of any of these five components was given 2 points for every new finding in each lobe. Thus, for all five components, maximum score of a baseline CT scan was 60 points (6 lobes × 5 components = 30 × 2 points for each component) whereas in a follow‐up CT the maximum was 90 points (6 lobes × 5 components = 30 × 3 points for worsening of each component) thus change, if any, was detected.

2.6. Inclusion and Exclusion Criteria

2.6.1. Inclusion Criteria

pwCF 12–60 years of age with CF related lung disease who had completed ETI therapy for 1 year and had chest CT within 1‐month of initiation (baseline) and within 1‐month of completing 1‐year therapy with ETI.

2.6.2. Exclusion Criteria

Patients without the diagnosis of CF, and patients with non‐CF lung disease were excluded.

3. Statistical Analysis

Sample size estimates were based on a two‐sided, 95% confidence interval (CI) for intraclass correlation coefficient (ICC) for inter‐rater reliability, using a two‐way random effects ANOVA model. For all calculations, a sample ICC of 0.8 was assumed, which represents excellent concordance. With three raters, 18 subjects were required to achieve a CI of width ≤ 0.3, or 37 subjects to achieve a CI of width ≤ 0.2. Thirty pwCF were ultimately included. All sample size calculations were performed using PASS software version 20.0 (NCSS, LLC., Kaysville, UT).

Variables were summarized using frequency (proportion) for categorical variables or median (interquartile range [IQR]) for continuous variables. All ratings were complete in the final data set. Chest CT scores were visualized at baseline, 1‐year follow‐up, and as a combined baseline/follow‐up score using violin plots.

The strength of interrater agreement was measured between three reviewers using ICC because total chest CT, BEC, BWT, AT, MP, and PLD scores were on ordinal scales [17]; kappa statistics were not appropriate for composite score outcomes because they were not dichotomous/nominal variables after combining scores from lobes and lung sides. Specifically, a two‐way random‐effects ICC model was applied. Interrater agreement was calculated for the baseline CT, 1‐year follow‐up CT, and baseline and 1‐year CTs combined. ICC estimates were interpreted as poor agreement for 0‐0.49, moderate for 0.5–0.75, good for 0.76–0.9, and excellent for 0.91–1. ICC was calculated among the group of three reviewers and pairwise between reviewers. When there were very unequal distributions across a few ICC score values or when all ratings were identical, we presented proportion of rater agreement instead of ICC [18]. Intrarater reliability was computed using similar ICC methods for the chest CT components and total score for each reviewer.

To summarize chest CT scores, mean scores were computed among the three reviewers. Spearman correlations between baseline ppFEV₁ and total score, BEC, BWT, AT, MP, and PLD for the baseline, 1‐year follow‐up, and combined chest CTs among all patients. Comparisons of chest CT scores at baseline, 1‐year follow‐up, and the change from baseline to 1‐year were conducted using Wilcoxon signed rank tests in the total cohort and by baseline disease severity subgroup. Changes from baseline to 1‐year follow‐up were evaluated for the total score and five components using the Wilcoxon signed rank test in the total cohort and by disease severity subgroup. All statistical analyses were performed in R version 4.4 (R Core Team, Vienna, Austria) with reproducible programming in R Markdown.

4. Results

4.1. Demographics

The cohort included 30 pwCF with a median (IQR) age of 18 (14, 27) years, and a slightly higher percentage of females (n = 17, 57%). All pwCF were Caucasian. Among them, twenty‐one pwCF (70%) were homozygous for F508del and 8 (27%) were heterozygous for F508del and one had another genotype. All received ETI for 1 year. Baseline mean (SD) ppFEV₁ in the total cohort was 67% (22). Among pwCF, 10 (33%) had mild disease (ppFEV_I ≥ 80%), 10 had moderate lung disease (ppFEV₁ between 50‐ < 80%) and remaining 10 had severe lung disease (ppFEV₁ ≤ 50%).

4.2. Interrater Reliability

Interrater reliability between three raters, as determined by ICC scores in the total cohort, was good. At baseline, total score ICC (95% CI) agreement was 0.83 (0.72, 0.91), at 1‐year follow‐up it was 0.72 (0.56, 0.85) and in combined (both baseline and 1‐year follow‐up CT scans, total of 60 CT scans) it was 0.83 (0.72, 0.91) (p < 0.001 in all). For individual components, combined score in all CT scans, both BEC and MP ICC revealed good interrater agreement of 0.83 (0.72, 0.91) for both while PLD and BWT had moderate interrater agreement 0.7 (0.54, 0.83) and 0.56 (0.35, 0.74) respectively. Air trapping had poor interrater agreement 0.44 (0.22, 0.65) (Table 2). ICC agreement between reviewers was compared and in all CT scans, agreement was either good or excellent. With individual variables, similar patterns were noted with the best agreement between reviewers on bronchiectasis and mucus plugging (between good and excellent) and lowest agreement on air trapping (Table 3).

Table 2.

Interrater agreement.

	Baseline		1‐Year		Combined
Scorea	ICC (CI)b	p valuec	ICC (CI)b	p valuec	ICC (CI)b	p valuec
Bronchiectasis (BEC)	0.88 (0.8, 0.94)	< 0.001	0.72 (0.56, 0.85)	< 0.001	0.83 (0.72, 0.91)	< 0.001
Bronchial wall thickening BWT)	0.54 (0.32, 0.72)	< 0.001	0.42 (0.19, 0.63)	< 0.001	0.56 (0.35, 0.74)	< 0.001
Air trapping/mosaic (AT)	0.35 (0.13, 0.58)	< 0.001	0.37 (0.14, 0.59)	< 0.001	0.44 (0.22, 0.65)	< 0.001
Mucous plugging (MP)	0.83 (0.71, 0.91)	< 0.001	0.68 (0.5, 0.82)	< 0.001	0.83 (0.72, 0.91)	< 0.001
Parenchymal lung disease (PLD)	0.68 (0.5, 0.82)	< 0.001	0.67 (0.49, 0.81)	< 0.001	0.7 (0.54, 0.83)	< 0.001
Total	0.83 (0.72, 0.91)	< 0.001	0.72 (0.56, 0.85)	< 0.001	0.83 (0.72, 0.91)	< 0.001

^a 30 pwCF and 3 raters.

^b ICC interpretation: poor, 0–0.49; moderate, 0.50–0.75; good, 0.76–0.90; excellent, 0.91–1.00.

^c ICC p values < 0.05 indicate the ICC between raters is significantly greater than zero.

Table 3.

Interrater agreement among three reviewers.

	Baseline						1‐Year						Combined
	1&2		2&3		1&3		1&2		2&3		1&3		1&2		2&3		1&3
Reviewers Scorea	ICC (CI)b	p valuec	ICC (CI)b	p valuec	ICC (CI)b	p valuec	ICC (CI)b	p valuec	ICC (CI)b	p valuec	ICC (CI)b	p valuec	ICC (CI)b	p valuec	ICC (CI)b	p valuec	ICC (CI)b	p valuec
Bronchiectasis (BEC)	0.96 (0.92, 0.98)	< 0.001	0.88 (0.76, 0.94)	< 0.001	0.8 (0.62, 0.9)	< 0.001	0.97 (0.94, 0.98)	< 0.001	0.63 (0.36, 0.81)	< 0.001	0.51 (0.19, 0.74)	0.002	0.98 (0.95, 0.99)	< 0.001	0.78 (0.59, 0.89)	< 0.001	0.71 (0.47, 0.85)	< 0.001
Bronchial wall thickening (BWT)	0.77 (0.57, 0.88)	< 0.001	0.69 (0.44, 0.84)	< 0.001	0.24 (−0.12, 0.55)	0.094	0.9 (0.8, 0.95)	< 0.001	0.35 (0, 0.63)	0.026	0.13 (−0.24, 0.46)	0.251	0.85 (0.71, 0.93)	< 0.001	0.59 (0.3, 0.78)	< 0.001	0.28 (−0.09, 0.58)	0.065
Air trapping/mosaic (AT)	0.82 (0.65, 0.91)	< 0.001	0.24 (−0.12, 0.55)	0.093	−0.03 (−0.39, 0.32)	0.574	0.92 (0.84, 0.96)	< 0.001	0.07 (−0.29, 0.41)	0.356	−0.03 (−0.38, 0.33)	0.565	0.89 (0.78, 0.94)	< 0.001	0.23 (−0.14, 0.54)	0.11	0.05 (−0.31, 0.39)	0.402
Mucous plugging (MP)	0.91 (0.82, 0.96)	< 0.001	0.84 (0.69, 0.92)	< 0.001	0.74 (0.52, 0.87)	< 0.001	0.94 (0.88, 0.97)	< 0.001	0.6 (0.31, 0.79)	< 0.001	0.51 (0.19, 0.73)	0.002	0.96 (0.92, 0.98)	< 0.001	0.78 (0.59, 0.89)	< 0.001	0.75 (0.54, 0.87)	< 0.001
Parenchymal lung disease (PLD)	0.88 (0.77, 0.94)	< 0.001	0.69 (0.44, 0.84)	< 0.001	0.47 (0.14, 0.71)	0.004	0.91 (0.82, 0.96)	< 0.001	0.61 (0.33, 0.79)	< 0.001	0.52 (0.2, 0.74)	0.001	0.92 (0.85, 0.96)	< 0.001	0.7 (0.45, 0.84)	< 0.001	0.5 (0.18, 0.73)	0.002
Total	0.94 (0.87, 0.97)	< 0.001	0.85 (0.7, 0.92)	< 0.001	0.71 (0.47, 0.85)	< 0.001	0.98 (0.95, 0.99)	< 0.001	0.62 (0.34, 0.8)	< 0.001	0.54 (0.23, 0.75)	< 0.001	0.97 (0.94, 0.99)	< 0.001	0.79 (0.6, 0.89)	< 0.001	0.71 (0.48, 0.85)	< 0.001

^a 30 pwCF and 2 raters.

^b ICC interpretation: poor, 0–0.49; moderate, 0.50–0.75; good, 0.76–0.90; excellent, 0.91–1.00.

^c ICC p values < 0.05 indicate the ICC between raters is significantly greater than zero.

4.3. Intrarater Reliability

Intrarater reliability, done 4–6 weeks apart on 10 pwCF (20 CT scans) as determined by ICC scores in combined total cohort was excellent ranging from 0.92 to 0.98 in all three reviewers. Agreement at baseline was between moderate and excellent, ranging from 0.62 to 0.99 and at follow‐up CT was excellent, ranging from 0.95 to 1 (Table 4). In individual variables, in combined group, ICC agreement was excellent for BEC and good to excellent for PLD and MP and moderate for BWT. For AT, two reviewers had moderate to excellent agreement while one reviewer had poor intrarater agreement (Table 4).

Table 4.

Intrarater reliability.

Timeb	Reviewer	ICC (CI) BECc	p value d	ICC (CI) BWTc	p value d	ICC (CI) ATc	p value d	ICC (CI) MPc	p value d	ICC (CI) PLDc	p value d	ICC (CI) Totalc	p value d
Baseline	1	1a	NA	1a	NA	0.72 (0.21, 0.92)	0.006	1a	NA	1a	NA	0.99 (0.95, 1)	< 0.001
Baseline	2	1a	NA	1a	NA	0.89 (0.63, 0.97)	< 0.001	0.9a	NA	0.46 (−0.2, 0.83)	0.078	0.94 (0.79, 0.99)	< 0.001
Baseline	3	1a	NA	0.9a	NA	0.58 (−0.03, 0.88)	0.03	0.6a	NA	0.14 (−0.5, 0.69)	0.336	0.62 (0.03, 0.89)	0.021
1‐Year follow‐up	1	0.91 (0.69, 0.98)	< 0.001	0.99 (0.95, 1)	< 0.001	0.74 (0.26, 0.93)	0.004	0.84 (0.48, 0.96)	< 0.001	0.88 (0.6, 0.97)	< 0.001	0.96 (0.86, 0.99)	< 0.001
1‐Year follow‐up	2	0.94 (0.79, 0.99)	< 0.001	0.98 (0.93, 1)	< 0.001	0.99 (0.97, 1)	< 0.001	1a	NA	0.98 (0.92, 0.99)	< 0.001	1 (0.99, 1)	< 0.001
1‐Year follow‐up	3	0.9a	NA	0.63 (0.04, 0.89)	0.019	0.21 (−0.45, 0.72)	0.264	0.86 (0.53, 0.96)	< 0.001	0.88 (0.58, 0.97)	< 0.001	0.95 (0.8, 0.99)	< 0.001
Combined	1	0.98 (0.91, 0.99)	< 0.001	1 (0.99, 1)	< 0.001	0.71 (0.19, 0.92)	0.007	0.97 (0.88, 0.99)	< 0.001	0.98 (0.94, 1)	< 0.001	0.98 (0.93, 1)	< 0.001
Combined	2	0.94 (0.79, 0.99)	< 0.001	0.98 (0.93, 1)	< 0.001	0.94 (0.79, 0.99)	< 0.001	0.98 (0.93, 1)	< 0.001	0.75 (0.26, 0.93)	0.004	0.98 (0.93, 1)	< 0.001
Combined	3	0.9a	NA	0.63 (0.04, 0.89)	0.019	0.41 (−0.25, 0.81)	0.105	0.75 (0.28, 0.93)	0.004	0.52 (−0.11, 0.86)	0.049	0.92 (0.72, 0.98)	< 0.001

Abbreviations: AT, air trapping; BEC, bronchiectasis; BWT, bronchial wall thickening; MP, mucous plugging; PLD, parenchymal lung disease.

^a Proportion agreement instead of ICC.

^b 10 pwCF and 2 repeated ratings.

^c ICC interpretation: poor, 0–0.49; moderate: 0.50–0.75; good, 0.76–0.90; excellent: 0.91–1.00.

^d ICC p values < 0.05 indicate the intrarater agreement is significantly greater than zero.

4.4. Correlation of FEV₁ With Chest CT Scores

Baseline ppFEV₁ correlated well with total mean chest CT scores among three reviewers at baseline with spearman correlation of −0.68, at 1‐year follow‐up −0.68, and in combined total scores −0.75 (p < 0.001 at all time points). Similarly for individual components, in combined scores, the strongest correlations were noted with BEC (−0.71), PLD (−0.73) and MP (−0.68) (Table 5).

Table 5.

Correlation of FEV₁ with chest CT scores.

Chest CT	Totala	BECa	BWTa	ATa	MPa	PLDa
Baseline	−0.68 (< 0.001)	−0.59 (< 0.001)	−0.57 (0.001)	−0.49 (0.006)	−0.6 (< 0.001)	−0.78 (< 0.001)
1‐Year follow‐up	−0.68 (< 0.001)	−0.7 (< 0.001)	−0.45 (0.013)	−0.45 (0.012)	−0.66 (< 0.001)	−0.67 (< 0.001)
Combination	−0.75 (< 0.001)	−0.71 (< 0.001)	−0.44 (0.015)	−0.49 (0.006)	−0.68 (< 0.001)	−0.73 (< 0.001)

Abbreviations: AT, air trapping; BEC, bronchiectasis; BWT, bronchial wall thickening; MP, mucous plugging; PLD, parenchymal lung disease.

^a Spearman correlation coefficient (p value) between mean chest CT score among 3 reviewers and baseline ppFEV₁.

4.5. Change in Chest CT Scores With 1‐Year of ETI Therapy

In our study group, our scoring system was able to detect significant improvement after 1‐year therapy with ETI and median (IQR) total scores improved from 49 (42, 54) to 23.5 (18, 31.7) (p < 0.001) and improvement was also noted in all three subgroups based on disease severity. Among those with ppFEV₁ ≤ 50%, median (IQR) total scores improved from 54.3 (48, 56.7) to 29.7 (23, 35.7) (p = 0.004), among those with ppFEV₁ between 50% and 79% median (IQR) scores improved from 52.3 (49.3, 54) to 29.2 (23.3, 30.7) (p = 0.002), and among those with baseline ppFEV₁ of ≥ 80% median (IQR) total scores improved from 24.7 (15.3, 47.3) to 7.8 (3.7, 18.0) (p = 0.002) (Table 6). Similarly, significant improvement was noted in the total group in all individual components (BEC, BWT, MP, AT, and PLD) with 1‐year therapy with ETI (Table 6). Significant improvement was also noted in all components in all these groups with ETI therapy except in BEC among those with ppFEV₁ ≤ 50% and PLD among those with ppFEV₁ ≥ 80% (Table 6).

Table 6.

Change in chest CT scores with one‐year of ETI therapy by disease severity.

	Total cohort			Baseline ppFEV1 ≤ 50			Baseline ppFEV1 > 50‐ < 80			Baseline ppFEV1 ≥ 80
Characteristic	Baseline N = 30a	1‐Year follow‐up N = 30a	p valueb	Baseline N = 10a	1‐Year follow‐up N = 10a	p valueb	Baseline N = 10a	1‐Year follow‐up N = 10a	p valueb	Baseline N = 10a	1‐Year follow‐up N = 10a	p valueb
BEC	12 (11.3, 12)	10.5 (8, 12)	< 0.001	12 (12, 12)	11.8 (10, 12)	0.059	12 (12, 12)	11.2 (10.3, 12)	0.022	9.7 (3.3, 12)	3.8 (2.7, 8.7)	0.009
BWT	12 (11.3, 12)	5.3 (3.3, 6)	< 0.001	12 (12, 12)	5.7 (3.7, 7.3)	0.006	12 (12, 12)	5.8 (5, 6)	0.006	9 (6.7, 12)	2.7 (1.3, 6)	0.006
MP	11.7 (8.7, 12)	3.3 (1, 6)	< 0.001	12 (10.7, 12)	4.7 (3.7, 7)	0.008	12 (9.3, 12)	4.3 (2, 6)	0.006	5.3 (0.7, 10.7)	0.3 (0, 1.7)	0.018
AT	7 (2.7, 8)	3 (0, 3.7)	< 0.001	8 (5.3, 8.7)	3.7 (0.7, 4)	0.013	7.7 (6.7, 8)	3 (2.7, 3.7)	0.006	2.7 (0.7, 6.7)	0 (0, 2)	0.009
PLD	8.3 (4.7, 12)	3.5 (0.7, 5.3)	< 0.001	12 (8, 12)	5.5 (3.7, 6.3)	0.010	8.7 (8.7, 10.7)	3.7 (2.3, 5)	0.008	0.7 (0, 4.7)	0 (0, 1.3)	0.11
Total	49 (42, 54)	23.5 (18, 31.7)	< 0.001	54.3 (48, 56.7)	29.7 (23, 35.7)	0.004	52.3 (49.3, 54)	29.2 (23.3, 30.7)	0.002	24.7 (15.3, 47.3)	7.8 (3.7, 18)	0.002

Note: To minimize each parameter score, we used the mean score among the 3 reviewers.

Abbreviations: AT, air trapping; BEC, bronchiectasis; BWT, bronchial wall thickening; MP, mucous plugging; PLD, parenchymal lung disease.

^a Median (Q1, Q3).

^b Wilcoxon signed rank test.

5. Discussion

This study validated good interrater and intrarater agreements using the new scoring system not only in total score but also in individual component scores. This scoring system was able to detect the extent and severity of lung disease and was also able to detect change with ETI therapy among the total study group and those with mild, moderate and severe lung disease. In pwCF with severe lung disease, our scoring system was able to detect meaningful change with ETI therapy in BWT, MP, PLD and AT but not in BEC. In pwCF with mild or moderate lung disease, our scoring system was able to detect meaningful change in BEC, BWT, MP and AT. PLD was very minimal at baseline in those with mild lung disease and no significant change was noted after 1‐year therapy. Unlike currently available scoring systems, ours does not use complex algorithms. Our scoring system showed good correlation between chest CT scores and ppFEV₁ and the ability to quantify change with therapy and can be used to quantify change with time.

Our scoring system is different from others. The Bhalla [2] scoring system used total score from an aggregate of nine components and divided CT into eighteen bronchopulmonary segments and score was based on labelling each component as mild, moderate, or severe and scored from 1 to 3, depending on the number of generations of bronchial divisions involved and then subtracting the CT score from the total possible score to determine the patient score. The Brody scoring system [4, 5] requires high resolution CT (HRCT) both inspiratory and expiratory films, which may require sedation and even mechanical ventilation in younger patients, and it measured five components in six lobes and determined extent by score (1‐3) in each lobe by labelling each of the five component/abnormalities by disease involvement up to 1/3rd, 1/3‐2/3rd and > 2/3rd of the lobe. Central and peripheral areas were measured separately. Severity was also measured using a multiplier for components which range from 2 to 5 levels. Scores of all the lobes were added to produce a total score which could be as high as 243. In these scoring systems, each segment/lobe disease severity/extent for each component is divided into three levels (< 1/3rd or 1/3‐2/3rd or > 2/3rd of each lobe). Thus, patients with 5% or 25% of lobe involved will still have the same score for that component. The same is also true in patients with involvement from 40% to 60% or 70%–100% of each component in each lobe. Because different levels of disease severity will have the same numerical severity, true disease severity is not evident.

More recently, in 2015 the Perth‐Rotterdam Annotated Grid Morphometric Analysis (PRAGMA) method was introduced by Rosenow [12] and his team including radiologists and computer scientists specifically targeting early CF lung disease in infants and young children with mild disease. It is a quantitative CT analysis method to measure percent of disease. It requires both inspiratory and expiratory films and is usually done under general anesthesia. In this protocol, a square grid is overlaid on 10 equidistant axial slices between lung apex and base. Grid cell size is about one‐twentieth of the lung width at the carina, measured with electronic clippers. Grid cells are annotated in a hierarchical manner and the primary outcomes from this method are the volume proportion of the lung with airway disease (% Dis), determined by dividing the number of cells annotated with either BEC, MP, AWT by the total number annotated cells (excluding atelectasis) in inspiratory films and AT in expiratory films [12, 13, 14], making it even more difficult for clinicians to use.

Scoring systems are becoming more complicated in search of perfection, but by doing so, they are more susceptible to errors and increasingly less user friendly. Our scoring system tries to minimize these issues by making scoring simpler by quantifying disease severity as presence or absence of five components/abnormalities in each lobe. Disease worsening is defined as either spread to other lobes or increase in extent of disease within each lobe on comparison CT. Other CT scoring systems define change with increase in size/spread of the disease, but if disease is more severe but still localized to same area then change in severity cannot be quantified by the current scoring systems. In our system, for change, we compare with previous CT scan and change is measured both by spread to other lobes or change in severity within each lobe. If disease is increased within the same area, it can still be measured as increased disease even if it did not spread within the lobe or to the other lobes. Our scoring system is simpler but equally effective in quantifying disease and detecting change with time with good correlation with ppFEV₁.

Previous studies looked at both interrater and intrarater agreements using many scoring systems including the Bhalla and Brody scoring systems and revealed marked variabilities in both cumulative (total) scores and among individual components [5, 6, 7, 8, 9, 10]. In our study using our scoring system both interrater and intrarater agreements were either similar or better when compared with currently available scoring systems.

Bronchiectasis is most characteristic of CF lung disease and many scoring systems have shown consistent interrater agreement in many studies [4, 5, 6, 7, 8, 9, 10] as it can be detected easily and can be used as a reliable marker of progression of lung disease on CT scans and can be seen on CT scans even before any worsening in FEV₁. Studies have shown that advanced BEC may be irreversible in CF [19, 20, 21] thus it is not clear if it can be used as a marker for disease reversibility in CF. Other CT components in CF such as BWT, MP, and AT are precursors/risk factors. Advanced BEC is end‐stage of the disease process, thus improvement in other components and even improvement in early BEC may be indicative of reversal of CF lung disease. Changes in these components may be possible before advanced bronchiectasis takes place thus these components reflecting early lung disease may be used as surrogate markers for improvement over time. Thus early lung disease may be reversible especially MP, and BWT and in both children and adults advanced BEC may be less reversable [20, 21, 22, 23, 24, 25]. Studies have shown marked improvement in MP, BWT and AT with ETI therapy though improvement in BEC was limited [20, 21].

Studies have shown that though lung function and chest CT measure different pathologies with different sensitivities, they have some correlation. Brody et al. [4] revealed the correlation coefficient between ppFEV₁ and overall CT scores to be −0.46 while with BEC it was −0.37 and with AT correlation was −0.38 [4]. In another study, de Jong et al. [8] compared correlation between five CT scoring systems and ppFEV₁ in 25 pwCF and noted correlation between total CT scores and ppFEV₁ ranging from −0.69 to −0.73 (Brody scoring system: −0.69, Bhalla scoring system: −0.73). In our scoring system, correlation between total chest CT score and ppFEV₁ was −0.75 (p < 0.001) and there was good correlation with BEC −0.71 (p < 0.001) and mucus plugging −0.68 (p < 0.001). Correlation was weakest with BWT −0.44 (p = 0.015) and with AT it was −0.49 (p = 0.006) even though these were all inspiratory films and air trapping is better defined in expiratory films.

Our study has many limitations. Our study cohort is of adolescents and young adults but lack younger children. Thus efficacy of our scoring system in younger children with very mild disease is yet to be determined. This is a single center study. More studies with different geographic areas and those with milder disease and younger age need to be done. We correlated our findings with ppFEV₁ which is the gold standard clinical parameter, but it is possible that newer testing measures, such as Lung Clearance Index (LCI) may have even more robust correlations especially in children with milder disease.

One of the perceived limitations of our scoring system might be that parameters in the initial CT are scored using a binary system, as are the parameters in the follow‐up CT if the initial CT is normal. We acknowledge that our scoring system has reduced granularity, especially for initial CT for which each parameter can receive only seven possible scores across the lobes (0, 2, 4, 6, 8, 10, 12). Having said that, if initial CT is abnormal, then the follow‐up CT parameters are evaluated on an ordinal scale from 0 to 3 points. Each parameter can therefore receive 18 possible scores across the lobes (1, 2, … 18), so there is much greater granularity in the follow‐up CT. We acknowledge that like other scoring systems our scoring system is also designed to assess change between two time points. Scores will always be relative to a baseline/reference CT scan, as the follow‐up scan is evaluated in comparison to the initial scan. Having said that, our scoring system can be used for longitudinal tracking of the disease. It can be done in one of the two ways.

• 1.When applying our scoring system to three time points, for example (initial CT scan, follow‐up CT scan 1, follow‐up CT scan 2), initial CT scan can be used as the reference for both follow‐up time points to ensure fair scoring between follow‐up CTs. We acknowledge that this may bias further change (from follow‐up 1 to follow‐up 2) toward the null in longitudinal analyses; for example, bronchiectasis in the right upper lobe of lung may be abnormal for initial CT, improve at follow‐up CT 1 and further improve at follow‐up CT 2 but bronchiectasis scores for both follow‐up scans would be the same because they are relative to the initial scan. At any time, any follow‐up CT can also be compared with the initial CT (CT scan 1) to determine overall change over time.
• 2.When applying our scoring system to three time points, for example (initial CT scan (scan #1), follow‐up CT scan (scan #2), second follow‐up CT scan (scan 3). In this case, first initial CT scan can be used as the reference for 2nd CT scan and 2nd CT scans can be used as a reference for 3rd CT scan and on and on monitoring longitudinal scoring and disease progression or improvement over time. Comparing any 2 scans will be able to document change between the two thus will have different total score and different score for each component and/or lobe if there is any change (improvement or deterioration) between the two time points to document change over time. At any time point, any CT scan can also be compared with the initial CT scan (scan #1) to define change from the baseline.

The Sheikh system is most effectively applied when there are two defined CT scans in which to evaluate change. We hope users of our scoring system will thoughtfully consider the relative nature of our system when designing future studies and in their clinical applications. At any time, any follow‐up CT can also be compared with the initial CT (CT 1) to determine overall change over time.

CT scans do increase radiation exposure, thus repeated CT scanning to evaluate change with time do raise concern [19, 26]. If frequent CT scans are needed, the goal should be to keep the cumulative lifelong radiation exposure as low as possible. The radiation dose of the thin‐section CT scans can be up to 12 times that of a chest radiograph [27]. Attempts are being made to decrease radiation dose while maintaining quality of the scans. Low radiation dose CT scans are now available, which can produce acceptable images with very low radiation and some as low as a chest radiograph [28, 29, 30, 31, 32]. Low radiation dose CT scans and easy to use scoring systems which can not only identify and quantify lung disease but can also detect change in lung disease with either therapy or time can be the future of monitoring CF lung disease.

6. Conclusion

This study validated good interrater and intrarater agreements using our new scoring system not only in total score but also in individual component scores. Our scoring system can detect extent and severity of lung disease and is able to detect change with ETI therapy among the total cohort and among those with mild, moderate, and severe lung disease.

Author Contributions

Shahid Sheikh: conceptualization, investigation, writing – original draft, methodology, writing – review and editing, supervision, data curation, project administration. Mariah Eisner: methodology, validation, visualization, formal analysis, writing – review and editing, software. Melissa Holtzlander: conceptualization, investigation, writing – review and editing, methodology, project administration, supervision, data curation. Sehyr Imran: investigation, writing – review and editing, methodology, data curation. Alpa Patel: investigation, writing – review and editing, methodology, supervision. Karen McCoy: resources, supervision, data curation, project administration, writing – review and editing, visualization, methodology, investigation.

Conflicts of Interest

Dr. McCoy reports grants from 4D Molecular Therapeutics, AbbVie, Aridis Pharmaceuticals, Armata Pharmaceuticals, Boehringer Ingelheim, Cystic Fibrosis Foundation, Corbus Pharmaceuticals, Eloxx Pharmaceuticals, Insmed, Laurent Pharmaceuticals, Novoteris, Proteostasis Therapeutics, Savara, Translate Bio, Vertex Pharmaceuticals, all outside the submitted work. All the related funding is given to her institution of employment for the above‐listed entities to support research trials. She received no direct money, and none of these relationships are in conflict with this manuscript. The remaining authors declare no conflicts of interest and no relevant disclosures.

Data Availability Statement

Data sharing is not applicable to this article as no new data were created or analyzed in this study.