Inter- and intrascanner variability of pulmonary nodule volumetry on low-dose 64-row CT: an anthropomorphic phantom study

Abstract

Objective:

To assess inter- and intrascanner variability in volumetry of solid pulmonary nodules in an anthropomorphic thoracic phantom using low-dose CT.

Methods:

Five spherical solid artificial nodules [diameters 3, 5, 8, 10 and 12 mm; CT density +100 Hounsfield units (HU)] were randomly placed inside an anthropomorphic thoracic phantom in different combinations. The phantom was examined on two 64-row multidetector CT (64-MDCT) systems (CT-A and CT-B) from different vendors with a low-dose protocol. Each CT examination was performed three times. The CT examinations were evaluated twice by independent blinded observers. Nodule volume was semi-automatically measured by dedicated software. Interscanner variability was evaluated by Bland–Altman analysis and expressed as 95% confidence interval (CI) of relative differences. Intrascanner variability was expressed as 95% CI of relative variation from the mean.

Results:

No significant difference in CT-derived volume was found between CT-A and CT-B, except for the 3-mm nodules (p<0.05). The 95% CI of interscanner variability was within ±41.6%, ±18.2% and ±4.9% for 3, 5 and ≥8 mm nodules, respectively. The 95% CI of intrascanner variability was within ±28.6%, ±13.4% and ±2.6% for 3, 5 and ≥8 mm nodules, respectively.

Conclusion:

Different 64-MDCT scanners in low-dose settings yield good agreement in volumetry of artificial pulmonary nodules between 5 mm and 12 mm in diameter. Inter- and intrascanner variability decreases at a larger nodule size to a maximum of 4.9% for ≥8 mm nodules.

Advances in knowledge:

The commonly accepted cut-off of 25% to determine nodule growth has the potential to be reduced for ≥8 mm nodules. This offers the possibility of reducing the interval for repeated CT scans in lung cancer screenings.

Lung cancer is the primary cancer in males and the second most common cancer in females worldwide, causing 18% of the total number of deaths [1]. Many lung cancers are found at a relatively late stage, resulting in a 5-year survival of only 15% or less [2]. Low-dose CT is a promising screening method for early detection of lung cancer [3–7]. The first result indicates that CT lung cancer screening can reduce lung cancer-specific mortality [8].

In lung cancer screening, treatment decisions usually depend on pulmonary nodule size for the nodules at first detection and on the growth rate at follow-up [4]. Therefore, it is essential to assess the nodule size and growth rate accurately and reproducibly [9,10]. Variability has been found in CT-derived nodule size assessment [11,12]. In view of the current practice of patients frequently undergoing follow-up examinations, sometimes not on the same scanner, reliable inter- and intrascanner reproducibility of nodule volumetry is important.

However, previous studies reported inconsistent results regarding the reproducibility of nodule volumetry. Some in vitro studies have been performed in which artificial nodules were placed at known locations in a thoracic phantom without pulmonary vessels [13–15]. Some of these studies were based on older generation CT scanners [13,16]. These studies generally showed a small margin of variability in nodule volumetry for software from different vendors. On the other hand, in vivo studies have shown that variability can be considerable, with variability up to 25% for 15 to 500 mm³ nodules [11,17–19]. A study to investigate inter- and intrascanner variability under optimally controlled conditions, yet resembling human lungs, using a more realistic phantom, has not been performed. Nowadays, 64-row multidetector CT (64-MDCT) scanners are most commonly utilised, as well as in lung cancer screenings. The variability of nodule volumetry of these scanners impacts nodule management, e.g. the interval of repeated CT scanning. As an extension to our recent study on observer detection and accuracy of manual and semi-automated volumetry [10], the focus of this study is on reproducibility between and within 64-MDCT systems. We assessed the inter- and intrascanner variability of pulmonary nodule volumetry on low-dose 64-MDCT, using randomly placed solid nodules in an anthropomorphic thoracic phantom with a background of pulmonary vasculature.

MATERIALS AND METHODS

Phantom

This study was performed using an anthropomorphic thoracic phantom (Lungman; Kyoto Kagaku, Tokyo, Japan) with an artificial thoracic wall, heart, mediastinum, diaphragm and lungs with pulmonary vasculature (Figure 1). This phantom simulates an accurate life-sized anatomical model of a male thorax. Soft-tissue substitute materials were made of polyurethane resin composites. Synthetic bones were made of epoxy resin, with X-ray absorption rates close to those of human tissue. The space between the pulmonary vessels in the thoracic cavity contained air.

Figure 1.

External view (a) and internal artificial heart and lungs (b) of the anthropomorphic thoracic phantom.

Moreover, we used five commercially available artificial spherical pulmonary nodules with a smooth surface in five diameters (3, 5, 8, 10 and 12 mm, corresponding to a volume of 14, 65, 268, 524 and 905 mm³), made of polyurethane resin (Figure 2). CT density was +100 Hounsfield units (HU) at 120 kV.

Figure 2.

Artificial spherical pulmonary nodules with five different diameters (from left to right: 12, 10, 8, 5 and 3 mm, corresponding to volumes of 905, 524, 268, 65 and 14 mm³).

CT imaging

Two 64-MDCT systems (CT-A: Sensation 64; Siemens, Forchheim, Germany and CT-B: Brilliance 64; Philips, Best, Netherlands) were utilised. The low-dose CT acquisition protocol in the Dutch–Belgian Randomized Lung Cancer Screening Trial (NELSON) was applied [4]. The protocol for CT-A was spiral acquisition at 120 kV, 20 mAs, rotation time 0.5 s, pitch 1.5 and slice collimation 2×32×0.6 mm, field of view 300 mm; images were reconstructed at a slice thickness of 1 mm and a slice increment of 0.7 mm, using a medium-smooth B30f image reconstruction kernel. The protocol for CT-B was spiral acquisition at 120 kV, 20 mAs, rotation time 0.5 s, pitch 1.39 and slice collimation 64×0.625 mm, field of view 300 mm; images were reconstructed at a slice thickness of 1 mm and a slice increment of 0.7 mm, using a medium-smooth B kernel. Both scanners were routinely calibrated.

Artificial nodules were randomly positioned in the pulmonary vessels of the phantom. All nodules were firmly held in the vessels without using glue. To avoid the influence from thoracic walls on nodule segmentation, all nodules were attached to pulmonary vessels and none of the nodules was attached to pleura or positioned subpleurally. After randomly positioning a pre-determined set of one to three nodules in the artificial lungs, CT image acquisition was performed. 13 different combinations of nodules were set up so that each nodule was examined in total 5 times. The CT examination was repeated three times for each nodule configuration, with a small translocation (approximately 5–10 mm) and rotation (approximately 5–10°) of the phantom in between each examination to simulate participant movement. The thoracic phantom was also examined once without pulmonary nodules, serving as a control examination to exclude false-positive findings. Thus, per CT system, 40 examinations were performed, including 39 test examinations and 1 control examination.

Therefore, each nodule was examined 30 times (5 different locations, 3 repeating examinations and 2 CT systems). Furthermore, all nodules were examined in air, on the CT table, to confirm the visibility of the artificial pulmonary nodules on low-dose CT without the artificial thoracic background.

Quantitative image analysis

The reconstructed data were evaluated on a workstation (Leonardo; Siemens). Three observers participated in image analysis, including two radiologists specialised in thoracic diagnostic imaging with 8 years' experience (Observer A and Observer B) and one resident with 2 years' experience (Observer C). All the examinations were assessed twice separated by at least 1 month by all three observers. The data were presented in the same order for the double reading. Observer A evaluated the examinations of CT-A and CT-B. Observer B evaluated the examinations of CT-A. Observer C evaluated the examinations of CT-B. The three observers were blinded to information about the presence, properties and location of the artificial pulmonary nodules. Since each nodule was scanned 15 times on each CT system and measured twice, in total at most 30 paired evaluations for each nodule could be included for interscanner analysis. Also, 30 evaluations for each nodule could be included for intrascanner analysis.

The observers were asked to report whether there were pulmonary nodules present or not. If a potential nodule was observed, the images were compared with the images of the control CT examination without a nodule to confirm that it was not a false-positive finding caused by pulmonary background structures. If comparison with the control CT confirmed the presence of the pulmonary nodule, a dedicated commercial software tool (LungCARE™; Siemens) was utilised to semi-automatically measure the volume of the detected nodules. The accuracy of nodule volumetry of this software was shown to be very high [20].

In addition, image noise was assessed by measuring the standard deviation (SD) of the mean number of Hounsfield units in a circular region of interest (ROI) with a radius of 10 mm. The ROIs were placed in the artificial left ventricle, avoiding image artefact [21]. 10 examinations per CT scanner were randomly selected and 3 ROIs were measured in each examination.

Statistics

The normality of data was assessed by the Kolmogorov–Smirnov test. Influential effects on nodule volumetry were investigated using univariate analysis of a general linear model. The dependent variable was CT-derived volume. The fixed factors were CT systems (CT-A and CT-B) and observers (Observer A, Observer B and Observer C). Differences in CT-derived volumes between the two CT systems were evaluated by a paired sample t-test.

Besides interscanner variability, intermeasurement variability between two rounds of measurements was assessed. The measurements of Observer A were considered as the first round, and those of Observer B and Observer C were considered as the second round. Interscanner and intermeasurement reliability of CT-derived volume was expressed as an intraclass correlation coefficient (ICC). An ICC value >0.90 was considered as high agreement, in the range of 0.75–0.90 as moderate and <0.75 as low [22]. Interscanner and intermeasurement variability of CT-derived volume was explored using Bland–Altman plots. Relative difference was calculated as (a−b)/m₁×100%, in which a and b were CT-derived volumes on CT-A and CT-B, respectively, or CT-derived volumes on the first and second round of measurement. m₁ was the mean of a and b. The 95% confidence interval (CI) of relative difference was calculated as a mean relative difference ±1.96 SD.

Since multiple image acquisitions and measurements were performed for an individual nodule, intrascanner variability was expressed as a variation coefficient by calculating SD/m₂, where m₂ was the mean of 30 evaluations of each nodule [16]. Relative variation from the mean was calculated. The 95% CI of relative variation was calculated as (±1.96 SD)/m₂×100% [23]. A 95% CI <25% was considered as good agreement for inter- and intrascanner variability [17].

Results were given as mean ± SD. p<0.05 was considered as statistically significant. All statistical analyses were performed using SPSS® v. 20 (IBM Corp., Armonk, NY).

RESULTS

All the nodules were detected and measured by the two observers. No false-positive nodules were found. Representative CT images of the anthropomorphic thoracic phantom are shown in Figure 3. CT-derived volume for each artificial nodule conformed to a normal distribution (test for normality p>0.05).

Figure 3.

Axial (a), coronal (b) and sagital (c) images of the anthropomorphic thoracic phantom with a semi-automatically measured solid pulmonary nodule (d).

Volumetry

Univariate analysis showed that neither CT system nor observer was a significant factor influencing volumetry (p>0.05). No significant difference in CT-derived volume was found between CT-A and CT-B (p>0.05) except for the smallest 3 mm nodules (p<0.05). Semi-automated volumetry demonstrated underestimation from the actual volume. Overall, CT-derived volume was underestimated by 9.1±7.9% (p<0.001). The noise level in CT-A was 21.8±1.3, significantly lower than the 24.0±1.3 in CT-B (p<0.01).

Inter- and intrascanner variability

Interscanner reliability of CT-derived volume was high (ICC=1.000, p<0.001). Interscanner variability of CT-derived volume decreased at larger nodule size (Table 1 and Figure 4). Inter-scanner variability was low for nodules ≥8 mm in diameter in which the maxima of the 95% CI was 4.9%.

Table 1.

Interscanner variability of CT-derived volume for artificial spherical solid pulmonary nodules

Nodule characteristics		Relative difference
Diameter (mm)	Volume (mm3)	Mean (%)	95% CI (%)
3	14	16.7	−19.3, 41.6
5	65	7.2	−5.6, 18.2
8	268	0.9	−2.3, 4.0
10	524	2.2	−0.5, 4.9
12	905	1.3	−1.1, 3.7

CI, confidence interval; SD, standard deviation.

Relative difference was calculated as (volume on CT-A−volume on CT-B)/mean×100%. 95% CI was calculated as mean±1.96SD.

Figure 4.

Bland–Altman plots of the relative interscanner difference in semi-automated volumetry for solid (+100 HU) nodules in five sizes (3, 5, 8, 10 and 12 mm in diameter, corresponding to volumes of 14, 65, 268, 524 and 905 mm³) as measured on two CT systems, CT-A and CT-B, from two different vendors. The dotted lines indicate the 95% confidence interval. The grey zone indicates a relative difference of ±25%, which is commonly used to exclude systematic errors, thus to exclude a growing nodule on subsequent CT scans.

Intrascanner variability also decreased at larger nodule size (Table 2 and Figure 5). Intrascanner variability of CT-derived volume was also low for nodules ≥8 mm in diameter in which the maxima of the 95% CI was 2.6%.

Table 2.

Intrascanner variability of CT-derived volume for artificial spherical solid pulmonary nodules

Nodule characteristics		CT-A		CT-B
Diameter (mm)	Volume (mm3)	Variation coefficient (%)	95% CI of relative variation (%)	Variation coefficient (%)	95% CI of relative variation (%)
3	14	14.6	−28.6, 28.6	14.6	−28.6, 28.6
5	65	3.9	−7.6, 7.6	6.8	−13.4, 13.4
8	268	1.1	−2.2, 2.2	1.1	−2.2, 2.2
10	524	1.3	−2.6, 2.6	0.7	−1.4, 1.4
12	905	0.6	−1.3, 1.3	1.1	−2.2, 2.2

CI, confidence interval; SD, standard deviation.

Variation coefficient was calculated as SD/mean×100%. 95% CI of relative variations was calculated as (±1.96SD)/mean×100%.

Figure 5.

Grouped plots of the relative intrascanner variation in semi-automated volumetry for solid [+100 Hounsfield units (HU)] nodules in five sizes (3, 5, 8, 10 and 12 mm in diameter, corresponding to volumes of 14, 65, 268, 524 and 905 mm³) on CT-A (a) and CT-B (b). The grey zone indicates a relative difference of ±25%, which is commonly used to exclude systematic errors, thus to exclude a growing nodule on subsequent CT scans.

Intermeasurement variability

Intermeasurement reliability of CT-derived volume between two rounds of measurement was high (ICC=1.000, p<0.001). Intermeasurement variability of CT-derived volume decreased at larger nodule size (Table 3 and Figure 6). Intermeasurement variability was low for nodules ≥8 mm in diameter in which the maxima of the 95% CI was 3.2%.

Table 3.

Intermeasurement variability of CT-derived volume for artificial spherical solid pulmonary nodules

Nodule characteristics		Relative difference
Diameter (mm)	Volume (mm3)	Mean (%)	95% CI (%)
3	14	−3.5	−34.5, 27.5
5	65	−4.6	−17.0, 8.1
8	268	−0.1	−3.2, 3.1
10	524	−0.3	−3.0, 2.5
12	905	0.1	−2.7, 2.9

CI, confidence interval; SD, standard deviation.

Relative difference was calculated as (volume in the first round of measurement−volume in the second round)/mean×100%. 95% CI was calculated as mean±1.96 SD.

Figure 6.

Bland–Altman plots of the relative intermeasurement difference in semi-automated volumetry for solid [+100 Hounsfield units (HU)] nodules in five sizes (3, 5, 8, 10 and 12 mm in diameter, corresponding to volumes of 14, 65, 268, 524 and 905 mm³) as evaluated on two rounds of measurement (Observer A in the first round, Observers B and C in the second round). The dotted lines indicate the 95% confidence interval. The grey zone indicates a relative difference of ±25%, which is commonly used to exclude systematic errors, thus to exclude a growing nodule on subsequent CT scans.

DISCUSSION

In this anthropomorphic phantom study investigating inter- and intrascanner variability, intermeasurement variability of CT-derived nodule volume and variability of CT-derived volume decreased at increasing nodule size to a maximum of 4.9% for ≥8 mm nodules. Good inter- and intrascanner agreement was found for nodules ≥5 mm in diameter but not for 3 mm nodules.

One-dimensional measurement (diameter) of a target lesion is commonly utilised [24], yet accuracy and reproducibility of diameter measurement to assess pulmonary nodule size are low [10]. Also, discrepancies in manual assessment of lesion size can significantly influence therapeutic decisions [25]. By contrast, semi-automated volumetry using dedicated software shows better reproducibility than manual assessment of nodule size [26,27]. However, even semi-automated volumetry suffers from some variability [11,12]. It is well known that reproducibility of CT-derived volume of pulmonary nodules is influenced by a number of factors, such as CT equipment, nodule characteristics and pulmonary surroundings [9,11]. In view of the current practice of lung cancer screening in which multiple CT systems are used, management of systematic errors is essential to reach a reliable result.

Variability of CT-derived volume by the semi-automated method decreased with the larger nodule size. In previous in vivo studies, pulmonary nodules were commonly analysed as a whole, without assessing variability by nodule size categories [17,18]. Also, inconsistent results were reported about dependency of volumetry variability on nodule size. Rampinelli et al [19] found that volumetry for larger nodules was more reproducible, based on 83 nodules with a mean volume of 220±241 mm³, yet Marchianò et al [28] observed that variability was not affected by nodule volume based on 233 nodules of 99±127 mm³. In those in vivo studies, different volume ranges of included nodules have likely caused these inconsistent results. We performed the current study based on five discrete and well-defined sizes of nodules from 14 to 905 mm³, which is the common nodule volume range in lung cancer screening [4], and found that variability does depend on nodule size.

Currently, a threshold of 25% is commonly utilised as the minimal volume change for growing solid nodules [3]. That in vivo result was based on interscan variability observed in one CT system for nodules 15–500 mm³ [17]. Our results were based on two CT systems, which is closer to the practice of lung cancer screening with multiple CT systems involved. In our study, the maxima of 95% CI of variability was <18% for ≥5 mm (65 mm³) solid nodules and <5% for ≥8 mm (268 mm³) nodules. This suggests that the threshold of 25% can be used for solid nodules of ≥5 mm and is a conservative threshold. The threshold of 25% could potentially be decreased for larger pulmonary nodules of ≥8 mm. Because large variability was found for 3 mm nodules, the threshold of 25% is not likely to be usable for 3 mm nodules. However, the likelihood of <5 mm nodules to be malignant is very low [29].

Intermeasurement variability between two rounds of evaluation was assessed in this study instead of interobserver variability between two observers. In lung cancer screening, images are often read twice in the same examination, commonly by different paired observers. For example, in the NELSON trial, each participant was assessed twice in the central and the local site. Two radiologists with more than 6 years' experience performed the evaluation in the central site, and 13 radiologists or residents with variable experiences in the local sites [30]. To closely simulate this clinical practice, we chose one observer for the first round of measurement and two observers for the second measurement.

A significant volumetry difference was observed between CT-A and CT-B in 3 mm nodules but not in the larger nodules. This was caused by different properties between the scanners from both vendors, i.e. image acquisition and reconstruction parameters. Because volumetry analysis of smaller nodules is more affected by noise, we quantified the image noise in the images from both CT scanners. Higher image noise degrades nodule segmentation, thus leading to underestimated CT-derived volume [31]. Because CT-B had a higher noise level than CT-A, this could very well explain the observed lower CT-derived volume in CT-B than in CT-A for the 3 mm nodule.

Semi-automated volumetry systematically underestimated the nodule volume when using a low-dose CT acquisition protocol. This finding concords with previous studies based on the same phantom in three different CT scanners [10,32]. It is well known that accuracy of nodule volumetry depends on many factors, such as CT acquisition and reconstruction algorithm. De Jong et al [33] reported that low-dose CT yielded lower volumes than normal-dose CT. It is likely that our low-dose setting contributed to the systematic volume underestimation. However, given the systematic nature of this error, this is less important in evaluating the growth rate in lung cancer screening.

Clinical implications

Changes in nodule size that exceed measurement variability are important to determine the actual growth, which can impact treatment decision and estimation of therapy outcome [34,35]. In lung cancer screening trials, volumetry and diameter methods to assess the pulmonary nodule size and growth have been applied. Semi-automated volumetry was utilised in the NELSON trial [36]. Manual diameter measurement was utilised in some earlier launched trials, such as the National Lung Screening Trial [8]. Since the determination of nodule growth depends on the nodule size, a volumetry method with less variability can potentially provide more accurate and timely treatment decisions and a better outcome estimation. We found that semi-automated volumetry showed good inter- and intrascanner variability. Thus, the semi-automated method is reliable for pulmonary nodule assessment in lung cancer screening. Moreover, no difference was found between two CT systems from different vendors for ≥5 mm nodules. As the follow-up of screened participants can last for an extensive period, different CT systems from different vendors may be used. A direct comparison of nodule volumes obtained from different CT systems seems valid, at least for the CT systems investigated in this study.

To determine a growing nodule, inter- and intrascanner variability should be optimised to minimise systematic errors. This allows for the determination of a cut-off value to assess with confidence whether nodule volumes have actually increased. For an indeterminate nodule, lung cancer screening trials commonly use a follow-up CT of 3–4 months to evaluate the nodule growth, based on a minimum increase of 25% [6]. The threshold of 25% can likely be lowered for nodules of ≥8 mm in diameter when using semi-automated volumetry and 64-MDCT scanning, since the variability was only 4.9% in this study. If based on a minimum increase of 5% to determine growth, the follow-up interval can potentially be reduced to 3–4 weeks. This can also limit the time of participant anxiety.

Limitations

Firstly, although the anthropomorphic phantom more closely resembles the human thorax than previously published phantoms, degenerative changes in lung tissue were not present in the phantom. Fibrosis, emphysema and consolidations can influence the nodule volumetry. Additional patient factors were not present either, such as breathing variability, variable habitus and patient size. A phantom with these variable characteristics is difficult to make. Our phantom study represents the variability of nodule volumetry in a relatively optimal environment, although the variability might be higher in vivo. Also, only spherical nodules of five discrete sizes were used. Although pulmonary nodules of these sizes are common findings in lung cancer screening [4], the nodule shape is more often irregular and non-spherical in vivo, with a continuous size range. A study extension would therefore be variability assessment for irregular nodules.

Secondly, we evaluated only one CT acquisition protocol. That protocol has been widely applied in lung cancer screening [3]. However, variability of nodule volumetry depends on acquisition and algorithm in individual settings [7,35]. When discussing the results, we linked to the studies based on the same CT acquisition protocol [5,10,11,17]. Thus, our results are comparable to those previous data. A further extension would be the evaluation of different CT acquisition protocols.

CONCLUSIONS

Different 64-MDCT scanners in low-dose settings yield a similar volumetry of artificial pulmonary nodules between 5 mm and 12 mm in diameter. Inter- and intrascanner variability and intermeasurement variability of CT-derived volumes of spherical solid pulmonary nodules decrease with increasing nodule size. In lung cancer screening, the commonly accepted cut-off of 25% to determine the nodule growth has the potential to be reduced for nodules of ≥8 mm in diameter. This phantom study offers potential for reducing the interval for repeated CT scans in lung cancer screening.