Imprecision in Automated Volume Measurements of Pulmonary Nodules and Its Effect on the Level of Uncertainty in Volume Doubling Time Estimation

Abstract

Background

Detection of small indeterminate pulmonary nodules in clinical practice is increasing, largely because of increased utilization and improved imaging technology. Although there currently exists software for CT scan machines that automate nodule volume estimation, the imprecision associated with volume estimates is particularly poor for nodules ≤ 6 mm in diameter, with greater imprecision associated with increasing CT scan slice thickness. This study examined the effects of the volume estimation error associated with four CT scan slice thicknesses on estimates of volume doubling time for solid nodules of various sizes.

Methods

Data reflecting the accuracy of 1,624 automated volume estimations were obtained from experiments incorporating volume estimation software, performed on a commercially available lung phantom. These data informed mathematical simulations that were used to estimate imprecision around VDT estimates for hypothetical pairs of volume estimates for a given solid pulmonary nodule observed at different time points.

Results

The confidence intervals around the VDT estimates were extremely wide for 2.50- and 5.00-mm slice thicknesses, often encompassing values traditionally associated with both benignity and malignity for simulated 1- and 2-mm growths in diameter.

Conclusions

Because of the inaccuracy in automated volume estimation, the confidence a clinician should have in estimating VDT should be highly dependent on the degree of observed growth and on the CT scan slice thickness. The performance of CT scanners with slice thicknesses of ≥ 2.5 mm for assessing growth in pulmonary nodules is essentially inadequate for 1-mm changes in nodule diameter.

Due to improvements in CT scanning technology, small pulmonary nodules are being detected with increasing frequency. ¹ Nodules < 1 cm in diameter often pose a dilemma to clinicians and patients because these nodules may be difficult to biopsy, and alternative imaging techniques such as positron emission tomography/CT scanning may not improve diagnostic yield. Although not yet widely implemented, volume doubling time (VDT) measurements have the potential to become used more frequently in clinical practice as a surrogate measure of potential malignancy for such lesions, especially given the increased utilization of CT scanning. For solid nodules 3 to 9 mm in diameter, a doubling of volume occurs with 0.8- to 2.3-mm diameter growth, respectively; however, the inherent inter- and intraobserver variability of manual measurements, even in larger nodules, ² makes confident interpretation of small incremental change difficult.

Studies in patients suggest that three-dimensional volumetric measurement is a viable alternative to manual two-dimensional measurement,^{3, 4, 5} providing a direct volume estimate with high reproducibility that can be used to improve reader confidence in the presence or absence of growth. Other studies involving patients suggest that smaller nodules exhibit a greater tendency to grow than larger nodules, and that there may be large within-patient variation in the clinical behavior for pulmonary metastases. ⁶ Furthermore, shape characteristics of the nodule and slice thickness may also create significant volume differences.^{7, 8}

Segmentation refers to the process in which the computer chooses the borders of the nodule in order to calculate volume. Depending on nodule characteristics and adjacent structures, this process may arbitrarily include parts of vessels or additional lung parenchyma, and studies in phantoms suggest that volume estimation can be limited by variations due to nodule segmentation.^{9, 10, 11} These studies suggest that in solid nodules < 6 mm, automated volume estimation may tend to overestimate true volume (that it may be somewhat inaccurate) and may vary substantially for nodules of the same size (indicating a degree of imprecision). Although phantom experiments are not a substitute for studying nodule growth in vivo, they do provide a unique opportunity to incorporate “gold standard” volumetric measurements because the nodule volumes can be measured directly in the laboratory.

Small differences between estimated and true nodule VDTs resulting from inaccurate and imprecise volume estimations may be clinically insignificant, but large differences may yield uncertainty in doubling times that make benign and malignant nodules indistinguishable, rendering indeterminate evaluations. Our goal was to use data from phantom CT scan experiments to examine how imprecision in solid nodule volume estimation associated with nodule size and CT scan slice thickness impacts uncertainty in VDT estimates.

Materials and Methods

This study was approved by both the Medical University of South Carolina Institutional Review Board and by the US Army Medical Research and Materiel Command.

Experimental Data

These experiments, involving 1,624 volume estimations, have been reported in greater detail in a previous article. ⁹ To estimate the degree to which CT scan slice thickness impacts volume estimation for small indeterminate pulmonary nodules, experiments were performed on a single commercially available lung phantom using models of 29 lobular nodules made from dental wax, placed in the thorax cavity amid ground cork, which was used to simulate the lung parenchyma. These lobular nodules were designed to better reflect true lung nodule shapes, and ranged from 3.0 to 15.9 mm in average diameter. Data from all of these nodules were incorporated into our models and simulations. The reference standard for each nodule's volume was calculated using its mass, measured using a digital balance scale, and its density, measured using a technique involving repeated weighings in air and distilled water. CT scans were performed on a 16-slice CT scan system (GE Lightspeed; GE Medical Systems; Milwaukee, WI), and nodule volumes were estimated using reading software (Lung Volume Computer Assisted Reading [VCAR]; GE Medical Systems). The experimental CT scan protocol involved a CT detector array of 16 × 0.625 mm with reconstructed slice thicknesses of 0.625, 1.25, 2.5, and 5 mm. All measurements were performed with a 36-cm field of view and the BONE reconstruction kernel, which had previously been shown ⁹ to be the least biased kernel among several examined.

Defining Volume Estimation Accuracy and Precision

Volume estimation accuracy was defined by the percentage overestimation of the reference standard volume, dividing the difference between the estimated VCAR volume and the reference standard volume by the reference standard volume. The theoretical range of the accuracy measurement was thus −100% to infinity. Because our experiments were originally designed to examine the impact of other CT scan reconstruction parameters (eg, field of view) on nodule volume estimation, there were eight measurements and volume estimations for each nodule, using eight different scans with multiple nodules included on each scan. The precision associated with a volume estimate for a nodule of a given diameter was thus defined as the SD of the percentage overestimations of the reference standard volume. Note that larger values for this measure are indicative of larger SDs; thus, smaller precision estimates are desirable.

Model Assumptions

In modeling volume estimation precision as a function of nodule diameter and CT scan slice thickness, we made several assumptions. First, we assumed that precision is improved with larger nodule diameters because with larger diameters there are more opportunities for the CT scanner to slice through the nodule, providing more data to the VCAR software. Second, we assumed that for a fixed nodule diameter, precision is improved with decreasing CT scan slice thickness because smaller CT scan slice thicknesses would also result in more slicing opportunities. In concordance with the first two assumptions, an exponential decline (nonlinear) regression model was selected, in which the volume estimation precision (γ) associated with the ith slice thickness for a nodule of diameter (D) is of the form:

$$\gamma [iD]=A[i]\times \exp (BD)$$

where A[i] is a fixed-effect intercept parameter estimated for the ith slice thickness, exp(BD) is Euler's number (2.718) raised to the BD power, and B is an estimated parameter reflecting the rate of decline in imprecision with increasing nodule diameter. Using the exponential decline model aided us in conforming to our first assumption, and allowing the A[i] values (but not the B values) to vary by slice thickness helped us to conform to our second assumption. An exponential decline model is useful when modeling a dependent variable that decreases in value asymptotically toward a fixed value with increasing values of an independent variable. Note that our precision measurements are SDs, which, by definition, are always greater than zero. Using the regression model above, the parameters were estimated with a statistical software package (SAS, version 9.1; SAS Institute; Cary, NC) using the lung phantom CT scan experiment data.

Simulations To Estimate Confidence in VDT

Once the regression parameters were estimated, we conducted a series of mathematical simulations to demonstrate how error in volume estimates on hypothetical pulmonary nodules observed from CT scans performed at successive points in time translates into uncertainty in the VDT estimate. The simulations involved several steps:

• 1.First, input variable values were identified, including the time between the hypothetical CT scan nodule measurements, the CT scan slice thickness of interest, and the observed nodule diameters (D₁, D₂) at each of the two points in time.
• 2.Using the results from the nonlinear regression models, the precision γi[Dj] of each of the VCAR volume estimates associated with each diameter was computed.
• 3.
  Assuming that the observed volume at time 1 came from a distribution that was normally distributed with a mean equal to the nodule's true mean at time 1:

  $$V_1\left(True\right)=\frac{4}{3}\times \pi \times {\left(\frac{D_1}{2}\right)}^3$$
  (volume of a sphere) and (SD)

  $$\gamma \left[i{D}_1\right]\times {\text{V}}_{\text{1}}\left(\text{True}\right)$$
  (see Appendix A for further explanation) and that the observed volume at time 2 also came from a normally distribution with a mean equal to the nodule's true mean at time 2:

  $$V_2\left(True\right)=\frac{4}{3}\times \pi \times {\left(\frac{D_2}{2}\right)}^3$$

  and (SD)

  $$\gamma \left[i{D}_2\right]\times {\text{V}}_{\text{2}}\left(\text{True}\right)$$

  we obtained a sample from each of those distributions. Through these sampling procedures we introduced a certain degree of uncertainty into each of the VCAR volume estimates given the CT scan slice thickness of interest and observed diameters.
• 4.
  The VDT estimate was then computed using the modification by Lindell et al ¹² of the tumor growth model by Schwarz, ¹³ where

  $$VDT=\frac{Time\hspace{0.17em}\times \hspace{0.17em}\text{log}\left(2\right).}{\text{log}\left(\frac{{\text{V}}_2}{{\text{V}}_1}\right)}$$

  During this process, if a nodule was estimated by chance to have shrunk over the time of interest, we assigned a maximum VDT of 18,250 days (50 years) [Such occurrences were extremely rare.]

• 5.Steps 2 to 4 were repeated a total of 100,000 times to obtain the 2.5th, 50th, and 97.5th percentiles for each hypothetical VDT of interest. (The 2.5th and 97.5th percentiles provided a 95% confidence interval [CI] around the estimated VDT.) The number of repetitions needed was determined empirically to ensure the stability of the resulting percentiles.

To reflect a variety of relevant clinical scenarios, simulations were performed to demonstrate the effect of the volume estimation error on hypothetical nodules growing in diameter from 4 to 5 mm, 4 to 6 mm, 4 to 8 mm, 5 to 6 mm, and 9 to 10 mm. We also used the results of our simulations in separate regression analyses involving the parameters of interest to help produce equations that could be used to calculate VDT lower confidence limit (LCL) and upper confidence limit (UCL).

Simulations To Estimate Effects of Volume Measurement Imprecision on Malignancy Assessment

To examine the phenomena of how error in a nodule's volume estimates influences the probability of true doubling (ie, that it is malignant), we used similar simulation techniques. For these analyses, however, we assumed a fixed initial nodule diameter of 5 mm (corresponding volume for spherical nodule = 125 mm³), varied the CT scan slice thickness, and varied the hypothetical subsequent diameter up to 7 mm (corresponding volume for spherical nodule = 343 mm³), observed at some later point in time. Thus, for a specified follow-up diameter within a given simulation, the nodule may or may not have truly doubled in volume, depending on the precision of the follow-up measurement. By repeating this process 100,000 times across the range of observed follow-up diameters, we were able to calculate the proportion of times (ie, probability) that the nodule truly doubled in volume.

Results

Figure 1 displays the results of the nonlinear regression estimation modeling the volume estimation precision by nodule diameter and CT scan slice thickness. There were no data points corresponding to 5-mm CT scan slices for nodules < 4 mm in diameter; thus, the associated curve for the 5-mm slices only exists for diameters ≥ 4 mm. The model's predicted volume estimation precision was identical for the 0.625- and 1.25-mm CT scan slices; thus, they were grouped together for further comparisons. Volume estimation precision was 4.7-fold and 5.8-fold higher (worse), respectively, among 2.5- and 5.0-mm CT scan slices when compared to the 0.625/1.25-mm CT scan slices. This is reflected in the graph by the fact that the regression lines corresponding to the 2.5- and 5.0-mm CT scan slices are closer to one another than to the line for the 0.625/1.25-mm CT scan slices. Note that the volume estimation precision curves decline (ie, improve) with greater nodule diameter and decreased CT scan slice thickness.

Figure 1.

Simulation results: volume estimation precision by nodule diameter and CT scan slice thickness. Larger precision estimates are associated with greater variability in volume estimation. The curves represent the predicted results obtained from fitting the nonlinear regression model to the data points (▪, 5.00-mm CT scan slices; □, 2.50-mm CT scan slices; •, 1.25-mm CT scan slices; ○, 0.625-mm CT scan slices).

Median VDT estimates, which exactly matched the true VDT values, along with corresponding 95% CIs for each CT scan slice thickness of interest for a variety of hypothetical growth patterns, are displayed in Table 1. Because VDTs are always > 0 (resulting in skewed distributions), their CIs are not necessarily symmetric. For each hypothetical growth pattern, the VDT CIs get wider as the CT scan slice gets thicker. For example, if a nodule were observed to grow from 4 to 5 mm over the course of 180 days, the VDT estimate would be 186 days. However, if the observations were made using 5-mm CT scan slices, we would have much less confidence in the VDT estimate than had the observations been made using 1.25-mm slices. In fact, in this instance the 95% CI for the 5-mm slices (326 − 127 = 199 days) is more than six times wider than that for the 1.25-mm slices (202 − 173 = 29 days).

Table 1.

Estimated Nodule VDT by Slice Thickness and Various Observed Nodule Growth Patterns^*

	Growth, d
Patterns	From 4–5 mm	From 5–6 mm	From 4–6 mm	From 4–8 mm	From 9–10 mm
Observed nodule diameter growth over 90 d
VDT median	93	114	51	30	197
VDT 95% CI
0.625/1.25 mm	86–102	106–124	49–53	29–31	189–206
2.50 mm	65–159	81–183	42–64	27–33	163–248
5.00 mm	59–190	76–215	40–68	26–34	157–264
Observed nodule diameter growth over 180 d
VDT median	186	228	103	60	395
VDT 95% CI
0.625/1.25 mm	171–205	211–248	98–107	59–61	378–413
2.50 mm	128–314	163–367	84–128	54–67	326–497
5.00 mm	119–380	153–430	80–136	52–68	313–529
Observed nodule diameter growth over 360 d
VDT median	373	457	205	120	789
VDT 95% CI
0.625/1.25mm	342–409	422–496	197–214	117–123	756–826
2.50 mm	258–631	325–732	167–257	107–134	653–994
5.00 mm	239–760	305–853	159–272	104–137	626–1,061

^*Note that the estimated VDT medians are equal, what would be predicted by the modification of Lindell et al ¹² of the tumor growth model of Schwartz ¹³ using the diameters and times of interest.

Table 1 also emphasizes the fact that a 1-mm increase in nodule diameter carries entirely different clinical implications depending on the baseline diameter. For example, a 4-mm diameter nodule that grows 1 mm in 180 days implies a VDT estimate of 186 days; a 9-mm diameter nodule that grows 1 mm in 180 days implies a VDT estimate of 395 days. Likewise, the more relative growth that is observed, the more precise the VDT estimates are. For example, if a 4-mm diameter nodule increases (grows) to 5 mm, the 95% CIs for the VDT estimates are much wider than those for the situation in which a 4-mm diameter nodule grows to 8 mm in diameter. General regression formulas for the LCL and the UCL for VDT, explaining 97.1% and 93.9% of the variation in LCL and UCL, respectively, are listed in Appendix B.

Often clinicians are concerned with whether an observed pulmonary nodule has doubled in volume. Figure 2 demonstrates how the probability that a 5-mm nodule has doubled varies given subsequent observed diameters. Note that for a 5-mm nodule to double in size, it would have to grow in diameter to 6.3 mm. According to our data, if a 5-mm nodule is observed to grow to 6 mm (ie, not having doubled in volume) using 5-mm CT scan slices, the simulations suggest there is actually a 14% chance that the nodule has truly doubled in volume. However, if a 5-mm nodule is observed to grow to 6 mm using 1.25-mm CT scan slices, the simulations suggest there is virtually no chance that the nodule has truly doubled in volume. Likewise, if a 5-mm nodule is observed to grow to 6.5 mm using 5-mm CT scan slices, the simulations suggest there is only a 77% chance that the nodule has truly doubled in volume, whereas if the same observations were made using 1.25-mm CT scan slices, it is almost certain that the nodule has truly doubled in volume.

Figure 2.

Simulation results: probability that a 5-mm diameter nodule has doubled in volume given subsequent observed diameters and CT scan slice thicknesses.

Discussion

Volumetric analysis has been proposed as a method for assessing growth of small solid pulmonary nodules, having implications for incidentally detected nodules and nodules detected during cancer staging studies. Our study reaffirms the critical nature of using thin collimation for accurate measurement of diameter, volume, and analysis of doubling time, but it also shows the pitfalls of trying to assess growth of small solid pulmonary nodules with slice thicknesses commonly used in clinical practice. These phantom pulmonary nodule experiments in conjunction with mathematical simulations provide insight into what can be learned from repeated CT scan screens on intermediate pulmonary nodules and highlights and quantifies what should be obvious: errors in nodule volume estimation translate into uncertainty with VDT estimation. The CIs around the VDT estimates were extremely wide for 2.50- and 5.00-mm CT scan slice thicknesses, often encompassing values traditionally associated with both benignity and malignancy^{14, 15} for simulated 1- and 2-mm growths in diameter. The confidence clinicians should have in estimating VDT should be highly dependent on the degree of observed growth and on the CT scan slice thickness.

Our results support the general recommendations of follow-up for small indeterminate pulmonary nodules set out by the Fleischner Society. ¹⁴ Even for tumors with a doubling time of 90 days, our data suggests that the follow-up intervals set forth in the Fleischner guidelines ¹⁴ are reasonable to reliably detect growth that would imply the need for more aggressive intervention. Our findings are also consistent with those of Ko et al, ¹⁶ who noted that knowledge of the precision of volume estimation is vital for correct clinical nodule growth interpretations. Moreover, our findings emphasize the importance of thin sections to detect growth for lesions growing at an intermediate and slow pace (180 to 360 days doubling time) because variability associated with thick sections overlaps benign and malignant growth rates. In this study, no comparison was made between manual diameter measurements and automated volumetric measurements. However, Erasmus et al ² note that large variation exists with manual diameter measurements. In clinical practice, mean diameter measurements are used frequently for assessing nodule growth and are likely the source of even greater error than that observed with automated volumetric measurements. Figure 3 highlights several key clinical implications of our study.

Figure 3.

Key clinical messages.

Even with relatively thick sections, however, automated volumetric software can provide a relatively confident window of expected doubling time in the malignant range for relatively rapidly growing lesions (90 days doubling time) with 1 mm of growth. This suggests that even small lesions with a higher likelihood of malignancy (eg, small nodules in patients with extrathoracic neoplasms) may be re-imaged to detect growth earlier than set forth by the Fleischner Society criteria. ¹⁴ Overall though, our study has shown that 2.5- and 5.0-mm slice thicknesses are generally inadequate for following small nodules, particularly those < 6 mm where a 1-mm change in diameter would approximately represent a volume doubling. Perhaps future versions of automated volumetric software could incorporate our study findings so that nodule growth reports could include CIs for doubling time and/or probabilities for true nodule doubling.

Several previous studies^{4, 7, 17, 18, 19} have also addressed the slice thickness issue, often using overlapping reconstruction algorithms. Our previous work suggests that the mean percentage volume overestimation is similar for both spherical and lobulated nodules. ⁹ Although we did not investigate spiculated nodules, it has recently been suggested that overestimation of volume is even greater in such instances. ⁷ We chose to use nonoverlapping slices, which may have affected our results at both 2.5- and 5.0-mm collimation. Although it appears this is necessary with larger slice thicknesses, at 1.25-mm slice thickness and below, overlap does not substantially affect nodule volume measurements.

Our study has limitations that arise from the use of a lung phantom. In clinical practice, volume of nodules for consecutive scans may vary up to 25%. ²⁰ Additionally, the contribution of depth of inspiration to error cannot be assessed. For noncalcified nodules > 3 mm, the change in volume from expiration to inspiration ranges from 17 to 23%.^{21, 22} There are likely many other patient characteristics that may influence volume estimation in vivo (eg, nodule density) that cannot be accounted for in the context of a phantom study. We also used nodules suspended in cork that greatly facilitated segmentation. In clinical practice, adjacent vessels may be included in the volume analysis creating another source of variability. ²³ The relative contribution of adjacent structures can lead to spurious changes in volume and complicate the interpretation of growth or shrinkage. A human-adapted approach in which the radiologist interactively adjusts the contours to exclude attached vessels may produce greater accuracy for actual volume.^{24, 25} The downside of this approach, however, would be a potential loss of accuracy and precision. Also, our models, although based on data from solid lobulated nodules, were used to predict uncertainty for spherical nodules. Thus, in practice, errors surrounding the calculated VDT are going to be larger, and our results represent a “best-case” scenario. Additionally, our volume estimations were based on a single (ie, VCAR) software package and may not necessarily be generalizable to other volume estimation algorithms.

Our findings emphasize the clinical importance of understanding the variability in solid lung nodule volume estimates. Errors in clinical judgments concerning these nodules, including both false positives and false negatives, have significant consequences for patients. A clinical example involves the patient with an extrathoracic malignancy and a pulmonary nodule. Mischaracterization of growth assigned to an incremental change of 1 mm or spurious volume doubling may lead to an interpretation of metastatic disease, when, in fact, the patient truly has localized and potentially curable disease. As radiologists and other clinicians caring for patients with such nodules continue to embrace and rely on increasingly advanced technologies, it is important for them to understand the nature of the limitations and levels of uncertainties.

Appendix A: Proof That the SD of the Observed Volume Is the SD of the Percent Overestimation of Volume Times the True Volume

Let X equal the percentage overestimation of volume (V), and let γ represent the SD of X.

By definition,

$$X=\frac{VObserved-VTrue}{VTrue}$$

Then, by algebra

$$VObserved=VTrue+\left[X\times VTrue\right]$$

Thus, we know that the variance of Vobserved is the variance of

$$VTrue+\left[X\times VTrue\right]$$

Using statistical theory and the fact that Vtrue is a constant (thus having no variance), it can further be shown that the variance of Vobserved is

$${\left(VTrue\right)}^2\times Variance\left(X\right)$$

Because the SD of a random variable is the square root of its variance, we thus know that the SD of Vobserved is

$$\gamma \times VTrue$$

Appendix B: General Regression Formulas for the LCL and UCL for VDT

In these equations, THICK is CT scan slice thickness (range, 1.25 to 5.0 mm), TIME is the length of follow-up time in days, and D1 and D2 are diameter measurements at time 1 and time 2, respectively. Because the entries in Table 1 were determined by simulations, they do not necessarily match what would be found using these formulas. However, these formulas might help clinicians with CT scan slice thicknesses, time frames, or nodule growths that do not coincide with what is presented in Table 1.

$$LCL=-4.4930-7.8709\times THICK+0.0821\times \text{TIME}+6.1321\times D_1-\frac{0.8226}{In\left(\frac{D_2}{D_1}\right)}+\frac{0.1661\times TIME}{In\left(\frac{D_2}{D_1}\right)}$$

$$UCL=28.3163+20.3725\times THICK-0.2441\times TIME-19.4718\times D_1+\frac{2.8151}{In\left(\frac{D_2}{D_1}\right)}+\frac{0.3822\times TIME}{In\left(\frac{D_2}{D_1}\right)}$$