Observer Performance for Detection of Pulmonary Nodules at Chest CT over a Large Range of Radiation Dose Levels

Abstract

Background

There is a wide variation in radiation dose levels that can be used with chest CT in order to detect indeterminate pulmonary nodules.

Purpose

To compare the performance of lower-radiation-dose chest CT with that of routine dose in the detection of indeterminate pulmonary nodules 5 mm or greater.

Materials and Methods

In this retrospective study, CT projection data from 83 routine-dose chest CT examinations performed in 83 patients (120 kV, 70 quality reference mAs [QRM]) were collected between November 2013 and April 2014. Reference indeterminate pulmonary nodules were identified by two nonreader thoracic radiologists. By using validated noise insertion, five lower-dose data sets were reconstructed with filtered back projection (FBP) or iterative reconstruction (IR; 30 QRM with FBP, 10 QRM with IR, 5 QRM with FBP, 5 QRM with IR, and 2.5 QRM with IR). Three thoracic radiologists circled pulmonary nodules, rating confidence that the nodule was a 5-mm-or-greater indeterminate pulmonary nodule, and graded image quality. Analysis was performed on a per-nodule basis by using jackknife alternative free-response receiver operating characteristic figure of merit (FOM) and noninferiority limit of −0.10.

Results

There were 66 indeterminate pulmonary nodules (mean size, 8.6 mm ± 3.4 [standard deviation]; 21 part-solid nodules) in 42 patients (mean age, 51 years ± 17; 21 men and 21 women). Compared with the FOM for routine-dose CT (size-specific dose estimate, 6.5 mGy ± 1.8; FOM, 0.86 [95% confidence interval: 0.80, 0.91]), FOM was noninferior for all lower-dose configurations except for 2.5 QRM with IR. The sensitivity for subsolid nodules at 70 QRM was 60% (range, 48%–72%) and was significantly worse at a dose of 5 QRM and lower, whether or not IR was used (P < .05). Diagnostic image quality decreased with decreasing dose (P < .001) and was better with IR at 5 QRM (P < .05).

Conclusion

CT images reconstructed at dose levels down to 10 quality reference mAs (size-specific dose estimate, 0.9 mGy) had noninferior performance compared with routine dose in depicting pulmonary nodules. Iterative reconstruction improved subjective image quality but not performance at low dose levels.

© RSNA, 2020

Online supplemental material is available for this article.

See also the editorial by White and Kazerooni in this issue.

Summary

For depicting indeterminate pulmonary nodules size 5 mm or greater, radiation dose levels for chest CT can be reduced to one-third of the radiation dose level used in the National Lung Cancer Screening Trial.

Key Results

• ■ Radiation dose reductions to one-third of that in the National Lung Cancer Screening Trial were possible for chest CT examinations performed to depict indeterminate nodules 5 mm or larger, and yielded noninferior performance compared with standard radiation doses (P < .05).

• ■ Sensitivity in detecting part-solid pulmonary nodules at radiation dose levels less than 0.8 mGy or size-specific dose estimate of 0.9 mGy were inferior to routine-dose levels, with 25% lower sensitivity or more for radiation dose levels corresponding to CT dose index 0.4 mGy or less (P < .05).

Introduction

Chest CT is one of the most common diagnostic tasks in CT imaging (1). Chest CT has the ability to depict early lung cancer in patients who are high risk and to reduce lung cancer mortality (2). It is also used to depict pulmonary metastases and to plan subsequent surgical or medical therapies (3). Despite its frequent use, there is substantial variability in radiation dose used for chest CT (4).

The National Lung Cancer Screening Trial used a lower-dose chest CT technique with an effective tube current–time product of 20–30 mAs for the average-sized patient, which resulted in an effective dose of approximately 1.5 mSv (5). Numerous subsequent studies have examined the ability of chest CT, when performed at much lower doses, to depict indeterminate pulmonary nodules, which can be followed up over time to detect early-stage lung cancer (6–9). As a working definition, a pulmonary nodule is classified as indeterminate if there are no definite benign morphologic findings (10). Challenges to the implementation of low-dose chest CT include the potential for interobserver variability and the difficulty in identifying ground-glass or part-solid (ie, subsolid) pulmonary nodules (11–13). A recent systematic analysis (13) concluded that iterative reconstruction (IR) improves image quality for low-dose chest CT compared with filtered back projection (FBP). Mounting evidence in abdominal CT demonstrates that IR has limited ability to preserve observer performance at lower radiation doses for low-contrast lesions (in which the CT number is similar to that of the anatomic background) (14). Part-solid nodules at chest CT may present a similar low-contrast diagnostic challenge. Although the majority of part-solid nodules resolve at follow-up CT (15), persistent pulmonary subsolid nodules may represent a broad spectrum of neoplastic abnormalities (16).

We recently performed a pilot feasibility study examining routine-dose chest CT and numerous lower-radiation-dose levels to determine the lowest radiation dose levels with noninferior performance compared with routine-dose chest CT in depicting indeterminate pulmonary nodules (17). The purpose of our study was to examine a larger number of patients by using a multireader design to estimate the ability of varying radiation dose levels to help detect indeterminate pulmonary nodules 5 mm or greater and determine whether IR is necessary.

Materials and Methods

Patients

After our institutional review board approved this retrospective, case-control, Health Insurance Portability and Accountability Act–compliant study, we archived CT image and projection data from patients who underwent clinically indicated chest CT between November 2013 and April 2014. This work does not contain any overlap with previously published studies. No patients or experiments in this study were reported previously.

Patients were included if they underwent routine-dose chest CT with a 128-slice multidetector CT platform for clinical indications, if they provided consent to the retrospective use of clinical imaging and medical records for research purposes, and if imaging and projection data were successfully archived. A subspecialized thoracic radiologist (D.L.L., with >20 years of clinical experience after thoracic radiology fellowship training) reviewed all images along with the correlative medical record. Patients with airspace disease, more than 10 pulmonary nodules, primary or secondary lung neoplasms, or previous thoracic surgery were excluded. As we sought to detect decreases in observer performance associated with radiation dose reduction, patients with known primary or secondary lung neoplasms at the time of the study were excluded because these patients often had postsurgical changes or other findings (eg, bulky adenopathy, osseous metastatic disease) that might have affected observer performance. Additionally, we used a case-control study design to select cases (ie, patient examinations with indeterminate pulmonary nodules) that would reflect the discriminatory ability of lower radiation doses to display difficult-to-detect pulmonary nodules.

The primary diagnostic task under consideration was the detection of indeterminate pulmonary nodules 5 mm or greater. Because of substantial interobserver variability in chest CT, any decrease in observer performance due to dose can potentially be overshadowed by the underlying variability of detection at baseline (18,19). Larger nodules are more likely to minimize reader variability (20). Additionally, the detection of nodules smaller than 5 mm either incidentally or during lung cancer screening is of questionable importance because those nodules do not typically require specific follow-up (21,22). Cases consisted of chest CT scans in patients with indeterminate pulmonary nodules 5 mm or larger identified by nonreader thoracic radiologists (D.L.L. and R.S.K.) unblinded to the medical record, which included follow-up chest CT. Control cases consisted of CT scans in patients without indeterminate pulmonary nodules 5 mm or larger. A nonreader subspecialized thoracic radiologist (D.L.L.) then reviewed archived chest CT scans to select as many chest CT examinations with part-solid indeterminate pulmonary nodules 5 mm or larger as possible. An overall study schema is shown in Figure 1.

Figure 1:

Flowchart of overall study schema. MDCT = multidetector CT.

Image Acquisition and Reconstruction

Chest CT was performed with similar CT systems (Somatom Definition Flash and Somatom Definition AS+; Siemens Healthineers, Erlangen, Germany). Scans were acquired at 120 kV with a single x-ray source, with automatic exposure control (CareDose4D; Siemens Healthineers) setting of 70 quality reference mAs (QRM), 0.5-second rotation time, and 128 × 0.6-mm detector configuration. Lower-dose CT projection data were created by using a validated noise insertion tool (23–26), which accounts for the automatic exposure control data, electronic noise, and the bowtie filter. Noise-inserted projection data were then created and loaded back onto the original CT system for image reconstruction. Before the initiation of the study, simulated very-low-dose images that used the noise-insertion tool were created corresponding to 3%–4% of the routine dose in a thoracic phantom, comparing image noise between very-low-dose CT acquisition versus simulated low-dose images, with results demonstrating 6% or smaller difference in image noise.

Images were reconstructed with an FBP B50 kernel or an IR I50 kernel, with a strength setting of 2 (I50–2, Sinogram Affirmed Iterative Reconstruction or SAFIRE; Siemens Healthineers), slice thickness of 1.5 mm, and reconstruction increment of 1.0 mm. The following six conditions were selected for the reader study by using the CT system’s automatic exposure control setting to control radiation dose and image appearance: 70 QRM with FBP (the routine clinical examination), 30 QRM with FBP, 10 QRM with IR, 5 QRM with FBP, 5 QRM with IR, and 2.5 QRM with IR. In addition to the thin axial images, thick axial maximum intensity projection images were also generated for each dose level by using the same reconstruction kernel as the corresponding axial images, with a 20-mm slice thickness and a 2.5-mm increment.

Image Evaluation by Subspecialized Thoracic Radiologists

Three subspecialized thoracic radiologists (A.M.G.S., R.M.L., and D.B.W., with 17, 13, and 3 years of experience, respectively, as subspecialized thoracic radiologists at our institution) were selected to be radiologist readers. Each reader had previously participated in standardized reader training (1).

In each reading session, thoracic radiologists evaluated chest CT scans according to a predetermined randomization schedule, viewing each patient’s scans only once per session, with sessions separated by at least 3 weeks. Radiologists panned up and down through images by using a previously described proprietary computer workstation (1). When a potential indeterminate pulmonary nodule was identified, the cursor was used to draw a line across the greatest nodule diameter and the length was displayed on the screen. Radiologists then rated confidence that the nodule was an indeterminate pulmonary nodule 5 mm or larger by using a scale from 0 to 100 (eg, scores 1–25 reflected a low degree of confidence in an indeterminate pulmonary nodule and 75–100 indicated a high degree of confidence and the need for follow-up). Readers were asked to characterize each indeterminate nodule as solid or part solid. Image quality scores were also assigned, as previously described (1,27).

Reference Standard

Two nonreader subspecialized thoracic radiologists (D.L.L., R.S.K., each with >20 years of clinical experience following subspecialized thoracic radiology fellowship training) evaluated routine-dose CT images by using the same computer workstation and examined all correlative information in the medical record, including comparison to previous and follow-up chest CT examinations. Each radiologist completed a review of the images independently. The original interpretation for each CT study was reviewed as a further search for nodules not detected by either nonreading radiologist. All initial discrepancies in the independent interpretations were resolved by consensus review to determine the presence or absence of a pulmonary nodule. Disagreements were not recorded. To characterize pulmonary nodules, a nodule was classified as part solid if at least one radiologist marked the nodule and characterized it as part solid.

Statistical Analysis

The sample size for this study was determined as a part of a two-stage study design, with results of stage 1 previously published (1). The original sample size calculations determined 83 patients were needed for this stage of the study.

Matching of reference and reader detections was performed by the principal investigator (J.G.F., with 20 years of experience as a clinical radiologist). Comparison between routine-dose chest CT (FBP, 70 QRM) and lower-dose reconstruction configurations was performed by using jackknife alternative free-response receiver operating characteristic (JAFROC) figure of merit (FOM) noninferiority analysis.

JAFROC FOM analysis used the reader confidence scores assigned by the readers and their circumscribed imaging findings (1). The JAFROC FOM uses correct localization and identification of proven lesions per patient, in addition to false-positive markings (nonlesion localizations). The JAFROC FOM ranges from 0 to 1, representing the probability that the rating of the highest rated and correctly identified metastasis in a positive patient case exceeds that of the highest rated nonlesion localization in a healthy control case (15). Multiple findings in positive patient cases were weighted according to the reciprocal of the number of findings (27). FOMs were calculated for every dose level and reader. The comparisons of FOMs were estimated by using the Hillis improvement (28) to the method by Dorfman et al (29) with the modeling assumption of fixed readers–random cases by using the RJafroc package v1.0.1 (R version 3.4.2; Vienna, Austria). Noninferiority of lower-dose configurations was represented by calculating the difference between routine-dose CT and lower-dose configurations, with an a priori limit of noninferiority set at −0.10 (30). Noninferiority was demonstrated if the lower limit of the 95% confidence interval is greater than −0.10.

Typical measures of diagnostic accuracy were provided by using a cutoff reader confidence score of 10. For per-patient specificity, there could be no reader circumscriptions with a confidence level greater than 10 in negative patient cases. For these measures, generalized estimating equations with independent working covariance matrices were used to estimate the pooled estimate across the three readers for each imaging strategy. A subanalysis was performed for part-solid nodules.

For the qualitative image quality ratings (eg, overall impression of diagnostic image quality), a summary score was the mean image quality ranking of the three readers. Tests for differences in image quality across dose and reconstruction were facilitated by generalized estimating equation models with independent working covariance matrices. All pairwise comparisons were performed after Tukey adjustment. Post hoc comparisons of the quality summary score were considered descriptive and were not adjusted for multiple testing across doses.

Finally, the relationship between lesion size (diameter) and mean confidence score was assessed at each dose and reconstruction by Spearman rank correlation coefficient, imputing a confidence of 0 if a nodule was missed. P values less than .05 were considered to indicate statistical significance.

Results

Eighty-three patients underwent routine-dose chest CT (mean age, 47 years ± 18 [standard deviation]; 47 men and 36 women). There were 66 indeterminate pulmonary nodules (21 were part-solid nodules) 5 mm or larger in 42 patients (mean age, 51 years ± 17; 21 men and 21 women). The median size of the indeterminate pulmonary nodules was 8.0 mm (mean, 8.6 mm ± 3.4; interquartile range, 6.5–9.7 mm). Forty-one control patients did not have indeterminate pulmonary nodules larger than 5 mm. The size profile did not differ between part-solid and solid nodules (part-solid vs solid nodules: mean size, 9.3 mm ± 4.3 vs 8.3 mm ± 3.0, respectively). The mean volume CT dose index and size-specific dose estimate at routine dose were 5.3 mGy ± 2.0 and 6.5 mGy ± 1.8, respectively.

Table 1 shows the pooled JAFROC FOMs for each radiation dose level. Figure 2 is a forest plot that demonstrates the estimated difference in the JAFROC FOM between routine-dose CT and the lower-dose configurations. It shows that all lower-dose configurations, except for the 2.5 QRM with IR configuration, had noninferior performances.

Table 1:

Pooled JAFROC FOMs

Figure 2:

Forest plot demonstrates estimated differences with 95% confidence intervals (CIs) in the jackknife alternative free-response receiver operating characteristic figure of merit between routine-dose chest CT and lower-radiation-dose configurations. The dotted line represents the limit of noninferiority set prior to the study. Because the lower limit of the 95% CI exceeds the limit for 30 quality reference mAs (QRM) with filtered back projection (FBP), 10 QRM with iterative reconstruction (IR), and 5 QRM configurations, these configurations are noninferior to routine dose at 70 QRM. Eff = effective.

Table 2 shows the generalized estimating equation sensitivity and specificity estimates per patient in addition to overall and part-solid per nodule sensitivity estimates. Table E1 (online) shows the point estimates for the individual readers. Table 2 shows that per-patient and per-nodule sensitivity drops slowly with declining radiation dose, but that specificity remains virtually unchanged despite a greater than 25-fold dose difference between the 2.5 QRM and 70 QRM configurations. The sensitivity for subsolid nodules is significantly decreased for all dose configurations less than 10 QRM (P < .05; Figs 3, 4).

Table 2:

Per-Patient and Per-Lesion Sensitivity and Specificity

Figure 3:

Axial lung CT images in a 44-year-old man with a 14.8-mm part-solid nodule (arrows) in the left lower lobe. Three thoracic radiologist readers detected the nodule on images corresponding to, A, an automatic exposure control setting of 70 quality reference mAs (QRM) reconstructed with filtered back projection, but only one reader detected the nodule at, B, 10 QRM with iterative reconstruction, and no readers detected the nodule with, C, 5 QRM and, D, 2.5 QRM with iterative reconstruction. Note how it is difficult to perceive the ground-glass portions of the nodule at the lower-dose levels shown in C and D.

Figure 4:

Axial lung CT images in a 22-year-old man with a 6.8-mm part-solid nodule (arrows) in the left upper lobe. The nodule was detected by two readers on images corresponding to, A, an automatic exposure control setting of 70 quality reference mAs (QRM) with filtered back projection and two readers at, B, 10 QRM with iterative reconstruction. However, the nodule was not detected at, C, 5 QRM with filtered back projection or, D, iterative reconstruction. Note how it is difficult to see the ground-glass portions of the nodule at the lower-dose levels shown in panels C and D.

Figure 5 shows that lower-dose configurations demonstrated lower overall diagnostic image quality, greater image noise, and worsening sharpness (P < .001). For image quality comparisons that evaluate 5 QRM with IR versus 5 QRM with FBP, diagnostic image quality was superior with IR (P < .001), image noise was significantly reduced (P = .004), and image sharpness was slightly but significantly reduced (P < .001). The Spearman correlation between reader confidence in nodule detection and size indicated that for all radiation dose levels, nodule size correlated significantly with reader confidence (P < .05).

Figure 5:

Scatter plot shows different image-quality metrics for routine (70 quality reference mAs [QRM]) and lower-dose configurations. Lower-dose configurations demonstrated significantly lower overall diagnostic image quality, greater image noise, and worsening sharpness for all pairwise comparisons after Tukey adjustment for all comparisons. At the 5-QRM dose level, all three image quality metrics were significantly improved with iterative reconstruction (IR) compared with filtered back projection (FBP). Black dots represent the mean image quality rating for each dose level and reconstruction combination along the horizontal axis.

Discussion

We compared observer performance for detection of indeterminate pulmonary nodules between our routine-dose chest CT and images at five lower-radiation-dose levels. We found that radiation dose could be reduced by one-seventh (to radiation dose levels approximately one-third that in the National Lung Cancer Screening Trial) while still yielding noninferior performance compared with routine radiation dose (P < .05). Importantly, however, sensitivity for detection of part-solid pulmonary nodules at lower-radiation-dose levels were inferior to routine-dose levels, with an estimated decrease in sensitivity of over 25% (P < .05), with iterative reconstruction (IR) unable to improve performance at these very-low-dose levels.

Our study showed that subsolid nodules can be detected without a decrement in performance at the National Lung Screening Trial dose level (2). However, very-low-dose levels resulted in loss of detection of some part-solid nodules, as demonstrated previously (31,32). Our finding that IR was not helpful at lower-dose levels differs from some previous results (33–35), but is consistent with that of Vardhanabhuti et al (36) and others (20), and a prior meta-analysis concluding that dose can be reduced to less than 1 mSv when IR is used for a variety of indications including detection of pulmonary nodules (37). In our study, performance for detection of part-solid nodules was not only preserved at the approximate National Lung Cancer Screening Trial dose by using automatic exposure control (30 QRM, approximately 1.5 mSv) but also was preserved down to the 10-QRM level (CT dose index, 0.8 mGy; size-specific dose estimate, 0.9 mGy).

IR has a limited ability to preserve diagnostic performance for low-contrast diagnostic tasks in the abdomen and head (ie, for detection of liver metastases and cause of acute neurologic deficit) (14,23,38,39). Similarly, CT attenuation of ground-glass or part-solid nodules is only slightly different than background pulmonary parenchyma. Because IR has image contrast–dependent spatial resolution, boundaries of part-solid pulmonary nodules can be obscured at low doses. Radiologists should be aware that even though image quality appears to be improved with IR at these very-low-dose levels, part-solid nodules may be underdetected.

The results of our study also have practical application to emerging methods of CT noise reduction in thoracic CT, namely convolutional neural networks, which are becoming commercially available. In one recent study (40), deep learning reconstructions and IR provided similar but superior results compared with FBP. Further examination is needed to assess whether convolutional neural network–based noise reduction is able to improve performance for low-contrast diagnostic tasks, such as the detection of part-solid nodules. Instead of performing large reader studies, task-based model observers, such as channelized Hoteling observers could be used to assess performance of lower-dose images with convolutional neural network–based noise reduction (41,42). These model observers characterize the human visual response and provide an efficient and objective measure of diagnostic image quality, which has been demonstrated to be well correlated with human reader performance for both phantom-based and patient-based lesion detection tasks (41–44).

Our study had limitations. It relied on only three readers; consequently, we plan to conduct a larger study that uses more patients and readers. A validated noise insertion algorithm was used to create the lower-dose CT projection data; however, this highly accurate method has permitted rapid assessment of many dose levels and facilitated adoption of research results into our clinical practice. Our reference standard relied on experienced subspecialized thoracic radiologists with access to clinical and follow-up information, which was variable, particularly for negative studies. Readers did not examine nonaxial multiplanar reformations, and the extrapolation of performance results from our subspecialized readers to general radiologists is inappropriate. The retrospective case-control study design we used was designed to exacerbate potential performance differences between radiation dose levels by using a cohort enriched with a greater proportion of subsolid nodules so that performance estimates could not be extrapolated to other settings.

In conclusion, for the purpose of depicting indeterminate pulmonary nodules 5 mm or larger, radiation dose levels for chest CT can be reduced by a limited amount beyond that employed in the National Lung Cancer Screening Trial. We show that very low radiation doses result in an inability to depict a significant proportion of part-solid indeterminate pulmonary nodules, with iterative reconstruction improving subjective image quality but not observer performance at very low doses.