Tomosynthesis, when used in a clinical radiology setting with both thoracic and nonthoracic specialists, can improve detection of pulmonary nodules and case management decisions when compared with conventional chest radiography.
Abstract
Purpose
To conduct a multi-institutional, multireader study to compare the performance of digital tomosynthesis, dual-energy (DE) imaging, and conventional chest radiography for pulmonary nodule detection and management.
Materials and Methods
In this binational, institutional review board–approved, HIPAA-compliant prospective study, 158 subjects (43 subjects with normal findings) were enrolled at four institutions. Informed consent was obtained prior to enrollment. Subjects underwent chest computed tomography (CT) and imaging with conventional chest radiography (posteroanterior and lateral), DE imaging, and tomosynthesis with a flat-panel imaging device. Three experienced thoracic radiologists identified true locations of nodules (n = 516, 3–20-mm diameters) with CT and recommended case management by using Fleischner Society guidelines. Five other radiologists marked nodules and indicated case management by using images from conventional chest radiography, conventional chest radiography plus DE imaging, tomosynthesis, and tomosynthesis plus DE imaging. Sensitivity, specificity, and overall accuracy were measured by using the free-response receiver operating characteristic method and the receiver operating characteristic method for nodule detection and case management, respectively. Results were further analyzed according to nodule diameter categories (3–4 mm, >4 mm to 6 mm, >6 mm to 8 mm, and >8 mm to 20 mm).
Results
Maximum lesion localization fraction was higher for tomosynthesis than for conventional chest radiography in all nodule size categories (3.55-fold for all nodules, P < .001; 95% confidence interval [CI]: 2.96, 4.15). Case-level sensitivity was higher with tomosynthesis than with conventional chest radiography for all nodules (1.49-fold, P < .001; 95% CI: 1.25, 1.73). Case management decisions showed better overall accuracy with tomosynthesis than with conventional chest radiography, as given by the area under the receiver operating characteristic curve (1.23-fold, P < .001; 95% CI: 1.15, 1.32). There were no differences in any specificity measures. DE imaging did not significantly affect nodule detection when paired with either conventional chest radiography or tomosynthesis.
Conclusion
Tomosynthesis outperformed conventional chest radiography for lung nodule detection and determination of case management; DE imaging did not show significant differences over conventional chest radiography or tomosynthesis alone. These findings indicate performance likely achievable with a range of reader expertise.
© RSNA, 2016
Introduction
Digital tomosynthesis has been applied to a variety of applications, including breast imaging (1–3), orthopedic imaging (4–8), urologic imaging (9), and chest imaging (10–23). Digital chest tomosynthesis has been available and cleared by the U.S. Food and Drug Administration since 2006. Early clinical studies have shown it has the potential to triple detection sensitivity of pulmonary nodules over that of conventional chest radiography when used by experienced thoracic radiologists (11,12).
Prior to the development of tomosynthesis, dual-energy (DE) imaging was used as an improvement over conventional chest radiography for the detection and classification of pulmonary nodules by reducing the visual clutter associated with the ribs and providing information on patterns of nodule calcification (24–34). In general, however, DE imaging has shown more modest improvement in detection accuracy than has been reported more recently for tomosynthesis.
In this report, we describe a study conducted to compare the performance of these modalities implemented with a commercially available device cleared by the U.S. Food and Drug Administration. The primary analyses were used to compare tomosynthesis and conventional chest radiography, both with and without DE imaging, for nodule detection accuracy and case management decisions. Secondary analyses were used to evaluate five subsets of nodules: those visible to radiologists who provided ground truth findings (ie, the “truth panel”) on conventional chest radiographs; nodules not visible to the truth panel on conventional chest radiographs; solid nodules; calcified nodules; and nodules located in the upper lung. Radiologists from a variety of specialty training backgrounds were used to gauge the effect of the various modalities on detection accuracy in a broad context of clinical readers. In this article, we describe a multi-institutional, multireader study to compare the performance of tomosynthesis, DE imaging, and conventional chest radiography for pulmonary nodule detection and management.
Materials and Methods
Financial support for this study was provided by GE Healthcare. Nonemployee authors and an independent contract research organization had unrestricted control of the data and information submitted for publication.
Imaging data were collected from subjects at four tertiary care sites—three in the United States and one in Sweden. The study (ClinicalTrials.gov no. NCT00963651) was approved by the institutional review board of each institution, and informed consent was obtained from each subject prior to enrollment. The study was compliant with the Health Insurance Portability and Accountability Act. Subjects enrolled were referred for, or had recently undergone, chest computed tomography (CT) as part of their standard clinical care for suspicion of a pulmonary nodule or other indications unrelated to pulmonary nodules. No subject underwent a CT examination solely for the purposes of participating in this study. Appendix E1 (online) lists inclusion and exclusion criteria, study population demographics, and the disposition of cases excluded from the final study cohort. There were 187 subjects who provided consent and were enrolled in the study. Of these, 158 subjects were found to qualify for inclusion in the final study sample; 115 had one or more CT-confirmed pulmonary nodules, and 43 subjects had no nodules.
After the clinical CT examination, radiographic images, including conventional posteroanterior (PA) and lateral chest radiographs, DE radiographs with both tissue and bone images, and digital tomosynthesis images, were acquired for research purposes. All radiographic images were acquired by using a commercial device cleared by the U.S. Food and Drug Administration (XR656 with VolumeRAD and DE applications; GE Healthcare, Chalfont St Giles, United Kingdom). The subjects were positioned for the PA chest radiographic examination, and then low-energy (60-kVp) and high-energy (120-kVp) radiographs for the DE examination were acquired, with the high-energy image from the DE pair serving as the conventional PA radiograph per standard procedure. Tomosynthesis images were then acquired at 120 kVp with the subject in the same PA orientation, totaling 60 low-exposure images acquired as the tube moved over 30°. Subjects were then positioned to undergo lateral chest radiography by using the standard protocol for each institution (110–130 kVp). Beam filtration of 0.2-mm Cu was used for tomosynthesis image acquisition to minimize the dose consistent with good image quality (35), and automatic phototiming was used for all image acquisitions. Radiation dose for each subject was estimated from recorded tube current–time product values on the basis of a previous analysis of effective dose (36). Mean effective doses for the various modalities were 0.06 mSv (range, 0.02–0.19 mSv) for two-view (PA and lateral) conventional chest radiography and 0.10 mSv (range, 0.07–0.41 mSv) for tomosynthesis. By comparison, effective doses of 4.9 mSv (range, 1.4–18.2 mSv) were estimated for the clinical CT examinations of each subject (37,38).
Ground truth was established by three experienced thoracic radiologists. Details about the process of establishing ground truth are provided in Appendix E2 (online).
Observers and Training
Five radiologists (all nonauthors) served as readers for images obtained with the three modalities being studied (conventional chest radiography, DE imaging, and tomosynthesis). These observers were intentionally selected from different areas of specialty training to provide a range of expertise that might reflect that commonly found in a general radiology setting. The training specialties for these five readers were thoracic, neuroradiology and cardiovascular, pediatric and thoracic, abdominal and interventional, and nuclear medicine, with 26, 12, 14, 2, and 15 years of experience, respectively. The time these readers spent interpreting chest images in their regular practices ranged from 10% to 100%. Prior to the study, only one reader who had trained in thoracic imaging was experienced in the interpretation of DE images; none had prior experience with tomosynthesis. The five readers did not serve as the truth panel of the CT data and did not have the CT data available to them when interpreting the conventional chest radiographs, DE images, and tomosynthesis images. Observers were shown a set of training images prior to beginning the study (see Appendix E2 [online]).
Observer Study
For measuring the accuracy of nodule detection and localization, the study design included use of a free-response receiver operating characteristic (FROC) method (39–42). Readers reviewed images from the three modalities on medical-grade flat-panel monitors cleared by the U.S. Food and Drug Administration, which were located at the facilities of the contract research organization, by using a standard viewing environment (OsiriX; Pixmeo, Geneva, Switzerland) in a setting with reduced ambient lighting. The window and level settings were adjustable to the observers’ preferences. A free-response marking paradigm was used in which observers freely marked the location of lesions without cues such as designated regions of interest, and they used the measurement tool to draw a line through the center of the nodule to indicate its diameter. They also rated each nodule candidate on a five-point likelihood scale of confidence that the noted opacity was a pulmonary nodule (score of 1, 1%–5% confidence; score of 2, 6%–20% confidence; score of 3, 21%–50% confidence; score of 4, 51%–95% confidence; and score of 5, >95% confidence).
Readers were blinded to the clinical case history and were presented with only the images for review. Images were presented in block fashion, with approximately 75 cases in each of two viewing sessions. Each viewing session consisted of two blocks, one block with PA and lateral conventional chest radiographs, followed by DE images of the same subjects; the other block consisted of tomosynthesis images, followed by DE images of the same subjects. The order of blocks and case presentations was varied randomly among readers. No images of a given subject were reviewed with both conventional chest radiography and tomosynthesis images in a given session, and the sessions were scheduled at least 1 month apart to avoid memory bias. No time limit was imposed on the image interpretation, although readers were encouraged to maintain a mean pace of 7 minutes per case. For the conventional chest radiographs, readers were instructed to mark a nodule on either the PA image or the lateral image, depending on which image better depicted the lesion. Readers had the opportunity to revise markings on the PA conventional chest radiographs and tomosynthesis images after viewing the subsequent DE images of each case; such revised markings were stored separately as interpretations for the DE modality.
Readers also recorded a per-case indication for each modality of what clinical management action, if any, they would take on the basis of nodule findings alone. This case management determination was based on Fleischner Society guidelines for clinical management of pulmonary nodules (43) and involved a decision about whether further imaging would be recommended. A score of 1 to 5 was also recorded to reflect the confidence of the reader in making the determination that the case requires follow-up (score of 1, 0%–5% confidence; score of 2, 6%–20% confidence; score of 3, 21%–50% confidence; score of 4, 51%–95% confidence; and score of 5, >95% confidence). Appendix E3 (online) lists the specific questions asked of the readers for both nodule localization and case management.
Methods for reconciliation of location in the PA orientation relative to that on the CT images and methods for quality control and adjudication of certain cases are provided in Appendix E2 (online).
Measures of Performance
Four combinations of modalities were evaluated in this study: (a) conventional chest radiography alone, (b) conventional chest radiography plus DE imaging, (c) tomosynthesis alone, and (d) tomosynthesis plus DE imaging. DE imaging was not compared alone, since in current clinical practice it is used only as an adjunct to conventional chest radiography. These combinations were compared in three primary and five secondary analyses listed herein, with details provided in Appendix E4 (online).
Primary analyses.—In the first primary analysis, accuracy of nodule detection and localization was measured on a nodule-level basis (44), as given by maximum lesion localization fraction (LLF), maximum nonlesion localization fraction (NLF), and weighted jackknife alternative FROC (JAFROC) figure of merit (FOM); these metrics were further analyzed according to nodule diameter groupings: nodules of all diameters (3–20 mm), nodules 3–4 mm, nodules larger than 4 mm to 6 mm, nodules larger than 6 mm to 8 mm, and nodules larger than 8 mm to 20 mm. A second primary analysis was conducted to measure accuracy of nodule detection on a case-level basis by ignoring localization, which included highest-rating inferred sensitivity, highest-rating inferred specificity, and highest-rating inferred area under the trapezoidal receiver operating characteristic (ROC) curve (AUC). The third primary analysis was conducted to evaluate the accuracy of the case management decision for each modality, including case management sensitivity, case management specificity, and case management AUC.
Secondary analyses.—Secondary analyses were performed by evaluating nodule localization FOMs on the following subsets of nodules: (a) nodules visible to the truth panel on the conventional chest radiographs (ie, nodules for which locations could be identified by the truth panel on conventional chest radiographs), (b) nodules not visible to the truth panel on the conventional chest radiographs (ie, nodules for which locations on conventional chest radiographs had to be estimated), (c) solid nodules, (d) calcified nodules, and (e) nodules located in the upper lung (both upper lobes).
Statistical Analysis
Statistical analysis for nodule localization was performed by using the JAFROC approach (44) (JAFROC software, version 4.2.1; www.devchakraborty.com). The algorithm used for significance testing, applicable to all FOMs in this study and implemented as a subroutine in JAFROC software, was developed at the University of Iowa (45–47). JAFROC software was appropriately modified to allow this algorithm to be used with non–area-based FOMs, such as maximum LLF and others. All significance testing was done at α (probability of a type I error) equal to .05. For a modality pairing to be declared significant, two conditions had to be met: The P value of the F test used to compare all modality pairings and the P value of the individual modality pairing both had to be less than α. For the individual size subgroup comparisons, a Bonferroni-corrected α value of .05/5/3, or .0033, was used to account for multiple comparisons (five diameter ranges, including 3–20 mm, and three measures of performance), which conservatively maintain the overall α value at .05, and a similar Bonferroni correction was applied to comparisons involving the secondary analysis. Statistical analysis for the case management question involved the use of standard multiple reader and multiple case analysis by using the Dorfman-Berbaum-Metz procedure (45) as updated by Hillis et al (46,47) at an α level of .05.
The levels of significance of reader and modality-reader effects were calculated by using the analysis of variance table output of JAFROC. To study the effect of reader expertise, the readers were separated into two groups: group 1 consisted of the two readers who were thoracic radiologists, and group 2 consisted of the remaining readers. The weighted JAFROC FOM was averaged over all readers within each group, and a bootstrap analysis was conducted to determine an empirical 95% confidence interval (CI) for the FOM difference between the two groups.
Results
Summary Statistics
The number of nodules visible to the truth panel on CT images in the various diameter categories are given in Table 1. In total, 115 of the 158 subjects had at least one nodule, with 516 total true nodules in the data set. When considering markings on the images from the modalities being studied, there was a mean of 0.409 lesion localization (ie, lesion-level “true-positive”) markings per abnormal case and mean of 0.287 and 0.393 nonlesion localization (ie, lesion-level “false-positive”) markings per normal and abnormal case, respectively, averaged over all readers and treatment modality combinations.
Table 1.
Summary Statistics for All Nodules Visible to the Truth Panel on CT Images
*Case findings were considered “normal” for a given diameter range if there were no nodules of that size; however, the subject may have a nodule(s) of a different size.
Effect of DE Imaging
There were no comparisons where any DE-augmented modality showed statistically significant improvement (P value was greater than the Bonferroni-corrected α value in all cases) in any FOM over the corresponding modality without DE imaging. For this reason, the subsequent result summaries will focus on comparisons of tomosynthesis versus conventional chest radiography.
Primary Analysis: Nodule Detection and Localization on a Nodule-Level Basis
The results for all nodules (3–20 mm) are summarized in Figure 1 and Table 2. Maximum LLF was found to be 3.55 times greater (P < .001), and the JAFROC FOM was found to be to be 1.20 times greater (P = .001) with tomosynthesis than with conventional chest radiography. Table 3 summarizes the results for lesion-level results analyzed according to nodule size. Maximum LLF on a per-lesion basis was significantly higher for tomosynthesis than for conventional chest radiography in all diameter groups, ranging from 2.14 times better (P < .001) with nodules larger than 8 mm to 20 mm to 7.50 times better (P < .001) with nodules larger than 4 mm to 6 mm. As a measure of overall accuracy, JAFROC FOM on a nodule-level basis showed significant improvement with tomosynthesis over conventional chest radiography at the larger than 6 mm to 8 mm level (1.16 times, P < .001) and larger than 8 mm to 20 mm level (1.28 times, P < .001) nodule diameters. Figure 2 summarizes the detection sensitivity on a nodule-level basis for tomosynthesis and conventional chest radiography.
Figure 1a:
Nodule-level localization results for all nodules (3–20-mm diameter). (a) Bar graphs show raw FOMs for each modality combination, and (b) box and whisker plots show the differences of given FOMs for each paired modality comparison. Error bars indicate 95% CIs. Modality combinations are as follows: 1, conventional chest radiography; 2, conventional chest radiography plus DE imaging; 3, tomosynthesis; and 4, tomosynthesis plus DE imaging. MAX_LLF = maximum LLF, MAX_NLF = maximum NLF.
Table 2.
Summary of Nodule-Level Localization Results for All Nodules (3–20-mm diameter)
Note.—Shown are the FOM values averaged over readers and the ratios of the FOM of the modality indicated by the row index, divided by that indicated by the column index. For example, maximum LLF was a factor of 3.55 times larger for tomosynthesis than for conventional chest radiography, and JAFROC FOM was 1.20 times larger. Numbers in parentheses are the P values of the individual modality comparison t tests (where P values are indicated, the P value of the overall F test is also smaller than the Bonferroni-corrected α).
*Significant ratios (P values less than α = .05/5/3 = .0033 after Bonferroni correction).
Table 3.
Summary of the Lesion-Level Results from FROC Data for Nodules in Different Size Ranges
Note.—Shown are the FOM values averaged over readers and the ratios of the FOM of the modality indicated by the row index, divided by that indicated by the column index. The P value display convention is the same as in Table 2.
*Significant values (individual comparison α = .05/5/3 = .0033).
Figure 2:
Bar graph shows the detection sensitivity on a per-nodule basis as a function of nodule size. Results show sensitivity as determined by maximum LLF (MAX LLF) for all nodules visible to the truth panel on CT images. All differences between conventional chest radiography (CXR) and tomosynthesis (TS) are significant (P < .001 for nodules 3–20 mm; P = .002 for nodules 3–4 mm; P < .001 for nodules >4 mm to 6 mm; P < .001 for nodules >6 mm to 8 mm; and P < .001 for nodules >8 mm to 20 mm). LAT = lateral.
Figure 1b:
Nodule-level localization results for all nodules (3–20-mm diameter). (a) Bar graphs show raw FOMs for each modality combination, and (b) box and whisker plots show the differences of given FOMs for each paired modality comparison. Error bars indicate 95% CIs. Modality combinations are as follows: 1, conventional chest radiography; 2, conventional chest radiography plus DE imaging; 3, tomosynthesis; and 4, tomosynthesis plus DE imaging. MAX_LLF = maximum LLF, MAX_NLF = maximum NLF.
When considering all evaluations of nodule localization accuracy, there were no significant differences in measures of specificity for any of the comparisons in any of the size groups (P value was greater than the Bonferroni-corrected α in all cases).
Figure 3 illustrates the improvement in nodule detection with tomosynthesis relative to conventional chest radiography and DE imaging.
Figure 3a:
Improved nodule visibility with tomosynthesis. A 7-mm lower-left-lobe CT-confirmed nodule was not visible on the (a) conventional chest radiograph, (b) DE tissue image, or (c) DE bone image but was visible on the (d) tomosynthesis image (arrow). (e) Axial CT images obtained with two different window settings (mediastinal window and lung window) show the nodule (arrow on the right image).
Figure 3b:
Improved nodule visibility with tomosynthesis. A 7-mm lower-left-lobe CT-confirmed nodule was not visible on the (a) conventional chest radiograph, (b) DE tissue image, or (c) DE bone image but was visible on the (d) tomosynthesis image (arrow). (e) Axial CT images obtained with two different window settings (mediastinal window and lung window) show the nodule (arrow on the right image).
Figure 3c:
Improved nodule visibility with tomosynthesis. A 7-mm lower-left-lobe CT-confirmed nodule was not visible on the (a) conventional chest radiograph, (b) DE tissue image, or (c) DE bone image but was visible on the (d) tomosynthesis image (arrow). (e) Axial CT images obtained with two different window settings (mediastinal window and lung window) show the nodule (arrow on the right image).
Figure 3d:
Improved nodule visibility with tomosynthesis. A 7-mm lower-left-lobe CT-confirmed nodule was not visible on the (a) conventional chest radiograph, (b) DE tissue image, or (c) DE bone image but was visible on the (d) tomosynthesis image (arrow). (e) Axial CT images obtained with two different window settings (mediastinal window and lung window) show the nodule (arrow on the right image).
Figure 3e:
Improved nodule visibility with tomosynthesis. A 7-mm lower-left-lobe CT-confirmed nodule was not visible on the (a) conventional chest radiograph, (b) DE tissue image, or (c) DE bone image but was visible on the (d) tomosynthesis image (arrow). (e) Axial CT images obtained with two different window settings (mediastinal window and lung window) show the nodule (arrow on the right image).
Primary Analysis: Nodule Detection on a Case-Level Basis by Ignoring Localization
The results are summarized in Figure 4 and Table 4. Highest-rating inferred metrics from the free-response data showed that highest-rating inferred sensitivity was 1.49 times greater (P < .001), and the highest-rating inferred AUC was 1.21 times greater (P < .001) with tomosynthesis than with conventional chest radiography. Both nodule-level and case-level analyses showed roughly comparable improvements in overall nodule detection accuracy (ie, JAFROC FOM and highest-rating inferred AUC, respectively) with about 20% improvement for tomosynthesis over conventional chest radiography. There were no significant differences in nodule- or case-level specificity, as given by maximum NLF and highest-rating inferred specificity, respectively, between any of the four modality combinations when considering the whole complement of nodules as a group.
Figure 4a:
Case-level results inferred from FROC data for all nodules (3–20-mm diameter). (a) Bar graphs show the raw FOMs for each modality combination, and (b) box and whisker plots show the differences in given FOMs for each paired modality comparison. Error bars indicate 95% CIs. Modality combinations are as follows: 1, conventional chest radiography; 2, conventional chest radiography plus DE imaging; 3, tomosynthesis; and 4, tomosynthesis plus DE imaging. HR_AUC = highest-rating inferred AUC, HR_SE = highest-rating inferred sensitivity, HR_SP = highest-rating inferred specificity.
Table 4.
Summary of the Case-Level Results Inferred from FROC Data for All Nodules (3–20-mm Diameter)
Note.—Shown are the FOM values averaged over readers and the ratios of the FOM of the modality indicated by the row index, divided by that indicated by the column index. The P value display convention is the same as in Table 2.
*Significant values (individual comparison α = .05/3 = .017).
Figure 4b:
Case-level results inferred from FROC data for all nodules (3–20-mm diameter). (a) Bar graphs show the raw FOMs for each modality combination, and (b) box and whisker plots show the differences in given FOMs for each paired modality comparison. Error bars indicate 95% CIs. Modality combinations are as follows: 1, conventional chest radiography; 2, conventional chest radiography plus DE imaging; 3, tomosynthesis; and 4, tomosynthesis plus DE imaging. HR_AUC = highest-rating inferred AUC, HR_SE = highest-rating inferred sensitivity, HR_SP = highest-rating inferred specificity.
Primary Analysis: Case Management Decision
Table 5 summarizes the results in the measures associated with case management accuracy, and Figure 5 shows case management ROC curves for the four modality combinations. The AUC in this ROC analysis was significantly higher for tomosynthesis than for conventional chest radiography, both with DE imaging (1.16 times, P = .001) and without DE imaging (1.23 times, P < .001), meaning that overall case management accuracy was higher with tomosynthesis than with conventional chest radiography, with or without DE imaging. There were no significant differences in case management specificity in any of the modality comparisons.
Table 5.
Summary of the Results for the Case Management ROC Analysis
Note.—Data are FOM values averaged over readers and the ratios of the FOM of the modality indicated by the row index, divided by that indicated by the column index. The P value display convention is the same as in Table 2.
*Significant values (individual comparison α = 0.05/3 = 0.017).
Figure 5:
Trapezoidal ROC curves for the case management data averaged over readers. Modality combinations are as follows: 1, conventional chest radiography (CXR); 2, conventional chest radiography plus DE imaging; 3, tomosynthesis (TS); and 4, tomosynthesis plus DE imaging. The lower two curves are conventional chest radiography with and without DE imaging, and the upper two curves are tomosynthesis with and without DE imaging; the darker curve in each pair includes DE imaging. (Differences in AUCs 3 > 1, 4 > 1, 3 > 2, and 4 > 2 were significant. The differences 2 > 1 and 3 > 4 were not significant.)
Secondary Analyses
Table 6 lists the summary statistics for the categories of nodules. By evaluating only the nodules visible to the expert truth panel on conventional chest radiographs, maximum LLF was 2.40 times larger (P < .001), highest-rating inferred AUC was 1.18 times larger (P = .001), and JAFROC area was 1.30 times larger (P < .001) with tomosynthesis than with conventional chest radiography when evaluating all nodules together (3–20 mm). When considering the more difficult situation of nodules not visible to the expert panel on conventional chest radiographs, maximum LLF and highest-rating inferred AUC were 9.06 (P < .001) and 1.20 (P < .001) times larger with tomosynthesis than with conventional chest radiography, respectively (for all nodules). Thus, the sensitivity for detection of all nodules (3–20 mm) on a per-nodule basis, given by maximum LLF, was better for tomosynthesis than conventional chest radiography, regardless of the subtlety of the lesion, although the improvement was greatest for the most subtle lesions (ie, the ones not visible to the expert panel on conventional chest radiographs). The improvement in overall accuracy per case, as inferred by the AUC calculated with the highest-rated lesions in each case (ie, highest-rating inferred AUC), showed improvements of roughly 20%, regardless of the lesion subtlety (when considering all nodules).
Table 6.
Summary Statistics for All Nodules (3–20 mm) Visible to the Truth Panel on CT Images, Split according to Nodule Characteristics
*Case findings were considered “normal” for a given category if there were no nodules in that category; however, the subject may have nodule(s) in other categories.
Several significant differences were noted in secondary analyses on the basis of type or location of nodules. For the subset of solid nodules, per-nodule sensitivity (maximum LLF) and per-case highest-rating inferred sensitivity were 3.45 times larger (P < .001) and 1.51 times larger (P < .001) for tomosynthesis than for conventional chest radiography, respectively (all nodules). Overall accuracy measures were also significantly higher with tomosynthesis than conventional chest radiography; JAFROC FOM and highest-rating inferred AUC were 1.14 (P = .001) and 1.19 (P < .001) times larger, respectively, for the subset of solid nodules (all nodules). Nodules located in the upper lungs showed 2.65 times larger maximum LLF (P < .001) and 1.16 times larger highest-rating inferred AUC (P < .001) on the tomosynthesis images than on the conventional chest radiographs (all nodules). No significant differences in the evaluated metrics were found in the subset of calcified nodules across modalities. None of the specificity comparisons were significantly different from zero.
Reader and Modality-Reader Effects
By using the weighted JAFROC FOM for all nodules, the reader effect was highly significant (P < .001), but the modality-reader effect did not reach significance (P = .232). When the weighted JAFROC FOM difference between the two reader groups (thoracic and other radiologists) was analyzed for individual modalities, none of the differences were significant (P > .05); however, when the weighted JAFROC FOM was further averaged for each group over all modalities, the thoracic radiologists showed significantly improved performance over the other radiologists (FOM difference of 0.055; 95% CI: 0.004, 0.072). Similar analysis applied to the case management decisions demonstrated insignificant (P > .05) reader and modality-reader effects.
Discussion
This study was a binational multi-institutional evaluation of chest tomosynthesis in a prospectively designed, randomized, case-controlled study, with readers blinded to ground truth findings. It included prospective evaluation of all nodules (with CT confirmation) in eligible subjects rather than just suspected nodules noted on conventional chest radiographs (18,21), and it therefore reflects the type and range of nodules encountered clinically, including those not visible on conventional chest radiographs. In this study, radiologist readers were also used from a wider range of specialty training than those in many previously published reports in which trained thoracic radiologists were predominantly used (11,12,15–18,20,21,48,49). Analysis of the effect of reader experience by using the most sensitive measure (weighted JAFROC FOM) showed that there was no significant difference between the performance of the thoracic and nonthoracic radiologists for any of the individual modalities. Only when the measures were averaged over all of the modalities did we see the expected result that trained thoracic radiologists significantly outperformed the nonthoracic radiologists.
The findings of this study are in agreement with those of previous studies of chest tomosynthesis that have all shown a significant improvement in pulmonary nodule detection with tomosynthesis relative to conventional chest radiography. It is noteworthy that these studies have consistently shown improvement with tomosynthesis given the range of subject populations, reader experience, and methods used (Appendix E5 [online]).
This study demonstrated significant improvements in nodule detection sensitivity (maximum LLF) and accuracy (highest-rating inferred AUC) with tomosynthesis relative to conventional chest radiography in subcategories both of nodules visible to the truth panel on conventional chest radiographs and nodules not visible to the truth panel on conventional chest radiographs. It should be noted that the secondary analyses involving nodules visible or not visible to the truth panel on conventional chest radiographs presented challenges in terms of defining which markings to include as nonlesion localizations. Ideally, in such subcategories of lesions, a separate blinded image review would have been conducted to include only nodules within that category (ie, readers would have separately interpreted images of nodules marked as visible or not visible on conventional chest radiographs by the truth panel, and therefore, any nonlesion localization would have included only those markings for which readers thought that the truth panel would have seen or not seen lesions on conventional chest radiographs). Such a separate set of readings was impractical, and the readers interpreted images from all cases without regard to any supposition about the visibility to the truth panel. Therefore, the nonlesion localizations in both the subtle and not-subtle categories included the same full set of nonlesion markings. This practical limitation means that there is potential bias in the FOMs involving nonlesion localizations for these two categories, although it is likely that relative measures of performance between modalities would be less affected by any such bias.
Secondary analyses also yielded significant improvements in performance with tomosynthesis relative to conventional chest radiography for categories of solid nodules and nodules in the upper lung. The finding of improved performance with tomosynthesis over conventional chest radiography for nodules in the upper lobes is important because these nodules are often obscured on conventional chest radiographs by bone (rib, clavicle, and scapula), and previous studies have shown that nodules in these obscured areas account for most of the cancers missed at conventional chest radiograph interpretation (50). With regard to calcified nodules, no significant difference was found for any of the comparisons in this study; however, the number of calcified nodules (n = 52) was small when compared with the total number of nodules (n = 516), largely owing to the study design, which did not specifically target inclusion of calcified nodules. A larger study of calcified nodules will be required to address specific comparisons of relative performance with that type of nodule.
There are several limitations to this study. First, there were too few subsolid nodules to draw conclusions about the relative performance of tomosynthesis, DE imaging, and conventional chest radiography for ground-glass opacity nodules (n = 40) or other subsolid nodules (n = 44) or for other lesions, such as pleural plaques. Second, in our study, we did not evaluate nodules on the basis of whether they were proven to be malignant. Third, an enriched data set of abnormal cases relative to normal cases was used that is different than the prevalence of nodules encountered in clinical practice. Fourth, because the exclusion criteria eliminated any subjects with potentially confounding structures, we do not know from this study alone how patients with central venous catheters, pacemakers, pleural fluid, pneumonia, and other similar hardware or conditions would benefit from tomosynthesis for the detection of pulmonary nodules. Additionally, five subjects with motion artifacts were excluded from this study. Other reports have indicated that patient motion can substantially affect the performance of tomosynthesis for nodule detection (49,51,52), and therefore, the effect of motion artifacts warrants further investigation. Last, we used our own population to determine the threshold distance for a correct lesion, which could affect the relative results had a different distance threshold been selected.
Several clinical implications can be drawn from this study. First, this study suggests improved performance with tomosynthesis over conventional chest radiography, regardless of the specialty training of radiologists, implying that results of the current study are likely to describe a lower bound on the performance that could be expected in general use (ie, general radiologists across multiple institutions). Second, the greatest improvement in detection sensitivity is in the nodule size range right at the threshold for follow-up according to Fleischner Society guidelines, which might imply that tomosynthesis is well tuned to the clinical tasks associated with nodule management. Third, while nodule detection and case management questions are certainly related, the questions were treated independently to better clarify the role improved detection performance may play in overall clinical management decisions. Tomosynthesis provides superior performance to conventional chest radiography not just when considering nodules in isolation but also when considering cases as a whole and leads to better case management decisions than conventional chest radiography alone.
In conclusion, this study shows that tomosynthesis, when used in a clinical radiology setting with both thoracic and nonthoracic specialists, can improve the detection of pulmonary nodules and case management decisions compared with conventional chest radiography. These results should augment the findings of previous studies in which only thoracic radiologists were used to suggest that tomosynthesis may have clinical value for a wide range of radiologist users in the detection and management of pulmonary nodules.
Advances in Knowledge
■ Maximum lesion localization fraction (LLF) was higher for tomosynthesis than for conventional chest radiography in all nodule diameter categories (for all nodules, maximum LLF was 0.135 for tomosynthesis and 0.038 for conventional chest radiography; ratio was 3.55-fold greater, P < .001; 95% confidence interval [CI]: 2.96, 4.15).
■ Case-level sensitivity for nodule detection was higher with tomosynthesis (0.614 sensitivity) than with conventional chest radiography (0.412 sensitivity) for all nodules (ratio was 1.49-fold greater, P < .001; 95% CI: 1.25, 1.73).
■ Case management decisions showed better overall diagnostic accuracy with tomosynthesis (area under the receiver operating characteristic curve, 0.719) than with conventional chest radiography (0.584; ratio was 1.23-fold greater, P < .001; 95% CI: 1.15, 1.32) and no significant difference in specificity (P = .441).
Implication for Patient Care
■ Tomosynthesis provided improved clinical performance over chest radiography for pulmonary nodule detection and case management for patients who met the inclusion criteria in this study, including such factors as not having objects in or around the lungs that might produce artifacts, not having active lung or pleural disease that could obscure pulmonary nodules, and being in good enough physical condition to stand motionless and suspend respiration during the imaging procedure.
APPENDIX
Acknowledgments
Acknowledgments
The authors gratefully acknowledge the technical staff who collected the clinical images: Brenda Prince, RT(R), Duke University Medical Center; Karen Carter, MS, RT(R), Luanne Newman, RT(R), and Eileen Schmidtke, RT(R), University of Michigan; Sally Friend, MPH, University of Washington; and Lena Björneld, RN, PhLic, Gull-Britt Möhlenbrock, RN, and Helén Milde, RN, Sahlgrenska University Hospital. The authors also acknowledge the readers who participated anonymously in the observer study. Xuetong Zhai, MS, provided modifications to the JAFROC software for this project.
Received March 5, 2015; revision requested April 21; revision received September 3; accepted September 29; final version accepted May 3, 2016.
This study was supported by GE Healthcare and was implemented through sponsored research agreements with all four clinical sites. Acquisition and tabulation of data were provided by a contract with the American College of Radiology Image Metrix, Philadelphia, Pa. Funding for developing the JAFROC method by D.P.C. was supported by grants from the U.S. Department of Health and Human Services, National Institutes of Health (R01-EB005243 and R01-EB008688).
Disclosures of Conflicts of Interest: J.T.D. Activities related to the present article: author received a grant from GE Healthcare. Activities not related to the present article: disclosed no relevant relationships. Other relationships: author has a U.S. patent, a joint inventorship between GE and Duke University. H.P.M. Activities related to the present article: author received payment from ACR Image Metrix for consulting and travel expenses. Activities not related to the present article: disclosed no relevant relationships. Other relationships: disclosed no relevant relationships. J.M.S. Activities related to the present article: disclosed no relevant relationships. Activities not related to the present article: disclosed no relevant relationships. Other relationships: author is an employee of GE Healthcare. D.P.C. Activities related to the present article: author received payment from ACR Image Metrix for consulting. Activities not related to the present article: disclosed no relevant relationships. Other relationships: disclosed no relevant relationships. E.A.K. Activities related to the present article: author received payment from ACR Image Metrix for consulting and travel expenses. Activities not related to the present article: disclosed no relevant relationships. Other relationships: disclosed no relevant relationships. G.P.R. Activities related to the present article: author received payment from ACR Image Metrix for consulting. Activities not related to the present article: disclosed no relevant relationships. Other relationships: disclosed no relevant relationships. J.V. disclosed no relevant relationships. M.B. disclosed no relevant relationships.
Abbreviations:
- AUC
- area under the trapezoidal ROC curve
- CI
- confidence interval
- DE
- dual energy
- FOM
- figure of merit
- FROC
- free-response ROC
- JAFROC
- jackknife alternative FROC
- LLF
- lesion localization fraction
- NLF
- nonlesion localization fraction
- PA
- posteroanterior
- ROC
- receiver operating characteristic
References
- 1.Niklason LT, Christian BT, Niklason LE, et al. Digital tomosynthesis in breast imaging. Radiology 1997;205(2):399–406. [DOI] [PubMed] [Google Scholar]
- 2.Poplack SP, Tosteson TD, Kogel CA, Nagy HM. Digital breast tomosynthesis: initial experience in 98 women with abnormal digital screening mammography. AJR Am J Roentgenol 2007;189(3):616–623. [DOI] [PubMed] [Google Scholar]
- 3.Rafferty EA, Park JM, Philpotts LE, et al. Assessing radiologist performance using combined digital mammography and breast tomosynthesis compared with digital mammography alone: results of a multicenter, multireader trial. Radiology 2013;266(1):104–113. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Gazaille RE, 3rd, Flynn MJ, Page W, 3rd, Finley S, van Holsbeeck M. Technical innovation: digital tomosynthesis of the hip following intra-articular administration of contrast. Skeletal Radiol 2011;40(11):1467–1471. [DOI] [PubMed] [Google Scholar]
- 5.Duryea J, Dobbins JT, 3rd, Lynch JA. Digital tomosynthesis of hand joints for arthritis assessment. Med Phys 2003;30(3):325–333. [DOI] [PubMed] [Google Scholar]
- 6.Hayashi D, Xu L, Roemer FW, et al. Detection of osteophytes and subchondral cysts in the knee with use of tomosynthesis. Radiology 2012;263(1):206–215. [DOI] [PubMed] [Google Scholar]
- 7.Machida H, Yuhara T, Tamura M, et al. Radiation dose of digital tomosynthesis for sinonasal examination: comparison with multi-detector CT. Eur J Radiol 2012;81(6):1140–1145. [DOI] [PubMed] [Google Scholar]
- 8.Yoo JY, Chung MJ, Choi B, et al. Digital tomosynthesis for PNS evaluation: comparisons of patient exposure and image quality with plain radiography. Korean J Radiol 2012;13(2):136–143. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Mermuys K, De Geeter F, Bacher K, et al. Digital tomosynthesis in the detection of urolithiasis: diagnostic performance and dosimetry compared with digital radiography with MDCT as the reference standard. AJR Am J Roentgenol 2010;195(1):161–167. [DOI] [PubMed] [Google Scholar]
- 10.Dobbins JT, 3rd, Godfrey DJ. Digital x-ray tomosynthesis: current state of the art and clinical potential. Phys Med Biol 2003;48(19):R65–R106. [DOI] [PubMed] [Google Scholar]
- 11.Dobbins JT, 3rd, McAdams HP, Song JW, et al. Digital tomosynthesis of the chest for lung nodule detection: interim sensitivity results from an ongoing NIH-sponsored trial. Med Phys 2008;35(6):2554–2557. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Vikgren J, Zachrisson S, Svalkvist A, et al. Comparison of chest tomosynthesis and chest radiography for detection of pulmonary nodules: human observer study of clinical cases. Radiology 2008;249(3):1034–1041. [DOI] [PubMed] [Google Scholar]
- 13.Dobbins JT, 3rd, McAdams HP. Chest tomosynthesis: technical principles and clinical update. Eur J Radiol 2009;72(2):244–251. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Kim EY, Chung MJ, Lee HY, Koh WJ, Jung HN, Lee KS. Pulmonary mycobacterial disease: diagnostic performance of low-dose digital tomosynthesis as compared with chest radiography. Radiology 2010;257(1):269–277. [DOI] [PubMed] [Google Scholar]
- 15.Quaia E, Baratella E, Cioffi V, et al. The value of digital tomosynthesis in the diagnosis of suspected pulmonary lesions on chest radiography: analysis of diagnostic accuracy and confidence. Acad Radiol 2010;17(10):1267–1274. [DOI] [PubMed] [Google Scholar]
- 16.Jung HN, Chung MJ, Koo JH, Kim HC, Lee KS. Digital tomosynthesis of the chest: utility for detection of lung metastasis in patients with colorectal cancer. Clin Radiol 2012;67(3):232–238. [DOI] [PubMed] [Google Scholar]
- 17.Yamada Y, Jinzaki M, Hasegawa I, et al. Fast scanning tomosynthesis for the detection of pulmonary nodules: diagnostic performance compared with chest radiography, using multidetector-row computed tomography as the reference. Invest Radiol 2011;46(8):471–477. [DOI] [PubMed] [Google Scholar]
- 18.Quaia E, Baratella E, Cernic S, et al. Analysis of the impact of digital tomosynthesis on the radiological investigation of patients with suspected pulmonary lesions on chest radiography. Eur Radiol 2012;22(9):1912–1922. [DOI] [PubMed] [Google Scholar]
- 19.Johnsson ÅA, Fagman E, Vikgren J, et al. Pulmonary nodule size evaluation with chest tomosynthesis. Radiology 2012;265(1):273–282. [DOI] [PubMed] [Google Scholar]
- 20.Terzi A, Bertolaccini L, Viti A, et al. Lung cancer detection with digital chest tomosynthesis: baseline results from the observational study SOS. J Thorac Oncol 2013;8(6):685–692. [DOI] [PubMed] [Google Scholar]
- 21.Quaia E, Baratella E, Poillucci G, Kus S, Cioffi V, Cova MA. Digital tomosynthesis as a problem-solving imaging technique to confirm or exclude potential thoracic lesions based on chest x-ray radiography. Acad Radiol 2013;20(5):546–553. [DOI] [PubMed] [Google Scholar]
- 22.Lee G, Jeong YJ, Kim KI, et al. Comparison of chest digital tomosynthesis and chest radiography for detection of asbestos-related pleuropulmonary disease. Clin Radiol 2013;68(4):376–382. [DOI] [PubMed] [Google Scholar]
- 23.Hwang HS, Chung MJ, Lee KS. Digital tomosynthesis of the chest: comparison of patient exposure dose and image quality between standard default setting and low dose setting. Korean J Radiol 2013;14(3):525–531. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24.Ishigaki T, Sakuma S, Ikeda M. One-shot dual-energy subtraction chest imaging with computed radiography: clinical evaluation of film images. Radiology 1988;168(1):67–72. [DOI] [PubMed] [Google Scholar]
- 25.Oestmann JW, Greene R, Rhea JT, et al. “Single-exposure” dual energy digital radiography in the detection of pulmonary nodules and calcifications. Invest Radiol 1989;24(7):517–521. [DOI] [PubMed] [Google Scholar]
- 26.Ergun DL, Mistretta CA, Brown DE, et al. Single-exposure dual-energy computed radiography: improved detection and processing. Radiology 1990;174(1):243–249. [DOI] [PubMed] [Google Scholar]
- 27.Sone S, Kasuga T, Sakai F, et al. Chest imaging with dual-energy subtraction digital tomosynthesis. Acta Radiol 1993;34(4):346–350. [PubMed] [Google Scholar]
- 28.Kelcz F, Zink FE, Peppler WW, Kruger DG, Ergun DL, Mistretta CA. Conventional chest radiography vs dual-energy computed radiography in the detection and characterization of pulmonary nodules. AJR Am J Roentgenol 1994;162(2):271–278. [DOI] [PubMed] [Google Scholar]
- 29.Ricke J, Fischbach F, Freund T, et al. Clinical results of CsI-detector-based dual-exposure dual energy in chest radiography. Eur Radiol 2003;13(12):2577–2582. [DOI] [PubMed] [Google Scholar]
- 30.Fischbach F, Freund T, Röttgen R, Engert U, Felix R, Ricke J. Dual-energy chest radiography with a flat-panel digital detector: revealing calcified chest abnormalities. AJR Am J Roentgenol 2003;181(6):1519–1524. [DOI] [PubMed] [Google Scholar]
- 31.Uemura M, Miyagawa M, Yasuhara Y, et al. Clinical evaluation of pulmonary nodules with dual-exposure dual-energy subtraction chest radiography. Radiat Med 2005;23(6):391–397. [PubMed] [Google Scholar]
- 32.Rühl R, Wozniak MM, Werk M, et al. CsI-detector-based dual-exposure dual energy in chest radiography for lung nodule detection: results of an international multicenter trial. Eur Radiol 2008;18(9):1831–1839. [DOI] [PubMed] [Google Scholar]
- 33.Kashani H, Varon CA, Paul NS, et al. Diagnostic performance of a prototype dual-energy chest imaging system ROC analysis. Acad Radiol 2010;17(3):298–308. [DOI] [PubMed] [Google Scholar]
- 34.Sabol JM, Avinash GB, Nicolas F, Claus B, Zhao J, Dobbins JT, III. Development and characterization of a dual-energy subtraction imaging system for chest radiography based on CsI: Tl amorphous silicon flat-panel technology. Proc SPIE 2001;4320:399–408. [Google Scholar]
- 35.Dobbins JT, 3rd, Samei E, Chotas HG, et al. Chest radiography: optimization of x-ray spectrum for cesium iodide-amorphous silicon flat-panel detector. Radiology 2003;226(1):221–230. [DOI] [PubMed] [Google Scholar]
- 36.Sabol JM. A Monte Carlo estimation of effective dose in chest tomosynthesis. Med Phys 2009;36(12):5480–5487. [DOI] [PubMed] [Google Scholar]
- 37.International Electrotechnical Commission . Medical electrical equipment—part 1: general requirements for basic safety and essential performance. IEC 60601-1. 3rd ed. Geneva, Switzerland: International Electrotechnical Commission, 2005. [Google Scholar]
- 38.U.S. FDA Code of Federal Regulations, Title 21 (Apr 2012).
- 39.Bunch PC, Hamilton JF, Sanderson GK, Simmons AH. A free-response approach to the measurement and characterization of radiographic-observer performance. J Appl Photogr Eng 1978;4:166–171. [Google Scholar]
- 40.Chakraborty DP. New developments in observer performance methodology in medical imaging. Semin Nucl Med 2011;41(6):401–418. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 41.Chakraborty DP, Berbaum KS. Observer studies involving detection and localization: modeling, analysis, and validation. Med Phys 2004;31(8):2313–2330. [DOI] [PubMed] [Google Scholar]
- 42.Chakraborty DP. Validation and statistical power comparison of methods for analyzing free-response observer performance studies. Acad Radiol 2008;15(12):1554–1566. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 43.MacMahon H, Austin JH, Gamsu G, et al. Guidelines for management of small pulmonary nodules detected on CT scans: a statement from the Fleischner Society. Radiology 2005;237(2):395–400. [DOI] [PubMed] [Google Scholar]
- 44.Chakraborty DP. JAFROC: software for planning and analyzing FROC studies. http://www.devchakraborty.com. Published 2013. Accessed June 1, 2016.
- 45.Dorfman DD, Berbaum KS, Metz CE. Receiver operating characteristic rating analysis. Generalization to the population of readers and patients with the jackknife method. Invest Radiol 1992;27(9):723–731. [PubMed] [Google Scholar]
- 46.Hillis SL. A comparison of denominator degrees of freedom methods for multiple observer ROC analysis. Stat Med 2007;26(3):596–619. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 47.Hillis SL, Berbaum KS, Metz CE. Recent developments in the Dorfman-Berbaum-Metz procedure for multireader ROC study analysis. Acad Radiol 2008;15(5):647–661. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 48.Zachrisson S, Vikgren J, Svalkvist A, et al. Effect of clinical experience of chest tomosynthesis on detection of pulmonary nodules. Acta Radiol 2009;50(8):884–891. [DOI] [PubMed] [Google Scholar]
- 49.Kim SM, Chung MJ, Lee KS, et al. Digital tomosynthesis of the thorax: the influence of respiratory motion artifacts on lung nodule detection. Acta Radiol 2013;54(6):634–639. [DOI] [PubMed] [Google Scholar]
- 50.Quekel LG, Kessels AG, Goei R, van Engelshoven JM. Miss rate of lung cancer on the chest radiograph in clinical practice. Chest 1999;115(3):720–724. [DOI] [PubMed] [Google Scholar]
- 51.Johnsson ÅA, Vikgren J, Svalkvist A, et al. Overview of two years of clinical experience of chest tomosynthesis at Sahlgrenska University Hospital. Radiat Prot Dosimetry 2010;139(1-3):124–129. [DOI] [PubMed] [Google Scholar]
- 52.Johnsson ÅA, Vikgren J, Båth M. Chest tomosynthesis: technical and clinical perspectives. Semin Respir Crit Care Med 2014;35(1):17–26. [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.