Abstract
This study aimed to compare the diagnostic effectiveness of computer-aided detection (CAD) software (OnGuard™ 5.2) in combination with hardware-based bone suppression (dual-energy subtraction radiography (DESR)), software-based bone suppression (SoftView™, version 2.4), and standard posteroanterior images with no bone suppression. A retrospective pilot study compared the diagnostic performance of two commercially available methods of bone suppression when used with commercially available CAD software. Chest images from 27 patients with computed tomography (CT) and pathology-proven malignant pulmonary nodules (8–34 mm) and 25 CT-negative patient controls were used for analysis. The Friedman, McNemar, and chi-square tests were used to compare diagnostic performance and the kappa statistic was used to evaluate method agreement. The average number of regions of interest and false-positives per image identified by CAD were not found to be significantly different regardless of the bone suppression methods evaluated. Similarly, the sensitivity, specificity, and test efficiency were not found to be significantly different. Agreement between the methods was between poor and excellent. The accuracy of CAD (OnGuard™, version 5.2) is not statistically different with either DESR or SoftView™ (version 2.4) bone suppression technology in digital chest images for pulmonary nodule identification. Low values for sensitivity (<80 %) and specificity (<50 %) may limit their utility for clinical radiology.
Keywords: Computer-aided detection, Pulmonary nodule, Digital chest radiography, Image analysis method, Dual-energy subtraction
Introduction
Chest radiography is one of the most commonly used forms of radiologic examination. Several authors have suggested that computer-aided detection (CAD) may help radiologists to detect pulmonary nodules. However, high numbers of false-positives identified by early versions of CAD software on standard posteroanterior (PA) chest images limited the clinical utility of this technology [1–3]. Meziane et al. [4] indicated that CAD software improved recall rates and diagnostic accuracy particularly in cases with small pulmonary nodules and inexperienced readers.
De Boo et al. questioned the clinical utility of CAD when used with chest images. The authors suggested that while CAD improved the sensitivity of nodule detection, it did not improve overall cancer detection rate since differentiation between subtle true and false positives was difficult even with CAD annotation [5].
In support of this position, a recent systematic review of the literature suggested that CAD is useful for breast cancer diagnosis, but provided little assistance for diagnosis of lung cancer [6]. Other researchers [7, 8] determined that a large proportion of the false-positives and missed lung cancer cases occurred because of bony structures in the chest, in particular clavicles and rib crossings.
The literature on the subject suggests that the use of bone suppression improves the diagnostic accuracy of digital chest radiography. Many authors advocate the suppression of ribs and clavicles in digital chest images to improve malignant nodule detection by (CAD) the software. Currently, two methods exist to suppress bone and other calcified structures in digital chest radiographs. One method is hardware based, dual-energy subtraction radiography (DESR, GE Healthcare), and the other method is software based (SoftView™/ClearRead™, Riverain Medical).
Oda et al. [9] determined that the use of DESR significantly improved the diagnostic performance of radiologists to detect pulmonary nodules. Other authors, [10–12] indicated that hardware-based bone suppression (DESR) removed the presence of bony structures in digital chest radiographs and significantly improved the sensitivity of CAD performance in pathology-proven cases with pulmonary nodules while reducing the false-positive rate. Similarly, Freedman et al. [13] suggested that software-based removal of bony structures also improved the rate of lung cancer detection. The results of these studies suggest that both methods of bone suppression when used with digital chest images improve the diagnostic efficacy of CAD by improving sensitivity and reducing the number of false-positives.
While the aforementioned studies suggest that CAD in combination with bone suppressed images improves the diagnostic accuracy of pulmonary nodule detection, DESR and SoftView™/ClearRead™ have not been directly compared. Further, while many earlier versions of CAD software from the same manufacturer (OnGuard™) have been studied [14], it has not been determined if the newer versions of CAD software (OnGuard™ 5.2) would improve the diagnostic efficiency of the incorporated CAD software when used with an already bone suppressed image provided by DESR.
Recently, Riverain Technologies changed the name and combinations of their products. The bone suppression software formerly known as SoftView™ version 2.4 is now known as ClearRead™. The combination bone suppression and CAD software product formerly known as Unison™ (a combination of SoftView™ version 2.4 and OnGuard™ version 5.2) is now known as ClearRead + Detect™. The stand-alone version of CAD software formerly known as OnGuard™ (versions 1.0–5.2) is no longer commercially available but was utilized for this study as OnGuard™ version 5.2. Unlike previous versions, OnGuard™ version 5.2 incorporated a bone suppression algorithm similar to SoftView™ to better identify areas of interest prior to applying the CAD markings to the radiograph.
The purpose of this study was to compare the diagnostic effectiveness and agreement of (A) use of OnGuard™ version 5.2 with a bone suppressed image provided by dual-energy subtraction, (B) a CAD + bone suppression software product (Unison™/ClearRead + Detect™) with a standard PA image and (C) OnGuard™ version 5.2 with a standard PA image.
Materials and Methods
Institutional review board approval was obtained from University Hospitals Case Medical Center for this project. The approval for an informed patient consent was waived and patient records were handled in compliance with Health Insurance Portability Accountability Act (HIPAA) regulations. All patient images and records were maintained on encrypted storage devices to maintain HIPAA compliance. All chest images for this study were provided by a Revolution XRd/Definium™ digital radiography unit (General Electric Medical Systems).
Patient Selection
Medical records and images from University Hospitals Case Medical Center were reviewed from 2005 through 2008. Patients with either pulmonary nodules confirmed by 16- or 64-slice computed tomography (CT) or pathology-proven lung carcinoma were selected. Individuals with single lung nodules 8–34 mm in size were selected as this size range is most likely to be missed by a radiologist. The nodules were located in a variety of locations in the lungs. This sample included 27 patients with CT and pathology-proven malignant nodules (mean age 71 ± 13, 17 females, 10 males).
In individuals with malignant nodules, the size and location were measured and marked on standard PA radiograph by an expert radiologist with 24 years of experience. The experienced radiologist also rated the nodules for subtlety, using a Likert scale of 1–5 with 1 being the least subtle and 5 the most subtle. None of the nodules were rated with a subtlety score of 5. Additionally, 25 individuals (mean age 46.2 ± 15, 10 females and 15 males) without malignant nodules as determined by 16- or 64-slice CT were included as negative cases.
Soft Tissue Image Generation
DESR, one form of bone suppressed image generation, was performed on a Revolution XRd/Definium™ digital radiography unit (General Electric Medical Systems). This radiography unit consisted of a 41 × 41-cm2 amorphous silicon-based flat panel detector. DESR is performed using the acquisition of a low-energy 60-kVp PA chest radiograph taken after a 150-ms delay and a high-energy 120-kVp radiograph. Subtracted and bone-enhanced images are also presented after post-processing of the high- and low-energy radiographs. A standard PA radiograph is obtained only from the 120-kVp acquisition.
Bone Suppression and CAD Software
Unison™/ClearRead + Detect™ and OnGuard™ version 5.2 were used to identify potential regions of interest (ROI) that may be suspect malignant nodules. The CAD component of the OnGuard™ CAD system identified ROI by imposing circular markings on the radiographs. Examples of CAD markings on the two bone suppression types and PA images are shown in Fig. 1. The circular marks were centered about a detection location that signified the identification of a probable malignant nodule.
Fig. 1.
True positive markings on same chest image: (A) DESR + OG 5.2™, (B) PA + Unison™/ClearRead + Detect™ and (C) PA + OG 5.2™
For identification of true positive detection and sensitivity, the known central point of the known nodule location must have been enclosed by the circular marking method of the CAD software and greater than 50 % of the radiologist outlined nodule must have been also enclosed by the CAD mark. All other ROIs were determined to be false-positives including those generated on CT-proven negative cases. Figure 2 depicts true positive and false positive example markings.
Fig. 2.
True and false-positive markings on subject with disease (TP = true positive; FP = false-positive): (A) DESR + OG 5.2™, (B) PA + Unison™/ClearRead + Detect™ and (C) PA + OG 5.2™
Statistical Analysis
MedCalc Software™ (version 12.3) was used to calculate test efficiency. Differences in ROI production and false-positive rate between the combination of CAD and the various image types were evaluated by the Friedman test. Differences in test sensitivity, specificity, and efficiency were evaluated using the McNemar test. The association between subtlety score, nodule location, and detection in the various bone suppression/CAD systems was measured using the chi-square test. Agreement between the imaging analysis groups was evaluated by Cohen's kappa. P values less than 0.05 were considered significant.
Results
The findings of the study indicate that the average number of ROI and average false-positives per image (FPPI) were not statistically significantly different between the three image group types (Table 1). FPPI and ROI rates, however, were reduced from previously reported versions of the CAD software used in previous studies [14].
Table 1.
Comparison of average ROI and false-positives per image for image analysis methods
| Average ROI/image | SD | Average FP/image | SD | |
|---|---|---|---|---|
| CAD image type | ||||
| DESR + OG 5.2™ | 1.115 | 0.98 | 0.73 | 0.97 |
| PA + Unison™ ClearRead + Detect™ | 1.31 | 1 | 0.9 | 0.96 |
| PA + OG 5.2™ | 1.289 | 0.99 | 0.9 | 0.98 |
| Friedman test | F = 0.67, p = 0.51 | F = 0.47, p = 0.63 | ||
The comparison of sensitivity, specificity, and test efficiency is listed in Table 2. Overall differences in sensitivity and specificity between the three different CAD image analysis methods were not found to be significant when McNemar's test was used (Table 3). Utilization of the chi-square test did not identify any significant association between the subtlety and detection by the various bone suppression method/CAD system combinations. However, location of the nodule (upper, middle, or lower lung) was associated with nodule identification (χ2 = 6.8, p = 0.033). Nodules in the upper lung were less likely to be detected by the CAD/bone suppression systems evaluated when compared to middle or lower lung nodules.
Table 2.
Diagnostic performance of the image analysis type
| Image analysis type | Sensitivity (%) | Specificity (%) | Test efficiency (%) |
|---|---|---|---|
| DESR + OnGuard™ 5.2 | 74.07 | 48.00 | 61.54 |
| PA + Unison™/ClearRead + Detect™ | 77.78 | 32.00 | 55.77 |
| PA + OnGuard™ 5.2 | 77.78 | 32.00 | 55.77 |
Table 3.
Pairwise comparisons of the image analysis types
| Image analysis type comparison | (McNemar's test significance (p value)) |
|---|---|
| DESR + OnGuard™ 5.2 vs PA + Unison™ | 0.30 |
| DESR + OnGuard™ vs PA + OnGuard™ | 0.33 |
| PA + Unison™ vs PA + OnGuard™ | 1.00 |
Agreement levels between the image analysis methods were between poor and excellent using the criteria of Fleiss [15]. DESR + OnGuard™ agreement with the other bone suppression/CAD methods was poor while PA + OnGuard™ and PA + Unison™ had excellent agreement (Table 4). However, overall diagnostic performance of DESR + OnGuard™ was better than either PA + Unison™ or PA + OnGuard™ but was not found to be statistically significantly different.
Table 4.
Agreement between image analysis methods
| Image analysis type comparison | Cohen's kappa | Fleiss agreement category |
|---|---|---|
| DESR + OnGuard™ vs PA + Unison™ | 0.253 | Poor |
| DESR + OnGuard™ vs PA + OnGuard™ | 0.341 | Poor |
| PA + Unison™ vs PA + OnGuard™ | 0.902 | Excellent |
Discussion
There are several methods for the suppression of rib, clavicle, and other bony structures in digital chest radiographs. Currently, bony structures can be removed from digital images via either hardware- or software-based methodologies. While previous research had evaluated the use of CAD and either hardware- or software-based methods separately, the two methods of bone suppressed image generation had not been directly compared. The results of this study suggest that the number of ROI and FPPI has been dramatically reduced from earlier versions of CAD used in conjunction with DESR [11, 12, 14] and also when compared to earlier software-based methods of bone suppression [13]. Additionally, sensitivity and specificity were found to be improved over previous versions evaluated in the literature [11, 13, 14]. The majority of false-positives recorded by the CAD/bone suppression systems tested were caused by misidentification of remnant rib crossings and shadows. Although the reduced numbers of ROI and FPPI suggest that current versions of CAD software could be more valuable to the clinical radiology community, the values for sensitivity (<80 %) and specificity (<50 %) are considered less than desirable for diagnostic tests.
Further, the near identical diagnostic performance of the latest version of OnGuard™ with the new bone suppression algorithm and Unison™/ClearRead + Detect™ ingeminates the previously reported improvement in CAD performance when using bone suppression methods. The results of this study suggest that it is possible to use PA or DESR chest images with either Unison™/ClearRead + Detect™ or the most current version of OnGuard™ with equally effective results. DESR + OnGuard™ version 5.2 had improved but not significantly different performance when compared to PA + OnGuard™ and PA + Unison™/ClearRead + Detect™. It is possible that the dual bone suppression image manipulation (DESR + the bone suppression algorithm in OnGuard™ version 5.2) reduced potential areas for erroneous ROI placement. Further investigation of this finding in a larger population of patients will be required.
The same PA chest images when used with Unison™/ClearRead + Detect™ and OnGuard™ version 5.2 had identical diagnostic performance and excellent but not identical agreement (Table 4). Only 2 cases out of 52 (3.8 %) were found to be different between the two image analysis methods. The reason for this discrepancy is unclear. Agreement between PA chest images when used with Unison™/ClearRead + Detect™ and OnGuard™ version 5.2 the DESR image analysis set was poor according to the criteria of Fleiss [15].
The cost between hardware- and software-based systems for soft tissue image generation is significantly different with post-processing software (SoftView™/ClearRead™) being substantially lower in cost than hardware-based (DESR) systems. However, DESR offers additional benefits with the production of an enhanced bone image (Fig. 3) that can assist in the detection of calcified structures [16] including potential coronary artery calcification [17, 18]. This enhanced bone image, which is produced with a nominal additional radiation dose, may provide additional diagnostic utility to the radiologist that is not available with software-based bone suppression methods.
Fig. 3.
Enhanced “bone” image from DESR
The question of whether or not to utilize CAD with or without soft tissue image generation methods remains germane however, even though results from the National Lung Cancer Screening Trial (NLST) suggest that spiral low-dose CT is superior to chest radiography in the identification of lung cancer, particularly at earlier stages of disease. First, the NLST study did not evaluate either BSI or the utility of CAD as diagnostic tools to improve nodule detection in chest X-rays. Further, several authors suggest that spiral CT screening is not without its own problems and may also provide an unwanted number of false-positives making the results clinically less than desirable [19, 20].
The results of this study indicate that no statistically significant differences exist in diagnostic performance when either PA or DESR chest images were used with the most current editions of CAD software alone (OnGuard™ version 5.2) or a combination bone suppression/CAD product (Unison™/ClearRead + Detect™). The results also suggest that DESR chest images when analyzed by the latest version of CAD software have improved, but not significantly different diagnostic performance when compared to PA images used with the same CAD software or PA images used with combination software-based bone suppression/CAD product.
This pilot study was limited by a relatively small sample size and restricted by a limited nodule size range that would be of the most value to clinical radiologists. Since the diagnostic performance of the three image analysis schemes was not found to be significantly different, the choice of hardware- or software-based bone suppression technology when used with the latest versions of CAD software may be relegated to cost or the ability to utilize additional image types from hardware-based systems.
Abbreviations
- DESR
Dual-energy subtraction radiography
- ROI
Region of interest
- CAD
Computer-aided detection
- BSI
Bone suppressed image
- PA
Posteroanterior
- FPPI
False-positives per image
References
- 1.Kobayashi T, Xu XW, MacMahon H, Metz CE, Doi K. Effect of a computer-aided diagnosis scheme on radiologists' performance in detection of lung nodules on radiographs. Radiology. 1996;199(3):843–848. doi: 10.1148/radiology.199.3.8638015. [DOI] [PubMed] [Google Scholar]
- 2.Bley TA, Baumann T, Saueressig U, Pache G, Treier M, Schaefer O, et al. Comparison of radiologist and CAD performance in the detection of CT-confirmed subtle pulmonary nodules on digital chest radiographs. Invest Radiol. 2008;43(6):343–348. doi: 10.1097/RLI.0b013e318168f705. [DOI] [PubMed] [Google Scholar]
- 3.Kasai S, Li F, Shiraishi J, Doi K. Usefulness of computer-aided diagnosis schemes for vertebral fractures and lung nodules on chest radiographs. AJR Am J Roentgenol. 2008;191(1):260–265. doi: 10.2214/AJR.07.3091. [DOI] [PubMed] [Google Scholar]
- 4.Meziane M, Obuchowski NA, Lababede O, Lieber ML, Philips M, Mazzone P. A comparison of follow-up recommendations by chest radiologists, general radiologists, and pulmonologists using computer-aided detection to assess radiographs for actionable pulmonary nodules. AJR Am J Roentgenol. 2011;196(5):W542–W549. doi: 10.2214/AJR.10.5048. [DOI] [PubMed] [Google Scholar]
- 5.De Boo DW, Uffmann M, Weber M, Bipat S, Boorsma EF, Scheerder MJ, et al. Computer-aided detection of small pulmonary nodules in chest radiographs: an observer study. Acad Radiol. 2011;18(12):1507–1514. doi: 10.1016/j.acra.2011.08.008. [DOI] [PubMed] [Google Scholar]
- 6.Eadie LH, Taylor P, Gibson AP. A systematic review of computer-assisted diagnosis in diagnostic cancer imaging. Eur J Radiol. 2012;81(1):e70–e76. doi: 10.1016/j.ejrad.2011.01.098. [DOI] [PubMed] [Google Scholar]
- 7.Shah PK, Austin JH, White CS, Patel P, Haramati LB, Pearson GD, et al. Missed non-small cell lung cancer: radiographic findings of potentially resectable lesions evident only in retrospect. Radiology. 2003;226(1):235–241. doi: 10.1148/radiol.2261011924. [DOI] [PubMed] [Google Scholar]
- 8.Monnier-Cholley L, Carrat F, Cholley BP, Tubiana JM, Arrive L. Detection of lung cancer on radiographs: receiver operating characteristic analyses of radiologists', pulmonologists', and anesthesiologists' performance. Radiology. 2004;233(3):799–805. doi: 10.1148/radiol.2333031478. [DOI] [PubMed] [Google Scholar]
- 9.Oda S, Awai K, Funama Y, Utsunomiya D, Yanaga Y, Kawanaka K, et al. Effects of dual-energy subtraction chest radiography on detection of small pulmonary nodules with varying attenuation: receiver operating characteristic analysis using a phantom study. Jpn J Radiol. 2010;28(3):214–219. doi: 10.1007/s11604-009-0411-7. [DOI] [PubMed] [Google Scholar]
- 10.Li F, Hara T, Shiraishi J, Engelmann R, MacMahon H, Doi K. Improved detection of subtle lung nodules by use of chest radiographs with bone suppression imaging: receiver operating characteristic analysis with and without localization. AJR Am J Roentgenol. 2011;196(5):W535–W541. doi: 10.2214/AJR.10.4816. [DOI] [PubMed] [Google Scholar]
- 11.Balkman JD, Mehandru S, DuPont E, Novak RD, Gilkeson RC. Dual energy subtraction digital radiography improves performance of a next generation computer-aided detection program. J Thorac Imaging. 2010;25(1):41–47. doi: 10.1097/RTI.0b013e3181aa34ed. [DOI] [PubMed] [Google Scholar]
- 12.Szucs-Farkas Z, Patak MA, Yuksel-Hatz S, Ruder T, Vock P. Improved detection of pulmonary nodules on energy-subtracted chest radiographs with a commercial computer-aided diagnosis software: comparison with human observers. Eur Radiol. 2010;20(6):1289–1296. doi: 10.1007/s00330-009-1667-0. [DOI] [PubMed] [Google Scholar]
- 13.Freedman MT, Lo SC, Seibel JC, Bromley CM. Lung nodules: improved detection with software that suppresses the rib and clavicle on chest radiographs. Radiology. 2011;260(1):265–273. doi: 10.1148/radiol.11100153. [DOI] [PubMed] [Google Scholar]
- 14.Meziane M, Mazzone P, Novak E, Lieber ML, Lababede O, Phillips M, et al. A comparison of four versions of a computer-aided detection system for pulmonary nodules on chest radiographs. J Thorac Imaging. 2012;27(1):58–64. doi: 10.1097/RTI.0b013e3181f240bc. [DOI] [PubMed] [Google Scholar]
- 15.Fleiss JL, Levin B, Paik MC. The Measurement of Interrater Agreement. Statistical Methods for Rates and Proportions [Internet]. John Wiley & Sons, Inc.; 2003. p. 598–626. Available from: http://dx.doi.org/10.1002/0471445428.ch18
- 16.Kuhlman JE, Collins J, Brooks GN, Yandow DR, Broderick LS. Dual-energy subtraction chest radiography: what to look for beyond calcified nodules. Radiographics. 2006;26(1):79–92. doi: 10.1148/rg.261055034. [DOI] [PubMed] [Google Scholar]
- 17.Gilkeson RC, Novak RD, Sachs P. Digital radiography with dual-energy subtraction: improved evaluation of cardiac calcification. AJR Am J Roentgenol. 2004;183(5):1233–1238. doi: 10.2214/ajr.183.5.1831233. [DOI] [PubMed] [Google Scholar]
- 18.Mafi JN, Fei B, Roble S, Dota A, Katrapati P, Bezerra HG, et al. Assessment of coronary artery calcium using dual-energy subtraction digital radiography. J Digit Imaging. 2012;25(1):129–136. doi: 10.1007/s10278-011-9385-y. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Zompatori M, Mascalchi M, Ciccarese F, Sverzellati N, Pastorino U: Screening for lung cancer using low-dose spiral CT: 10 years later, state of the art. Radiol Med. 2013, doi:10.1007/s10278-012-9565-4 [DOI] [PubMed]
- 20.Greenberg AK, Lu F, Goldberg JD, Eylers E, Tsay JC, Yie TA, et al. CT scan screening for lung cancer: risk factors for nodules and malignancy in a high-risk urban cohort. PLoS One. 2012;7(7):e39403. doi: 10.1371/journal.pone.0039403. [DOI] [PMC free article] [PubMed] [Google Scholar]