Which Phantom Is Better for Assessing the Image Quality in Full-Field Digital Mammography?: American College of Radiology Accreditation Phantom versus Digital Mammography Accreditation Phantom

Abstract

Objective

To compare between the American College of Radiology (ACR) accreditation phantom and digital mammography accreditation phantom in assessing the image quality in full-field digital mammography (FFDM).

Materials and Methods

In each week throughout the 42-week study, we obtained phantom images using both the ACR accreditation phantom and the digital mammography accreditation phantom, and a total of 42 pairs of images were included in this study. We assessed the signal-to-noise ratio (SNR) in each phantom image. A radiologist drew a square-shaped region of interest on the phantom and then the mean value of the SNR and the standard deviation were automatically provided on a monitor. SNR was calculated by an equation, measured mean value of SNR-constant coefficient of FFDM/standard deviation. Two breast radiologists scored visible objects (fibers, specks, and masses) with soft-copy images and calculated the visible rate (number of visible objects/total number of objects). We compared SNR and the visible rate of objects between the two phantoms and calculated the k-coefficient for interobserver agreement.

Results

The SNR of the ACR accreditation phantom ranged from 42.0 to 52.9 (Mean, 47.3 ± 2.79) and that of Digital Phantom ranged from 24.8 to 54.0 (Mean, 44.1 ± 9.93) (p = 0.028). The visible rates of all three types of objects were much higher in the ACR accreditation phantom than those in the digital mammography accreditation phantom (p < 0.05). Interobserver agreement for visible rates of objects on phantom images was fair to moderate agreement (k-coefficients: 0.34-0.57).

Conclusion

The ACR accreditation phantom is superior to the digital mammography accreditation phantom in terms of SNR and visibility of phantom objects. Thus, ACR accreditation phantom appears to be satisfactory for assessing the image quality in FFDM.

INTRODUCTION

Full-field digital mammography (FFDM) systems have many physical and clinical advantages when compared to film-screen mammography (1-5). In terms of physics, the FFDM separates the processes of image acquisition, processing, and display. Because image acquisition and display are separated, each can be optimized (1). In addition, digital detectors have a linear response to X-ray intensity, in contrast to the sigmoidal response of film-screen mammography. Therefore, digital detectors provide a broader dynamic range of density and higher contrast resolution (1). In terms of clinical aspects, FFDM has greater accuracy for the diagnosis of breast carcinomas in women under 50 years of age, women with radiographically-dense breasts, and premenopausal or perimenopausal women (2, 3). The FFDM demonstrates improved image quality when compared directly with film-screen mammography (4). In addition, the FFDM is superior to the analog film-screen mammography for the detection and morphologic characterization of microcalcifications (4, 5). Therefore, FFDM has been widely used and the FFDM image quality control is playing an increasingly important role in the field of radiology.

For the mammography accreditation program, the American College of Radiology (ACR) has approved Regional Medical Imaging model 156, Nuclear Associates model 18-220, and Computerized Imaging Reference Systems Inc. (CIRS) model 015 as an ACR mammography accreditation phantom (ACR Phantom). The ACR Phantom is the standard phantom for the Mammography Quality Standards Act (MQSA) and ACR quality control programs. The ACR Phantom was designed to test the performance of a mammographic system by a quantitative evaluation of the system's ability to image small structures similar to those found clinically. Objects within the phantom simulate calcifications, fibrous calcifications in ducts, and tumor masses. The ACR Phantom has been used for assessing image quality in both film-screen mammography and FFDM (6-8). As a result of a recent ACR requirement, phantom images should be obtained with the ACR Phantom and read by a hardcopy, without zooming or rotation of FFDM systems (7). However, Huda et al. (9) reported that the ACR Phantom was unsatisfactory for assessing image quality in FFDM and should be modified to have the appropriate range and sensitivity of current digital systems. The authors demonstrated that intraobserver variability was greater than interobserver variability in the detection of fibers, ACR phantom specks, a various change in the tube voltage, FFDM current influenced the average glandular doses and lesion detectability of the phantom. There are several commercially available digital mammography accreditation phantoms. The digital mammography accreditation phantom (Digital Phantom) was designed and devised along the different physical mechanisms of film and digital detectors. In spite of the development of several Digital Phantoms, there are no established criteria of these phantoms for image quality control. In addition, to our knowledge there is no published report to compare the usefulness between the ACR Phantom and the Digital Phantom for quality control in FFDM.

The purpose of this study was to compare the ACR Phantom and Digital Phantom in terms of the signal to noise ratio (SNR) and the visibility of phantom objects for assessment of image quality on FFDM.

MATERIALS AND METHODS

Mammography System

We used a Selenia FFDM system manufactured by Hologic Cooperation (Denver, CO, USA) for phantom imaging. The X-ray spectrum was generated by using a molybdenum target, and a molybdenum (30 µm) or rhodium (60 µm) filter. The digital detector was composed of amorphous selenium and absorbed X-ray photons and directly converted into an electronic signal. This flat-panel detector contains an array of electrodes and thin-film transistor switches for readout. The detector has two types of active imaging areas; 18 × 24 cm with 2560 × 3328 image pixel matrix and 24 × 29 cm with 3328 × 4096 image pixel matrix. For phantom imaging in the current study, we used a field of 18 × 24 cm. The pixel size of the detector was 70 µm and the bit depth was 14. The limited spatial frequency was 7.14 cycles per millimeter. The FFDM system had a Lorad HTC® (High Transmission Cellur; Hologic Cooperation, Denver, CO, USA) grid. The dynamic range was 400 : 1 in X-ray exposure with linear response.

We used a Coronis® 5MP display system (Barco, Duluth, GA, USA) with two 5-million-pixel gray-scale Liquid Crystal Display (LCD) monitors. The maximum luminance of the monitors was 800 cd/m². The resolution was 2048 × 2560 pixels and the pixel pitch was 0.165 mm. Digital images had the window width and level settings adjusted to optimize the image display. To ensure that images were displayed with the highest possible fidelity, the display system was calibrated with a dual-head BarcoMed® 5MP2FH display controller (Barco) and MediCal® Pro software (Barco).

Phantom Imaging

From June 2007 to March 2008, each week we obtained phantom images with both the ACR Phantom and Digital Phantom. A total of 42 pairs of phantom images were included in this study. Table 1 demonstrates characteristics of 2 phantoms. A standard ACR Phantom and a commercially available Digital Phantom were used for phantom imaging. As a standard ACR Phantom, we used a Nuclear Associates model 18-220 (Fluke Biomedical Radiation Management Services, Cleveland, OH, USA) (Fig. 1A). The 4.4 cm-thick phantom is made of a 7 mm wax block insert containing 16 sets of test objects, a 3.4 cm thick acrylic base, and a 3 mm thick cover. The phantom approximates a 4.5 cm compressed breast of average glandular/adipose composition. Included in the wax insert are aluminum-oxide specks that simulate microcalcfications. Six different nylon fibers simulate fibrous structures and 5 different size lens-shaped masses that simulate tumors. The ACR Phantom contains 6 fibers with diameters of 1.56, 1.12, 0.89, 0.75, 0.54, and 0.40 mm; 5 speck groups with 6 specks in each group, with speck diameters of 0.54, 0.40, 0.32, 0.24, and 0.16 mm; and 5 masses with decreasing diameters and thickness of 2.00, 1,00, 0.75, 0.50, and 0.25 mm.

Table 1.

Comparison of Characteristics between ACR Phantom and Digital Phantom

Note.- ACR Phantom = American College of Radiology Mammography Accreditation Phantom, Digital Phantom = Digital Mammography Accreditation Phantom

Fig. 1.

Two mammography accreditation phantoms. Left is ACR Phantom (Nuclear Associates model 18-220) and right is Digital Phantom (CIRS Model 015DM). Digital Phantom has extended top edge allows ease of positioning on recumbent biopsy units. A. ACR Phantom. B. Digital Phanom. ACR = American College of Radiology

As a commercially available Digital Phantom, we used a CIRS Model 015DM (Computerized Imaging Reference Systems Inc., Norfolk, VA, USA) (Fig. 1B). This phantom machine was designed similarly to the CIRS Model 015, which was accepted as an ACR Phantom. The 4.4 cm thick Digital Phantom is made of a 7 mm wax block insert containing 12 sets of test objects chick acrylic base, and a 3 mm thick cover. All of this together approximates a 4.2 cm compressed breast of average glandular/adipose composition. Ingredients of the included objects are the same as the ACR Phantom, however, the size and number of objects within the phantom are different. The Digital Phantom contains 4 fibers with diameters of 0.89, 0.75, 0.54, and 0.40 mm; 4 speck groups with 6 specks in each group, with speck diameters of 0.54, 0.32, 0.24, and 0.16 mm; and 4 masses with decreasing diameters and thickness of 1,00, 0.75, 0.50, and 0.25 mm. An experienced mammography technician obtained phantom images at 28-29 kVp and 58-69 mAs for both phantoms.

Image Assessment

We assessed the SNR and visible rates of objects in each phantom image. SNR is defined as the comparison of the desired signal level to the level of noise. A higher SNR provides a better image. A phantom was placed on the image detector and exposed. One radiologist drew a square-shaped region of interest in the background between first and second fibers of the phantom and then recorded the mean value of SNR with standard deviation for the regions of interest (Fig. 2). Our mammography unit automatically provided these values. The SNR values were calculated based on the equation;

Fig. 2.

Calculation of signal to noise ratio (SNR). Radiologist drew square-shaped region of interest (arrow) in background of phantom and then recorded mean value of SNR and standard deviation. SNR values were calculated based on following equation:

The constant coefficient of FFDM in our mammography unit was 50. SNR was obtained for both the ACR Phantom and Digital Phantom in the same way.

For assessment of visible rates, two breast radiologists who had between four and ten 10 years of mammography experience scored visible objects (fibers, specks, and masses) with soft-copy images based on the guidelines by the ACR (6). Digital images had the window width and level settings adjusted to optimize the image display. Two breast radiologists evaluated phantom images independently. The numbers of each type of object were different between the ACR Phantom and the Digital Phantom. The ACR has provided scoring criteria for the ACR Phantom; however, there have been no scoring criteria for the Digital Phantom. Thus, we calculated visible rates of fibers, specks, masses, and total objects to compare visibility between two phantoms. Visible rates were obtained as follows. Two radiologists independently scored each visible object and then obtained mean values of each visible object and total visible objects. The visible rate was calculated by the equation;

Each visible rate of fibers, specks, masses, and overall visible rates were obtained respectively for each radiologist and for each phantom.

Statistical Analysis

For comparison of the SNR and visible rates between the ACR Phantom and Digital Phantom, we used the student t test. Interobserver agreement between 2 radiologists for the visible rates was assessed by using the weighted kappa coefficients. A finding was considered statistically significant if the p value was less than 0.05. All statistical analyses were performed using SAS version 9.12 (SAS Institute, Cary, NC, USA). The statistical analyses of the data were supervised by a biostatistician.

RESULTS

Signal to noise ratio of the ACR Phantom ranged from 42.03 to 52.88 (mean, 47.33 ± 2.79) and that of Digital Phantom ranged from 24.84 to 53.99 (mean, 44.08 ± 9.93). This difference was statistically significant (p value = 0.028) (Fig. 3).

Fig. 3.

SNR and visible rates on ACR and Digital Phantom images of week 23.

A. On ACR Phantom, SNR is 45.23 and overall visible rate is 0.93 (15/16). Visible rates of fibers, specks, and masses are 1.00 (6/6), 0.80 (4/5), and 1.00 (5/5). B. On Digital Phantom, SNR is 32.47 and overall visible rate is 0.58 (7/12). Visible rates of fibers, specks, and masses are 0.50 (2/4), 0.50 (2/4), and 0.75 (3/4). Therefore, ACR Phantom is superior to Digital Phantom in terms of both SNR and visibility of phantom objects. SNR = signal to noise ratio, ACR = American College of Radiology

Table 2 demonstrates comparison of mean values of visible objects between the ACR Phantom and the Digital phantom. Mean values of visible objects on each phantom were presented as mean ± standard deviation. The number of each object on the ACR and the Digital Phantoms were different, thus, visible rates were obtained for comparison of visibility between 2 phantoms (Fig. 3). Mean values of visible objects were divided by the total number of objects within a phantom for visible rates. Each visible rate of fibers, specks, masses, and overall visible rates on the ACR Phantom and Digital Phantom for the 2 independent radiologists were also shown in Table 2. Overall visible rates (0.83 for radiologist 1 and 0.82 for radiologist 2) of the ACR Phantom were superior to those (0.75 for radiologist 1 and 0.72 for radiologist 2) of the Digital Phantom; these differences were statistically significant (p value = 0.018). In addition, the visible rates of each object were improved on the ACR Phantom compared to those calculated on the Digital Phantom. The differences in visible rates of fibers were statistically significant (p value = 0.009).

Table 2.

Comparison of Mean Values and Visible Rated of Visible Objects between ACR Phantom and Digital Phantom

Note.- Mean values are presented as mean ± standard deviation.

^*Visible rate

Visible rates are calculated by the equation,

The ACR has provided scoring criteria for the ACR Phantom and required minimum number of visualized fibers, specks, and masses to be 4, 3, and 3, respectively; however, there have been no scoring criteria for Digital Phantom.

ACR Phantom = American College of Radiology Mammography Accreditation Phantom, Digital Phantom = Digital Mammography Accreditation Phantom

Two radiologists independently evaluated phantom images, thus, we assessed interobserver agreement. Table 3 demonstrates the weighted kappa coefficients for interobserver agreement in terms of visible rates of objects on phantom images. Weighted kappa coefficients ranged from 0.34 to 0.57, therefore, the 2 radiologists showed fair to moderate agreement.

Table 3.

Weighted Kappa Coefficients for Interobserver Agreement in Visible Rates on Phantom Images

Note.- ACR Phantom = American College of Radiology

Mammography Accreditation Phantom, Digital Phantom = Digital Mammography Accreditation Phantom

DISCUSSION

The National Cancer Institute in the United States has reported that the steady decline in breast cancer mortality since 1990 has been attributed to early detection and the increased use of hormonal and adjuvant chemotherapies. The early detection has resulted in improved rates of survival, which has increased by 13% since the mid-1970s (10). Mammography screening has played an essential role in the early detection of breast cancer. There is reasonable evidence to support the position that the improved quality of mammography techniques in the United States is a result of accreditation and MQSA coupled with an increase in annual screening compliance have contributed to this early detection and improved survival rates.

Based on the recent MQSA and accreditation program for FFDM of ACR, 27% of mammography units in the United States were FFDM and the growth rate per month of FFDM was 6% in 2007 (9). ACR requires mammography equipment quality control and annual surveys according to the U.S. Food and Drug Administration regulations. ACR accreditation testing for FFDM includes clinical image review of fatty and dense breasts, phantom image review, dose, processor quality control or laser quality control. Criteria for the accreditation program in clinical image review, phantom image review, and dose is the same for digital as with film-screen mammography. For phantom image review of FFDM, only the ACR Phantom is accepted as film-screen mammography. For phantom and clinical image reviews, soft-copy images are not accepted in ACR accreditation testing and only hard-copy images are accepted. Interpretive radiologists must print the images as close to true size as possible within ± 25% error and read without zooming or rotating.

In Korea, the Korean Institute for Accreditation of Medical Images (KIAMI) plays similar roles to the ACR and MQSA in the United States. In addition, the KIAMI requests the ACR Phantom for phantom image review in both film-screen mammography and FFDM. However, the KIAMI accepts soft-copy images for phantom and clinical image review and allows for adjustment of window levels, zooming, or rotation. Previous reports about diagnostic performance between hard-copy and soft-copy readings for FFDM demonstrated that soft-copy reading using 3-mega pixel and 5-mega pixel LCDs is comparable to hard-copy reading for detecting breast cancers (11-13). The soft-copy system can improve performance in terms of both speed and diagnostic accuracy through improvements in monitor technology. In the current study, we used soft-copy phantom images using 5-mega pixel LCD and allowed adjustment of image windows, zooming, or rotation during image interpretation.

The usefulness of the ACR Phantom for accreditation testing of FFDM has not been deeply investigated. A few commercially available Digital Phantoms have been introduced for quality control in FFDM. For assessment of image quality of FFDM systems, the clinical applications of FFDM systems should be understood. FFDM systems have advantages for mammography-guided procedures such as stereotactic biopsy and localization. FFDM is very convenient for these procedures, because radiologists can review the mammographic images on a monitor in real time during procedures with no waiting for film development. The Digital Phantom which was used in the current study was designed to evaluate image quality in FFDM systems, not only for routine mammographic examination but also for stereotactic biopsy and localization. The phantom is a compact version and miniaturization of the mammographic accreditation phantom for coverage of various range of field of view in the FFDM systems. Also it was devised along the different physical mechanisms that distinguish film and digital detectors. However, the Digital Phantom does not provide definitive criteria to determine appropriate image quality. In addition, Huda et al. (9) reported that the ACR Phantom was unsatisfactory for assessing image quality in FFDM and should be modified to have the appropriate range and sensitivity for current digital systems. Therefore, our goal was to compare the ACR Phantom and the Digital Phantom in terms of SNR and the visibility of objects on the phantoms and to assess the usefulness of the ACR Phantom for image quality control in FFDM.

As a Digital Phantom, we chose the CIRS Model 015DM, which is the most similar digital mammography evaluation phantom to the ACR Phantom. The CIRS Model 015DM has the same ingredients and thickness as the ACR Phantom. There are 2 differences between the CIRS Model 015DM and the ACR Phantom. First, the ACR Phantom has a total of 16 objects (6 fibers, 5 specks, and 5 masses); on the other hand, the Digital Phantom has a total of 12 objects (4 fibers, 4 specks, and 4 masses). The ACR Phantom has 6 fibers with diameters of 1.56, 1.12, 0.89, 0.75, 0.54, and 0.40 mm; 5 speck groups with 6 specks in each group, with speck diameters of 0.54, 0.40, 0.32, 0.24, and 0.16 mm; and 5 masses with decreasing diameters and thickness of 2.00, 1,00, 0.75, 0.50, and 0.25 mm. The Digital Phantom does not have 2 fibers with diameters of 1.56 and 1.12 mm, a speck group with a diameter of 0.40 mm, and a mass with a diameter of 2.00 mm when compared with the ACR Phantom. Therefore, the average size of objects in the Digital Phantom is smaller than in the ACR Phantom. However, we performed soft-copy reading and allowed zooming and adjustment of window width and levels in the current study. In addition, we used visible rates to compare the visibility of objects between 2 phantoms to overcome the difference in the number of objects. Second, there is a little difference in the composition. Both phantoms have average glandular/adipose composition, however, the ACR Phantom approximates a 4.5 cm compressed breast and the CIRS Model 015DM approximates a 4.2 cm compressed breast. Thus, the Digital Phantom has slightly thinner breast composition and this may have an influence on the image quality.

Both overall visible rates and visible rates of each object on the ACR Phantom were superior to those on the Digital Phantom for both radiologists. In addition, the ACR Phantom had significantly better visible rates to evaluate fibers. These might be related with containing fibers' size. The Digital Phantom does not include 2 thicker fibers, 1.56, 1.12 mm. Even though we allowed image adjustment that included size and window settings during phantom image review, the thickness of fibers may influence the detection. Visible rates on the ACR Phantom were superior to those on FFDM for the evaluation of masses and specks, however, these results were not statistically significant. Therefore, although the Digital Phantom did not contain larger diameters of specks and masses, differences of visible rates between the two phantoms were not significant in the current study. Based on our results, we would not hesitate to recommend the use of the ACR Phantom for accreditation testing of FFDM in terms of visibility of phantom objects, if we obtain the phantom image with appropriate kVP and mAs settings. For routine applications of the Digital Phantom in the accreditation testing, we suggest setting up the different criteria from the ACR Phantom with regards to the visibility of objects because the ACR and Digital Phantoms have different numbers and sizes of objects.

In SNR, our results demonstrated that the ACR Phantom was superior to the Digital Phantom. Both mean values and variability of SNR for the ACR Phantom were significantly superior to those for the Digital Phantom (range and mean of the ACR Phantom, 42.03 - 52.88, 47.33 ± 2.79 vs. range and mean of the Digital Phantom, 24.84 - 53.99, 44.08 ± 9.93). For optimization of FFDM, detective quantum efficiency, modulation transfer function, contrast-to-noise ratio, and SNR are widely used. SNR and contrast-to-noise ratio are used for the evaluation of image quality. In the current study, we used the SNR for assessment of phantom image quality. Noise reduces image quality and sensitivity for detecting breast lesions. In FFDM, the same detector is used repeatedly and fixed pattern noise can be developed. We obtained 42 pairs of phantom images with an ACR and a Digital Phantom on the same detector and the possibility of fixed pattern noise was the same between two phantoms. Therefore, better SNR of the ACR Phantom means that the ACR Phantom can provide better image quality than the Digital Phantom.

In this study, the ACR Phantom had better SNR values and visible rates than the Digital Phantom. The Digital Phantom can be used for quality control of stereotactic biopsy units. Smaller object size of the Digital Phantom may not interfere with image quality because stereotactic biopsy units have smaller field of views than FFDM. However, for the routine application of the Digital Phantom in assessment of image quality of FFDM, appropriate criteria should be settled in the near future. In addition, SNR values of the Digital Phantom also should be improved.

In the current study, phantom images were displayed on the 5 mega-pixel monitors and 2 radiologists independently evaluated images with the window width and level settings individually optimized for viewing fibers, specks, and masses on the 2 phantoms. We obtained satisfactory phantom images at 28-29 kVp and 58-69 mAs and all scores of the 3 objects on the ACR Phantom were higher than the criteria of the ACR. On the other hand, Huda et al. (9) demonstrated the ACR phantom was unsatisfactory for assessing image quality in FFDM, because of intraobserver variability for lesion detection on the phantom and a large change of image quality according to the various settings of the kVP and mAs. They evaluated phantom images in various kVp (24-34 kVp) and mAs (5-500 mAs) settings. They reported that the detection of objects was essentially constant between 26 and 34 kVp reaching a plateau level that corresponded to the best performance and visibility at 80 mAs. Detection performance was reduced at 24 and 25 kVp for all 3 types of objects, because of increased background noise. Our results were concordant with the study by Huda et al. (9). If we perform the ACR Phantom imaging at the adequate kVp and mAs settings, we can obtain and keep the best quality of the ACR Phantom image. We would recommend 28-29 kVp and 58-69 mAs for ACR Phantom imaging based on our results.

In conclusion, the ACR Phantom is superior to the Digital Phantom in terms of both SNR and phantom object visibility. Therefore, the ACR Phantom would be a satisfactory phantom for assessment of image quality in FFDM, if we keep the appropriate kVp and mAs settings during phantom imaging.