Image Quality Improvement after Implementation of a CT Accreditation Program

Abstract

Objective

The purpose of this study was to evaluate any improvement in the quality of abdominal CTs after the utilization of the nationally based accreditation program.

Materials and Methods

Approval was obtained from the Institutional Review Board, and informed consent was waived. We retrospectively analyzed 1,011 outside abdominal CTs, from 2003 to 2007. We evaluated images using a fill-up sheet form of the national accreditation program, and subjectively by grading for the overall CT image quality. CT scans were divided into two categories according to time periods; before and after the implementation of the accreditation program. We compared CT scans between two periods according to parameters pertaining to the evaluation of images. We determined whether there was a correlation between the results of a subjective assessment of the image quality and the evaluation scores of the clinical image.

Results

The following parameters were significantly different after the implementation of the accreditation program: identifying data, display parameters, scan length, spatial and contrast resolution, window width and level, optimal contrast enhancement, slice thickness, and total score. The remaining parameters were not significantly different between scans obtained from the two different periods: scan parameters, film quality, and artifacts.

Conclusion

After performing the CT accreditation program, the quality of the outside abdominal CTs show marked improvement, especially for the parameters related to the scanning protocol.

Generally, every medical institution has a quality control program to maintain imaging quality. Additionally, other international programs for the improvement of diagnostic imaging quality exist, such as the American College of Radiology (ACR) in the United States of America (1). There are also quality control programs with a legal influence, such as the Mammography Quality Standards Act (MQSA) of the ACR. However, to our knowledge, the only accreditation program for the improvement of computed tomography (CT) imaging quality (in addition to any self-assessment or other recommendations) that is performed as a program of national laws is the Korea Institute for Accreditation of Medical Image (KIAMI) program in Korea (2). KIAMI was established in 2004 under the Ministry of Health and Welfare to evaluate the quality of medical images of mammography, CT, and magnetic resonance imaging, and to perform medical imaging quality tests with the goal of improving national health. The CT scanners that were evaluated in this quality accreditation program included approximately 1,600 scanners from every year since 2005. Annual documentation and hands-on evaluation (every 3 years) were performed according to the regulations of KIAMI. The CT accreditation program involves reviews of the standards of a facility, the personnel information, and the quality control procedure records, as well as an evaluation of both phantom and clinical images (2-6).

Studies comparing any improvements in the quality of the clinical images prior to and after the implementation of the law of the CT accreditation program have not yet been conducted. Therefore, the purpose of this study was to analyze and compare the clinical imaging quality in abdominal CT scans before and after the implementation of the accreditation program and to evaluate the effectiveness and limitations of the CT accreditation program.

MATERIALS AND METHODS

Our Institutional Review Board (Catholic Medical Center, Seoul, Korea) approved the protocol of this retrospective study, and the requirement for informed consent was waived. The study was comprised of outside abdominal CT scans referred to our five hospitals from outside hospitals, from 2003 to 2007. The five hospitals are affiliated with the Catholic Medical Center of The Catholic University of Korea. We randomly chose two months per year, for a total of 10 selected months, and all of the abdominal CT scans from outside hospitals or clinics during these months were included as part of the analysis. A total of 1,011 CT scans were collected; of which, 582 were from male patients, 419 were from female patients, and 10 were from patients of unidentified genders. The ages of the patients ranged from one to 97 (mean, 56.6) years old, and 23 of the subjects' ages were unidentified.

We used KIAMI's clinical test image data sheet (2, 4) to evaluate the clinical images (Table 1). Two experienced radiologists from each hospital (a total of 10 abdominal radiologists with 5 to 31 years of experience) reviewed the outside abdominal CT scans, and both reviewers graded them using the modified clinical test image data sheet in consensus.

Table 1.

Clinical Test Image Data Sheet for Abdominal CTs, as Conducted by KIAMI

Note.-KIAMI = Korea Institute for Accreditation of Medical Image

CT Clinical Image Evaluation Sheet (Table 1)

Testing of the CT images was performed using the clinical test image data sheet generated by KIAMI in Korea. The clinical test image data sheet is prepared separately according to the areas of the body (head/neck, chest, abdomen regions). Our study performed CT clinical image testing on abdominal CTs. Table 1 shows the clinical test image data sheet for abdominal CTs, as conducted by KIAMI (2, 4). The two main categories are the general and imaging parameters, followed by the scores set by each sub-category. The general parameters are divided into four sub-categories, namely the identifying data (6 points), display parameter (8 points), scan parameter (4 points), and film quality (2 points), yielding a total potential score of 20 points. The passing grade for this program is a score over 12 points, which is 60% of the perfect score. The imaging parameters are divided into six sub-categories; namely, the artifact (12 points), scan length (7 points), spatial and contrast resolution (20 points), window width and level (10 points), optimal contrast enhancement (21 points), slice thickness (10 points), which add up to a potential total score of 80 points. A passing grade for this program is a score over 48 points, which is 60% of the perfect score. The total score is the sum of the general parameters and the imaging parameters, which adds up to a potential of 100 points. However, the total score is not the important factor to consider for passing the test of the program. As stated above, the general and imaging parameters should each be over 60% of the perfect score to pass the test of the accreditation program. One separate box at the bottom of the sheet, which is not present on the clinical test image data sheet designed by KIAMI, was added to facilitate conducting research. The box on the bottom has an item to check for the subjective 3-point scale: grade 1, unacceptable when degraded image quality is significant enough for impairment of diagnosis; grade 2, fair when there is some degraded image quality but no impairment of diagnosis; grade 3, optimal when the CT image quality is optimal before subjective scoring the CT images. The reviewers found and agreed in consensus on the CT images of each score, and these images were used as a standard for scoring. The reviewers graded the overall quality of CT images using the 3-point scale.

Analysis

The CT images were divided into two periods of 'before' (2003 and 2004) and 'after' (from 2005 to 2007) the implementation of the accreditation program.

First, we checked whether the referred CT images were soft copies. We evaluated any changes between the two periods according to the parameters of the clinical image evaluation and subjective assessment. We correlated the subjective assessment of image quality with the score from the clinical image evaluation. We also analyzed the results according to the size of the medical institution: general hospital, hospital, or clinic. This classification was based on the medical service law of Korea (5). General hospitals are defined as medical institutions equipped with more than 100 beds, and more than nine medical departments. Hospitals are defined as medical institutions established by medical doctors and equipped with facilities having more than 30 beds. Clinics are defined as medical institutions which do not otherwise meet the criteria for a hospital or general hospital. We evaluated the appropriateness of the kVp and mAs of the CT images from before and after implementation of the CT accreditation program. Finally, we compared the failure rate and average score of the imaging parameters in the failed group, before and after the program. We defined failure as not passing either general parameters (less than 12 points) or imaging parameters (less than 48 points). We analyzed the 'before' and 'after' time periods and among groups according to the size of medical institution data by the student's t test and one-way ANOVA (analysis of variance) using the SPSS software (Version 13.0, Statistical Package for the Social Sciences, Chicago, IL). We used the MedCalc software (MedCalc 10.4.8, Mariakerke, Belgium) for comparisons of the proportion of soft copies and unidentified institutions between the two periods. We used the Mann-Whitney U test for the comparison of the subjective assessment and the failure rate between the two periods (before, after). P values less than 0.05 were considered indicative of a significant difference.

RESULTS

We reviewed a total of 1,011 outside abdominal CT scans, with 328 before and 683 after the implementation of the accreditation program. Before program implementation, there were 171 hard copies (52%) and 157 soft copies (48%), as opposed to 217 hard copies (32%) and 466 soft copies (68%) after program implementation. We collected significantly more soft copies of outside abdominal CT scans after the implementation of the program (p < 0.001) compared to before.

Table 2 shows the average scores of the general and imaging parameters. The identifying data and display parameters were significantly greater after implementation of the program (p < 0.001). The scan parameters and the film quality were not statistically different before or after implementation. Of the imaging parameters evaluated, scan length, spatial and contrast resolution, window width and level, and slice thickness were significantly greater after program implementation. The artifact showed no statistically significant difference between the two periods. The optimal contrast enhancement was actually lower after the implementation of the accreditation program. The average scores for the sums of general parameters (p = 0.004), imaging parameters (p < 0.001), and the total score (p < 0.001) after the implementation of the accreditation program, were significantly higher than those from before.

Table 2.

Average Scores of General and Imaging Parameters

After implementing the accreditation program, there was a decrease in percentage in grade 2 (30% before, 19% after) and an increase in grade 3 (60% before, 70% after), with an overall statistically significant improvement in scores (p = 0.013) for the subjective assessment. However, although not statistically significant, grade 1 showed a slightly greater percentage (10% before, 12% after, p = 0.646). Table 3 shows the average scores of the general parameters, imaging parameters, and total score according to the grade of the subjective assessment. As for the unaccepted cases, the means of the general parameters and total score were 17.6 points, and 63.6 points, respectively, which are both higher than the passing grades (12 and 60 points), however the mean of the imaging parameters was 46.0 points, which is not higher than the passing grade (48 points).

Table 3.

Average Scores of General Parameters, Imaging Parameters, and Total Score According to Results of Subjective Assessment

Note.-KIAMI = Korea Institute for Accreditation of Medical Image, SD = standard deviation

We also analyzed the results according to the size of the medical institution. Before the implementation of the accreditation program, 217 CT images (66%) were referred from general hospitals, 37 (11%) from hospitals, and 33 (10%) from clinics. We could not find any information on the medical institution for the remaining 41 CT images (13%), which were classified as the unidentified group. After the accreditation program, 444 CT images (65%) were referred from general hospitals, 88 (13%) from hospitals, 82 (12%) from clinics, and 69 (10%) from unidentified medical institutions. The proportion of collected CT scans according to the size of the medical institution and the proportion from unidentified institutions are not statistically different between the 'before' and 'after' time periods (p = 0.190).

Figure 1 shows the average total scores for the outside abdominal CTs from each group according to the size of the medical institution. After the implementation of the CT accreditation program, the average total scores from the general hospital (before: 86.2 ± 10.93 points, after: 90.2 ± 8.48 points, p < 0.001) and the hospital (before: 80.8 ± 9.06 points, after: 85.0 ± 12.69 points, p = 0.041) were significantly higher than the scores prior to program implementation. However, the average total scores from clinics (before: 74.8 ± 12.20 points, after: 74.5 ± 16.48 points, p = 0.925) and the unidentified group (before: 74.4 ± 15.30 points, after: 78.1 ± 16.48 points, p = 0.241) showed no significant difference between the 'before' and 'after' time periods. The total scores among groups according to the size of the medical institutions were compared and the total score from general hospitals was found to be significantly higher than at hospitals, clinics, or unidentified groups both before (p = 0.042) and after (p < 0.001) the accreditation program implementation. However, the total score from hospitals was not different from that of clinics or unidentified groups prior to the implementation of the accreditation program. After implementing the accreditation program, a significant difference in scores was shown between hospitals and clinics (p < 0.001), as well as between hospitals and unidentified groups (p = 0.001). The total scores from before and after implementation of the accreditation program were not different between the clinics and unidentified groups.

Fig. 1.

Bar graph showing average total scores of outside abdominal CT images referred from general hospitals, hospitals, clinics, and unidentified group from before and after accreditation program.

The average peak kilovoltage (kVp) and milliampere second (mAs) prior to the implementation of the accreditation program were 120.87 and 271.54, respectively, compared to 121.92 and 249.32, respectively after the program, which were both not significantly different (p = 0.622, p = 0.458).

The failure rates before and after the implementation of the program were 10% and 9%, respectively and were not significantly different (p = 0.967). The average scores of the imaging parameter in the failed group were 38.5 ± 7.6 points before and 37.6 ± 6.8 points after and was not significantly different (p = 0.442).

DISCUSSION

During the 10 selected months taken from 2003 to 2007, 1,011 outside abdominal CT scans referred to our five hospitals from outside hospitals. More soft copies were collected after the implementation of the CT accreditation program, thereby suggesting that, over these years in Korea, with the increased use of the PACS, the number of soft copies increased compared to hard copies. In addition, this pattern suggests that, to pass the newly initiated CT accreditation program, old CT equipment was to be replaced with the PACS rather than with a printer.

After the implementation of the CT accreditation program, the scores of the identifying data and display parameters in the general parameters were noticeably improved. The reason for this improvement is likely that these parameters could be readily inserted without special equipment. Nonetheless, among the general parameters, the scores of the scan parameter and the film quality were not significantly improved. The scan parameter asks whether the kVp and mAs are appropriate. The kVp and mAs values could not be shown in a lot of old equipment. Using the old equipment, these parameters were not automatically displayed on the image, and some institutions added these parameters as a text on the image. Moreover, some equipment was not adjusted to the new standards after the year 2000, for instance for Y2K. Therefore, the old equipment should be replaced by newer versions. The film quality was not a problem in either time period.

Among the imaging parameters, the scan length, window width and level, and slice thickness were significantly greater after the implementation of the accreditation program. These parameters are notably closely related to the protocol of CT scanning. These improvements can be interpreted as the result of efforts by each institution to be accredited by the KIAMI program through the education of responsible technologists. Furthermore, this improvement implies that the program has an educational effect and that it should be a part of the recommended standard CT protocol.

Even after the implementation of the CT accreditation program, the score of the artifact did not significantly improve. The score of the optimal contrast enhancement actually decreased after the program's implementation. These parameters are related to equipment performance and are different from parameters that could be improved simply by reorganizing the protocol and educating radiology technicians. The improvement of parameters such as artifacts and optimal contrast enhancement requires the replacement of old machines with new ones, or better maintenance of quality control, both of which take time and money.

The parameters of the documentation of the lung window setting on the basal lung and information about contrast injection require little effort for improvement, but they are constantly a low scoring category. Institutions give this only minor attention, probably due to the low scoring and the pass/fail nature of the program.

The grading score of the subjective assessment, added by us, improved after the program's implementation, as did the average scores of the general parameters, imaging parameters, and total score. Because the KIAMI program is one with pass/fail scoring and is not used for the grading of imaging quality, the average scores of the general parameters and total scores of outside abdominal CTs that were graded as unacceptable by the subjective assessment were shown to be higher than the passing grade set by the KIAMI program. However, the average score of the imaging parameters of outside abdominal CT images that were graded as unacceptable was comparable to passing grade (48 points) set by KIAMI. Considering that the imaging parameters are very important factors in the evaluation of CT images, KIAMI's abdominal CT accreditation program could sufficiently function as the proper standard of evaluation of CT imaging quality. In addition, adding a new program for quality grading would be helpful for the continuous improvement of image quality.

When the total scores were compared according to the size of the institution, general hospitals and hospitals significantly improved their total scores after the implementation of the program, but the total score of clinics did not improve. This demonstrates that CT quality control has not been improved in clinics, which is likely due to the limitation of personnel and financial resources. In addition, according to current Korean law, it is clearly stated that a CT could be performed without a full-time radiologist and with only one radiology technician when using CTs installed prior to 2003. Furthermore, a CT could be performed even with the supervision of just one part-time radiologist for CTs installed after 2004. This may not be a big problem in general hospitals or hospitals; however, in clinics without a radiologist or with a part-time radiologist, CT image quality control may not be performed properly, and this may be responsible for the delay in improvement of CT quality in clinics. Considering that clinics contribute the most basic service in the Korean medical service, appropriate actions may be needed for quality improvement.

After implementing the accreditation program, the failure rate of clinical image evaluation according to KIAMI's sheet was approximately 10%, with no significant difference with the period prior to the implementation of the accreditation program. Furthermore, there was no significant difference in the percentage of unacceptable subjective image quality between the two periods. The reason for this is possibly due to the fact that most of the CT equipment that failed before continued to be used after optimizing and passing the test; however, these machines possibly also often failed the following year. So perhaps a specific new accreditation program targeting prior failed CT equipment is needed, because these machines are intrinsically of poor quality and are temporarily passing the accreditation criteria.

One limitation of our study is the difference in the evaluated time periods (two years before and three years after the implementation of KIAMI). We chose three years for the 'after' time period because the KIAMI clinical image evaluation is performed three years after accreditation. On the other hand, the PACS implementation began after 2003, and any evaluation of film from before this period was logistically difficult; thus, the only choice for the 'before' period was two years. The other limitation of our study is that the results of the scan parameter category of kVp and mAs indicated no significant change before and after the imposition of KIAMI. Thus we can presume no reduction in dose as a result of this quality improvement. Because the amount of radiation received during a CT exam may be harmful, it is important to reduce the dose in concurrence with keeping on the image quality for proper diagnosis. Though KIAMI has no references about radiation dose, the new standards for radiation dose such as CT dose index and dose length product should be incorporated into the KIAMI program.

In conclusion, after the implementation of the CT accreditation program, the quality of outside abdominal CTs improved markedly, especially for parameters that were directly related to the CT protocols. The accreditation program by national law improved the quality of abdominal CTs and had a measurable educational effect on radiologists and radiology technicians.