Abstract
This study compared a five-category ordinal scale and a two-alternative forced-choice subjective rating of image quality preferences in a multiabnormality environment. 140 pairs of laser-printed posteroanterior (PA) chest images were evaluated twice by three radiologists who were asked to select during a side-by-side review which image in each pair was the “better” one for the determination of the presence or absence of specific abnormalities. Each pair included one image (the digitized film at 100 μm pixel resolution and laser printed onto film) and a highly compressed (∼60∶1) and decompressed version of the digitized film that was laser printed onto film. Ratings were performed once with a five-category ordinal scale and once with a two-alternative forced-choice scale. The selection process was significantly affected by the rating scale used. The “comparable” or “equivalent for diagnosis” category was used in 88.5% of the ratings with the ordinal scale. When using the two-alternative forced-choice approach, noncompressed images were selected 66.8% of the time as being the “better” images. This resulted in a significantly lower ability to detect small differences in perceived image quality between the noncompressed and compressed images when the ordinal rating scale is used. Observer behavior can be affected by the type of question asked and the rating scale used. Observers are highly sensitive to small differences in image presentation during a side-by-side review.
Key Words: image quality, observer performance, study methodology, ratings
Full Text
The Full Text of this article is available as a PDF (1.0 MB).
Footnotes
This work is supported in part by grants from the National Cancer Institute (CA60259, CA66594, and CA67947).
References
- 1.Rosenthal MS, Good WF, Costa-Greco MA, et al. The effect of image processing on chest radiograph interpretations in a PACS environment. Invest Radiol. 1990;25:897–901. doi: 10.1097/00004424-199008000-00005. [DOI] [PubMed] [Google Scholar]
- 2.Metz CE. Some practical issues of experimental design and data analysis in radiological ROC studies. Invest Radiol. 1989;24:234–245. doi: 10.1097/00004424-198903000-00012. [DOI] [PubMed] [Google Scholar]
- 3.Rockette HE, Obuchowski NA, Gur D, et al. Effect of experimental design on sample size. Proc SPIE. 1991;1446:276–286. doi: 10.1117/12.45282. [DOI] [Google Scholar]
- 4.Obuchowski NA, Zepp RC. Simple steps for improving multiple-reader studies in radiology. AJR. 1996;166:517–521. doi: 10.2214/ajr.166.3.8623619. [DOI] [PubMed] [Google Scholar]
- 5.Holbert JM, Staiger M, Chang TS, et al. Selection of processing algorithms for digital image compression: A rank order study. Acad Radiol. 1995;2:273–276. doi: 10.1016/S1076-6332(05)80183-7. [DOI] [PubMed] [Google Scholar]
- 6.Good WF, Gur D, Feist JH, et al. Subjective and objective assessment of image quality—a comparison. J Digit Imag. 1994;7:77–78. doi: 10.1007/BF03168426. [DOI] [PubMed] [Google Scholar]
- 7.Good WF, Maitz GS, Gur D. Joint photographic expert group compatible data compression of mammograms. J Digit Imag. 1994;7:123–132. doi: 10.1007/BF03168505. [DOI] [PubMed] [Google Scholar]
- 8.Britton CA, Gabriele OF, Chang TS, et al. Subjective quality assessment of computed radiography hand images. J Digit Imag. 1996;9:21–24. doi: 10.1007/BF03168564. [DOI] [PubMed] [Google Scholar]
- 9.Nill NB. A visual model weighted cosine transform for image compression and quality assessment. IEEE Trans Commun COM. 1985;33:551–557. doi: 10.1109/TCOM.1985.1096337. [DOI] [Google Scholar]
- 10.Ngan KN, Leong KS, Singh H. Cosine transform coding incorporating human visual system model. Proc SPIE. 1986;707:165–171. [Google Scholar]
- 11.Kelley DH. Visual contrast sensitivity. Opt Acta. 1977;24:107–129. [Google Scholar]
- 12.Thaete FL, Fuhrman CR, Oliver JH, et al. Digital radiography and conventional imaging of the chest: A comparison of observer performance. AJR. 1994;162:575–581. doi: 10.2214/ajr.162.3.8109499. [DOI] [PubMed] [Google Scholar]
- 13.Hamdan MA, Jensen DR. A bivariate binomial distribution and some applications. Austral J Statist. 1976;18:163–169. doi: 10.1111/j.1467-842X.1976.tb01292.x. [DOI] [Google Scholar]
- 14.Hamdan MA, Nasro MO. Maximum likelihood estimation of the parameters of the bivariate binomial distribution. Commun Statist Theor Method. 1986;15:747–754. doi: 10.1080/03610928608829149. [DOI] [Google Scholar]
- 15.Rockette HE, Gur D, Metz C. The use of continuous and discrete confidence judgments in receiver operating characteristic studies of diagnostic imaging techniques. Invest Radiol. 1992;27:169–172. doi: 10.1097/00004424-199202000-00016. [DOI] [PubMed] [Google Scholar]
- 16.Gur D, Rockette HE, Good WF, et al. Effect of observer instruction on ROC study of chest images. Invest Radiol. 1990;25:230–234. doi: 10.1097/00004424-199003000-00004. [DOI] [PubMed] [Google Scholar]