The Medical Image Perception Society Update on Key Issues for Image Perception Research

Abstract

To guide future investigations, the 2007 Medical Image Perception Society meeting held panel discussions to consider the current state of our knowledge of medical image perception and identify important questions to advance our understanding.

The Medical Image Perception Society (MIPS) is composed of scholars studying the processes of perception and recognition of information in medical images. Membership includes radiologists, psychologists, statisticians, physicists, engineers, and others in this growing research community. The members represent universities, hospitals, private companies, and government agencies (eg, National Institutes of Health [NIH], U.S. Food and Drug Administration). Every 2 years, MIPS holds a scientific conference to exchange current research and to conduct tutorials and workshops. The first conference was held in 1985 and the MIPS was incorporated in 1998. The purpose of the society is to promote medical image perception research. The conference enjoys growing participation and attendance, offering a chance for students to interact with seasoned perception researchers in a retreat-workshop setting.

In 1994, the NIH, the National Cancer Institute, and the Conjoint Committee on Diagnostic Radiology established priorities for medical image perception research (1). The priorities identified were to develop psychophysical models for the detection of abnormalities on medical images, to improve understanding of the mechanisms of perception important for interpreting medical images, to develop aids for interacting with displays that enhance perception, to study alternatives to sequential presentation of cross-sectional imaging data, and to perform methodologic research to improve the evaluation of medical imaging systems. These priorities continue to guide research.

The MIPS meeting periodically dedicates a session of its conference to reconsider these priorities, update them, and formulate new goals for research. For example, articles were published in 1998 that were based on an earlier MIPS conference and updated the aims of medical image perception research (2,3). Those goals included mathematical modeling of the detection task, gaining a better understanding of visual search and the nature of expertise, and developing perceptually based standards for image quality, computer-based aids to image perception, and quantitative methods for describing natural images and for measuring human detection and recognition.

The most recent MIPS conference, held in 2007 at the University of Iowa, formulated new thrusts for medical image perception research. Radiologists work in a changing environment. Analog film images are already rare. The exchange of images from one specialty to another to enhance decision making will continue to increase as information systems become more sophisticated and integrated. New types of digital images, acquired with new technologies and processed in new ways, appear every year. The burgeoning field of molecular imaging (4), which unites molecular biology and in vivo imaging, is likely to play a greater clinical role in the future but we know little to date on how best to present this new type of image data to clinicians. Regardless of these changes, the interaction of the eye-brain system with visually presented medical image data will remain at the core of radiology. Continued investigation of this complex perceptual recognition and interpretation process will be needed to offer the most useful and effective presentation of imaging information to physicians and to improve their detection and classification of disease.

It is difficult to directly monitor the ways in which image perception research has been incorporated into clinical practice, but it is clear that the numerous studies conducted over the years have had an impact. For example, many of the studies that facilitated the transition from analog film to soft-copy digital image reading and picture archiving and communication systems were based on a fundamental understanding of human perception. The Digital Imaging and Communications in Medicine (DICOM) gray-scale standard display function is based on the principle of perceptual linearization (5). The specific display function selected by DICOM is based on the Barten model and offers the additional advantage of perceptual linearization (6–9). Perceptual linearization optimizes the display by producing a tone scale in which equal changes in driving level yield changes in luminance that are perceptually equivalent across the entire luminance range. Studies have demonstrated that a perceptually linearized display does indeed improve diagnostic accuracy (10), and now it is inconceivable that electronic displays would be used for diagnostic applications without proper DICOM calibration.

To guide future investigations, at the 2007 MIPS meeting panel discussions were held to consider the current state of our knowledge of medical image perception and to identify important questions to advance our understanding. Discussions reviewed the current state of the human perception of medical images, observer modeling, visual search, display issues, and technology evaluation tools. There were two discussion leaders for each of these areas. Some of the general questions considered were: Does new imaging technology obviate or necessitate perception research? How can visual search be studied in advanced imaging? What is the value of observer modeling for clinical radiologists? How will the stimulus be defined for new and innovative observer models? What is the future of computer-aided diagnosis? Are there innovations in the pattern recognition literature that will make a difference? Will computer-aided diagnosis research be reabsorbed by the more general image processing effort? What standards will be applied to color displays, and how will we know whether they perform adequately for gray-scale images? Should there be broader standards for displays used in low-cost reading (eg, personal digital assistants and cell phones)? Is there consensus on the appropriate methods for measuring observer performance? How should the use of more modern data analysis methods be encouraged? What are the limitations of our current receiver operating characteristic (ROC) methods?

The following sections present the results of these discussions.

Human Perception of Medical Images

Background

The pattern recognition provided by radiologists is a fundamental determinant of performance of diagnostic systems (11–13). As long as radiologists interpret medical images, the limitations of human detection of abnormalities will influence the overall performance of diagnostic systems. Abnormalities on medical images are sometimes missed (14–16). We need to know the causes of these errors to find ways of eliminating them (11). We need to better understand the cognitive and perceptual mechanisms that underlie the discovery and reporting of abnormalities, as well as the formulation of diagnoses. By studying how the observer allocates attention across images, we may inform the training of radiologists, the development of effective machine readers, and ultimately provide better images and display devices.

New Research Questions

(a) In current clinical practice, radiologists rarely get feedback about their errors in diagnosis. Will this change with the Integrated Health Enterprise? If so, will feedback change how radiologists interpret images (17? Radiologists interpret more imaging studies and more images in each study (18). This increase in workload creates the potential for errors based on perceptual and cognitive overload. Computer-aided diagnosis (CAD) and other automated image analysis tools may help, but the tools currently do not seem to help all radiologists and are not available for many interpretation tasks (19,20). Perception research may help us to better guide the development of interpretation aids. (b) As the field of molecular imaging is so new and so different from what radiologists and other clinicians are used to, what viewing formats for molecular imaging are most effective?

Observer Modeling

Background

Observer modeling can be used to assess image quality throughout the imaging chain and to predict observer performance (21–23). Although observer models do not eliminate the need for observer studies, they can be used to narrow the range of experimental conditions that need to be examined with actual observers. In a model of the ideal observer, a theoretical observer takes advantage of all information contained in an image. These models have traditionally served as the basis for evaluating performance of image acquisition. The value of observer models has been underappreciated by the clinical community as shown by the small number of articles published in clinically oriented radiology journals. Greater applicability to clinical radiology has been a recent goal of this field. Newer versions of visual discrimination models more closely simulate the human visual system and recent investigations use more elements of real clinical images and abnormalities rather than just mathematically simulated backgrounds and targets (24).

New Research Questions

(a) Are observer models useful for studying clinical radiology and radiologists? Can observer models be used to model individual radiologists? (b) Can we use observer models for training? If we could show trainees what an ideal observer would do for each diagnostic modality and task, we might be able to produce more rapid perceptual learning. (c) Most observer modeling is still very basic, using simulated signals in simulated or real backgrounds without visual search. Radiology images rely increasingly on dynamic and complex display. How can ideal observer models be extended to clinical modalities?

Visual Search

Background

The experienced radiologist processes a substantial amount of diagnostic information in a short time, but not without moving his or her eyes over the images to gather visual information (11–13). Perception researchers have used eye-tracking technology extensively to study visual search in radiology (25–29). If we can understand visual search of medical images, we may be able to explain why errors of diagnosis occur and what can be done to improve performance. We also may be better able to train residents to become expert radiologists (30,31).

New Research Questions

(a) Is the eye-tracking technology sufficiently easy and robust for routine use in the clinic to provide feedback? (b) Can we begin to study visual search of more complex imaging such as scrolling through computed tomographic (CT) sections or three-dimensional images? (c) Can CAD be improved by the study of human eye-position recording? Can the CAD focus on some aspects of images compensate for the lack of attention by human readers?

Display Issues

Background

Ergonomic and human factors issues of displays and display interfaces are becoming more important to radiologists as new technologies emerge and the clinical environment becomes more computer based (32,33). For example, fatigue of radiologists interpreting electronic display images has been considered by a few researchers, but we are only beginning to determine exactly how radiologists and radiology residents are affected (34–36). If they are affected and diagnostic accuracy is impaired as a function of fatigue, we need to understand, for example, whether we need to develop countermeasures at the physiologic level (eg, oculomotor control) or at the decision-making level (eg, improved use of computer-aided decision tools).

New Research Questions

(a) Does the wide variability of the types and configurations of displays offered by different vendors affect the perception of images? For example, does diagnostic accuracy differ as a function of using a 3-megapixel monochrome liquid crystal display versus a 3-megapixel monochrome display? If only color is considered, is diagnostic performance influenced if the color display technology is based on in-plane switching (IPS) using cold cathode fluorescent lamp (CCFL) and light-emitting diode backlights, a vertical alignment using CCFL, or a twisted nematic using CCFL? What are the implications for clinical practice and laboratory research? Perception research should help us understand these variations and optimize the radiology reading environment. (b) Is it worth developing training sets of imaging studies with proved diagnoses to reduce differences among radiologists in detection accuracy?

Technology Evaluation Methodology

Background

Over the years, many improvements in receiver operator characteristic (ROC) methods have been introduced at the Far West/MIPS meetings, as well as other psychophysical and statistical methods (37–40). Feedback from the MIPS audience has proved useful to developers of new technology evaluation methods because MIPS conference attendees often make extensive use of these methods in their own research.

New Research Initiatives

(a) We need additional ROC designs, such as split-plot design and unbalanced designs, and to be able to handle missing data. In some multifactor study designs it is often not possible to completely randomize the order of the runs or trials within the block or study session, so the randomization becomes restricted. This often results in a generalization of the randomized block design called split-plot design. Unbalanced designs refer to studies in which, for example, not all subjects provide responses to every question or not all subjects complete all parts of a study. The result is that there are missing data. In radiology observer studies this can be a common problem, since radiologists are often quite busy and sometimes are not able to complete large studies. Instead of eliminating them from the analysis and discarding the data that was acquired, it would be useful to have statistical methods that could deal with the missing data. (b) Scoring limitations of the classic ROC approach are still with us. Patients often have multiple lesions or abnormalities on their images. We are in need of a widely accepted analytic method to cope with multiple responses. Moreover, there are diagnostic tasks that require more than just a binary decision; further exploration of three-class and multiclass decision paradigms is needed. (c) Are there better ways to establish disease state truth?

Benefits to Clinical Radiology

The ultimate goal of researchers involved in the development and evaluation of better hardware and software for the presentation of medical image data to the clinician is not necessarily to provide clinicians with the most “beautiful” image, but rather with the image that allows them to render the most accurate and timely interpretation. It is important to appreciate that clinical image interpretation is a moving target—technology has changed and has altered the nature of image interpretation. Our continued study of medical image perception and the general interaction of clinicians with medical imaging examinations remains a critical element of improving health care.