Visual Attention and Applications in Multimedia Technologies

Abstract

Making technological advances in the field of human-machine interactions requires that the capabilities and limitations of the human perceptual system are taken into account. The focus of this report is an important mechanism of perception, visual selective attention, which is becoming more and more important for multimedia applications. We introduce the concept of visual attention and describe its underlying mechanisms. In particular, we introduce the concepts of overt and covert visual attention, and of bottom-up and top-down processing. Challenges related to modeling visual attention and their validation using ad hoc ground truth are also discussed. Examples of the usage of visual attention models in image and video processing are presented. We emphasize multimedia delivery, retargeting and quality assessment of image and video, medical imaging, and the field of stereoscopic 3D images applications.

I. INTRODUCTION

Selective attention is nature’s answer to a problem that is present in all but the simplest organisms and increasingly also in machines: information overload. To work efficiently in a variety of complex environments, animals and machines are equipped with an array of sensors, all of which are needed in one situation or another to assure survival of the animal or proper function of the machine. In any given situation,, however, only a subset of the sensory input is needed and it would be wasteful (and in many cases practically impossible) to process all sensory input at all times. Therefore, selection has to be made which sensors are relevant at a given time, and only information provided by those is allowed access to central processing resources, Frequently, even the input stream from one sensor may be overwhelmingly rich. For instance, all visual input to the human brain¹ is provided by about 10⁶ retinal ganglion cells per eye. Assuming a maximal firing rate of these neurons of about 100 Hz results in a channel capacity of 100 Mbits per second per eye. Indeed, analyses of spike train statistics of visual input to the brain in primates [1], carnivores [2] and insects [3] confirm that the rate of the transmitted information is within an order of magnitude of the channel capacity. This torrent of information cannot be, and does not have to be, processed in detail. Instead, only a fraction of the instantaneously available information is selected for detailed processing while the remainder is discarded.

The filtering process is called selective attention and its mechanisms have been studied systematically for well over a century [4]–[6]. The first parallel stages of sensory processing are followed by a bottleneck that restricts the amount of information allowed to proceed to more central processing stages [7], [8]. Information processing in these later stages occurs sequentially rather than in parallel. This allows the application of powerful algorithms to the selected parts of the input that would be too costly to implement for all of sensory input.

For instance, search for a “singleton” target (that is distinguished from distractors by one feature, e.g., by its color) is usually a parallel process (with search times nearly independent of the number of distractors) while search times for “conjunctive” targets (that can be distinguished from distractors only be considering more than one feature, e.g., color and orientation) increase linearly with the number of distractors, suggesting a serial search. Treisman and colleagues argue in their Feature Integration Theory [9] that identification of conjunctive targets requires to bind its various features to a coherent object, a task that cannot be performed by elementary feature maps but requires the resources of a more powerful attentional mechanism. This mechanism is not available in parallel for the whole visual field but needs to be applied sequentially. A more differentiated view of visual search has emerged since Treisman’s original theory, e.g., refs [10]–[13], but it is generally accepted that visual processing consists of a parallel stage that is fast but relatively simple, followed (if the task requires it) by application of a more powerful mechanism that needs to be applied sequentially to one (or possibly a few) parts of visual input. Exploitation of this limitation of the human visual system is the basis for applications in multimedia which is the topic of this paper.

In section II, we discuss mechanisms of selective selective attention in primate vision and existing computational models. In section III, we focus on some multi-media applications without seeking for exhaustiveness, and we conclude in section IV.

II. VISUAL ATTENTION MECHANISMS AND COMPUTATIONAL MODELS

In this Section, we introduce detailed computational models of selective attention and some of their limitations. We define two dichotomies, overt vs. covert attention in Section II-A and bottom-up vs. top-down attention in Section II-B. In section II-C, we briefly discuss difficulties in obtaining ground truth for model predictions.

A. Overt versus covert visual attention

Due to the much higher resolution in the center of the retina compared to its more peripheral regions, humans and other primates usually direct their center of gaze towards the most relevant areas of the visual scene. This generates a series of fixations (and smooth eye movements although the latter are not often discussed in the context of selective attention) called “overt attention,” since allocation of the high-resolution resources in the fovea can be easily observed by following the person’s eyes, most conveniently and quantifiably with an eye tracker. It has been proposed that far-reaching conclusions can be drawn about the state of the human mind by analyzing the details of this so-called “scan path” [14], [15].

Primates, however, do not have to attend compulsively to objects in their center of gaze. As discovered early on both experimentally [4] and through introspection [6], humans are able to focus their attention to peripheral locations, away from their center of gaze. An illustration of this process is a car driver who fixates the road while simultaneously and covertly monitoring road signs and lights that appear in the retinal periphery. Since this redirection of attention is not visible immediately, it is referred to as covert attention.

There are many experimental paradigms that can determine the movements of the covert focus of attention but none is as convenient, fast, and easy to understand as tracking the eyes of an observer; in other words, measuring his or her overt attentional state. Fortunately, although the locations of overt and covert attention can be dissociated, as discussed, psychophysical evidence shows that an eye movement to a new location is necessarily preceded by focal attention to this locationq [16]–[21]. This makes it possible to easily obtain a close correlate of overt attentional selections by recording eye movements which thus serve as a proxy for shifts of covert attention. Of course, prediction of eye movements is also of immense interest by itself and of great practical interest, including for multimedia applications, Section III. Frequently, models for covert attention are, explicitly or implicitly, used to predict eye movements.

B. Bottom-up versus top-down attention

Attentional selection is a central part of perception and cognition. As such, it is influenced by many factors, both internal and external to the observer. What is attended depends, for instance, on the observer’s motivation and the specific task he or she is performing. In a set of classic experiments, Yarbus [22] showed that eye movements (overt attention) of the same observer viewing the same visual scene differ dramatically depending on what information the observer is looking for in the scene. Attentional selection that depends on the internal state of the observer is referred to as “top-down attention.” It is very difficult to develop biologically realistic detailed models of such mechanisms which may include influences such as the personal history of the observer.

On the other hand, “bottom-up” selection only depends on the visual input provided instantaneously or in the very recent past (as in immediately preceding frames of a movie). As such, it is not only much easier to control but it is also easier to quantify the correlation between input and resulting behavior. For this reason, Koch and Ullman [23] proposed that bottom-up attention is a suitable candidate for detailed computational models of selective attention. Specifically, they proposed that bottom-up attention is directed to salient parts of the visual scene and they proposed the concept of a saliency map. This is a topographic map of the visual field whose scalar value is the saliency at the respective location. Saliency is computed at multiple scales from the local differences in visual submodalities (color, orientation, …). If both the basic premise that bottom-up attention is attracted by salience as well as their concept how salience is computed are correct, attentional control is then reduced to finding the local maxima in the saliency map and assigning the successively visited foci of attention to those maxima in order of decreasing peak value. This results in a “covert attentional scan path,”, see Figure 1 for an illustrative example, in analogy to the sequence of eye movements in overt attention.

Fig. 1.

Examples of scanpath (a), Region of Interest Map and corresponding content (b), Fixation density Map and corresponding content (c).

This conceptual idea of attentional control by a saliency map was subsequently implemented in biologically realistic computational models [24]–[26]. Over the last decade and a half, these models have been refined, tested and applied by a large number of groups. Borji and Itti [27] provide an excellent overview of the current state-of-the-art of visual attention modeling including a taxonomy of models (information theoretical, cognitive, graphical, spectral, pattern classification, Bayesian,…).

The simplicity of the original saliency map model makes it attractive both conceptually as well as for applications but it also engenders limitations. For instance, it has been found that eye movements are typically oriented towards the centers of objects, rather than their borders which is where bottom-up saliency peaks [28]. Such deviations can be explained, at the cost of slightly higher complexity, by directing attention to proto-objects, rather than purely spatially defined regions of the visual scene [28], [29].

While bottom-up influences are thus important, it is clear that in many situations top-down attention plays a role, too. One consequence of the saliency map model is that its first selections in a new scene should agree better with observed eye movements than later ones, since less top-down guidance is expected to exist for input never seen before; this was confirmed experimentally [30]. It is also important to distinguish the term “salience” and “importance” (as in, e.g., Region of Interest/Importance, RoI) which are frequently considered synonyms in the signal processing literature. While both visual salience and visual importance denote the most visually “relevant” parts of the scene, it is useful to reserve the term “salience” for strictly bottom-up influences, while “important” areas can be selected based on both bottom-up and top-down criteria. The two mechanisms are thus driven by a different combination of sources. The interplay between these mechanisms has been studied showing that their relationship might vary along viewing time [31].

Even though top-down influences play an important role in attentional selection, we have already discussed that developing computational models of top-down attention in as much detail as for bottom-up attention is virtually impossible. Some progress has been made for parts of the general problem, for instance for finding objects [32]. This field must be considered, however, as being in its infancy. For instance, it is known that not only the properties of objects and their immediate surrounds but the interaction between objects on the image scale as well as the “gist” of the scene [33], [34] strongly influence search patterns and response times [35]. On the other hand, it was shown that low-level saliency is significantly predictive not only of eye movements but, surprisingly, even for conscious decisions of what observers consider interesting [36]. The fact that the very simple quantities computed in the original saliency map [24], [25] significantly influence human behavior after conscious deliberation and after many seconds of response time engenders hope that these easily and cheaply computed models and their derivatives can be useful for technical applications, even when humans are “in the loop,” as in multi-media applications.

C. Visual attention models and ground truth

Developing and testing computational models of visual attention depends on the availability of ground truth. Many studies rely on fixation density maps (FDM) generated from eye-tracking experiments (see Figure 2 for an illustration of the process leading to the generation of FDM). Consequently, most of the models are supposed to address mainly overt visual attention. Nevertheless, recommendations to properly generate FDM are still missing. Several eye-tracking FDM databases have been made publicly available corresponding to experiments conducted independently in different conditions. The question of corresponding viewing time is particularly critical regarding top-down and bottom-up competition, while rarely considered. How the difference between various experimental set up to obtain FDM may impact image processing applications has been recently investigated [37].

Fig. 2.

Steps for transforming eye-tracking data into a Fixation Density Map. After gathering the raw data (top), saccades are identified and fixation locations are determined (center). The fixation map is then obtained by convolving fixation locations with a Gaussian whose size is determined by a combination of mean eye tracking error and the size of the human fovea (bottom).

Computational models of attention produce very different predictions for FDM (see examples in Figure 3). How to quantitatively compare the performance of different models given the ground truth is another topic of research, see ref. [38] for a recent study proposing several metrics to assess model performance. It should also be noted that using FDM as ground truth may not be warranted for all models since some are designed to explain aspects of visual attention mechanisms that are not reflected in FDM.

Fig. 3.

Examples of FDM generated by visual attention models: original content (a), AIM model [?] (b), STB model [28] (c), SR model [39] (d)

Given these caveats, the usage of a given visual attention models should be achieved cautiously in image processing applications, considering the model type (e.g: top-down vs bottom-up) but also its performance regarding a given application context. Better characterization of a model should lead to comprehensive recommendation of proper usage.

III. APPLICATIONS OF VISUAL ATTENTION MODELS IN IMAGE AND VIDEO PROCESSING

In the following, we give an overview of applications of models of visual selective attention. In Sections III-A and III-B, we discuss multimedia delivery. Section III-C is devoted to re-targeting, Section III-D to quality assessment, and Section III-E to applications in medical imaging. Finally in Section III-F we discuss stereoscopic 3D images.

A. Multimedia Delivery: improving source coding

Several stages of the media delivery chain can benefit from insights into visual attention mechanisms. The first attempts were applied to selective compression of image and video contents. A survey on this topic can be found in [40]. Sensitivity and resolution reduction of the human visual system as a function of eccentricity is one the that could benefits to improve compression performance once salient location identified [43]. Selective compression is based on two priors: a prior of selection, that defines the most informative areas of an image, and a prior of compression that defines the coding nature and bit rate allocation strategy. Compression rate (prior of coding), and consequently the visual quality, can be differentially adapted to different image areas depending on the level of attention devoted to them by the human observers (prior of selection). The importance of a given image region can be computed based on the contribution of different features (contrast in color, orientation, intensity, …) [24], [41], [42] or in a simplified version under the assumption that human faces attract attention [43]. There are two principle approaches to prioritize coding of different image areas using saliency information. The first is the indirect approach [44] in which the graphical contents is pre-processed. Image agreas are selectively encoded according to their saliency, e.g., by low-pass filtering less important regions. The choice of preprocessing methods needs to be compatible with the coding scheme, especially with the quantization operator.

The direct approach is applied in block based coding methods. Bit rates are allocated to each macro block separately according to a visual saliency criterion (see Figure 4). Most of the time, this is achieved by changing the quantization parameters. This can be done using conventional RDO (Rate Distortion Optimization) techniques [45]–[48] or by providing a map based on a preceding analysis of the contents [49].

Fig. 4.

Distribution of encoding cost of natural scenes (shown in a) for a conventional H.264 coding (b) and a saliency based approach (c) (from O. Le Meur, P. Le Callet, D. Barba Selective H.264 video coding based on a saliency map, http://people.irisa.fr/Olivier.Le Meur). Color coded pixels show the cost in the respective areas. The color scale at the bottom is common for all panels in rows b and c.

With the recent availability of low-cost, consumer-grade eye trackers, visual attention-based bit allocation techniques for network video streaming have been introduced [50]. To improve the efficacy of such gaze-based networked systems, gaze prediction strategies can be used to predict future gaze locations to lower the end-to-end reaction delay due to the finite round trip time (RTT) of transmission networks. Feng et al. [50] demonstrated that the bit rate can be reduced by slightly more than 20% without noticeable visual quality degradation even when end-to-end network delays ares as high as 200ms.

In another approach [51], the audio component is also taken into account to improve RoI encoding based on the observation that sound-emitting regions in an audio-visual sequence typically draw a viewer’s attention.

B. Multimedia Delivery: Improving Resilience to Transmission Errors

Packets in a video bitstream contain data with different levels of importance from the visual information point of view. This results in unequal amounts of perceived image quality degradation when these packages are lost. Quality assessment experiments with observers have demonstrated that the effect of a lost packet depends on the spatio-temporal location of the visual information coded in the packet. Perceived quality degradation is lowest when the loss affects regions of “noninterest” [52]–[54]. Visual attention based error resilience or RoI based channel coding methods are consequently good candidates to attenuate the perceptual quality loss resulting from packet loss. In the context of highly prediction based coding technologies such as H.264/AVC, for good compression performance there is a high dependency between many parts of the coded video sequence. However, this dependency comes with the drawback of allowing a spatio-temporal propagation of the error resulting from a packet loss. RoI based coding should also consider attenuating the effect of this spatiotemporal dependency when important parts of the bitstream are lost. As part of the H.264/AVC video coding standard, error resilience features such as Flexible Macroblock Ordering (FMO) and Data Partitioning (DP) can be exploited to improve resilience of salient regions of video content. DP partitions code slice into three separate NAL (Network Abstract Layer) units, containing each different part of the slice. FMO allows the ordering of macroblocks in slices according to a predefined map rather than using the usual raster scan order. Coupled with RoI-based coding, FMO is can be used to gather RoI macroblocks into a single slice [55]. An alternative approach [56] consist in confining the RoI in separate slices to prevent error propagation within a picture and then constraining the coding prediction process in the RoIs to avoid that the resulting loss distortion reaching RoIs in other pictures.

C. Image and Video retargeting

With the recent explosion of commonly available device types (tablet, smart phone, large displays, …), formats (3D, HD, Ultra HD, …) and services (video streaming, image database browsing, …), the visual dimension of multimedia contents viewed by a human observer can vary enormously, resulting in the stimulation of very different fractions of his or her visual field. Depending on display capacity and the purpose of the application, contents often need to be repurposed to generate smaller versions, with respect to image size, resolution, frame rate, … A common way to achieve this goal is to dramatically down-sample the picture homogeneously, as in thumbnail modes. This often yields poorly rendered pictures since important objects of the scene may be no longer recognizable. Alternatively, content repurposing techniques perform content-aware image resizing, for example by seam carving [57]. Saliency based image re-targeting (or content repurposing or reframing techniques) algorithms have been proposed following this idea: identify important regions of interest and compute the reduced picture centered on these parts [58], [59] (see figure 5 for an illustration). More recently, dynamic (i.e. time changing) thumbnails have been introduced using a dynamic computational model of visual attention [60]. Rubinstein and colleagues [61] have evaluated many image re-targeting algorithms both objectively and subjectively and demonstrated the value of saliency based cropping approaches.

Fig. 5.

Process of saliency-based reframing [58]. The saliency-based thumbnail focuses on the most relevant image parts.

D. Image and Video quality assessment

Perceptual objective image quality assessment uses an algorithm that evaluates the quality of pictures or video as a human observer would do based on the properties of the human visual system. Visual attention is one of the features that can be considered based on the rationale that an artifact is likely more annoying in a salient region than in other areas [62]. Most of objective quality assessment methods can be decomposed in two steps. Image distortion is first locally (pixel-base, block-based, …) evaluated resulting in a distortion map. In the second step, a pooling function is used to combine the distortion map values into a single quality score value. An intuitive idea to improve quality assessment methods using visual attention information is to give greater weight at the pooling stage to degradation appearing in salient areas than in non-salient areas [63], [64]. Initial approaches consisted in weighting the distortion map using local saliency values before computing a linear or non linear mean. More recent studies, based on eye tracking data, demonstrated that this simple weighting is not very effective [65], [66] in the case of compression artifacts. Nevertheless, such approaches can lead to significantly improved performance in the case of non-uniformly located distortions such as those due to transmission impairments [67]. Alternative weighting methods have been introduced for compression artifacts with varying success [68], [69]. In ref. [70], more complex combinations of saliency map and distortion are introduced, assuming that weights should be a function of both saliency value and distortion level. You etal [71], [72] revisit the problem at the distortion level for video content. Distortion visibility can be balanced according to the human contrast sensitivity function. As the latter is spatially non uniform, gaze estimation should be considered to properly apply it.

Another open issue is which parts of the original content and its distorted version should be used for estimating the saliency map. Artifacts themselves may affect the deployment of visual attention; they may, for instance, attract attention [73]. Moreover, objective quality measures are expected to correlate with the outcomes of quality assessment experiments performed by observers. To obtain comparison data, observers need to perform specific tasks. Such tasks are likely to affect the visual attention deployment compared to a free-viewing [74]–[76].

E. Medical imaging

Over the past twenty years, digital medical imaging techniques (Computed Tomography, Magnetic Resonance Imaging, Ultrasound, Computed Radiography/Digital Radiography, Fluoroscopy, Positron Emission Tomography, Single Photon Emission Computed Tomography, …) have revolutionized healthcare practice, becoming a core source of information for clinicians to render diagnostic and treatment decisions. Practical analysis of medical images requires two basic processes: visually inspecting the image (involving visual perception processes, including detection and localization tasks), and performing an interpretation (requiring cognitive processes). Unfortunately, interpretation is not error-free and can be affected by the observer’s level of expertise and by technological aspects. Moreover, a side effect of the dramatic increase in the availability and use of medical images is a shortage of qualified image reading experts. It is likely that the time per image that is available for interpretation will continue to decrease in the future. Expertise in medical image reading therefore needs to be seen under the two aspects: accuracy and speed [77]. Understanding how clinicians read images, how they develop expertise throughout their careers, and why some people are better at interpreting medical images than others are crucial questions that are related to visual attention.

Such knowledge represents great potential to develop better training programs and create new tools that could enhance and speed up the learning process. A longitudinal study [77] of pathology residents during their development of expertise in reading slides of breast biopsies used eye tracking experiments at the beginning of each of their three years of residency, documenting changes of their scan paths as they increased the level of their experience. The data showed that search patterns changed with each successive year of experience. Over time, residents spent significantly less time per slide, made fewer fixations, and performed less examination of non-diagnostic areas. Similar findings have been obtained in radiology on multi-slice images such as Computer Tomography scans (CCT) [78] or multi sequences Magnetic Resonance Imaging (MRI) [79]. Figure 6 shows an example of scanpath and gaze fixations in the case of multiple MRI sequences.

Fig. 6.

Scanpath and gaze fixations on multiple MRI sequences. Shown are different MRI sequences (gray) taken from one patient’s head, overlaid with eye movement data of a clinical expert (green). Lines are saccades and shaded circles signify fixations, with the diameter proportional to viewing time. It is seen that the image reader uses several sequences, implying comparison of different source of information from the different representations in several panels.

F. Stereoscopic 3D: new opportunities for visual attention

A key factor required for the wide-spread adoption of services based on stereoscopic images will be the creation of a compelling visual experience for the end-user. Perceptual issues and the importance of considering 3D visual attention to improve the overall 3D viewing experience in 3DTV broadcasting have been discussed extensively [80]. Integrating visual attention at source and channel coding level represents limited adaption compared to 2D case. More interestingly, content production offers new original opportunities to make use of insights in visual attention mechanisms, especially dealing with perceptual concept such as visual comfort. Comfortable viewing conditions, e.g., zone of comfortable viewing, of stereoscopic content is linked to several factors such as accommodation-vergence conflict, range of depth of focus and range of fusion [81], [82]. A seminal study by Wopking [83] suggests that visual discomfort increases with high spatial frequencies and disparities, partially because the limits of stereoscopic fusion increase as a result of the decreased spatial frequency. More generally, it appears that blurring can have a positive impact on visual comfort because it reduces the accommodation-vergence conflict, limiting both the need for accommodation and the effort to fuse [84], [85]. Simulating depth-of-field (DOF) is a way to take advantage of the retinal defocusing property in order to improve visual comfort, by artificially blurring images to a degree that corresponds to the relative depth from fixated objects. As reported by Lambooij et al. [86], “three essential steps are required for proper implementation of a simulated DOF: localization of the eye positions, determination of the fixation point and implementation of blur filters to non-fixated layers.” This procedure has been applied in virtual reality environments but has drawbacks in more general contexts since it affects depth cue integration between retinal disparity and areas with high amounts of blur [87]. Blurring effects can also be used for 3D content to direct the viewer’s attention towards a specific area of the image that could meet a comfortable viewing zone. In gaming and in the computer graphics community, visual attention modeling has attracted a growing interest. Visual attention models have been used to produce a more realistic behavior of a virtual character, to improve interactivity in 3D virtual environments, and to improve visual comfort when viewing rendered 3D virtual environments [88]–[90].

Due to geometry issues related to depth rendering, adaptation from a cinema environment to the home environment is far from being an automatic, straightforward process for 3D content production. Automated content-based post-production or post-processing tools to help adapt 3D content to television are expected to be developed. 3D visual attention models can be employed to provide the area of interest and convergence plane to drive the content repurposing of stereoscopic content. In addition, the adaptation of the scene depth can be used to improve visual comfort. To reduce both visual discomfort and fatigue, the convergence plane is usually continuously set to the main area of interest, as the latter is moving across different depth levels. A way to reduce eye strain is to modify the convergence plane of the main area of interest to place it on the display plane, i.e., by adapting the content disparity. Such visual attention based adaptive rendering of 3D stereoscopic video has been proposed using a 2D visual attention model [91].

IV. CONCLUSION

Visual attention is attracting a high level of interest in the vision science community. In this paper, we have demonstrated that this research interest is highly penetrating the Information and Communication Technology (ICT) field with some successful outcomes although there are still challenges ahead. One caveat is that, as in any trans-disciplinary approach, one has to assure that concepts from one research field are properly used when appropriated by another. For instance, in the image processing community, the terms “salience” and “importance” (or Visual Salience and Region of Interest/Importance) have sometimes been considered synonymous, while, as stated, they should be distinguished. Both denote the most visually “relevant” parts of the scene. However, the concepts differ as they may refer to two different mechanisms of visual attention: bottom-up vs. top-down. While the interaction between ICT and vision science is intensifying, the ICT community needs to assure carefully that the proper tools (models, validation protocols, databases, …) are used for the proper needs.

Biographies

Patrick Le Callet received both an M.Sc. and a PhD degree in image processing from Ecole polytechnique de l’Université de Nantes. He was also a student at the Ecole Normale Superieure de Cachan where he sat the “Aggrégation” (credentialing exam) in electronics of the French National Education. He worked as an Assistant Professor from 1997 to 1999 and as a full time lecturer from 1999 to 2003 at the Department of Electrical Engineering of Technical Institute of the University of Nantes (IUT). Since 2003 he teaches at Ecole polytechnique de l’Université de Nantes (Engineering School) in the Electrical Engineering and the Computer Science departments where is now a Full Professor. Since 2006, he is the head of the Image and Video Communication lab at CNRS IRCCyN, a group of more than 35 researchers. He is mostly engaged in research dealing with the application of human vision modeling in image and video processing. His current centers of interest are 3D image and video quality assessment, watermarking techniques and visual attention modeling and applications. He is co-author of more than 140 publications and communications and co-inventor of 13 international patents on these topics. He also co-chairs within the VQEG (Video Quality Expert Group) the “Joint-Effort Group” and “3DTV” activities. He is currently serving as associate editor for IEEE transactions on Circuit System and Video Technology, SPRINGER EURASIP Journal on Image and Video Processing, and SPIE Electronic Imaging.

Ernst Niebur graduated with an MS degree (Diplom Physiker) from the Universität Dortmund, West Germany. He received a Post-Graduate Diploma in Artificial Intelligence from the Swiss Federal Institute of Technology (EPFL), Switzerland, and the Ph.D. degree (Dr és sciences) in physics from the Université de Lausanne, Switzerland. His dissertation topic was a detailed computational model of the motor nervous system of the nematode C. elegans.

Niebur was a Research Fellow and a Senior Research Fellow at the California Institute of Technology, Pasadena, and an Adjunct Professor at Queensland University of Technology, Brisbane, Australia. He joined the faculty of Johns Hopkins University in 1995 where he is currently a Professor of Neuroscience in the School of Medicine, and of Brain and Psychological Sciences in the School of Arts and Sciences. He uses computational neuroscience to understand the function of the nervous system at many levels.

Niebur was the recipient of a Seymour Cray (Switzerland) Award in Scientific Computation in 1988, an Alfred P. Sloan Fellowship in 1997, and a National Science Foundation CAREER Award in 1998.