On the number of perceivable blur levels in naturalistic images

Abstract

Blur is a useful cue for depth. Natural images contain objects at a range of depths whose depth can be signaled by their perceived blur. Here, to evaluate the usefulness of blur as a depth cue, we estimate the number blur levels that observers can perceive simultaneously. To estimate this value, observers discriminated and classified dead leaves patterns that contained a controlled distribution of blur levels but are more complex or naturalistic than stimuli typically used in blur research. We used a 2-IFC discrimination task, in which observers reported the interval that contained more blur levels and a classification task, in which observers reported the number of perceived blur levels. In both tasks, observers could not discriminate or classify more than four levels of blur in the stimulus reliably. In isolation from other cues, blur may provide only a coarse cue to depth and add limited depth information when present in natural scenes with complex distributions of blur and multiple depth cues.

1. Introduction

Blur is often present during the viewing of scenes. Image blur has the drawback in that it can affect perceived image quality (Ahumada, 1993), but it has been used artistically by photographers (Massey and Bender, 1996) in tilt-shift photography to delight viewers. In biological optical systems, blur places a fundamental limit on the information that can be transmitted (Banks et al., 1987). Retinal blur drives accommodation (Ciuffreda, 1991; Ciuffreda et al., 1990; Fisher and Ciuffreda, 1988) and has been implicated in eye growth and the development of myopia and hyperopia (Walker et al., 1978; Hodos and Kuenzel, 1984).

Blur has the ability to signal the range of depths present in a natural image (Fisher and Ciuffreda, 1988) and presenting images at a single focal plane may be the reason why 3D displays look flatter than one would expect (Watt et al., 2005). Recently, (Maiello et al., 2014) examined fusion and vergence in stereoscopic images of natural scenes captured with a plenoptic camera. The addition of gaze-contingent dioptric blur, proportional to the real depths of objects in scenes, reduced the time to fusion and vergence instability. Burge and Geisler (2011) showed that defocus blur is a useful depth cue, but their stimuli contained only one level of blur (i.e., there was no variation in depth within their images) which is generally not true of natural scenes. A question that remains unanswered is how useful blur is for signaling depth when processing complex scenes that contain multiple levels of depth or blur. At least three factors each place constraints on the utility of blur as a depth cue, how finely blur can be discriminated, the number of discrete blur levels that can be perceived simultaneously, and the ability of blur to signal the sign (whether an object is blurred because it is front/behind the point of interest).

Observers can discriminate small changes in image blur (Murray and Bex, 2010; Mather and Smith, 2002; Wang and Ciuffreda, 2005; Watson and Ahumada, 2011) which indicates that blur discrimination between two levels of blur is unlikely to be a major constraint for blur as a depth cue. The second factor, the number of discrete blurs levels that can be encoded and signaled simultaneously by the visual system, has yet to be investigated and addressing this question is the chief aim of this paper. Natural scenes typically contain objects at a range of depths which results in variation in blur across the retina. The ability of the visual system to use the variation in blur as a cue to the multiple depth planes depends on the number of depths that the visual system can signal simultaneously in a glance. Finally, it is not clear whether the sign of depth can be signaled by blur. There is some evidence (Nguyen et al., 2005) that the blur from chromatic aberration can be used to determine the sign of blur, but whether other blur cues (e.g., high-order monochromatic aberration) can signal the sign of depth from blur is unknown.

Although the functional impact of blur of blur is usually assessed with visual acuity, the influence that blur has on image quality is ultimately evaluated by observers when viewing natural scenes which are much richer stimuli than the edges and letters typically used to study perceived blur. The use of natural images to study perceived blur has been complicated by the lack of a single standard computational approach to estimate perceived image blur from image statistics; several methods that can estimate local blur have been described (see Elder, 1999; Marziliano et al., 2002; Wang and Simoncelli, 2003; Vu et al., 2012; Taylor and Bex, under review, for examples) but no single method has been adopted as a standard. Given the lack of a standard method for measuring and manipulating local blur in natural sciences, we adopted a simpler and more controllable stimulus that captures some of the characteristics of natural scenes as a compromise. We used naturalistic ’dead leaves’ stimuli (see Figure 1) that contains the range of contrast values present in natural scenes, edges at a variety of orientations, occlusions, and a variety of blur levels. The use of these stimuli allows us to investigate local perceived blur as a depth cue in a highly controlled way. However, this method does introduce some limitations including the difference between Gaussian versus dioptric blur, the identity between near and far blur and a failure to match blur to an individuals optics, which may have been learned by the observer (Nguyen et al., 2005).

Figure 1.

Examples of the dead-leaves patterns used in Experiments 1 and 2. Each image shows a dead leaves patch that contains a number of blur levels. The first column shows dead leaves patches with one through four blur levels, and the second column the dead leaves patches with five through eight blur levels.

The work in this paper was inspired by the literature on transparency perception for motion and depth stimuli. Several researchers have measured how many stimulus levels can be simultaneously perceived for moving dot patterns (see Mulligan, 1992; Edwards and Greenwood, 2005; Edwards and Rideaux, 2012) and depth images defined by stereoscopic disparity (Weinshall, 1989; Akerstrom and Todd, 1988; Gepshtein and Cooperman, 1998; McKee and Verghese, 2002; Parker and Yang, 1989;Wallace and Mamassian, 2004;Weinshall, 1991; Tsirlin et al., 2008). Coherent unidirectional motion can be perceived in random dot stimuli containing broad distributions of direction signals (Williams and Sekuler, 1984) and direction discrimination thresholds in random dot stimuli are as low as 1–2 degrees (De Bruyn and Orban, 1988). However, when multiple direction stimuli are combined, observers are only able to perceive at most two directions (Edwards and Nishida, 1999; Edwards and Greenwood, 2005) or three if additional depth (Greenwood and Edwards, 2006b) or speed (Greenwood and Edwards, 2006a) segmentation cues are added.

Similarly in depth perception, while stereoacuity can be extremely fine most people can perceive disparities of only a few arc seconds (Coutant and Westheimer, 1993), but only two to four stereoscopic depth planes can be perceived simultaneously. We take a similar approach in our measurement of the number of blur levels that can be perceived simultaneously in dead-leaves patterns. Dead-leaves patterns match the statistics of natural images (Lee and Mumford, 2001) and are generated from occlusions, similar to the provenance of edges in natural scenes (Ruderman, 1997). Dead-leaves like the random dot patterns used in the work on motion and stereoscopic depth perception described above, are limited in that they lack the richness and variety of natural scenes (e.g., object, perspective, and surfaces cues) but arguably carry enough information to make general conclusions about the perception of blur as the random dot stimuli do for motion or stereoscopic depth perception. Moreover, by eliminating these cues, dead leaves patterns allow us to study basic sensory mechanisms in isolation of other visual cues. Our primary aim was to measure the number of blur levels that can be perceived; our stimulus isolates perceived blur from stereo (Wardle et al., 2012) surface gradients, segmentation (Wallis and Bex, 2011), texture (Cao et al., 2010), monocular depth cues (other than occlusion), and motion. Despite sacrificing some naturalism and introducing limitations on generalization, the dead-leaves stimuli we use are more naturalistic than other synthetic stimuli used in past blur research and allow us to address our key question – in the absence of other depth cues, how many levels of perceived blur can be estimated by the visual system simultaneously?

2. Methods

2.1. Observers

Four male observers participated in the experiment (including both authors), all of whom wore their normal correction and had previous experience in psychophysical tasks. Their ages ranged from 23–45 and one observer, PJB, is an amblyope. All observers were tested binocularly. The experiments were approved by the Harvard Medical School Internal Review board and carried out in accordance with the Code of Ethics of the World Medical Association (Declaration of Helsinki).

2.2. Apparatus & Stimuli

A PC with an NVIDIA Quadro FX 4600 10-bit graphics card was used to drive a Sony PVM-2541 OLED display. The display was 54.3 cm wide by 30.5 cm high and set to a resolution of 1920 by 1080 pixels. The display was confirmed to be able to produce 10-bit gray scale resolution (Elze et al., 2013) and linearized with a PhotoResearch PR-655 photometer. The display was set to a mean luminance of 80 cd/m² and a maximum luminance of 140 cd/m² to avoid saturation artifacts (Elze et al., 2013). Stimuli were generated under Ubuntu Linux 12.04 LTS with MATLAB 2011a (64-bit) and the Psychophysics Toolbox 3 (Brainard, 1997; Pelli, 1997; Kleiner et al., 2007).

Dead leaves patterns (Bordenave and Gousseau, 2006; Lee and Mumford, 2001) like those shown in Figure 1 were used as stimuli. The dead leaves were constructed from a set of 128 ellipses each drawn with a pseudo-random center position, orientation, aspect ratio, and luminance. The ellipse centers were positioned within a 256 by 256 patch that subtended approximately 7 by 7 degrees of visual angle in the center of the screen at the viewing distance of 60 cm.

In the dead leaves stimuli, each ellipse was independently blurred with a Gaussian low pass filter. There were eight levels of Gaussian blur, σ 1, 2, 4, 8, 16, 32, 64, 128 cy/image (90’, 45’, 22.5’, 11.3’, 5.6’, 2.8’, 1.4’, 0.7’, respectively). For each dead leaves stimulus, a random subset of one to eight blur levels was selected from this range and assigned to the elements of the pattern. Thus the total number of different blur levels in any dead leaves stimulus was between one and eight. In Figure 1, example stimuli containing one through eight levels of blur are shown. The number of elements with a given assigned blur level was equated when possible, but for blur levels that produced remainders, the remaining ellipses were assigned a random blur levels without replacement from the set. For example, in a patch with three blur levels, of for example 2, 16, and 64 cy/image, the two remaining ellipses could be assigned levels 2 and 64, but not 2 and 2.

2.3. Procedure

Observers completed blur classification and discrimination tasks. Each trial began with a white fixation point, then observers viewed dead leaves for 500 ms and responded with a key-press how many blur levels were present in the stimulus, one through eight. After reporting the number of blur levels, observers then reported the level of confidence on a scale of 1–8 of their response. Observers then received feedback, via a change in color of the fixation point (green for correct, red for incorrect) whether their classification response was correct. The feedback did not indicate whether their estimate was too low or too high. After a brief pause, the fixation point was re-drawn and the next trial began. All observers ran at least 250 classification trials for each blur level for a total of 2,000 trials (Observer GMM completed double the number of trials).

The discrimination task was usually completed on the same day in a counter-balanced order with the classification task. Two dead leaves stimuli were viewed sequentially in a 2-IFC paradigm. One of the intervals, at random across trials, contained a target stimulus the other contained a foil stimulus. The target stimulus was a dead-leaves patch composed of 128 ellipses that were individually blurred with N different blur leaves, where N was between 2 and 8. The blur levels were randomly selected from the set of 8 blur levels, as illustrated in Figure 1. The foil stimulus contained N−1 blur levels. Each patch was presented for 500 ms, with each stimulus separated by a blank interval of 500 ms. Observers reported, by pressing one of two keyboard buttons, whether the first or second interval contained a stimulus with a greater number of blur levels. Observers then rated their confidence of their discrimination response (on a scale from 1–8). After their confidence judgment, they were provided with feedback (green if their choice of interval was correct, red if it was incorrect). Observers ran at least 250 discrimination trials for each blur level for a total of 2,000 trials (Observer GMM did double the number of trials).

3. Results

3.1. Discrimination d′

We used standard 2-IFC signal detection analyses (Macmillan and Creelman, 2004) to compute d′ for our discrimination task. The results were checked for interval biases, which have been suggested to contaminate estimates of sensitivity by introducing bias and altering sensitivity (Yeshurun et al., 2008); we used a procedure to test for this recommended by (Yeshurun et al., 2008) and Figure 3 shows that none were found. If the 2-IFC procedure introduced bias and/or sensitivity changes the points in 3 would deviate substantially from the red line.

Figure 3.

Proportion correct in the first interval plotted against that found second interval for our four observers.

Figure 2 shows how d′ decreases as the number of blur levels in the stimulus increases. When the number of blur levels is increased to four d′ is near zero. Confidence intervals, calculated via the method described by Macmillan and Creelman (2004), were computed at each blur level pairing and are, in most cases, smaller than the symbols. To test whether the lower-limit of approximately four perceivable blur levels was task dependent, we also measured observers classification performance.

Figure 2.

Blur level discrimination results from four observers. d′ is highest when the observer discriminates a patch with one blur level from two blur levels. Sensitivity then drops substantially when the correct patterns contains three or four blur levels. With greater than four blur levels, observers’ sensitivity is approximately at chance.

3.2. Classification d′

Classification d′ was computed using previously described methods (Macmillan and Creelman, 2004; Braida and Durlach, 1972) but because it is not commonly used a brief summary will be provided here. Each of the eight stimuli and eight responses combine to generate a stimulus-response matrix, in which each row represents the stimulus presented and each column the observers’ response. Each entry in the matrix is then computed by normalizing the number of responses by the number of stimulus presentations and adjacent entries in the matrix of z -transformed proportion of responses can be subtracted to (e.g., two-blur levels minus) yield a d′ value.

Figure 4 shows d′ versus the number of blur levels in the stimulus. The data are similar to those from the discrimination task; the ability to classify the number of blur levels in a pattern is best when there are a small number of blur levels present in the signal. When the number of blur levels exceeds three or four, observers are unable to classify the patterns correctly. Classification is an intrinsically more difficult task given that the observers are uncertain which of eight signals are presented on any given trial.

Figure 4.

Blur level classification results from four observers. Classification d′ was calculated using the method described in the text.

3.3. Comparing discrimination and classification sensitivity

To determine if the task influenced the number of perceivable blur levels, we compared our discrimination and classification data using a SDT framework. This framework has been used previously to relate discrimination and classification (Braida and Durlach, 1972; Macmillan, 1987) for several tasks.

Discrimination in a 2IFC discrimination task is limited by two factors, expressed in the equation:

$$d\prime =\frac{\mathrm{\Delta }\mu }{\beta }$$ (1)

In the equation, Δμ is a sensory distance that is determined how far apart any two stimuli are on a perceptual dimension, and β is the variance of a general noise in the sensory system.

Classification is a more complex task than discrimination under a SDT framework because it is influenced by the observer’s ability to remember the range of signals that can be presented. This difference in difficulty is reflected in Equation 2.

$$d_{\mathit{\text{classifcation}}}^{\prime }=\frac{\mathrm{\Delta }\mu }{{({\beta }^2+G{R}^2)}^{\frac{1}{2}}}$$ (2)

In Equation 2, GR² is a term that describes how memory for the range of possible signals influences d′, β² is the sensory distance squared from Eq. 1. A discrimination task is a much simpler task than classification because it does not require a term for memory to model sensitivity. In a classification task, a memory of the range of possible signals that could be presented is required, and if memory for the range of possible signals was a factor for our observers, the task may place a limit on the number of blur levels that could be perceived. To test whether the complexity of the classification task influence observers’ sensitivity to the number of blur levels in the patch, we compared the sensitivity between the two tasks by calculating cumulative d′ for each task (the sum of d′ across all the blur levels).

Figure 5 shows cumulative d′ for four observers in both the classification and discrimination tasks. The difference between cumulative d′ for all observers in the two tasks is not significantly different from zero. The lack of difference in sensitivity between the two tasks demonstrates that the relative insensitivity to the number of blur levels is not specific to the task and does not depend whether observers are performing a discrimination or classification, lending generality to our finding that observers can only perceive a limited number of blur levels.

Figure 5.

Classification results from four observers. The difference of classification d′ and classification d′ for each of the blur levels was calculated and the mean computed. The error bars are within-subject 95% confidence intervals. If number of perceivable blur levels does not differ between the two tasks confidence interval ought to include zero (red line).

4. Discussion

We used a set of naturalistic and controllable dead-levees stimuli to measure the number of blur levels that can be perceived simultaneously. We found that observers have difficulty discriminating images that contain greater than four unique levels of blur. Our classification results show that observers can not reliably classify more than three blur levels. Our signal detection analyses demonstrate that our results generalize from discrimination to classification. Despite classification being a more difficult task from a signal detection point of view, our analysis showed no evidence of task being a limiting factor on the number of perceivable blur levels. In all cases the blur levels we used are easily discriminable when presented in an image containing a single blur level (Wang and Ciuffreda, 2005), but when several levels of discriminable blur are present, observers are unable to differentiate more than three or four levels. One interpretation of these results is that image blur is a coarse cue for the perception of depth that can only signal a limited number of unique depths. However, it is important to be clear about the limitations inherent in these experiments, limitations that may constrain how generalizable the conclusions from our results could be.

Our choice of stimulus and procedure, specifically our choice of spatial (patch size) and temporal (stimulus duration) parameter, place two limits on the generality of our findings. These limitations are comparable to the majority of psychophysics research in general, nevertheless, it is worthwhile to recognize these limitations and discuss how they constrain our conclusions. The stimulus size and duration we employed are comparable to many previous studies of blur discrimination, but it is possible that observers could discriminate and/or classify a greater number of blur levels if the stimulus duration were larger or longer (or unlimited). Indeed, the number of planes that can be perceived increases (up to a maximum of four) when the stimulus duration is increased (Akerstrom and Todd, 1988). Also, with our relatively brief stimulus presentation, observers could not make more than one or possibly two eye-movements during the stimulus presentation. These parameters intentionally precluded observers from building up a representation of depth based on blur estimates across multiple fixations. In real environments, observers are generally able to make multiple fixations in a scene, although natural selection pressures may favor species that can make rapid, accurate, and precise decisions about the positions of predators or prey. A larger stimulus size and/or duration could perhaps allow observers to extract more information about the number of blur levels from our stimulus. Thus, we wish to make it clear that we do not intend to generalize our findings beyond the spatial/temporal parameter space tested. However, within this parameter space there is a clear and measurable limit to the number of blur levels that can be perceived simultaneously.

The stochastic local spatial structure of our stimulus also places a limit on the generalizablity of our conclusions. It is possible that if our stimulus contained surface structure, then many more blur levels could be perceived simultaneously. It is also possible that a only a few levels of blur could be used to signal depth when they act together with other two- and three-dimensional depth cues. For example vergence/accommodation are both powerful cues to depth (Held et al., 2012a) and may over-shadow the depth signal that blur provides. Our experiments were not designed to measure cue combination for stimuli containing multiple depth cues, but to isolate the role of blur as a depth cue, thus it remains to be seen if either of the above conjectures are supported.

It is worth considering the interpretation that our stimuli, especially when they contain elements with largest blur level, do not carry the information necessary to reveal blur as a fine cue to depth in our tasks. For example, Ciuffreda et al. (2006) observed that if letters or text had greater than approximately 1.5 to 2 diopters of blur the stimulus had, in their terms, unrecognizable blur. There are several reasons to doubt that the dead-leaves stimulus lacks the information to provide observers with blur as a fine depth cue because blur in the stimulus is unrecognizable. First, unlike experiments that use letters as stimuli, the spatial structure of the individual blurred elements in our stimuli is irrelevant to our tasks; whether classifying or discriminating the patterns, the observer was not required to identify or recognize any individual element of the stimulus only to report the overall number of blur levels perceived. That the stimuli are composed of blurred ellipses which are probably not individually identifiable or recognizable themselves in the pattern, irrespective of the amount of blur added to them, argues against an interpretation that the presence of the most blurred of the elements reduced the amount of information available. Second, experiments that measure the discrimination of the mean blur of dead-leaves patterns argue against the idea that there is too little information in the stimulus to perform the task, we previously found that all the levels of blur contained within the stimulus were individually discriminable from one another (Taylor and Bex, 2011). Finally, the stochastic nature of the stimulus means that, even at a number of levels that generally can not be discriminated or classified, our dead-leaves stimulus may not have contained the most blurred elements. Specifically, about half the time the four level blur dead-leaves stimulus would not contain the blurriest elements (i.e., elements whose blur may be unrecognizable) and one third of the time it would not contain the two largest blur levels. Thus, the above factors taken together make the idea that the patterns did not contain sufficient information to discriminate or classify the number of blur levels an improbably/unlikely interpretation of our main findings.

What might be the functional role of the depth signal provided by blur? Recently, blur has been shown to draw faster fixation and manual responses under some experimental conditions (Enns and MacDonald, 2013), implying that blur can be used flexibly depending on the observers’ task goals. Perhaps, blur provides a first-pass depth cue to the observer that operates quickly but coarsely to provide depth information to guide behavior.

Our results are reminiscent of studies of the number of motion directions that can be perceived simultaneously, although small directional differences (approximately 1–2 degrees) can be discriminated in random dot stimuli (De Bruyn and Orban, 1988), observers are only able to identify 2 directions of motion in overlapping random dot stimuli (Edwards and Nishida, 1999; Edwards and Greenwood, 2005). The number of motion direction signals that can be perceived simultaneously is limited by thresholds for bi-directional motion. Adding more motion direction signals becomes equivalent to adding motion noise in direction selective channels, leading to the coherence required to perform the task exceeding that which can be detected and a ceiling of around two or three directions that can be perceived simultaneously (Edwards and Nishida, 1999; Edwards and Greenwood, 2005). This does not mean that all stimuli containing multiple motion directions appear as noise - clearly global patterns of radial and rotational motion can be perceived as moving coherently, even though they contain directions spanning 360 degrees. We find a similar limit for the detection of multiple blur levels. This result implies that the mechanisms that are used to estimate image blur may be broadly tuned or sum across multiple spatial and/or orientation channels (Taylor et al., 2009, 2014) and therefore respond to adjacent blur levels in a complex stimulus. A population of mechanisms, such as spatial frequency channels whose responses vary with blur, can encode small differences in the blur of sequential spatial or temporal intervals containing a single level of blur (Watson and Ahumada, 2011). However, the responses of the same population to single stimuli containing multiple levels of blur in close proximity (as in natural images) may be highly similar, i.e. within the range of normal response variability, when more than 3 blur levels are present. As with global patterns of radial or rotational motion, it is possible that if a broad distribution of blur levels are present in a single image, but organized in a spatially coherent global pattern as in perspective gradients, many levels of blur may be perceived. However, to our knowledge, nobody has yet examined whether observers could detect quantization of the blur levels in such stimuli.

Recent work (Burge and Geisler, 2011; Held et al., 2012a) has shown that blur may be a useful cue to depth. There is some controversy in the literature: Vishwanath (2012) criticized Held et al. (2012a) and asserted that observers may have been using a 2D strategy (i.e., which patch is blurrier), because blur was always positively correlated with depth in their stimuli. Held et al. (2012b) responded by noting that in 3D scenes a 2D strategy is generally available. Although adding gaze-contingent dioptric blur to stereoscopic natural images reduces the time to fusion and vergence instability including blur in these images decreases stereoscopic acuity because it removes high spatial frequencies that are necessary for fine depth discrimination (Maiello et al., 2014). However, because blur increases with distance from the plane of accommodation (i.e., objects closer to the observer’s plane of fixation or further from it) the sign of blur must be available to be useful. It is not currently known if observers are able to infer the sign of blur from their own optical aberrations, although there is some evidence that the blur resulting from chromatic aberration carries information about the sign (Nguyen et al., 2005). Of course, our stimuli that were 2D but our results can be interpreted that blur, if it is a cue that contributes to 2D depth, is a very coarse depth cue.

Blur as a depth cue may be coarse as our results suggest, but our findings are limited by our choice of stimuli and procedure, our exploration of the parameter space (e.g., patch size, stimulus duration), the lack of stereo depth cues in the dead leaves (Wardle et al., 2012, e.g.,), and how we applied blur to the dead-leave elements. Using a stimulus with natural surfaces, stereo cues, and another type of blur may lead to larger estimates of the number of perceivable blur levels. In conclusion, our finding that there is a limit to the number of reliably simultaneously discriminable/classifiable blur levels places a possible constraint on how blur might operate as a depth cue, but our choice of stimulus leaves open many branches to explore in this field.