Change blindness for cast shadows in natural scenes: Even informative shadow changes are missed

Abstract

Previous work has shown that human observers discount or neglect cast shadows in natural and artificial scenes across a range of visual tasks. This is a reasonable strategy for a visual system designed to recognize objects under a range of lighting conditions, since cast shadows are not intrinsic properties of the scene – they look different (or disappear entirely) under different lighting conditions. However, cast shadows can convey useful information about the three-dimensional shapes of objects and their spatial relations. In this study, we investigate how well people detect changes to cast shadows, presented in natural scenes in a change blindness paradigm, and whether shadow changes that imply the movement or disappearance of an object are more easily noticed than shadow changes which imply a change in lighting. In Experiment 1, a critical object's shadow was removed, rotated to another direction, or shifted down to suggest the object was floating. All of these shadow changes were noticed less often than changes to physical objects or surfaces in the scene, and there was no different in detection rates for the three types of changes. In Experiment 2, shadows of visible or occluded objects were removed from the scenes. Although removing the cast shadow of an occluded object could be seen as an object deletion, both types of shadow changes were noticed less often than deletions of the visible, physical objects in the scene. These results show that even informative shadow changes are missed, suggesting that cast shadows are discounted fairly early in the processing of natural scenes.

Shadows are ubiquitous features of everyday scenes, but we rarely give them much attention, except in some rare circumstances. For example, we may notice a particularly dramatic long shadow at sunset, or actively search for shadows when looking for a shady spot to sit on a hot day. But in general we have little reason to attend to shadows in daily life, and we don't seem to encode much information about shadows: for example, we are generally poor at recognizing impossible shadows in scenes (Ostrovsky, Cavanagh, & Sinha, 2005) and worse at detecting changes to shadows than changes to objects (Wright, 2005). However, most of this previous work looked at scenes where the shadow information was largely redundant, and changing a shadow would not have changed anything about the meaning of the scene. Here we look at whether people neglect shadow changes that do affect the meaning of the scene, e.g., by changing the number of objects or their spatial relations.

In these experiments, we focus on cast shadows, which are the shadows a light-occluding object casts onto an external surface, such as the ground. These should be distinguished from attached shadows, which are the gradations in shading on the surface of an object, and self shadows, which are shadows cast by one part of an object onto another part of its own surface. All shadows are produced when light emanating from a source interacts with objects in its path. The brightness (or luminance) that we perceive when looking at an object or surface is the product of the amount of light hitting that surface (illuminance) and the actual color of the surface (reflectance).

In order to recognize objects under a variety of lighting conditions, a visual system must be able to subtract out the effects of illumination in order to perceive the true surface colors of objects (Land & McCann, 1971; Adelson, 2000, Rolls & Stringer, 2006; DiCarlo & Cox, 2007). Cast shadows, in particular, should be discarded because they are produced by a change in illumination only: the surface appears darker because it receives less light, not because there's any actual difference in the surface reflectance. Experiments have shown that people discount shadow information across a variety of visual tasks, although there is some debate about the mechanisms involved in this discounting process. People are poor at identifying lighting inconsistencies in complex object arrays or detecting incorrect cast shadows in natural scenes (Ostrovsky, Cavanagh, & Sinha, 2005). In change blindness tasks, people are slower to detect a disappearing cast shadow than a disappearing object (Wright, 2005). People seem to discount cast shadows in search tasks, in that they are slower to detect an odd shadow in an array than an equivalent odd object (Rensink & Cavanagh, 2004). This could be taken as evidence for an early visual processing stage that identifies cast shadows in images and corrects for them, but later experiments suggest that cast shadows are retained in early vision and that the discounting occurs later, as part of the object recognition process (Porter, Tales, & Leonards, 2010). Furthermore, other studies have shown rapid, “pop-out” search for odd shadows that suggest a different lighting direction or change in object depth, which provides more evidence that cast shadow information is available in early visual processing (Elder, et al., 2004). The presence or absence of cast shadows seems to have no effect on object recognition from photographs (Braje, Legge, & Kersten, 2000), but later experiments using a more varied set of computer-generated objects found that recognition is slightly slowed when cast shadows are absent or incorrect (Castiello, 2001). Experiments using novel objects have found a small recognition penalty when the cast shadow information changes between learning and test (Tarr, Kersten, & Biilthoff, 1998; Leek, Davitt, & Cristino, 2015). This suggests that some cast shadow information is retained during object processing, though it may only be useful for identifying unfamiliar, artificial objects.

Although cast shadows give incorrect information about the surface on which they're cast, they can give very useful information about the casting objects. In particular, cast shadows can provide 3D information that isn't otherwise available in the scene and can be used to disambiguate the positions of objects in depth (Mamassian, Knill, & Kersten, 1998). For example, cast shadows can be used to determine whether an object is resting on a surface or floating above it, or determine which of two ambiguous surfaces is supporting the other – assuming light from above, an object can cast a shadow onto a supporting surface, but not vice versa. People's perception of depth and spatial position of objects in scenes seems to be very dependent on cast shadows, and will perceive the movement of an object's cast shadow as movement of that object in depth, even when other cues in the scene contradict that interpretation (Kersten, Mamassian, & Knill, 1997).

Cast shadows are potentially useful to the visual system because they provide a second two-dimensional projection of the scene (Casati, 2004; Dee & Santos, 2010). They depict the shapes of objects as “seen” from the point of view of the light source, which must necessarily be a different point of view than the one observed by the eye (Da Vinci, ca. 1508, 1978). A second projection of the scene can provide useful 3D information, which is why shadows are so valuable for disambiguating object depth and establishing spatial relationships between objects and surfaces. In this sense, shadows are somewhat similar to reflections – they provide incorrect information about the surface on which they're cast (a shadowed surface isn't really dark and a mirror doesn't actually contain depth), but provide a second view of the scene which may contain information that's not otherwise visible.

In this study, we investigate the discounting of shadows in natural scenes using a change blindness paradigm similar to one previously used to study reflections in natural scenes (Sareen, Ehinger, & Wolfe, 2014). The experiments make use of the “flicker paradigm” of Rensink, O'Regan & Clark (1997) in which two versions of a scene alternate with a brief blank period between each presentation. The observer attempts to locate the difference between the two frames. Under these conditions, even quite large changes in a scene can go unnoticed for many seconds. For change to be noticed, the item being changed must be attended before and after the change. With very simple displays like arrays of colored squares, observers are able to keep track of the state of 3 or 4 items from frame to frame (Luck & Vogel, 1997; 2013). With scenes, there are typically many more items that could be the focus of attention so change blindness becomes a way of assessing what attracts attention in a scene, at least when the observer is looking for a change. In detecting change, low-level salience does not appear to be as important as the meaning of items in a scene (Stirk & Underwood, 2007) and changes in the existence or position of an object are more readily detected than changes in surface properties like color (Aginsky & Tarr, 2000). Wright (2005) looked at change blindness for deletion of cast shadows, and found that observers were not good at detecting them. Apparently, having a shadow go out of existence is not like having an object go out of existence. People may be particularly insensitive to this type of shadow change because it doesn't change the meaning of the scene. It does add a lighting inconsistency, but people are poor at recognizing lighting inconsistencies in natural scenes (Ostrovsky, Cavanagh, & Sinha, 2005). In the present study, we consider cases where a change to a cast shadow affects the spatial relations of objects in the scene (Experiment 1) or effectively deletes an implied object from the scene (Experiment 2). We compare these types of informative shadow changes to object changes and less informative shadow changes such as deletion or rotation of cast shadows, which add a lighting inconsistency but do not alter the meaning of the scene.

Experiment 1

We first looked at whether there were different detection rates for different types of cast shadow changes in scenes. We compared two types of shadow changes which should not significantly affect the meaning of the scene – shadow deletion and rotation – to a change which arguably does change the gist of the scene: shifting a cast shadow downward to suggest that the casting object is floating. This turns an ordinary scene into an impossible scene with an object that violates the laws of physics. These shadow manipulations were compared to two control conditions in which changes occurred to objects or surfaces in the scene.

Methods

Participants

Twenty-one people participated in the online experiment on Amazon Mechanical Turk. Participants were based in the U.S. and had a good track record on the Mechanical Turk site (at least 100 HITs completed and an acceptance rate of at least 95%). All participants gave informed consent before starting the task. Participants were paid $1.00 upon completing the task and an additional $0.10 for each correct response, resulting in an average total payment of about $14.00.

Stimuli

The stimuli images were photographs of everyday scenes. They were 24 bit color and scaled to a resolution of 1024 pixels wide by 768 pixels high (a few images had a different aspect ratio but were scaled to the maximum possible size within this box). There were 60 images used in the experimental trials and an additional 150 images used as “fillers” to disguise the fact that the main experimental manipulation involved shadows.

We created a second version of each image by changing one object or region using Photoshop. There were five change conditions for the experimental images and each image appeared in all five conditions, but an individual subject would only see one of the five versions. The five conditions for a single example image are shown in Figure 1. On object change trials, a random object in the scene would either disappear or change color (each type of change occurred on half of experimental trials). On shadow deletion trials, one object's shadow would disappear. On shadow rotation trials, the same shadow would be deleted and redrawn as though the object were lit from another direction. On floating shadow trials, the shadow was shifted downward to suggest that the object was floating above its supporting surface. On surface change trials, a patch with the same size, shape, and color values as the original shadow was added to a flat surface in the scene. This patch was located away from objects so it would not be mistaken for a cast shadow, and was blurred to match the original shadow's penumbra. The images used in this experiment (and Experiment 2) are available online as part of the Change Blindness (CB) database at http://search.bwh.harvard.edu/new/CBDatabase.html (Sareen, Ehinger, & Wolfe, in press).

Figure 1.

An example image (a) and the change conditions (b-f) used in Experiment 1. In the flicker change blindness paradigm, the original image is shown alternately with one of these modified versions, which a brief blank screen in between. The changes in this image are: (b) the girl's shorts change color, (c) a dark patch appears in the sand near the bottom right, (d) the girl's shadow is deleted, (e) the girl's shadow changes direction, (f) the girl's shadow is moved so that she seems to float above the sand.

Design

Each participant saw 6 images from each of the five experimental conditions; the image selection was random and counterbalanced across subjects so each image occurred equally often in each condition across the experiment. These 30 experimental trials were randomly interleaved with 150 filler trials. The filler trials were the same for every subject and included 114 trials with object changes (half were object deletions and the other half were object color changes), and 36 trials with no change. Trial order was randomized for each subject. Across the experiment, 80% of trials had a change, and 16.67% of those changes involved shadows.

Procedure

Participants were told that the experiment would involve finding changes in scenes and given an example of a change blindness trial with an object deletion. Images were presented in a web browser using jsPsych (de Leeuw, 2015). On each trial, participants saw a pair of images, presented alternately one after the other for one second each with a 250 ms blank in between. On each trial, the starting image was chosen at random. This sequence repeated 24 times (one minute) or until the participant pressed a response key on the keyboard: “Y” if they saw a change, or “N” if they thought there was no change. Keypress responses and reaction times were recorded.

Next, participants were shown one of the two images from the change blindness sequence. This probe image was always the original image from the pair (the one with the original shadow, or the original object present) except on surface change trials, which showed the modified image with the extra surface feature present. Participants were asked to click on the location of the change. If the participant had not detected a change, they were asked to skip this step. There was no time limit on this step and reaction times were not recorded.

After each trial, participants were shown the number of trials remaining. They could press a key to start the next trial, or quit the experiment and return to it later. Participants were encouraged to complete the trials on their own time and take breaks whenever they wished. There was no feedback after the trials.

Results

One participant who detected only a single change during the experiment was dropped and replaced. The data was screened for very short response times (< 250 ms) and very long response times (> 3 standard deviations above the mean responses time, not including trials that timed out), but no trials were dropped due to these criteria.

We compared accuracy, responses times, and the number of timed-out trials across the five experimental conditions, shown in Figure 2. Experimental trials were marked “correct” if the participant responded “yes” and correctly marked the location of the change, and “incorrect” if the participant responded “no” or failed to mark the change location, or if the trial timed out with no response. A one-way repeated-measures ANOVA showed a significant effect of condition on accuracy (F(4,19) = 10.89, p < 0.001). Bonferroni-corrected pairwise t-tests showed significant differences between each of the two most accurate conditions (object change, surface change) and each of the three least accurate conditions (shadow deletion, shadow rotation, floating shadow), but no significant differences within either of these groups. A similar analysis was performed to compare reaction times on correct trials (four participants were dropped from this analysis because they had no correct trials in at least one condition). A one-way ANOVA showed no significant difference in reaction times across conditions (F(4,15) = 1.88, p = 0.125). There was also no significant difference in number of trials that timed out across the experimental conditions (F(4,19) = 0.41, p = 0.802).

Figure 2.

Response time (left), accuracy (center), and percent of trials which timed out across the five types of changes in Experiment 1. Error bars represent standard error of the mean.

Most of the errors were failures to detect the changes. These are trial on which participants either said that there was no change or the trial timed out before they detected the change. Only 2.6% of the errors (28 trials) were location errors: people said there was a change but clicked on a wrong object or region. We checked whether there were any trials where participants misinterpreted the shadow change as an object change and clicked the object, but there were no errors of this type. The low incidence of location errors is probably due to the reward scheme in our task: participants received bonuses for correct localization or correctly rejecting the trial as a “no change” trial. There was no bonus for correctly detecting a change but marking the wrong location, which probably encouraged participants to guess “no change” if they thought there was a change but could not identify what was changing.

We also looked at whether there were any effects of the number of image pixels affected by the change or the spatial location of the changes across the five experimental conditions. Although we tried to keep the size and eccentricity of the changes similar across the conditions, these cannot be perfectly matched in natural scenes. For each condition, we created a mask image in which the changing pixels were set to one and all other pixels in the image were set to zero. For the conditions with moving shadows (shadow rotation and floating shadow), the mask included all pixels that were part of the shadow in one version of the changing image but not the other. To measure the area of the image which had changed in each condition, we computed the number of pixels in each mask. To measure the eccentricity of the changing region, we computed its center of mass from the mask and took the distance in pixels from that point to the center of the image. The average area and eccentricity of changes in the experimental conditions are shown in Figure 3. One-way repeated-measures ANOVAs showed that there was a significant difference in both area (F(4,59) = 13.48, p < 0.001) and eccentricity (F(4,59) = 3.48, p < 0.01) of changes across the experimental conditions. However, these differences do not seem to predict the observed differences in accuracy: the object and surface changes were detected most frequently but were generally smaller in area than changes that were detected less frequently (shadow rotation and floating shadow). The object changes were also, on average, less central than changes in the other conditions.

Figure 3.

Comparison of the size and position of changes in Experiment 1. Error bars represent standard error of the mean.

Twelve participants (60%) mentioned that they noticed or were looking for shadow changes when debriefed at the end of the experiment. Although none of them thought the experiment was primarily about shadow perception, this does suggest that at least some participants may have been paying extra attention to shadows. Figure 4 shows the accuracy across conditions for the participants who mentioned shadow changes in their debriefing versus those who did not. The former group seems to have noticed more of the shadow changes, but both subsets of participants show a significant main effect of condition on accuracy in separate ANOVA analyses (participants who reported looking for shadows: F(4,44) = 3.65, p < 0.05; participants who did not: F(4,28) = 9.88, p < 0.01). This suggests that even participants who were aware of or actively looking for shadow changes showed some effect of the experimental manipulation.

Figure 4.

Change detection performance of participants who mentioned shadow changes when debriefed after Experiment 1 (left) versus participants who did not mention shadow changes (right). Error bars represent standard error of the mean.

Experiment 2

In a second experiment, we looked at whether cast shadow deletions which imply an object deletion are more noticeable than shadow deletions which do not affect objects.

Methods

Participants

24 people participated in the online experiment on Amazon Mechanical Turk. Participants were based in the U.S. and had a good track record on the Mechanical Turk site (at least 100 HITs completed and an acceptance rate of at least 95%). All participants gave informed consent before starting the task. Participants were paid $1.00 upon completing the task and an additional $0.10 for each correct response, resulting in an average total payment of about $14.00.

Stimuli

36 new stimuli images were created for Experiment 2. As in Experiment 1, the stimuli images were photographs of everyday scenes in 24 bit color with a resolution of 1024 pixels wide by 768 pixels high. Three variations of each image were created in Photoshop; these are shown in Figure 5. For the object deletion trials, a single object was removed from the scene. For the shadow deletion trials, the shadow of a visible object was removed from the scene. For the hidden object shadow deletion trials, the shadow of an occluded or out-of-frame object was deleted from the scene. In each scene, the two critical shadows were cast by the same type of object (one visible and one occluded).

Figure 5.

Example image (a) and the change conditions (b-d) used in Experiment 2. In this image, the changes are (b) silverware by the leftmost plate is removed, (c) the shadow of the glass on the left is removed, (d) the shadow of the glass behind the flowers is removed.

Experiment 2 included 162 filler images. Most of the filler images from Experiment 1 were reused, except for a few which had been repurposed as experimental images or dropped because we determined that they were near-duplicates (reflections or differently-cropped versions) of other filler images.

Design

Each participant saw six images from each of the three experimental conditions; the image selection was random and counterbalanced across subjects so each image occurred equally often in each condition across the experiment. These 18 experimental trials were randomly interleaved with 162 filler trials which were the same for every subject and included 63 object, 63 object color changes, and 36 trials with no change. Trial order was randomized for each subject. As in Experiment 1, 80% of the trials in Experiment 2 had a change. Shadow changes were half as common as in Experiment 1, occurring on 8.3% of trials in Experiment 2.

Procedure

The instructions and experiment procedure were exactly the same as in Experiment 1.

Results

One trial was dropped due to a very long response time (> 3 standard deviations above the mean responses time, not including trials that timed out). We compared accuracy, responses times, and the number of timed-out trials across the three experimental conditions, shown in Figure 6. There was no significant difference in response time across the three conditions (oneway repeated-measures ANOVA: F(2,23) = 0.75, p = 0.49) but there was a massive difference in accuracy (F(2,23) = 25.2, p < 0.001). Bonferroni-corrected pairwise t-tests indicated that people were significantly better at detecting object changes than either type of shadow change, but there was no significant difference in accuracy across the two types of shadow change. There was also a significant difference in the percentage of trials that timed out across the three experimental conditions (F(2,23) = 4.35, p < 0.05). Object change trials timed out less frequently than either type of shadow change trial, but only the difference between the object change condition and shadow change (visible object) condition was significant.

Figure 6.

Response time (left), accuracy (center), and percent of trials which timed out across the three types of changes in Experiment 2. Error bars represent standard error of the mean.

As in Experiment 1, most of the errors were failures to detect the changes: people said there was no change, or the trial timed out before they detected the change. Only 1.9% of the errors (19 trials) were location errors, and these appeared to be random guesses.

The average size and eccentricity (distance from the center of the image) of the changes across the three experimental conditions are shown in Figure 7. An one-way repeated-measures ANOVA showed no significant differences in the size (F(2,35) = 0.24, p = 0.78) or eccentricity (F(2,35) = 2.38, p = 0.10) across conditions. There was a tendency for the object deletion changes to be smaller and farther from the image center than the shadow changes, even though they were the most easily detected, so size and eccentricity differences probably do not explain our results.

Figure 7.

Comparison of the size and position of changes in Experiment 2. Error bars represent standard error of the mean.

As in Experiment 1, participants were asked at the end of the experiment if they noticed any patterns to the changes in the scenes. Only one person mentioned shadow changes.

Discussion

In two experiments, we found that people are worse at detecting changes to cast shadows than changes to objects or surfaces in scenes. We expected this would be true for certain types of changes such as deletion of a cast shadow or rotation of a shadow to suggest an object is lit from the wrong direction. In these cases, the change in the shadow doesn't really alter the meaning of the scene. The change introduces a lighting inconsistency, but people are known to be poor at detecting wrong shadows in scenes (Ostrovsky, Cavanagh, & Sinha, 2005). However, people are no better at detecting shadow movement that implies an object is floating impossibly above a surface or shadow deletions that imply an object has been deleted from a scene. These types of changes could be interpreted as object or scene gist changes, but they are significantly less noticeable than changes to the physical objects in the scene.

People may fail to notice changes to shadows because shadows are less salient than other parts of a scene. However, the surface changes in Experiment 1 had similar local contrast to the shadow changes, so they should have been equally non-salient. Nevertheless, they were noticed significantly more often than shadow changes. In addition, it is unclear that low-level feature saliency alone drives change detection in natural scenes: higher-level scene semantics may play a more important role (Stirk & Underwood, 2007).

The results of this study are similar to previous findings by Sareen, Ehinger, and Wolfe (2014), which compared detection rates in a change blindness task when changes occurred to physical objects in the scene or to the same objects' reflections in mirrors. People were significantly worse at detecting changes to reflections, even when the changing object was only visible in the reflection. This is also somewhat surprising given that other studies have shown that objects visible only in a reflection are treated as more “real” that the reflections of objects which are also visible in the scene; for example, when counting the number of a given object in a scene, people will include reflections if the object is not visible in the scene, but they never count reflections otherwise (Chesney & Gelman, 2015).

Both cast shadows and reflections are redundant when the object casting them is clearly visible in the scene. Although there are cases where a cast shadow or reflection provides useful information about an object, these may be fairly rare in everyday life, and it may be more efficient for the visual system to ignore this information and focus on processing the physical objects and surfaces in the scene. This processing should discount shadows and reflections, since these are produced by lighting alone and do not represent the true color or shape of the surface.

Even when cast shadows are informative, using that information may require some prior processing of the objects in the scene. In order to correctly interpret shadow information, the visual system must be able to determine which shadows are cast by which objects, a problem known as the shadow correspondence problem (Dee & Santos, 2011). In this experiment, people may not have noticed the scene-altering shadow changes because they were not able to process the scenes deeply enough to solve this problem. For example, to recognize a shadow change which deletes a hidden object as an “object deletion,” people may need to process both the objects and shadows in the scene and then compare them to realize that, some of the time, there is an extra shadow that doesn't correspond to any visible object. It's also possible that people do process shadows unconsciously, but for whatever reason this information does not become consciously available for change detection, while information about objects in the scene is consciously available.

Another way to think about these results is to note that attention is preferentially directed to objects (Egly, Driver, & Rafal, 1994; Fiebelkorn, Saalmann, & Kastner, 2013) or at least to “proto-objects” (Resnink, 2000; Russell, et al., 2014). Shadows are not objects and may attract less attention. Thus changes to shadows may be detected more slowly and/or less successfully. Displaced shadows, acting as surface markings, may be treated more like objects and thus changes to those markings are found more readily.

Taken with the previous study by Sareen and colleagues, these change blindness results support the idea that cast shadows and reflections are discounted in early visual processing in favor of “real,” physical objects and surfaces. Deeper processing of shadows and reflections may occur somewhat later, after the visual system has already built a rough sketch of the scene. This later processing may reconcile hypotheses about the physical objects in the scene with the information available in shadows and reflections and resolve ambiguities such as a mismatch in the number of objects shown and the number of cast shadows, or inconsistencies in light direction across objects and shadows.