Human memory for real-world solid objects is not predicted by responses to image displays

Abstract

In experimental psychology and neuroscience, computerized image stimuli are typically used as artificial proxies for real-world objects to understand brain and behavior. Here, in a series of five experiments (N=165), we studied human memory for objects presented as tangible solids versus computerized images. We found that recall for solids was superior to images, both immediately after learning, and after a 24-hour delay. A ‘realness advantage’ was also evident relative to three-dimensional (3-D) stereoscopic images, and when solids were viewed monocularly, arguing against explanations based on the presence of binocular depth cues in the stimulus. Critically, memory for solids was modulated by physical distance, with superior recall for objects positioned within versus outside of observers’ reach, whereas recall for images was unaffected by distance. We conclude that solids are processed quantitatively and qualitatively differently in episodic memory than are images, suggesting caution in assuming that artifice can always substitute for reality.

Scientists have traditionally studied the human brain and behavior by presenting observers with pictorial stimuli, either as printed photographs or as digital images displayed on a computer monitor. Pictures are ubiquitous in scientific research because they are easy to find, easy to manipulate for visual properties such as luminance and contrast, and because their timing is straightforward to control. Yet while picture perception and image interaction are becoming increasingly common in the modern world, the human brain has evolved over the course of millennia to allow us to perceive and interact with real-world solids in natural 3-D environments (Cisek & Kalaska, 2010; Gibson, 1979; Heft, 2013).

Although artificial stimuli such as pictures are used widely as proxies for real-world solid objects, recent evidence suggests that there are differences in how observers respond to the different stimulus types. Experiments that have specifically tested the effect of physical realism show that it can amplify behavioral effects (Bushong et al., 2010; Chainay & Humphreys, 2001; Gerhard et al., 2016; Gomez et al., 2017; Holler et al., 2019; Romero et al., 2017; Sensoy et al., 2021), and brain responses (Freud et al., 2017; Marini et al., 2019), thereby increasing the likelihood of uncovering effects that might otherwise be missed by relying on artificial stimuli. Quantitative benefits of realism have been reported across cognitive domains including perception, recognition, attention, memory, valuation, and decision-making, as well as in the developing brain.

Nevertheless, previous studies have left open two fundamental questions: which characteristics of solid objects drive the performance differences, and are solids processed in a qualitatively different way than are pictures? Qualitative differences would be revealed by different or unique patterns of behavior, or activation in different brain areas, for solids versus pictures (Snow & Culham, 2021). For example, solid objects could modulate behavior in ways that that would not otherwise be apparent when studying responses to image displays. Qualitative differences are important because they raise questions about whether results from studies of artificial stimuli generalize to predict and explain behavior and brain responses in real-world contexts.

With these questions in mind, we studied human memory to explore which characteristics of solids drive performance differences in comparison to artificial stimuli, and to determine whether such differences are quantitative, or also qualitative, in nature. Only one study to date has compared memory performance for real objects versus two-dimensional (2-D) photographs of objects using controlled conditions where the photos were matched closely in appearance and timing to their real-world counterparts. Snow et al., (2014) asked college students to remember a set of everyday objects, and afterwards, to report which of the items they remembered. Students who viewed the stimuli as solids performed markedly better on the recall task than did those who viewed the stimuli as colored photographs, or as line drawings (Snow et al., 2014). The results were silent, however, as to which aspect(s) of realism drove the memory advantage.

One of the most striking differences between real objects and pictures is the extent to which they elicit a percept of depth –the experience of the world as having three dimensions. Depth perception arises both from pictorial cues (sometimes referred to as ‘monocular’ depth cues because the information can be derived from a single eye), and from binocular cues which convey depth by combining signals from both eyes. One of the most powerful of these is binocular disparity – the spatial differences in the images seen by the left and right eyes (Julesz, 1971). The disparities between the left- and right-eye images are resolved to yield a unitary percept of 3-D shape in depth in a process known as stereopsis (Blakemore, 1970). Because 2-D images project similar visual information to each eye, when the images are integrated via stereopsis, the stimulus is perceived as being ‘flat’.

Solids and pictures also differ with respect to their position in the world. Solids are inextricably linked with their location in the 3-D environment. The distance of an object from an observer’s body, known as egocentric distance, is especially important because it determines the object’s relevance for action: real objects positioned within reach offer the potential for grasping and interaction, while those outside of reach do not. Unlike real objects, pictures do not convey precise information about an object’s distance from the body. Although the distance from the body to the picture frame or computer monitor is known, the distance of the object within the image is ambiguous (Proffitt, 2006). Furthermore, the distance of a depicted object is not relevant for genuine action because it is not a solid and it is therefore not actable.

Here, across a series of experiments, we examined how visual depth cues, and stimulus distance, influence episodic memory for real objects versus images of objects. Advancing on previous work with printed photographs (Snow et al., 2014), we developed novel apparatus and protocols that permitted a within-subjects design in which half of the to-be-remembered stimuli were presented to observers as real objects, and half as high-resolution computerized images of objects comparable to those used in other modern studies of human memory.

In what follows, we first show that real objects are more memorable than 2-D images of objects (Experiment 1). Next, we show that the memory advantage for real objects (versus 2-D images) is robust after a 24-hour delay (Experiment 2), confirming that the performance advantage reflects processes associated with storage of object information in episodic long-term memory rather than of short-term rehearsal in active working memory. Next, we examine the contribution of binocular disparity to the memory advantage for solids. We show that a memory advantage for real objects manifests both in comparison to 3-D stereoscopic images (Experiment 3), and when binocular cues are removed from the real objects by presenting the stimuli monocularly (Experiment 4). Finally, we show that whereas memory for real objects is modulated by stimulus distance, with superior recall for objects positioned near to, versus far from, the observer; the same is not the case for 2-D images of objects, for which memory is unaffected by stimulus distance (Experiment 5). Together, our results suggest that solids are processed quantitatively and qualitatively differently in human memory than pictures of objects.

Method

Participants

Participants were healthy college students who volunteered to participate in the research. All experiments were approved by the Institutional Review Board at the University of Nevada Reno, and the protocols met Declaration of Helsinki standards. Participants were recruited from undergraduate courses in the Psychology Department. All participants provided informed consent prior to the experiment and were compensated with either course credit or $10.

Sample size estimates for our experiments were based on results from Snow et al., (2014), who observed the ‘real object memory advantage’ (using real objects and printed photos) to be of medium to large effect size (ES). We used G*Power (Faul, 2006) to estimate a priori sample sizes required to achieve power = 0.8, at alpha = 0.05, for a medium ES (given that we used computerized images rather than printed photos). For Experiments 1-4 to detect a difference between two dependent means (matched-pairs) with two-tails, the estimated minimum sample size was N = 32 subjects; for Experiment 5 using Repeated Measures ANOVA, estimated minimum sample size was N = 34 subjects.

Participants were recruited and tested for each experiment over the course of a semester. The actual n in each study reflects the number of students who volunteered to participate in the study after it was advertised at the beginning of the semester. The enrollment ns met or exceeded the targets of our power analysis. The only exception was Experiment 2 (which was perhaps less popular among students due to the need to return to the lab after a 24-hr delay), which terminated with an end-of-semester sample of N=19, but which nevertheless yielded an ES of d = 0.61. One participant was excluded from Experiment 2 due to inability to remember any items from the study phase. Data from one participant in Experiment 3, and one from one Experiment 4, was excluded due to failure to follow task instructions. This yielded a final sample of 165 participants (Exp.1 N=40, mean age 20.4 years, 31 female, 9 male; Exp. 2 N= 19, mean age = 24.3 years, 16 female, 3 male; Exp. 3 N= 41, mean age = 20.2 years, 25 female, 16 male; Exp. 4 N= 32, mean age= 21.5 years, 24 female, 8 male; Exp. 5 N=33, mean age = 20.9 years, 23 female, 10 male); age and gender information was self-reported via pen-and-paper questionnaire with free-response boxes. None of the participants reported having a history of neurologic or psychiatric disorders, and the visual acuity of all participants was normal, or corrected to normal, as assessed by self-report.

Stimuli and Apparatus

The stimuli were 112 everyday real-world objects, and high-resolution computerized color images of the same items. Participants viewed half of the exemplars as real objects and the remainder as images.

In Experiments 1-4, the real objects were presented on a manually operated turntable (Fig. 1A). The turntable (2 m diameter) was divided into 20 sectors (62 cm D x 26 cm W) using vertical dividers (24 cm H). A large 152.5 cm x 127.5 cm vertical partition was set up between the subject and the turntable, through which was cut a rectangular viewing aperture (59 cm H x 23 cm W). The viewing aperture prevented participants from seeing stimuli on adjacent sectors of the turntable on each trial, but was large enough so as not to prevent manual access to the stimuli. Viewing distance from the participant to the stimulus was 50 cm.

Fig. 1.

Apparatus and stimuli used in the memory experiments. (A) Aerial schematic showing turntable used to present real objects and computerized images in Experiments 1-4. In the learning phase, participants were asked to attend to the stimuli and to try to remember as many items as they could. Memory for the items was probed in a subsequent free-recall task. The turntable was divided into 20 equal sectors. On real object trials, one object was visible to the observer on a sector of the turntable; on image trials an LCD monitor was used to depict an image of an object as it appeared when it was on the turntable. The monitor was mounted on a sliding platform that could be positioned within the viewing aperture on image trials (as illustrated) or retracted to the side behind a vertical divider on real object trials. (B) Half of the stimuli were presented to observers as real objects (example ‘toothbrush’, left panel), and half were presented as images on a computer screen (right panel). The object images were matched closely to their real-world counterparts for monocular depth cues, including lighting and shading, perspective, texture and background, as well as retinal size and distance from the observer. Head position was stabilized using a chin rest. Each object was presented once during the experiment, either as a real object or an image (real and image versions of ‘toothbrush’ are shown above for illustrative purposes only). Item display format was counterbalanced across participants. Observers did not know whether the upcoming item would be a real object or an image, and display format was not relevant for the subsequent recall task. (C) Viewing time for each stimulus was matched using computer-controlled liquid-crystal glasses. Each of the 112 items was presented for 3 sec (glasses ‘open’), followed by an 8 sec intertrial interval (ITI) (glasses ‘closed’). (D) In Experiment 5, the stimuli were presented on a rolling platform; half of the stimuli were presented at a ‘near’ distance (50 cm; real object shown, left panel) and half were presented at a ‘far’ distance (80 cm, real object shown, right panel).

To create the 2-D computerized image displays, we photographed each of the 112 real objects on a sector of the turntable. The stimuli were photographed using a Canon Rebel T2i DSLR camera using constant F-stop (an aperture setting that controls how much light enters the camera lens) and shutter speed to control exposure. The camera was mounted on a tripod positioned at the approximate height and distance of the participant when looking at the objects from straight ahead. Image size was adjusted to match the physical size of the real objects using Adobe Photoshop. The resulting images matched closely the real objects for retinal size, viewing angle, background, lighting and shadows (Fig. 1B). In Experiments 1-4 the 2-D images were presented on a 27” Acer G276HL LCD computer monitor. The monitor was mounted on a platform, with wheels mounted on the underside. The platform could slide horizontally between the vertical divider and the viewing aperture. On image trials the monitor was positioned within the viewing aperture (masking the turntable). On real object trials the monitor was retracted behind the divider, revealing the turntable. To construct the 3-D images for Experiment 3, two photographs were taken of each real object on the turntable. The photographs were captured using a forward-facing camera (50 cm from the screen) displaced 60 mm to the left, and 60 mm to the right of midline (120 mm total horizontal displacement). The images were viewed binocularly through active shutter glasses (3D Vision 2, NVIDIA) and displayed on an LCD monitor (120 Hz; Model VG278HE, ASUS, Beitou District, Taipei, Taiwan) with a screen resolution of 1,920 × 1,080 pixels. Participants wore the 3-D NVIDIA glasses during both the real object and image conditions. A second monitor positioned behind the vertical partition (perpendicular to the sliding monitor and not visible to the participants) was used to display instructions to the experimenter about the object identity and display format on the upcoming trial. The 2-D images for Experiment 5 were created by photographing each real object mounted on the platform, from a distance and viewpoint that matched that of the participant with the head mounted in the chin rest.

Stimulus presentation and timing was controlled using vision-occluding PLATO (Translucent Technologies Inc.) glasses that switch between transparent (open) and opaque (closed) states with millisecond accuracy (Fig. 1C). In Experiment 3 the computer-controlled liquid crystal glasses were worn in front of the active 3-D shutter glasses throughout the study phase. In Experiment 4 participants wore an eye patch over the non-dominant eye.

A Dell computer T1700 Intel i7 CPU running Microsoft Windows 7 and MATLAB R2013a (Mathworks Inc., Sherborn, MA, USA) was used to present the image stimuli, control the glasses, and to provide instructions to participants.

The stimuli in Experiment 5 were presented using a custom-designed moving platform (Fig. 1D). The platform consisted of a horizontal base (27 cm x 50.5 cm x 25 cm) backed vertically by a 22” Dell LCD computer monitor. Wheels were mounted to the underside of the base and allowed the platform to be moved rapidly between within (near) and outside of reach (far) stimulus positions between trials. Except for the rolling platform, all other stimuli and equipment for Experiment 5 were the same as the previous experiments.

Procedure

Learning Phase:

Using a within-subjects design, participants viewed half of the stimuli as real objects and half as computerized images. The identity of items in each display format was counterbalanced across participants: if a toothbrush was displayed as a real object for a given participant, it was not also displayed as a 2-D picture (and a different participant received the reverse format assignment). For Experiment 5, half of the stimuli in each format were presented at a near distance (50 cm, within reach) and the remainder were presented at a far distance (80 cm, outside of reach), and item identity in each distance condition was counterbalanced across participants. For all experiments, the order of trials in each Format (and Distance, Experiment 5) condition was randomized independently for each participant.

All trials in the learning phase had identical timing parameters, regardless of stimulus format (Fig. 1C). At the start of each trial the vision-occluding glasses opened for 3 sec, revealing the stimulus (a real object or an image). The glasses then closed for an inter-trial interval (ITI) of 8 sec. During the ITI the experimenter prepared the stimulus for the upcoming trial. In Experiments 1-4 the experimenter prepared the object on the turntable/monitor for the upcoming trial; in Experiment 5 the experimenter prepared the object on the platform/monitor and moved the platform to the correct distance. White noise (70 dB) was played within the testing room throughout the Learning Phase.

Participants were instructed at the start of the experiment to try to remember as many of the stimuli as possible, and that their recollection of the items would be probed after the Learning Phase.

Test Phase:

An object recall test was administered immediately after the learning phase, except for Experiment 2 in which participants returned to the laboratory after a 24-hour delay to complete the recall task. Participants were asked to recall as many items as possible from the learning phase. Participants entered the names of all objects that they could remember using a computer keyboard and monitor positioned within the viewing aperture (Experiments 1-4), or on the rolling platform (Experiment 5). No time limit was imposed. Participants completed one or more other behavioral tasks (which differed across experiments) after the recall task, for other ongoing studies, and which are not reported further here. The experiment took ~1 hour to complete per participant.

Data Analysis

For Experiments 1-4 performance was compared across Formats (real versus image) using planned two-tailed paired-samples t-tests. For Experiment 5 performance was compared across Formats (real versus image) and Distances (‘near’ versus ‘far’) using repeated-measures Analysis of Variance (ANOVA) with two follow-up planned orthogonal paired-samples t-tests. A Bonferroni correction (α = 0.025) for multiple comparisons was not applied to the follow-up given that our power analysis was based on the ANOVA (but note that even if the Bonferroni correction was applied it would not change the conclusions of the Experiment). The Greenhouse-Geisser correction for violations of sphericity was applied where appropriate for within-subjects analyses.

Transparency and Openness

We have reported how we determined our sample size (see Participants), all data exclusions, all manipulations, and all measures in the study. The data for all five experiments have been made publicly available via the Open Science Framework page (https://osf.io/qgh75/). The design and analysis plans for the experiments were not preregistered. The data were analyzed using SPSS statistical software; no additional computer code was used for the analyses. Further information about the stimuli used in these experiments can be obtained from the corresponding author (snow@unr.edu).

Results

Experiment 1: Real objects are more memorable than 2-D images of objects

Experiment 1 advanced on earlier work done with 2-D printed photographs of objects, which used a between-groups design (Snow et al., 2014). Here, we developed novel apparatus and protocols that permitted a within-subjects design in which half of the to-be-remembered stimuli were presented to observers as real objects, and half as computerized images of objects.

Memory performance for the real objects and 2-D images in the free recall task is shown in Fig. 2A. Observers recalled (% correct) significantly more items that were presented as real objects (M = 31.47%, SD = 10.61) compared to 2-D images (M = 26.78, SD = 7.76; t(39) = 3.81, p < 0.001, d = 0.60).

Fig. 2.

Bar graph showing object recall performance in Experiments 1-5. (A) Memory for real objects was superior to that of 2-D images immediately after study. (B) Memory for real objects was also superior to that of 2-D images after a 24-hr delay. (C) Real objects were more memorable than 3-D images of objects presented with binocular disparity. (D) Real objects were more memorable than 2-D images of objects when all stimuli were viewed monocularly. (E) Memory for real objects was modulated by the physical distance of the stimulus, with superior performance for objects positioned in the ‘near’ versus ‘far’ position, whereas memory performance for 2-D images was not influenced by distance. Bars illustrate mean percent (%) correct in the free-recall task; open circles represent individual data; error bars show 95% CIs of the mean; asterisks represent p-value (* p<0.05; ** p<0.005).

The results of Experiment 1 provide an important replication of the findings reported by Snow et al., (2014), and critically, they extend these results by demonstrating that a ‘real object advantage’ in memory is not only apparent in comparison to 2-D printed photographs, but that it is also apparent in comparison to high-resolution 2-D computerized images of objects comparable to those used in other modern studies of human memory. Our within-subjects design further confirms that the memory advantage for real objects reported by Snow et al., (2014) is not explained by a priori differences in memory ability across observers.

Experiment 2: Real objects are more memorable than 2-D images of objects after a 24-hr delay

Next, we tested whether the memory advantage for real objects versus 2-D images of objects extended across a temporal delay. We tested a new group of participants using identical procedures to those described in Experiment 1, except that participants returned to the lab 24-hours after the study phase to complete the free-recall task.

Free-recall performance for the real objects and 2-D images after a 24-hour delay is shown in Fig. 2B. As expected, recall performance was (qualitatively) poorer than in Experiment 1, indicating that it was more difficult for observers to recall the stimuli after the lengthy delay. Nevertheless, like Experiment 1, free recall for the real objects (M = 21.62, SD = 9.20) was significantly better than for the 2-D images (M = 16.17, SD = 7.88; t(18) = 2.65, p = 0.016, d = 0.61). These results confirm that the performance advantage for real objects (versus 2-D images) reflects superior episodic long-term memory rather than rehearsal of items in active working memory.

Experiment 3: Real objects are more memorable than are images of objects presented with binocular disparity

Experiment 3 tested the role of depth information from binocular disparity on memory performance for real objects versus computerized images. Although some studies have compared memory for 2-D versus virtual 3-D computer-generated displays, the findings have been mixed (Cockburn & McKenzie, 2002; Valsecchi & Gegenfurtner, 2012).

If the memory advantage for real objects (versus 2-D images) observed in Experiments 1 and 2 is driven by the presence of binocular disparity information in the real objects (but not in the 2-D images), then free recall performance should be comparable across formats when the images convey disparity information. Experiment 3 had an identical design to Experiments 1 and 2, except that the stimuli in the image condition were 3-D stereoscopic images presented using active shutter glasses (see Methods). The computer-controlled liquid crystal glasses were worn in front of the active 3-D shutter glasses throughout the study phase.

Free-recall for the real objects and 3-D stereoscopic images of objects is shown in Fig. 2C. As in Experiments 1 and 2, free recall for the real objects (M = 22.69, SD = 9.49) was significantly better than for the 3-D stereoscopic images (M = 19.64, SD = 7.83; t(40) = 2.31, p = 0.026, d = 0.36). These results seem especially surprising because stereoscopic images are more unusual stimuli than are real objects, making them more salient for attention and memory (Chun & Turk-Browne, 2007); but this is the opposite pattern of results than what we observed.

Experiment 4: Real objects are more memorable than are 2-D images of objects, even when they are viewed monocularly

Next, we contrasted memory for real objects versus 2-D images when binocular disparity information was eliminated from the real objects by presenting all the stimuli monocularly (see Methods). Participants viewed the stimuli using the dominant eye, with the non-dominant eye closed and covered with an eye-patch.

Free recall performance for the real objects and 2-D images viewed monocularly is shown in Fig. 2D. Once again, recall was significantly higher for the real objects (M = 25.67, SD = 8.71) compared to the 2-D images (M = 19.75, SD = 9.38; t(31) = 3.39, p = 0.002, d = 0.60). Together, the results of Experiments 3 and 4 provide powerful evidence that the superior memory performance for real objects versus images is not attributable to differences between the stimuli in depth information from binocular disparity, since the memory advantage for real objects is apparent both when disparity is added to the images (Experiment 3), and when disparity is removed from the real objects (Experiment 4).

Experiment 5: Memory for real objects, but not 2-D images, is modulated by stimulus distance

In Experiment 5 we tested whether memory for real objects and 2-D images is differentially modulated by stimulus position. Half of the items in each display format were presented 50 cm from the observer (‘near’ condition), and the remainder at 80 cm (‘far’ condition). The stimuli were within reach of the observer in the ‘near’ condition, but not in the ‘far’ condition. The stimuli were presented using a custom-designed rolling platform (see Figure 1D, and Methods). Trials in each format and distance condition were randomly interleaved during the study phase.

Free recall for real objects and 2-D images presented at the ‘near’ and ‘far’ distance is shown in Fig. 2E. A 2 x 2 repeated-measures ANOVA with the factors of Display Format (real, 2-D image) and Distance (near, far) revealed a significant main effect of Display Format (f(32) = 24.745, p < 0.001, n² = 0.436), in which free recall for the real objects (M = 26.68, SD = 10.46) was significantly better overall than for the 2-D images (M = 19.81, SD = 10.08). The main effect was qualified by a significant Display Format x Egocentric Distance interaction (f(32) = 7.071, p = 0.012, n² = 0.181). For real-objects, free recall was superior for stimuli at the near (M = 29.76, SD = 10.81) versus far (M = 23.59, SD = 10.10; t(32) = 3.051, p = 0.005) location. Conversely, free recall for the 2D-images remained unchanged across near (M = 19.16, SD = 10.37) versus far (M = 20.45, SD = 9.79) locations (t(32) = −0.72, p = 0.479).

The poorer recall performance for real objects in the far (versus near) position cannot be attributed to reduced retinal extent because this should have led to equally poor performance for the 2-D images whose retinal extent declined by the same amount, but this is not what we observed. The results also cannot be explained by reduced stereoscopic information for the real objects (but not the 2-D images) at the far position. First, our stimuli were presented at a maximum of 0.8 m in an illuminated testing room in which the ground plane and walls were visible. Under these conditions, depth estimates for real objects can be derived reliably from disparity at distances up to 3 m (Durgin et al., 1995; McKee & Taylor, 2010), and from perspective-based cues from scene features at distances up to 40 m (Allison et al., 2009; Gillam et al., 2011; Palmisano et al., 2010). Second, the memory advantage for real objects is evident when computer images of objects convey stereoscopic cues (Experiment 3), and when stereoscopic cues are eliminated from real objects (Experiment 4).

Discussion

Our experiments revealed two remarkable aspects of human object memory. First, objects received higher priority in episodic memory if they were seen as real-world solids than as images, even when the depictions appeared strikingly similar to real objects in 3-D depth. Second, the way objects were processed in memory depended on both stimulus format and physical distance. When the stimulus was a solid object, it was processed according to spatial position, with superior recall for objects positioned near to, versus far from, the observer. When the stimulus was a 2-D image, it was processed similarly in memory regardless of spatial position. Together, these results suggest for the first time that stimulus realism has both quantitative and qualitative effects on how objects are processed in episodic memory.

Our findings address which of the differences between real objects and pictures drive the quantitative effects on performance. Numerous studies have revealed differences in behavioral (Bushong et al., 2010; Chainay & Humphreys, 2001; Gerhard et al., 2016, 2021; Gomez et al., 2017; Holler et al., 2019; Holler et al., 2020; Holmes & Heath, 2013; Romero et al., 2017; Sensoy et al., 2021) and brain (Fairchild et al., 2021; Freud et al., 2017; Marini et al., 2019; Snow et al., 2011) responses to real versus artificial stimuli, yet there is intense interest in determining whether these effects are explained by differences between the stimuli in visual depth information or differences related to inherent tangibility (Snow & Culham, 2021). We consistently found that real objects were more memorable than images of objects, both when the images were presented with binocular disparity, and when the real objects were viewed monocularly –indicating that that the percept of three-dimensionality provided by stereopsis is not sufficient to boost memory for images (relative to real objects), nor to eliminate the memory advantage for real objects (relative to planar images of objects). These results, which dovetail with findings that perceptual complexity does not explain memory performance for pictorial stimuli (Brady & Störmer, 2022; Brady et al., 2016; Konkle et al., 2010; Snow et al., 2014), demonstrate that the memory advantage for real objects is not simply explained by the amount of depth or shape information conveyed by the stimulus.

Instead, long-term memory processes for objects appear to depend on whether the available visual cues signal in concert with one-another that the object has depth, thereby revealing if it is a tangible solid. For real objects, depth information from all available monocular and binocular sources signal in agreement that the object is a solid; for pictures, conflicts between different depth inputs reveal that the stimulus is inherently flat. Although stereoscopic displays elicit a compelling depth percept, lens accommodation remains fixed at the depth of the screen, which detracts from performance (Hoffman et al., 2008). Our results provide a striking demonstration of how such subtle cue conflicts can also influence memory. Subtle environmental features in our experiments also signaled that the stimuli on image trials were not real (such as the outline of the monitor), but like the depth cues, they were orthogonal to the observer’s task. Either way, our results demonstrate that human observers are exquisitely sensitive to visual cues that signal whether an object is inherently solid or flat (Banks et al., 2016), and that this influences how object information is processed in memory.

Indeed, we found that the inherent solidity or flatness of the stimulus had qualitative effects on episodic memory, with solid objects (but not object images) processed according to their distance from the observer. Classical theoretical frameworks divide visual processing across distinct cortical pathways; a ventral pathway dedicated to perception and object recognition, and a dorsal pathway dedicated to spatial cognition and visually guided actions (Goodale & Milner, 1992; Mishkin et al., 1983). Real objects and images trigger different responses in human dorsal brain areas involved in object coding and automatic motor preparation for action, even when the observer has no explicit plan to interact manually with the stimulus (Fairchild et al., 2021; Holler et al., 2019; Marini et al., 2019; Snow et al., 2011). Recruitment of dorsal networks could have facilitated memory for real objects in our experiments, especially for those within reach, by enhancing depth of processing at encoding (Craik & Tulving, 1975). Yet there is also mounting evidence from neurophysiology that information about objects, together with their spatial position in the world, is deeply integrated in the ventral processing pathway that supports perception and memory. In the domain of memory, the idea of a distinct anatomical division between object and spatial-action processing has given way to a more complex picture where object shape and spatial information are closely integrated –not only in anterior temporal memory structures such as the hippocampus, but also along much of the ventral stream (Connor & Knierim, 2017; Verhoef et al., 2012). Importantly, while retinotopic organization declines from posterior to anterior ventral cortex (Brewer et al., 2005), spatial information may be preserved along this pathway by recoding local retinotopic information into relative dimensions that can represent an object’s position in the real world (Connor & Knierim, 2017; Hong et al., 2016).

An increasing number of studies have demonstrated that, compared with artificial stimuli, realism leads to quantitative effects on responses, such as more accurate object recognition, stronger attentional or gaze capture, and increased valuation (see Snow & Culham, 2021 for review). These patterns indicate that realistic stimuli can amplify or strengthen responses that might otherwise be less apparent in studies that use images. Given that artificial stimuli are easier to generate and present than real objects are, researchers could tackle this situation by continuing to rely on images but compensating for attenuated effects, for example, by designing experiments to maximize power. However, reality may also lead to qualitatively different patterns of responses – behaviors or brain responses that are uniquely different from what is observed using images. In this case, responses to real-world stimuli cannot be predicted by studying responses to artificial stimuli because the response patterns are not just different in magnitude, they are different in quality. In line with this possibility, our results from Experiment five demonstrate that stimulus distance modulated memory for real objects, whereas this was not the case for 2-D images of objects. Indeed, we found (Figure 2E) that memory performance was marginally (but not significantly) better for images in the ‘far’ location, whereas for real objects there was a significant decline in memory across the near vs far location. These results are especially compelling because our apparatus and protocol allowed for real object trials to be randomly interleaved with image trials during the learning phase, the stimulus exposure times were identical across displays formats, and the findings were obtained in a within-subjects design. The results from Experiment five support the conclusion that, in some cases, solid objects can modulate behavior in ways that may not otherwise be apparent when studying responses to image displays.

We are not aware of any other studies in which memory for images of objects has been contrasted with the monitor set at different egocentric positions. Although isolated images of objects do not convey absolute distance information, adding richer visual depth cues to the image may increase the likelihood that the stimulus is processed according to distance (Gandolfo, et al., 2023). For example, when familiar objects are presented in the context of a rich natural background in which a variety of other objects and environmental features are also visible, they appear to be processed (Josephs & Konkle, 2019) and represented in the brain (Josephs & Konkle, 2020) according to implied viewing distance. Of course, in the real world, as an object moves further from the observer, it subtends a correspondingly smaller visual angle on the retinae; yet our perception of the size of the object remains relatively constant. Interestingly, the size of familiar 2-D pictures of objects is inferred to be consistent with the physical size of those objects (Konkle & Oliva, 2011), and studies have shown that real-world size information is inferred automatically from 2-D images of objects during perception (Konkle and Oliva, 2012) and attention (Collegio et al., 2019). The size of basic 2-D shapes and line-drawings of objects has also been reported to influence response times to make old-new judgments about those stimuli in episodic memory (Jolicoeur, 1987). However, others have reported that the size of familiar objects on a monitor did not influence object naming times in a visual priming task (Biederman & Cooper, 1992). Studies that have directly compared responses to real versus artificial stimuli have found that relatively subtle changes in the physical size of familiar graspable real-world solid objects (e.g., an apple that is shown 50% larger or 50% smaller than its typical real-world size) has a striking influence on whether patients with visual agnosia can identify (name) the object, whereas analogous changes in the size of 2-D or 3-D images of the same objects, or of basic shapes that have no real-world size associations, has no influence on object identification in these patients (Holler et al., 2019). Similarly, the real-world size of real object toys, but not the size of matched 2-D images of those toys, influences looking preferences in healthy human infants (Sensoy et al., 2021). Future studies will shed light on the conditions under which real-world size and distance information might be automatically derived from objects depicted as images.

Limitations

These considerations raise the question of whether the solids in our experiments were processed according to physical distance per se, or relevance for action. Animal neurophysiology highlights physical distance as a key modulator, and recent studies in humans highlight the importance of physical size (Holler et al., 2020; Holler et al., 2019; Sensoy et al., 2020). But physical distance and size also influence actionability: they reveal which objects are graspable and which are not (Gibson, 1979). Relevance for genuine motor action influences behavior and brain responses. For example, Gomez & Snow (2017) found that real objects captured attention more so than did 2-D or 3-D images of objects. However, the stimulus-related effects on attentional capture were absent both when the real objects and 2-D images were positioned outside of the observer’s reach (dovetailing with the effect of distance in our memory task), and also when the stimuli were displayed within reach but behind a transparent barrier that prevented in-the-moment manual interaction with the objects. Actionability has similar effects on cortical responses to real objects, as measured by EEG. When observers look at (but do not touch) everyday tools, real-world exemplars elicit stronger desynchronization of the mu (μ) rhythm –a neural signature of automatic motor preparation to act –than do matched 2-D photos of the same tools (Marini et al., 2019). However, with a transparent barrier in place, the difference in μ desynchronization across formats is sharply attenuated (Fairchild et al., 2021).

Although tests of the ‘distance’ versus ‘actionability’ explanations present an avenue for future research into the effects of realism, different properties of real objects may trigger different processing mechanisms, and those processes may unfold over different time scales. After all, real objects are highly multidimensional; they have a definite weight, surface texture, taste, smell, and compliance, unlike images which do not have these properties. In the EEG study by Fairchild et al., (2021) a barrier attenuated μ desynchronization differences between formats early after stimulus onset, but not later in the trial after stimulus offset. This led the authors to speculate that immediate effects of realness are driven by in-the-moment actionability, while more slowly evolving effects are driven by inherent properties of real objects such as distance or physical size (which remained stable in their study, irrespective of the barrier). These findings, and others (Droll & Eckstein, 2009), caution that the nature and timing of the observer’s task (i.e., where responses are collected immediately, versus later, after the stimuli are presented) may determine which characteristics of real objects drive performance. Indeed, one prediction is that real objects are relatively susceptible (and images more resilient) to the effects of task and timing because of their multidimensional qualities.

Conclusions

In summary, our results provide compelling evidence that real objects are processed both quantitatively and qualitatively differently than images of objects in episodic memory. Our findings converge with recent evidence from human and animal studies that point to the idea that cognitive and neural systems support naturalistic vision, where information about (solid) objects is processed conjointly with spatial position –possibly because of implications for action. Images, which are disembodied abstractions, may either not adequately engage, or not fully leverage, the spatio-temporal nature of object processing in the brain. Our results raise intriguing questions about the limits and potential for virtual / augmented reality to mimic realness, and about whether awareness or belief that a stimulus is real is a necessary condition for performance differences (Banks et al., 2016). For example, the perception of reachable space surrounding the body can be extended, or ‘remapped’, following motor training with a real tool, but the same does not happen after motor training with a virtual reality tool (Ferroni, et al., 2022). The effect of realness on memory advances various translational predictions, including that real objects may be preferable to pictures for facilitating learning and memory in the classroom (Strouse & Ganea, 2021), for maximizing sensitivity in neuropsychological evaluations (Beaucage et al., 2020; Hampstead et al., 2010), and perhaps for facilitating performance in individuals for whom memory function is disrupted due to brain injury (Sirigu et al., 1991), developmental disorder (Humphreys & Riddoch, 1999), natural aging (Tran et al., 2021), or neurodegenerative conditions (Clemenson & Stark, 2015). Our protocols demonstrate how real-world stimuli can be used in experimental contexts to maximize ecological validity without sacrificing experimental control (Romero & Snow, 2019). Our findings underscore the idea that to fully understand the human brain and behavior, the traditional ‘build-up’ approach of using minimalist stimuli and gradually adding complexity, should be combined with a ‘tear-down’ approach where the starting point is feature-rich, multidimensional reality, and components are gradually removed to see which matter (Snow & Culham, 2021). In the context of human memory, at least, our results indicate that performance for real objects is superior to that of images of objects, and that under some circumstances responses to real-world solids cannot be predicted by responses to artificial image displays.

Constraints on Generality

A memory advantage for real objects versus 2-D images of objects has been observed in English-speaking male and female college-aged students, both in a previously published paper using a between-subjects design (Snow et al., 2014), as well as here in repeated experiments using a within-subjects design. Therefore, we would expect these results to generalize to English-speaking adult populations. In both studies, a wide range of everyday familiar objects served as stimuli. A direct replication would test memory for everyday real objects versus closely-matched pictures or computer images of those objects, when the objects are presented as isolated stimuli on a display surface. We have no reason to believe that the results depend on other characteristics of the participants, materials, or context.

Public Significance Statement.

Scientists study vision, thinking, and memory by running controlled laboratory experiments; results from the laboratory are then generalized to understand brain function in real-world contexts. In laboratory studies of human memory, scientists measure responses to visual stimuli (such as common objects) which are typically presented as images on a computer screen, rather than as real-world solid objects. Here, we show that memory for real-world objects cannot be predicted based on responses to artificial image displays. This work suggests that stimulus realism influences how information is processed in the brain, questioning how well results from studies of artificial stimuli generalize to explain behavior in the natural world.