Significance
Using electroencephalography and virtual reality, our research provides a unique perspective on the centuries-old open-ended debate in cognitive neuroscience and philosophy on the relationship among cognition, movement, and environment. Our results indicate that cortical potentials vary as a function of bodily affordances reflected by the physical environment. First, the results imply that cognition is inherently related to the potential movement of the body; thus, we posit that action is interrelated with perception, actively influencing the perceivable environment. Second, these results indicate that moving in space is to continuously construct a prediction of a world of affordances, suggesting that architects take up the continuity of spaces, given that the unfolding of bodily movement alters perception and experience.
Keywords: sensorimotor, predictive processing, mobile brain/body imaging, architectural cognition, mobile EEG
Abstract
Anticipating meaningful actions in the environment is an essential function of the brain. Such predictive mechanisms originate from the motor system and allow for inferring actions from environmental affordances, and the potential to act within a specific environment. Using architecture, we provide a unique perspective on the ongoing debate in cognitive neuroscience and philosophy on whether cognition depends on movement or is decoupled from our physical structure. To investigate cognitive processes associated with architectural affordances, we used a mobile brain/body imaging approach recording brain activity synchronized to head-mounted displays. Participants perceived and acted on virtual transitions ranging from nonpassable to easily passable. We found that early sensory brain activity, on revealing the environment and before actual movement, differed as a function of affordances. In addition, movement through transitions was preceded by a motor-related negative component that also depended on affordances. Our results suggest that potential actions afforded by an environment influence perception.
The affordance of a given spatial environment—defined as the perception of possibilities for, or restraints on, an action that the environment offers—is essential for an agent to produce meaningful behavior. Thus, the affordance of the spatial environment become a central concept for humans interacting with their world. The term “affordance” was introduced by Gibson (1) and subsequently refined by various authors, including Clark (2), who defined it as “the possibilities for use, intervention, and action which the physical world offers a given agent and are determined by the ‘fit’ between the agent’s physical structure, capacities, and skills and the action-related properties of the environment itself.” In light of emerging theories of embodied cognition, the perception of the environment may be dependent on proprioceptive mechanisms. According to predictive processing, a neuroscientific-based theory of embodied cognition (3–5), motor systems, similar to perceptual processes, aim at canceling out continuously incoming bottom-up sensory signals with top-down predictions. In this perspective, movement emerges as a result of an active inference that attempts to either minimize motor trajectory prediction errors by acting, and thus perceiving the unfolding of the predicted movement, or by changing perception itself (6–8).
From the standpoint of active inference, motor systems suppress errors through a dynamic interchange of prediction and action. In other words, there are two ways to minimize prediction errors: to adjust predictions to fit the current sensory input and to adapt the unfolding of movement to make predictions come true. This is a unifying perspective on perception and action suggesting that action is both perceived by and caused by perception (9). Thus, action, perception, and cognition coordinate to move the body in ways that conform to a transitional set of expectations (10).
The claim that we seek to investigate in the present study is that perception is rooted in action, creating an action–perception loop informed by dynamically (top-down/bottom-up) generated prediction errors. Ultimately, the argument is that perception is not the sole result of sensing the physical world but unfolds as an ongoing interaction between sensory processes and bodily actions. Such a claim has philosophical and neuroscientific significance, because the neural dynamics underlying perception would be intimately dependent on the affordances of a given environment.
To investigate this claim further, we used electroencephalography (EEG) recordings to address the neural dynamics of action–perception interactions through affordance manipulations in architectural experiences. More specifically, we investigated the affordances of transitions, which are ideal candidates due to their dynamic nature concerning the duration of altering one condition to another (11). Here we confine transitions to the passage between spaces, which according to the enactivists’ proposed action–perception loop will be an experience dependent on the affordances offered by the passage itself. Because of the dynamic nature of architecture, an essential part of transitions and experiencing architecture is the ability to act (12). Traditionally, investigations of architectural experiences are phenomenological—the description of phenomena in how experience gives access to a world of space and time (13–16). Such descriptions specifically find the movement of the individual to be an expression of a holistic experience of architecture (13, 14), linking the nature of movement to architectural experiences (17).
Transitions in architecture depend on voluntary movement, and thus a prerequisite for any transit is a goal, which calls for action planning. Coarsely, three parameters compose a transition: a motivated goal, a change in the physical environment, and the unfolding of action. All three parameters are interdependent, because reaching a goal depends on the affordance offered by an environment and also propels the body in space, contributing to experience. Thus, architectural transitions include the attenuation of an agent’s experience through movements and the way in which such movements animate the body through environmental changes.
Data from neuroscientific experiments addressing this issue might contribute to discussions centered on philosophical questions on how we relate to the world. For a long time, enactivists implicated the reciprocal dependency of the living organism as a self-organized living system and the embedded body in a world for cognition (18–20). Enactivism is rooted in phenomenology (12, 21), similar to prominent architectural theorists who put body, action, and cognition central to experience. Active inference is closely related to enactivism, in the sense that we act to perceive, and vice versa. Such a thesis rests on a hierarchical and dynamic model of the world, which temporally dissociates lower sensorimotor inferences from higher motivated goals as fast and slow, respectively (22). Fast lower sensorimotor inferences depict processes of affordances, which thus must be present in early stages of perception. Hierarchical affordance competition (HAC) (23) takes the temporal aspect of affordances much further by suggesting that cortical activity relates to the immediate decision of action selection, which occurs fluently during movement. Such an account of temporally extended affordances is in accordance with active inferences.
To investigate the impact of environmental affordances on early sensory processing in actively transiting humans, we used a mobile brain/body imaging (MoBI) approach (24–26), recording brain activity with EEG synchronized to movement recordings and head-mounted virtual reality (VR). This approach allows for the investigation of brain dynamics of participants perceiving an environment and the transitions contained therein, as well as brain dynamics during the transitions themselves. Previous studies investigating event-related potential (ERP) activity in stationary participants demonstrated slow cortical potentials to indicate anticipative motor behavior (overview in ref. 27). Known motor-related cortical potential (MRCP) components include the readiness potential (28), contingent negative variation, and stimulus-preceding negativity (29), which can be seen as indicators of predictive behavior (30). MRCPs are negative-going waveforms preceding an actual or imagined motor execution. However, these negative components are associated with multiple processes, including sensory, cognitive, and motor systems.
Bozzacchi et al. (31) attempted to measure the affordances of a physical object by evaluating whether the anticipated consequence of action itself influences the brain activity preceding a self-paced action. They compared MRCPs of situations in which it was possible to reach out and grasp a cup versus situations in which it was made impossible to grasp the cup by tying the hands of the participant. A motor execution was forced at all times. In situations where it was impossible to grasp the cup, the authors reported an absence of early activity over the parietal cortex and found instead increased activity over the prefrontal cortex. The results were interpreted as reflecting an awareness of the inability to execute a goal-oriented action.
Closely related to MRCPs is the postimperative negative variation (PINV), a negative-going waveform present following an imperative stimulus. The PINV, which reflects the immediate motor execution related to onset of an imperative stimulus, has been observed during experiments investigating learned helplessness or loss of control (32, 33). Thus, the PINV allows the linkage of MRCPs to the readiness to act (34).
If an enactive account of perception, action, and cognition is correct, then affordances are intimately related to higher hierarchical levels through low-level perceptual cues. Such an account would situate the processing of affordances at a similar stage as early perceptual processes and should reveal differences in sensory and motor-related ERPs associated with the perceived affordance of an environment. To investigate whether brain activity is altered depending on affordances offered by the environment, we presented human observers with environmental stimuli that allowed or prohibited a transition from one room to the next. To this end, the participants were presented with a view into a room containing one door of different widths, allowing or prohibiting a transition into the next room and thus providing different affordances. We expected to find differences in cortical responses to covary as a function of affordances over sensory and motor areas. In addition, we expected to see differences in MRCPs as a function of the environmental affordances when the participants walked through the door or remained in the same room.
Methods
Participants.
Twenty participants (9 females) with no history of neurologic pathologies were recruited from a participant pool of the Technical University of Berlin, Germany. All participants provided signed informed consent expressing knowledge of the experimental protocol, which was approved by the Ethics Committee of the Technical University of Berlin. Participants received either monetary compensation (10 €/h) or accredited course hours. The mean age of participants was 28.1 y (σ = 6.2 y). All participants had normal or corrected-to-normal vision, and none had a specific background in architecture (no architects or architectural students). One participant was excluded due to technical issues of the experimental setup.
Paradigm Description.
The experiment was conducted at Berlin Mobile Brain/Body Imaging Laboratories (BeMoBIL) in an experimental rooms with an area of 160 m2. The size of the virtual space was 9 m × 5 m, with room sizes of 4.5 m × 5 m for the first room and 4.5 m × 5 m for the second room. The participants performed a forewarned (S1-S2) Go/NoGo paradigm (pseudorandomized 50/50) in the VR environment that required them to walk from one room to a second room. Doors of different widths, ranging from unpassable (20 cm, Narrow) to passable (100 cm, Mid) to easily passible (1,500 cm, Wide), manipulated the transition affordance between rooms.
The experiment was a 3 × 2 repeated-measures design including the factors door width (Narrow, Mid, Wide; pseudorandomized) and movement instruction (Go, NoGo). A total of 240 trials per participant were collected, with 40 trials for each of the factor levels. In one trial, the participants started in a dark environment on a predefined starting square (Fig. 1). After a random intertrial interval (mean = 3 s, σ = 1 s), the “lights” were turned on, and the participants faced a room with a closed door. The participants were instructed to wait (mean = 6 s, σ = 1 s) for a color change of the door, with a change to green indicating a Go trial and a change to red indicating a NoGo trial. In the case of a green door, the participants walked toward the door, which slid aside. On entering the subsequent space, the participants were instructed to find and virtually touch a red rotating circle using the controller. The circle informed the participants that they had earned another 0.1 € for their basic reimbursement of 10 €/h. After each trial, the participants were asked to provide an emotional rating of their state irrespective of whether they transitioned through the door (Go condition) or remained in the same room (NoGo condition) without transition. To this end, the participants were instructed to go back to the starting square and complete a virtual Self-Assessment Manikin (SAM) questionnaire, using a laser pointer from the controller, and subsequently engage the response button located at the pointer finger to turn the lights off. The lights then turned back on automatically to start the next trial.
In Go trials, the participants were instructed to walk toward the door and into the second room even if the door was too narrow to pass. This was done to control for motor execution in the Go condition and to allow movement toward the goal irrespective of the affordance (passable vs. unpassable). A narrow opening was thus different from a NoGo trial, in the sense that a NoGo trial did not require any movement toward the door, whereas a Go trial always required approaching the door. When a participant touched the surrounding walls, the walls turned, indicating that the participant failed to pass and thus must return to the start square, complete the virtual SAM, and start the next trial.
Participants would quickly notice that the narrow door (20 cm) was impossible to pass without producing the warning feedback that they have failed to pass, and yet they were required to try passing. All participants underwent a training phase to become accustomed to the VR environment and the different conditions. The experimenter observed the participants from a control room, separated from the experimental space, using two cameras and a mirrored display of the VR environment to reduce interactions to a minimum during the experiments.
Subjective and Behavioral Data.
To investigate the subjective experience of the task, we introduced the participants to a virtual SAM questionnaire after each trial. The SAM is a pictorial assessment of pleasure, arousal, and dominance on a 5-point Likert scale (27). The manikin display ranges from smiling to frowning (pleasure), from a dot in the stomach to an explosion (arousal), and from being very small to very big (dominance). Participants were asked to self-assess their current state after each trial. Furthermore, regarding behavioral measures, we recorded the reaction time from the onset of the Go stimulus (door color change) to reaching the opening threshold itself, to assess the behavior. The data were analyzed using analysis of variance (ANOVA) with the width of the doors as a repeated-measures factor. In the case of violation of normality and homogeneity, corrected P values are reported. For post hoc analysis, the data were contrasted using Tukey’s honest significant differences (HSD) test.
EEG Recording and Data Analysis.
We investigated the impact of transitional affordances on human cognition and brain dynamics using a MoBI approach (24–26, 28) recording human brain dynamics in participants actively transitioning through virtual rooms. All data streams were recorded and synchronized using LabStreamingLayer (35). Participants wore a backpack, which held a high-performance gaming computer to render the VR environment (Zotac; PC Partner Limited) attached to two batteries and an EEG amplifier system. We combined a Windows Mixed Reality (WMR; 2.89″, 2,880 × 1,440 resolution, update rate at 90 Hz, 100° field of view with a weight of 440 g, linked to the Zotac computer through HDMI) headset and one controller (Acer) to display and interact with the virtual environment based on Unity (Fig. 2). Events for recordings of performance and physiological data were triggered by the position of the participant in the tracking space or by the respective response buttons of the remote control. Specific events, such as touching the wall, all button presses, transitioning through the door, answering the questionnaire, and all cases of “lights on” (and “lights off”), were synchronized with the recorded brain activity and the presented VR environment through LabStreamingLayer.
EEG data were acquired continuously with a 64-channel EEG system (eegoSports, ANT Neuro), sampled at 500 Hz. Impedances were kept below 10 kΩ. The computational delay generated by the interaction of ANT Neuro software, Windows Mixed Reality, and Unity was measured as 20 ms (σ = 4 ms), which was taken into account during the analysis by subtracting the average delay from each event latency. With a jitter of 4 ms, we considered the delay to have little to no impact on the ERPs. Offline analysis was conducted using MATLAB (MathWorks) and the EEGLAB toolbox (36). The raw data were bandpass-filtered between 1 Hz and 100 Hz and down-sampled to 250 Hz. Channels with more than five SDs from the joint probability of the recorded electrodes were removed and subsequently interpolated. The datasets were then re-referenced to an average reference and adaptive mixture independent component analysis (ICA) (37) was computed on the remaining rank of the data using one model with online artifact rejection in five iterations. The resultant ICA spheres and weights matrices were transferred to the raw dataset that was preprocessed using the identical preprocessing parameters as for the ICA dataset, except for the filtering, which used a bandpass filter from 0.2 Hz to 40 Hz. Subsequently, independent components reflecting eye movements (i.e., blinks and horizontal movements) were removed manually based on their topographic, spectral, and temporal characteristics.
Epochs were created time-locked to the onset of the room including the closed door (lights on) from −500 ms before to 1,500 ms after stimulus onset for Narrow, Mid, and Wide door trials. Similarly, another set of epochs was time-locked to the second Go/NoGo stimulus from −500 ms before to 1,000 ms after onset of the stimulus for Narrow, Mid, and Wide door trials. On average, 15% (σ = 10.8) of all epochs were automatically rejected when they deviated by >5 SDs from the joint probability and distribution of the activity of all recorded electrodes.
The visual evoked potentials and MRCPs were analyzed at central midline electrodes (Fz, FCz, Cz, Pz, POz, and Oz) covering all relevant locations, including the visual cortex and the motor cortex, as reported previously (31, 38). Because stimuli were distributed across the complete visual field and participants walked through the virtual spaces, we did not expect to see any lateralization of ERPs. All channels were analyzed; however, only three channels (FCz, Pz, and Oz) are discussed here, according to findings reported by Bozzacchi et al. (31). The analysis results of all six channels are provided in SI Appendix. For peak analysis of the P1-N1 complex, the grand average peaks were estimated, and individual peaks were defined as the maximum positive peak and negative peak in the time window surrounding the grand average P1 and N1 peaks (±10 ms from the peak), respectively. An automatic peak detection algorithm detected the peaks in the averaged epochs for each participant. Multiple peaks were detected and systematically weighted depending on the magnitude, the distance to the grand average peak latency as determined by visual inspection of grand average ERP, and the polarity. The algorithm is provided in SI Appendix. For anterior N140 and posterior P140, by visual inspection of the grand average ERPs, the estimated grand average latency was 140 ms, with a search window for individual peaks ranging from 50 to 200 ms. For the anterior P215 and posterior N215, the estimated grand average peak latency was 215 ms, with a search window for individual peaks ranging from 140 to 290 ms.
Mean peak amplitudes were analyzed by 3 × 3 repeated-measures ANOVA using the door width (Narrow, Mid, or Wide) and electrode as repeated measures. The results descriptions focus on the visual evoked P140 component at posterior electrodes (Pz, POz, and Oz) and the N140 component at frontal leads (Fz, FCz, and Cz) based on separate ANOVAs. For the N215 and P215 components at posterior electrodes (Pz, POz, and Oz) and frontal leads (Fz, FCz, and Cz), separate ANOVAs were computed in the time range of 140–290 ms. For the later motor-related potentials, an ANOVA was computed for the mean amplitude in the range of 600–800 ms. The data were analyzed using a 2 × 3 × 6 factorial repeated-measures ANOVA with the factors imperative stimulus (Go and NoGo), door width (Narrow, Mid, and Wide), and electrode location (Fz, FCz, Cz, Pz, POz, and Oz) within the time window (600–800 ms). For post hoc analysis, the data were contrasted using Tukey’s HSD. In cases of violation of sphericity, corrected P values are reported. All ANOVAs were computed as linear mixed models.
Results
We obtained subjective, behavioral, and electrophysiological data, with a focus on electrophysiology. All data underwent statistical processing and are presented here in the order processed.
Subjective Data: SAM Ratings.
The SAM questionnaire was completed for Go or NoGo and for all door conditions. A 2 × 3 factorial repeated-measures ANOVA with the factors imperative stimulus (Go and NoGo) and door width (Narrow, Mid, and Wide) for each emotional dimension of the SAM questionnaire revealed differences in the main effect for width in Arousal (F2,90 = 3.35, P = 0.0393, η2 = 0.048), Dominance (F2,90 = 10.03, P < 0.0001, η2 = 0.138), and Valence (F2,90 = 5.31, P = 0.0065, η2 = 0.073). For the imperative stimulus, differences were found for Arousal (F1,90 = 36.81, P < 0.0001, η2 = 0.266), Dominance (F1,90 = 25.26, P < 0.0001, η2 = 0.173), and Valence (F1,90 = 28.59, P < 0.0001, η2 = 0.196). Interaction effects revealed significant differences for Dominance (F2,90 = 4.14, P = 0.0189, η2 = 0.056) and Valence (F2,90 = 7.04, P = 0.0014, η2 = 0.096) but only tendencies for Arousal (F2,90 = 0.92, P = 0.4000, η2 = 0.0134). Post hoc contrasts using Tukey’s HSD test (Fig. 3) showed no significant differences for NoGo in Arousal but identified significant differences for Go between Narrow × Mid (P = 0.0386). For NoGo in Dominance, no significant differences were revealed as opposed to Go for Narrow × Wide (P < 0.0001), Mid × Wide (P = 0.0335), and Narrow × Mid (P < 0.0345). Similarly, for valence, in Go significant differences were revealed for Narrow × Mid (P = 0.0326), Narrow × Wide (P < 0.0001), with a tendency seen for Mid × Wide (P = 0.0625).
Behavioral Data: Door Approaching Times.
This analysis was possible only for Go trials, as it required actually approaching the door. The time it took participants from the Go stimulus to pass the door was calculated using one-way ANOVA with repeated measures for different door widths, which revealed a significant difference in door widths (F2,36 = 6.07, P < 0.0053, η2 = 0.232; Fig. 4). Post hoc comparison with Tukey’s HSD test showed no significant differences in behavior when approaching the Narrow doors compared with the Mid doors (P = 0.3073), but had a tendency to be slower when approaching Mid doors compared with Wide doors (P = 0.1312) and a significant difference between approaching Narrow doors compared with Wide doors (P = 0.0038), with significantly faster approach times for the Wide door condition.
Electrophysiology: Early ERP.
Posterior P140.
With onset of the lights that allowed participants to see the room including the door (i.e., lights on), the ERPs clearly demonstrated a P1-N1 complex that was most pronounced over the occipital midline electrode, with a first positive component around 140 ms, followed by a negative peak around 210 ms (Fig. 5; all six channels shown in SI Appendix, Fig. S1). At the frontal midline electrode, this pattern was inverted, and a negative component around 140 ms was followed by a positive peak observed around 215 ms. The 3 × 3 repeated-measures ANOVA on P140 amplitudes for posterior channels revealed significant main effects for both door width (F2,108 = 8.163, P = 0.005, η2 = 0.096) and channel (F2,36 = 15.868, P < 0.0001, η2 = 0.187). The interaction effect was not significant (F4,108 = 1.669, P = 0.1624). Post hoc comparisons using Tukey’s HSD test revealed significant differences in peak amplitudes at channel Oz between Narrow and Mid transitions (P = 0.0021) and between Narrow and Wide transitions (P = 0.0065) and at channel POz between Narrow and Wide transitions (P = 0.028).
Posterior N215.
The 3 × 3 repeated-measures ANOVA on N215 amplitudes for posterior channels revealed a significant main effect for the factor door width (F2,108 = 4.348, P = 0.0153, η2 = 0.066) but no significant impact for the factor channels (F2,36 = 0.0893, P = 0.9147, η2 = 0.001). Post hoc Tukey HSD contrasts revealed no significant differences for Pz and POz. However, similar to posterior P140, significant differences at Oz for the comparison of Narrow and Mid transitions (P = 0.0113) and for the comparison of Narrow and Wide transitions (P = 0.0372) were found (Fig. 6).
Anterior P215.
An inverse pattern was observed for amplitudes over anterior leads, with a main effect of door width that differed depending on the affordances (F2,108 = 11.071, P < 0.0001, η2 = 0.139). The main effect of channels also reached significance (F2,36 = 5.3627, P = 0.0092, η2 = 0.067). Tukey HSD contrasts revealed significant differences only between Narrow and Wide transitions for FCz (P = 0.0071) and Cz (P = 0.0214), with a tendency at Fz (P = 0.0717). The interaction was not significant.
Anterior N140.
The 3 × 3 repeated-measures ANOVA on N140 amplitudes for anterior channels revealed no significant main effect for the factor door width (F2,108 = 1.823, P = 0.1663, η2 = 0.024). In contrast, the main effect of channels reached significance (F2,108 = 8.109, P = 0.0012, η2 = 0.107). The interaction did not reach significance.
EEG–motor-related processes.
After onset of the imperative stimulus, a positive peak at anterior leads and a negative peak at posterior leads were observed. For the sake of brevity, this potential complex is referred to as the early postimperative complex (EPIC). Reflecting its similar cortical polarity to the P1-N1 complex, the EPIC was analyzed in a similar way, separating anterior leads (Fz, FCz, and Cz) from posterior leads (Pz, POz, and Oz), and detecting single peaks in individual averages.
Anterior EPIC.
A 2 × 3 × 3 repeated-measures ANOVA revealed significant differences in the main effect for widths (F2,270 = 4.21, P = 0.0157, η2 = 0.025), imperative stimulus (F1,270 = 23.66, P < 0.0001, η2 = 0.071), and channel (F2,36 = 6.70, P = 0.0033, η2 = 0.040). No interaction effect was observed. The post hoc Tukey’s HSD test revealed no significant differences between the transition widths for the various channels and for the imperative stimuli.
Posterior EPIC.
The identical ANOVA for the posterior potentials of the EPIC revealed no significant impact of transition width (F2,270 = 2.001, P = 0.1371, η2 = 0.013) or imperative stimulus (F1,270 = 2.30, P = 0.1298, η2 = 0.007). Significant differences in EPIC amplitudes were observed for the factor channel (F2,36 = 5.45, P = 0.0085, η2 = 0.035). Because topographical differences were not the focus of this study, no further post hoc contrasts were computed. No interaction was significant.
PINV.
In the preparation time before the onset of the door color change, indicating that the participant was either to walk through the door or remain in the same room, we observed no systematic negative going waveform as reported in previous studies (29, 39). However, after the onset of the color change, a pronounced positivity, the EPIC, followed by a long-lasting negative waveform over frontocentral locations was observed in the ERP (Fig. 7 ; all six channels shown in SI Appendix, Fig. S2). This negative waveform resembled a PINV, as described previously (32, 34, 40). The PINV component was observed at 600–800 ms after the imperative stimulus (color change of the door) and varied as a function of the affordance of the environment (door width). A global 2 × 3 × 6 factorial repeated-measures ANOVA was computed to analyze the MRCPs using Go/NoGo, width, and channel as repeated measures. ANOVA revealed significant differences in the main effect for Go/NoGo (F1,540 = 19.54, P < 0.0001, η2 = 0.039) and for channel (F5,90 = 16.69, P < 0.0001, η2 = 0.112). Significant differences were reported for the interaction effect of Go/NoGo × channel (F5,540 = 5.25, P = 0.0001, η2 = 0.035) and for width × channel (F10,540 = 2.61, P = 0.0042, η2 = 0.035). A tendency toward an interaction of the factors Go/NoGo × Width (F2,540 = 2.33, P = 0.0975, η2 = 0.006) was observed.
Post hoc contrasts using Tukey’s HSD test revealed significant differences only for the Go condition as opposed to the NoGo condition (Fig. 8). Similar to the early evoked potentials, differences were observed only at frontal and occipital sites and between Narrow and Mid doors over FCz (P = 0.0059) and Oz (P < 0.0001), as well as between Narrow and Wide doors at FCz (P = 0.0323) and Oz (P < 0.0001). No differences were observed between the Mid and Wide doors (SI Appendix, Fig. S3).
Discussion
The main goal of this study was to assess whether brain activity is altered depending on the affordances offered by the environment. If this were the case, then affordances should systematically modulate behavior and brain activity. Specifically, we hypothesized that perceptual processes would covary with the environmental affordances, leading to behavioral changes, and that MRCPs would vary as a function of affordances.
SAM and Approach Time.
The analysis of subjective ratings revealed significant differences among Go trials but no differences across NoGo trials for all ratings. Notably, in cases of NoGo, all participants perceived a similar scene standing in front of a red (NoGo) door, turning around, and answering the virtual SAM. Varying door sizes for Go trials yielded differences for Dominance, with Narrow doors more dominant than Mid doors and even more dominant than Wide doors. The increase in Dominance for Narrow doors is inversely reflected in Valence because we observed increasing values with increasing door widths. Regarding Arousal, participants reported less arousal for Narrow doors compared with Mid and Wide doors. Furthermore, it is noteworthy that Dominance for NoGo is relatively high in value compared with NoGo in Arousal and Valence, which score low and central values, respectively. Taken together, these findings indicate that subjective reports differ significantly depending on whether participants who received a Go actively moved through the rooms, implying an impact of action affective ratings of an environment. However, our findings should be considered with caution, as the subjective ratings might have been influenced by several factors beyond affordance, including monetary reward, different trial durations, physical activity, and varying skills of subjective/introspective emotional evaluation.
Thus, performance data might provide a better basis for interpreting the impact of affordances on behavior. The time it took participants to reach the door after the onset of the imperative color change varied according to the environmental affordance. Participants approached the Wide doors nearly significantly faster and significantly faster than the Mid and Narrow doors, respectively, while there was no significant difference between Mid and Narrow transitions. While the Wide door clearly offered a passage without greater demands regarding the motor plan and execution, the Mid door width, being ambiguously wide/narrow, might have triggered motor processes simulating a transition to estimate whether the door was passable. In this sense, the Mid and Narrow doors, causing uncertainty, might have delayed approach times due to increased processing demands. Admittedly, results derived from the approach time are limited, due in part to fatigue from performing a physically demanding task for a relatively long period, and in part to passing a door that is seemingly impossible to pass. This led participants to develop different approach strategies, such as twisting their bodies, peeking inside from different angles, or walking directly into the virtual wall to trigger a failed attempt, causing different delays. Given that no participant was told beforehand that one opening was impassable, participants’ enthusiasm and creativity diminished over the course of the trials when they learned that it was the Narrow door. However, the fact that participants in general spent significantly more time approaching the Narrow doors compared with Wide doors provides sufficient guidance for the analyses of cortical measures associated with these differences.
Cortical Measures.
Early evoked potentials.
As an initial insight into the association of affordances and cortical potentials, we analyzed the early visual-evoked potentials. We expected to find differences in the stimulus-locked ERP at occipital channels, reflecting differences in sensory processing of affordance-related aspects of the transition. Based on the assumption of fast sensorimotor active inferences that should be reflected in action-directed stimulus processing influencing not only sensory activity, but also motor-related activity, we hypothesized that we would also find differences in the ERP over motor areas in the same time window as sensory potentials (i.e., between 50 and 200 ms). As illustrated in the analysis, we found significant differences in amplitudes of the visually evoked P140 component over the central occipital electrode varying with the affordance of the transition. In addition, and in line with our hypothesis, we also found a difference over frontocentral leads starting around 50 ms and lasting until 200 ms after onset of the door display. Taken together, these findings indicate that no significant differences in peak amplitudes were found between the passable Mid and Wide doors, while peak amplitudes associated with both door widths significantly differed from those of the impassable Narrow doors.
Note that the visual scene of the three doors was comparable, as they contained the same physical contrasts in the Go and the NoGo condition. In addition, being merely introduced to the environmental setting, participants did not know whether they would have to attempt to pass. These results indicate that impassable doors with poor affordances produce significantly different early evoked potentials compared with passable doors, particularly at the frontocentral and occipital sites. Thus, environmental affordances, in terms of being able to program a trajectory to transit spaces, yield a significant measurable effect on early cortical potentials best pronounced over frontal and occipital sites at ∼200 ms after the first view of the environment.
Considering the affordance-specific pattern observed for the early P1-N1 complex, previous studies have shown that this visual-evoked potential complex reflects attentional processes associated with spatial or feature-based aspects of stimuli (41–45). Attended stimuli elicit larger P1-N1 amplitudes than unattended stimuli. Based on these findings, our results suggest that passable transitions were associated with increased attentional processing. Keeping this in mind, when viewing the affordance-specific pattern of the P1-N1 complex in light of active inferences (46), the difference confirms the assumption that perceptual processes covary with environmental affordances. In this sense, the amplitude difference might be credited to the process of actively inferring whether the body can move and transit at all, implying that visual attention is also guided by action-related properties of the environment. Similar to HAC (23) and active inference (22, 47), these findings are in line with parallel cortical processes integrating sensory information to specify currently available affordances.
How one might act on the environment is an ongoing process of resolving affordances, taking place as early as perceptual processes, which situates actions in an intimate position with perception. Such early processes are deeply involved in the conception and articulation of the environment for an agent, pointing toward the importance of movement in cognition, and of how an agent continuously enacts the world.
Motor-related potentials.
Although the ERP plots indicated an affordance trend of the EPIC, statistical tests revealed no significant differences. However, the Narrow door width elicited the greatest amplitude in both anterior positivity and posterior negativity. The increased amplitude associated with Narrow transitions can be interpreted as a reflection of the body simply not fitting, producing a prediction error because one is forced to interact with the transition. The nature of the PINV component has not been as well investigated as other ERP components, limiting the reliability of an interpretation. Some studies treat this component as modality-unspecific “electrocortical correlate of a cognitive state” (48). Gauthier and Gottesmann (49) hypothesized that the PINV, similar to affordances, acts as a marker of change in the psychophysiological state. Subsequently, the PINV has been used to investigate depression, schizophrenia, learned helplessness, and loss of control (32–34, 50, 51). Depressive and schizophrenic individuals exhibit an increased PINV that is explained as an increased vulnerability for loss of control, as well as increased anticipation for future events (32, 34, 40). It must be emphasized that affordances reflect actions directed toward the future. If an increased PINV reflects increased vulnerability for future events, as we observed for impassable doors, then the component might shed new light on the intentionality in affordances. Given the intention to pass, yet being deprived of doing so, seems to be reflected in the PINV. Casement et al. (34) suggested the PINV depends on lack of control as the state of having no influence, depriving the potential to act. This could explain the difference in the Narrow condition, as participants were instructed to attempt to pass at all times until failure, even for impassable openings, leading to a sense of loss of control.
Only in cases of Go did we observe a difference in the PINV component, which varied with the environmental affordances. Amplitudes of the component for Narrow doors differed significantly different from those for Mid and Wide doors, while the passable conditions did not differ among the doors. Furthermore, there were no significant differences in the PINV component in cases of NoGo, emphasizing the importance of the motor execution itself in evoking the PINV component. These results point toward the PINV component as an expression of the readiness to interact with the designed environment (i.e., less negative for passable doors and more negative for impassable doors), thus serving as a potential marker for the readiness to act given environmental affordances. Our results are also consistent with the observed increase in activity over frontocentral sites reported by Bozzacchi et al. (31), who concluded that the meaning of the action and awareness of being able to act—affordances—affect action preparation, which is here understood as the motor-related potential before movement onset. We argue that the PINV component might reflect a readiness aspect of affordances. This would mean that the PINV is not modulated by the perception that the door is different visual information, but reveals something about the readiness to act. For this reason, we find significant differences in cases of Go but not in cases of NoGo, and also for passable compared with impassable.
In light of HAC (23), a potential explanation for the absence of differences in the NoGo trials is related to the immediate action selection, which in all cases (Narrow, Mid, and Wide) is a simple turn to answer the questionnaire, and thus the task presents the participant with identical affordances. When instead given a Go, cortical processes require an action selection related to the anticipated motor trajectory, which differs according to the affordances of the door width. HAC suggests the higher levels bias the lower-level competitions, which operate at the level of action itself, through a cascade of expected next affordances. The lower levels have a continuous competition of how to satisfy the higher expectations. Action selection, executed while unfolding the planned movements in a continuous manner, depends on the expectation of next affordances.
Of note, regarding architectural experience, because the PINV component was expressed only in the Go condition (i.e., forced interaction with the environment), these findings support the importance of movement for architectural experience, in a sense that action or even only the perception of action possibilities alters brain activity. Visually guiding and propelling the body in space greatly influences the continuous emerging of affordances, which in turn affect the human experience. We found differences in frontocentral and occipital areas before movement through space, with the postimperative negative-going waveform most pronounced over FCz indicating involvement of the supplementary motor area (SMA), as reported by Bozzacchi et al. (31). Previous studies showed involvement of the SMA in visually guided actions (52), which is the essence of active inferences. The PINV can be generated independently from the reafferent signal, which is, in terms of active inference, understood as ascending (bottom-up) proprioceptive prediction errors (53). This suggests that the PINV component might reflect descending (top-down) predictions, making the SMA an essential area of the action–perception loop and thus crucial for processing continuous affordances. This might resolve the finding of frontocentral differences in Go trials only. The SMA is anatomically bridging the frontal cortex with the motor cortex—perhaps also functionally, as argued by Adams et al. (53), because this anatomical nature fits with the proposed hierarchical characteristics of forward and backward projections in active inferences.
Using VR to investigate cortical processes has its natural limitations, such as the absence of a physical body. Regarding the sense of body, which is at stake in the present study, it has been suggested that VR “may offer new embodied ways for assessing the functioning of the brain by directly targeting the processes behind real-world behaviors” (54), which is remarkably valid for the present study. Riva et al. (54) argued that the brain’s predictive capability immerses the body, and thus related processes, if the visual perception is in line with the body’s actions, for instance, by head movements and wandering. Through a process of trial and error, the brain and body adjust to VR. Furthermore, in terms of architecture, VR as a head-mounted display (55) and as a CAVE system (56) has been integrated into studies with bodily and environmental interests, yielding comparable results. However, VR in combination with neuroscientific methods remains a newer technique and thus must be used with care. It must be emphasized that the purpose of VR in the current experimental setup was to isolate and control the factor of interest. Future studies using MoBI in real-world environments are needed to investigate whether the results from VR can be generalized to the real world.
Conclusion
The present study provides strong evidence for affordances to be processed as early as perceptual processes, linking action and perception in a similar manner to active inference. The results point toward a conception of the brain that seems to deal with “how can I act” while in parallel processes referring to “what do I perceive” take place. These results thus support the assumption that perception of the environment is influenced by affordances and action itself, and thus affordances and action can influence the experience of an environment. Given the importance of affordances and action for brain dynamics, this further emphasizes and qualifies the general idea of enactivism as a holistic approach to investigating cognition. We do not claim that architectural affordances are directly represented as a specific ERP component; however, we provide evidence for an action–perception account of cognition, which systematically differentiates according to the definition of affordances.
As a note for architects, the fact that we are mobile and predictive beings suggests that architects should take the temporal aspect as seriously as the spatial aspect, given that the predictive process of unfolding bodily movement can alter the perception of space. Moving and transitioning in space is to continuously construct a prediction of a world that we perceive as dependent on our action potentials, which informs brain, body, and mind. Altering perception would ultimately lead spaces to have a potential physiological impact on users. Much remains to be uncovered in architectural cognition.
Supplementary Material
Acknowledgments
We thank Lukas Gehrke, Marius Klug, and Federica Nenna for their support with the technical setup, in particular Federica Nenna for her additional support in conducting the experiment. We also thank Andrea Jelic for important discussions and bringing essential insights of enactivism to our knowledge.
Footnotes
The authors declare no conflict of interest.
This article is a PNAS Direct Submission. E.R.M. is a guest editor invited by the Editorial Board.
Data deposition: All data are available in the Open Science Framework (https://osf.io/xywdh/), reference doi 10.17605/OSF.IO/XYWDH.
See Commentary on page 14404.
This article contains supporting information online at www.pnas.org/lookup/suppl/doi:10.1073/pnas.1900648116/-/DCSupplemental.
References
- 1.Gibson J. J., The Ecological Approach to Visual Perception (Psychology Press–Taylor & Francis Group, East Sussex, UK, 1986). [Google Scholar]
- 2.Clark A., An embodied cognitive science? Trends Cogn. Sci. 3, 345–351 (1999). [DOI] [PubMed] [Google Scholar]
- 3.Clark A., Surfing Uncertainty, Prediction, Action and the Embodied Mind (Oxford University Press, New York, 2015). [Google Scholar]
- 4.Friston K., Kilner J., Harrison L., A free energy principle for the brain. J. Physiol. Paris 100, 70–87 (2006). [DOI] [PubMed] [Google Scholar]
- 5.Hohwy J., The Predictive Mind (Oxford University Press, Oxford, UK, ed. 1, 2013). [Google Scholar]
- 6.Friston K., The free-energy principle: A unified brain theory? Nat. Rev. Neurosci. 11, 127–138 (2010). [DOI] [PubMed] [Google Scholar]
- 7.Friston K., The free-energy principle: A rough guide to the brain? Trends Cogn. Sci. 13, 293–301 (2009). [DOI] [PubMed] [Google Scholar]
- 8.Friston K., Mattout J., Kilner J., Action understanding and active inference. Biol. Cybern. 104, 137–160 (2011). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Friston K., Learning and inference in the brain. Neural Netw. 16, 1325–1352 (2003). [DOI] [PubMed] [Google Scholar]
- 10.Clark A., Whatever next? Predictive brains, situated agents, and the future of cognitive science. Behav. Brain Sci. 36, 181–204 (2013). [DOI] [PubMed] [Google Scholar]
- 11.OED Oxford English Dictionary (Oxford University Press, Oxford, UK, 2018). [Google Scholar]
- 12.Jelić A., Tieri G., De Matteis F., Babiloni F., Vecchiato G., The enactive approach to architectural experience: A neurophysiological perspective on embodiment, motivation, and affordances. Front. Psychol. 7, 481 (2016). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Holl S., Pallasmaa J., Pérez-Gómez A., Questions of Perception : Phenomenology of Architecture (William Stout Publishers, San Fransisco, ed. 1, 2006). [Google Scholar]
- 14.Pallasmaa J., The Embodied Image: Imagination and Imagery in Architecture (Wiley, West Sussex, UK, ed. 1, 2011). [Google Scholar]
- 15.Bachelard G., Jolas M., Stilgoe J. R., The Poetics of Space (Beacon Press, Boston, ed. 1994, 1994). [Google Scholar]
- 16.Norberg-Schulz C., Nightlands (MIT Press, Cambridge, Mass.; London, 1997). [Google Scholar]
- 17.Coburn A., Vartanian O., Chatterjee A., Buildings, beauty, and the brain: A neuroscience of architectural experience. J. Cogn. Neurosci. 29, 1521–1531 (2017). [DOI] [PubMed] [Google Scholar]
- 18.Varela F. J., Thompson E., Rosch E., The Embodied Mind: Cognitive Science and Human Experience (MIT Press, London, England, 2016), Revised ed. [Google Scholar]
- 19.Thompson E., Mind in Life: Biology, Phenomenology, and the Sciences of Mind (Belknap Press of Harvard University Press, London, England, ed. 1, 2007). [Google Scholar]
- 20.Maturana H. R., Varela F. J., The Tree of Knowledge : The Biological Roots of Human Understanding (Shambhala, Boston, ed. 1, 1992). [Google Scholar]
- 21.Gallagher S., Enactivist Interventions: Rethinking the Mind (Oxford University Press, Oxford, ed. 1, 2017). [Google Scholar]
- 22.Kiebel S. J., Daunizeau J., Friston K. J., A hierarchy of time-scales and the brain. PLoS Comput. Biol. 4, e1000209 (2008). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Pezzulo G., Cisek P., Navigating the affordance landscape: Feedback control as a process model of behavior and cognition. Trends Cogn. Sci. 20, 414–424 (2016). [DOI] [PubMed] [Google Scholar]
- 24.Makeig S., Gramann K., Jung T.-P., Sejnowski T. J., Poizner H., Linking brain, mind and behavior. Int. J. Psychophysiol. 73, 95–100 (2009). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25.Gramann K., et al. , Cognition in action: Imaging brain/body dynamics in mobile humans. Rev. Neurosci. 22, 593–608 (2011). [DOI] [PubMed] [Google Scholar]
- 26.Gramann K., Jung T.-P., Ferris D. P., Lin C.-T., Makeig S., Toward a new cognitive neuroscience: Modeling natural brain dynamics. Front. Hum. Neurosci. 8, 444 (2014). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27.Bradley M. M., Lang P. J., Measuring emotion: The self-assessment manikin and the semantic differential. J. Behav. Ther. Exp. Psychiatry 25, 49–59 (1994). [DOI] [PubMed] [Google Scholar]
- 28.Gramann K., Ferris D. P., Gwin J., Makeig S., Imaging natural cognition in action. Int. J. Psychophysiol. 91, 22–29 (2014). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29.Brunia C. H. M., CNV and SPN: Indices of Anticipatory Behavior in The Bereitschaftspotential, Jahanshahi M., Hallett M., Eds. (Springer, New York, 2003), pp. 207–227. [Google Scholar]
- 30.Di Russo F., et al. , Beyond the “Bereitschaftspotential”: Action preparation behind cognitive functions. Neurosci. Biobehav. Rev. 78, 57–81 (2017). [DOI] [PubMed] [Google Scholar]
- 31.Bozzacchi C., Giusti M. A., Pitzalis S., Spinelli D., Di Russo F., Awareness affects motor planning for goal-oriented actions. Biol. Psychol. 89, 503–514 (2012). [DOI] [PubMed] [Google Scholar]
- 32.Diener C., Kuehner C., Brusniak W., Struve M., Flor H., Effects of stressor controllability on psychophysiological, cognitive and behavioural responses in patients with major depression and dysthymia. Psychol. Med. 39, 77–86 (2009). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 33.Elbert T., Rockstroh B., Lutzenberger W., Birbaumer N., Slow brain potentials after withdrawal of control. Arch Psychiatr Nervenkr (1970) 232, 201–214 (1982). [DOI] [PubMed] [Google Scholar]
- 34.Casement M. D., et al. , Anticipation of affect in dysthymia: Behavioral and neurophysiological indicators. Biol. Psychol. 77, 197–204 (2008). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 35.Kothe C., Lab Streaming Layer (2014). https://github.com/sccn/labstreaminglayer. Accessed 5 April 2018.
- 36.Delorme A., Makeig S., EEGLAB: An open source toolbox for analysis of single-trial EEG dynamics including independent component analysis. J. Neurosci. Methods 134, 9–21 (2004). [DOI] [PubMed] [Google Scholar]
- 37.Palmer J. A., Kreutz-Delgado K., Makeig S., AMICA: An adaptive mixture of independent component analyzers with shared components (2011). https://sccn.ucsd.edu/∼jason/amica_a.pdf. Accessed 20 October 2018.
- 38.Bozzacchi C., Spinelli D., Pitzalis S., Giusti M. A., Di Russo F., I know what I will see: Action-specific motor preparation activity in a passive observation task. Soc. Cogn. Affect. Neurosci. 10, 783–789 (2015). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 39.van Boxtel G. J. M. Böcker KBE cortical measures of anticipation. J. Psychophysiol. 18, 61–76 (2004). [Google Scholar]
- 40.Klein C., Rockstroh B., Cohen R., Berg P., Contingent negative variation (CNV) and determinants of the post-imperative negative variation (PINV) in schizophrenic patients and healthy controls. Schizophr. Res. 21, 97–110 (1996). [DOI] [PubMed] [Google Scholar]
- 41.Hillyard S. A., Anllo-Vento L., Event-related brain potentials in the study of visual selective attention. Proc. Natl. Acad. Sci. U.S.A. 95, 781–787 (1998). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 42.Posner M. I., Dehaene S., Attentional networks. Trends Neurosci. 17, 75–79 (1994). [DOI] [PubMed] [Google Scholar]
- 43.Mangun G. R., Hopfinger J. B., Heinze H.-J., Integrating electrophysiology and neuroimaging in the study of human cognition. Behav. Res. Methods Instrum. Comput. 30, 118–130 (1998). [Google Scholar]
- 44.Gramann K., Töllner T., Müller H. J., Dimension-based attention modulates early visual processing. Psychophysiology 47, 968–978 (2010). [DOI] [PubMed] [Google Scholar]
- 45.Gramann K., Toellner T., Krummenacher J., Eimer M., Müller H. J., Brain electrical correlates of dimensional weighting: An ERP study. Psychophysiology 44, 277–292 (2007). [DOI] [PubMed] [Google Scholar]
- 46.Friston K. J., et al. , Dopamine, affordance and active inference. PLoS Comput. Biol. 8, e1002327 (2012). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 47.Friston K., Active inference and free energy. Behav. Brain Sci. 36, 212–213 (2013). [DOI] [PubMed] [Google Scholar]
- 48.Rockstroh B., Cohen R., Berg P., Klein C., The postimperative negative variation following ambiguous matching of auditory stimuli. Int. J. Psychophysiol. 25, 155–167 (1997). [DOI] [PubMed] [Google Scholar]
- 49.Gauthier P., Gottesmann C., Communications in electroencephalography. Electroencephalogr. Clin. Neurophysiol. 43, 449–583 (1977).95724 [Google Scholar]
- 50.Kathmann N., Jonitz L., Engel R. R., Cognitive determinants of the postimperative negative variation. Psychophysiology 27, 256–263 (1990). [DOI] [PubMed] [Google Scholar]
- 51.Klepeis N. E., et al. , The national human activity pattern survey (NHAPS): A resource for assessing exposure to environmental pollutants. J. Expo. Anal. Environ. Epidemiol. 11, 231–252 (2001). [DOI] [PubMed] [Google Scholar]
- 52.Picard N., Strick P. L., Activation of the supplementary motor area (SMA) during performance of visually guided movements. Cereb. Cortex 13, 977–986 (2003). [DOI] [PubMed] [Google Scholar]
- 53.Adams R. A., Shipp S., Friston K. J., Predictions not commands: Active inference in the motor system. Brain Struct. Funct. 218, 611–643 (2013). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 54.Riva G., Wiederhold B. K., Mantovani F., Neuroscience of virtual reality: From virtual exposure to embodied medicine. Cyberpsychol. Behav. Soc. Netw. 22, 82–96 (2019). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 55.Pasqualini I., Llobera J., Blanke O., “Seeing” and “feeling” architecture: How bodily self-consciousness alters architectonic experience and affects the perception of interiors. Front. Psychol. 4, 354 (2013). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 56.Vecchiato G., et al. , Electroencephalographic correlates of sensorimotor integration and embodiment during the appreciation of virtual architectural environments. Front. Psychol. 6, 1944 (2015). [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.