Attention and schema violations of real world scenes differentially modulate time perception

Ourania Tachmatzidou; Argiro Vatakis

doi:10.1038/s41598-023-37030-2

. 2023 Jun 20;13:10002. doi: 10.1038/s41598-023-37030-2

Attention and schema violations of real world scenes differentially modulate time perception

Ourania Tachmatzidou ¹, Argiro Vatakis ^1,^✉

PMCID: PMC10281981 PMID: 37340029

Abstract

In the real world, object arrangement follows a number of rules. Some of the rules pertain to the spatial relations between objects and scenes (i.e., syntactic rules) and others about the contextual relations (i.e., semantic rules). Research has shown that violation of semantic rules influences interval timing with the duration of scenes containing such violations to be overestimated as compared to scenes with no violations. However, no study has yet investigated whether both semantic and syntactic violations can affect timing in the same way. Furthermore, it is unclear whether the effect of scene violations on timing is due to attentional or other cognitive accounts. Using an oddball paradigm and real-world scenes with or without semantic and syntactic violations, we conducted two experiments on whether time dilation will be obtained in the presence of any type of scene violation and the role of attention in any such effect. Our results from Experiment 1 showed that time dilation indeed occurred in the presence of syntactic violations, while time compression was observed for semantic violations. In Experiment 2, we further investigated whether these estimations were driven by attentional accounts, by utilizing a contrast manipulation of the target objects. The results showed that an increased contrast led to duration overestimation for both semantic and syntactic oddballs. Together, our results indicate that scene violations differentially affect timing due to violation processing differences and, moreover, their effect on timing seems to be sensitive to attentional manipulations such as target contrast.

Subject terms: Psychology, Human behaviour

Introduction

Every object in the world has its own place. This arrangement of items not only is not random but is also characterized by a few rules. These rules are referred to as “syntactic” and “semantic” (as termed in¹) and indicate the appropriate spatial coordinates of several objects in a visual scene. Syntactic relations refer to the spatial relations of scenes and objects, while semantic relations within an object and a scene are about the probability, size, and position of the given object in the scene. In this latter case, knowing the object’s identity and function is required. A more recent and slightly different interpretation of semantic and syntactic relations among objects and scenes comes from Vo and Wolfe². According to them, semantics are about the object’s fitting into the global context of a scene, while syntactic relations refer to objects being at their expected location in a scene. This notion was further supported by the different neural representations of semantic and syntactic scene violations with the former producing negative deflections in the N300-N400 time window, while the latter positive deflections in the P600^2,3.

This so-called scene “grammar” is, thus, responsible for our understanding of the visual environment, the recognition of the objects in it, and, moreover, the execution of specific actions depending on the surroundings⁴. It can also assist one in inferring additional information, like the existence of other objects in the scene and their position in it⁵. However, what happens when scene grammar rules are violated, is equally interesting. An early study from Loftus and MacKworth⁶ showed that semantically inconsistent objects in a scene (i.e., an octopus in a farmyard) were fixated faster and for longer times compared to consistent with the scene objects. These findings along with more recent ones^7,8 suggest that objects that violate scene grammar might attract attention even before becoming fully identified by the observer. That is also known as the pre-attentive pop-out effect⁹. Yet, this effect has not always been reported, with some studies showing that when semantic and syntactic violations are applied, inconsistent objects are identified slower and less accurately than the consistent ones^1,10,11. In a more recent study, Vo and Henderson¹² investigated the degree of attentional capture via eye-movements when semantic and syntactic inconsistencies were present in real-world scenes. Their results did not support the pre-attentive pop out account, as they saw that there was no extrafoveal processing of inconsistent objects. However, they found that after fixation, gaze duration for inconsistent objects was longer and that first fixation was even longer for syntactically inconsistent objects compared to semantically inconsistent ones. Thus, suggesting that syntactic and semantic inconsistencies are processed in the fovea area (i.e., the area with the greatest visual acuity that is responsible for object identification) and, moreover, that a prolonged attention allocation is needed (even greater for the syntactic violations). This research points out the need to further investigate not only the perception of those violations as a function of attention, but also their potential differential effects on perception (i.e., semantic versus syntactic).

The effects of scene violations in perception have been investigated in the timing domain (note that we use the terms “timing” and “time perception” interchangeably to refer to the experienced or subjective temporal value that one utilizes to make their judgement; e.g.,¹³). For instance, Clarke and Porubanova¹⁴ examined the role of semantic knowledge for real-world scenes and objects, hypothesizing that violations of that knowledge would lead to duration overestimations, as a result of the increased neural processing needed for their encoding. They, thus, presented images of real-world scenes with semantic violations or no violations in a duration reproduction task. They also included an attentional manipulation by either drawing or distracting participants’ attention to the violations presented (i.e., participants were asked to report scene-object inconsistencies or the main character’s gender by pressing a key, respectively). Analyses showed that scene duration in the presence of a violation (i.e., semantic) was overestimated as compared to no violations, an effect that was independent of the attentional manipulation used. These findings were interpreted based on the neural energy model of time dilation¹⁵ that supports that perceived duration correlates positively with the amount of cognitive and neural processing necessary for stimulus encoding (i.e., coding efficiency; cf.^16,17). This model is an alternative to the idea that subjective time expansion is driven by attentional accounts (e.g.,^18–20). An example of the latter account can be found in Tse et al.’s²⁰ study that showed evidence of time dilation due to the allocation of attention to an unexpected stimulus (cf.^21–23). Specifically, using an oddball paradigm, where a low-probability stimulus (i.e., odd) is presented in a stream of high-probability stimuli (i.e., standards), the former was perceived as longer compared to the latter stimulation. Such accounts are, also, in line with Schweitzer et al.’s²⁴ study, where they utilized contextually-associated (e.g., standard was a pizza and odd a pizza cutter) and non-associated (i.e., standard was a pizza and odd a rubber duck) odds in an oddball paradigm, in which, as expected, both odds were overestimated as compared to standards, yet the contextually-associated odds exceeded the temporal dilation of the non-associated ones. [It must be noted here, however, that the objects in this study were presented in isolation and not in the context of a scene. Thus, this study does not really refer to a scene violation, but rather to a semantic congruence or incongruence of the stimulus stream presented in each trial]. It was argued that the larger dilation occurred due to the attraction of top-down attention to the contextually associated oddballs²⁴.

Given the prolonged attentional allocation accounts in the scene grammar violation literature (e.g.,¹²) and the time dilation noted for semantic-like scene violations¹⁴, one wonders whether any schema violation (i.e., violation of an existing mental construction about the real world; e.g.,²⁵) would lead to an expansion of timing and, moreover, whether attention is the main cognitive effect involved in this phenomenon. In the present pre-registered study (see pre-registration here https://doi.org/10.17605/OSF.IO/S3QT5), therefore, we aimed to investigate these issues by conducting two experiments. In Experiment 1, we utilized the oddball paradigm along with naturalistic visual scenes with or without violations, with the violations being of syntactic or semantic form²⁶. We hypothesized that in the presence of both semantic and syntactic violations, time dilation would occur due to the prolonged attentional allocation to the violation^14,24. We, also, expected that this dilation would be greater in the presence of a syntactic as compared to a semantic violation given that the initial encoding of the former requires more time than the latter (cf.¹²). In Experiment 2, we utilized the same setup as in Experiment 1, but we manipulated attention allocation to the target objects of the violation by increasing their contrast relative to their background, an effect known to modulate attention (cf.^27,28). We hypothesized that time dilation would be observed for both types of violations. Yet, we expected that the effect would be stronger in the increased contrast conditions due to an increased gaze allocation to the target²⁷. The results of these two experiments are expected to inform the scene grammar literature in terms of the conflicting accounts of pop-out or prolonged attentional processing and the timing literature in terms of the mechanisms underlying time dilation.

Experiment 1

Methods

Participants

G*Power²⁹ was used to perform an a priori power analysis for a one-sample t-test, comparing participants’ duration estimations with the standard duration. The effect size for this analysis was estimated based on Cohen’s guidelines³⁰. This indicated that the best estimate of the true population standardized mean difference was δ = 0.80, meaning that duration estimations will be reported as higher or lower from the standard duration. This effect size estimate was entered into the power analysis with the following input parameters: a (two-sided) = 0.05, power = 0.95. The power analysis results suggested that a N = 23 is required in this study to detect a difference between the mean of the estimated durations and the standard duration value.

A similar a priori power analysis was also performed for a repeated measures analysis of variance comparing duration estimation for real-world scenes with violations (i.e., syntactically, semantically) with duration estimations for each type of scene utilized. This indicated that the best estimate of the true population standardized mean difference was δ = 0.40, meaning that the duration estimations will be different among scenes with different violations and type. This effect size estimate was entered into the power analysis with the following input parameters: a = 0.05, power = 0.95. The power analysis results suggested that a N = 12 is required in this study to detect a difference between the two conditions with 95% probability.

We recruited twenty-nine participants, assuming that some participants may not follow the experimental instructions or complete the study. All participants were university students (23 female), aged between 19 and 32 years old (mean age = 21) with normal or corrected to normal vision. Participants participated voluntarily for their own interest or the extra credit course opportunities offered in the University through online advertisements and social media posts. When they expressed interest in participating, they were provided with detailed information about the experimental procedure, and they signed the informed consent documentation. This study was approved by the ethics committee of the Panteion University of Social and Political Sciences (protocol number: 33/27-6-2022). All methods included in the present study were performed in accordance with the institution’s relevant guidelines and regulations.

Apparatus

The experiment was programmed and run on OpenSesame 3.3³¹. The stimuli were presented on a 21.5-inch FUJITSU Display E22-8 TS Pro computer monitor, set at 1920 × 1080 resolution. The operating system was Windows 10.

Stimuli

Nine color images (i.e., 3 scenes with no violations and their respective scenes with 3 semantic and syntactic violations) with a standard 4:3 aspect ratio and 1.024 × 768-pixel resolution, captured in the real world (i.e., in several different apartments) and containing items that are essential to every household (toilet paper, cup, remote control etc.) were used. These essential items served as the target objects that allowed for the creation of the semantic and syntactic violations. In the semantic violation condition, a semantically inconsistent object was presented in a syntactically consistent location (i.e., a cup in the bathroom). In the syntactic violation condition, a semantically consistent object was presented in a syntactically inconsistent and physically impossible location (i.e., a floating cup in the kitchen; see Fig. 1). All images were taken from the SCEGRAM database, an image-set that contains standardized semantic and syntactic object-scene inconsistencies²⁶. We selected the semantic and syntactic item violations to create the semantic and syntactic oddballs, respectively. Images subtended a visual angle of approximately 10° × 11.1° at a viewing distance of approximately 60 cm.

Scenes with and without violations utilized in Experiment 1. The figure depicts the three scenes utilized in the no violation (i.e., standard), semantic violation (e.g., toilet paper in the washing machine), and syntactic violation (e.g., floating cup) condition. These images were adopted from the SCEGRAM database (Öhlschläger and Vo³⁶).

Design and procedure

After they sat in a chair in front of the computer monitor, the participants were instructed to fixate at the center of the computer screen. They completed 216 experimental trials, in which they observed 8 repeated presentations of a real-world scene image (i.e., a standard) and one more with a syntactic or semantic violation (i.e., an oddball) randomly presented between the 5^th and the 8^th position³². Images were repeatedly presented at fixation for 500 ms, with ISIs of 300 ms, while the oddballs’ duration varied between 300 and 700 ms, in steps of 50 ms (i.e., 9 different durations). After each trial, participants were asked to report whether the oddball’s duration was longer or shorter than the duration of the images preceding and succeeding it (i.e., standards) by pressing the “,” or the “.” keyboard key (using the same hand, the dominant one), respectively. Both keyboard keys were labelled as “LONG” and “SHORT”, respectively. Before the main experiment, the participants completed 10 practice trials to familiarize themselves with the procedure. For the practice trials, the same three scenes as in the experimental task were used.

Results and discussion

Four participants were removed from the analysis as they failed to follow or understand the given instructions (thus, ending up with a N = 25). For all the analyses, Bonferroni-corrected t-tests (where p < 0.05 prior to correction) were used for all post-hoc comparisons. When sphericity was violated, Greenhouse-Geisser correction was applied. The psychometric function for each participant was estimated (see Fig. 2), using the MATLAB R2013b software, based on the number of times they reported that the oddball was “longer” than the standard. A psychometric function was fitted to the relative frequencies of “longer” responses per oddball duration level. The cumulative density function (cdf) of a normal distribution was used as the mathematical model for the psychometric function, a modeling approach known as probit analysis (see³³). The psychometric function was used for estimating the point of subjective equality (PSE; see Fig. 3), which is defined as the oddball’s duration for which the probability of giving the answer “longer” was 50%. That is, an oddball with a longer duration estimate as compared to the standard is being underestimated, while an oddball with a shorter estimate than the standard’s is being overestimated. The PSEs of each participant were used for further statistical analyses.

Representative psychometric functions for two participants for the semantic (a) and syntactic (b) violation conditions in Experiment 1.

Participants’ Point of Subjective Equality (PSE) of the oddball for the two violation types (i.e., syntactic, semantic) in Experiment 1.

A one-sample t-test was run to determine whether participants’ PSEs were different to the point of objective equality (POE), that is 500 ms. The analysis showed that PSEs for oddballs with syntactic violations were significantly lower by a mean of 35.9 ms (95% CI [11.9–59.9]) than POE [t(24) = − 3.088, p = 0.005]. On the contrary, PSE for oddballs with semantic violations were significantly higher by a mean of 22.3 ms (95% CI [42–2.5]) than POE [t(24) = 2.329, p = 0.029]. There was no significant difference of the mean PSE for both oddball types from POE [t(24) = − 0.870, p = 0.393].

To further explore the above-mentioned results, data were analyzed using a repeated measures ANOVA between Violation Type (2 levels: syntactic, semantic) and Scene (3 levels: scene 1, scene 2, scene 3). The alpha level was set to 0.05 and the confidence interval to 95%. A significant main effect of Violation Type was obtained [F(1, 24) = 35.82, p < 0.001, η_p² = 0.60], with the oddballs containing semantic violation having a higher PSE (M = 524 ms) as compared to those with the syntactic ones (M = 465.6 ms). A main effect of Scene was also obtained [F(2, 48) = 7.13, p = 0.002, η_p² = 0.23], with scene 2 having higher PSE (M = 514.2 ms) than scene 1 (M = 486.3 ms) and scene 3 (M = 483.8 ms; see Fig. 4). The analysis also revealed an interaction between Violation Type and Scene [F(2, 48) = 6.05, p = 0.005, η_p² = 0.20]. Post hoc analysis with a Bonferroni adjustment revealed that scene 2 (M = 555.5, SD = 13.1) was underestimated at a larger degree as compared to scene 3 (M = 497.5, SD = 10.9; p = 0.001), but only in the case of semantic violations.

Mean Point of Subjective Equality (PSE) for each Scene per Violation Type (i.e., syntactic, semantic) in Experiment 1. The error bars represent the standard error of the mean. *p < .05.

Overall, the results of Experiment 1 showed that time dilation, indeed, occurs in the presence of syntactic violations, yet, contrary to our predictions, the opposite is true for the semantic violations. This could indicate the existence of differential processing mechanisms for syntactic and semantic violations². Moreover, we observed that in the presence of semantic violations, one of the scenes (i.e., scene 2; see Fig. 1) utilized was significantly underestimated. That could possibly mean that apart from the semantic violation itself, additional scene properties (such as brightness³⁴;) might affect the scenes’ percept. In Experiment 2, we extended our research by questioning whether the timing modulations observed were influenced by attentional accounts (inspired by¹⁴).