Post-determined emotion: motor action retrospectively modulates emotional valence of visual images

Abstract

Upward and downward motor actions influence subsequent and ongoing emotional processing in accordance with a space–valence metaphor: positive is up/negative is down. In this study, we examined whether upward and downward motor actions could also affect previous emotional processing. Participants were shown an emotional image on a touch screen. After the image disappeared, they were required to drag a centrally located dot towards a cued area, which was either in the upper or lower portion of the screen. They were then asked to rate the emotional valence of the image using a 7-point scale. We found that the emotional valence of the image was more positive when the cued area was located in the upper portion of the screen. However, this was the case only when the dragging action was required immediately after the image had disappeared. Our findings suggest that when somatic information that is metaphorically associated with an emotion is linked temporally with a visual event, retrospective emotional integration between the visual and somatic events occurs.

1. Introduction

The current theory of embodied cognition describes how bodily experience influences emotional processing [1–6]: our perception, cognition and decision-making processes interact with our sensory–motor experiences. For example, Casasanto & Dijkstra [7] reported that participants tended to recollect positive/negative memories while moving their hands upward/downward, respectively. Participants have also been found to judge sentences with a positive meaning more quickly when raising their facial muscles by holding a pen between their teeth [8]. Moreover, the mood of participants has been found to increase after they assumed an upright posture [9] and experienced upward illusory self-motion [10]. These studies suggest that upward movement and upright postures are associated with a positive valence, whereas downward movement and slouching postures are associated with a negative valence.

Positive/negative valences appear to be represented metaphorically in the spatial domain based on the association between bodily experience and emotional valence: in daily conversation, we express positive emotions using upward motions or positions in space (e.g. ‘I'm feeling up’) and negative issues with downward motions or positions (e.g. ‘I'm known to be a bottom feeder’) [7,11–13]. This space–valence association is found in diverse linguistic backgrounds, including Japanese [14]. Indeed, a number of studies have revealed that somatic information can influence ongoing and subsequent emotional processing on the basis of this space–valence metaphor [7–10]. However, whether somatic information can influence previous emotional processing retrospectively is unclear.

Our mental representation of an external event is shaped not only by prior or current information about the event but also by the incorporation of current information with information received after the event. The retrospective effect of the information about a physically subsequent event on the psychological response to a current event is known as postdiction [15]. Previous studies have demonstrated postdictive modulation of the location [15,16], size [17], shape [18] and orientation [19] of visual objects. Similarly, the perceived positions of tactile (e.g. [20,21]) and auditory (e.g. [22]) stimuli have been found to shift towards subsequent stimuli. Postdictive integration has also been found between cross-modal signals (e.g. [23–26]). Taken together, these findings suggest that, if postdictive integration of multimodal information occurs not only for perceptual processing but also emotional processing, the emotional valence of a visual event could be modulated by information from other modalities (e.g. somatesthesia) received after the visual event. We tested this prediction.

We sought to ascertain whether postdictive integration is present in the associations between bodily and emotional experiences. To address this issue, we tested whether the emotional valence of a visual event was modulated based on the vertical direction (upward or downward) of a subsequent arm movement. Previous postdiction studies have reported that the effect of a subsequent input on a preceding event was diminished when the time interval between the events was long (e.g. [19,20]). Therefore, we predicted that a visual event preceding an upward arm movement would be evaluated as more positive than a visual event preceding a downward arm movement when the temporal interval between the visual and action events was short. Conversely, when the temporal interval was long, we expected the emotional valence of the visual event to be similar for both upward and downward arm movements.

2. Experiment 1

(a). Methods

(i). Participants

Twenty-eight dextral volunteers participated in Experiment 1 (12 males and 16 females, mean age ± s.e.m. = 22.75 ± 0.57 years). All were naive as to the purpose of the experiment and reported that they had normal vision and motor function.

(ii). Apparatus

Stimuli were presented on a 19-inch touch screen. The resolution of the monitor was 1280 × 1024 pixels and the vertical refresh rate was 60 Hz. The presentation of the stimuli and collection of the data were computer controlled. The stimuli were generated via MATLAB with the Psychtoolbox extension [27,28]. Key inputs were acquired using a numerical keypad.

(iii). Stimuli

The stimuli consisted of fixation marks, a rating system, an action cue and emotional images (figure 1). The stimuli were displayed on a grey background (43.5 cd m⁻²) and presented at a viewing distance of 50 cm. Each fixation mark was composed of two white concentric rings (radii, 0.29° and 0.17°; luminance, 187 cd m⁻²). The rating system had rating values that ranged from −3 to 3 and seven grey boxes that were outlined in black (1.38° × 0.57°; inside, 43.5 cd m⁻²; outline, 0.24 cd m⁻²). The rating values were presented within the boxes and arranged in ascending order from left to right. The rating values and boxes were presented on the centre of the screen. When one of the rating values was selected, the colour of the box surrounding the selected value changed from grey to deep grey (10.4 cd m⁻²).

Figure 1.

Schematic of one experimental trial. ISI, inter-stimulus interval. (Online version in colour.)

The action cue comprised a fixation mark, black dot and two rectangles. The colour of the dot was black (0.24 cd m⁻²) and its radius was 0.57° (visual angle). The fixation mark and black dot were presented at the centre of the screen. The participants moved the dot on the screen using their right index or annular finger. The rectangles were red (CIE xy coordinates 0.641/0.355 and 46.4 cd m⁻²) and blue (CIE xy coordinates 0.139/0.076 and 13.7 cd m⁻²), and their size was 9.5° × 40.6°. One of the two coloured rectangles was an earmarked area, to which the participants had been instructed to move the dot before the experiment began. The other was a non-earmarked area. The colours of the earmarked and non-earmarked areas were counterbalanced across the participants. In the vertical session, the two rectangles were presented 7.32° above and below the centre of the screen, whereas in the control session they were presented 7.32° to the left and right of the centre of the screen.

In the vertical session, the earmarked area was presented in either an upper (upward condition) or lower (downward condition) position on the screen, and the non-earmarked area was presented in the other position. That is, the participants had to raise or lower their right arm to move the dot to the upper or lower location on the screen in the upward or downward conditions, respectively. In the control session, the earmarked area was presented either to the left or right of the centre of the screen, and the non-earmarked area was presented in the other position.

We chose 60 emotional images from the International Affective Picture System (IAPS) [29] for use in both the vertical and control sessions (table 1). The size of each image was 12.8° × 16.8°. The fixation mark was at placed at the centre of the image. Twenty images each were used as positive (mean valence rating 7.90; mean arousal rating 4.56), neutral (5.38; 4.32) and negative (2.88; 4.68) stimuli. For the valence rating scores for each image, we conducted a one-way between-groups analysis of variance (ANOVA) with emotional valence (positive, neutral and negative stimuli) as a factor. The results indicated a significant main effect of emotional valence (F_{(2, 57)} = 579.78, p < 0.001, ). Multiple comparisons using Ryan's method [30] revealed that the rating score of the positive stimuli was higher than that of the neutral and negative stimuli (t₅₇ = 17.10, p < 0.001, Cohen's d = 6.25, and t₅₇ = 34.05, p < 0.001, Cohen's d = 10.86, respectively). Moreover, the rating score of the neutral stimuli was higher than that of the negative stimuli (t₅₇ = 16.95, p < 0.001, Cohen's d = 4.76). Conversely, the arousal rating scores were not different across emotional valence (F_{(2, 57)} = 1.77, p = 0.18, ).

Table 1.

List of images (IAPS numbers).

emotional valence	positive	neutral	negative
IAPS numbers	1440	1302	2682
	1441	1675	2722
	1460	2635	2750
	1463	5455	2753
	1710	5661	2800
	1750	5900	2900
	1920	7058	6242
	2040	7182	9000
	2050	7211	9110
	2091	7242	9220
	2209	7247	9280
	2260	7248	9290
	2311	7249	9330
	2332	7484	9342
	2340	7487	9390
	2388	7495	9620
	4626	7496	9830
	5594	7504	9902
	5760	7620	9925
	5833	8465	9926
mean valence	7.90	5.38	2.88
mean arousal	4.56	4.32	4.68

(iv). Procedure

The experiment was conducted in a dark room. The participant placed their hands on a desk. They used their left hand to make responses on a numerical keypad. Figure 1 shows the timeline of the trials. The participant initiated each trial by pressing the start key on the numerical keypad. The fixation mark was then presented for 500 ms on the centre of the screen. Following this, an emotional image was presented for 500 ms on the centre of the screen. We used two inter-stimulus intervals (ISIs) between the emotional image and the action cue: 0 or 2000 ms. When the ISI was 0 ms, the action cue was presented immediately as the emotional image disappeared. When the ISI was 2000 ms, the screen was left blank for 2000 ms after the disappearance of the emotional image, and then the action cue was presented. In the upward and downward conditions, the participant was asked to move the dot to the earmarked area using their right finger. After either 1500 ms had passed or the dot had reached the earmarked or non-earmarked area, the action cue was replaced by the rating system, which was presented at the centre of the screen. The participant was then asked to rate the emotional valence of each image using a 7-point scale from −3 (strongly negative) to 3 (strongly positive) using selection keys and a decision key.

To fully differentiate the somatic from the visual events, we chose to present a blank screen for 2000 ms in the long ISI condition. A previous study reported that a (partial) sense of agency lingered even when the ISI between a visual stimuli and an action was beyond 1000 ms [31]. Considering this, we added an additional 1000 ms to the ISI to reduce the chance of visuo-somatic grouping.

Thirty images (10 positive, 10 negative and 10 neutral images) were assigned to one action cue condition and the other 30 images were assigned to the other action cue condition, such that 60 trials were performed in each session. The trial order was randomized across the participants. Each participant completed both the vertical and control sessions, and the order of the sessions was counterbalanced across the participants. Before each session, the participants completed 20 practice trials. Because the aim of the practice trials was to teach the participants to move the dot on the screen, we presented a neutral image (the same for each trial) in the practice trials. This image was not included in the stimulus set for the test sessions. Each participant was assigned to one of two ISI conditions: 12 participated in the 0 ms ISI condition and 16 in the 2000 ms ISI condition.

(b). Results and discussion

We excluded the ratings of emotional valence for trials in which the dot did not reach the earmarked area within 1500 ms or the participant moved the dot to the non-earmarked area. Thus, we discarded 0.3% of all of the data. To examine the effect of the dot-moving action on the emotional evaluation of the images, we first calculated the valence bias by subtracting the average ratings from the control condition from those in the upward and downward conditions (figure 2). If the valence bias in a direction condition was 0, then we considered the emotional valence of the image in that condition to be comparable to that in the control condition. Negative values indicated that the emotional images were evaluated more negatively in the upward/downward conditions compared with the control condition, whereas positive values indicated that the emotional images were evaluated more positively in the upward/downward conditions compared with the control condition. We conducted a two-way mixed ANOVA with arm movement direction as a within-groups factor and the ISI as a between-groups factor. The results showed a significant main effect of arm movement direction (F_{(1, 26)} = 6.93, p = 0.01, ), while the main effect of the ISI was not significant (F_{(1, 26)} = 1.75, p = 0.20, ). The significant main effect of arm movement direction indicates that the valence bias was higher in the upward condition (M = 0.006) than in the downward condition (M = −0.08). The interaction between arm movement direction and ISI was significant (F_{(1, 26)} = 5.68, p = 0.02, ). Post hoc tests revealed that a simple main effect of arm movement direction was significant in the 0 ms condition (F_{(1, 11)} = 10.78, p < 0.01, ), whereas this effect was not significant in the 2000 ms condition (F_{(1, 15)} = 0.04, p = 0.85, ). Briefly, when the ISI was 0 ms, the valence bias was higher in the upward condition (M = 0.08) than in the downward condition (M = −0.09) and when the ISI was 2000 ms, the valence bias was similar in the upward (M = −0.05) and downward (M = −0.06) conditions. Moreover, a simple main effect of ISI was significant in the upward condition (F_{(1, 26)} = 5.83, p = 0.02, ), whereas this effect was not significant in the downward condition (F_{(1, 52)} = 0.39, p = 0.54, ). To determine whether upward and downward movements positively and negatively biased the valences of the images in the 0 ms condition, respectively, we conducted one-sample t-tests between 0 and the valence bias of each direction. The results showed that the valence biases were significantly different from 0: the image was evaluated positively in the upward condition (t₁₁ = 2.20, p = 0.05, Cohen's d = 0.90), and negatively in the downward condition (t₁₁ = 2.15, p = 0.05, Cohen's d = −0.88).

Figure 2.

The results of Experiment 1. The grey and white bars show the valence bias in the downward and upward conditions, respectively. Error bars denote s.e.m.

As predicted, upward and downward arm movements affected the evaluation of a previously presented emotional image when the ISI was 0 ms, while this effect was not observed when the ISI was 2000 ms. These results suggest that when a somatic event that is metaphorically associated with an emotion occurs immediately after a visual event that evokes an emotion, the subsequent somatic information is integrated postdictively with the previous information given by the visual image. Eventually, the emotional valence of the image is modulated on the basis of space–valence metaphor processing (e.g. [7,11,13]). However, when the visual and somatic events are separated by a long interval, the somatic information is not integrated with the visual information, and thus postdictive modulation of emotional valence does not occur.

Another possible explanation is that this modulation is not based on the space–valence metaphor, but instead, on a response bias stemming from the link between vertical space and the numerical values on the rating scale (up = large number, down = small number). If the results of Experiment 1 were due to this response bias, then the subsequent action should modulate not just the emotional evaluation but also the evaluation of other stimuli properties that are unrelated to emotion. We addressed this issue in Experiment 2.

3. Experiment 2

In Experiment 2, we asked participants to evaluate the density of objects in emotional images. If the results of Experiment 1 were owing to the response bias derived from the link between vertical space and the numerical values of the rating scale, then the subsequent action should affect both the participants' evaluations of object density and evaluations of emotional valence.

(a). Methods

(i). Participants

Twenty-eight dextral individuals, none of whom participated in Experiment 1, completed Experiment 2 (9 male and 19 female, mean age ± s.e.m. = 23.93 ± 0.66 years). The participants were naive as to the purpose of the experiment and reported that they had normal vision and motor function.

(ii). Apparatus, stimuli and procedure

The apparatus, stimuli and procedure were identical to those in Experiment 1 except that the participants were asked to report how densely they felt that objects were distributed in the images, instead of rating the intensity of the emotional valence. To do this, the participants used a 7-point scale from −3 (low density) to 3 (high density). Each participant was assigned to one of two ISI conditions: 12 participated in the 0 ms ISI condition and 16 in the 2000 ms ISI condition.

(b). Results and discussion

As in Experiment 1, we excluded ratings obtained for incorrect and incomplete trials (as a result, we discarded 0.3% of all of the data) and calculated the density bias (figure 3). We conducted a two-way mixed ANOVA with arm movement direction as a within-groups factor and the ISI as a between-groups factor. We found that the main effect of arm movement direction (F_{(1, 26)} = 0.24, p = 0.63, ), ISI (F_{(1, 26)} = 0.78, p = 0.39, ), and the interaction between arm movement direction and ISI (F_{(1, 26)} = 0.03, p = 0.87, ) were not significant. Therefore, the subsequent action did not appear to influence the evaluation of object density in the previously viewed image. A previous study reported that human judgement is susceptible to bias when the object being judged is ambiguous (e.g. [32]). In this study, the density evaluation required ambiguous judgements because the images that we used varied widely in terms of content: they ranged from natural and daily scenes to abstract art and were thus unfit for comparative density evaluations. Considering this, we expected it more likely that the density bias would significantly differ between the direction conditions in comparison with the valence bias of Experiment 1 if the effect of arm movement stemmed simply from the response bias. We found the contrary: while the effect of arm movement was present in the valence bias in Experiment 1, this effect was absent in the density bias in Experiment 2. Taken together, these results enabled us to dismiss the possibility that the response bias solely accounted for the results of Experiment 1.

Figure 3.

The results of Experiment 2. The grey and white bars show the density bias in the downward and upward conditions, respectively. Error bars denote s.e.m.

4. General discussion

Our results indicate that motor action postdictively modulated the emotional valence of a visual event. Moreover, we established that this postdictive modulation was not owing to a response bias stemming from the link between vertical space and numerical values on a rating scale. Our findings suggest that, on the basis of space–valence metaphor processing, bodily experience affects previous emotional processing as well as ongoing and subsequent emotional processing [7–10].

When we interpret an event in the external world, we use not only information obtained during the event but also information received after the event [15]. To date, the postdictive effect has been observed in several sensory modalities, including visual [15–19], auditory [22] and tactile [20,21] perception. Moreover, several studies have reported postdictive integration of audio–visual [24,25] and tactile–visual [23,26] signals. The present study provides further evidence: subsequent somatic information contributed to the emotional valence of a preceding visual event. Thus, it appears that postdictive integration of multimodal inputs is not limited to sensory processing, but also applies to emotional processing. This phenomenon may exist for other types of cognitive processing as well.

Previous studies have shown that visual categorization concerning facial expressions is affected by auditory inputs (e.g. [33–35]). Tactile inputs also influence the evaluation of emotional expressions when the evaluator has taken oxytocin [36]. In these studies, the multimodal signals were input simultaneously. To the best of our knowledge, this is the first study to demonstrate that the emotional valence of a visual event can be modulated by a subsequent somatic event. Our findings suggest that emotional experience is formed through the integration of multimodal information received both simultaneously and within a short subsequent temporal window.

One might argue that our findings were related to the mood induced by the vertical action, which was metaphorically associated with emotion. Specifically, via mood congruence, an action-induced positive or negative mood might have directly affected evaluations in the rating phase. If this were the case, then the effect of mood should have been equivalent, such that emotional valence was modulated in both the 0 ms and the 2000 ms conditions. This would occur because the temporal distance from the action to the evaluation was equivalent between the ISI conditions. However, our data did not reflect this situation, leading us to reject this possibility.

The association between vertical direction and valence, i.e. ‘positive is up/negative is down’, may be established through the use of metaphorical expressions in daily life (e.g. ‘feeling down’ and ‘high on life’; see [11]) or through implicit learning of associations between concurrent bodily and emotional experiences (e.g. when feeling good, we are upright; e.g. [1,2]). These notions raise a possibility: if people in some cultures use the opposite metaphorical expression or circumstances, i.e. where the co-occurrence between bodily and emotional experiences opposes that of the participants in this study, the association between vertical space and emotional valence should also be reversed. In this case, the postdictive effect of vertical actions on the emotional valences of images might be opposite to that observed in our study: that is, downward and upward actions would bias the emotional valence of images positively and negatively, respectively. However, the association that ‘positive is up/negative is down’ is found in diverse linguistic backgrounds [14], and thus it may be difficult to find a culture that holds a reverse association. Moreover, although it may be possible to reconstruct the space–valence association by teaching people that ‘negative is up/positive is down’, the association ‘positive is up/negative is down’ is salient and steady [14], making reconstruction unlikely. Future methodological advances might enable us to empirically investigate how cultural and individual differences in metaphorical expressions contribute to the construction of metaphorical associations.

Although we found that a motor action induced postdictive modulation of the emotional valence of a visual event, the processing stage at which this postdictive integration occurred remains unclear. There are two possibilities: a representative [17,37] or decisional (e.g. [38]) stage. The former possibility would imply that the subsequent information modulated the mental representation of the preceding event. In this case, modulation of the emotional valence occurs regardless of the presence of an evaluation task. The latter possibility implies that when we make a decision during a task, we use information related to the task. When our participants evaluated the emotional valence of the visual event, they might have used somatic information that was metaphorically associated with emotion and located temporally close to the visual event. As a result, the somatic information might have biased their evaluation regarding the emotional valence of the visual event. Future studies might clarify the stage (i.e. representative or decisional) at which upward/downward actions are integrated with emotional valences of a visual event.

Ethics statement

The protocol of this was approved by the ethical committees of Kyushu University (approval number: 2013-005). We conducted the experiments according to the guidelines established by the Helsinki Declaration. We obtained written informed consent from all participants.

Data accessibility

We submitted our data as electronic supplementary material.

Author contributions

All of the authors designed the experiment. K.S. conducted the experiment and analysed the data. All authors discussed the data and wrote the paper.

Funding statement

This work was supported by a Grant-in-Aid for JSPS Fellows (#266025) given to K.S., a Grant-in-Aid for Challenging Exploratory Research (#26540067) given to Y.Y., and Kyushu University Interdisciplinary Programs in Education and Projects in Research Development (#26806 and #26307) given to Y.Y. and K.M.