Action Recognition Depends on Observer’s Level of Action Control and Social Personality Traits

Abstract

Humans recognize both the movement (physical) goals and action (conceptual) goals of individuals with whom they are interacting. Here, we assessed whether spontaneous recognition of others’ goals depends on whether the observers control their own behavior at the movement or action level. We also examined the relationship between individual differences in empathy and ASD-like traits, and the processing of other individual’s movement and action goals that are known to be encoded in the “mirroring” and “mentalizing” brain networks. In order to address these questions, we used a computer-based card paradigm that made it possible to independently manipulate movement and action congruency of observed and executed actions. In separate blocks, participants were instructed to select either the right or left card (movement-control condition) or the higher or lower card (action-control condition), while we manipulated action- and movement-congruency of both actors’ goals. An action-congruency effect was present in all conditions and the size of this effect was significantly correlated with self-reported empathy and ASD-like traits. In contrast, movement-congruency effects were only present in the movement-control block and were strongly dependent on action-congruency. These results illustrate that spontaneous recognition of others’ behavior depends on the control scheme that is currently adopted by the observer. The findings suggest that deficits in action recognition are related to abnormal synthesis of perceived movements and prior conceptual knowledge that are associated with activations in the “mirroring” and “mentalizing” cortical networks.

Introduction

The ability to recognize behavior of other agents is critical for successful social interaction, reproduction and survival [1]. Recognition of other individuals’ behavioral goals can occur either in a deliberate or a spontaneous manner [2], and at distinct levels of either physically defined movements or conceptually defined actions [3,4]. This ability to recognize others’ behavior is not equally present in all individuals. A profound disability in processing own and other individuals’ goals is associated with social and communicative challenges that are often observed in individuals diagnosed with autism spectrum disorder (ASD) [5-14].

The term ‘goal’ has often been used in a ill-defined manner within the field of action research [15,16]. A useful distinction in the alleged goal hierarchy [17-21] can be made between the level of physical movement goals which are associated with the kinematics of individual goal-directed movements (e.g., reaching towards the right), and conceptual action goals, which reflect functional expectations that govern movement execution (e.g., reaching to an object to drink coffee). Action goals include information about the expected external outcome of an action [22,23] and rely on prior conceptual knowledge [24] regarding relationships between objects in the environment and their purpose (e.g., cups are used to drink coffee). While physical movement goals typically involve a one-to-one perceptuo-motor mapping between the end-goal location and the observed movement kinematics [15,25-28], action goals may involve one-to-many mappings where a conceptual goal could be associated with expectation of multiple movement goals, dependent on the context in which behavior is embedded. In the present article, we use congruency effects between others’ and own behavior at the movement goal and at the action goal levels as an index for spontaneous goal recognition.

Neurally, the perceptuo-motor based account of action recognition suggests that the processing of other individuals’ goal-directed movements entails an automatic mapping to the observer’s own movement representations in the “action observation network/mirror neuron system (AON/MNS)” [6,7,29-31]. Prior behavioral research on action in social settings has mainly focused on movement goals, while neglecting the more complex conceptual action planning involved in everyday behavior [3]. In line with the perceptuo-motor account, it has repeatedly been shown that observation of another individual’s movement influences the observer’s subsequent execution of similar movements [32-36].

In contrast, in order to explain guidance of more complex social behavior, conceptual accounts of action understanding focus on the processing of other individuals’ conceptual action goals [37-39]. Cortical processing of action goals has been associated with activations in the “mentalizing/theory of mind network” comprising temporal and midline circuits, including medial parietal and prefrontal cortices and the superior temporal sulcus [40,41]. Recent behavioral work suggests that conceptual action goal congruency can have a powerful effect on subsequent action execution [4,42,43]. For example, we have recently suggested that during joint action observers rely on the same mechanism to process both their own action goals and the action goals of other individuals [4]. The results from this study also showed that when an observer’s behavioral control depends on the coactors goals (i.e., observer needed to match or mismatch coactors action goals), observed movement goals influence movement execution only when their action goals match.

In contrast to the deliberate processing of others’ conceptual action goals [4], in many social situations individuals’ brains are spontaneously processing others’ behavior while simultaneously controlling their own action selection [2]. For example, spontaneous recognition of others’ behavioral goals appears in a situation in which you are having breakfast with your colleague. While you are reaching to the right for your knife, you might still be able to recognize your colleague’s movement goal towards a cup as well as the fact that she is having coffee; even though these goals are not directly related to the selection and control of your own behavior.

The current study had two primary aims. First, we assessed whether spontaneous goal recognition, as indexed by movement and action congruency effects, depends on the level of the control scheme adopted by the observer (i.e., movement or action). Second, to investigate a possible relationship between social personality traits and the level of social interaction, we asked whether these traits were related to congruency indices of action and movement goal recognition. We used a modified version of the card-selection paradigm employed by Ondobaka and colleagues (2012) to manipulate the congruency between participant’s and coactor’s conceptual action goals and physical movement goals. In the original experiment, participants needed to explicitly recognize coactor’s behavior in order to match or mismatch their action goal. In the current study, however, participants were explicitly instructed (written instructions prior to each block) to adopt specific types of control scheme independent of the coactor’s behavior. In 4 different blocks of trials, participants adopted a movement goal (i.e., movement-control condition) or an action goal (i.e., action-control condition) which guided the selection of one of two cards that were presented on a touch screen in front of them. In the movement-control blocks participants selected a card by focusing on its physical location (i.e., left or right), and in the action-control blocks they selected the card with the higher or lower value. During both the movement-control condition and the action-control condition participants observed the behavior of a confederate coactor who, immediately prior to the action of the participant, choose his own card. After the card-game, participants completed the Interpersonal Reactivity Index (IRI) [44] and the Autism-Spectrum Quotient (AQ) [45] that assessed their self-reported social personality traits. Whereas the IRI assesses individual differences in empathy on four subscales, the AQ provides a measure of the degree to which an individual with normal intelligence shows autistic traits. In short, the aim of this study was to test whether spontaneous recognition of other individuals’ behavioral goals is contingent on the behavioral control scheme adopted by the observer. Moreover, we were interested to examine whether individual differences in empathy and ASD personality traits would relate to movement goal congruency effects, or rather to conceptual action goal congruency effects.

Materials and Methods

Participants and apparatus

The study was approved by the Ethics Committee for Behavioural Research of the Social Sciences Faculty at Radboud University Nijmegen and is in accordance with the Helsinki declaration. All participants provided their written consent. Sixteen healthy (3 male), right-handed [46] participants were recruited from the Radboud University’s student population (mean age: 23,8 years; range: 21-31 years). Participants seated at a custom-built table (length = 120 cm, width = 80 cm) facing a male confederate coactor. A 19-in. touch screen (Elo Touch, Elo Touch Systems, Menlo Park, CA) and one start button on each long side of the screen were embedded in the table at a level even with the tabletop (see Figure 1 for an illustration of the experimental setup). Start buttons and the touch screen were connected to a PC using Presentation Software (Neurobehavioral systems Inc., Albany, CA). The program enabled us to detect stimulus onset, release of the start button (movement onset time) and contact with the touch screen (arrival time) with millisecond accuracy. Reaction time (RT) was calculated by subtracting the stimulus onset time from the movement onset time on each trial.

Figure 1. Experimental setup and illustration of a action/congruent-movement/congruent trial.

Trials started with all four cards face down. After a variable delay (0.5–2.5 s), the two cards on the coactor's side were revealed. This triggered the coactor to make a selection, which caused the two cards on the participant's side to be revealed immediately. At this point, after having scanned all the revealed cards for the no-go cue (i.e., card with the value 2) the participant then made their choice according to the pertinent instruction for the given block. Immediately after selecting their cards by touching the screen, the coactor and participant placed their index fingers on their start buttons; the next trail was initiated when the coactor and then the participant had their fingers on the start buttons. The trial illustrated here is action congruent because both the coactor and the participant chose the card with the higher value; it is movement congruent because both the coactor and the participant moved to their right.

Procedure, design, and stimuli

All participants performed the experimental task with the confederate coactor. On each trial, four cards were presented faced down, together with the start cue (Figure 1 ). Next, the start cue disappeared and two cards were revealed to the confederate coactor, who had to select the card with either the higher or the lower value. So that the coactor would choose higher and lower cards at random, he was presented with either a high or low auditory cue at the beginning of each trial (i.e., the word higher or lower, audible only to the coactor through the earphones). The purpose of the start cue was solely to indicate the start of the trial. After a variable delay (0.5–2.5 s), the reminder cue disappeared, and the cards on the coactor's side of the screen were revealed. When the coactor chose one card by using his right index finger to press it on the touch screen, the cards on the participant's side of the screen were instantaneously revealed (see Figure 1). At this point, the participant used his or her right index finger to select a card according to the block instruction. Immediately after selecting their cards by touching the screen, the coactor and participant placed their index fingers on their start buttons. Finally, the next trail was initiated when the coactor and then the participant had their fingers on the start buttons. Participants were uninformed regarding whether the coactor had to select higher/lower card or left/right card. At the beginning of each block, the participant received an instruction to either select the left or the right card (i.e., movement-control condition) or the higher or the lower card (i.e., action-control condition) of the two cards presented in front of them. Additionally, both the participant and the coactor were instructed to adopt a predefined upright body posture, with the head in line with the centre of the touch screen and the start button. Furthermore, participants were explicitly instructed to constantly pay attention and be as fast and as accurate as possible throughout the experiment.

To prevent the possible influence of ‘low level’ perceptual effects on RTs, the mapping of the cards was randomized in such a way that the 4 cards were always perceptually dissimilar. We took care that the lower/higher card appeared equally often on the left/right for both the coactor and the participant. Furthermore, the numerical distance between the value of the cards on both sides was always one. We also balanced the sum of the two cards such that it was equally likely to be higher or lower on any given trial. The card selected by the participant was independent of the card selected by the confederate coactor. However, in order to ensure that participants paid attention to the actions of the confederate coactor, they were required to scan all the revealed cards on the table and to refrain from making a response when a card with the value “2” (i.e., no-go cue) was revealed. This requirement was equal for all trials and did not bias participants’ performance towards any particular condition. No-go cards appeared in 23% of the trials in each block, and had an equal probability of appearing on the participant’s or the coactor’s side. At the level of movement goals, the responses of the coactor and the participant could be physically congruent (movement-congruent condition) or physically incongruent (movement-incongruent condition) from an egocentric perspective. Similarly, at the level of apparent action goals, the responses of the coactor and the participant could be action-congruent (both move to the high/low card) or action-incongruent (e.g., one moves to the high card while the other moves to the low card).

We used similar 20 combinations of card values in all the conditions, while manipulating the spatial alignment of the target cards (i.e., whether the cards were presented at the same or different side from an egocentric perspective). The total of 104 trials (80 go and 24 no-go) from each of the 4 blocks (i.e., left, right, higher or lower control condition) were randomly drawn from this pool. The block instructions that manipulated movement-control (“Select the left/right card, unless a card with the value “2” appears on the screen”) or action-control (“Select the higher/lower card, unless a card with the value “2” appears on the screen) were presented on the screen for 10 s and were followed by the instruction to place the right index fingers at the start button to begin the block. Block order was counterbalanced between subjects using a pseudo-random sequence that resulted in 16 different combinations of the four blocks. We have excluded the no-go trials and trials in which participants or the coactor selected the incorrect card (1.0% of participants’ responses and 0.0% of the coactor’s responses) and in which response times (RTs) were more than 2.5 standard deviations above the mean (1.1% of participants’ responses and 1.2% of the coactor’s responses). After exclusion of participants´ and coactor´s responses that exceeded 2.5 SDs of each condition, mean response times were submitted to a 2 (control level: action or movement) x 2 (action-congruency: congruent or incongruent) x 2 (movement-congruency: congruent or incongruent) analysis of variance (ANOVA). After the participants completed the experiment, we asked them to fill in two self-report questionnaires. In order to assess the presence of individual empathic traits we used the empathic concern and the perspective taking subscales of the Interpersonal Reactivity Index [44]. To assess the individual ASD-like traits we used the Dutch version of the Autism-Spectrum Quotient [45].

Results

Participants responded faster in the movement-control condition (mean RT = 410 ms, SEM = 21) compared to the action-control condition (mean RT = 602 ms, SEM = 18), F(1,15)= 224.55, p < .001; η² = .94). Overall, we observed a significant action-congruency effect, participants were faster to select a card that had a relative value congruent to the coactor’s card value (mean RT = 503 ms, SEM = 19) as compared to the incongruent one (mean RT = 510 ms, SEM = 19), F(1,15)= 11.91, p = .004; η² = .44. Mean response times in movement-congruent trials (mean RT = 506, SEM = 19) were not significantly different than those in the movement-incongruent trials (mean RT = 506, SEM = 19), F(1,15) = 0.00, p = .99; η² = .00. A significant three-way interaction indicated that movement-congruency was dependent on action-congruency, solely in the movement-control condition (control level × action-congruency × movement-congruency, F(1,15) = 5.89, p = .03; η² = .28, Figure 2 ). To further examine the nature of the three-way interaction effect, we separately analyzed participants’ response times from the action-control and movement-control conditions.

Figure 2. Participants’ mean response times in the action (conceptual)-control and movement (physical)-control conditions as a function of action (conceptual)-congruency and movement (physical)-congruency.

Error bars represent standard errors of the mean (SEM). The asterisks indicate significant differences between conditions, * = p < .05, ** = p < .005.

Response times in movement-control condition

In the movement-control condition, we found a significant two-way interaction between action-congruency and movement-congruency, F (1, 15) = 32.90, p < .001; η² = .69 (see Figure 2 ), indicating an interdependency between movement and action levels of processing. There was a marginally significant effect of action-congruency, F (1, 15) = 3.41, p = .09; η² = .19; and no significant effect of movement-congruency, F (1, 15) = .02, p = .90; η² = .00.

To further specify the interaction effects in the movement-control condition, we ran post-hoc t tests to investigate movement-congruency effects separately for action-congruent and action-incongruent trials. When the chosen cards of the participant and coactor matched in relative value (i.e., their action goals were congruent), congruent movements were executed faster (mean RT = 400 ms, SEM = 21; see Figure 2 ) than incongruent movements (mean RT = 418 ms, SEM = 21), t (1, 15) = 4.50, p < .001, d = -.22. When the relative value of the cards selected by the two actors was incongruent, incongruent movements were executed faster (mean RT = 402 ms, SEM = 22; see Figure 2 ) than congruent movements (mean RT = 422 ms, SEM = 21), t (1, 15) = 2.53, p = .02, d = .23. In other words, when action goals matched, participants moved faster in the same direction as the confederate coactor, but when action goals clashed, movements in the opposite direction from the coactor were executed faster.

Response times in action-control condition

In the action-control condition, participants responded faster in action-congruent trials (mean RT = 597 ms, SEM = 19 ms; see Figure 2 ) than in the action-incongruent trials (mean RT = 608 ms, SEM = 18), F (1, 15) = 6.35, p = .02; η² = .30. We observed neither significant movement-congruency effect, F (1, 15) = 0.02, p = .90; η² = .00, nor an interaction between movement-congruency and action-congruency, F (1, 15) = 2.08, p = .17; η² = .12.

Accuracy scores

The participants showed high accuracy rates throughout the experiment. In the movement-control condition they made on average 0.15 % errors and in the action-control condition their average error rate was 0.11 %. The analysis of participants’ error rates revealed no significant results, all p’s > .1.

Coactor’s response times

To examine whether the RTs of the coactor differed depending on the experimental conditions we ran a 2 (control level: action or movement) × 2 (action-congruency: congruent or incongruent) × 2 (movement-congruency: congruent or incongruent) ANOVA. As expected, the analysis showed no significant main effects, nor interaction effects (all ps > .1). These results indicate that the responses of the coactor were equally fast in all experimental conditions. We found no evidence that the RTs of the coactor influenced the RT effects of the participants.

Correlation analysis between individual ASD-like and empathy trait scores and behavior

We ran a Spearman’s correlation analysis to examine whether individual differences in ASD-like traits and empathy were associated with the degree of individual action-congruency and movement-congruency effects. Individual congruency effects were obtained by calculating the overall differences between individual participants’ incongruent and congruent response times from action and movement congruency conditions. We used the acquired self-report scores from Autism-Spectrum Quotient [AQ, 45] and empathic concern (EC) and perspective taking (PT) subscales of Interpersonal Reactivity Index [IRI, 44] to assess individual personality differences in the tested pool of participants. EC scores [mean = 16.25 (SD = 3.49)] and PT [mean = 16.62 (SD = 4.99)] scores were comparable to the mean Dutch population scores [mean EC = 18.09 (SD = 4.23); mean PT = 17.29 (SD = 4.30)]. Similarly, participants’ AQ scores [mean = 13.50 (SD = 6.85)] were in line with the original control population scores [mean = 17.60 (SD = 6.40)]. As the mean score in the ASD population is 35.80 (SD = 6.50), a score of 32 on the AQ scale is suggested as a useful cutoff score for distinguishing between high-functioning individuals with ASD and controls [45]. All our participants had scores that were below 32 and were thus within the normal range.

Individual action-congruency effects were positively correlated with the empathy scores (r _s (16) = .55, p = .03) and perspective taking scores (r _s (16) = .41, p = .11) derived from the Interpersonal Reactivity Index (IRI), and negatively correlated with the Autism-Spectrum Quotient (AQ), r _s (16) = -.59, p = .02 (Figure 3 ). That is, individuals that scored high on empathy and low on ASD-like traits were more strongly influenced by the conceptual action goals of the coactor. We did not observe any significant correlations between individual movement-congruency effects and either empathy scores, r _s (16) = .05, p = .87, or AQ scores, r _s (16) = -.12, p = .67. Similarly, no significant correlations were observed between the other two subscales of the IRI (fantasy and personal distress; all ps > .2) and either the movement- or action-congruency scores. In sum, the size of action-congruency effects, but not movement-congruency effects was related to the measured personality differences.

Figure 3. Participants’ individual action-congruency scores (left) and movement-congruency scores (right) as a function of their self-reported Autism spectrum quotient scores (AQ; top) and empathic concern scores (IRI; bottom); r values denote spearman’s rho correlation coefficients.

Significant correlations, as indicated by the * were found for action-congruency scores, but not for movement-congruency scores, (p < .05).

Discussion

In social settings, humans constantly observe and expect behavior of other individuals in their surroundings. Here, we show that the spontaneous recognition of other individuals’ behavioral goals is contingent on the behavioral control scheme adopted by the observer during action observation. Specifically, when observers guided their behavior towards physical movement goals, their performance was affected by the expected conceptual action goals and the movement goals of the coactor. When observers guided their behavior based on action goals, their performance was only sensitive to the coactors action goals. These findings are informative regarding the functional action hierarchy that underlies action and movement control of behavior in social interaction. Moreover, we report a relationship between individual differences in the size of action congruency effects and individual differences in empathy and ASD personality traits, while no relationship was found for movement congruency effects.

Shared hierarchy for action recognition and action control

There is a plethora of evidence that supports hierarchical models of neurocognitive processing, but a general characterization of the different components within each distinct functional and neural hierarchy is still missing [16,21,47-50]. Our novel finding, namely, that how observers control their behavior alters the influence of observed behavior on their movements, is consistent with the idea that the multitiered system underlies both action control and action recognition. The data is in line with and builds upon previous work demonstrating that, in situations in which observer’s behavior is directly related to others’ behavior, performance is altered by congruency between observer’s and coactor’s movement goals e.g. [33,35] and action goals [4,43]. Our findings indicate that action goal congruency between the coactors plays an important role in social interaction, irrespective of the level of behavioral control. Movement goal congruency appeared to be only relevant in cases where the observer controlled their behavior at the movement level.

One interesting finding to emerge from the present study is that a mismatch between observers’ and coactors’ action goals resulted in slower response times for congruent movement goals. This reversal of the movement-congruency effect is in line with the recent findings [28,51], which showed that one-to-one associations between observed and executed action can be modulated by the context in which the actions are embedded. Our findings extend previous work by demonstrating that a spontaneous recognition of other individual’s potential action goals (e.g., choosing a higher card of the two) [22,23,52] affects observer’s movement execution and alters the influence of other’s movements on the observer’s movement execution [15,25-28]. One could argue that participants ‘automatically’ took into account the mapping of the numbers, irrespective of movement- or action-congruency. However, we assert that merely visual mapping of the cards cannot explain the reported findings. Spatial mapping of the cards should be regarded as a necessary, but not sufficient, contextual condition for adaptive perceptuo-motor coupling during both action control and action recognition. The interpretation rather requires the assumption that participants have conceptual (relational) model (which of the two cards is higher) of the objects in the environment, which dynamically modulates the perceptuo-motor mapping that underlies the body-world interaction [53].

A probabilistic conceptuo-perceptual framework [53] for action control and action recognition offers a possible explanation for the reversal of the movement-congruency effects. This framework assumes that both control and recognition of action depend on combining prior conceptual information about the relationships between the objects and the immediate perceptuo-motor information regarding movement selection. In the present experiment, incongruent conceptual expectations (i.e. mismatch between the conceptual goals of the actor and coactor) may have facilitated the selection of a movement direction incongruent to that of the coactor, or, alternatively inhibited the selection of a congruent movement. Similar, but opposite facilitatory and inhibitory effects would be predicted in the case that the actor’s and coactor’s conceptual expectations matched (i.e., both going for the low card, or both going for the high card). The conceptuo-perceptual probabilistic framework described here can contribute to a better understanding of the mechanisms that underlie flexible selection and recognition of movements in social situations [28,53,54].

A lot of controversy exists regarding the exact neural instantiation of action recognition [29,37,38,55-58]. Our behavioral findings point to the notion that a dynamic interplay between mentalizing and AON/MNS brain networks might underpin spontaneous recognition of observed everyday behavior [21,41,59,60]. The results of the current study are in line with dynamic multitiered accounts of action representation [17,19,21,61,62] and are consistent with the idea that processing behavior can be viewed as the synthesis of immediate movement information and prior conceptual knowledge. The formalization of appropriate multitiered models and illuminating their neural correlates is critical to understanding guidance of everyday social behaviors [3,21,52,63-65].

Relation between personality traits and hierarchical goal processing

The neurocognitive basis of social impairments in ASD is currently heavily debated in cognitive neurosciences [31,66,67]. Both perceptuo-motor theories [6,7,12,31] and conceptual theories [9,11,39,68,69] have been proposed to explain the social and communicative deficits typically observed in the ASD population. Here, we showed that self-reported autistic characteristics and empathy are related to spontaneous recognition of conceptual action goals, rather than physical movement goals. Studies on the association between the ability to process human movements and ASD have reported mixed results [66,67]. Some researchers found evidence for impaired processing of physical movement goals in ASD [6,7,13,14], whereas others were unable to find such an association, e.g. [ 70,71].

We show a relationship between conditions characterized by social deficits and the ability to process conceptual action goals, which are distinct from the corresponding physical movements [13,25,70,71], but pertain to overarching prior knowledge driving purposeful action [23,52,63-65]. Synthesis of conceptual knowledge that is associated with brain activity in the ‘mentalizing’ regions [72] and the observed movement goals, processed by the parieto-frontal ‘mirroring’ circuit [29,31] is necessary for the inference of others’ object-related conceptual action goals [3,21,41,65,73]. Nevertheless, given the intertwined nature of conceptual and perceptuo-motor representations, the lack of correlation between movement goal recognition and ASD characteristics should not be taken as direct evidence against the perceptuo-motor explanations for impairments in action inference. Rather, the findings raise the interesting possibility that ASD symptoms result from dysfunctional spontaneous integration of observed goal-directed movements with the observer’s contextually activated conceptual knowledge, which might entail the interaction between the cortical networks for “mentalizing” and “action observation” [2,21,41,59,60]. Present results suggest that social learning at a conceptual level, or rather the interaction between conceptual level and observed movements, might be impaired in the ASD population. Adopting the conceptuo-movement hierarchical approach of action processing [53] could lead to novel insights regarding such aberrant social interaction. For example, a key tenet of the conceptuo-perceptual approach, that processing of behavior entails a synthesis of immediate movement information and prior conceptual knowledge, could guide development of new forms of therapy for ASD individuals.

Conclusion

In sum, our findings are consistent with the idea that spontaneous recognition of others’ behavioral goals depends on the observer’s level of behavioral control. We suggest that empathy and ASD-like personality characteristics might be related to an impairment in the synthesis of incoming perceptuo-motor information and prior conceptual knowledge related to functional outcomes in the world. The results also indicate a need to further investigate the roles of parieto-frontal “action observation” and medial-temporal “mentalizing” brain networks that are known for their involvement in processing movement and conceptual information.

Funding Statement

The present study was supported by the NWO Top grant (NWO: 407-11-040). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.