Fronto‐parietal coding of goal‐directed actions performed by artificial agents

Abstract

With advances in technology, artificial agents such as humanoid robots will soon become a part of our daily lives. For safe and intuitive collaboration, it is important to understand the goals behind their motor actions. In humans, this process is mediated by changes in activity in fronto‐parietal brain areas. The extent to which these areas are activated when observing artificial agents indicates the naturalness and easiness of interaction. Previous studies indicated that fronto‐parietal activity does not depend on whether the agent is human or artificial. However, it is unknown whether this activity is modulated by observing grasping (self‐related action) and pointing actions (other‐related action) performed by an artificial agent depending on the action goal. Therefore, we designed an experiment in which subjects observed human and artificial agents perform pointing and grasping actions aimed at two different object categories suggesting different goals. We found a signal increase in the bilateral inferior parietal lobule and the premotor cortex when tool versus food items were pointed to or grasped by both agents, probably reflecting the association of hand actions with the functional use of tools. Our results show that goal attribution engages the fronto‐parietal network not only for observing a human but also a robotic agent for both self‐related and social actions. The debriefing after the experiment has shown that actions of human‐like artificial agents can be perceived as being goal‐directed. Therefore, humans will be able to interact with service robots intuitively in various domains such as education, healthcare, public service, and entertainment.

1. INTRODUCTION

In the near future, nonbiological agents such as humanoid robots will increase their range of activities by not only collaborating with human operators as coworkers in factories and performing household tasks, but will be used as teachers, assistants in shopping malls, receptionists, guides at museums, and nannies/playmates for children (Beran, Ramirez‐Serrano, Vanderkooi, & Kuhn, 2015; Breazeal, Dautenhahn, & Kanda, 2016; Reiser, Jacobs, Arbeiter, Parlitz, & Dautenhahn, 2013; Robinson, MacDonald, & Broadbent, 2014). Robots could also be used to fulfil task‐related purposes in the health care sector, for example, to assist elderly or disabled people in daily tasks and enable them to live more autonomously (Bedaf, Gelderblom, & Witte, 2015; Rabbitt, Kazdin, & Scassellati, 2015). Unlike traditional automation systems used in industrial production, which require strength, precision and speed and need to be isolated from human workers to ensure safety, domestic service robots will be placed in close proximity to humans. For a robot to be effective and safe in an assistive or collaborative role, it is important that the human is able to easily and quickly understand the robot's goals by observing its actions. Ideally, this understanding will come in an intuitive manner, similar to how humans are innately able to communicate their goals and intentions nonverbally. The implicit reading of robot's goals would happen naturally, with no need of specific training or instructions which might be required for the elderly or patients with cognitive impairments.

A number of studies has indicated that during the interaction with biological agents, we understand their goals by internally simulating their actions on the neuronal level (Fogassi et al., 2005; Iacoboni et al., 2005). In monkeys, this simulation could be investigated using single neuron recordings (Rizzolatti, Fogassi, & Gallese, 2001). There are neurons in the monkey premotor cortex which discharge when it executes a specific object‐directed action but also when it observes the same action being performed by an experimenter (Ferrari, Gallese, Rizzolatti, & Fogassi, 2003). Due to their involvement in off‐line internal simulation of the observed action, these neurons are called “mirror neurons” (Rizzolatti and Sinigaglia, 2016). There is a growing body of evidence from noninvasive neurophysiological techniques and brain imaging studies that a similar action observation‐execution matching mechanism based on the mirror neuron system (MNS) is also present in the fronto‐parietal brain regions of humans (Buccino, Binkofski, & Riggio, 2004a; Rizzolatti and Craighero, 2004; Van Overwalle, 2009), although there is some criticism (Hickok, 2009).

There are indications, that when we physically interact with other people, we understand what our collaboration partners are going to do next by unconsciously simulating their actions (Frith and Frith, 2006). The ability to predict actions is especially important in case of service robots which act autonomously (in contrast to industrial robots or transport vehicles such as automobiles) to avoid collisions and injuries. It has also been suggested, our tendency to unconsciously simulate the actions of the collaboration partner is strongly related to the pleasantness of interaction (Carr, Iacoboni, Dubeau, Mazziotta, & Lenzi, 2003; Chartrand and Bargh, 1999), which might be an important factor in the everyday interaction with robotic companions.

Brain imaging makes it possible to investigate specifically the unconscious human responses to observation of robotic actions (Chaminade and Cheng, 2009). For natural interaction with robots, it would be optimal, if we would use similar brain areas to simulate their actions as we use to simulate the actions of other humans (Sciutti, Bisio, & Nori, 2012). Furthermore, we should be able to attribute goals to robotic motor actions using a similar brain network as we use to attribute goals to human actions. The first fMRI study, which compared brain activity triggered by action observation of robotic versus human actions, has suggested that internal simulation of actions is triggered only by biological agents (Tai, Scherfler, Brooks, Sawamoto, & Castiello, 2004). Accordingly, it has also been shown that there is less activity in areas associated with empathy and less emotional distress when people observe abusive behavior directed to a robot in comparison to a human (Rosenthal‐von der Pütten et al., 2014). However, more recent investigations have reported internal action simulation, although weaker than for human actions, for non‐goal‐directed robotic actions (Bisio et al., 2014; Hofree, Urgen, Winkielman, & Saygin, 2015). Some other studies found either similar or even stronger fronto‐parietal activations for robotic versus human grasping actions (Cross et al., 2012; Gazzola, Rizzolatti, Wicker, & Keysers, 2007; Oberman, McCleery, Ramachandran, & Pineda, 2007; Saygin, Chaminade, Ishiguro, Driver, & Frith, 2012). Despite this evidence for the ability to recognize actions performed by artificial agents using a similar brain mechanism (indicating that we understand the movement patterns of a robot), it is not clear whether attribution of various goals to artificial agents modulates activity in similar fronto‐parietal areas as during goal attribution to humans. Investigating this issue is of crucial importance, because discrimination between different goals on the behalf of the observer is a step toward recognizing an artificial agent as being intentional (Hamilton and Grafton, 2006). It can further be considered a building block for the ascription of human properties to the agent, making the interaction with robots more intuitive, easy, and human‐like (Carter, Hodgins, & Rakison, 2011; Chaminade and Cheng, 2009; Gazzola et al., 2007; Oberman et al., 2007).

To answer whether we discriminate goals of robots using the same brain networks we use for differentiating goals of humans, we designed an experimental paradigm based on the one used in studies investigating goal understanding in children (Sodian and Thoermer, 2004; Woodward, 1998) and adults (Hamilton and Grafton, 2006). In this paradigm subjects were presented with an agent who performed grasping or pointing actions directed to two different object‐targets. In our experiment, we used items from categories tools and foods as targets and asked the subjects to guess the possible goals underlying the observed actions.

The choice of categories “tools” and “food items” has been motivated by the fact that the well‐known function of the object‐target suggests a further goal, such as eating in case of food, or using a tool to fix something. Further, grasping tool items presumes a high variety of future actions which require hand precision and therefore a different and more complex pattern of activation in the fronto‐parietal areas than grasping food items, which requires activation of muscle groups located in the head. Additionally, there are a sufficient number of various food items which can be counterbalanced when compared to various tool items. Finally, there are studies which have investigated brain activity related to grasping tools (Creem‐Regehr and Lee, 2005; Valyear, Gallivan, McLean, & Culham, 2012; Vingerhoets, 2014) and food items (Iacoboni et al., 2005; de Lange, Spronk, Willems, Toni, & Bekkering, 2008), so that our results can be compared with findings from previous investigations.

An action may be defined as a sequence of motor acts which are executed one by one to reach the final goal. The two early papers (Gentilucci et al., 1988; Rizzolatti et al., 1988) provided details about the relative selectivity of motor neurons` discharge in monkeys during action execution. Thus, single cell recordings from monkeys show that there are distinct subpopulations of neurons coding subgoals of goal‐directed actions, such as reaching, grasping with the hand, holding, and bringing to the mouth (Gallese, Fadiga, Fogassi, & Rizzolatti, 1996; Rizzolatti et al., 1988). Accordingly, when the monkey grasps a peanut, its reaching neurons are activated first, followed by its grasping neurons, holding neurons, and finally the bringing‐to‐the‐mouth neurons. The different subpopulations of neurons were also activated when the monkey observed the experimenter grasping a piece of food (Umiltà et al., 2001). Thus, an ordinary action such grasping a tool can lead to activation of different sets of neurons, depending on the final action goal (grasp‐to‐use, grasp‐to‐hand over or grasp‐to‐replace, depending on the context (for comparison, see Iacoboni et al., 2005)). Accordingly, it was shown that different cerebral mechanisms are involved in preparing actions based on their desired outcome or based on a required initial goal of action (Majdandžić et al., 2007). Further, activity in the fronto‐parietal areas was modulated by the motoric complexity of the observed actions, suggesting that these areas are responsible for providing a fairly accurate simulation of observed actions by mimicking internally the level of motoric complexity (Molnar‐Szakacs, Kaplan, Greenfield, & Iacoboni, 2006).

Previous studies have investigated how the brain responds when the observers are asked to attribute goals to human grasping actions which are aimed at the same target object but are performed to reach a different outcome based on the context (e.g., drinking from a cup vs cleaning the cup) (Iacoboni et al., 2005). Comparing the brain activity depending on the predicted action outcome showed a different pattern of activation (differential activity) in the fronto‐parietal areas which possess sensory and motor properties: the inferior parietal lobule (IPL), the ventral premotor cortex (PMC), and the inferior frontal gyrus (IFG) (Hamilton and Grafton, 2006; Iacoboni et al., 2005; de Lange et al., 2008). These regions are supposed to be involved in goal prediction by covert und unconscious neural simulation of the observed actions. We therefore hypothesized that in our study the activity in these regions will be modulated by the goal of the action. Further, if we attribute goals to human‐like artificial agents through neural simulation, the activity in these regions should be modulated in the similar way for observation of robotic actions. On the contrary, activations in visual brain areas would indicate that we attribute goals to robotic actions based on their visual properties, without simulating them, like for example in case of observation of a barking dog (Buccino et al., 2004b).

While the influence of agent`s nature on action understanding has been extensively investigated, it is still an open question whether action simulation and therefore goal understanding depends on whether the goal is self‐related like in the case of grasping (the agent intends to act on the object himself), or other‐related, like in the case of pointing. Pointing is a communicative gesture which is used to draw attention of the observer to a certain target. The actor might want to communicate that the observer should act on the object or that he/she intends to act on the object himself. It has been proposed that we are able to simulate observed actions on the neuronal level only if we believe that they belong to the reasonable repertoire of the actor (Buccino et al., 2004b). Thus, although we know that service robots grasp and manipulate objects because they are programmed to do so, we might not necessarily consider a communicative gesture such as pointing to be reasonable for a robotic agent, making it less easy to attribute a goal to it. To answer whether we attribute differential goals not only to self‐related (grasping) but also to other—related social (pointing) robotic actions using similar brain networks as in case of human actions, in addition to grasping we included pointing actions in our design.

2. METHODS

As action simulation is supposed to take place in areas which are active both during action observation and action execution (Rizzolatti and Sinigaglia, 2010), in our study, we focused only on areas with both motor and sensory properties. To localize execution‐related brain areas, prior to the action observation experiment, subjects performed grasping actions in first imaging session and participated in an action observation imaging session months later. The conjunction of the results of the action execution session in combination with action observation served as ROI mask for the results from the action observation session. As the masking was based only on the results of grasping actions, but not pointing actions that were observed in the second imaging session, we additionally conducted a whole‐brain analysis for all results as a control and comparison.

The action execution session always took place prior to the action observation session with an interval of several months. With this design we aimed at decreasing the vividness of motor imagery of the subjects performing the task themselves, which may arise from the memory of having recently performed the action execution session in the MR scanner. Unfortunately, there is no possibility to suppress motor imagery completely.

2.1. Participants

Twenty healthy right‐handed individuals with normal or corrected‐to‐normal vision participated in this study (age range: 21–39 years; mean = 26.6 years; SD = 4.2; 10 females). All participants were recruited from the university and local community. They gave written informed consent and were monetarily compensated for their time. The study was performed in accordance with the Declaration of Helsinki and approved by the ethics committee of the medical faculty of the Ludwig‐Maximilians‐University Munich.

2.2. Action execution session (localizer for motor regions)

2.2.1. Conditions

The action execution session consisted of two action execution conditions and one object observation condition (Figure 1). The control condition was used later for masking (Figure 3). In the action execution conditions (duration: 10 s), the subjects grasped objects belonging to two different categories: food items and tools. In the object observation condition (duration: 10 s), subjects only observed these objects. An additional condition (duration: 10 s), “grasping a cylindrically shaped block”, was used in the analysis of a separate study (Kupferberg et al., 2012), so that altogether there were 4 conditions in the actions execution experiment.

Figure 1.

Experimental design of the action execution (a) and action observation (b) sessions. (a) Each of the 3 runs of the action execution session was composed of 2 blocks for grasping tool (T), food (F), and plastic cylinder (B) separated by an object observation trial (control condition). We employed a block design, in which the subjects had to grasp an object belonging to the same category (tool, T; food, F; or cylindrical block, B) during each block 3 times repeatedly to reduce the cognitive demands caused by frequent task changes. The order of the blocks (T, F, B) was randomized in every run. (b) Each of the 4 runs of the action observation session was composed of 5 blocks of action observation for each of the two agents (the figure depicts ½ of a typical run). The action observation blocks were separated from each other by 5 blocks of static controls. During each action block, 4 action trials (grasping “G” or pointing “P”) of the same agent (human “H” or robot “R”) targeted at objects from one of the two categories (tool “T” or food “F”) were presented in randomized order. During the static agent observation (control), 3 trials were presented. These trials depicted either a static human or static robot either without objects or with objects. Both action observation and static trials lasted 5 s and were separated from each other by 1s grey screen. During action observation, the order of different action types (grasping or pointing) and the object category (tool, food, or block) were counterbalanced over every run. The order of blocks depicting either agent in action and static agents were randomized over every run. Abbreviations: RGT = robot grasping tool, RPF = robot pointing to food; RGB = robot grasping block; RPT = robot pointing to tool; RS = static robot without objects, HSO = static human with objects; HGT = human grasping tool; HGF = human grasping food; HGB = human grasping block; HPF = human pointing to food [Color figure can be viewed at http://wileyonlinelibrary.com]

Figure 3.

The creation of the ROI mask. In the first step of creating the ROI mask, we performed the conjunction analysis between the sessions (a) “action execution” and (b) “action observation.” Before the images for action observation and execution were entered in the conjunction analysis, we subtracted the baselines in each session (baseline with objects and object observation respectively) from them. To restrict our analyses only to fronto‐parietal areas of motor significance, (c) the results of the conjunction were further masked with human anatomical regions of interest (ROIs) which were chosen from the Wake Forest University Pick Atlas: (d) the left/right rostral inferior parietal lobule (IPL), left posterior inferior frontal gyrus (pIFG), and the bilateral ventral premotor cortex (PMC). The resulting “ROI mask” was used as a mask for all future voxelwise analyses. Voxels exceeding a statistical threshold of p < .05 (FDR‐corrected for multiple comparisons) are presented, overlaid on a single‐subject MNI template [Color figure can be viewed at http://wileyonlinelibrary.com]

2.2.2. Temporal structure and design

Each experimental session was segmented into 3 runs each lasting 4.5 min. The structure of a typical run can be seen in Figure 1a.

2.2.3. Stimuli

During the session, subjects were presented with a wooden board with pairs of tools and a food items on each side, and a cylindrical block in the middle (Figure 2a).

Figure 2.

Setup and stimuli. In the action execution session, subjects lying in the scanner could observe the objects fixed to the apparatus (b) through a system of mirrors (a). The apparatus could be rotated by the means of a knob on one side of it. The figures (c–h) depict the examples of screenshots from movies presented to the subjects in the action observation session. These screenshots show the robot and the human performing a grasping action (c, f), a pointing action (d, g), and a static agent observation condition (e, h) [Color figure can be viewed at http://wileyonlinelibrary.com]

Six MRI‐compatible mock tools and six food items were used as stimuli. By using a variety of tools and foods as target objects and therefore counterbalancing we tried to control for differences within each category concerning object affordances, superficial object features such as size, shape, and color, and different finger configurations during grasping.

The object to be grasped in each trial was indicated by the letter “W” (Werkzeug, tool), “E” (Essen, food), or “K” (Kreis, plastic cylindrical block) written on the plastic block. If a fixation cross was depicted on the block, the subjects were to perform the control task, in which they were instructed to observe the objects while fixating on the cross. This condition was later subtracted from the action execution condition.

2.2.4. Setup

During the experiment, subjects lay supine within the magnet and wore headphones to reduce the noise from the scanner. Through a system of mirrors they could view the stimuli and their hand without moving their head (Figure 2b). As illustrated in Figure 2a, the apparatus consisted of a table, to which thin wooden boards (30 × 20 cm) with different object pairs were attached by means of velcro straps. The apparatus was placed approximately 10 cm above the subject's pelvis to locate the objects at a comfortable and natural grasping distance. The middle part of the apparatus (on which the wooden boards were fixed) could be rotated by means of a plastic knob at the side of the table. At the beginning of each new trial, the experimenter (standing at the side of the subject) rotated the apparatus and thus presented the subject with a new pair of objects. The experimenter received auditory instructions when to start a new trial by earphones connected to the computer in the control room. While the subjects grasped the objects, the experimenter removed the old board and attached another board with new objects to the back side of the table.

During grasping, the subjects moved their right arm to the target location and grasped the object, lifted it, and put it back. After returning the object to its place, the subjects placed the hand on the starting position and waited for the next trial to begin. While grasping, the subjects were instructed to think about the usual use of this object. This procedure has been performed for the following reason. The execution of the grasping actions directed at different object categories in the artificial lab environment may make them devoid of the meaning they get in real life. Thinking about the object's function while executing the grasping action should activate the representation of the goal typically associated with using this object in the daily life.

Between grasping movements, subjects held their right hand at the level of their navel and put their right index finger on a response button. The extended left arm was oriented parallel to the trunk in a relaxed position. To minimize head movements, the subject's upper body and head were fixed to the scanner bed by a wide fabric belt and a narrow fabric strap respectively. The right arm of the subjects was also supported by appropriate pillows and restrained by a belt to minimize movements of the arm and hand during force production. This arm belt allowed full motion of the wrist (to grasp and reach any object comfortably), but limited motion at the elbow and the shoulder (however, enough to move the lower arm from the resting position toward the stimuli). The subjects were instructed to start the first grasping action as soon as the experimenter turned the board with objects toward them.

Prior to the scanning session, subjects were extensively trained to grasp tool and food items with as little arm and head movement as possible. After the session, the subjects were debriefed about how easily they could think about the use of the object which they grasped.

2.3. Action observation session

During the action observation session, subjects were asked to carefully observe series of video sequences presenting grasping and pointing actions performed by either a human or a robot. Video sequences were rear‐projected onto a screen positioned in the scanner while the subjects saw the images through a mirror located above their head.

2.3.1. Conditions

The experiment included 4 action observation test conditions (observing grasping tool, pointing to tool, grasping food item, pointing to food item), 1 dynamic control condition (observing grasping a cylindrical block) and 2 static control conditions (observing static agents sitting in front of table with target objects lying on the table/agents sitting at an empty table) for the robot and human respectively. Therefore, there were 14 conditions altogether. The condition “observing the agent sitting at an empty table” was used in a separate study (Kupferberg, 2013). Figure 2c–h depicts examples of grasping, pointing, and static conditions.

2.3.2. Temporal structure and design

The action observation session consisted of 4 runs. Each run lasted 7 min and consisted of 5 blocks of action observation conditions for each agent (grasping and pointing actions, including the dynamic control) and 5 blocks of the static control condition of each agent. The organization of a ½ of a typical run can be seen in Figure 1b. For each agent, there were altogether 16 grasping actions aimed at different food items, 16 grasping actions aimed at tool items and the same amount of pointing actions. In the dynamic control condition, there were 16 grasping actions directed at the plastic cylindrical block. After observation of each pointing and grasping action (during the 1 s grey screen), the subjects had to indicate whether they were able to attribute a goal to each of the actions by pressing the left (goal clear) or right (goal unclear) button. After the static control conditions, the subjects had to press either one of the two buttons randomly.

The dynamic control condition representing the robot and the human grasping a plastic cylinder (this condition was presented during the action blocks) and the static condition representing the human and the robot sitting behind the table (static human/static robot, see Figure 2e,h) were used in percent‐signal change analysis to test for the effect of appearance and kinematics of robotic and human movement. The static control condition depicting the agents sitting behind the table, on which objects from different categories were located (static human with objects/static robot with objects) was used to reveal areas related to action observation while controlling for the possible activity of canonical mirror neurons, which respond to presentation of graspable objects alone (Rajmohan and Mohandas, 2007).

2.3.3. Stimuli and setup

Similar to the action execution experiment, tools and food items (14 different pieces in each category) served as stimuli. During the grasping action, the actor in the video lifted his right hand from the starting position on the table, grasped one of the objects, lifted it, transported it to the middle of the body, and looked at it for 2 s, after which the trial ended. The average timing for each segmented grasping action for the human agent was as follows: (a) turning the head toward the object: 600 ms; (b) reaching for the object: 450 ms; (c) grasping the object: 300 ms; and (d) lifting the object: 700 ms. For the robotic agent, the times were as follows: (a) turning the head toward the object: 500 ms; (b) reaching for the object: 500 ms; (c) grasping the object: 300 ms; and (d) lifting the object: 700 ms. During the pointing condition, the actor pointed at the object with the index finger and returned the hand to the starting position on the table.

The side of the table at which the objects were positioned and the category of the objects (tool and food items) were randomized. The plastic block was always located in the middle between two other objects.

2.3.4. Debriefing on goal attribution

After the session, participants were debriefed on goal attribution to every observed action for both agents. Specifically, they were presented with a picture of every object shown in the session and asked to write down their opinion about “what the respective agent was intending to do” for grasping and pointing actions separately. Further, the subjects were asked about the ease of goal attribution to the robotic in comparison to human agent: “Did you find it equally easy to think of a future action outcome in case of human and robot (yes/no)?” In case of a negative response the subjects had to comment on their answer by naming the agent whose goals were more difficult to understand. Further, the subjects were asked about the naturalness of the robot movement: “Did you find the movement of the robot natural? (yes/no).” Finally, we asked the participants about the ease of goal attribution to pointing in comparison to grasping actions: “Did you find it equally easy to think of a future action outcome in case of pointing and grasping?” Again, if a negative response was given, the subjects had to comment on their answer by naming the action in which the goal attribution seemed more difficult.

2.4. Data acquisition

All BOLD‐sensitive (blood‐oxygen‐level‐dependent) fMRI images were acquired on a 3 T whole‐body scanner (GE Signa HDx) using a standard echo‐planar imaging (EPI) sequence and an 8‐channel radiofrequency (RF) head coil for signal reception. The sequence had an echo time (TE) of 60 ms, matrix: 96 × 96; field of view (FOV): 220 mm, and a voxel size of 2.3 × 2.3 × 3.5 with no gap. All slices were oriented parallel to the anterior–posterior commissure. In the action execution session, there were 37 interleaved slices with a repetition time (TR) of 2.25, and in the action observation session, there were 39 interleaved slices with a TR of 2.34. During each experimental session, a T1‐weighted anatomic reference volume was acquired using a 3D acquisition sequence.

2.5. Data analysis

2.5.1. Data preprocessing

Behavioral data (button presses during the action observation session) were analyzed using MATLAB. Image analysis was performed using MATLAB (Mathworks Inc., Natick, MA) and SPM5 (Wellcome Department of Cognitive Neurology, University College London). During preprocessing, images of action execution and action observation sessions were corrected for head movements by alignment to the mean image across all runs. The data for each subject were coregistered to the individual anatomical image and then segmented into MNI standard coordinate space. The data were also smoothed by a Gaussian filter (8 mm FWHM).

2.5.2. First level analysis

To analyze the activations in the action execution and action observation sessions, the regressors of interest for the all conditions (Figure 1) were entered into a general linear model (GLM) as boxcar functions convolved with the canonical hemodynamic response function. Head motion parameters were included in the model as covariates of no interest. We normalized the global signal intensity and applied an appropriate high‐pass temporal filter to remove low‐frequency drifts independent of the stimulus‐induced signal changes in the action observation session. The high‐pass filter was calculated as two times the stimulus period (the longest time duration in seconds before the same condition was repeated: 143 s). In the action execution session, the control condition was not explicitly modeled. In the action observation experiment, all control conditions were modeled (Figure 2e,f,i,j), as we had 3 control conditions (2 static and 1 dynamic) for every agent. Images of parameter estimates for the contrasts of interest were created for each subject.

Individual contrast images from the first level analysis of the action observation session were entered into 2 different group level GLMs to test for the effects of action goal (eating/using a tool), action type (self‐related goal in grasping/social goal in pointing), and agent (human/robot). The first model contained the factors agent, action type. and goal and was aimed at investigating how the activity in the fronto‐parietal areas is modulated depending on the action goal, nature of the agent and the action type (Model 1). To investigate whether the agent‐related difference in brain activity was based on the superficial difference in their appearance or emerges only during movement, a second sets of contrasts with factors agent (human/robot) and state (static control/dynamic control, see Section 2.3.1) was also performed (Model 2).

2.5.3. Masking with ROIs and whole‐brain analysis

As related fronto‐parietal areas of motor significance are supposed to be active both during action observation and action execution, a conjunction analysis using the conjunction null (Nichols, Brett, Andersson, Wager, & Poline, 2005) was used to assess activation common to the execution and observation of grasping (anatomical overlaps, see Figure 3). To restrict our analyses only to fronto‐parietal areas, the results of the conjunction analysis were further masked with human anatomical regions of interest (ROIs). These ROIs were chosen from the Wake Forest University Pick Atlas (Maldjian, Laurienti, Kraft, & Burdette, 2003): the left/right rostral inferior parietal lobule (IPL), left posterior inferior frontal gyrus (pIFG), and the bilateral ventral premotor cortex (PMC). The resulting “ROI mask” was used as a mask for all voxelwise analyses (ROI‐analysis). The cluster size and the anatomical description of each region are depicted in Table 1.

Table 1.

Significant cluster activations for the ROIs

ROI	Region	Hemi	Cluster size	p (FWE)	p (FDR)	Z score	x	y	z
IPL	AIP	L	514	.000	.000	5.36	−45	45	−27
	AIP	L		.000	.000	5.27	−57	51	−27
	SMG	L		.002	.000	4.25	−30	36	−63
IPL	SPL	R	95	.000	.000	5.84	−39	51	30
	SPL	R		.000	.000	5.39	−48	51	27
	SI	R		.000	.000	4.86	−36	42	33
IFG	BA 45	L	58	.008	.007	3.42	12	0	−45
	BA 44	L		.011	.007	3.33	9	27	−57
	BA 44	L		.017	.007	3.19	15	9	−39
PMC	BA 6	L	293	.000	.000	5.67	9	48	−3
	BA6	L		.000	.000	5.59	0	60	−24
	BA 6	L		.000	.000	5.34	−6	48	−27
	BA 6	R	197	.000	.000	5.39	−9	48	24
	BA 6	R		.000	.000	5.31	−9	57	27
	BA 6	R		.000	.000	4.84	−3	48	39
	BA 6	R	59	.000	.000	4.92	12	48	3

Note. Abbreviations: FWE = familywise error rate; FDR = false discovery rate; IPL = inferior parietal lobule; AIP = anterior intraparietal sulcus; SMG = supramarginal gyrus, SPL = superior parietal lobe; SI = primary somatosensory cortex; BA = Brodman area; IFG = inferior frontal gyrus. The locations of the regions were determined using the Juelich anatomical atlas implemented in the FSL viewer. Coordinates are listed in MNI atlas space.

Apart from the ROI analysis we performed a whole‐brain analysis without the masking procedure for all described contrasts. This analysis is described in more detail in the supplemental information.

2.5.4. Calculating percent‐signal change in ROIs and ANOVAs

To investigate the effect of goal, action type and agent during action observation across entire anatomical brain regions, we calculated the mean percent signal change each ROI (right IPL, left IPL, PMC, and left IFG) for each condition and subject. To this end we compared the mean parameter estimates in these regions to the mean intensity over all voxels in the brain as a measure of percent signal change. Individual mean percent signal change values for each ROI in each condition were entered in 2 different repeated‐measures ANOVAs.

Based on Model 1 (Section 2.5.2), the first ANOVA (ANOVA I) was performed on the factors agent (human/robot), action type (grasping/pointing) and goal (tool/food). Further, based on Model 2, the second ANOVA (ANOVA II) was performed on the factors state (static control condition/dynamic control condition), and agent (human/robot). We used the FWE‐correction based on random field theory (Worsley et al., 1996). To keep the family‐wise error rate of these two ANOVAs at 0.05, we corrected for the multiple comparisons by dividing the significance level of 0.05 by 2, resulting in a new corrected significance level of p = .025.

3. RESULTS

3.1. Behavioral results from action observation

The analysis of button presses during the action observation session showed that subjects were able to attribute a certain goal to the majority of pointing and grasping actions (human: 93% of all test trials; robot: 91% of all trials, see Figure 4a). Overall, the reaction time for being able to attribute a goal to an action (513.0 ± 98.2 ms) was significantly (p = .004) larger than for not being able to attribute a goal (387.5 ± 131.0 ms).

Figure 4.

Behavioral results. (a) Subjective ability to attribute a certain goal to human and robot during observation of grasping and pointing. (b) Difficulty level in attributing goals to human (H) versus robotic (R) agent. (c) Goals attributed to robotic agent for grasping food. (d) Subjective perception of the naturalness of robot movement

The subjective reports indicated that for 50% of the subjects it was more difficult to attribute a goal to grasping and pointing actions of the robot directed to food items in comparison to the human agent (Figure 4b). Further, 35% of the participants found it more difficult to attribute goals to the robot grasping and pointing to the tool items. However, McNemar's mid‐p test for paired nominal data showed no significant difference for goal attribution to tool versus food items [p = .125]. Finally, 8 subjects (40% of all subjects) reported that the movements of the robot seemed more unnatural than human movements (Figure 4d).

Further, the debriefing of the subjects has shown that the goal attributed to the robot and the human during grasping tools, but not food items, was identical for both agents for every subject. In case of the tool items, the attributed goal was appropriate to the tool`s purpose (e.g., fixing something, repairing something, measuring something, gluing parts together). When observing the human agent grasping a food item, the attributed goal was “eating” for all subjects. When observing the robot, the attributed goal in case of grasping food was either “giving/offering the food item to another agent” (13 subjects, 65%; p < .05) or “eating” (7 subjects, 35%; p < .05), even though eating is not a meaningful action for an artificial agent (Figure 4c).

3.2. Functional imaging results from action execution

To restrict our analyses to fronto‐parietal areas with motor and sensory properties, we first identified the brain areas involved in execution of grasping actions (motor localizer on the whole brain level). To this end, we created the contrast (referred to as action execution contrast) from the conditions “grasping food and tool items” and “observation of tool and food items.” This contrast revealed activations in the bilateral IFG, the premotor cortex (PMC), the motor cortex, the IPL (including the AIP), the primary somatosensory cortex (SI), the secondary somatosensory cortex (SII), the superior frontal gyrus (SFG), the middle frontal gyrus (MFG), the middle temporal gyrus (MTG), the inferior temporal gyrus (ITG), the lateral occipital cortex (LOC), insula, cerebellum, lingual gyrus, V1, V3, and V4 (for locations of these areas, see Figure 3a).

3.3. Functional imaging results from action observation

To display the activations for observation of grasping we created the contrast (referred to as action observation contrast) from the conditions “observation of a human agent grasping foods and tools” and “observation of static human agent with tool and food items located on the table” (control). This contrast resulted in the activation in the bilateral premotor cortex, the IPL (including the AIP), the superior parietal lobe (SPL), SFG, ITG, LOC, insula, lingual gyrus, the left IFG (pars opercularis and pars triangularis), the left angular gyrus, the left MFG, and the right superior temporal sulcus (STS) (Figure 3b).

3.4. Conjunction analysis of action execution and action observation

For the conjunction analysis needed for masking the results, brain activation during grasping foods and tools was compared to the activation triggered by observation of a human agent grasping foods and tools (Figure 3).

The conjunction analysis of observation of grasping and execution of grasping (after subtracting the control conditions as described above) activated the bilateral SFG, the PMC, the IPL including the AIP, ITG, LOC, lingual gyrus, SI, SII, and temporo‐occipital fusiform gyrus (Figure 3). Further activations on the right side were located in STG and parahippocampus and on the left side in the frontal lobe, the supramarginal gyrus (SMG), the pIFG (pars opercularis, pars triangularis), angular gyrus, insula, the MFG, paracingulate, and cingulate gyri.

3.5. Main effects of goal, action type and agent

Both the whole‐brain analysis and the ROI analysis (to which we refer to as “masked”) based on Model 1 and Model 2 have shown similar activations in the fronto‐parietal areas including IPL, IFG and PMC for the effects of goal, action type and agent (Figures 5 and 6 and Supporting Information, Figure 2).

Figure 5.

Modulation of the fronto‐parietal activity by attribution of different self‐referenced (grasping) and other‐referenced (pointing) goals. (a) The activations (after masking) for actions directed at tools versus actions directed at food items were stronger in the bilateral IPL and PMC. (b) Mean percent signal change in the right IPL, bilateral PMC, left IPL and left IFG (averaged over 20 subjects) during observation of actions directed at tools and foods performed by the human (dark grey) and by the robot (light grey). Error bars represent the standard error of the mean [Color figure can be viewed at http://wileyonlinelibrary.com]

Figure 6.

Modulation of fronto‐parietal activity by observation of action type and agent. Activation for observation of (a) “grasping minus pointing” and (b) “robot minus human” were stronger in the bilateral IPL and PMC. All displayed activations are masked by the ROI mask. (c) Mean percent signal change (averaged over 20 subjects) during observation of grasping and pointing actions of robot (light grey) and human (dark grey) directed at both tools and food items in the right IPL, PMC, left IPL, and left pIFG. Error bars represent the standard error of the mean [Color figure can be viewed at http://wileyonlinelibrary.com]

In the following sections, the results of the ROI analysis are considered. Voxels exceeding a statistical threshold of p < 0.05 (FDR‐corrected for multiple comparisons using the Benjamini & Hochberg procedure; Chumbley and Friston, 2009) are presented, overlaid on a single‐subject MNI template. The nomenclature of anatomical structures located outside the ROIs follows the Harvard–Oxford structural atlas and the Jülich histological atlas (Eickhoff et al., 2007).

3.5.1. Discrimination between different action goals

The percent‐signal change ANOVA I revealed a main effect of goal in the left IPL [F(1,19) = 9.41; p = .006], right IPL [F(1,19) = 41.51; p < .001] and in the bilateral PMC [F(1,19) = 10.91; p = .004] (Figure 5b and Table 2). Activations in the ROIs for the contrast “grasping/pointing directed to tools–grasping/pointing directed to foods” are depicted in Figure 5a. The whole‐brain analysis of the same contrast has shown very similar fronto‐parietal activations and additionally in visual areas (Supporting Information, Figure 2e). The contrast “grasping/pointing to food–grasping/pointing to tools” did not show any activations after masking, but there were activations in the visual areas on the whole brain level (Supporting Information, Table 1).

Table 2.

Main effects and interactions

	Left IFG	Right IPL	Left IPL	Bilateral PMC
Main effects
Goal	F(1,19) = 1.18 p = .29	F(1,19) = 41.51 p < .001*	F(1,19) = 9.41 p < .006*	F(1,19) = 10.91 p < .004*
Action type	F(1,19) = 5.11 p = .36	F(1,19) = 83.31 p < .001*	F(1,19) = 42.81 p < .001*	F(1,19) = 13.99 p < .001*
Agent	F(1,19) = 0.72 p = .41	F(1,19) = 9.08 p < .007*	F(1,19) = 5.21 p < .034*	F(1,19) = 1.71 p < .21
Interactions
Action type x Goal	F(1,19) = 0.28 p = .33	F(1,19) = 0.13 p =2.54	F(1,19) = 1.48 p = .24	F(1,19) = 1.06 p = .32
Agent x Goal	F(1,19) = 0.01 p = .94	F(1,19) = 0.32 p = .57	F(1,19) = 0.016 p = .90	F(1,19) = 1.65 p = .21
Action type x Agent	F(1,19) = 0.38 p = .85	F(1,19) = 8.40 p = .009*	F(1,19) = 6.32 p = .021*	F(1,19) = 2.61 p = .12
Action type x Goal x Agent	F(1,19) = 2.96 p = .102	F(1,19) = 0.63 p = .44	F(1,19) = 0.14 p = .72	F(1,19) = 0.01 p = .91

Main effects and interactions from the ANOVA I (goal × action type × agent). Significant results are marked with an asterisk (*).

There were no significant interactions of agent × goal or action type × goal in any of the ROIs, suggesting that discrimination of the action goal was independent from the agent and took place both in actions with self‐related and other‐related goal.

In case of the robot grasping a food item, the goal of “using” the food item according to its well‐known purpose has been attributed to the robot only by 7 of 20 subjects (p > .05), even though eating is not a meaningful action for a robot. To investigate the difference between the group of subjects who either attributed the goal of “giving the food item to someone” or the goal of “eating” to the robot, we performed a t test (in SPM) using contrast images for observation of grasping food items. However, this t test revealed no difference in brain activation depending on the attributed goal (whole‐brain analysis, FDR corrected). We assume that the number of subjects was not sufficient to generate significant results.

3.5.2. Discrimination between different action types

The ANOVA I revealed increased activations for grasping versus pointing in the left [F(1,19) = 42.81; p < .001] and right IPL [F(1,19) = 83.31; p < .001] and in the bilateral PMC [F(1,19) = 13.99; p = .001] (Table 2 and Figure 6a,c). No brain areas were activated stronger for pointing than for grasping after masking. On the whole‐brain level, there were activations in the middle temporal gyrus and visual areas for pointing versus grasping (Supporting Information, Table 2).

Debriefing after the experiment has shown that 50% of the subjects interpreted the pointing gesture as the goal of the agent to communicate to the observer that he is going to use the object himself while other 50% interpreted the gesture as a request directed to the observer to grasp the object. We performed a t test (in SPM) on the contrast images of “pointing to tool & pointing to food items for both human and robotic agent” between two subject groups: those who regarded the pointing action as the agent's goal to direct the observer's attention to the object versus those who interpreted pointing as the agent's desire to grasp the object himself. This t test did not reveal any significant activations (FDR corrected), perhaps due to low statistical power.

3.5.3. Discriminating between different agents

The comparison of observation of robotic versus human actions revealed stronger activations for the robotic actions in the left IPL [F(1,19) = 5.21; p = .034] and right IPL [F(1,19) = 9.08; p = .007] (Figure 6b and Table 2).

Further, ANOVA I has shown an interaction effect of action type × agent in the right IPL [F(1,19) = 8.40; p = .009] and left IPL [F(1,19) = 6.32; p = .021], indicating that the activation for observation of the human and robotic agent was different depending on whether the agent executed pointing or grasping. To investigate this further, we performed two posthoc ANOVAS: one for the action of grasping and one for the action of pointing. The first post‐hoc ANOVA with factors agent and goal performed for grasping and has shown an effect of agent in the right IPL [F(1,19) = 14.96; p < .001] and left IPL [F(1,19) = 13.74; p < .001]. The second post‐hoc ANOVA using the factors agent and goal for pointing has revealed no significant effects. This result indicates that the increase in activity associated with observing the robot was specific only for grasping actions.

Additionally, ANOVA II, which included the factors state (static control condition/dynamic control condition), and agent (human/robot), has shown the main effect of agent in the left IPL [F(1,19) = 13.51; p = .002] and right IPL [F(1,19) = 6.82; p = .017] and the main effect of state in IPL and PMC [left IPL F(1,19) = 16.11; p = .001; right IPL F(1,19) = 18.73; p < .001; bilateral PMC F(1,19) = 19.01; p < .001]. There was no effect of agent in the left posterior IFG. The interaction agent × state was significant in the left IPL [F(1,19) = 8.4; p = .009] and bilateral PMC [F(1,19) = 8.9; p = .008], indicating that the agent‐related activation depended on whether the agent was in a static (not moving) or a dynamic (grasping a cylindrical block, see Section 2.3.1). To further investigate this interaction, we compared the percent signal change values in left IPL and bilateral PMC between the robot and the human in the dynamic control condition and in the static control conditions, respectively (paired t test). We found increased activation for the observation of the robot as compared to the human agent in the left IPL [t(19) = 3.7; p = .02] and PMC [t(19) = 2.2; p = .04] in the dynamic control condition. However, comparison of brain activation during observation of static videos depicting the robot versus the human did not show any difference. This result indicates that the increase of brain activity in the ROIs when observing the robot in comparison to the human agent was not caused by its nonbiological appearance but by its pattern of movement.

3.6. Summary of the main findings

The main aim of this study was to assess whether fronto‐parietal activity during observation of robot's target‐directed actions such as grasping (self‐related goal) and pointing (other‐related goal) is modulated by attributing differential goals depending on the functional use of the target object. The comparison of actions directed to tools versus food items showed activations in the bilateral IPL and PMC (results masked by ROI‐mask) in both grasping and pointing and in both human and robotic agent (Figure 5a,b). We therefore suggest that we use the same brain networks to attribute self‐related and social goals to observed robotic actions as in case of human action. To our knowledge, no previous study ever reported this finding.

We further investigated whether we can differentiate between grasping and pointing actions of humans using similar brain mechanisms as for differentiating human actions. For both agents, we have shown a bilateral signal increase in the IPL and PMC when masked with the ROI‐mask for grasping versus pointing (Figure 6a,c). The whole‐brain analysis showed a broader activation including bilateral IPL, PMC, and visual areas for grasping versus pointing (Supporting Information, Figure 2a). For pointing versus grasping, there were small activations in the visual areas (Supporting Information, Figure 2b).

When observing grasping actions, the activation was stronger in bilateral IPL for the robot in comparison to the human agent (Figure 6b,c). However, we found no difference for robot versus human when we compared the brain activity elicited by observation of their pointing actions (see Section 3.5.3, second paragraph). Further, agent‐dependent brain activation in PMC and left IPL was absent for observation of static images of the agents indicating that it was not merely the difference of their appearance which caused it.

4. DISCUSSION AND CONCLUSIONS

4.1. Modulation of fronto‐parietal brain activity by the action goal

This study was conducted to investigate whether observation of actions of an inanimate agent directed to different object categories while attributing goals to them leads to neural activations in the same fronto‐parietal areas brain regions as in case of a human agent. We have used two object categories: tool and food items.

We found increased activation in the same parts of bilateral IPL and PMC of subjects independent of the agent while observing grasping tool versus food items and pointing to food versus tool items. This indicates that attribution of different goals to the robot takes place through motor simulation on the part of the observer. It is therefore not based solely on visual system like it may happen if the action cannot be simulated in case of nonconspecifics (Buccino et al., 2004b). Our findings demonstrate the possibility of easy and intuitive interaction with humanoid robots in the future.

Our results are consistent with behavioral studies which demonstrate automatic imitation (Press, Bird, Flach, & Heyes, 2005, 2006, 2007) and motor interference (Kupferberg et al., 2012) during action observation of artificial agents. Also single unit data in monkeys showed that fronto‐parietal brain areas discriminate between goals (grasp to eat or grasp to replace), but there is little evidence that they discriminate between agents (Bonini et al., 2010; Fogassi et al., 2005).

In contrast to previous studies (Hamilton and Grafton, 2006; Ramsey and Hamilton, 2010), the action goal was not defined by the shape of the target, which requires a certain grasp configuration. Rather, we were interested in the possible future outcome of grasping an object from a certain object category, based on the range of action possibilities which this object is associated with (Humphreys, 2001). This is important since in most daily life situations, grasping an object is an initial component of a broader intentional behavior, in which the object is likely to be used to achieve a subsequent final outcome. We suggest that potential action streams such as “eating something,” “using a tool to fix something,” “giving an item to someone else,” or “pointing to an object to direct the observer” can be considered a generic form of goal attribution. We further suggest that this is reflected in the goal‐specific activity within the fronto‐parietal network (Fogassi et al., 2005; Iacoboni et al., 2005; Umiltà et al., 2001). Consistent with that, the increased activation in the anterior intraparietal sulcus (AIP) and supramarginal gyrus (SMG) for observation of grasping tools versus food items is consistent with the involvement of these regions in the planning of skillful use of tools (Binkofski et al., 1998; Valyear, Cavina‐Pratesi, Stiglick, & Culham, 2007). Further, the AIP is responsible for extracting affordance information from the visual stream and can function as an active memory in which a set of affordances are updated as the plan of action unfolds (Fagg and Arbib, 1998; Oztop and Arbib, 2002).

The anterior SMG is suggested to code tool actions in terms of causal relationships between the intended use of the tool and the results obtained by using it (Peeters et al., 2009). The activations in the IPL might also represent imagined object manipulation during tool use (Hermsdörfer, Terlinden, Mühlau, Goldenberg, & Wohlschläger, 2007; Moll et al., 2000). This interpretation is consistent with the behavioral data, showing that observation of grasping and pointing to a tool led to an expectation on the part of the observer that the agent will use it according to its well‐known function in the future stream of actions. The internal simulation of robotic actions (Gallese and Goldman, 1998; Rizzolatti and Craighero, 2004) allows the subjects to predict the future outcome of it as if they were in the place of the observed agent and, consequently, to fairly automatically attribute goals to it.

Our results have shown that we attribute not only self‐related (grasping) but also other‐related social goals (pointing) to robotic actions using similar brain networks as for human actions. This indicates internal coding of attributed goal to a communicative (social) action depending on action target. This result might seem surprising, as previous studies failed to demonstrate the attribution of social intention to a robotic hand (Castiello, 2003; Sartori, Becchio, Bulgheroni, & Castiello, 2009) or failed to show simulation of a communicative action produced by a robotic hand (Liepelt, Prinz, & Brass, 2010). Further, a sudden social request from a robotic agent had no influence on the kinematics of a preplanned action (Sartori et al., 2009). Yet, we do not consider our findings in contrast with the previous ones, since we used an anthropomorphic human like robot instead of a robotic hand. It seems that when we see a humanly shaped agent, it is very difficult not to have the “feeling” of understanding of its goal.

4.2. Modulation of fronto‐parietal brain activity by the action type

The observation of grasping, regardless of the agent, activated bilateral IPL and PMC stronger than pointing (Figure 6a). This indicates that we differentiate between different action types of both artificial and human agents using the same brain networks.

The anatomical location of activations is in line with previous studies, which showed increase of the IPL activity for executed (Frey, Vinton, Norlund, & Grafton, 2005) or observed (Grafton, Arbib, Fadiga, & Rizzolatti, 1996; Pierno et al., 2009) grasping versus observed pointing actions. Activations in these areas of motor significance might be explained by increased demands on motor control during internal simulation of observed grasping requiring precise finger coordination during the grip and lifting. Similar to observation of grasping, also execution of grasping leads to a higher activation of IPL than reaching or pointing movements (Culham et al., 2003; Gallivan, McLean, Smith, & Culham, 2011).

4.3. Modulation of fronto‐parietal brain activity by different agents

We found increased activity in the fronto‐parietal areas responding to observation of robotic grasping actions but no difference between human and robot neither for the static pictures of agents nor for pointing actions. The increased activity for observing the robot grasp might be caused by its unusual hand movement trajectory (for details see supplementary methods).

The predictive coding framework (Jakobs et al., 2009; Neal and Kilner, 2010; Rao and Ballard, 1999) claims that when we observe agents, we generate expectations about how they might move based on our own motor system, the shape of the agent's body (Buccino et al., 2004b), the goal of the action (Gallese and Goldman, 1998) or/and environmental context (Liepelt et al., 2010). Any deviation of sensory input from this prediction might lead to a higher prediction error observed as signal increase (Friston, 2010). Indeed, the robot's grasping trajectory, but not pointing trajectory, was perceived as being unnatural by 40% of the subjects (Figure 4d). This was due to the robot's joint configuration, which required grasping the objects from above rather than from the side, as humans would usually do it. Increased IPL activity for observation of unnatural robotic movement is consistent with findings which show increased fronto‐parietal activity for observation of humanly impossible movements (Costantini et al., 2005; Romani, Cesari, Urgesi, Facchini, & Aglioti, 2005), movements performed by robots (Gazzola et al., 2007; Oberman et al., 2007) or robot‐like human motion (Cross et al., 2012). In case of pointing, robot's hand trajectory did not cause increased activation, as it is natural to point to an object from above.

The whole‐brain analysis has shown an increased activity in posterior superior temporal sulcus (pSTS), a region predominantly involved in the processing of biological motion (Thompson, Clarke, Stewart, & Puce, 2005), for human versus robotic actions (Supporting Information, Figure 1d). This activation can be explained by the fact that robotic movements looked less biological than human movements. It has also been shown that the extent to which the observer considers the agent to be intelligent can modulate the activation in this area (Takahashi et al., 2014).

4.4. Present results in light of other studies

Our study has a number of advantages in comparison to other studies which examined how we process goal‐directed actions of artificial agents on the neuronal level. First, no studies have investigated goal attribution to pointing actions performed by a robot. Additionally, most of these studies have not used an action execution control experiment for localizing brain regions of motor significance (except for Gazzola et al., 2007). Finally, and most important, no studies have tested which brain regions are involved in goal attribution to robotic actions directed to two different object categories. Instead, they directly compared observation of grasping actions directed to various objects performed by a human to those performed by artificial agents. Most of these studies demonstrated an increased activity in the IPL and PMC when observing robotic motion compared to natural human motion (Cross et al., 2012; Gazzola et al., 2007; Oberman et al., 2007; Saygin et al., 2012). The deactivation in the fronto‐parietal network while observing robotic actions as seen in Miura et al. (2010) and Tai et al. (2004) might be due to a highly unnatural robot configuration and movement decreasing the possibility of action simulation on the part of the observer (Kupferberg et al., 2012).

4.5. Possible implications for social robotics and neuroscience

Taken together, this evidence indicates that, as far as simple collaborative behaviors are concerned, actions of a humanoid robot are processed similarly to human actions and trigger a similar response in the human partners. In line with it, a recent electroencephalography study has shown that we can empathize with humanoid robots using similar neural networks as we use to empathize with other humans (Suzuki, Galli, Ikeda, Itakura, & Kitazaki, 2015).

Social robotics aims at developing robots that will assist humans in their daily lives, making the need for understanding their goals a crucial aspect for their development. The fact that the same modulations of fronto‐parietal activity depending on the action goal took place both during observation of human and robotic actions suggests that the internal simulation of the observed action does not depend on whether we categorize the agent as animate. Further, in accordance with Bisio et al. (2014), we demonstrated that the agent does not need to exhibit a biological motion trajectory for goal attribution on the part of the observer. Thus, similar to earlier studies (Gazzola et al., 2007), our results suggest that interactions with robots will be likely based on the same basic biological and behavioral implicit mechanisms on which human interactions are based. Therefore, in the future, there is potential for robots to be used in a variety of scenarios such as personal assistance duties (Young, Hawkins, Sharlin, & Igarashi, 2008), education and therapy of children with autism spectrum disorder (Robins, Dautenhahn, Te Boekhorst, & Billard, 2005), elderly care, assistance in independent living (Tapus, Tapus, & Mataric, 2009), neurorehabilitation and physiotherapy (Obo, Loo, & Kubota, 2015). In the field of neuroscience, robots can be used to investigate basic aspects of motor interaction, as using artificial agents guarantees full control on timing, movement trajectory and repeatability (Sciutti et al., 2013).

4.6. Limitations and outlook

When it comes to goal coding in the fronto‐parietal regions, it might be possible that higher brain activity for grasping tools versus food items is caused not by different goals, but by retrieving different object‐related information. However, two arguments speak against this explanation. First, objects from both categories were visible in every test condition and control conditions were subtracted from both action execution and action observation sessions prior to creating the conjunction mask. Second, as we chose tool and food items with a different size and shape, the possibility that fronto‐parietal activity was caused by different physical and visual features of objects is unlikely. An earlier study has shown that mere observation of tools versus other graspable objects led to activations in the left fusiform gyrus and not fronto‐parietal areas (Creem‐Regehr and Lee, 2005).

A further limitation of our study is the fact that in the motor localizer (action execution session) we used only grasping, but not pointing actions. The masking procedure thus biases the statistics in favor of grasping to pointing. However, studies have shown that grasping leads to a higher activation than pointing mainly in parietal brain areas (Culham et al., 2003; Gallivan et al., 2011; Pierno et al., 2009). Further, the whole‐brain analysis without masking procedure revealed that the contrast “pointing‐grasping” showed activations only in the middle temporal gyrus and visual areas (Supporting Information, Figure 2b).

Further, the instruction to guess the agents' goals could have biased the participants to believe that both actors have pursued certain goals. However, we tried to reduce this bias by informing the participants that goal attribution to every object and to every agent was not expected and that after every trial they had a choice between two buttons for successful goal attribution versus inability to attribute a goal. Indeed, both in case of the robot and the human, there were some trials in which the participants could not attribute a goal to the observed action (Figure 4a). Also, goals attributed to human and robotic actions were different in case of grasping food items. Indeed, only one‐third of subjects attributed the goal of “eating” to the robot (as it was always the case for the human action) and two‐third of the subjects attributed the goal of giving/offering the food to someone else. It is also plausible that the actual perception of goals while the action unfolds and self‐reports on perceived goals may somewhat dissociate. One might “feel” like the robot intends to eat while watching it grasping a banana, and yet respond “no” to the subsequent, direct question on whether one felt that the robot wanted to eat. The reflective nature of answering the question may have led subjects to “rethink” and respond negatively to attributing goals to robots.

While the robot used in this study was clearly not a living organism, it was similar in overall form to a human. Therefore, humans could easily match their own bodily configuration with that of the robot. Future studies will need to address this issue in a more detailed way by using different forms of animate and inanimate motion and appearance.

We thought about using a repetition suppression design for the experiment but decided against it for two reasons. First, the electrophysiological evidence for adaptation in the action observation (or mirror neuron) system is relatively inconclusive, and the fMRI studies using repetition suppression have yielded mixed results (Kilner and Lemon, 2013). Second, and more importantly, we had a large number of different conditions in the action observation experiment that we were interested in comparing. This made performing the experiment as a repetition suppression experiment challenging because we would have needed to use an event‐related design and therefore increased our number of trials to reach a comparable statistical power. Therefore, we opted for the block‐design instead, focusing on comparisons between our different experimental manipulations and maximizing the power for these comparisons. However, we believe that in future studies, the habituation paradigm as used in Hamilton and Grafton (2006) and Majdandzic, Bekkering, van Schie, and Toni (2009) would be a great way to increase the resolution of studying shared activation.

Supporting information

Additional Supporting Information may be found online in the supporting information tab for this article.

Supporting Information

Kupferberg A, Iacoboni M, Flanagin V, et al. Fronto‐parietal coding of goal‐directed actions performed by artificial agents. Hum Brain Mapp. 2018;39:1145–1162. 10.1002/hbm.23905