Human Cognition in Interaction With Robots: Taking the Robot’s Perspective Into Account

Abstract

Objective

The present study investigated whether and how different human–robot interactions in a physically shared workspace influenced human stimulus–response (SR) relationships.

Background

Human work is increasingly performed in interaction with advanced robots. Since human–robot interaction often takes place in physical proximity, it is crucial to investigate the effects of the robot on human cognition.

Method

In two experiments, we compared conditions in which humans interacted with a robot that they either remotely controlled or monitored under otherwise comparable conditions in the same shared workspace. The cognitive extent to which the participants took the robot’s perspective served as a dependent variable and was evaluated with a SR compatibility task.

Results

The results showed pronounced compatibility effects from the robot’s perspective when participants had to take the perspective of the robot during the task, but significantly reduced compatibility effects when human and robot did not interact. In both experiments, compatibility effects from the robot’s perspective resulted in statistically significant differences in response times and in error rates between compatible and incompatible conditions.

Conclusion

We concluded that SR relationships from the perspective of the robot need to be considered when designing shared workspaces that require users to take the perspective of the robot.

Application

The results indicate changed compatibility relationships when users share their workplace with an interacting robot and therefore have to take its perspective from time to time. The perspective-dependent processing times are expected to be accompanied by corresponding error rates, which might affect—for instance—safety and efficiency in a production process.

Sheridan (2016) summarizes today’s robot applications in four areas. First, robots fulfill a social function such as entertaining humans. Second, robots are used as automated shuttle systems in the transport sector. Third, robots serve as remotely controlled teleoperators, which perform tasks as an extended arm of the human in places that are difficult to access for the human. And fourth, in an industrial context robots are often programmed to perform routine tasks under human supervision. However, at a rapid pace, further areas of life are being opened up for robots. These applications have in common that new forms of human–robot interaction (HRI) are emerging, the mechanisms of which are still little understood.

Initial attempts in classification of HRI have differentiated according to the type of interaction between humans and robots (e.g., Onnasch et al., 2016; Scholtz, 2002). The loosest form of interaction is characterized by a coexistence of human and robot, in which both interaction partners meet only occasionally and have no common task. In contrast, in cooperative and collaborative forms of interaction, robots work simultaneously and often in close proximity with humans. In cooperative interactions, the subtasks of human and robot are not directly interdependent and the distribution of tasks is clear. Human–robot collaboration describes the closest form of interaction in which each interaction partner performs successive subtasks.

A human–robot collaboration would be given, for example, if a robot first moves a nail into a position that is then driven in by a human, or vice versa. As described in this example, physical human–robot collaboration often takes place in shared workspaces. As a result, the human and the robot’s movement trajectories may overlap, which then again generates the requirement for an extensive functional safety concept. From an industrial point of view, the goal of collaborative human–robot systems is to combine the human ability to solve imprecisely defined tasks with the advantages of robots, such as precision, power, and endurance (ISO/TS 15066, 2016). In order to harness these synergies, the basic mechanisms by which humans interact physically with a robot must be understood.

Since the actions of both task partners must be coordinated, work areas in which robots and humans work in close physical proximity are particularly demanding in terms of performance, safety, and comfort (Lasota et al., 2014). In order to create fluid physical interactions, efforts have been made to adapt the motions of the robot to human behavior (e.g., Hoffman & Breazeal, 2007; Hoffman, 2019). Lasota and Shah (2015) showed for a collaborative HRI that performance was improved with a robot which was programmed to adapt its movements to human motion. During the experiment, it was the participant’s task to arrange screws, which were then brushed by the robot. The robot’s trajectories were either adapted to the motions of the participant or the robot executed the task with standard motion planning. In addition to increased performance of both interaction partners with regard to faster and more simultaneous actions with less idle time and greater distance, participants also rated the collaboration with the human-aware robot as safe, more comfortable, and more satisfying.

Besides the efforts to adapt the robot’s movements to the human, it is conceivable that human interaction partners adapt to the behavior of the robotic operational characteristics. To the best of our knowledge, however, the possible behavioral changes caused by interacting with the robot have not yet been investigated. Compared to subjective measures, the investigation of human behavioral changes in response to robotic operational characteristics has the advantage that it offers an insight into human cognition unaffected by the participant’s own judgment.

As in social interaction with others, it is possible that humans integrate the spatial perspective of the robot into their own behavior and in this sense adopt its perspective (perspective taking; e.g., Flavell et al., 1981). Since the type of interaction with the robot differs for applications, the question arises as to how human behavior is influenced by different interactions with the robot. An established tool for measuring human behavior is stimulus–response (SR) compatibility. SR compatibility is a key determinant of human performance in various types of technical systems (for an overview of SR compatibility, see Proctor & Vu, 2006; for an overview of applications, see Proctor & Van Zandt, 2018). Humans show faster reaction times (RTs) and lower error rates when the position of the stimulus and the reaction are spatially compatible (compared to incompatible positions). For example, if participants use left- and right-sided button presses to respond to stimuli that appear on the left or right side of a computer screen, performance is better if the stimulus and response location are compatible (e.g., a left-sided button press to a left-sided stimulus) than if they are incompatible (e.g., a right-sided button press to a left-sided stimulus).

The general significance of this finding is underlined by the fact that the performance advantage of spatially compatible SR mappings is also given when the position of the stimulus is task irrelevant (Simon effect; Simon & Small, 1969). Since compatible SR relationships enable the user to intuitively interact with technical systems and lead to better performance, interfaces should be designed to be SR compatible. Incompatible SR arrangements are more likely to cause slow or incorrect reactions and should be avoided in all kinds of human–machine interactions.

Models of spatial SR compatibility assume that the performance advantage in compatible reactions is the result of the overlap of stimulus and response dimensions (e.g., Kornblum et al., 1990). A stimulus feature automatically activates the overlapping response in the sense of a direct route. The direct route, for example, facilitates left reactions to left stimuli and vice versa. For the selection of an incompatible response, however, the automatic activation via an indirect, rule-based route must first be overcome, which causes performance losses in incompatible SR relationships.

In interactions with other humans, participants’ compatibility relations to objects can shift from the egocentric perspective to the perspective of the allocentric referent (e.g., Cavallo, et al., 2017; Müsseler et al., 2019). Participants showed spatial compatibility relations that could be expected from the perspective of their interaction partner, but not from their own. A common finding is that two objects arranged in a vertical line from the actor’s perspective are treated as if they were on the left and right sides when a 90° rotated coactor is in horizontal position to the objects (e.g., Böffel & Müsseler, 2019a; Freundlieb et al., 2016).

The present study investigated the conditions under which perspective taking in the interaction with robots took place or did not take place. In case the robot has a different spatial position than the human, perspective taking toward the robot would systematically change the behavior of the human interaction partner. SR arrangements, which were compatible from the robot’s perspective, would lead to better human performance, whereas incompatible arrangements would result in slower and more erroneous reactions. Thereby, it is possible that SR relations, which could be expected from the human position, could be neglected in favor of the robot’s position.

The study investigated whether there are regular changes in SR compatibility when participants interact with a robot within a physically shared workplace. In this case, we expected the participants to show reaction tendencies that could be expected from the perspective of the robot, but not from their own. Since the size of the compatibility effect, and thus the extent of performance loss in incompatible SR mappings, is crucial in practice, this article further examines the extent of possible compatibility relationships with different forms of HRI.

In two laboratory experiments, we investigated whether different interactions with a mobile robot in one physically shared workplace had regular influence on human performance in spatial compatibility tasks. Exemplary HRIs were investigated in which participants controlled the robot (remote-controlled robot, Experiment 1) or in which they monitored the actions of the robot (supervised robot, Experiment 2). Both scenarios were each compared with a condition in which at the same spatial position a noninteracting robot was presented.

Experiment 1

We compared two groups of participants who performed the same task in an identical setup while interacting with a robot or performing the task independently of the robot’s movements. One group of participants steered the robot into a position and then moved its left or right gripper by pressing left or right button. The remote control of the robot was compared to a situation where the robot executed its movements automatically without interacting with the human.

Based on previous studies (e.g., Böffel & Müsseler, 2019a; von Salm-Hoogstraeten et al., 2020), a compatibility effect relative to the perspective of the robot was predicted. Furthermore, we expected the compatibility relationships to be influenced by the necessity to consider the robot’s perspective. A pronounced robot compatibility effect was predicted in the remote-control condition where participants had to perform the task from the robot’s perspective. A comparatively smaller or missing robot compatibility effect was expected when the human and robot performed the tasks without interacting with each other.

Method

Participants

Forty-eight students (8 men and 40 women, M age 22.10 years, SD 3.35 years, 2 left-handed, 46 right-handed) of RWTH Aachen University participated and received course credits. Half of the group (5 men and 19 women, M age 21.21 years, SD 1.77 years, 1 left-handed, 23 right-handed) were randomly assigned to steer the robot. In the experimental session of the other half of the participants (3 men and 21 women, M age 23.00 years, SD 4.26 years, 1 left-handed, 23 right-handed) the robot acted automatically. The sample size was determined before the data collection and is the same as in previous studies of our lab (e.g., Böffel & Müsseler, 2019b; von Salm-Hoogstraeten et al., 2020). This research complied with the ethical principles of the Declaration of Helsinki and was ethically approved by the Ethics Committee of the Faculty of Arts and Humanities of RWTH Aachen University (2020_002_FB7_RWTH Aachen). Informed consent was obtained from each participant. Participants of both experiments were mainly female which could limit the transferability of the results of our study as there are findings suggesting an influence of gender in spatial abilities (e.g., Tarampi et al., 2016). In compatibility experiments, however, these differences are comparably small and there are pronounced compatibility effects in both women and men (e.g., Stoet, 2017).

Apparatus and stimuli

The experimental program was run on a MacBook Pro with MATLAB software (Mathworks, version R2018b) with the Psychtoolbox-3 extension (Kleiner et al., 2007). The stimuli were a light blue disc (RGB 98 193 254) and a dark blue disc (RGB 36 115 254) with a diameter of 3.5 cm, which were displayed on a horizontally arranged 24.1” display (Wacom DTH-2400 Cintiq 24HD Touch). The tablet was mounted 77 cm from the floor and covered with a 110 × 90 cm large plate on which the robot moved. The plate had a central cutout of 15 × 30 cm through which the tablet was visible. In the middle of the visible screen, a black fixation cross was displayed on a gray background (RGB 231 230 230). The discs appeared in vertical line to the position of the participant, 4.3 cm above or below the fixation cross. In the behavioral experiment, the participants reacted by pressing a left- and a right-sided button (distance between the buttons 13 cm). To manually steer the robot into the experimental positions, the robot could be moved by four additional buttons (forward, backward, left, right).

The robot was built with the Lego Mindstorms EV3 platform and was controlled via Wi-Fi. The target positions of the motors were sent via User Datagram Protocol to the robot. Position control of the motors was executed via MATLAB Simulink using the MATLAB and Simulink support package for Lego MINDSTORMS EV3 hardware (v.18.2.1 and v.18.2.2) on the robot itself. The size of the robot was about 23 × 16.5 × 10.5 cm (left side, Figure 1). Two chain drives enabled 360° radius movements, and right and left grippers could be moved from horizontal position to 60° down and back to horizontal position (tip movement, Figure 1, left side).

Figure 1.

The robot is shown on the left. The rotation of the right gripper during the tip movement is shown as an example. The experimental setup is sketched on the right side. The robot moved from the start position (1) to the left (2) or right (4) experimental position. After half of the experiment, the robot changed sides (3). Depending on the conditions, the robot moved automatically to the experimental positions or was controlled by the participant. Therefore, two arrows indicated the desired position either on the left or on the right side.

Procedure

Noninteracting robot

The participants who were randomly assigned to perform the task with a noninteracting robot were first instructed that the robot would autonomously move to the left/right experimental position (Figure 1, right side). It took the robot about 14 s to reach the experimental position, which was indicated by two arrows on the screen. The robot was arranged such that its left and right grippers were near the upper/lower stimulus position. The order in which the robot moved to the experimental condition was balanced between the participants (either 1-4-3-2 or 1-2-3-4; Figure 1, right side).

After the robot had arrived at the experimental position, the first part of the compatibility experiment began (Figure 2). The compatibility experiment consisted of two parts with six blocks of 32 trials each. The first block in each part of the experiment was defined as practice and excluded from the analysis. Participants were instructed to categorize the color of the disc by pressing the left- or the right-sided button and were informed that the robot executed other commands during the task. The button to color mapping remained the same during the entire experiment and was balanced between participants.

Figure 2.

The spatial compatibility task performed by the participants in the experiment. The participants had to categorize the colors of the discs (dark and light blue) which appeared on upper/lower position by pressing a left/right button. Each trial in which the participant pressed the left/right button in response to a disc which was near the left/right gripper of the robot was compatible (exemplarily shown on the left). Trials where participants pressed a left/right button in response to a disc near the robot’s right/left gripper were incompatible (exemplarily shown on the right). The robot changed the position (left side vs. right side of the figure) after the first half of the experiment.

Every trial started with the presentation of the disc and the fixation cross, the latter being displayed throughout the trial. The robot made a random tip movement with the left or the right gripper, 500 ms after the disc was presented (it moved the left gripper in 16 trials of each block). The disc was displayed until the participant and the robot had performed their actions. When the participants pressed the correct button, the next trial started in 1500 ms. Incorrect reactions, reaction faster than 100 ms, and reaction slower than 1500 ms were followed by a beep feedback (two beeps with a frequency of 720 Hz with a duration of 50 ms each and separated by 50 ms), which sounded 500 ms after the reaction of the participant and the robot were completed. The feedback extended the inter-trial interval by 500 ms. In every block both locations and colors of the discs were repeated 8 times. After six blocks the robot automatically moved to the next experimental position. Once the robot changed its position, the task instructions were repeated and then the second half of the experiment was conducted.

Remote-controlled robot

The participants who performed the task by remotely controlling the robot were first instructed to move the robot to the left or right experimental position. The left and right wheel of the robot should be steered to the same position as described for the noninteracting robot. When the robot was in the experimental position, the participants performed the same color categorization task as the participants who performed the task with the noninteracting robot. The only difference was that the remote-controlled robot executed the participant’s reactions by tipping with its left or right gripper when the participants pressed the left or right button. In every trial, the disc was displayed until the robot had performed the participant’s action. The feedback was identical to the noninteracting robot but sounded immediately after the participants reacted. After half of the trials, the participants steered the robot to the other experimental position and then performed the second half of the experiment. For both groups of participants, the experiment took about 30 min.

Design

The experimental conditions formed a 2 × 2 × 2 design with the within-subject factors of Robot Compatibility (compatible, incompatible) and Robot Position (left position, right position) and the between-subject factor of Interaction Type (remote-controlled robot, noninteracting robot). The factor Robot Compatibility was composed of the following trials: All trials consisting of a left reaction to a stimulus near the left gripper of the robot or a right reaction to a stimulus near the robot’s right gripper were compatible from the robot’s perspective. Incompatible from the robot’s perspective were all trials in which a left response was given to a stimulus near the robot’s right gripper or in which a right response was given to a stimulus near the robot’s left gripper (Figure 2). The dependent variables were RTs and percentages of errors (PE).

Data treatment

In 4.5% of trials a wrong button was pressed. With regard to Kirk (1995, p. 106) error probabilities were arcsine-transformed for statistical analysis. For easier interpretation, the untransformed values are given in the figures and in the text. For the analysis of the RT data, only trials where the participant pressed the correct button were considered. Further, 3.8% RT outliers were identified and excluded using Tukey’s criterion (i.e., RTs 1.5 times the interquartile range below the first quartile or above the third quartile; Tukey, 1977). RT and PE data were averaged and submitted to two repeated-measures analyses of variance (ANOVAs). To compare compatible and incompatible conditions, post hoc pairwise comparisons were calculated (t-tests, always two-tailed).

Results

For the RT data, the factor Robot Compatibility was significant, F(1, 46) = 41.27, p < .001, ${\eta }_p^2$ = .47, with a mean RT of 452 ms in compatible trials and 469 ms in incompatible trials. Furthermore, the predicted interaction between Robot Compatibility and Interaction Type was significant, F(1, 46) = 11.33, p = .002, ${\eta }_p^2$ = .20 (Figure 3). When the robot was controlled by the participant, the compatibility effect relative to the robot’s perspective was 26 ms, 444 versus 470 ms, t(23) = −5.82, p < .001. With the noninteracting robot, the robot compatibility effect decreased to 8 ms, 460 versus 468 ms, t(23) = −2.83, p = .009. The factor Robot Position was neither significant as a main effect nor did it interact. Thus, the robot compatibility effect relative to the spatial perspective of the robot was present both when the robot was on the left and on the right side.

Figure 3.

Mean reaction times and percentages of errors as a function of Robot Perspective (left vs. right) and Robot Compatibility (compatible vs. incompatible). The left side of the figure shows the compatibility effects when the robot was remotely controlled which are larger than in the condition where the robot did not interact (right side of the figure). Asterisks indicate significant pairwise comparison between robot compatible and incompatible trials (t-tests, two-tailed, * p < .05, ** p < .01, *** p < .001, n.s.: not significant). Error bars show within-subject 95% confidence intervals from normalized data (Cousineau, 2005).

For the PE data, there was a main effect of Robot Compatibility, F(1, 46) = 23.78, p < .001, ${\eta }_p^2$ = .34 (3.61% vs. 5.44%). The interaction between Robot Compatibility and Interaction Type was also significant, F(1, 46) = 18.12, p < .001, ${\eta }_p^2$ = .28. When participants controlled the robot, there was a significant difference between compatible and incompatible trials, 2.84% versus 6.02% , t(23) = −5.84, p < .001. This effect was eliminated with the noninteracting robot, t(23) = −.50, p = .62 (4.38% vs. 4.87%). The factor Robot Position was neither significant as a main effect nor did it interact. Thus, the robot compatibility effect relative to the spatial perspective of the robot was present both when the robot was on the left and on the right side.

Discussion

In sum, participants took the spatial position of the robot and coded the stimuli as left and right relative to the grippers of the robot. Moreover, the spatial compatibility effect was clearly stronger in RTs when the participants controlled the robot than when the robot performed the actions automatically. This difference was even more pronounced in the PE: When the robot did not interact with the participants, a robot compatibility effect was not observed at all. The results show that the change in the spatial compatibility relationship by the robot is dependent on whether participants had to interact with the robot during the task and thus if it was necessary to consider the robot’s perspective. To ensure that this finding could also be applied to other types of HRI, Experiment 2 was run in which participants supervised the actions of the robot.

Experiment 2

Two conditions were compared in which the participants performed the same color categorization task as described in Experiment 1. It was manipulated whether the participants had to supervise the actions of the robot during the task (supervised robot) or performed the task independently of the robot (noninteracting robot). A larger robot compatibility effect was predicted when supervising the robot than when human and robot did not interact. Since when supervising the robot participants had to monitor the actions of the robot while performing the color categorization task themselves, dual-task costs were expected compared to the noninteracting condition (e.g., Pashler, 1994).

Method

Participants

A new sample of 24 students (3 men and 21 women, M age 22.42 years, SD 2.70 years, 2 left-handed, 22 right-handed) participated in the experiment.

Apparatus, stimuli, and procedure

Apparatus and stimuli were identical to those of Experiment 1. The participants first completed the task with a noninteracting robot and then had to supervise the robot. At the beginning of every experiment the robot automatically moved to the experimental positions as depicted in Figure 1. The participants were then asked to complete the task with the noninteracting robot. The only task of the participants was to categorize the color of the disc (dark blue or light blue) appearing in upper or lower position (see Experiment 1). Similar to the noninteracting robot of Experiment 1, the robot performed a tip movement with its left or right gripper 500 ms after the disc appeared. In 21 of 32 trials the automatic acting robot performed the same task as the participant (e.g., a left movement with the gripper when the participant had to press the left button). In the remaining trials the robot moved the other gripper (e.g., the right gripper when the participant’s task was to press the left button). The participants were informed that the robot automatically performed the same task but made a few mistakes.

In the second part of the experiment the participants supervised the robot. In every trial the participants were instructed to perform two tasks. First, they had to categorize the color of a disc. As in the noninteracting condition, the robot performed the same task as the participants by moving the left or right gripper 500 ms after the disc appeared. The participants were to monitor whether the robot was moving the correct gripper (21 of 32 trials in each block) or moved the wrong gripper (11 trials). If participants noticed the robot moved the incorrect arm, they were instructed to press the left and right button simultaneously as soon as they completed the color categorization task.

In each trial, a disc appeared next to the fixation cross until the participant pressed a button and the robot moved one arm. After a disc disappeared there was a 2000 ms interval in which the fixation cross was presented. The interval aborted when the left and right button was pressed simultaneously. If the correct response was given in the color categorization task and participants correctly monitored the robot the next trial started after 1500 ms. Feedback for the categorization task was given after the monitoring interval. In case of false alarm or a miss in the monitoring task, an additional beep sounded (880 Hz with a duration of 50 ms) and the next trial started with a delay of 500 ms.

Design and data treatment

The behavioral data were preprocessed in the same way as in Experiment 1. In 2.7% of trials a wrong response was given by the participants. Further, 4.4% of RT outliers were determined using Tukey’s criterion (Tukey, 1977). RT and PE data were submitted to a 2 × 2 × 2 repeated-measures ANOVA including the within-subject factors Interaction Type (supervised, noninteracting), Robot Position (left position, right position), and Robot Compatibility (compatible, incompatible).

Results

For RT data there was a main effect of Interaction Type, F(1, 23) = 60.20, p < .001, ${\eta }_p^2$ = .72. Participants responded 91 ms faster if they were not also supervising the robot (456 vs. 547 ms). The factor Robot Compatibility was also significant, F(1, 23) = 57.49, p < .001, ${\eta }_p^2$ = .71, and interacted with the factor Interaction Type, F(1, 23) = 29.66, p < .001, ${\eta }_p^2$ = .56 (Figure 4). There was a pronounced robot compatibility effect when participants had to supervise the robot, 454 versus 532 ms, t(23) = −7.56, p < .001, and only a weak tendency for a compatibility effect was observed for the noninteracting robot, 454 versus 458 ms, t(23) = −1.96, p = .062.

Figure 4.

Mean reaction times and percentages of errors as a function of Robot Position (left vs. right) and Robot Compatibility (compatible vs. incompatible). The left side of the figure shows the compatibility effect when supervising the robot, which is larger than for the noninteracting robot (right side of the figure). Asterisks indicate significant pairwise comparison between robot compatible and incompatible trials (t-tests, two-tailed, * p < .05, ** p < .01, *** p < .001, n.s.: not significant). Error bars show within-subject 95% confidence intervals from normalized data (Cousineau, 2005).

The factor Robot Compatibility was also significant for PE, F(1, 23) = 19.78, p < .001, ${\eta }_p^2$ = .46, and interacted with the factor Interaction Type, F(1, 23) = 10.42, p = .004, ${\eta }_p^2$ = .31. Here, too, a robot compatibility effect was found for the supervised robot, 1.5% versus 3.91%, t(23) = −5.97, p < .001, but not for the noninteracting robot, 2.2% versus 3.04%, t(23) = −1.34, p = .194.

Discussion

As is common for dual tasks, the results showed slower RTs when participants had to supervise the actions of the robot during the categorization task. Of more importance were the compatibility relationships shown by the participants with both interaction types. A robot compatibility effect in PE and RT was found when participants supervised the robot and had to take its perspective during the task. Irrespective of whether the robot was on the left or right side, participants coded the disc as left- and right-sided relative to the left and right grippers of the robot. This effect did not occur if a robot was presented with whom participants did not interact during the task. Thus, Experiment 2 showed that the size of the compatibility effect from the robot’s perspective depended on whether the robot’s perspective needed to be taken during the task.

General Discussion

Robots find their way into more and more areas of life. A multitude of applications are conceivable in which humans and robots interact in close proximity and have to coordinate their movements. In order to ensure that the robot could be optimally adapted to human action, it is plausible to incorporate cognitive mechanisms of action control, such as SR compatibility, into the prediction. We observed pronounced effects of SR compatibility in the interaction with a remotely controlled and monitored robot, but significantly reduced or even diminished effects when the robot did not interact, was not controlled, or was not supervised. Although the robot worked in the same physical position in all conditions, depending on the degree of interaction and the associated need to consider the perspective of the robot, different SR compatibility effects were observed from the robot’s perspective.

Similar as in the interaction with virtual avatars or other humans (e.g., Böffel & Müsseler, 2019a; Cavallo, et al., 2017; Freundlieb et al., 2016), the perspective of the robot served as a reference relative to which the spatial position of the stimuli was recoded. The participants coded the stimuli as left- and right-sided relative to the grippers of the robot with which they interacted. From the perspective of the robot, spatially compatible SR mappings were linked to improved human performance, while incompatible mappings led to increased response times and higher error rates.

Changes in human SR compatibility relationships due to the interaction with a robot were exemplarily demonstrated when participants remote-controlled the robot, when the participant supervised the actions of the robot, and when the human and robot performed different tasks. In all tested scenarios, the robot was located in the same spatial proximity to the participants, which made it possible to investigate the changes in behavior solely through different forms of interaction with the robot. Both experiments showed that participants encoded the stimuli from the perspective of the robot when the perspective of the robot had to be considered during the task. Large compatibility effects were observed when participants were in control of or supervised the robot’s actions. But even if the robot and the human did not interact, there were tendencies that the participants included the spatial position of the robot in their spontaneous reaction tendencies. The latter finding indicates that, even in scenarios where humans and robots perform different tasks in close proximity, the perspective of the robot automatically influences human action, suggesting that our findings could also be applied to further scenarios in which human and robot share the same workspace.

Nevertheless, the size of the compatibility effect depended on whether the participants had to take the robot’s perspective during the task. When the robot was remotely controlled by the human, the recoding of the objects relative to the robot’s perspective, and thus the compatibility effect from the robot’s perspective, was increased. Presumably the referential coding of the robot’s perspective was strengthened by the compatible response effect of the robot’s gripper (e.g., a left button press leads to a tip of the left gripper). In analogy to the use of tools, response effect congruency is assumed to be a cause of compatibility effects (e.g., Müsseler & Skottke, 2011). Ultimately, the need to consider the robot’s perspective when controlling it made the robot a salient reference point from which the stimulus positions were re-encoded.

In addition, greater compatibility effects were found from the robot’s perspective when participants supervised the robot, compared to the condition in which the robot did not interact. The participants’ task was to monitor the robot and respond if the robot made an incorrect movement. This passive monitoring of the robot is required in a number of applications, for example, when humans have no direct control over the robot’s movements but have to intervene in case of system failure. The costs of shared attention were reflected in Experiment 2 in performance losses when supervising the robot versus not interacting with the robot. Further, Experiment 2 allowed the conclusion that even when monitoring the robot, the human interaction partner was influenced by the actions of the robot, which was systematically reflected in its own actions. In order to monitor the robot, participants had to take its perspective into account. As a result, participants integrated the robot’s perspective into their own behavior to such an extent that they showed behavioral tendencies from the robot’s perspective.

As an important design criterion for human–machine interactions, SR relationships of control elements should be as compatible as possible with the natural human behavior (e.g., a left button press on an alarm on the left side of the screen). Thereby, the common approach is to consider spatial compatibility relationships from the user’s perspective. Our results indicate that this could not be sufficient when humans interact physically with robots in a shared workspace, as it is the case in human–robot collaboration. We were able to show that participants adopted the perspective of the robot and showed SR compatibility relationship from its perspective. It was no longer the human perspective that was decisive, but the perspective of the robot. The practical implication of this finding is that when establishing compatible SR relationships in a shared workplace where humans have to take from time to time the perspective of a robot, SR relationships from the perspective of the robot can be crucial for human behavior and must therefore be taken into account.

Consider the following example. At a workplace ergonomically designed from the user’s perspective, the individual work steps require compatible manual actions. If a collaborative robot would be subsequently installed to support the worker, the robot introduces a different perspective to the situation. Our results indicated changed compatibility relationships when the users take the perspective of the robot. Previously error-free and fast processing times could now deteriorate if the SR mapping is incompatible from the perspective of the robot. Note that the compatibility-dependent changes in the RTs are accompanied by corresponding error rates, which have even greater effects on safety and efficiency of the production process.

Applications

In order to design future HRI ergonomically, it is necessary to consider how human cognition is influenced by interaction with robots. Especially when humans and robots work in a shared workspace, the actions of both partners must be precisely coordinated to avoid motion conflicts (e.g., Hoffman, 2019; Hoffman & Breazeal, 2007; Lasota et al., 2014; Lasota & Shah, 2015). The study emphasizes the importance of integrating knowledge about human information processing into the design of HRI and other forms of human–machine interactions by showing that spatial compatibility relationships are systematically changed through interaction with a robot. This knowledge could, for example, be used to cooperate cognitive processes in robot motion planning.

Limitations and Further Research

The two grippers of our robot formed a left and right dimension, which enabled us to clearly distinguish the spatial references of robot and human. In the variety of robot applications, this is only one possible type of robot. Further studies, for example, with industrial robot arms, could examine whether our findings could be transferred to other types of robots.

Conclusion

In this paper, we investigated adaptations of SR compatibility effects when humans interacted with robots in a shared workspace. Based on our findings, we concluded that human behavior in interaction with the robot adapted to the spatial perspective of the robot. The participants showed responses that could be expected from the perspective of the robot they were interacting with, but not from their own. The perspective of the robot changed the spatial compatibility relationships to objects, which was reflected in a performance change in terms of response times and error rates. Further, the spatial reference coding depended on whether the perspective of the robot had to be considered in order so solve the task. In this case, SR compatibility relationships were defined by the perspective of the robot and not by the human’s own position. Human SR compatibility relationships in human–robot interaction could thus be (co-)determined by the spatial perspective of the robot.

Key Points

• The study investigated possible changes in human SR compatibility relationships when working with a robot in a shared workspace.

• Robust changes in the human SR compatibility relations due to the spatial perspective of the robot were observed when humans had to take the perspective of the robot during the task.

• The changes in SR compatibility relationships were pronounced when humans had to control the robot remotely or supervise it. However, there was also a tendency to take the robot’s perspective, if the robot did not interact with the human.

• Based on our results, we conclude that interaction with a robot could modify human SR compatibility relationships, which should be considered when designing workspaces where humans and robots physically interact.

Author Biographies

Sophia von Salm-Hoogstraeten is a member of the Department of Work and Cognitive Psychology at RWTH Aachen University (Germany). She received her master’s degree in psychology from RWTH Aachen University (Germany) in 2018.

Jochen Müsseler is a full professor of psychology at the RWTH Aachen University and head of the Department of Work and Cognitive Psychology. He received his PhD in psychology from the University of Bielefeld (Germany) in 1986 and his postdoctoral degree (habilitation) in psychology from the Ludwig Maximilians University of Munich (Germany) in 1995.

ORCID iD

Sophia von Salm-Hoogstraeten https://orcid.org/0000-0002-3551-3979