Visual control of steering through multiple waypoints

Abstract

Effective locomotion often requires the ability to navigate within complex environments at speed, moving smoothly through multiple waypoints while avoiding obstacles. If actors consider only one waypoint at a time, they may be forced to make jerky steering adjustments, collide with obstacles, or miss waypoints altogether. Findings from previous studies suggest that humans do use information from beyond the most immediate waypoint but leave open questions about how such information is used to control steering and speed of self-motion. The present study was designed to test three hypotheses about how humans anticipate multiple upcoming waypoints. Subjects performed a simulated drone-flying task in which they used a game controller to steer through a series of three gates: Gates 0 and 1 that were centered on the longitudinal axis and separated by a fixed distance, and Gate 2 at a distance, angle, and orientation that was manipulated across trials. In Experiment 1, when the drone was programmed to simulate a more sluggish vehicle, subjects initially veered away from Gate 1 before turning back, setting up a smoother trajectory through Gate 1 toward Gate 2. Trajectories between Gates 0 and 1 varied with the angle and distance (but not orientation) of Gate 2. In Experiment 2, when the drone was more agile, subjects flew more directly toward Gate 1 regardless of the position of Gate 2, rapidly decelerating and turning sharply toward Gate 2 at the last moment. Taken together, the results support the hypothesis that actors are attuned to their action capabilities and guide their movements so as to maintain the ability to successfully steer through upcoming waypoints within those capabilities. Such behavior is consistent with the more general theory of affordance-based control.

Supplementary Information

The online version contains supplementary material available at 10.1038/s41598-025-15880-2.

Introduction

Among the most compelling examples of the importance of motion perception are those that involve high-speed steering through densely cluttered environments. Movement of the body within such spaces generates patterns of visual motion (i.e., optic flow) that provide information about the speed and direction of self-motion^1,2 the immediacy of collisions with potential obstacles³ and openings that afford safe travel^4,5. The ability to detect such information is integral to activities such as mountain biking, slalom skiing, and drone racing in humans^6–10 and is part of the natural repertoire of birds and flying insects as they navigate wooded spaces in search of food^5,11–13.

An important goal of research on this topic is to develop models that capture how information in optic flow is used to perceive and select traversable routes and modulate locomotor speed and direction to avoid obstacles and reach goals^14–17. Such models are useful for addressing more general questions about the neural mechanisms and behavioral strategies that underlie visual-motor control^18,19 and promoting synergies with research on vision-based control in autonomous robots^20,21. Among the challenges of developing such models is understanding how actors use information from multiple objects at different time horizons and how the use of that information is shaped by the actor’s movement capabilities. This is the specific aim of the present study.

We focused on a variant of the task of steering through clutter in which specific gaps between obstacles are identified as waypoints through which the actor must pass. We also removed obstacles from the scene. Paring down the task in this way strips away the route selection and obstacle avoidance elements of the task but provides an ideal context for studying the control of steering as a continuous, ongoing behavior. It also allows for the kinds of systematic experimental manipulations that are needed to address the aforementioned specific aims.

The role of information from future waypoints

Up to this point, the majority of research on the visual control of steering has focused on steering to a single goal^22,23 or along winding roads^24–26. Much less is known about steering through a series of predefined waypoints (e.g., gates). An important open question about this task concerns the use of information from waypoints that lie beyond the most immediate one. Under some conditions, it may be sufficient to focus entirely on the most immediate waypoint (WP_N) and disregard future waypoints (WP_N+1, WP_N+2,…) until passing through WP_N. In other situations, however, focusing only on WP_N may result in poor performance (jerky steering, reduced speed, or failing to successfully intercept waypoints). Previous research on the visual control of steering provides some insight into the use of information from future waypoints. These findings can be distilled into three main points.

First, when steering through multiple waypoints, humans typically shift their gaze to WP_N+1 between 0.5 and 2.0 s before passing through WP_N. Such behavior has been observed in the context of a simulated slalom cycling task¹⁰ a slalom skiing task⁶ and a simulated drone-piloting task in which subjects had to steer a quadcopter through a series of gates²⁷. These findings suggest that humans begin to use information about subsequent waypoints before reaching the most immediate waypoint.

Second, performance degrades when lookahead distance is reduced to the point that only one waypoint is visible at a time. In the aforementioned drone-piloting study, Jansen and Fajen²⁷ manipulated how far in advance WP_N+1 appeared before or after the subject passed through WP_N. Performance (e.g. speed, smoothness of steering) was significantly negatively impacted when future gates were not rendered at least 1 ½ segments ahead, i.e. when the subject was halfway between WP_N−1 and WP_N. This is similar to the effects of reduced lookahead distance that have been reported in studies of automobile driving along a winding road²⁴ and walking over complex terrain^28–30.

Third, evidence from previous studies suggests that people start to make trajectory adjustments based on WP_N+1 prior to reaching WP_N^8,10,27. In the two most recent of these studies^8,27 subjects guided a drone through a series of gates. Using linear regression, the authors found that the angular position (but not orientation) of WP_N+1 was a significant predictor of the drone’s heading at WP_N. As subjects reached WP_N, their heading was biased toward WP_N+1, suggesting that they began to adjust their heading in the direction of WP_N+1 before reaching WP_N.

Hypotheses about the control of steering through multiple waypoints

Taken together, these findings provide strong evidence that people detect and use information about WP_N+1 before reaching WP_N. However, how such information is used to guide steering in an anticipatory manner remains unclear. Consider, for example, the finding that heading at WP_N was positively correlated with the angular position of WP_N+1^8,27. Such behavior could have resulted from a strategy in which subjects steer directly toward WP_N and turn toward WP_N+1 shortly before reaching WP_N (red curve in Fig. 1A). Alternatively, subjects could have adopted a strategy in which they initially veer away from WP_N before turning back, setting up a more direct path to WP_N+1 (blue curve in Fig. 1A). Both behaviors rely on information about WP_N+1 and capture heading at WP_N but reflect the use of qualitatively different control strategies. As such, a better understanding of how behavior between WP_N−1 and WP_N is affected by the position and orientation of WP_N+1 would provide insight into the control strategies that humans use to steer through multiple waypoints. Next, we consider three hypotheses about how humans use information about WP_N+1 to guide steering toward WP_N.

Fig. 1.

(A) Two potential trajectories through a series of three waypoints. (B) Depiction of the weighted-combination of attractors hypothesis, according to which the agent’s heading is drawn toward the two upcoming waypoints. (C) Depiction of the constant-curvature hypothesis, according to which the agent’s heading and path curvature are adjusted to match the tangent direction and curvature of the constant radius arc that travels through the agent and both waypoints.

Weighted combination of attractors

The first hypothesis is that heading is attracted by a weighted combination of WP_N and WP_N+1 (Fig. 1B). This strategy could be captured within the behavioral dynamics framework^16,31 which treats goals as attractors of heading, obstacles as repellors of heading, and trajectories as emerging in real-time from their interactions. Zhao and Warren³² proposed a variant of the original behavioral dynamics model in which both waypoints serve as attractors of heading, with their relative strength of attraction captured by a weight that varies with distance to WP_N. Initially, heading is attracted solely by WP_N. As the agent draws closer to WP_N, the weight for WP_N begins to decrease and the weight for WP_N+1 begins to increase. The weights continue to change in this manner until WP_N+1 is the sole attractor of heading at the moment that the agent reaches WP_N. An agent guided by the weighted-combination strategy through a series of waypoints would produce a trajectory qualitatively similar to the red curve in Fig. 1A, wherein the agent steers directly towards WP_N and then turns toward WP_N+1 just before passing through WP_N. Variations of this trajectory could be achieved by modifying how the weights vary with distance to WP_N. For example, a trajectory in which the turn toward WP_N+1 is initiated only after reaching WP_N could be captured by abruptly switching the weights between 0 and 1 at the moment the agent reaches WP_N. However, because waypoints can only attract heading in this model, there is no weighted combination of WP_N and WP_N+1 that could produce the blue trajectory in Fig. 1A in which the agent initially turns away from WP_N.

Similar behavior could result from a steering-by-gaze model in which heading is attracted by the current gaze direction^10,33. Although such a strategy is not strictly speaking a weight-combination strategy, the resulting trajectories would be qualitatively similar if one assumes that actors initially fixate WP_N and then switch to fixating WP_N+1 shortly before reaching WP_N.

Constant-curvature strategy

The second hypothesis is that the agent’s heading and path curvature are attracted by the constant radius path that passes through WP_N, and WP_N+1. The agent and the two WPs comprise three points, which define an arc of constant curvature with a tangent direction relative to the agent’s heading (Fig. 1C). The directions of and distances to the two waypoints are sufficient to specify the curvature and tangent direction of this constant-radius path. The agent could then adjust its path curvature to null the difference between the current and required heading, as well as the difference between the current and required curvature. This strategy can be understood as a generalization of the Pure Pursuit control law³⁴ which is well-known in robotics and autonomous vehicles research, to the case of multiple waypoints. Unlike the weighted-combination strategy, this strategy produces trajectories that initially veer away from WP_N before turning back, similar to the blue curve in Fig. 1A. In addition, the lateral displacement of the trajectory would increase with the angle and decrease with the distance of WP_N+1 relative to WP_N.

Maintaining the ability to steer through waypoints

The third hypothesis is that actors move so as to maintain their ability to steer through waypoints given the limits of their action capabilities. During high-speed steering, the set of potential trajectories is constrained not only by the layout of waypoints but also by the actor’s capabilities to change their speed and direction of self-motion. Consider, for example, performing the task depicted in Fig. 1A with a more sluggish vehicle that is slow to change direction and speed. Steering directly to WP_N may leave the actor in a state from which it would be impossible to reach WP_N+1 given the limits of their action capabilities. However, they could maintain their ability to steer through WP_N+1 by initially veering away from WP_N and then turning back (similar to the blue trajectory in Fig. 1A). On the other hand, if the vehicle is more agile, it may be possible to follow a more direct path to WP_N (similar to the red trajectory in Fig. 1A) and still reach WP_N+1 without having to exceed the limits of one’s action capabilities. As such, whether subjects steer directly toward WP_N and then turn or veer away from WP_N and then back may depend on the dynamics of the controlled vehicle.

Such findings would support the use of a control strategy that depends on the dynamical constraints of the body or vehicle, which is not a characteristic of the attractor-based hypothesis or constant-curvature hypothesis. A model-based control strategy could capture sensitivity to action limits by assuming that actors form and make use of internal models^25,35–37. Alternatively, such effects could reflect the use of an affordance-based control strategy^38–42 which assumes that actors are tuned to the limits of their action capabilities and guide their movements to ensure that the task remains possible given these limits.

Experimental paradigm

The current study was designed to test the predictions of these three hypotheses by examining how steering behavior leading up to WP_N was affected by WP_N+1. The task was performed in a simulated environment and required subjects to guide a quadcopter through a series of three gates, while the relative angle, distance, and orientation of the third gate was manipulated across trials. The study comprised two experiments with different subjects, using an identical paradigm except for the drone dynamics. In Experiment 1, the drone was more sluggish; maximum turning velocity was lower, the range of speeds was narrower, and it took longer to change speeds. In Experiment 2, the drone was more agile. Analyses focused on the trajectories that subjects followed between the first two gates. Specifically, we examined how these trajectories varied with the angle, distance, and orientation of the third gate as well as the dynamics of the drone.

Methods

Participants

There were 19 participants in Experiment 1 (10 men, seven women, one non-binary, and one not identified) whose ages ranged from 18 to 24 with a mean of 19.6 years. In Experiment 2, there were 20 participants (14 men, five women, and one non-binary) whose ages ranged from 19 to 27 with a mean age of 21.0 years. All subjects had normal or corrected-to-normal vision, and none had any conditions that would impair use of a controller. All subjects gave their written informed consent before participation and the protocol (#2069) was approved by the Rensselaer Polytechnic Institute Institutional Review Board. The experiment was performed in accordance with the Declaration of Helsinki. 29 subjects received course credit for participation and 10 received monetary compensation.

Hardware

The virtual environment was generated in Unity 2019.4.29f1 running on an Alienware Aurora R12 (Dell, Round Rock, TX) equipped with an NVIDIA GeForce RTX 3090 graphics card (NVIDIA, Santa Clara, CA) and 3.50-GHz 11th Gen Intel Core i9-11900KF processor (Intel, Santa Clara, CA). The environment was displayed on a 69 cm × 30 cm LG UltraWide monitor (LG Electronics, Seoul, South Korea) at 2560 × 1080 resolution with a 60-Hz refresh rate. Subjects were seated approximately 1 m away from the monitor, which subtended a horizontal visual angle of 38° and a vertical angle of 17° when viewed from that distance. An Xbox Series X Wireless Controller (Microsoft, Redmond, WA) was used to control movement of the drone.

Virtual environment

The virtual environment was created in Unity using GaiaPro, an asset from the Unity Asset Store (author Procedural Worlds), and consisted of a 130 m-wide, linear, grass-textured path bordered by trees. Three gates (0, 1, and 2) were placed in the environment. Gates 0 and 1 were positioned along the centerline of the path and spaced 50 m apart, and Gate 2 was at an angle, distance, and orientation that varied across trials (Fig. 2). Gates 0 and 2 were square with sides of length 3 m and Gate 1 was spherical with a diameter of 3 m. The shape of Gate 1 was chosen to avoid restricting the drone’s approach angle. All three gates were vertically centered at the same height as the quadcopter’s camera.

Fig. 2.

Top-down view (A) and a first-person view (B) of the simulated environment. Subjects were instructed to steer through all three gates in order. Gates 0 and 2 were square. Gate 1 was spherical.

The simulated drone and drone controller were assets purchased from the Unity Asset Store (Drones Bundle Package and FPV Drone controller; author Mario Haberle). The drone had a simulated mass of 0.3 kg and the drone camera had a field of view of 70°. The rotation (yaw) rate of the drone was controlled by the x-axis of the left joystick and forward/backward thrust was controlled by the y-axis of the right joystick. The altitude of the drone was constant at 5.5 m relative to a fixed reference level in the simulated environment.

Although the simulated drone was locked to a fixed height, the vehicle dynamics were more similar to real drone flying than to steering a car. For one, subjects could independently manipulate rotational velocity and translational velocity. The drone could, for example, rotate in place without translating at all. Subjects could also maintain the same speed while changing rotational velocity to perform a turn or manipulate the velocity of the drone to affect path curvature (a slower moving drone would make a tighter turn than a faster drone). There were other small differences, such as the fact that forward and backward thrust affected the pitch of the drone. The control mapping was less complex than real-life drone controls for ease of use by novice subjects, but without loss of the sensation of flying a drone.

The session framework utilized the Unity Experiment Framework (UXF) package to initialize experiment settings and for trial management and data logging⁴³.

Design and procedure

In both experiments, subjects were first briefed on the content of the study and written consent was obtained. Subjects then filled out a short demographics questionnaire. A Pupil Core eye tracker (Pupil Labs) was fitted and calibrated to the subject. The subjects first spent a minimum of five minutes flying around a basic forested environment to become familiar with the controller and drone dynamics. There was subsequently a block of training trials with the same basic task as in the main experiment, but unique conditions. Subjects then completed the main experiment. They were allowed to take breaks between blocks if they desired to minimize fatigue.

The design of both experiments was the same, with three independent variables corresponding to the angle, distance, and orientation of Gate 2 relative to Gate 1 (Fig. 3). There were six angle conditions (+/– 30°, 50°, 70°); three distance conditions (10 m, 25 m, 40 m); and three orientation conditions (0°, +/– 20°). Each condition was repeated four times for a total of 216 trials. Trials were presented in six blocks of 36 trials each with the conditions for each trial selected randomly without replacement from the 54 combinations of Gate 2 angle, distance, and orientation.

Fig. 3.

Top-down, schematic view illustrating how the angle, distance, and orientation of Gate 2 (red lines) relative to Gate 1 (green circle) were manipulated across trials.

Prior to the start of each trial, the drone was teleported to a stationary position 10 m before Gate 0. The initial heading of the drone relative to the centerline varied randomly within +/– 2.5°. Gates 0 and 1 were immediately visible but Gate 2 did not appear until later in the trial. Once the controller joysticks were in a neutral position for 2 s, the drone moved forward at the default speed. The drone’s passage through Gate 0 triggered the appearance of Gate 2. Subjects were instructed to fly through all three gates to the best of their ability. If the drone missed Gate 0 or Gate 1 or flew into the trees that bordered the path, the trial was reset to the beginning. If the drone missed Gate 2, the trial was not reset. Instead, it was automatically terminated and the drone was teleported to the starting position for the next trial.

The parameters that affected the dynamics of the drone varied across the two experiments. The linear speed of the drone was affected by three forces: drag, a baseline thrust when the right joystick was in the neutral position, and thrust applied whenever subjects displaced the right joystick from the neutral position. The rotational speed of the drone was affected by angular drag and the force applied when subjects displaced the left joystick from the neutral position. Drag and angular drag were each simulated using a function that reduced linear or angular velocity, respectively, proportional to its magnitude.

Across experiments, the magnitude of translational thrust and the gain on the rotation joystick were manipulated. This resulted in different acceleration and speed ranges for both translation and rotation. In Experiment 1, the default speed of the drone was 20 m/s and subjects could speed up to 25 m/s or slow down to 10 m/s, with a maximum acceleration of 16 m/s². The maximum angular velocity was 54 °/s. In Experiment 2, the default speed of the drone was 25 m/s and subjects could speed up to 30 m/s or slow down to 0 m/s, with a maximum acceleration of 24 m/s². The maximum angular velocity was 78 °/s. Minimum turn radius (r), which is a function of maximum angular speed (ω) and linear speed (v) (i.e., r = v/ ω) also varied across experiments. In Experiment 1, minimum turn radius was 7.35 m at minimum speed and 21.22 m at default speed. In Experiment 2, the drone was able to rotate in place at minimum speed, so the minimum turn radius was 0 m; at default speed (25 m/s), 18.36 m.

Data analyses

Post processing was performed using custom R scripts and data recorded by the Unity program. At every frame, Unity recorded the 3D position and orientation of the drone and gates relative to the environment. These data were used to recover other variables, such as instantaneous velocity. The drone’s change in position over a number of frames was used to calculate speed and heading. Lateral deviation was found as the distance from the centerline (z axis) along the x axis. Unity also recorded any objects with which the drone collided, as well as the drone’s passage through gates. A gate was recorded as successfully passed through if the drone entered the collider in the center of the gate, regardless of whether the drone collided with the frame of the gate. Trial resets from missing Gate 1 were also recorded (trials were not reset if the subject missed Gate 2). Main effects and interactions were tested using a repeated-measures ANOVA in R. Prior to calculating the 95% confidence intervals for the plots, we removed the between-subjects variance using the method introduced by Loftus and Masson⁴⁴. This is because in both experiments, the independent variables were manipulated within subjects. As such, the statistical analyses (repeated-measures ANOVAs) focused on how individual subjects were affected by the different gate angles, distances, and orientations. Overall differences between individuals are factored out in a repeated-measures ANOVA. Removing between-subjects variance prior to calculating the 95% confidence intervals ensures that the confidence intervals reflect the component of variance that was relevant to the statistical analyses.

The time series of (x, z) drone positions at every frame was used to create plots of the drone’s average trajectory in each condition. Trials in which subjects did not successfully pass through all three gates were excluded. Prior to averaging, we interpolated the time series of drone coordinates on each individual trial and took the mean x-coordinate at 0.5 m intervals along the z-axis. Additionally, the x-coordinate of the drone’s position was flipped on trials with negative Gate 2 angles (i.e., Gate 2 appeared on the left of the centerline). Trials were then averaged over repetitions within each subject, within each distance x angle combination, and over the sign (±) of Gate 2 angle, resulting in a total of 9 unique conditions (3 angles x 3 distances). Data were averaged across orientations because the effects of orientation on measures of steering behavior between Gates 0 and 1 were weak to negligible, as explained below. Lastly, we averaged across all subjects, resulting in a mean trajectory for each condition. The time series of instantaneous velocity at every frame was averaged within and across subjects in the same way as the drone’s position. The resulting speed series was used to determine the distance between points on the trajectory plot. This distance was scaled according to the speed range of all subjects.

Results

Basic measures of task performance

This first section focuses on the frequency of trials in which subjects missed Gates 1 and 2, which serves as a basic measure of performance on the main task. In Experiment 1, the overall mean number of trials out of 216 in which subjects missed Gate 1 was 47.49 (SD = 25.25) and the mean number of trials in which subjects missed Gate 2 was 37.21 (SD = 35.10). Misses of both Gate 1 and Gate 2 significantly increased with angle (Gate 1: F_{2, 36} = 26.79, p < 0.001, η²_G = 0.17; Gate 2: F_{2, 36} =11.41, p < 0.001, η²_G = 0.19) (see Fig. 4A, C). Misses of Gate 1 were not significantly affected by distance (F_{2, 36} = 2.93, p = 0.07, η²_G = 0.01) but Gate 2 misses decreased with distance (F_{2, 36} = 14.38, p < 0.001, η²_G = 0.10). The angle x distance interaction had a significant but weak effect (Gate 1: F_{4, 72} = 2.95, p < 0.05, η²_G = 0.02; Gate 2: F_{4, 72} = 4.48, p < 0.01, η²_G = 0.04). There was no significant effect of orientation (Gate 1: F_{2, 36} = 0.07, p = 0.84, η²_G < 0.01; Gate 2: F_{2, 36} = 0.15, p > 0.05, η²_G < 0.01).

Fig. 4.

Mean proportion of trials in which subjects missed Gate 1 (A, B) and Gate 2 (C, D) for each angle and distance condition in Experiment 1 (left) and Experiment 2 (right). Error bars indicate 95% confidence intervals with between-subject variance removed.

Misses were much less frequent in Experiment 2, of both Gate 1 (M = 19.55, SD = 14.42) and Gate 2 (M = 9.25, SD = 7.31). Although the general pattern of Gate 1 misses was similar to that in Experiment 1, only the main effect of angle reached significance (F_{2, 38} = 7.29, p < 0.01, η²_G = 0.03) (see Fig. 4B and D). There was no effect of distance (F_{2, 38} = 0.30, p = 0.74, η²_G < 0.01) or orientation (F_{2, 86} = 0.42, p = 0.66, η²_G < 0.01) and no significant interactions. The difference compared to Experiment 1 was more dramatic for Gate 2 misses. There were very few misses in any condition and neither angle (F_{2, 38} = 0.34, p > 0.05, η²_G = 0.002) nor distance (F_{2, 38} = 1.52, p > 0.05, η²_G = 0.01) had a significant main effect. However, there was a weak but significant effect of orientation on Gate 2 misses (F_{2, 38} = 3.49, p < 0.05, η²_G = 0.01), and the angle x distance x orientation interaction was significant (F_{8, 152} = 1.75, p < 0.05, η²_G = 0.04).

These results align with the intuition that gates that were at larger angles and smaller distances were more difficult to reach, especially with a sluggish vehicle. Subjects reduced their speed in such conditions (as explained below), possibly to maintain the affordance of goal passage, but were still more prone to missing gates. Variations in the orientation of Gate 2 could have affected task difficulty, but the effects of orientation were weak at best. Subjects were able to consistently achieve task success in both experiments; however, when the drone was more sluggish (Experiment 1), the overall proportion of missed gates was higher.

Trajectory measures

The next set of analyses focus on the trajectories that subjects generated as they travelled from Gate 0 to Gate 1. Trajectories from Gate 1 to Gate 2 are not directly relevant to the main focus of the present study. As such, they were analyzed separately and are reported in the Supplementary Material (see Figs. S1 and S2). The key findings are depicted in Fig. 5, which shows the mean trajectories for each combination of Gate 2 angle and distance in both experiments. This figure shows both the shapes of the trajectories as well as mean speed, which is represented by the spacing between dots (see data analysis section of Methods for details). Note that the mean trajectories in Fig. 5 do not include unsuccessful trials. However, the mean trajectories based on all trials (both successful and unsuccessful) were similar (see Fig. S3 in the Supplementary Material section). Figures S4 and S5 in the Supplementary Material section illustrate the mean trajectories for each individual subject across the nine conditions.

Fig. 5.

Mean trajectories on successful trials averaged across subjects for each angle and distance condition in Experiment 1 (A, B, C) and Experiment 2 (D, E, F). Note that the plot is stretched along the x-axis. This exaggerates the degree of lateral deviation but allows for better visualization of the difference in deviation across conditions, which is the more important finding.

In Experiment 1 (Fig. 5A–C), subjects initially turned away from Gate 1 in the direction opposite Gate 2, before turning back. Maximum lateral deviation consistently occurred ~ 75% along the segment between Gate 0 and Gate 1 and was significantly more negative (corresponding to deviation in the direction opposite from Gate 2) at larger angles (F_{2, 36} = 23.15, p < 0.001, η²_G = 0.12) and smaller distances (F_{2, 36} = 9.17, p < 0.001, η²_G = 0.05) (see Fig. 6A). There was also a significant but weak effect of orientation (F_{2, 36} = 5.09, p < 0.05, η²_G = 0.03) on maximum lateral deviation but none of the interactions were significant.

Fig. 6.

Mean maximum lateral deviation from the centerline for each angle and distance condition in Experiment 1 (A) and Experiment 2 (B). Overall speed between Gate 0 and Gate 1 for each angle and distance condition in Experiment 1 (C) and Experiment 2 (D). Drone heading (from centerline) at Gate 1 for each angle condition in Experiment 1 (E) and Experiment 2 (F). All plots in Fig. 6 are based on data from successful trials. However, plots of the same measures based on all trials (both successful and unsuccessful) were similar (see Fig. S6 in the Supplementary Material section). Error bars indicate 95% confidence intervals with between-subject variance removed.

There were significant main effects of both angle and distance on mean speed between Gates 0 and 1 (Fig. 6C). Mean speed significantly decreased with angle (F_{2, 36} = 21.48, p < 0.001, η²_G = 0.34) and increased with distance (F_{2, 36} = 15.71, p < 0.001, η²_G = 0.11). In addition, the angle x distance interaction was significant (F_{2, 36} = 3.69, p < 0.01, η²_G = 0.02), although the effect size was quite small. There was no significant effect of orientation (F_{2, 36} = 1.37, p = 0.27, η²_G < 0.01) and none of the other interactions were significant.

Lastly, the drone’s heading at the moment it reached Gate 1 was consistently biased toward Gate 2 (Fig. 6E). Heading angle was significantly greater at larger angles (F_{2, 36} = 81.06, p < 0.001, η²_G = 0.57) and smaller distances (F_{2, 3} = 37.01, p < 0.001, η²_G = 0.12). The main effect of orientation was not significant (F_{2, 36} = 2.53, p > 0.05, η²_G = 0.01), but the distance x angle (F_{4, 72} = 7.44, p < 0.01, η²_G = 0.04) and distance x orientation (F_{4, 72} = 4.30, p < 0.001, η²_G = 0.03) interactions were significant.

To summarize, in Experiment 1, subjects initially turned away from Gate 1 in the direction opposite Gate 2 by an amount that was systematically related to the angle and distance of Gate 2. They decelerated as they started to turn back toward Gate 1 and then accelerated as they came out of the turn. At the moment they reached Gate 1, their heading was biased toward Gate 2 by an amount that depended on Gate 2 angle and distance. In general, when Gate 2 was in a more extreme position (larger angle, shorter distance), subjects made larger adjustments to their trajectory such that their heading at Gate 1 was more closely aligned with Gate 2.

Subjects followed much more direct trajectories in Experiment 2 (Fig. 5D–F). Maximum lateral deviation (overall M = −0.11 m) was much less than it was in Experiment 1 (overall M = −2.22 m) and occurred later, at about 85% to Gate 1. There was a significant effect of angle (F_{2, 38} = 5.17, p < 0.05, η²_G = 0.03) but not distance (F_{2, 38} = 0.04, p = 0.96, η²_G < 0.01) or orientation (F_{2, 38} = 1.36, p = 0.27, η²_G = 0.01) (see Fig. 6B). Only the angle x distance interaction was significant (F_{4, 76} = 2.96, p < 0.05, η²_G = 0.02).

As in Experiment 1, mean speed significantly decreased with angle (F_{2, 38} = 46.61, p < 0.001, η²_G = 0.33) and increased with distance (F_{2, 38} = 31.04, p < 0.001, η²_G = 0.23) (see Fig. 6D). The angle x distance interaction was also significant (F_{2, 38} = 3.92, p < 0.01, η²_G = 0.02), but neither the main effect of orientation (F_{2, 38} = 0.19, p = 0.83, η²_G = 0.00) nor any of the other interactions were significant.

Heading at Gate 1 was less biased toward Gate 2 and less dependent on Gate 2 position than it was in Experiment 1 (Fig. 6F). Heading was significantly affected by Gate 2 angle (F_{2, 38} = 19.44, p < 0.001, η²_G = 0.11) but the effect size was much weaker compared to Experiment 1. Heading also significantly decreased as Gate 2 distance increased (F_{2, 38} = 44.03, p < 0.001, η²_G = 0.31) but was not affected by orientation (F_{2, 38} = 0.45, p > 0.05, η²_G = 0.001). There were no significant interactions.

To summarize, although the trajectories that subjects generated in Experiment 2 were qualitatively similar to those in Experiment 1, there were some major differences. Subjects made much smaller lateral deviations in Experiment 2, indicating that they followed more direct trajectories to Gate 1. They also waited longer until they were close to Gate 1 before making steering adjustments and decelerating. As such, when they reached Gate 1, their heading was more closely aligned with their initial heading. The largest steering adjustments were made after passing through Gate 1.

Discussion

Revisiting the hypotheses

In this section, we consider the support that the findings provide for each of the three hypotheses presented in the introduction: (1) Heading is attracted by a weighted combination of WP_N and WP_N+1; (2) Heading and path curvature are attracted by the constant radius path through WP_N and WP_N+1; (3) Actors adjust steering and speed so as to maintain the ability to steer through waypoints.

We found no evidence to support the weighted-combination of attractors hypothesis, which predicts that subjects steer directly toward WP_N and turn towards WP_N+1 shortly before reaching WP_N. In Experiment 1, subjects made large deviations in the direction opposite Gate 2 before turning back toward Gate 1. Even in Experiment 2, in which subjects followed more direct trajectories to Gate 1, they still made lateral deviations (albeit, of a smaller magnitude) in the direction opposite Gate 2. Since waypoints can only attract heading under this hypothesis, the observed behavior allows us to rule out this hypothesis.

The constant-radius path hypothesis states that heading and path curvature are attracted by the heading and path curvature of the constant-radius path through WP_N and WP_N+1. This hypothesis predicts trajectories similar to those produced by subjects in Experiment 1 but cannot capture the more direct trajectories that subjects produced in Experiment 2.

The trajectories produced by subjects across the two experiments are most consistent with a strategy of maintaining the affordance of steer-ability through waypoints. In Experiment 1, when the drone was more sluggish (i.e., when changes in direction and speed took longer), steering directly toward Gate 1 would leave subjects in a position from which it would be difficult to reach Gate 2. (Note that it was physically possible to reach Gate 2 after steering more or less directly toward Gate 1. One of the co-authors of this study, who has substantial experience with this task, confirmed this by completing a session with the drone parameters set to those of Experiment 1 while attempting to minimize lateral deviation between Gates 0 and 1. Their mean lateral deviation was − 0.27 m and they only missed gates on 8 of 216 trials.) Although it was physically possible to reach Gate 2 after directly approaching Gate 1, doing so required precisely timed changes in steering and speed, leaving a very small margin of error. By initially veering away from Gate 1 and turning back to approach Gate 1 from a more oblique angle, subjects reached Gate 1 while maintaining their ability to steer to Gate 2.

In Experiment 2, when the drone’s dynamics allowed for more rapid changes in direction and speed, the boundaries for task success expanded. Subjects could approach Gate 1 more directly without compromising their ability to reach Gate 2. This required making a sharp turn upon reaching Gate 1, but such turns were possible because the drone could decelerate and turn more rapidly. Taken together, the evidence strongly suggests that steering and speed are controlled in an anticipatory manner that takes into account both the layout of waypoints and the action capabilities of the actor.

Such behavior could be interpreted as evidence of the use of internal forward and inverse models of the vehicle in accordance with model-based control. However, theoretical arguments have been raised against accounts that rely on internal models and the strength of empirical evidence for model-based control has been questioned⁴⁵. Such arguments motivate attempts to capture the observed behavior without resorting to internal models. One such approach is affordance-based control, which posits that actors attempt to maintain the possibility of task success given the limits of their action capabilities and is supported by evidence from previous studies of braking³⁸ fly ball catching⁴¹ and locomotor interception^40,42. Of course, we cannot rule out other information-based control strategies beyond the two-attractors version of the behavioral dynamics model and the constant-curvature model. For example, future research may reveal other variations of the behavioral dynamics model that could capture the tendency of subjects in Experiment 1 to initially turn away from Gate 1 in anticipation of also having to steer through Gate 2. However, such a model would also have to capture differences in behavior of actors who are tuned to more sluggish versus more agile vehicles. This is beyond the scope of behavioral dynamics in its present form but could motivate an integration of the behavioral dynamics and affordance-based frameworks.

Effects of waypoint N + 1 angle, distance, and orientation

Although the observed effects of WP_N+1 angle and distance may seem intuitive, it is worth noting that they rule out the use of a simple heuristic-based strategy. For example, given that Gates 0 and 1 were always in the same position, subjects could have adopted a simple heuristic in which they detected whether WP_N+1 was on the left or right side, and made a stereotyped, initial deviation in the opposite direction before turning back. If this was the case, trajectories would have varied based on the side of WP_N+1, but not on its angle or distance. The fact that the angle and distance manipulations had such a strong and systematic effect on the control of both steering and speed provides evidence against the use of such a heuristic strategy. Instead, the results support the view that movements are continuously coupled to the available information about the two upcoming waypoints.

Orientation manipulation

In contrast to the effects of angle and distance, the effects of WP_N+1 orientation were weak and inconsistent. Across the various measures of behavior, very few main effects and interactions involving orientation were significant, and those that were significant had small effect sizes. This does not necessarily mean that humans are incapable of taking the orientation of future waypoints into account. It is possible that in the present study, the visual information about the orientation of Gate 2 was too difficult to detect when subjects were in a position to use it. For example, in the condition in which Gate 2 was at an angle of 30° and a distance of 40 m from Gate 1, it was ~ 87 m away from subjects when it first appeared. At that distance, it may have been too difficult for subjects to perceive whether the gate was at an orientation of −20°, 0°, or + 20°.

It is also possible that the range of orientation angles in the present study was not sufficiently large to require subjects to adapt their behavior to orientation. Interestingly, an analysis of the trajectories between Gates 1 and 2 revealed very similar behavior across orientation conditions within a given angle x distance combination (see Supplementary Materials). This suggests that subjects did not adapt their trajectories after Gate 1 to approach Gate 2 from a perpendicular angle. Nevertheless, subjects successfully passed through Gate 2 on 83% of trials in Experiment 1 and 96% of trials in Experiment 2. Perhaps if we had chosen a wider range of orientations, subjects would have adapted their behavior based on Gate 2 orientation.

Nevertheless, it is worth noting that this is the third recent study in which the effects of WP_N+1 orientation have been tested and found to be weak or not statistically significant. In the two previous studies^8,27 subjects guided a simulated drone through a series of gates at various relative angles, distances, and orientations. They fit a linear model with approach angle to Gate N as the outcome variable and the relative angular position and orientation of Gate N + 1 as the predictors. Angular position accounted for 23–31% of the variance in approach angle to Gate N but orientation accounted for less than 3% of the variance. Taken together, these findings suggest that if actors are capable of using information about the orientation of WP_N+1, the effects on behavior are not as robust as those of angle and distance.

Missed gates

Although subjects successfully passed through both gates on the majority of trials (see Fig. 4), performance was far from perfect, especially in the most extreme conditions (i.e., when Gate 2 angle was large and Gate 2 distance was short). How can such failures be understood from an affordance-based perspective, which asserts that actions are controlled so as to maintain the possibility of task success? After all, whenever subjects missed a gate, there was a point during the approach to that gate at which task success changed from possible to impossible. From an affordance-based perspective, such failures can result from motor variability – that is, adjustments to steering or speed that were intended to maintain the possibility of task success but did not because they were larger, smaller, slower, or faster than intended. Failures can also occur when actors are not perfectly tuned to the limits of the action capabilities, which must be learned through experience⁴⁶. For example, if an actor overestimates their ability to turn or slow down, they may perceive that the task is still within their capabilities when it is not. Both factors likely contributed to the higher rates of failure when Gate 2 was positioned at a larger angle and smaller distance. Such conditions required actors to make larger, more frequent, and precise changes to steering and speed, and to perform the task closer to the limits of their action capabilities.

Future research

The main conclusion of the present study is that when humans steer through multiple waypoints, they control their direction and speed so as to maintain their ability to steer through the next two waypoints. An important goal for future research is to formalize this strategy in the form of a model that could be simulated to generate trajectories through waypoints. Such a model would need to capture how trajectories are affected not only by the layout of waypoints but also the actor’s capabilities to change direction and speed. Fajen⁴⁷ outlined an approach to developing such a model, but further work is needed.

Another important goal for future research concerns how actors come to learn (and continually relearn) their action capabilities. How did subjects in Experiment 1 know well before reaching Gate 1, that to steer through that gate and the next one, they would need to initially turn away from and then back toward Gate 1? How did subjects in Experiment 2 know that they could steer more or less directly toward Gate 1 and still reach Gate 2?

In a previous study using a simulated braking task, Fajen⁴⁶ found that subjects were capable of rapidly recalibrating to changes in their action capabilities, and that such recalibration is driven by information in the optical consequences of one’s movements rather than the outcome of trial. However, the braking task used in that study was simpler in that subjects only had control over one degree of freedom – the rate of deceleration. In the present study, subjects had independent control over two degrees of freedom – speed and rotation rate. Furthermore, their ability to steer through waypoints (i.e., their affordances) depended on both degrees of freedom and how they interacted. For example, the controller in the present study was programmed such that rotation rate was independent of speed. As such, the ability to rapidly decelerate in Experiment 2 made it possible for subjects to make sharp turns, which in turn made it possible for them to steer directly toward Gate 1 and still reach Gate 2. Had the controller dynamics been programmed in such a way that speed and rotation rate covaried as they do in a car, subjects’ ability to make sharp turns would not have been affected by their ability to decelerate. Understanding how actors adapt to changes in their action capabilities and learn their affordances in these more complex scenarios is an important next step for research on this topic.

Supplementary Information

Below is the link to the electronic supplementary material.

Author contributions

A.J. and B.F. designed the experiment, A.J. conducted the experiment, A.J. and B.F. analyzed the data, A.J. and B.F. drafted, reviewed, and edited the manuscript.

Data availability

Raw data from both experiments are provided on the Open Science Framework here https://osf.io/ytf5e/?view_only=e0941ece6f564245ba0d0e856f7a0067.

Declarations

Competing interests

The authors declare no competing interests.