Brain–computer interface control with artificial intelligence copilots

Abstract

Motor brain–computer interfaces (BCIs) decode neural signals to help people with paralysis move and communicate. Even with important advances in the past two decades, BCIs face a key obstacle to clinical viability: BCI performance should strongly outweigh costs and risks. To significantly increase the BCI performance, we use shared autonomy, where artificial intelligence (AI) copilots collaborate with BCI users to achieve task goals. We demonstrate this AI-BCI in a non-invasive BCI system decoding electroencephalography signals. We first contribute a hybrid adaptive decoding approach using a convolutional neural network and ReFIT-like Kalman filter, enabling healthy users and a participant with paralysis to control computer cursors and robotic arms via decoded electroencephalography signals. We then design two AI copilots to aid BCI users in a cursor control task and a robotic arm pick-and-place task. We demonstrate AI-BCIs that enable a participant with paralysis to achieve 3.9-times-higher performance in target hit rate during cursor control and control a robotic arm to sequentially move random blocks to random locations, a task they could not do without an AI copilot. As AI copilots improve, BCIs designed with shared autonomy may achieve higher performance.

The only control source in most traditional motor brain–computer interfaces (BCIs), including computer cursor^1–3 and robotic arm^4–8 control, are decoded neural signals (Fig. 1a). However, many tasks we perform are goal oriented, where movements are directed towards target locations, such as search bars, buttons and icons on a computer (Fig. 1b), or physical objects such as cups, chips, door handles, keys on a table or blocks (Fig. 1c). In these cases, knowing the user’s goal, a deliberation between a finite number of possible goals, largely determines the movement. Once a goal is known, human actions are often stereotyped and could be aided via an artificial intelligence (AI) copilot. This approach is called shared autonomy, where a human user (pilot) and an AI copilot share control to achieve higher performance^9–14. But how do we infer the user’s goal? We reason that in addition to neural signals, there are other information sources, including task structure, historical movements and computer vision (CV), that can be used to infer a user’s goal and subsequently aid their movements (Fig. 1d).

Fig. 1 |. In an AI-BCI, AI copilots use task information to improve BCI performance.

a, In traditional BCIs, neural signals are decoded to control an end effector (for example, a computer cursor or robotic arm). b, In computer tasks, there can be a finite number of selectable targets. In web browsing, goals include editing the URL, changing the web page, closing the web page, entering a search query or hitting the search button. These goals could be inferred through CV. In this work, our cursor control task is a BCI benchmark, the centre-out 8 task, in which there are nine potential targets. c, In robotic arm tasks, finite goals often correspond to physical objects. An example may be choosing to drink water or grabbing a bag of chips, both of which could be inferred through CV. In this work, our robotic arm task is to pick and place blocks, where the goals are pick locations (blocks) and place locations (crosses). d, In an AI-BCI, the BCI and AI copilot combine to produce task-informed actions. The AI copilot uses information, including task priors, past movements, CV and neurally decoded signals, to infer potential goals and movements. It then helps to complete the task. e, The goal of the AI-BCI is to increase the task performance without additional cost to participants. Panels a and d created with BioRender.com.

For example, consider taking a drink from a cup on a table. An AI copilot, with (1) a camera that sees the cup and (2) task structure knowledge that humans typically grab cups to drink with their mouths can then process a (3) decoded motor command (such as move left; Fig. 1c) to infer the user wants to drink from the cup. The copilot would then help the user grab the cup with the appropriate force and bring it to the user to drink. We call this kind of architecture an AI-BCI (Fig. 1d). Similar kinds of reasoning could be made to infer goals for a computer task. For example, when using a search engine, one’s goal would probably be to select ‘search’ after entering a query into a search bar (Fig. 1b). For typing, past context gives strong information regarding the next character selection^15–17. What if a task does not have a structured goal (such as free drawing with a computer cursor)? Even in tasks without structured goals, an AI copilot can help avoid universally bad or unlikely actions (such as very erratic movements during drawing). If there is truly no available task information, then our proposed AI-BCI simplifies to a traditional BCI. However, we emphasize that in tasks with goal-directed movements, an AI-BCI can and should use additional contextual and task-specific information to outperform a traditional BCI.

Although an AI copilot could improve both invasive and non-invasive BCIs, we demonstrate that this shared autonomy approach can improve the performance of a non-invasive BCI system (Fig. 1e). Our goal is to enable a participant with paralysis to proficiently control computer cursors and robotic arms. Towards this end, we make two key contributions. First, we develop an electroencephalography (EEG) decoding architecture that uses the nonlinear features of a convolutional neural network (CNN) as the observations of Kalman filter (KF) similar to the recalibrated feedback intention-trained KF (ReFIT-KF)¹⁸, enabling online closed-loop decoder adaptation (CLDA)^19–21 (Fig. 2a). We characterize the performance of this decoder architecture across days. Second, we demonstrate two copilots, one for cursor control trained with deep reinforcement learning (RL; Fig. 2b) and another for robotic arm control with concurrent CV to automatically detect and grasp or place objects (Fig. 2c). We show that AI copilots significantly increase performance in a computer cursor and robotic arm task. We anticipate AI-BCI performance will continue to improve with advancing AI copilots, providing a complementary axis to improve BCIs beyond traditional approaches of decoder design^18,22–26 and neural adaptation^19,20,27.

Fig. 2 |. CNN-KF and AI-BCI decoding framework for centre-out 8 and robotic arm tasks.

a, CNN-KF architecture. The CNN uses the EEGNet architecture, whose final hidden state is the observation of a ReFIT-like KF. The KF is updated using CLDA. b, The cursor copilot is trained with deep RL and synthetic KF output velocities to drive cursor movements towards one of the potential centre-out goals. c, The robotic arm copilot detects pick-and-place locations with CV. The copilot has a simple policy: when close to a block, it picks it up, and when close to a cross, it places any held object. d, Open-loop training task. Text corresponding to one of four actions is presented for participants to execute (without paralysis) or attempt (with paralysis). The order of the four actions is randomized. e, Decorrelated closed-loop training task. On each trial, one of the four actions is placed at any of eight radial locations. The 2D decoded cursor is projected onto the one-dimensional line connecting the centre to the target. The correctly decoded action moves the cursor towards the target, whereas the opposing decoded action moves the cursor away from the target. The target is presented for 2 s and the cursor movement period is 5 s. Panels a and c created with BioRender.com.

A nonlinear and adaptive EEG decoder

We performed experiments with three healthy participants (H1, H2 and H4) and one spinal cord injury participant (S2) with T5 complete paraplegia and having no movement in his legs. All experiments were approved by the University of California, Los Angeles Institutional Review Board. Our first task is cursor control. We evaluated the performance on a common intracortical BCI cursor control benchmark, a centre-out task with eight targets (referred to as the centre-out 8 task) and a 500-ms hold time^2,18,21. To achieve continuous control, we trained decoders to translate EEG activity corresponding to four discrete behaviours into continuous cursor velocities. Our decoding approach combines deep learning with traditional CLDA approaches from invasive BCIs. The decoder was a CNN (the EEGNet²⁸) followed by a velocity KF performing ReFIT-like corrections^18,22, which we call a CNN-KF (Fig. 2a).

Our design was guided by two principles. First, deep learning architectures empirically achieve higher performance in both offline²⁸ and closed-loop experiments, where they outperform a traditional linear classifier²⁹. We incorporated an empirically successful CNN adapted from EEGNet²⁸. Second, we found that adaptation was important to update the decoder parameters in response to EEG non-stationarities. For example, it was possible to decode whether EEG activity was recorded in the first or second half of a session (Extended Data Fig. 1). As characterized below, biases frequently arose in fixed decoders. Neural networks are not straightforward to adapt in closed-loop experiments because noisy gradient descent steps are not guaranteed to improve the decoder. We instead froze the CNN, and used its nonlinear features as the observations of a linear KF. The decoded state of the KF was the cursor (x, y) velocity. We updated the KF parameters in CLDA using the parameter sufficient statistics^19–21 after ReFIT-like innovations¹⁸. This enabled us to (1) perform nonlinear decoding and (2) adapt the decoder in real time to the most recent data. Please refer to the ‘EEGNet (CNN) training’, ‘Kalman filtering’ and ‘CLDA’ sections, Extended Data Table 1 and Supplementary Sections 1–3 for additional details on this decoder architecture, training and CLDA.

To train the CNN-KF, participants first performed an open-loop training task. Participants were prompted with one randomly chosen action out of four total actions (Fig. 2d). These four actions corresponded to left, right, up and down movement classes, where the respective corresponding actions were left hand, right hand, both hands and feet (H1, H2 and H4); and left leg, right leg, both legs and still (S2). We then trained an initial ‘seed decoder’ from these data (Extended Data Fig. 2). This was not the final decoder because we found CNNs may decode features related to eye movement. In closed-loop control, eye movements are highly correlated with cursor and/or target location. For example, when the user performs a left motor action and the cursor correctly moves from the centre to the left-positioned target, the user also typically looks to the left side of the screen. Activity related to these eye movements is present in the EEG. To ensure that our performance was due to sensorimotor decoding and not eye movements, we performed a second training task in which the presented targets and kinematics were decoupled from motor intent (decorrelated closed-loop training task; Fig. 2e and Methods). We prompted a random motor intent (left leg; Fig. 2e) to a random target position (up right) and trained a CNN classifier to decode the motor intent. We confirmed through two analyses that our CNN decoded sensorimotor activity and not eye movements (Extended Data Fig. 3 and Supplementary Fig. 1). Because the CNN-KF freezes the CNN and only adapts the KF, this architecture and training paradigm yields a nonlinear, adaptable architecture that decodes sensorimotor activity without eye movement signals.

CNN-KF control across days

We evaluated the performance of the CNN-KF across days. Participants performed the centre-out 8 task on 5 separate days, distributed over anywhere from 9 to 36 calendar days (Supplementary Table 1). Similar to ref. 21, we initially evaluated the centre-out 8 performance with large targets that were progressively shrunk after the participant achieved good performance. Circular targets were positioned 8.2 cm from the centre target, and the initial target diameters were 7 cm, meaning that the target could be acquired by moving a minimum distance of 4.7 cm. Participants needed to hold the centre of the cursor over the target for 500 ms tosuccessfully acquire it. When the participant acquired targets at a rate higher than ten targets per minute at over 90% success rate, we increased the task difficulty by decreasing the target diameter (4.7 cm, 3.5 cm and 2.9 cm). All participants reached the 3.5-cm target size. The overall performance occasionally decreased when the target sizes were decreased. Nevertheless, our goal with these experiments was to quantify performance over weeks rather than maximize performance.

All participants achieved CNN-KF control of the BCI cursor across five separate sessions, each on a different calendar day (Fig. 3a,b). Two key metrics of performance in the centre-out 8 task are the percentage of successfully acquired radial targets and the average time to acquire each radial target. Participants were always able to complete the centre-out 8 task at a high success rate (median: 88% for S2 and 100% for all healthy participants; Fig. 3d). Trial time expectedly varied based on the target size, since smaller targets are more challenging to successfully hold over. Across all sessions, the median trial time was 10.1 s for S2 and between 7.0 s and 9.6 s for healthy participants (Fig. 3d). Because the target size is changing, we also quantified the Fitts information transfer rate (ITR), which factors the target size, success rate and acquisition time (Extended Data Fig. 4 and Extended Data Tables 2 and 3). It is possible that the performance could increase with longer-term CLDA²¹.

Fig. 3 |. Performance of CNN-KF for the centre-out 8 task.

a, Success percentage and trial time for participant S2 on five separate sessions in performing the centre-out 8 task. The KF statistics of the CNN-KF were incremented during all trials, but were only used to update the decoding parameters before and after each adaptation trial. Only trials from centre to radial targets are included, and each set of eight trials is compiled to a single data point. Lines show the Gaussian-smoothed averages. Decoder retraining and the one omitted day due to lower performance (when the participant felt paraesthesia) are indicated in magenta and cyan, respectively. b, Same data as a, but for the three healthy participants. c, Half-sided violin plots showing the distribution of x- and y-decoded velocities on each day of the centre-out 8 task for CNN-KF (solid line) and CNN only (dashed line). The CNN-KF adapts to velocity biases that emerge in the CNN over days. d, Success percentage and trial time across all days. The median of each box plot is shown in cyan. Box plots show Q1, median and Q3. Whiskers show the minimum and maximum of the data points that are within [Q1 – 1.5IQR, Q3 + 1.5IQR] (IQR, interquartile range), and outliers are shown outside of that range (n = 50, 86, 67, 69 blocks of 8 trials each). e, Saliency maps for participant with paralysis S2. Different hemispheric patterns are activated for each of the four classes. f, Saliency maps for healthy participant H1.

All healthy participants achieved plug-and-play performance, where the same decoder could be adapted day after day without recalibration on the decorrelated closed-loop training task (Fig. 3b). For S2, we decided to retrain the decoder on experimental session 3, which followed a gap of eight calendar days, and session 4, which followed a gap of six calendar days (Fig. 3a, magenta lines). We also observed that the decoder did not need to be retrained for session 5 (gap of 19 calendar days). Between experimental sessions 3 and 4, we omitted one session in which participant S2 experienced very strong phantom pain (Fig. 3a, cyan line). On this day, we were unable to train a high-performance decoder and S2 was unable to perform a full experimental session. We also observed that CLDA on the KF helped to remove biases in decoded velocity due to drifting EEG activity. We show the distribution of decoded velocities when using CNN versus CNN-KF across all participants and experimental sessions (Fig. 3c). We observed substantial drift in these distributions for some participants with the CNN (Fig. 3c, dotted lines), but the CNN-KF velocity distribution was relatively unbiased across sessions. Together, these results demonstrate that our decoder approach is performant across days, and that performance can even be sustained in the same decoder through adaptation, although CNN recalibration may sometimes be required.

What neural features guided proficient decoding? To answer this question, we computed spatial saliency maps that exhibit which EEG electrodes contributed most for each motor class. These saliency maps are based on computing the gradient of the cursor’s velocity update with respect to the magnitude of each electrode’s activity (Methods). In EEG activity, movement leads to event-related desynchronization, resulting in lower alpha and beta power in contralateral motor areas during movement³⁰. Active regions therefore tend to demonstrate a decrease in EEG activity. We found that as a result of the chosen action sets, the activation of the left or right hemisphere near the motor cortex largely drove cursor movements. The left hemisphere was more active during right movements, the right hemisphere was more active during left movements, both hemispheres were active during up movements and both hemispheres were less active during down movements (Fig. 3e,f shows the saliency maps for S2 and H1, respectively). In summary, these experiments demonstrate that a participant with paralysis can achieve two-dimensional (2D) EEG cursor control through the hemispheric modulation of cortical activity.

Cursor control copilot more than doubles target hit rates in centre-out 8 task

We hypothesized that an AI copilot that helps infer user goals could increase the task performance. In the centre-out 8 task, there were only nine possible goals, corresponding to nine potential target locations. This imitates, in spirit, how many applications often have only a handful of selectable goals (such as buttons on a computer screen (Fig. 1b) or keys on a keyboard³¹). Our goal was to build a copilot that can infer the user’s goal and help acquire it. Although the possible goals were known a priori in the centre-out 8 task due to the task structure, for tasks such as using a computer browser, the possible goal locations can be inferred via CV.

We trained a deep RL copilot with proximal policy optimization³² to assist the user in performing the task. This copilot observed the BCI decoder trajectories and velocities, and used these inputs to infer what goal the user was trying to acquire, aiding movement towards that goal. The copilot input was the CNN-KF-decoded velocities and cursor positions throughout the trial (Methods and Extended Data Table 4). A key advantage of using the CNN-KF-decoded velocities, and not the high-dimensional EEG, was that this enabled us to train the copilot in a simulated environment in which we generated synthetic CNN-KF velocities, removing the need for large experimental datasets. The copilot outputted a probability over potential goals, and guided the cursor velocity through a charge field proportional to the probability³¹. The copilot therefore influenced the velocity of the cursor at every time step. We trained this copilot in a custom environment in which the copilot received positive rewards for movement towards the correct goal and negative rewards for movement away from the correct goal (Methods and Extended Data Fig. 5).

We performed our most challenging task condition in which targets were 2.3 cm in diameter with a 500-ms hold time. Participants performed the task in subsessions of 10–15 min without and with the copilot (interleaved and without copilot first), and without adaptation. Although the copilot was trained in a simulated environment, we found it generalized to increase the closed-loop BCI centre-out 8 task performance. Distance-to-target plots¹⁸ averaged across all the single trials summarize the performance improvement (Fig. 4a); the cursor copilot reduced the average distance to target faster than without the copilot. For example, in S2, the AI-BCI achieved the same average distance to target as the BCI at end of trial (20 s) within only 5 s. Ultimately, this had the effect of significantly reducing the time it took to complete a trial (Fig. 4b). The copilot additionally increased the success percentage of correct target acquisition. S2 and all healthy participants achieved a significantly higher target success rate with the copilot (Fig. 4c). Together, the increased success rate and decreased trial time means the cursor copilot increased the average goal acquisition rate of the task by a factor of 2.1 times in healthy participants and 3.9 times in S2 (Fig. 4d).

Fig. 4 |. An AI copilot improves centre-out 8 performance.

a, Distance-to-target plots averaged across all trials on the centre-out 8 task for S2 and all healthy participants (H). The bolded line corresponds to the average dial-in time. b, Average trial time of all radial target trials (two-sided Wilcoxon rank-sum test; *P < 0.05, **P < 0.01, ***P < 0.001). Box plots show Q1, median and Q3. Whiskers show the minimum and maximum of data points that are within [Q1 – 1.5IQR, Q3 + 1.5IQR] (n = [4, 8], [56, 57] 8-trial blocks ([no copilot, copilot]), P = 0.00658 for S2, P = 3.46 × 10⁻¹⁵ for H). c, Average success percentage of radial target trials (n = [4, 8], [56, 57] 8-trial blocks ([no copilot, copilot]), P = 0.0108 for S2, P = 4.78 × 10⁻⁹ for H). d, Average hit rate (successful acquisition of radial target goals), that is, goals per minute (n = [4, 8], [56, 57] 8-trial blocks ([no copilot, copilot]), P = 0.00658 for S2, P = 1.18 × 10⁻¹³ for H). Centre targets, which occurred after every radial target, are not included. e, Average time to first touching the target for radial trials (n = [23, 58], [430, 426] trials ([no copilot, copilot]), P = 0.0917 for S2, P = 0.477 for H). f, Average dial-in time for radial trials (n = [23, 58], [430, 426] trials ([no copilot, copilot]), P = 4.48 × 10⁻⁶ for S2, P = 2.02 × 10⁻⁴⁷ for H). g, Average path efficiency (n = [4, 8], [56, 57] 8-trial blocks ([no copilot, copilot]), P = 0.00658 for S2, P = 1.50 × 10⁻¹⁹ for H). h, Summary scatter plot of the mean trial time, dial-in time and first touch time, with the copilot generally improving performance. Each dot represents a subsession of 10–15 min. i,j, Average trajectories towards each radial goal with and without the copilot for S2 (i) and H1 (j). The copilot helps improve path efficiency towards the goal. NS, not significant.

How did the copilot achieve this increased performance? First, we note that the copilot was able to correctly infer the participant’s goal on every successful trial. If the copilot did not, the AI-BCI would never acquire the correct goal. Second, the copilot significantly decreased the trial time. This could be achieved through two means: ballistic control (time it takes to first touch a target) or fine control (time it takes to dial-in to the target) could be improved. Although we found that the median time to reach a target was less using the copilot, this difference was not statistically significant (Fig. 4e). However, there was a significant reduction in dial-in time to acquire the goal, consistent with the cursor copilot aiding the acquisition of goals. In particular, although a traditional BCI had difficulty dialling in to a small target, the AI-BCI helped in significantly reducing this dial-in time from a median of 4.15 s to a median of 0.05 s for S2 (Fig. 4f). A breakdown of trial time reduction is shown in Fig. 4h. Third, we found that the cursor copilot also helped to make path trajectories more efficient, that is, closer to a straight line towards the target (Fig. 4g). Average trajectories for all centre-out reach conditions are shown in Fig. 4i,j, reflecting straighter trajectories when using the copilot compared with no copilot. Together, these results show that a cursor copilot is able to correctly infer user goals and aid their completion, increasing goal acquisition rates on a centre-out 8 task by 2.1 times to 3.9 times.

A robotic arm copilot enables proficient control in a sequential pick-and-place task

Although some goals may be task defined, they may also appear in random locations. In these situations, AI-BCIs can also use CV to infer goals. We used this approach to develop a robotic arm AI copilot, used in two pick-and-place tasks (Fig. 5a–c). The copilot used CV to infer the location of potential goals and keep track of the objects’ locations (Fig. 5d). The goals were pick locations (blocks) and place locations (crosses). We inferred the locations of all blocks and crosses using a camera (Intel, RealSense D455) paired with a foundation model for open-set object detection (Grounding DINO)³³. Our robotic arm copilot was programmed to identify all goal locations and aid control in real time by precisely grasping a block when the robotic arm was within 2.54 cm of the block and placing any held block when the robotic arm was within 2.54 cm of a cross, or the block’s starting location. This robotic arm copilot therefore aided in executing grasp-and-place actions in the vicinity of CV-inferred goals.

Fig. 5 |. An AI copilot improves the control of a robotic arm for pick-and-place tasks.

a, Setup of the robotic arm experiments. Each participant was seated in front of a table containing the robotic arm and a monitor showing the control orientation. b, Paired pick-and-place task. Participants must move a random block to a random target cross on each trial using a hover strategy or with the CV copilot. c, Sequence pick-and-place task. On each trial, the locations of target crosses were scattered, and blocks were randomly dropped onto the table. Participants were then presented with a random sequence of four block–target pairs, and were instructed to pick and place each block on its respective target in the order given. d, CV object detection using Grounding DINO for the paired (left) and sequence (right) pick-and-place tasks. e, Trial-level success percentage for hover and copilot conditions (n = 4, averaged across n = 32, 32, 16 and 32 trials for participants S2, H1, H2 and H4, respectively). f, Total time required to successfully pick and place blocks on their respective targets. No successful trials were recorded for S2 with the hover condition. Box plots show Q1, median, and Q3. Whiskers show the minimum and maximum of data points that are within [Q1 – 1.5IQR, Q3 + 1.5IQR], and outliers are shown outside of that range (n = [0, 10], [10, 25], [3, 8] and [12, 30] trials for participants S2, H1, H2 and H4, respectively, for [hover, copilot] conditions). g, Bar chart showing the percentage of blocks that were placed on the correct target at the end of each trial of the randomized pick-and-place task. h, Box plot showing the total time for all trials (n = 7, 15, 10, and 10 trials for participants S2, H1, H2, and H4, respectively). i, Stacked bar chart showing the proportion of correct, out-of-order placements at the correct location, error, and error correction pick-or-place actions. j, Trajectories showing S2’s successful completion of the task.

We first evaluated the extent to which CV aided performance. We compared a traditional BCI and an AI-BCI in a pick-and-place task, where four differently coloured blocks were placed in a line and four differently coloured crosses were placed in another line (Fig. 5b). We called this the robotic-arm paired pick-and-place task (Methods). On each trial, participants were asked to pick up a random block and move it to a random place location. The traditional BCI was able to give a command to pick or place via hold-time selection, whereas the AI-BCI relied on CV and the decoded location of the robotic arm to help execute the pick-and-place actions. Trials were only successful if the participant picked up the correct block and placed it at the correct cross. When participants were aided by the copilot, their rate of successful trials increased for every participant (Fig. 5e). The average time for successful trials also decreased for every participant when using the copilot (Fig. 5f and Supplementary Fig. 2). Participant S2 was unable to perform any successful trials without the robotic CV copilot.

We next used this copilot to demonstrate proficient control for a sequential pick-and-place task. We performed a task in which we randomly dropped four blocks and four crosses on a table (Fig. 5c). We then generated a random sequence pairing each block with a cross, and asked the participant to pick and place each block at its respective cross in order. We called this the robotic-arm sequence pick-and-place task (Methods). We counted errors when users picked up any block that was not the next block or placed blocks at incorrect locations. These errors could be ‘corrected’ by replacing the block at its original location, and picking up an incorrectly placed block and placing it at its assigned location, respectively. All participants successfully performed the task, placing each block at the final correct location either 93% (S2) or 100% of the time (healthy participants; Fig. 5g). The median time it took to complete a sequence (requiring eight goals) was 392 s for S2 and 136 s for healthy participants (Fig. 5h and Supplementary Fig. 2), and the majority of actions were correct actions or corrected an incorrect action (Fig. 5i). This sequence time therefore includes the time spent making errors and correcting them. Supplementary Video 1 shows S2 successfully performing this task, with trajectories plotted in Fig. 5j. Together, these results demonstrate that participants are able to use a copilot to autonomously and proficiently perform a sequential pick-and-place task with a non-invasive EEG AI-BCI.

Discussion

Our results demonstrate that shared autonomy can significantly increase the BCI performance in cursor control and robotic arm tasks. This approach provides a complementary axis to increase BCI performance, and as AI copilots improve, this has the potential to further improve the AI-BCI performance. One factor for improved performance was that the AI helped the user to achieve goals. In the centre-out task, the AI copilot significantly reduced the dial-in time, helping the user select an inferred target. In the robotic arm task, the AI copilot helped the user to grasp and place blocks. These approaches are complementary to other techniques to reduce the dial-in time^{3,18,21,34–36} that could be combined with AI copilots.

Our performance was also due to stable real-time EEG decoding through an adaptive CNN-KF. For our tasks, we found that we only needed to reliably decode four classes. The four classes we chose to decode predominantly reflected the different combinations of hemispheric activity near the motor cortex. However, EEG performance also exhibits substantial differences in performance across participants^37,38. Further, it may be possible that in patients with amyotrophic lateral sclerosis or other diseases, there may not be strong activation for movement-related actions due to motor neuron death. For other participants who do not exhibit decodable activity for the four classes we prompted, it may be necessary to identify different behaviour sets that can produce four decodable classes. There are many candidates to try to decode four classes, including different limbs, wrist motor imagery^39,40 and cognitive imagery^41,42. Future work should characterize how reliably four classes can be decoded based on disease condition or severity of paralysis, and increase the decoding robustness. It is also unclear whether S2 performed worse than healthy participants because of his injury or because the control signals were derived from the legs instead of the hands. Although we suspect that his injury plays a role in worse performance, future work may also determine which actions result in consistently higher decoding accuracy. Relatedly, we note that we were unable to achieve good decoding performance with S2 on one experimental day. S2 anecdotally commented that the decoding performance is inversely related to pain and paraesthesia sensations, and achieved the lowest performance when pain and paraesthesia were especially noticeable. As pain and paraesthesia may negatively impact BCI decoding, future work should investigate how to increase EEG robustness to these factors.

We emphasize that AI copilots help reduce the required number of neurally decoded degrees of freedom (DOFs) to perform relatively high-dimensional tasks. The use of AI copilots allows users to perform tasks beyond the complexity of a previously reported randomized grasp task⁴³. In that study, users grasped objects by holding the robotic arm’s position over each randomly placed block for 2 s, with block positions explicitly made known in their control paradigm (in contrast to how our block positions were inferred via CV). For more complex tasks than these, including interacting with particular objects, training an AI copilot to handle such interactions should reduce the DOF required to handle objects (for example, allowing the AI copilot to determine how to control a robotic hand to correctly grasp a cup). Along these lines, prior studies demonstrate robots that execute actions corresponding to classified goals^44,45. One limitation of this approach is that the copilot does not simultaneously share control with the user, but rather executes a set of actions corresponding to a goal. This means (1) online error correction by the human is more challenging, (2) task performance is upper bounded by the copilot, (3) there may be frustration and lack of engagement due to less direct control by the human and (4) the AI-BCI does not reduce to a traditional BCI in the absence of a task structure. This contrasts our shared autonomy approach, where the user and copilot are constantly interacting and, in the absence of directed goals, the AI-BCI simplifies to a traditional BCI.

We anticipate that future copilots may provide substantially more assistance. This is the goal of shared autonomy, which aims to use copilots to help increase human performance on various tasks by sharing control through task execution^9–13. Prior work on shared autonomy for robotic control predominantly focused on using joystick and similar input modalities to teleoperate high-DOF robots with low-DOF inputs^46–52. One of the key challenges in translating these approaches to BCI control is that neural decoding is relatively noisy with potentially slower temporal resolution. Future work should develop more advanced copilots that assist in executing robotic arm trajectories towards objects and precise manipulation based on object dynamics^12,53. When there is no information about the movement to be performed, an AI copilot can still help avoid universally bad states. We also believe shared autonomy approaches should maintain user autonomy to avoid frustration and deleterious effects on performance. We recently developed a control-sharing approach called interventional assistance, which may facilitate user–copilot cooperation by improving user autonomy and performance¹⁴. This approach, defining an intervention function, may help to improve cooperation—and stave off competition—between the user and the copilot. Finally, shared autonomy may enable more advanced robots, including those trained with large-scale datasets^54–59, to act as copilots, enabling AI-BCIs to carry out even more complex movement tasks.

Shared autonomy can improve both non-invasive and invasive BCIs. We note that invasive BCIs outperform non-invasive BCIs, including this study. We compared the Fitts ITR achieved in this study with prior invasive electrocorticography²¹ and Utah array BCIs² (Extended Data Table 3). We caveat that these Fitts ITR comparisons are imperfect, particularly when tasks have differences; for example, tasks with shorter hold times should achieve higher Fitts ITR. Further, because Fitts ITR does not directly take into account the success rate, it is unsuitable as a stand-alone metric of performance. We also emphasize that it was not a goal of this study to optimize the Fitts ITR; we did not attempt to find task parameters that would yield higher Fitts ITRs. Future work may aim to standardize tasks. Future work may also use AI copilots to increase the performance of invasive BCIs.

Recent studies have demonstrated high-performance intracortical BCI communication by decoding speech^60–62. Our work is complementary: we focus on the restoration of movement, for two reasons. First, many with paralysis (for example, spinal cord injury below C4) do not lose speech function but would be helped by motor BCIs. Second, many tasks we perform are fundamentally motor based, including controlling computer cursors and robotic arms. Although speech to movement is possible, it is less precise, responsive and discreet. For example, controlling a computer cursor through speech commands (like move left, left, up, up, down and right) may be prone to error with relatively crude online correction.

In general, we expect AI copilots to perform better when they better infer user intent; if they cannot decode user intent, then they will not be able to help complete the goal. Future work may therefore build better decoders and thereby increase performance. For example, it may be possible that large-scale EEG recordings may increase the performance of decoders, as observed in the decoding language⁶³ and in electromyography motor decoding⁶⁴. Because EEG is non-invasive, it is relatively easier (compared with invasive BCIs) to collect larger-scale datasets that could result in new non-invasive BCI foundation models. This approach may both reduce calibration time and increase performance. Future non-invasive BCIs may also record other non-invasive signals, including eye gaze^65,66, posture and electromyography⁶⁷. Although our results show that sensorimotor EEG signals can drive non-invasive BCIs, decoding other available non-invasive signals may significantly increase performance, especially in participants in which EEG modulation may be relatively weak.

Methods

EEG recording

We recorded EEG signals from a 64-channel cap at a sampling rate of 1,000 Hz with the eego rt amplifier (ANT Neuro, Waveguard Original, International 10/10 layout). We measured the impedance of each electrode at the start of every experimental day to ensure that none was over 200 kΩ, with a target of 20 kΩ or less. M1 and M2 electrodes were omitted from the recording. EOG, Fp1, Fpz and Fp2 electrodes were recorded but omitted from decoding because they contain strong eye movement information. We filtered the EEG data (for details, see ‘Data preprocessing’), which were then sent as inputs to our real-time system, named real-time synchronous Python (RaSPy).

Real-time experimental setup

Our open-loop task operated at 100 Hz (10-ms ticks) and our closed-loop tasks operated at 20 Hz (50-ms ticks). These tasks are described below. In tasks with real-time decoding, we decoded the most recent 1,000 available samples (1 s) of EEG data. Cursor control tasks were displayed on a 20-inch monitor (48 cm × 27 cm) with a refresh rate of 60 Hz. Each participant was seated with their eyes positioned approximately 55–76 cm from the screen. We recorded the eye gaze position at 60 Hz (Tobii Pro Nano). The eye tracker was calibrated at the start of each day and whenever large changes in seating position occurred between sessions. Robotic arm control tasks were performed using the Panda arm (Franka Emika) running at 1,000 Hz.

Tasks

Participants performed up to five tasks. Two tasks, open-loop text and decorrelated closed-loop tasks, were used to train the decoders. One task, centre-out 8 target selection, was used to quantify the decoder performance for cursor control. Two tasks, robotic-arm paired pick-and-place and robotic-arm sequence pick-and-place tasks, were used to quantify the decoder performance for robotic arm control. Participants were asked to sit up straight and rest their hands in a comfortable position for all tasks.

Open-loop text (referred to as the open-loop task).

We presented text corresponding to four actions. We randomized the presentation of the actions in groups of four. The participant was instructed to repetitively perform an action corresponding to the presented text. The presented texts for the healthy participants were left hand, right hand, both hands or feet. Healthy participants overtly moved the limb corresponding to the presented text (for more details on why we used overt movements in healthy participants, please see the ‘Experimental protocol’ section). The presented texts for the spinal cord injury (SCI) participant were left leg, right leg, both legs or still. The SCI participant was asked to attempt to move the paralysed limbs or to remain still during the respective prompts. The SCI participant displayed no observable limb movements.

Decorrelated closed loop (referred to as the decorrelated task).

The goal of this task is to decorrelate eye movements from motor actions. This task utilized a decoder trained from the open-loop task that controlled the position of a cursor.

At the start of a decorrelated trial, a cursor was displayed at the centre of the screen. A text prompt (from the same set used in the open-loop task) appeared inside a rectangular box at one of eight ordinal target locations. The text prompt and its target location were uncorrelated. For example, left leg could appear at any of the eight locations for that trial. The cursor moved towards the target if the decoded action was the same as the prompted action. For example, if the left leg was at the up right target (45°; Fig. 2h), then any left cursor movements moved the cursor up right. The 2D velocities were constrained onto the straight line segment connecting the centre of the screen to the outer edge of the target to restrict its movement.

All the decoders used for closed-loop control were trained using data from the decorrelated task to minimize the influence of eye position on sensorimotor decoding.

Centre-out 8 target selection (referred to as the centre-out 8 task).

Centre-out 8 task data were collected in subsessions of at least 10 min, until a multiple of eight centre-out trials was completed. Participants were allowed to take breaks between subsessions. Most breaks were between 5–15 min. During the centre-out 8 task, the user controls a cursor to acquire eight targets equally spaced on the circumference of a circle. The position of the cursor was bounded to a square with the x position between [–1, 1] units and the y position between [–1, 1] units (total side length, 2 units). The target diameter varied from 0.6 units (7 cm) to 0.2 units (2.3 cm). To acquire a target successfully, the centre of the decoded cursor had to be held contiguously over the target for 500 ms. The trial timed out if the participant was not able to acquire the target within 24 s. After each centre-out target, the next prompted target was at the centre of the screen. Statistics for the centre-out 8 task are only computed on the eight centre-out trials to be agnostic to target location and are averaged over each set of eight trials.

Robotic-arm paired pick and place (referred to as the robotic paired task).

We set up four place locations (crosses, spaced approximately 13 cm apart along a line) and four pick locations (blocks of 2.9 × 2.9 × 2.9 cm³, also spaced approximately 13 cm apart, and approximately 28 cm away from each cross). We then generated a trial in which a randomly chosen block was paired with a randomly chosen cross. To successfully complete a trial, the participant had to control the robotic arm to (1) move to the block, (2) grasp the block, (3) move the block to the correct cross and (4) place the block on the cross. The trial was otherwise failed. The participant had 40 s to move the arm to grasp the block before the trial was failed. If the participant grasped any block, they would then have 40 s to place the block before the trial was failed. The time during which the arm initiated but had not yet completed an action did not count towards the 40-s limit. Trials were performed in sets of eight. For the first four trials of each set, participants performed all four trials with (or without) an AI copilot. For the last four trials of each set, participants received the same prompt as the first four trials of the set, and performed the trials without (or with) an AI copilot. Whether each set began with or without an AI copilot was randomly determined (by a coin flip with P = 0.5). Participants were informed whether they performed the task with or without an AI copilot so that they could adjust their control strategy accordingly.

Participants controlled the 2D endpoint of the robotic arm above the blocks and crosses, with altitude as a function of distance to the closest block/cross. Participants were shown a live camera view from an overhead camera of the workspace (Fig. 5a). Participants were also given the option to rotate this top-down view of the workspace; S2 chose up and H1, H2 and H4 chose left to correspond to the direction pointing away from the base of the robot. The maximum 2D speed of the robotic arm was set to 5 cm s⁻¹. The robotic arm would attempt a pick/place whenever it remained in any 6-cm radius circle for 4 s (hover condition), or was within 2.54 cm of a relevant goal’s centre (CV AI copilot condition). We tested several radii and times for the hover condition, choosing the combination that was subjectively preferred. In the hover condition, the average location was continuously updated, allowing for minor fine-tuning of the final pick/place location. The arm could attempt but fail to pick up or place a block if there was no block to be picked up, if the block was misaligned or if there was a collision with a block. The blocks could be placed anywhere on the table, though the CV copilot would only place blocks on a target.

Robotic-arm sequence pick and place (referred to as the robotic sequence task).

During the sequence pick-and-place task, the participant controlled a robotic arm to pick and place four coloured blocks (pick locations) onto four crosses (place locations). The four crosses were placed on the table without any fixed arrangement. The four blocks were then simultaneously dropped on the table so that their positions were random, and only moved if a block was out of bounds of the workspace or was too close to any other block or target (approximately 5 cm). A random sequence was generated, assigning each of the four blocks uniquely to the four place locations in a particular order. The subject had to control the robotic arm endpoint to place each block at each cross in the specified correct order. If the user picked up a block out of order, they were instructed to replace the block at its original location. If this was subjectively too difficult or not possible (due to the occupation of place locations), they were instead instructed to place the out-of-order block at its respective target location. Trials ended when all four blocks were placed, when 40 s of moving time elapsed and the trial timed out, or when users declined to finish the trial. On some sequence task trials, the gripper dropped the block or failed to place the block due to improper calibration. These trials were excluded from the analysis.

Experimental protocol

Participant selection.

All experiments were approved by the University of California, Los Angeles Institutional Review Board, and we received informed consent from all participants. S2 gave consent to release the images and videos in this manuscript. Three healthy participants and one participant with paralysis with SCI (paraplegic, T5 complete) participated in the full duration of this study. As stated in the ‘Discussion’ section, controlling the CNN-KF used four decodable classes to control right, left, up and down 2D cursor movements. Furthermore, participants exhibited varying levels of EEG modulation for particular actions³⁸, something that can be affected by several factors including mind–body awareness³⁷. Although any four decodable classes could be used to control our BCI, we only tested the set of prompted behaviours described in the task description of open-loop text (for healthy: left hand, right hand, both hands, legs; for SCI: left leg, right leg, both legs, still). If a participant did not achieve high decoding accuracy on these four actions from the open-loop task, we did not perform subsequent experiments to identify a behaviour set that generated four decodable states, as this search was open ended. Rather, we decided to only perform experiments on participants whose activity for the prompted action set could be decoded by a CNN-KF.

Training data collection.

Each participant performed the open-loop task for approximately 20 min. EEG activity corresponding to each of the four possible motor actions was used to train a CNN-KF (see the ‘EEGNet (CNN) training’ and ‘Kalman filtering’ sections). There were two important differences between the open-loop and closed-loop cursor control subsessions. First, there was real-time feedback on the decoder’s output in the form of the cursor movement during closed-loop cursor control. Second, the participant’s eyes followed the cursor during closed-loop cursor control, resulting in eye movement correlated to the task. To ensure that the KF was not adapting to eye-movement-based EEG artefact during the cursor control subsessions, we included a decorrelated closed-loop subsession so that the EEGNet CNN learned to ignore the eye movements.

Analysis showed that each participant’s eye movements during the decorrelated task were minimally correlated with cursor and target information (Extended Data Fig. 3 and Supplementary Fig. 1), thereby minimizing the effective contribution of eye movements to decoding. A new EEGNet was trained using the data from the decorrelated subsession. This trained EEGNet was used in the following centre-out-8 closed-loop subsessions. To control the movement of the cursor, the SCI participant was asked to do the action tasks: left leg for left, right leg for right, both legs for up, and still for down. The healthy participants moved the left hand, right hand, both hands and legs to move the cursor to the left, right, up and down, respectively.

Although our SCI participant did not have any overt movement of his paralysed limbs during BCI control, we instructed healthy participants to overtly move their hands, in contrast to imagined movements. Overt movements have been previously used in several monkey BCI models^18,36 that were successfully translated to human participants with paralysis^2,3. Our choice of overt versus imagined movements for healthy participants centres around different considerations in using non-disabled participants to model participants with paralysis, something also considered in intracortical BCI animal models⁶⁸. Overt movements have proprioceptive feedback⁶⁹. Imagined movements have low variance along dimensions of neural activity that produce movement, whereas overt movements have high variance in these dimensions⁷⁰. When attempting movements, participants with paralysis generate activity that would have produced movement without their injury or disease. We therefore decoded the EEG activity corresponding to overt movements (healthy participants) or attempted movements with no overt behaviour (participant with paralysis). Please note that all of our BCI experiments worked with S2, who displayed no overt movement of his paralysed limbs during BCI control.

Data preprocessing

We filtered the EEG data using a fourth-order Butterworth low-pass filter (cutoff, 40 Hz) cascaded with a fifth-order Butterworth high-pass filter (cutoff, 4 Hz) and notch filters at [60 Hz, 60 Hz, 120 Hz, 180 Hz] with quality factors of [10, 4, 5, 2], respectively. Following this, we applied a band-pass fifth-order Butterworth filter to each of the remaining channels, allowing frequencies between 4 Hz and 40 Hz to pass through. This range contains the typical brain oscillations of interest, such as theta, alpha and beta waves, as well as attenuating slow drifts and high-frequency noise. We then used a next-nearest-neighbour Laplacian spatial filter over the decoded electrodes. Finally, we normalized the data for each channel to have unit root mean square for faster convergence in the subsequent neural network. This normalization aims to increase robustness to signal amplitude variance that occurs across recording sessions.

EEGNet (CNN) training

We used a CNN, EEGNet²⁸, to classify the EEG signals. EEGNet parameters, which were the same for all the participants, are listed in Extended Data Table 1. During online decoding, the most recently available 1,000 samples of EEG were resampled to 100 Hz (scipy.signal.resample) before being input to the CNN.

Additionally, for compatibility with the KF so that error distributions more closely resembled the Gaussian distribution, the EEGNet models were trained with a scaled mean square error (MSE) loss with targets corresponding to one-hot vectors of each class. The MSE loss was scaled by a factor of 100 to improve training stability. Accuracy for models trained with the one-hot MSE loss was defined by labelling the prediction of each data point as the argmax over the four classes. This method is equivalent to a linear discriminant analysis classifier with fixed mean and covariance and with trained feature maps. Data used to train the EEGNet were also used to train a seed KF for subsequent closed-loop subsessions.

Kalman filtering

The discrete-time linear KF with no inputs assumes a state-observation linear dynamical system defined as

$${\mathbf{s}}_t=\mathbf{A}{\mathbf{s}}_{t-1}+{\mathbf{w}}_t,$$ (1)

$${\mathbf{y}}_t=\mathbf{C}{\mathbf{s}}_t+{\mathbf{q}}_t.$$ (2)

In this instance, ${\mathbf{s}}_t$ is the cursor (or robotic arm) kinematic state vector at time step $t$ representing the position and velocity; ${\mathbf{y}}_t$ is the observation vector (EEGNet hidden state); $\mathbf{A}$ and $\mathbf{C}$ are the state transition and observation matrices, respectively; and ${\mathbf{w}}_k$ and ${\mathbf{q}}_k$ are the state and observation noise, respectively. The KF decodes according to

$${\widehat{\mathbf{s}}}_t=\left(\mathbf{A}-{\mathbf{K}}_t\mathbf{C}\mathbf{A}\right){\widehat{\mathbf{s}}}_{t-1}+{\mathbf{K}}_t{\mathbf{y}}_t,$$ (3)

where ${\mathbf{K}}_t$ is the Kalman gain at time $t$. The linear KF is minimum MSE optimal when noise ${\mathbf{w}}_t$ and ${\mathbf{q}}_t$ are Gaussian and independent of time.

To adapt this formulation to four-class decoding, we separate velocity into two components per axis: $v^{+x}$, $v^{-x}$, $v^{+y}$ and $v^{-y}$. A positive $v^{+x}$ corresponds to a positive $x$-axis velocity, whereas a positive $v^{-x}$ corresponds to a negative $x$-axis velocity. The kinematic state vector is then defined as

$${\mathbf{s}}_t={\left[p_t^x,p_t^y,v_t^{+x},v_t^{-x},v_t^{+y},v_t^{-y},1\right]}^{\top },$$ (4)

for $x$ position $p^x$ and $y$ position $p^y$ (a.u.). At each time step, we additionally clip the $L_1$ norm of velocity components ${\left[v_t^{+x},v_t^{-x},v_t^{+y},v_t^{-y}\right]}^{\top }$ to be at most 1.

We set $\mathbf{A}$ and $\mathbf{W}$ as follows:

$$\mathbf{A}=\left(\begin{array}{ccccccc}\mathbf{1}& \mathbf{0}& \mathbf{\gamma }\cdot \text{d}\mathbf{t}& -\mathbf{\gamma }\cdot \text{d}\mathbf{t}& \mathbf{0}& \mathbf{0}& \mathbf{0}\\ \mathbf{0}& \mathbf{1}& \mathbf{0}& \mathbf{0}& \mathbf{\gamma }\cdot \text{d}\mathbf{t}& -\mathbf{\gamma }\cdot \text{d}\mathbf{t}& \mathbf{0}\\ \mathbf{0}& \mathbf{0}& \mathbf{a}& \mathbf{0}& \mathbf{0}& \mathbf{0}& \mathbf{0}\\ \mathbf{0}& \mathbf{0}& \mathbf{0}& \mathbf{a}& \mathbf{0}& \mathbf{0}& \mathbf{0}\\ \mathbf{0}& \mathbf{0}& \mathbf{0}& \mathbf{0}& \mathbf{a}& \mathbf{0}& \mathbf{0}\\ \mathbf{0}& \mathbf{0}& \mathbf{0}& \mathbf{0}& \mathbf{0}& \mathbf{a}& \mathbf{0}\\ \mathbf{0}& \mathbf{0}& \mathbf{0}& \mathbf{0}& \mathbf{0}& \mathbf{0}& \mathbf{1}\end{array}\right)$$ (5)

$$\mathbf{W}=\left(\begin{array}{ccccccc}0& 0& 0& 0& 0& 0& 0\\ 0& 0& 0& 0& 0& 0& 0\\ 0& 0& w& 0& 0& 0& 0\\ 0& 0& 0& w& 0& 0& 0\\ 0& 0& 0& 0& w& 0& 0\\ 0& 0& 0& 0& 0& w& 0\\ 0& 0& 0& 0& 0& 0& 0\end{array}\right)$$ (6)

The gain $\gamma$ was set to 0.5, $\text{d}t$ was the tick length (50 ms), $a$ was set to ${\left({0.825}^{1/0.2}\right)}^{\left(\text{d}t\right)}$, consistent with ref. 21, and $w$ was set to the MSE loss averaged over all dimensions from training the EEGNet. Additionally, for stability during decoding, we normalized the $L_1$ norm of the velocity components to have a maximum value of 1 before performing the state update. This effectively sets a maximum velocity of 5.84 cm s⁻¹ (0.5 a.u. s⁻¹).

CLDA

The hidden state of the EEGNet trained from the decorrelated task was used as the observations of a linear KF, whose state represented the position and velocity of the displayed cursor. We performed CLDA to adapt the KF through the course of each session⁷¹. Although internal statistics were updated at each tick, online decoder parameters were only updated after each trial of an adaptation block or after the end of the first evaluation block during each session. Except for some earlier sessions (H2 days 1–5 and S2 day 1), the position information was removed from the inferred state before adaptation, so that the KF was effectively a velocity KF^18,22. Later sessions for S2 were readapted to past data, with all position information removed from the inferred state. For all sessions, the rows of the observation matrix C corresponding to the position were set to zero. As with ref. 18, the rows and columns of the a posteriori-estimated covariance matrix were set to zero at every iteration.

Retraining or CLDA for continuous BCIs benefits from a ReFIT-like KF innovation that augments the training data by defining the velocity component of the inferred state as a vector pointing from the cursor to the target. However, in our context, when the user is commanding one of four actions rather than continuously attempting to move their arm, this may not be optimal. For example, when trying to reach the up right target (45°) from the origin, both an up and right command are equally correct. Instead of a single state at each time step, the inferred state is represented by a probability distribution over up to two plausible states. For example, when the cursor is at coordinates (0, 0) and the target is at coordinates (0.15, 0.35), we assign a probability of 0.3 to the state ${\left[0,0,1,0,0,0,1\right]}^{\top }$ (right) and a probability of 0.7 to state ${\left[0,0,0,0,1,0,1\right]}^{\top }$ (up). Formally, and dropping the time-step subscript for convenience, the inferred state was defined by

$$\Delta \mathbf{p}={\mathbf{p}}_{\text{target}}-{\mathbf{p}}_{\text{cursor}},$$ (7)

$$\overline{\Delta {\mathbf{p}}_1}=\frac{\mathbf{p}}{{‖\Delta \mathbf{p}‖}_1},\overline{\Delta {\mathbf{p}}_2}=\frac{\mathbf{p}}{{‖\Delta \mathbf{p}‖}_2},$$ (8)

$${ϵ}_{\text{H}}=\cos \left({\theta }_{\text{snap}}\right),{ϵ}_{\text{L}}=\sin \left({\theta }_{\text{snap}}\right),$$ (9)

$$P^x=\left\{\begin{array}{ll}0,\hfill & \text{if}\left|{\overline{\Delta {\mathbf{p}}_2}}^x\right|<{ϵ}_{\text{L}}\hfill \\ \text{sign}\left({\overline{\Delta {\mathbf{p}}_1}}^x\right)\hfill & \text{if}\left|{\overline{\Delta {\mathbf{p}}_2}}^x\right|\ge {ϵ}_{\text{H}}\hfill \\ {\overline{\Delta {\mathbf{p}}_1}}^x\hfill & \text{otherwise}\hfill \end{array}\right.,P^y=\left\{\begin{array}{ll}\text{sign}\left({\overline{\Delta {\mathbf{p}}_1}}^y\right)\hfill & \text{if}\left|{\overline{\Delta {\mathbf{p}}_2}}^y\right|\ge {ϵ}_{\text{H}}\hfill \\ 0\hfill & \text{if}\left|{\overline{\Delta {\mathbf{p}}_2}}^y\right|<{ϵ}_{\text{L}}\hfill \\ {\overline{\Delta {\mathbf{p}}_1}}^y\hfill & \text{otherwise}\hfill \end{array}\right.,$$ (10)

$$\eta =\min \left({‖\Delta \mathbf{p}‖}_1/\alpha ,1.0\right),$$ (11)

$${\mathbf{s}}^x=\left[\begin{array}{c}0\\ 0\\ \eta \cdot \left(P^x\ge 0\right)\\ \eta \cdot \left(P^x<0\right)\\ 0\\ 0\\ 1\end{array}\right],{\mathbf{s}}^y=\left[\begin{array}{c}0\\ 0\\ 0\\ 0\\ \eta \cdot \left(P^y\ge 0\right)\\ \eta \cdot \left(P^y<0\right)\\ 1\end{array}\right],$$ (12)

where ${\mathbf{p}}_{\text{target}}$ and ${\mathbf{p}}_{\text{cursor}}$ are the ($x$, $y$) positions of the target and cursor, respectively; ${\theta }_{\text{snap}}={22.5}^{\circ }$ is the angle under which the vectors are snapped to the nearest direction; ${\mathbf{s}}^x$ and ${\mathbf{s}}^y$ are the states corresponding to velocities on the $x$ and $y$ axes, respectively; and $\eta$ acts as a scale factor parameterized by $\alpha =0.2\text{a}.\text{u}$. The $x$-axis state ${\mathbf{s}}^x$ was represented with probability $\left|P^x\right|$, and ${\mathbf{s}}^y$ was represented with probability $\left|P^y\right|$. During the decorrelated closed-loop subsession, the inferred states were set as ${\left[0,0,⫾\left(\text{right}\right),⫾\left(\text{left}\right),⫾\left(\text{up}\right),⫾\left(\text{down}\right),1\right]}^{\top }$, where is the indicator function. For example, when the down action was prompted, the inferred state was set to ${\left[0,0,0,0,0,1,1\right]}^{\top }$. Future work to optimize the inferred state could further improve performance.

On some days, the closed-loop performance of a previous day’s decoder dropped acutely at the start of a centre-out 8 session. On those days, we collected data from a new decorrelated training subsession to train a new EEGNet. We then adapted the KF resulting from the new EEGNet to a centre-out 8 dataset that had already been collected (S2 day 4 decoder adapted to day 3 data). This reduced the amount of recalibration time needed when a new EEGNet was trained. We note that although past data from the centre-out and other 2D tasks could be used to train an EEGNet with more data, we did not do so in this work to minimize the effect of eye movements on decoding.

Cursor copilot

Overview.

The cursor copilot aids the user’s cursor control by directly influencing the cursor movement with its own copilot output velocity ($v_t^{\text{CP},x}$, $v_t^{\text{CP},y}$) , which is added to the cursor velocity derived from CNN-KF $\left(v_t^{\text{KF},x},v_t^{\text{KF},y}\right)=\left(v_t^{\text{KF},+x}-v_t^{\text{KF},-x},v_t^{\text{KF},+y}-v_t^{\text{KF},-y}\right)$. We set

$$\left({\tilde{v}}^x,{\tilde{v}}^y\right)={\alpha }_{\text{KF}}\cdot \left(v_t^{\text{KF},x},v_t^{\text{KF},y}\right)+{\alpha }_{\text{CP}}\cdot \left(v_t^{\text{CP},x},v_t^{\text{CP},y}\right),$$ (13)

where ${\alpha }_{\text{KF}}=0.6$ and ${\alpha }_{\text{CP}}=0.3$.

The copilot used a long short-term memory (LSTM) recurrent neural network to learn representations over historical inputs (Supplementary Section 4). The input to the copilot network was the cursor’s current position and velocity. The goal information (that is, the prompted target) was not given to the copilot. Rather, the copilot had to infer the user’s goal (one out of the nine candidate centre-out 8 targets) from the user’s past decoded movements. Given the dynamic task information, the copilot therefore implicitly inferred the user’s intended goal and influenced the cursor’s velocity to move more efficiently towards the goal.

Input and output.

For our experiment, we made a design choice of using a policy network that input the CNN-KF velocity and the cursor’s trajectory to output the copilot velocity ($v_t^{\text{CP},x}$, $v_t^{\text{CP},y}$) . The LSTM component of the policy network has recurrence, enabling it to model the representations of the cursor trajectory. The copilot thus only observed the cursor positions and the current KF velocity, ${\mathbf{s}}_t^{\text{copilot}}=\left(p_{\text{t}}^x,p_{\text{t}}^y,v_t^{\text{KF},x},v_t^{\text{KF},y}\right)$, at every time step. At the beginning of each trial, the LSTM hidden state was reset to zeros. These values updated as the cursor moved. Overall, the LSTM network mapped an observation ${\mathbf{s}}_t^{\text{copilot}}\in {ℝ}^4$ to an action in ${\mathbf{a}}_t\in {ℝ}^9$. This action influenced the movement of the cursor by using a charge-based system with electric charges on all nine target positions. To define this charge-based system, we first define a matrix ${\mathbf{p}}_{\text{targets}}\in {ℝ}^{2\times 9}$ defining the ($x$, $y$) positions of all nine targets:

$${\mathbf{p}}_{\text{targets}}=\left[\begin{array}{cccccc}0.7& -0.7& 0& 0& \cdots & 0\\ 0& 0& 0.7& -0.7& \cdots & 0\end{array}\right].$$ (14)

As described in the main text, the centre-out 8 task has nine defined targets that are fixed. Our design incorporates this task-specific information; however, in general computer usage, this copilot can use candidate target locations ${\mathbf{p}}_{\text{targets}}$ inferred via CV. Here $a_t^i$ denotes the $i$th element of ${\mathbf{a}}_t$, and every element is output by the LSTM. Each element $a_t^i\in \left[-1,1\right]$. We then define the charge assigned to target $i$ at time $t$ as

$$c_t^i=\frac{a_t^i+1}{2},$$ (15)

implying that each charge $c_t^i\in \left[0,1\right]$. We then use these charges to define a force that each target exerts on the cursor, mimicking Coulomb’s law. We let hyperparameter $q$ denote the charge of the cursor, and compute the force exerted by each of the nine targets as

$$F_t^i=K\frac{q\cdot c_t^i}{{‖{\mathbf{p}}_{\text{targets}}^i-{\mathbf{p}}_{\text{cursor,}t}‖}^2+\epsilon },$$ (16)

where ${\mathbf{p}}_{\text{targets}}^i$ is the $i$th column of ${\mathbf{p}}_{\text{targets}}$, ${\mathbf{p}}_{\text{cursor,}t}$ is the current cursor position at time $t$, hyperparameter $K$ is a fixed constant and hyperparameter $\epsilon$ avoids division by zero. The values of all hyperparameters are given in Extended Data Table 4. We then compute the charge exerted by the $i$th target on the cursor at time $t$ as

$${\mathbf{v}}_t^{\text{CP},i}=F_t^i\cdot \left({\mathbf{p}}_{\text{targets}}^i-{\mathbf{p}}_{\text{cursor,}t}\right),$$ (17)

which is a vector $\in {ℝ}^2$. The total influence of all targets on the cursor is

$${\mathbf{v}}_t^{\text{CP}}=\sum _{i=1}^9{\mathbf{v}}_t^{\text{CP},i}.$$ (18)

This sum was the copilot output velocity, ${\mathbf{v}}_t^{\text{CP}}=\left(v_t^{\text{CP},x},v_t^{\text{CP},y}\right)$, which influenced the cursor towards the inferred goal.

Cursor task rewards.

The policy network was trained via proximal policy optimization³². The reward signal at each time step, $r_t$, was defined by the proximity of the cursor to the target, where

$$d_t={‖{\mathbf{p}}_{\text{correct}\text{target}}-{\mathbf{p}}_{\text{cursor},t}‖}_2,$$ (19)

$$r_t=k\cdot \left(d_t-d_{t-1}\right).$$ (20)

The hyperparameter $k$ was chosen so that $r_t$ in each trial was between –1 and 1. Note that the cumulative reward is a state function. Future rewards were discounted at $\gamma =0.989$ to encourage the copilot to acquire the target in the fewest number of steps. The copilot value network $\left(V\left(\mathbf{s}\right)\right)$ estimated the expected rewards to go from state $\mathbf{s}$.

Training.

Training the copilot involves generating decoded velocities, ${\mathbf{v}}_t^{\text{KF}}$, from the CNN-KF. However, optimizing the copilot with online human-in-the-loop experiments is very expensive in time. We therefore modelled CNN-KF-decoded velocities to accelerate copilot training. We call the model that generates CNN-KF-decoded velocities the surrogate human control policy because it models decoded velocities from human BCI users. The surrogate human control policy, denoted by $p_{\text{surrogate}}\left({\mathbf{v}}_t^{\text{KF}}\right)$, approximates the distribution of human cursor control velocities, $p_{\text{human}}\left({\mathbf{v}}_t^{\text{KF}}\right)$, when performing the centre-out 8 task. We designed the surrogate policy to mimic four features of the human policy:

1. The human attempts to move the cursor in the correct direction. To model this, we defined an intended velocity as

   $${\mathbf{v}}_t^{\text{int}}=0.5\cdot \frac{{\mathbf{p}}_{\text{correct}\text{target}}-{\mathbf{p}}_{\text{cursor},t}}{{‖{\mathbf{p}}_{\text{correct}\text{target}}-{\mathbf{p}}_{\text{cursor,}t}‖}_2}.$$ (21)

2. Due to the slow dynamics of EEG activity, changing cursor directions only occurred after a modest delay. We, therefore, modelled a switch in the decoded direction at time $t$ as occurring over an interval between $\left[t,t+\text{delay}\right]$ and linearly interpolated velocities within [${\mathbf{v}}_t^{\text{int}}$, ${\mathbf{v}}_{t+\text{delay}}^{\text{int}}$]. The random delay was drawn from a uniform distribution, $U\left[300\text{ms},600\text{ms}\right]$.

3. When a new direction is decoded, the CNN-KF-decoded velocities can be erroneous, which we modelled via angular noise in the decoded velocity. We modelled this as

   $${\mathbf{v}}_t^{\text{unintended}}=\left[\begin{array}{cc}\cos \left(\theta \right)& -\sin \left(\theta \right)\\ \sin \left(\theta \right)& \cos \left(\theta \right)\end{array}\right]{\mathbf{v}}_t^{\text{int}},\theta \approx 𝓝\left(0,\sigma \right),$$ (22)

   over the switch direction interval $\left[t,t+\text{delay}\right]$. In our simulations, $\sigma =\text{π}/3$.

4. Even when CNN-KF-decoded velocities go in the correct direction, there is stochasticity in cursor movements. We modelled this via independent additive Gaussian noise. This produced the final surrogate simulated velocity:

   $${\mathbf{v}}_t^{\text{KF}}={\mathbf{v}}_t^{\text{unintended}}+{\ell }_t,{\ell }_t\approx 𝓝\left(0,\right).$$ (23)

   In our simulations, $\Sigma =0.03\mathbf{\text{I}}$.

   With this model of $p_{\text{surrogate}}\left({\mathbf{v}}_t^{\text{KF}}\right)$, we trained the copilot without a human in the loop. In copilot training, we also increased the hold time to 2 s to encourage the copilot to hold the cursor close to the target even after acquiring it. All hyperparameters are reported in Extended Data Table 4.

Centre-out 8 performance metrics

We report how each performance metric for centre-out 8 was computed.

• Trial: data for where the participant must navigate the cursor from the centre to a radial target. The navigation from a radial target to the centre target is not counted as a trial.

• Trial time (s): the time from trial initiation to trial completion, either by acquisition or by timeout.

• Hold time (ms): the amount of time that the centre of the cursor had to continuously overlap the target for successful acquisition (set to 500 ms).

• Success %: the percentage of successfully acquired radial targets, typically aggregated across a block of eight trials.

• Hit rate (goals per minute): the number of successfully acquired radial targets (goals) per minute. No part of the trial was excluded.

• First touch time (s): the time it takes the centre of the cursor to first touch the prompted radial target.

• Dial-in time (s): equal to (trial time – hold time – first touch time) for successful trials only¹⁸. It is equal to zero if the cursor immediately overlaps the target for 500 ms (hold time) after first touching it, and is positive otherwise.

• Path efficiency (%): the straight-line distance between the targets divided by the total distance travelled by the cursor, multiplied by 100%. Its maximum value, 100%, occurs when the cursor takes the shortest path between targets.

• Fitts’ ITR (bits per second): Fitts’ ITR⁷² is defined as

  $$\text{ITR}={\log }_2\left[\frac{D+S}{S}\right]/\left(\text{trial}\text{time}-\text{hold}\text{time}\right).$$ (24)

The numerator is known as the index of difficulty.

Spatial saliency maps

To visualize the contribution of each EEG channel’s activity towards decoding, we computed saliency maps over channels. Saliency maps represent the importance of input channels for driving the cursor in one of the four cardinal directions (right, left, up and down). Spatial saliency maps $\delta {\mathbf{h}}_i$ for each direction were calculated as

$$\delta {\mathbf{h}}_i={\left.\frac{\partial \left({\mathbf{e}}_i^{\text{T}}\left(\Delta \mathbf{s}\right)\right)}{\partial \mathbf{\kappa }}\right|}_{\mathbf{\kappa }=1},$$ (25)

$$=\mathbf{h}\odot \frac{\partial \left({\mathbf{e}}_i^{\text{T}}\left(\Delta \mathbf{s}\right)\right)}{\partial \mathbf{h}},$$ (26)

$$\Delta \mathbf{s}=\left(\mathbf{A}-{\mathbf{K}}_{\mathbf{\kappa }}\mathbf{C}\mathbf{A}\right)\mathbf{y},$$ (27)

where $\mathbf{h}$ is an input or intermediate layer activation of the decoder, ${\mathbf{e}}_i$ is the unit vector for index $i$, $\mathbf{\kappa }$ is a unit scaling vector to collect gradients, $\mathbf{y}=\text{CNN}\left(\mathbf{\kappa }\odot \mathbf{h}\right)$ is the observation vector and $\odot$ is the broadcasted Hadamard product operator. We averaged $\delta {\mathbf{h}}_i$ over the time dimension to produce saliency maps for the preprocessed inputs and for inputs to the EEGNet depthwise layer. Saliency maps were then projected using MNE’s plot_topomap function⁷³.

Robotic arm and CV copilot

We interfaced with the Franka Emika Panda 3 robotic arm using custom C++ and Python code, and placed a camera above the working space of the robotic arm. We calibrated the CV system using four marked locations on the table. A separate process used Grounding DINO³³ to find the pixel locations corresponding to the text prompts for small cube and cross, identifying the block and target locations (collective goals).

The CV copilot identified each block and target and transformed their pixel locations to the robotic arm’s coordinate frame, creating an internal model of the environment. The states of each object, that is, the locations of blocks and whether they were held by the gripper, as well as locations of targets and whether any block occupied each target, were continuously tracked by the copilot. When the robotic arm’s position was within a 2.54-cm-radius circle of an available goal, the CV copilot initiated the appropriate pick/place action for that goal. The copilot did not initiate the placement of blocks onto already occupied targets, but did allow blocks to be placed at the location from which they were picked up. This contrasts with the hover condition, in which all the initiated pick/place actions were attempted but often failed.

Reporting summary

Further information on research design is available in the Nature Portfolio Reporting Summary linked to this article.

Extended Data

Extended Data Fig. 1 |. Within-day same-class CNN training confusion matrices.

a, Confusion matrix for decoding the left class in the first half (0–10 min) or second half (10–20 min) of the open loop session in test data. Although the motor intent is the same, EEG activity can be easily decoded from the first vs second half of the session, indicating decodable shifts in EEG activity. b, Same as a, but for the right class.

Extended Data Fig. 2 |. CNN training confusion matrices.

Confusion matrices for the seed decoder from the open loop session that was used in the decorrelated session for a-c, healthy (H1, H2, H4) and d, SCI (S2) participants. All results are computed on test set data.

Extended Data Fig. 3 |. Eye gaze position was not reliably decoded from decoder hidden state replayed for the 1D decorrelated closed-loop task.

Prediction of eye tracker gaze location from decoder hidden state during the decorrelated closed-loop task for a range of temporal offsets. Data for each session were split into 5 folds without shuffling. We trained linear regression models to predict gaze location from decoder hidden state, with one model trained from data for each motor prompt (right, left, up, down) for each offset for each data split. Lines show the maximum average coefficient of determination, by averaging each value across all 5 folds, then taking the maximum over all prompts. A delay of zero means that gaze data and hidden state are aligned as recorded, and a positive delay means that the gaze data at a certain timestamp is aligned with the hidden state recorded at a future timestamp. Black lines show the maximum average values, with individual averages shown as lighter colored lines. In general, coefficient of determinations were negative, meaning that these regression models could not predict the eye gaze data better than the mean on validation data. This provides strong evidence that the CNN hidden state did not contain decodable eye movement signals.

Extended Data Fig. 4 |. Fitts ITR for CNN-KF control.

a, Fitts ITR computed using acquisition time minus hold time (similar to Gilja et al., 2015), Fitts ITR computed using first touch time (as in Silversmith et al., 2021), and first touch time across days. Each circle represents one 8-trial block. b, same as a but for healthy participants.

Extended Data Fig. 5 |. Cursor copilot learns the center-out 8 task during synthetic training.

a, PPO policy loss, value function loss, and rewards over training for the cursor copilot. The copilot increases rewards through training, as well as b, in an evaluation test environment, where the copilot was frozen every 8192 training steps and evaluated on the center-out and back task. c, Success percentage, d, trial time, e, target hit rate, and f, Fitts ITR on the center-out 8 task over the course of training. Figure c, d, e, and f represent cumulative results of 8 copilots trained under the identical hyperparameter settings. The dark gray represents the mean, while the light gray band shows the standard error of the mean (SEM). These demonstrate that the copilot learns to use the surrogate KF signals to perform the center-out 8 task. Please note that these numbers are in general lower (for example, success percentage does not reach 100%) because the copilot task was more challenging, having a 2 second target hold time to encourage goal acquisition behavior (see Methods).

Extended Data Table 1 |.

Detailed hyperparameters of EEGNet

Parameter	Description	Value
# Temporal Filters	Number of temporal filters in the first convolutional layer	8
Temporal Filter Size	Size of each temporal filter (time × frequency)	(1, 51)
# Spatial Filters	Number of spatial filters in the spatial convolutional layer	2
Spatial Filter Size	Size of each spatial filter (channels × 1)	(58, 1)
Average Pool Size (1st)	Size of the average pool filter in the first block	(1, 3)
Separable Conv Size	Size of the separable convolution in the second block	(1, 16)
Average Pool Size (2nd)	Size of the average pool filter in the second block	(1, 16)
Dropout	Dropout rate applied for both layers	0.5
Learning Rate	Initial learning rate	0.001
Learning Rate Schedule	Reduction factor when validation accuracy does not improve	0.1
Weight Decay	Weight decay applied during optimization	0.0001
Batch Size	Number of samples in each batch	32

Architecture and training hyperparameters for EEGNet are shown, as well as a brief description for each.

Extended Data Table 2 |.

Within-day rank-sum comparisons of Fitts ITR show limited differences across sessions (days)

S2
Day \ Day		1 (n = 8)		2 (n = 14)		3 (n = 15)		4 (n = 7)
	U =	p=
2 (n = 14)	1.502	0.133
3 (n = 15)	0.129	0.897	−1.353	0.176
4 (n = 7)	2.199	0.028	1.119	0.263	1.798	0.072
5 (n = 6)	−0.258	0.796	−1.485	0.138	−0.234	0.815	−2.143	0.032
H1
Day \ Day		1 (n = 14)		2 (n = 23)		3 (n = 19)		4 (n = 20)
	U =	p=
2 (n = 23)	2.380	0.017
3 (n = 19)	3.315	0.001	0.518	0.604
4 (n = 20)	2.030	0.042	−0.536	0.592	−1.180	0.238
5 (n = 10)	2.284	0.022	0.588	0.557	0.367	0.714	1.056	0.291
H2
Day \ Day		1 (n = 10)		2 (n = 15)		3 (n = 17)		4 (n = 16)
	U =	p=
2 (n = 15)	−0.111	0.912
3 (n = 17)	3.163	0.002	3.304	0.001
4 (n = 16)	2.635	0.008	2.925	0.003	0.540	0.589
5 (n = 9)	2.123	0.034	2.176	0.030	−0.135	0.893	−0.793	0.428
H4
Day \ Day		1 (n = 11)		2 (n = 17)		3 (n = 14)		4 (n = 18)
	U =	p=
2 (n = 17)	−0.165	0.869
3 (n = 14)	−0.219	0.827	−0.476	0.634
4 (n = 18)	0.809	0.418	1.848	0.065	2.963	0.003
5 (n = 9)	1.861	0.063	2.614	0.009	3.780	0.000	2.623	0.009

Fitts ITR values were calculated with first touch time, and were directly averaged across each 8-trial center-out 8 block. p-values below the Bonferroni-corrected threshold of 0.005 (10 multiple comparisons) are bolded (two-sided rank-sum test). n represents the number of 8-trial blocks recorded on each day.

Extended Data Table 3 |.

Comparison of Fitts ITR with prior work

Comparison with Silversmith 2021
Participant	Best Day Fitts ITR (FTT)	Best: % of Silversmith Best	Average Day Fitts ITR (FTT)	Average: % of Silversmith Average
S2	0.176	41.9	0.156	48.6
H1	0.354	84.2	0.324	101.2
H2	0.263	62.6	0.224	70.0
H4	0.282	67.2	0.228	71.4
Silversmith	0.42	100	0.32	100

Comparison with Gilja 2015
Participant	Best Day Fitts ITR (calculated according to Gilja)	Best: % of T6	Best: % of T7	Best: % of S3	Average Day Fitts ITR (calculated according to Gilja)	Average: % of T6	Average: % of T7	Average: % of S3
S2	0.133	14.3	14.5	46.8	0.108	11.7	11.8	38.2
H1	0.236	25.4	25.7	83.2	0.217	23.4	23.6	76.5
H2	0.181	19.5	19.7	63.8	0.159	17.0	17.2	55.8
H4	0.188	20.2	20.4	66.1	0.159	17.1	17.3	56.1
Gilja T6	0.93	100.0	101.1	327.5
Gilja T7	0.92	98.9	100.0	323.9
Gilja S3	0.284	30.5	30.9	100.0

For Fitts ITR comparisons, values were taken from Fig. 1 in ref. 21 and from Fig. 1c and Supplementary Table 1 in ref. 2 using their respective methods for calculating Fitts ITR. For comparison with Silversmith et al., we averaged Fitts ITR across all blocks within each day, then averaged the ITR across day averages. For comparison with Gilja et al., we calculated ITR as the average index of difficulty (IOD) divided by the average time within each day, then averaged the ITR across day averages. This calculation used first touch time and Gilja et al.’s definition of IOD.

Extended Data Table 4 |.

Detailed hyperparameters of RL cursor copilot

Reward and Action Hyperparameter
Parameter	Description	Value
${\alpha }_{\text{KF}}$	Contribution of KF-CNN	0.6
${\alpha }_{\text{CP}}$	Contribution of RL copilot	0.3
$K$	Electrostatic constant	1
$q$	Fixed electric charge on cursor	1
$\epsilon$	Epsilon stabilization in the electrostatic equation for avoiding division by zero	0.01
$k$	Linear reward scale	66.67
$\gamma$	Reward discount factor	0.98901
Copilot Training Hyperparameter
Parameter	Description	Value
learning rate	Copilot learning rate	0.0003
rollout buffer	Number of steps stored before gradient update	2048
batch size	Mini batch size	512
total learning steps	Total number of steps taken for training	1,200,000
delay	Delay used in $p_{\text{surrogate}}\left({\mathbf{v}}_t^{\text{KF}}\right)$	$U\left[300ms,600ms\right]$
$\sigma$	Variance of angular noise in $p_{\text{surrogate}}\left({\mathbf{v}}_t^{\text{KF}}\right)$	$\pi /3$
$\Sigma$	Variance of additive Gaussian noise in $p_{\text{surrogate}}\left({\mathbf{v}}_t^{\text{KF}}\right)$	$0.03\mathbf{I}$

Training hyperparameters for the RL cursor copilot are shown, as well as a brief description for each. Details about the agent’s architecture can be found in Supplementary Section 4.

Supplementary Material

Supplementary information The online version contains supplementary material available at https://doi.org/10.1038/s42256-025-01090-y.

Data availability

Data related to this article are available via Zenodo at https://doi.org/10.5281/zenodo.15165133 (ref. 74). Source data are provided with this paper.

Code availability

Experimental and model training code is available via GitHub at https://github.com/kaolab-research/bci_raspy and via Zenodo at https://doi.org/10.5281/zenodo.15164641 (ref. 75). Plotting and analysis code is available via GitHub at https://github.com/kaolab-research/bci_plot and via Zenodo at https://doi.org/10.5281/zenodo.15164643 (ref. 76).