Wearable optomyography enables continuous neuroprosthetic control

Abstract

Wearable devices are increasingly used to enable human-machine interfaces, such as typing or cursor control, through wristbands that translate surface electromyographic (sEMG) signals into computer commands. However, traditional sEMG techniques face several limitations, including challenges with sensor fixation, signal cross-talk, instability over time, and susceptibility to electrical and mechanical artifacts. In this study, we propose an alternative approach to capturing and interpreting muscle activity using optomyography (OMG). Our OMG system – a wristband with 50 data channels, facilitates various computer mouse-like controls. Decoding is achieved through an efficient, compact, fully connected neural network trained on data from a center-out task performed with hand gestures. Eight able-bodied participants and one individual with limb loss successfully mastered OMG-based controls in tasks such as acquiring targets across various screen positions and playing Tetris. Performance improvements with training were assessed using metrics such as deviations from a straight trajectory, temporal deviation from an optimal path, and dwell time near the target prior to successful selection. These results highlight the potential of next-generation wearable devices to exceed conventional approaches in performance, accuracy, stability, and versatility.

Introduction

Human-machine interfaces encompass a variety of tasks in which humans operate or interact with computers, robots, autonomous vehicles, and other devices¹. These interactions can be facilitated with surface electromyography (sEMG) wristbands that generate commands from muscle contractions and enable direct muscle-to-machine control, bypassing traditional manual inputs such as buttons, sliders, joysticks, computer mice, etc. For the past five decades, this sEMG-based control has dominated as the method of detecting muscle contractions². In this approach, muscle electrical activity is sampled via electrodes placed on the skin overlying the muscles of interest.

Notwithstanding the success of such sEMG-based interfaces over the past several decades, their utility remains limited. This is true even with recent advances in low-cost, high-precision electronics and modern machine learning algorithms that transform muscle signals into control commands for devices and avatars in virtual reality^1,3,4. Even with a precise electrode placement, sEMG signals can be contaminated by external electrical and mechanical artifacts. Additionally, crosstalk between muscles is common, and activity in deep muscles is often underrepresented (Fig. 1).

Fig. 1.

OMG-based continuous real-time decoding. The OMG wristband (a) was placed on a ventral side of the forearm (b). The phototransistors were arranged to form four 17.6 x 17.6 mm squares. An IR LED was placed at the center of each of these squares. Both the phototransistors and IR LED had a transparent-silicone protective cover which acted as a lens. Each participant completed a calibration procedure where OMG recordings were conducted during hand movements (c). During the calibration, the participants performed the center-out task with 12 targets (d). The calibration data were then used to train a small multilayer perceptron (e). The perceptron predicted hand direction and fist clench state from the OMG signals. The real-time OMG-based control was implemented as a cursor-pointing game where the participants produced directional gestures to acquire targets with a screen cursor (f). The performance was evaluated with six metrics, five of them are displayed in (g), while the sixth is a mean number of misclicks across all game sessions. t, and are target acquisition time, ideal target acquisition time for the straight trajectory, and the duration inside the target proximity zone, respectively. s is the initial distance between the cursor and target in arbitrary units. and are the actual and straight-line coordinates of the cursor, respectively. k is the number samples per cursor trajectory on a given trial. designates coordinates of the target. R is the size parameter for the target proximity zone. It was set to 0.2.

Substantial improvements in muscle-based control systems are still needed, particularly in the field of upper-limb prostheses. More than 25 million people worldwide live with upper-limb amputations^5,6. Although sEMG-controlled robotic prostheses are commercially available, their use by amputees is hindered by their heavy weight and difficulties of control, leading many patients to abandon their use⁷. The other drawback is the absence of sensory feedback. Incorporation of feedback into the device could significantly improve quality of life by decreasing phantom limb pain^8–11. Additional challenges stem from the low amplitude of EMG signals and their susceptibility to noise⁴. Signal quality is further compromised by crosstalk between recording channels, which are intended to sample different muscles. In general, controlling a bionic prosthesis via EMG requires considerable mental effort¹².

During the last ten years, a range of new methods has emerged for sampling muscle activity^3,4. Among them, optomyography (OMG) is a non-invasive approach based on monitoring perfusion and oxygenation in contracting muscles¹³. Biological tissues exhibit relative transparency to light in the near-infrared range from 700 to 1000 nm^4,14. In a typical OMG setup, near-infrared light is emitted into the tissue via a light-emitting diode (LED), and a photodetector measures the intensity of scattered light in the surrounding tissue¹⁴. This technique offers a higher signal-to-noise ratio compared to EMG, is immune to electrical interference, and can be implemented using compact hardware. Several studies have already assessed the usage of OMG for decoding hand movements^15–20. These studies demonstrated the successful decoding of various hand gestures. However, no OMG system has yet demonstrated reliable, real-time decoding of multiple gestures.

In this study, we developed an OMG-based system which uses lightweight machine-learning algorithms for online decoding of hand gestures. The current implementation handles twelve directional gestures and two additional control gestures. This OMG system was tested in nine participants, including one amputee. They all learned to perform computer mouse-like operations like pointing to screen targets with a cursor and playing Tetris. We used six metrics to quantify user performance with this system and applied the Fitt’s law to connect the OMG performance with existing EMG studies.

Results

Gesture decoding

We implemented a neural network architecture that accurately decoded hand gestures. The decoding was visualized using a circular plot (Fig. 2), which displayed the predicted movement directions in polar coordinates.

Fig. 2.

The scatter plots showing how direction was decoded from the OMG data with the multilayer perceptron (MLP). Each dot corresponds to one sample of the OMG data. Polar angle represents the decoded direction. Each dot’s color corresponds to the instructed direction, and its distance from the center represents the perceptron’s output magnitude.

In this plot, dots represented the predicted cursor position and dot color indicated target identity. The directions were decoded mostly accurately, with errors occurring when the decoder misattributed the direction to a neighboring target. Thus, the circular plot for participant RK07 (Fig. 2g) shows that predictions for the 150°target were often attributed to the 120°target. A similar confusion was observed for the 0°and 30°targets. We also observed that the participants were not consistent when generating an angle of pronation or supination, particularly when their muscles were excessively relaxed. Despite these imperfections, the trajectories were decoded well as was evident from the low mean squared error (MSE). The MSE for the trajectories generated by the model was compared to the MSE obtained after these trajectories were randomly shuffled across targets, yielding a mean difference of .

The box plots in Supplementary Fig. S1 illustrate the model’s predictions for click gestures, showing a clear distinction between the fist clench and other gestures. To minimize false actuations, a subject-specific recognition threshold was empirically determined during calibration and applied before the sessions of real-time control began. These plots serve as an additional demonstration of the trained model’s ability to classify the fist clench gesture accurately.

Real-time decoding

All participants succeeded in the cursor-pointing task, which involved moving the cursor to a target and performing a click gesture. The participants acquired basic control skills during the first session and continued to practice in the subsequent sessions. Most metrics showed performance improvement from the first to the middle session. Specifically, the deviation of time completion time from the ideal performance decreased significantly (Fig. 3) for RK01 (t=3.2019, p=0.0025), RK03 (t=1.9934, p=0.0308), RK05 (t=1.7473, p=0.0488), and RK07 (t=2.6946, p=0.0074). For the related but distinct metric of trajectory deviation, as it is clearly seen in Supplementary Fig. S2, most participants showed improvement during the first five sessions, after which accuracy declined for some individuals. Two participants, RK03 and ARK02, had exemplary learning curves with steady improvement across sessions in terms of closeness to the ideal trajectory. Though, in the end trajectory deviation decreased significantly by the middle session for RK01 only (t=3.1952, p=0.0025).

Fig. 3.

The analysis of trial completion time relative to the ideal performance. The difference in trial completion times between the ideal and actual trajectories. The ideal completion time was calculated based on the assumption that cursor velocity was constant and maximal. The red line indicates overall trend of the median value.

Overall, trial pace decreased during the early sessions 1–5 and increased slightly toward the later sessions 6–8 (Fig. 4). The initial shortening of trial pace was observed for all participants except RK05. A significant shortening was found for RK07 (t=2.6714, p=0.0078). The results for RK03 (t=1.6383, p=0.0594) and RK08 (t=1.6528, p=0.0579) approached, but did not reach, the threshold for statistical significance.

Fig. 4.

Trial pace. Trial completion time divided by the initial distance between the cursor and the target. Each box represents metric distribution during every session. The red line indicates overall trend of the median value.

For the metrics evaluating target acquisition (Fig. 5), the performance improved for RK01 (t=2.2153, p=0.0199), RK03 (t=2.6946, p=0.0074), and RK07 (t=2.8643, p=0.0052). A similar trend was observed for the supplementary metric (catching time in Supplementary Fig. S3), although performance declined in later sessions for RK04–RK07.

Fig. 5.

Target acquisition path length. The number of trajectory points corresponding to the time interval from the cursor entering the target proximity till target acquisition. Each box represents metric distribution during every session. The red line indicates overall trend of the median value.

Click performance is illustrated in Fig. 6, showing participants’ efficiency in capturing the target with the fist clench gesture. Most participants required minimal clicks, though RK04, RK07, and notably ARK02 experienced greater difficulty.

Fig. 6.

Misclick count. Mean number of clicks across 9 game session for each participant. Each point represent mean value of verified fist clenches in the target’s proximity area with respect to the click actuation threshold.

These results indicate that participants were able to quickly acquire and improve cursor control using the OMG system. Their performance generally increased across the first half of the sessions, particularly for the metrics of the trajectory accuracy and target acquisition, but showed some decline in the later sessions, likely due to factors such as fatigue or armband displacement. To further evaluate the OMG-based mouse control, one normotypical participant (RK02) and the amputee (ARK02) played Tetris using the OMG system (Supplementary Video S1). Both participants successfully mastered the control and completed several rounds of the game, achieving scores of 1705 and 1683, respectively. Each game session lasted 7 to 10 minutes. Both participants demonstrated a high level of engagement during the task and provided positive overall feedback about the gaming experience.

ARK02 commented that it was a fun experience and that he was able to easily place the tetrominoes in the correct order to achieve good results at low speeds. However, as the game progressed, increasing speed made control more difficult, leading to occasional false directional gestures. He used the upward flexion gesture to rotate the tetrominoes, which he found to be easier and more reliable for control.

RK02 also enjoyed the game and was able to perform well at higher speeds, but struggled when rapid changes in both direction and rotation were required simultaneously.

Supplementary Fig. S4 shows the Fitts’s Law test and Supplementary Fig. S5 displays throughput and path efficiency of each subject over all trials. Most of the subjects has a high coefficient of determination (more than 0.9). It shows linear correlation between the index of difficulty and the trial completion time. It is notable that RK04 and RK06 struggled with the control, Both of them has a high variation in terms of completion time versus index of difficulty. In case of RK04 it can be noted on other metrics and the circular plot. Still, RK04 managed to improve the results over game sessions and has a better path efficiency than RK06. RK01, RK02, RK03, RK08 and ARK02 have a comparable results and these subjects show the best results in all measurements related to the completion time and trajectory with maximal throughput more than 0.5 bits/s and 75% of path efficiency.

Discussion and conclusion

In this study, we implemented an OMG-based control system that replicated computer mouse operations. The system relied on the optomyographic data and the neural network model that converted twelve hand gestures into cursor movement directions and used two additional gestures to generate control commands. Decoding was performed using a lightweight deep learning algorithm, with a calibration and training time of approximately five minutes. Eight able-bodied subjects and one amputee successfully learned to control a computer cursor using the OMG system. Our system is compatible with a variety of mobile devices, and the decoded control signal can be transferred to a personal computer (PC) or other devices.

The OMG system operation is based on the changes in the optical properties of underlying tissue. Alteration in blood volume during the muscle contractions change the amount of near-infrared light scattered back to the skin surface which is then detected by the sensors. In the first study that introduced OMG recordings, a wristband containing two OMG sensors was used to decode 14 gestures¹³. The same authors later developed a 5-channel OMG system that controlled a prosthetic arm²¹. In 2017 McIntosh, Marzo and Fraser developed a new optomyography wristband with 14 data channels¹⁵. They classified 12 unique gestures in an offline setting with high accuracy. These foundational studies introduced OMG as a human-machine interface technology, but their devices had low information transfer rates, enabling only discrete commands.

To the best of our knowledge, we are the first to implement continuous decoding with OMG sensors. We note that the term “continuous decoding” in the context of this study refers specifically to the generation of output for each individual OMG sample as opposed to the reconstruction of discrete, pre-segmented gestures. Our system operates in real time and supports immediate cursor control based on the sample-by-sample regression of hand direction and click probability. As such it performs continuous decoding rather than discrete-frame classification, resulting in a sensitive and smoothly operating human-machine interface. We showed that this control can enable mouse-like and gaming (i.e. Tetris) operations. Both normotypical participants and the amputee successfully mastered these operations where they controlled cursor speed and direction. We measured the decoding accuracy by comparing MSE against random predictions and used circular plots for further validation.

Real-time decoding is essential for practical application of OMG systems. Two prior studies achieved real-time operation: Maereg et al.¹⁶ streamed real-time OMG frames for gesture classification at 6–8 gestures per second; and Vilela de Godoy et al.²⁰ controlled a robotic hand in real time using 10 gestures, a five-channel OMG wristband, and a TMC-ViT model, achieving 93% accuracy. However, in both cases, gestures were represented by discrete OMG data frames. We argue that continuous decoding (i.e. decoding of each sample) is more practical than the previously implemented discrete decoding. In the center-out tasks, direction (calibrated on 12 gestures) was continuously decoded to enable smooth cursor movement. We used a multi-layer perceptron that processed 50 OMG channels. Prior to choosing the perceptron decoding, we ran pilot experiments where we tested decoding with a recurrent neural network. That decoder took three time points as an input. We hoped using this time window for data acquisition would improve the predictions. The decoding of direction indeed improved, but this method also introduced a latency between gesture onset and cursor response. Based on these initial tests we turned down the recurrent neural network and chose the perceptron as the faster and more reliable model.

Our implementation of the OMG interface is similar to the sEMG interface demonstrated by Lobov et al.²². In their study, a perceptron with one hidden layer of eight units controlled a computer cursor; the participants were required to draw a diamond-shaped figure. In the other study of an sEMG interface, Waris et al.²³ implemented a real-time robustness to evaluate the system’s clinical usability. Their 10 normotypical participants operated the system for seven days, where they used three control strategies based on a 15-node, single-layer perceptron. Four gestures were used to move the cursor to the target and hold it inside the target for 1 s. Four metrics were used to evaluate this performance.

Our OMG system enabled a mouse-like control that was evaluated with five metrics. All participants improved their real-time performance. In some cases, the performance degraded toward the end of the session, primarily due to a wristband displacement. Most of the participants were eager to complete the tasks quickly and contracted their muscles somewhat excessively. After five or more sessions, the wristband shifted and the acquired signal altered. Nonetheless, all participants eventually overcame these issues and performed well. An adaptive learning could be introduced in the future to compensate for the signal drifts caused by wristband displacements. Minor decoding errors occurred during the clench gesture when participants failed to sufficiently tense their muscles or produced muscle tension resembling wrist bending. Figure 6 illustrates this for subjects RK04 and RK07. Adjusting the click detection threshold helped to resolve this issue. ARK02 faced a similar problem in click identification but was able to adapt, improve and finish the sessions.

Based on these findings, we suggest that the OMG system holds potential for rehabilitation of individuals suffering from motor impairments. Amputees and post-stroke patients could use the system for prosthetic control or virtual-reality-based rehabilitation tasks. Gamification could further enhance these therapeutic paradigms.

Our study exposed several limitations of the OMG system. Some participants struggled with particular cursor directions. Additionally, the click gesture was harder to detect when the wrist was in a neutral position (RK04, RK05, RK07; Supplementary Fig.S2). The completion time degraded in these subjects toward the second part of the session. The amputee experienced difficulty selectively contracting muscles to generate directional commands. These issues could be mitigated by new data acquisition protocols incorporating additional signal filtering and sensor fusion (e.g., with IMUs). Improved gesture decoding models could also enhance recognition accuracy. Future work should aim to maintain a compact network size and memory efficiency suitable for embedded systems. Lastly, wristband displacement remains a challenge^14,15, highlighting the need for effective system recalibration.

The issues of stability and signal quality have been documented for sEMG interfaces, as well, so a comparison of the OMG and sEMG systems would be of interest in this respect. Nonetheless, some indirect comparisons could be done with the previously developed real-time cursor control sEMG systems^24–27. We used typical metrics including Fitts’s Law, throughput and path efficiency. Using the coefficient of determination on a Fitts’s law linearity check as a metric, most of our participants, including the amputee, had it above 0.9, which is similar to the sEMG control. (Note, however, that we measured a more challenging non-discrete index of difficulty.) Furthermore, our throughput and path efficiency metrics are the same or slightly lower than those reported for the sEMG systems, which could be due to the more complex task we used. Our participants controlled the cursor continuously over a relatively long time period. Additionally, the previous sEMG systems did not have click gestures or actions. A typical sEMG trial was considered complete when the participant acquired a target with the cursor and kept it over the target for 0.3–1.0.3.0 s. These sEMG experiments included a fixed, discrete set of target sizes and positions and index of difficulty was derived from that fixed set of values. To the best of our knowledge, only two of the reviewed systems^26,27 had a number of trials comparable to ours and needed for statistical analysis.

Overall, our OMG system performed on par with the previously reported sEMG systems. Given the differences of these systems’ operations, we anticipate that combining EMG and OMG could provide complementary advantages. With appropriate hardware integration, future studies could enable simultaneous EMG–OMG recordings, ultimately enhancing prediction accuracy and contributing to improved motor control in prosthetic systems and related applications.

A note should be made of our selection of 12 directional and 2 control gestures for OMG control. This selection worked well for our purposes as it allowed to form a sufficiently large command for continuous decoding in a rather complex task while relying on movements that were natural and easy to produce for the able-bodied participants. This design allowed us to test the system’s capacity to generalize across multiple movement directions and to integrate discrete click commands within continuous trajectories. Though, acknowledge that the ergonomic suitability and fatigue profile of this gesture set may differ across users. They also could be suboptimal for the individuals with constrained motor capabilities or higher-level amputations. Alternative strategies could be explored in the future for these individuals. The gesture sets should be optimized to make gestures non-fatiguing and distinct. Reduced the number of gestures could better suit the needs of disabled individuals.

We tested our system in only one transcarpal amputee, so it is too early to draw conclusions regarding the system’s usability in prosthetics. This individual performed well. Future work should test whether our current results are applicable to different levels of amputation. The wristband should be modified in the future to allow individualized electrode placement to accommodate residual limb anatomy. We foresee that the arrangement of sensors will be customized in the future for different amputation types. In the context of amputations, optimizing the penetration depth of IR light – currently limited to approximately 20 mm – could further enhance applicability. An important direction is to investigate whether varying penetration depths could reveal distinct correlates of muscle contraction. To this end, multiple sensors with several emitters could be employed, allowing the resulting signals to be treated as a multimodal measure.

Methods

Data acquisition

Optomyography

OMG data were acquired using a wristband (Motorica LLC) that incorporated an array of optical sensors. The wristband is a flexible printed circuit board (PCB) (Fig. 1a) attached to the cloth wristband. Four infrared light-emitting diodes (IR LED) KP-3216F3C served as the source of 940 nm light. The OMG signal was captured using ten phototransistors KP-3216P3C. They were placed at a distance of 17.6 mm from each other and 12.45 mm from the closest LED. The spectral response was in the range 420–1120 nm with peak sensitivity at 940 nm. Each IR LED was activated sequentially to sample 10 phototransistors voltages via an analog-to-digital converter (ADC). The algorithm provided up to 40 data channels for each LED. The other 10 channels were sampled when all LEDs were deactivated.

The wristband was positioned on the distal forearm so that the sensors rested on the ventral skin surface near the carpal tunnel (Fig. 1b). Fifty channels of OMG data were sampled at 33 ms intervals and transmitted via a 115200-baud serial port to a PC. Data were digitized using a 12-bit ADC, yielding values in the range of 0–4095.

Prior to the experiments, each channel was individually calibrated using sensitivity regulation potentiometers connected to phototransistors. Participants were instructed to maximally clench their fist, and resistance was adjusted so that the peak OMG signal closely approached 4095 value without causing a signal clipping due to saturation of the phototransistor.

Calibration task

A center-out task²⁸ was used to obtain the calibration data. Figure 1c displays a calibration procedure. In this task, participants followed a green target dot on the screen by performing directional gestures while wearing the OMG wristband. The target appeared at one of 12 locations (i.e. a clock face) marked by red dots, thereby requiring 12 distinct directional gestures. Two additional gestures were included: neutral wrist posture and fist clench. The center-out visual interface shown in Fig. 1d. The green dot first appeared at the screen center where it stayed for 2 s, and the participant was required to maintain the neutral wrist posture. Next, the green dot instantly moved to the 3 o’clock position and the participant was required to start pointing to that position. The green dot stayed at that position for 2 s after which it returned to the center. The participant had to assume the neutral position with the wrist. After the green dot stayed at the center for 2 s, it moved to the next peripheral location chosen in a counterclockwise order. After all 12 targets were presented, the participants clenched their fist. Here and later this gesture mimics mouse click. This calibration sequence was repeated 5 times.

In the pilot tests, we initially employed an 8-direction center-out scheme, following Georgopoulos et al²⁸. and our previous work²⁹. Offline analyss of this dataset showed good performance: the MLP model achieved low MSE values and SVM-based classification reached high accuracy. However, these results did not translate into stable real-time control. With eight directions, decoded movements tended to cluster around the cardinal and diagonal axes, which produced discrete jumps rather than continuous cursor motion. In practice, this limited the system’s ability to interpolate between neighboring directions and restricted the control to coarse, grid-like trajectories. To improve the angular resolution of decoding, we increased the number of movement directions to 12, corresponding to spacing between targets. This configuration has been applied in several previous studies^30–36 and provides a denser sampling of the movement space. Importantly, the same network architecture and training procedures were used in both the 8- and 12-direction pilot tests. The finer angular sampling yielded a more diverse set of training examples and improved the model’s ability to generalize across intermediate directions, which resulted in smoother real-time trajectories and more stable decoding during online control. Although the pilot nature of the study did not allow us to perform a full quantitative comparison between the two configurations, qualitative real-time performance consistently favored the 12-direction scheme. Based on these observations, we adopted the 12-direction set in the current study, together with two additional control gestures.

The default gestures are shown in Fig. 2. Participants were not explicitly instructed how to perform these gestures, nor were they restricted from using alternative movements. However, they were instructed that they should use consistent gestures for the same directions. In the case a participant needed a clarification regarding the required gestures we showed pictorial representation of Fig. 2.

Participants

The participants were 4 males and 4 females without motor disabilities (RK01, RK02,... RK08) aged 22 to 26 years and one male (ARK02) aged 38 years with the amputation of all fingers on both hands. All procedures performed were in accordance with the ethical standards of the responsible committee on human experimentation (institutional and national) and with the Helsinki Declaration of 1975, as revised in 2008 (5). Procedures were approved by the Institutional Review Board (Protocol No.12). Informed consent was obtained from all participants for being included in the study. Additional consent was acquired from ARK02 and RK02 to publish a supplementary video in an online open access publication.

Experimental sequence

Each participant underwent a standardized 2-hour experimental session consisting of three phases: calibration, model training, and testing. In the beginning of the session, participants received an explanation of the experimental conditions and the basic principles of OMG. After a participant gave an informed consent, the wristband was securely attached to the wrist using hook-and-loop fasteners. Additional double-sided tape was applied to the skin and clothing beneath the wristband to ensure stability throughout the experiment. Next, individual OMG channel calibration was performed for each participant through manual potentiometer adjustment. Before each phase, participants were also briefed on its objective and their required actions, without excessive detail. No restrictions were imposed on gesture execution; participants performed movements based on personal comfort.

The calibration and data acquisition phase lasted 6 minutes. Subsequently, a real-time testing environment was initiated, allowing participants to control a cursor in a manner similar to the actual task, ensuring system responsiveness. In some cases, calibration was repeated to guarantee reliable cursor control. This procedure took 5–15 minutes, after which the model was retrained using the full dataset.

The real-time task was run next. All 10 sessions were completed without recalibration. Participants were offered an optional 2-minute rest between sessions, though most of them completed the task without pauses.

The Tetris session was conducted separately and took place on a separate day, different from the other experiments. Two people participated. They performed the same calibration procedure as on the other experimental day. Each Tetris session lasted 5–10 minutes, continuing until the game termination (due to block stacking failure).

Data processing

Data labeling

To process the calibration data, each gesture was labeled with a number 0 to 13, with 0 corresponding to neutral posture, 1–12 designating directional gestures, and 13 corresponding to the fist clench (we also call it a click gesture). The transitional states between the directional and control gestures were labeled as a rest. Later, these labels were encoded as three-dimensional vectors to be used in the regression analysis. The first two elements represented the cosine and sine of the movement direction angle, respectively, and were set to zero for the neutral posture and the fist clench. Neutral posture data were included in the dataset as well. The third element encoded the mouse click.

Gesture decoding

To decode gestures, a neural network was trained using TensorFlow³⁷. The architecture used was a MLP with a single hidden layer of size 50. It is displayed in Fig. 1e. The input data size were . Where 32 is the batch size, 1 is the temporal window and 50 is the number of OMG channels. The channel value is an instantaneous measurement of each OMG channel without any additional preprocessing. The output layer generated 3-value vectors representing the sine, cosine of direction angle, and click probability. A hyperbolic tangent activation function was applied after both the hidden and output layers.

The dataset was divided into the training (67%) and validation (33%) sets. A standard Adam optimizer with MSE loss and learning rate 0.001 was used. The neural network was trained over 50 epochs.

During inference, the network was able to decode gestures with an inference speed lower than 33 ms which is equal to the minimal sampling rate of the wristband, considering only 50 samples without temporal context.

Real-time cursor control

The real-time task was designed to allow participants to control the on-screen cursor through directional gestures (Fig. 1), thereby emulating the operation of a computer mouse.

Each participant completed 9 sessions of the task. Each session consisted of 10 target-pointing trials. During each trial, a target appeared at a random screen position, and the cursor appeared at the screen center. The participants’ task was to place the cursor over the target and perform a grip gesture to mimic a ”capture” of the target. Following the completion of 10 trials (i.e., a session), the subject could relax before the start of the next session.

The real-time control of cursor movements was enabled by the OMG wristband readings transformed into movement directions. The visual interface was enabled with the matplotlib library³⁸, where arbitrary units (au) were used to measure distance. The working area size was 1 x 1 au. Based on pilot testing, the maximum cursor speed was set to 0.04 au/s, which participants reported as comfortable during experiments. The model was able to interpolate both sine and cosine values between the neutral posture state and the direction gestures. Thus, the system incorporated proportional speed control by multiplying the maximum speed by the predicted cosine and sine values. To enhance stability, we individually adjusted the mouse-click activation threshold for each participant.

Task performance was assessed using six metrics (Fig. 1) listed below.

1. Trial pace. Trial completion time divided by the initial distance between the cursor and the target.

2. Trajectory deviation from and ideal performance. Mean squared Euclidean distance between the straight trajectory to the target and the actual trajectory.

3. Time deviation from an ideal performance. The difference in trial completion times between the ideal and actual trajectories. The ideal completion time was calculated based on the assumption that cursor velocity was constant and maximal.

4. Target acquisition time. The time measured from the moment the cursor entered the target’s proximity until target acquisition. Proximity was defined as a 0.4 0.4 au square centered on the target.

5. Target acquisition path length. The number of trajectory points corresponding to the time interval from the cursor entering the target proximity till target acquisition.

6. Misclick count. The mean number of clicks which was used to catch the target when the cursor entered the target proximity.

For most subjects, the game metrics improved during the first half of the task. To assess this, we performed a series of independent one-tailed t-tests for each subject and each metric, comparing the first session with the middle (fifth) session. For the last sessions metrics were degrading due to distortions of sensors position which is addressed in the Discussion section.

In addition to the center-out task, a Tetris arcade game was implemented with the OMG control. Directional hand gestures controlled the left and right movements of the tetrominoes, while a downward gesture increased their falling speed. Fist clench and upward gesture both controlled the rotations of the tetrominoes. Both the able-bodied participants and the amputee learned this control (Supplementary Video S1).

Fitts’s law evaluation

Fitts’ law metrics were used to evaluate target reaching using the real-time continuous OMG decoding and to compare this performance with the previous reports on sEMG-based control. The Fitts’s law³⁹ and its Shannon’s extension were used. We started with computing the throughput, the metric of an overall performance for a given task, device and environmental conditions. The throughput was calculated for each reaching trial according to the equation (1).

1

where CT is the time the trial took to complete and ID is the index of difficulty calculated as (2).

2

In equation (2) D is the distance between the target and the initial cursor position and W is target width. In our experiments, target width was 0.1 au.

Next, we calculated the coefficient of determination , which characterized the correlation between the index of difficulty and trial completion time, assuming their linear relationship. For the typical sEMG systems, the index of difficulty takes discrete values whereas we had a continuous distribution of IDs. To approximate the discrete representation, we used 7 fixed bins on the ID axis and the correspomding mean completion times. The linear regression were performed with the Ordinary Least Squares method.

We also used the path efficiency metric. It is the ratio between the straight-line distance and the actual traveled distance.

Author contributions

R.K.: Data curation, Formal analysis, Investigation, Software, Visualization, Writing – original draft; G.S.: Conceptualization, Data curation, Investigation, Methodology, Project administration, Software, Writing – review & editing, Validation; M.S.: Validation, Software, Conceptualization, Methodology; M.L.: Resources, Validation, Funding acquisition, Supervision, Writing – review & editing

Data availability

Data used for the illustrations and supplementary materials are available at the following link: https://osf.io/zepcg/. Due to the requirements of the ethical committee, raw data would remain confidential and would not be shared.

Declarations

Competing interests

M.S. are employees of Motorica LLC. Motorica LLC is a private company, developing and producing functional prosthetics of upper limbs. R.K., G.S., M.S., have a patent application and/or registration related to this work. M.L. is member of Scientific Advisory Board for Neiry LLC which was not involved in this research and product development.

Supplementary Information

The online version contains supplementary material available at 10.1038/s41598-025-32646-y.