Novel Muscle Sensing by Radiomyography (RMG) and Its Application to Hand Gesture Recognition

Abstract

Conventional electromyography (EMG) measures the continuous neural activity during muscle contraction, but lacks explicit quantification of the actual contraction. Mechanomyography (MMG) and accelerometers only measure body surface motion, while ultrasound, CT-scan and MRI are restricted to in-clinic snapshots. Here we propose a novel radiomyography (RMG) for continuous muscle actuation sensing that can be wearable or touchless, capturing both superficial and deep muscle groups. We verified RMG experimentally by a wearable forearm sensor for hand gesture recognition (HGR). We first converted the sensor outputs to the time-frequency spectrogram, and then employed the vision transformer (ViT) deep learning network as the classification model, which can recognize 23 gestures with an average accuracy up to 99% on 8 subjects. By transfer learning, high adaptivity to user difference and sensor variation were achieved at an average accuracy up to 97%. We further extended RMG to monitor eye and leg muscles and achieved high accuracy for eye movement and body posture tracking. RMG can be used with synchronous EMG to derive stimulation-actuation waveforms for many potential applications in kinesiology, physiotherapy, rehabilitation, and human-machine interface.

Graphical Abstract

I. INTRODUCTION

A. Skeletal muscle monitoring

Monitoring skeletal muscle activities has many significant medical and commercial applications, including detection of muscle fatigue and injury, diagnosis of neuromuscular disorders, assessment for physical training and rehabilitation [1]–[3], human-computer interface (HCI) [4], and robotic control [5]. Electromyography (EMG) is presently the prevalent continuous muscle sensing method, which employs intramuscular needle electrodes or epidermal electrodes on the bare skin [6] to record the neural signals during muscle contraction. As the muscle condition will affect electrical pathways and neuronal depolarization, the EMG waveform will thus contain indirect muscle information as well. Mechanomyography (MMG) and accelerometers record the mechanical motion on the body surface, but lacks information in deep muscle layers [7]. Ultrasound monitoring requires body probes with surface preparation for impedance match [8]. Magnetic resonance imaging (MRI) and computer tomography (CT)-scan can obtain high-resolution muscle imaging, but is expensive and immobile, providing only short snapshots of muscle motion in a dedicated clinical setup [9].

Hence, a direct muscle activity sensor that can accurately and continuously track muscle contraction in both superficial and deep layers with high user comfort is a critical complement to EMG in many biomedical and HCI applications.

B. Hand gesture recognition by muscle monitoring

Hand gestures are controlled by complex muscle groups, many of which are extended from the forearms. Wearable sensors on wrist or arm bands for hand gesture recognition (HGR) [10][11] is of high interest to facilitate HCI [12][13] and a myriad of other applications [14][15]. Current HGR techniques however have many limitations. Vision-based system requires off-body line-of-sight (LoS) cameras, and is vulnerable to self and ambient obstruction [16]-[18]. Depth-perception methods demand excessive geometry reconstruction computation [19][20]. Gloved-based sensors can hinder hand motion, and are often inconvenient and uncomfortable [21]. Accelerometers can only transduce surface motion, and are impractical and cumbersome if deployed on fingers or phalanges [22]. The HGR radar, such as Google Soli [23], has to be in the LoS path to the target hand and can also suffer from path noises [24][25]. By monitoring muscle activities on the forearm, a versatile HGR system can be built with user convenience, such as the surface electromyography (sEMG), tracking the neural stimulation of forearm muscles [26]–[30]. However, sEMG is limited by the ambiguity in the skin potential, vital-sign interference, and requirement of numerous electrodes with high-quality contact to the bare skin, sometime raising health concerns in long-term wearing and suffering low user acceptance [31][32]. Electrical impedance tomography (EIT) recovers the interior impedance geometry of arm muscles, but spatial resolution is limited and cross-user generalization is often questionable [33].

C. Radiomyography for muscle activity sensing

In this paper, we propose a novel skeletal muscle sensor by radiomyography (RMG) that can non-invasively and continuously capture muscle contraction in various superficial and deep layers. RMG was previously speculated by electromagnetic (EM) simulation [34] as microwave impedance variation, and has been extended here by multiple-input-multiple-output (MIMO) near-field coherent sensing (NCS) radio signals [35][36] to measure the dielectric property change and dielectric boundary movement of nearby muscle groups, including both the backscatter and transmission channels. NCS couples ultra-high frequency (UHF) EM waves inside the body and reads out the internal organ and tissue motion signals as modulated antenna characteristics or scattering matrices [35]. As the near-field coupling is nonlinear and convoluted in the capture volume from different observation points, we explored spatially diverse channels to distinguish the detailed muscle action. MIMO provides N² observation channels in 3D from N sensing units on or above the body surface to enhance this critical spatial diversity with modest cost [36].

Radio-frequency (RF) signals in the UHF band, especially in the near-field region, will penetrate most dielectrics effectively without requiring direct skin contact. Therefore, our RMG system can be wearable over clothing or installed in a nearby off-body apparatus such as armrests and wrist pads.

To demonstrate that RMG can monitor the superficial and deep muscles, we carried out continuous recording of the complex forearm muscle contraction during various hand gestures by the MIMO channels, which provided rich information to decode the convoluted muscle activities. To validate HGR by RMG, we performed a human study of 8 participants with 23 gestures, including 8 basic gestures of fingers, palm, and wrist with multiple degrees of freedom (DoFs) and speeds. Various sensor modalities and forearm positions were also tested. After data segmentation, we transformed the 1D time waveforms to 2D time-frequency spectrograms using the short-time Fourier transform (STFT) and continuous wavelet transform (CWT). For gesture classification, we adopted vision transformer (ViT) as the deep-learning model [37] to compare with conventional convolutional neural networks (CNN). To provide a baseline comparison, we also benchmarked RMG with simultaneous sEMG for correlation and timing verification. To broaden the application scope, we also investigated leg RMG and radiooculogram (ROG), which tracked leg and eye muscle activities, respectively.

RMG can be applied to numerous applications. Gesture recognition and eye movement detection can be used as a middleware for HCI, such as virtual reality (VR) control and cybersickness reduction. In clinical applications, RMG can be used as assessment for voluntary and evoked muscle contraction, diagnosis of muscle disorders, and rehabilitation. RMG can be integrated with sEMG for possible diagnosis of neuromuscular disorders including Parkinson’s disease, as well as for feedback control of electromyostimulation (EMS). ROG can be further applied to rapid eye movement (REM) monitoring with eyes closed during sleep.

II. OPERATIONAL PRINCIPLES

A. Challenges for deep-layer muscle tracking

Muscles in a forearm are divided into anterior and posterior muscles, both containing superficial and deep layers. Hand gestures by the superficial muscle layers can be captured by motion sensors like accelerometers with tight skin contact. Deep-layer muscles are critical for HGR but can raise ambiguity for surface-based sensors. The forearm muscles actuating the hand gestures are listed in Table 1. For example, flexor digitorum profundus is the only major muscle that can flex the distal interphalangeal joints of the fingers, and four of deep posterior muscles are important for thumb and index finger movements. Hence, forearm muscle sensors for hand and wrist gestures will be able to differentiate hand gestures reliably only if all muscle groups, not just the surface ones, are included in the sensor readout. Here, RMG provides a new solution to detect muscle actuation in superficial as well as deep layers for accurate HGR.

TABLE I.

Major muscle groups generating the hand GESTURES.

Basic Gesture	Step 1	Major Muscles	Step 2	Major Muscles
Grasp	Extend 5 fingers	ED, EPL, EDM	Flex 5 fingers	FDP, FDS, FPL
Point Thumb	Extend thumb	EPL	Flex thumb	FPL
Point Index	Extend index	EI	Flex index	FDP, FDS
Point Ind.+Mid.	Extend ind.+mid.	EI, ED	Flex ind.+mid.	FDP, FDS
Point 4 Fingers	Extend 4 fingers	EI, ED, EDM,	Flex 4 fingers	FDP, FDS
Fist	Flex 5 fingers	FDP, FDS, FPL	Rest
Wrist Up	Extend wrist	ECU, ECRL,ECRB	Flex wrist	FCU, FCR
Wrist Down	Flex wrist	FCU, FCR	Extend wrist	ECU, ECRL,ECRB

Muscle groups: ECU: Extensor Carpi Ulnaris; ECRL: Extensor Carpi Radialis Longus; ECRB: Extensor Carpi Radialis Brevis; FCU: Flexor Carpi Ulnaris; FCR: Flexor Carpi Radialis; ED: Extensor digitorum; EDM: Extensor digiti minimi; FDS: Flexor Digitorum Superficialis; EPL: Extensor Pollicis Longus; EI: Extensor Indicis; FDP: Flexor Digitorum Profundus; FPL: Flexor Pollicis Longus.

Green font: Superficial; Blue font: Intermediate; Red font: Deep.

B. NCS: Near-field coupling inside the muscles

From near-field EM coupling, NCS directly modulates the superficial and deep muscle motion onto multiplexed radio signals. Previous radar-based systems often operated in the far field when the EM energy is mostly reflected at the body surface, so only the surface motion would be captured [38]. In comparison, NCS has more EM energy directed inside the body so the modulated signal from internal tissues and organs is significantly larger. In our previous studies, NCS has been validated for deep coupling into human body to monitor heart valve motion [36][39], femoral pulses [40], and diaphragmatic breathing [41][42].

RMG adopted the NCS principle for muscle monitoring. In the near-field region of the forearm, the sensing antenna is designed to couple more EM energy into the muscles with high signal-to-noise ratio (SNR). The dielectric change of the internal muscle groups during the manipulation of hand gestures will be represented as the RF antenna or channel characteristics in terms of the scattering (S) parameters. Owing to the high penetration capability of UHF in the near field, RMG can potentially monitor all muscle groups in the forearm.

C. MIMO: Rich N² channel by N points

In this work, we adopt MIMO to incorporate N² usable channels from N observation points [36]. MIMO is a mature RF technique where different transmitters (Tx) can be well isolated by either frequency or code multiplexing. Similar techniques can be employed by colors in vision and subcarriers in ultrasound, but RF MIMO offers higher channel isolation than optical or acoustic waves with much lower cost thanks to the mature wireless industry. N Tx can then be simultaneously received and demodulated by N receivers (Rx) to accomplish N² synchronous channels to fulfill the spatial diversity requirement to observe complex 3D geometry and motion. Due to tissue dispersion and near-field nonlinearity, the channel by Tx1–Rx2 would represent different information from Tx2−Rx1. Our RMG prototype utilized N = 4 sensing points around the forearm, and collected signals from 16 channels, and can be extended to more channels with modest system cost.

D. Electromagnetic simulation for RMG

We further demonstrated the sensing principle of RMG using a numerical simulation in CST Microwave Studio [43]. The human forearm model for EM simulation was constructed from the Tom anatomical model in the CST Bio Extension 4.0 library. This voxel-based forearm model has a resolution of 0.5 mm and contains accurate dielectric properties of the biological tissues in the UHF band including skin, muscle, blood, fat, and other tissues. Four dipole Tx antennas, as shown in Fig. 1(a), were deployed on the forearm circumference without direct contact, as was done in the box RMG experiment in the later section. Each Tx was driven with a 1-W input source at 1 GHz and 50-Ω source impedance. The power here was selected for normalization convenience, as the radiated power level was less than 1 mW in actual experiments. Fig. 1(b) shows the electric field magnitude originating from each of the four Tx antennas at the cross section of forearm. We can clearly observe that the electric field was strongly coupled into the layers of skin, fat, and muscles. The sensing locality of the RMG system can be observed by different antenna coupling into different nearby muscle groups, providing high observation diversity.

Fig. 1.

Demonstration of the near-field coupling principle in RMG by electromagnetic simulation. (a) The human forearm phantom in the CST Bio Extension 4.0 library with 4 dipole antennas around the arm circumference. (b) Electric field distribution in the cross section of forearm with excitation by antennas 1 – 4. (c) The normalized scattering (S) parameters of the four self backscatter channels for different muscle scales, and (d) of the selected cross channels. Different channels have dissimilar responses to enhance observation diversity.

To show the change in the antenna characteristics by the NCS principle during the different muscle contraction phases, we performed a mock muscle contraction in CST by isotropic muscle scaling of 1, 0.95, 0.9, and 0.85. Fig. 1(c) shows the normalized antenna reflections as the backscattering S parameters of S₁₁, S₂₂, S₃₃ and S₄₄, which have small percentage changes demanding differential extraction or direct-path cancellation [43]. Fig. 1(d) presented cross channels between 4 antennas. The magnitude of the cross channels is smaller, but contains distinctive features during muscle scaling.

III. System design

A. Experimental setup

The first RMG prototype employed four pairs of the sensing antennas attached to a wearable armband on the middle forearm, as shown in Fig. 2. Each sensing unit consisted of two monopole whip antennas (Taoglas TG.19.0112) mounted on a 3D-printed holder, and had a dimension of 69 × 17 × 11 mm.

Fig. 2.

The wearable RMG prototype by SDR. (a) System schematic; (b) Photo of a forearm placement. (c) Four sensing antenna pairs attached to the armband. (d) The cross section view of 4 measuring points. The third probe is on the anterior side of the forearm. (e) The transceiver setup by two SDR units.

The antennas were aligned in parallel to the forearm muscle for enhanced coupling. Unit 1 was placed close to extensor pollicis longus and flexor pollicis longus, which controlled extension and flexion of the thumb. Unit 2 was placed close to the extensor muscles, which produced extension at the wrist and fingers. Unit 3 was placed close to the flexion muscles, which were associated with pronation of the forearm, as well as flexion of the wrist and fingers. Unit 4 was close to flexor digitorum profundus which flexed the four fingers except the thumb. Multi-channel observation can help decode the convoluted muscle motion in various hand gestures by sensor proximity and rich MIMO channels.

The RMG transceiver was prototyped by two synchronized software defined radios (SDR, Ettus B210). The two SDRs were synchronized by an external local oscillator (LO, BG7TBL-GPSDO) with 10 MHz reference and 1 PPS (pulse per second) baseband synchronization. The SDRs were connected to a host computer through universal serial bus (USB), and the control software was implemented in LabVIEW. Each port of the MIMO system consisted of one Tx and one Rx, which was then connected to one sensing antenna pair. Each SDR supported two synchronous ports. Note that the present RMG on the armband were connected by cables to off-body SDR for fast and flexible prototyping of RF transceiver parameters. An all-in-one wireless unit of RMG can be a straightforward extension in the future [45], and further implementation by integrated circuits (IC) and custom packaging can make use of the present findings for product development with reduced size, power and cost.

The digital baseband tone f_BB of each Tx went through the digital-to-analog converter (DAC) and was then mixed with the carrier frequency f_RF in a standard quadrature scheme. The RF power was less than −10 dBm or 0.1 mW, well under the safety limits set by occupational safety and health administration (OSHA) in the UHF band [46]. The Tx signal was coupled into the forearm muscle groups, received by all Rx, and then demodulated and sampled by the analog-to-digital converter (ADC) to retrieve the baseband. We employed the quadrature scheme as the baseband tone f_BB, and the NCS signal can be represented by the amplitude and phase modulation on the quadrature signal as

$$NC{S}_{am}\left(t\right)=\sqrt{I_{Rx}{\left(t\right)}^2+Q_{Rx}{\left(t\right)}^2}$$ (1)

$$\mathit{\text{NC}}{S}_{ph}\left(t\right)=\mathit{\text{unwrap}}\left({\text{tan}}^{-1}\frac{Q_{Rx}\left(t\right)}{I_{Rx}\left(t\right)}-2\pi f_{BB}-{\theta }_0\right)$$ (2)

$$I_{Rx}\left(t\right)=A\left(t\right)\text{cos}\left(2\pi f_{BB}t+{\theta }_0\right)$$ (3)

$$Q_{Rx}\left(t\right)=A\left(t\right)\text{sin}\left(2\pi f_{BB}t+{\theta }_0\right)$$ (4)

where 𝜃₀ was the phase offset accumulated from the Tx−Rx signal chain and was not constant among different channels or setups. The antenna pair here can operate around 900 MHz and 1.8 GHz. Lower frequency often provided stronger penetration into human body and better signal coupling. Therefore, f_RF was selected at 900 MHz, where an in-body wavelength was close to the arm size around 5 cm assuming the relative dielectric constant of tissues around 36 – 64. The multiple Tx channels utilized frequency-division multiple access (FDMA) by setting f_BB =10, 25, 40, and 125 kHz, respectively, for Tx1−Tx4.

We configured the dual SDR as 4 self-channels and 12 cross channels. For example, Tx1 can be received by Rx1 as self backscattering, which was most affected by the muscle changes around Unit 1 to detect the extension and flexion of the thumb. Tx1 can also be received by Rx2−Rx4 as cross channels to collect information on the individual paths. All 16 channels are sampled at 10⁶ samples per second (Sps) to implement Tx FDMA, and further down-sampled to 500 Sps to retrieve NCS magnitude and phase.

B. Human study protocol

RMG was tested on 8 healthy participants as shown in Supplementary Table 1. The human study was approved by Cornell Institutional Review Board (IRB) under Protocol ID #1812008488, and conducted with the written consent of the participants. We designed 23 gestures including finger, palm, and wrist motions with various speeds and multiple DoFs as shown in Fig. 3. We had 8 basic dynamic gestures and 1 static resting gesture. Basic gestures were extended to three versions including quick, double quick, and slow, except that the gesture ‘Fist’ only had the slow version. These gestures are chosen for their common rendition in HGR, as well as for confirmation of deep muscle sensing. Every gesture was performed in a fixed time window of 5 s. All gestures excluding ‘Rest’ were dynamic and comprised two steps as described in Fig. 3. For the quick version, step 2 was performed immediately after step 1, while the slow version had a holding time around 2 s between steps 1 and 2. For each dynamic gesture, after step 2, the hand would relax back to the ‘Rest’ gesture (Supplementary Video). The on-off timing for each gesture motion inside the 5-s window was not fixed due to subject difference and variation in the response time for different repetitions. Each gesture was repeated around 30 times for each participant. The study procedures were divided into several repetitions of 5-min routines of two kinds. Routine 1 contained 16 finger-based gestures with 3 repetitions; Routine 2 contained 6 wrist-based gestures with 8 repetitions. The ‘Rest’ gesture was inserted between routines. Total recording time for each participant was around 1 hour. Participants occasionally made mistakes on the instructed gesture, and were suggested to report their mistakes after each 5-min routine. Routines with reported mistakes were removed from the datasets.

Fig. 3.

23 hand gestures used in the study protocol.

IV. Data Processing

After collecting data on multiple participants, we processed the raw data before feeding the output to the machine learning (ML) models for classification. The signal processing before learning helped de-noise the dataset and avoid overfitting, which will become apparent when comparison was made against the ML model based on raw waveforms. The data flow is shown in Fig. 4.

Fig. 4.

Schematic for data pre-processing to feed spectrogram into machine learning.

A. Multi-channel augmentation

From the MIMO configuration in RMG, we obtained 16 channels on a forearm. Each channel contained the baseband phase NCS_ph(t) and amplitude NCS_am(t) in the quadrature scheme on f_BB. In addition to employing phase and amplitude, we also augmented the original complex number as part of the information to retain the intricate relation between NCS_am(t) and NCS_ph(t) [44]. Therefore, from 16 MIMO channels, we had 48 temporal series in total.

B. Filtering and segmentation

The 48 1D waveforms in time was then processed by:

1. Bandpass filtering (0.1 Hz to 5 Hz) to eliminate the noises in the higher frequency.

2. Waveform normalization with center 0 and standard deviation 1.

3. Waveform segmentation into individual segments of T_seg = 5 s. Each segment now contained one gesture guided by voice instruction.

4. Waveform detrending by subtracting the best-fit linear line from the data within T_seg.

5. Annotation of the instructed gesture for each segment.

C. 1D waveforms to 2D spectrograms

We employed STFT and CWT to generate 2D spectrograms to feed into the ML model. Transformation of 1D time waveforms to 2D time-frequency spectrograms would bring forth significant improvement in accuracy. The ensemble of five 2D spectrograms from different transforms were incorporated into ML for classification. We explored two STFT outputs with different window lengths (T_win1 = 0.6 s and T_win2 = 1 s). Two window lengths can allow us to acquire information with different time and frequency resolutions [4]. CWT [5] takes advantage of multi-resolution analysis (MRA), which can effectively mitigate time-frequency resolution tradeoff. We used three different mother wavelets to capture different patterns and extend the feature diversity: 1) Ricker; 2) Gaussian; 3) Morelet.

Three columns in Fig. 5 represent gestures of (a) fast grasp, (b) double grasps, and (c) slow grasps in the 5-s segment. The first row of each column shows the RMG time waveform, and the second to fourth rows are STFT (T_win1 = 0.6 s), CWT1 (Ricker) and CWT2 (Gaussian) spectrograms. Note that STFT requires a short time window for n-point Fourier transform, so the starting and ending times of the spectrogram is truncated to 0.3 and 4.7 s.

Fig. 5.

Examples of 1D waveforms and transformed 2D spectrograms in three grasp-based gestures.

D. Classification by vision transformer

Though classical ML models can be computationally less expensive, algorithmic and hardware improvements in recent years have facilitated complex neural networks on embedded systems efficiently [47]. We implemented vision transformer (ViT) as the classification ML model, which has a deep-learning architecture inherited from the transformer model in natural language processing (NLP) [48] and is now gaining popularity in computer vision. To benchmark ViT performance, we also built a conventional CNN classifier.

Over the ViT architecture, patches of the 2D input image (size = 5) were constructed from the time-frequency spectrogram, and were then linearly embedded with dimension = 512. Position embedding was added, and the resulting vector sequences were fed to a standard transformer encoder. Inside the encoder, we had 6 transformer blocks, 16 heads in the multi-head attention layer, 64 dimensions of the multi-layer perceptron (MLP) (feed-forward) layer, and the dropout rate was set to 0.1. In CNN, each convolution layer was followed by a BatchNorm layer, and then 2 linear layers. Adam optimizer was used for both ViT and CNN.

V. Results and Analyses

A. Dataset composition

The final output dataset from all 8 participants consisted of 5,847 samples of 23 gestures. The ‘Rest’ gesture had 461 samples, each wrist-based gesture had around 283 − 288, each grasp-based gesture had 293 − 294, and each finger-based gesture had 215 − 222. Data exclusions were mostly due to reported mistakes by the participants, such as failing to follow the instruction on time and performing the wrong gestures.

B. The personal training model

We evaluated the classification accuracy of RMG by different cross-validation (CV) methods, feature sets and deep learning models. First, we built the personal training model for each participant. From individual person’s dataset, each gesture was repeated around 30 times, and the total sample number was around 700 − 800. K-fold (k = 7) CV was performed to estimate the mean accuracy for each participant. An overall accuracy was averaged on results from all participants. Fig. 6(a) shows the overall confusion matrix of ViT by the personal training model, which is normalized to the number of samples. RMG achieved an overall accuracy of 99.0% ± 0.48% for 23 gestures in the ensemble method by majority voting of all feature sets from 2 STFT and 3 CWT. Fig. 7(a) shows an example of the training and testing losses during model training on one subject. 6/7 of the overall data was trained, and 1/7 of the data was tested as unseen cases. We chose the learning rate of 10⁻⁴ and the epoch number of 20, and each epoch iteration has a batch size of 16. In the first 5 epochs, both training and testing losses decreased rapidly. In the following 5–20 epochs, training and testing losses both tended to be stable in low values, which indicated that the testing loss had a highly correlated decreasing pattern with the training loss and their values at the end did not have distinctive difference. Overfitting during training can often be spotted when the error on the training data decreased to a small value but the testing error increased in a reverse trend.

Fig. 6.

The confusion matrices showing the overall accuracy on all participants using (a) The personal training model; (b) Transfer learning on-the unseen participant by 1/5 of new data.

Fig. 7.

Performance of the personal training model. (a) Example of training and testing loss during model training (Epoch number=20, learning rate=10–4). (b) Accuracy using different portion of total dataset. (c) Accuracy using different transforms, (d) Accuracy for individual participants, (e) Accuracy by different ML models. (f) Accuracy using all RMG sensor units vs. individual sensor.

Fig. 7(b) then showed the accuracy when different portion of the dataset was chosen for training and testing. We changed to train on 4/5 (k = 5), 3/4 (k = 4), 1/2 (k = 2), and 1/4 (k = 4 with training and testing swapped). Though training on ¼ data had lower accuracy, when the model had at least half of the overall data to learn, the accuracy maintained at 96.9%. We can observe that the model performance did not degrade much even with limited training cases. These results indicated that there was no apparent overfitting in our classification model. We also compared the results using different feature sets individually and presented the results in Fig. 7(c). STFT2 used a longer time window length and thus had a higher frequency resolution than STFT1. CWT features generally outperformed STFT. CWT3 by Morelet wavelet achieved the highest accuracy of 98.6% among all individual feature sets. The ensemble method with the flexibility to choose among all alternatives achieved the highest accuracy. Multiple transforms on the same data can also be viewed as data augmentation in the learning process.

Our data were collected based on 5-min routines (around 60 repetitions of each gesture in total) in the hour-long study for each participant to allow some rest and sensor adjustment. Each subject had around 12 routines. Between the 5-min routines, the hardware would be reset, and the subjects would take off the sensor to rest. Small sensor position variation around 1–2 cm or rotation around 5–15° can occur between the sensor deployment. Due to the hardware restart and sensor position alteration, signals collected by different routines can have more variations. Therefore, we presented another CV process, where all routines were independently tested, i.e., the gestures in the same routine were never divided between training and testing. As shown in Fig. 7(d), routine-independent CV still achieved a high accuracy of 97.0% ± 1.27% for all 8 subjects. Compared with the random shuffle k-fold, routine-independent CV had lower mean accuracy and higher standard deviation across different subjects. This observation of maintaining high accuracy in routine-independent CV corroborated the system robustness against the hardware reboot and small position variation in practical scenarios.

ViT was compared to CNN in Fig. 7(e), where the accuracy dropped from 99.0% in ViT to 97.0% in CNN for personal training CV, and from 98.0% in ViT to 94.6% in CNN for routine-independent CV. To illustrate the advantage of 2D spectrogram, we also built a 1D-CNN model using the time waveforms directly. Accuracies dropped from 97.0% in 2D CNN to 88.5% in 1D CNN for personal training CV, and similarly from 94.6% to 88.0% for routine-independent CV. As ViT is often computationally more expensive than CNN, this comparison can be regarded as a tradeoff between accuracy and computing resources.

We adopted the MIMO setup for RMG in order to collect both self and cross channels. We also tested the HGR accuracy using only self channels as the input for the ViT model. Accuracy degraded from 99.0% to 95.0% in the personal training CV. Currently, we have 4 RF antennas functioning as sensing units on the armband. We further analyzed the accuracy degradation in Fig. 7(f) when the number of the sensing units was reduced. We chose one subject as an example where MIMO achieved higher accuracy than any other individual sensor used alone. Sensor 2 on the anterior side had the highest accuracy of 93.2%, while sensor 3 on the posterior side had the lowest accuracy of 69.9%. Large accuracy variation from different sensors was probably due to the different coupling to various muscle groups. In summary, the channel spatial diversity in MIMO played a critical role in HGR accuracy.

C. Transfer learning for unseen users

The HGR system must be robust to various practical conditions, especially for subject variation. Not only people perform hand gestures differently, but also the forearm size and muscle conditions have considerable distribution. Here, we adopted conventional transfer learning (TL) [49] where we leveraged a pre-trained model with large amount of data from multiple users to test on a new user with a small amount of individual training data. TL also has been widely used by other HGR systems with high generalization and low training burden [47][49]. We first generated the pre-trained model using all data from 7 participants. We then fine-tuned the model with 1/m data from the new participant as short personal calibration. The final model was tested on the rest (1 − 1/m) data. This CV process is similar to k-fold, but only one fold is for training, and (m − 1) folds are for testing. Fig. 8(a) shows the accuracies for all participants rotating as the new test case by the above TL strategy with m = 5. The model was entirely reset for each rotation. The averaged accuracy is 96.6%±0.74%, and the normalized confusion matrix is presented in Fig. 6(b).

Fig. 8.

Performance of transfer learning on unseen participants. (a) Accuracy using TL with m = 5 on individual participants. (b) Accuracy with and without TL for m = 4 or 5 by ViT and CNN.

ViT also outperformed CNN for our TL strategy [37]. Fig. 8(b) shows accuracies for m = 4 or 5 with and without TL in the ViT and CNN models. ViT achieved higher accuracy than CNN in every scenario. Direct learning from 1/m data without TL had much lower accuracy than the pre-trained model by TL. When the personal training set increased from ⅕ to ¼, accuracy also noticeably increased, indicating the trade-off between high accuracy and the amount of personal training data.

D. Variations in experimental designs

Apart from subject dependence, accuracy degradation can also be induced from the sensor placement on the forearm. To test the adaptability against large sensor position variation, we performed another test on one participant with the same protocol but with the sensor position moved to a higher position by d = 3 cm. We used the same TL strategy to achieve the result in the first two columns of Table II. After adopting TL, accuracy was boosted from 87.2% to 97.2%.

TABLE II.

Evaluation for position and design VARIATIONS.

	d = 3cm with TL	d = 3cm without TL	Notch	Box	Wrist
Accuracy (%)	97.2	87.2	99.0	97.4	95.8

For the human study above, we used the antenna-based sensing unit on the forearm. We also explored more sensor design variations. The first design in Fig. 9(a) was notch RMG, where the muscle motion was coupled to an RF coaxial cable with an open notch leaking out a portion of the EM energy [50]. The notch RMG has the potential to miniaturize the sensor size, and can be readily adapted to flexible wearables. The second design in Fig. 9(b) is a non-contact square box with the antenna sensors attached to the inside walls. Forearm can be placed into the box freely without direct contact. The third design in Fig. 9(c) is by the same sensing antenna, but placed on the wristband, which can be convenient for integration into the smart watch as a new input method. Present user interface by fingers on the smart watch display has been impeded by the small screen size, and hand gestures can be a promising alternative [22]. Table II presents the HGR accuracies using the above three design variations. Notch RMG showed the highest accuracy and can be favorable in certain applications. Box RMG can still attain reasonable accuracy by the non-contact setup, which further enhance the design flexibility over clothing or in armrests. Wrist RMG showed lower accuracy than the forearm placement because tendon and ligament motion had less dielectric contrast than the muscle motion.

Fig. 9.

Experimental setups of various designs: (a) A notch RMG; (b) A non-contact box; (c) A wristband; (d) Verification by the hand inside an RAM box; (e) Benchmark with sEMG with short +/− separation; (f) Slow grasp strength testing with sEMG and accelerometer.

To validate that strong RF coupling was from the forearm muscles and not from the direct hand motion in the radar mode, we conducted measurements with the hand inside a radar-absorption-material (RAM) box, as shown in Fig. 9(d), where minimal difference in collected waveforms and achievable HGR accuracy was observed.

VI. BENCHMARK WITH sEMG

We performed RMG with synchronous sEMG for the baseline comparison and physiological correlation. Notice that our sEMG setup had only one or two channels and was implemented by a commercial device without optimization. The performance of our sEMG study is expected to lag behind many state-of-the-art multi-channel implementations [27]. Nevertheless, the two sensing schemes can be complementary in operation to establish the complete physiological sequence of stimulation and actuation, as well as to study the neuromuscular disorders in the future.

A. RMG and sEMG placement

For the reference sEMG setup, we used BIOPAC MP36R with the SS2LB leads set and EL503 electrodes (BIOPAC Systems, Goleta, CA). Fig. 9(e) shows the experimental setup with RMG and EMG both on the forearm. Each EMG channel has 3 electrodes on skin as +, −, and ground. We used 2 sEMG channels on the anterior and posterior sides of the forearm. The ground electrodes for two sEMG channels were both placed close to a wrist spot with minimal muscles. RMG and sEMG channels were synchronized on Labview. We performed the same study protocol on two participants as Exp1 and Exp2 in Table III. The two participants had the same sEMG placement, where + and − electrodes were on the two longitudinal sides of the RMG armband to capture more differential signals with a large distance. Exp3 was the same participant as Exp1, but had a different sEMG placement where the + and − electrodes were on the lower position from the RMG armband. The smaller distance would measure only the muscles close to the electrodes with less voltage resolution.

TABLE III.

Accuracy comparison of RMG VS. SEMG

	Exp1	Exp2	Exp3	Mean
RMG	99.0%	98.5%	98.7%	98.7%
sEMG	68.2%	70.8%	66.7%	68.6%

B. RMG and sEMG waveform comparison

As our sEMG waveforms were noisy during the hand gestures, we added two pre-processing procedures: Enveloping the raw data by spline interpolation over local maxima, and smoothing by moving average [50]. The subsequent signal transformation and learning models were the same for sEMG and RMG. The overall HGR accuracy by 7-fold CV is shown in Table III. Accuracy of sEMG was relatively low in comparison with RMG in our setup, which may be caused by the small number of sEMG channels under the large number of gesture classes. Our sEMG implementation was mainly for comparative purposes and was far from ideal. A more comprehensive comparison with the literature results will be presented in next section.

As shown in Fig. 10, we also compared the averaged waveforms of different gestures obtained from RMG and sEMG using global dynamic time warping (DTW) [51]. Each gesture had a time segment of 5 s, while the y-axis was the normalized amplitude. The RMG waveforms were examples from Tx2-Rx2 and the sEMG from channel 2, both positioned on the posterior side of the forearm. For quick and double quick gestures, both RMG and sEMG presented sharp peaks corresponding to the fast muscle motion. However, compared with RMG, sEMG signals had longer duration of pulse waveforms and showed more tailing after the gesture motion terminated. For slow gestures, RMG showed a more consistent square-wave pattern from the holding period. The sEMG signal showed a shorter pulse duration for gestures that do not require continuous myoelectrical simulation such as the point-finger gestures. For other gestures that require continuous efforts to maintain the position such as the wrist up/down, the sEMG pulse duration was extended. During ‘Rest’ and between gestures with no intended hand motion, sEMG had more interference and ambiguity due to either hardware sources such as inconsistent electrode contact resistance or from biological sources such as the neural signals from vital signs [32]. In comparison, RMG is less susceptible to vital signs or noises from electrode contacts.

Fig. 10.

RMG and sEMG waveforms for various gestures by DTW averaging on all samples with the same gesture.

To compare the waveform features further, we performed peak detection in the 14 quick gestures. Fig. 11(a) shows the scatter plot of peak locations of quick gestures in synchronous RMG and sEMG in all samples, where the Pearson correlation coefficient r = 0.929 and the mean time difference is a delay of 0.183 s, i.e., RMG and sEMG have a high temporal correlation and a consistent time lag. This delay may indicate the time offset between neural stimulation and muscle actuation. Fig. 11(b) compares the feature of the pulse width, computed as the time duration between the points to the left and right of the half peak magnitude. Most data points are scattered above y = x line, which indicates that RMG waveforms have sharper peaks with less spreading during the quick gestures. Note that the few outliers are probably due to peak detection errors caused by the cases of questionable signal quality. For slow gestures, peak detection is not an appropriate comparison because the waveform features are not always consistent.

Fig. 11.

Scatter plots of RMG and sEMG for peak location and pulse width during quick gestures.

C. Timing and latency of RMG

RMG has ultra-low latency with the sampling rate readily over 10⁵ samples per second (Sps), which is important for dynamic HGR. Here, we performed the high-speed gesture tracking by RMG and sEMG. The participant followed a metronome of 150 beats/minute and performed the gesture of ‘point index and middle fingers’ with equal strength at each beat. The sensor setup was the same as Fig. 9(d). The waveforms from one of each RMG and sEMG channels are shown in Fig. 12(a). We can observe from the time waveforms that RMG had a consistent signal pattern corresponding to the quick motions, while sEMG had more fluctuations. Supplementary Table II also presents the statistics of estimated gesture frequency, which is very close to the ground truth with small standard deviation.

Fig. 12.

Waveforms recorded from RMG, sEMG, and accelerometer for (a) fast finger motion; (b) slow grasps in 3 times/min with equal strength.

Compared with surface-motion based sensors including MMG and accelerometers, RMG possesses the unique capability to capture deep muscle contraction. To further corroborate this claim, we tested a slow grip strength detection by RMG together with accelerometers and sEMG. As shown in Table I, during the grip motion, the main muscles include the flexor digitorum superficialis (intermediate), flexor digitorum

profondus (deep) and the flexor policus longus (deep) [52]. The participant performed firm holds on the hand dynamometer with a speed of 3 times/minute in equal strength as shown in Fig. 9(f). The waveforms from one of each RMG, sEMG, and accelerometer channels are shown in Fig. 12(b). RMG had a clear and stable signal pattern reflecting the strong and slow grip motion, while sEMG showed some ambiguity and the accelerometer presented even more noisy patterns. This is likely due to the different coupling strength to the deep muscle groups by different sensors. Supplementary Table II also shows the statistics of the estimated grip frequency, where RMG is accurate, and sEMG and accelerometer have more errors. The slow grip frequency is estimated by counting the number of detected cycles of pinching the hand then back to relaxation.

D. Extension to eye and leg RMG

To validate the general applicability of RMG to different skeletal muscles, we further extended the setup to wearable radiooculogram (ROG) on eyes and RMG on legs.

As shown in Figs. 13(a)(b), the ROG system integrated four notch RMG to a facemask around the eyes. A participant wearing ROG and electrooculorgram (EOG) was shown in Figs. 13(c)(d). In a human study of 5 subjects, participants were instructed to move eyes in four directions (up, down, left, and right) with eyes closed, all of which had 2 versions of moving once and twice. Hence, we had 8 distinctive eye movements, and each motion was performed in a time segment of T_seg = 5 s with around 24 repetitions for each participant (Supplementary Video). Then the training model within each participant was built and 7-fold CV was performed to estimate the mean accuracy. ROG achieved an overall accuracy of 94.2% (Supplementary Fig. 2(a)). ROG can monitor fine eye muscle activities with eyes open or shut. In the future, ROG can be applied for sleep REM and dream stage monitoring [53], and facilitate HCI applications using eye motion control.

Fig. 13.

Setup of the ROG and leg RMG systems. (a) One ROG sensor unit by a notched transmission line; (b) Four ROG sensor units on a mask; (c) ROG on a participant’s face; (d) EOG setup for baseline comparison; (e) One lower leg RMG sensor unit by whip antennas together with one EMG; (f) Four leg RMG sensor units on two legs.

Another extension is for monitoring lower leg muscles. We implemented 2 RMG sensing units on each leg with sEMG for reference, as shown in Figs. 13(e)(f). We tested 7 postures: 1) tiptoe standing; 2) tiptoe sitting; 3) reverse tiptoe standing; 4) reverse tiptoe sitting; 5) tiptoe sitting with only the right foot; 6) tiptoe sitting with only the left foot; 7) squat (Supplementary Video). Each posture was also performed in a time segment of T_seg = 5 s with around 34 repetitions. Leg RMG achieved accuracy of 100% for one participant using 7-fold CV (Supplementary Fig. 2(b)). RMG on lower legs can monitor body postures and can be applied for balance training and fall warning [54].

VII. DISCUSSION

A. Comparison to previous HGR works

A comparison of RMG to previous HGR systems is presented in Table IV. Li et al. [17] can achieve high accuracy of 98.5% for 8 classes, but requires off-body line-of-sight (LoS) cameras, and is vulnerable to change of light and background. Zhang et al. [33] used Electrical Impedance Tomography (EIT) to recover the interior impedance distribution of the tested arm, which was similar to RMG because both techniques monitored interior muscle activities. However, EIT needs many electrodes and suffers from reproducibility and accuracy degradation across users. Many previous efforts employed sEMG for HGR, which required direct skin contact and a large number of electrodes to achieve high accuracy. McIntosh et al. [30] successfully integrated pressure sensor with sEMG, but needed 8 wet electrodes and 4 pressure sensors. The recent work from Moin et al. [27] showed 92.9% for 21 gestures with high in-sensor adaptability, but required 64 electrodes integrated on the armband. In comparison, RMG only utilized 4 sensing units to achieve 16 channels by MIMO. In this work, we demonstrated the competitive RMG to recognize 23 gestures (including 8 basic gestures in three different speeds) with accuracy up to 99.0%. Notice that the number of hand gestures was different in various works due to the intended applications, and the larger number did not directly indicate higher sensor capability except for the increased complexity in the classification algorithm. Moreover, the wearable and armrest RMG setups without requiring direct skin contact or restricting the capture volume offer inherent operational advantages over sEMG and camera-based systems. Our choice of ViT classification on spectrograms also shows better performance than traditional ML models adopted in previous works.

TABLE IV.

Comparison to previous works

	Li 2019 [17]	Zhang 2016 [25]	Zhang 2015 [33]	McIntosh 2016 [30]	Savur 2016 [14]	Qi 2020 [29]	Côté-Allard 2019 [47]	Moin 2021 [27]	This work
Class	8	8	5	15	27	9	7/18	13/21	23
Subject	5	4	10	12	1	-	17/10	2	8
Sensor	Camera	FMCW Radar	EIT	sEMG+pressure	sEMG	sEMG	sEMG	sEMG	RMG
Model	CNN	CNN	SVM	SVM	Ensemble	GRNN	ConvNet	Neural	ViT
Accuracy	98.5%	96.0%	97% (hand) 87% (pinch)	95.8%	79.4%	95.3%	98.3% (7) 69.0% (18)	97.1% (13) 92.9% (21)	99.0%

B. Potential future improvements

1). Sensor hardware improvement

In future hardware implementations, we should be able to miniaturize RMG into convenient and comfortable packages as all-in-one wireless wearables, because the expected power consumption and data bandwidth are both very low in view of modern RF devices. The notch RMG offers a promising design path to reduce cost, form factors, and complexity, especially for integration with a wristwatch.

2). Real-time classification for HCI

For HCI in robotic and gaming control, real-time HGR with minimal latency is an important feature. Embedded learning capability with local signal processing and accurate HGR output of RMG will be attractive to many applications. The computational load of signal processing and ML classification was presented in Supplementary Note 3. With a given pre-trained ViT model, an inference on a single gesture presently took less than 1 ms for the execution time in a modest PC gaming console. Further custom hardware acceleration and algorithmic optimization can be applied to enable future real-time HGR.

3). Fusion with sEMG

sEMG can estimate neural stimulation of muscle actuation, and RMG can directly detect the actual muscle change. Thus RMG should not be viewed as a competition of sEMG, but the two sensors can be combined for a fuller physiological interpretation. Our consistent observation of the RMG delay from sEMG possibly indicated the non-trained muscle actuation without the participation of proprioceptive neurons, which can be promising for neuromuscular disorder diagnosis with RMG and sEMG fusion.

4). Closed-loop EMS control

A closed-loop control of EMS is another possible future application. EMS has long been employed to either supplement or substitute voluntary muscle stimulation in many settings of rehabilitation and electroceuticals [55]. However, inadequate EMS due to personal and daily differences can cause confusion of antagonistic and synergistic coordination of the muscle groups, and even induce serious spasm. For a more precise control on EMS, RMG can give feedback on actual muscle actuation to control the EMS signal with higher adaptability to personal and conditional variations.

VIII. CONCLUSION

We have reported a novel muscle monitoring technique, named as radiomyography (RMG), which can directly measure the muscle motion by coupling RF energy to superficial and deep internal muscles. Operation over clothing without direct skin touch enables convenient setup and comfortable operation.

The MIMO approach enriches the collected information with a relatively small number of sensing points. We implemented RMG as a wearable forearm sensor to accurately track forearm muscles. For the HGR purpose, we adopted ViT as the classification model and effectively boosted the accuracy up to 99.0% for 23 hand gestures tested on 8 participants. We further adopted TL to address cross-subject and operational variations. For HGR systems, RMG has lower cost, lower complexity, lower latency and less privacy issues than camera-based devices, as well as higher user comfort and accuracy than contact-based devices.

RMG has the inherent advantage to monitor internal muscles non-invasively. In the future, RMG and sEMG can be fused together to derive the closed-loop information of stimulation and actuation. RMG can potentially lead to new methods for assessment of muscle functions, monitoring of muscle fatigue, and diagnosis of neuromuscular disorders. RMG is also promising for other HCI applications including exoskeleton robotic control, virtual reality interface, and in-air gesture capture.

Biographies

Zijing Zhang received a B.Eng. degree in optoelectronic engineering from Huazhong, University of Science and Technology (HUST), Wuhan, China, in 2019. She received the Ph.D. degree in Electrical Engineering at Cornell University, Ithaca, NY, USA in May 2023. She is now a sensor engineer in Apple Inc., Cupertino, CA, USA.

Edwin C. Kan received B.S. from National Taiwan University in 1984, and M.S. and Ph.D. from University of Illinois, Urbana-Champaign in 1988 and 1992, respectively, all in electrical engineering. He has served in the Air Force, Taiwan, R. O. China as a second lieutenant from 1984 – 1986. In January 1992, he joined Dawn Technologies as a Principal CAD Engineer. He was then with Stanford University as a Research Associate from 1994 to 1997. In 1997, he joined School of Electrical and Computer Engineering, Cornell University, Ithaca, NY as an assistant professor, where he is now a professor.