Towards Real-Time, Continuous Decoding of Gripping Force From Deep Brain Local Field Potentials

Abstract

Lack of force information and longevity issues are impediments to the successful translation of brain–computer interface systems for prosthetic control from experimental settings to widespread clinical application. The ability to decode force using deep brain stimulation electrodes in the subthalamic nucleus (STN) of the basal ganglia provides an opportunity to address these limitations. This paper explores the use of various classes of algorithms (Wiener filter, Wiener-Cascade model, Kalman filter, and dynamic neural networks) and recommends the use of a Wiener-Cascade model for decoding force from STN. This recommendation is influenced by a combination of accuracy and practical considerations to enable real-time, continuous operation. This paper demonstrates an ability to decode a continuous signal (force) from the STN in real time, allowing the possibility of decoding more than two states from the brain at low latency.

I. Introduction

BRAIN Computer Interface (BCI) systems convert signals recorded from the brain into useful information (e.g. infer a command, estimate state of the brain) in real-time with potential applications in a number of areas. This is of particular interest for patients with a wide spectrum of motor deficits of different origin and severity. This includes wheel-chair control for patients with limited mobility [1], communication tool for patients with inability to speak (e.g. ALS or locked-in patients [2]), neural control of prosthetic limbs after traumatic amputation [3] and rehabilitative approaches after acute neurological disorders such as stroke [4]. For a BCI technology to be developed for widespread use beyond experimental settings, the signal recorded must remain stable over a long period and the conversion of signal to information must be robust, stable and reliable. The amount of information captured by a BCI system depends on the number of states that can be decoded, the accuracy of decoding and the speed at which this decoding occurs (latency).

A variety of signal modalities can be used for recording signals from the brain for BCI. These can be non-invasive and include the electroencephalogram (EEG) [5], functional magnetic resonance imaging (fMRI) [6], and functional near infra-red spectroscopy (fNIRS) [7], or invasive such as the electrocorticogram (ECoG), cortical single/multiunit recordings [8] and local field potentials (LFP) picked up using depth electrodes from deep brain structures [9]. EEG suffers from low signal to noise ratio (SNR) and significant volume conduction [10]. fNIRS suffers from poor temporal resolution. Practical considerations limit the use of fMRI outside experimental settings and this signal also provides poor temporal resolution. Furthermore, the majority of non-invasive BCI studies to date rely on decoding two states (e.g. distinction between movement/no movement [2]). Consequently, most have only been experimental in nature.

In contrast, BCI systems using invasively recorded signals have shown greater potential to restore functions important to everyday life, such as reaching and grasping [3], [11]. This is because of the richer information content of these signals. Yet a number of challenges still need to be addressed before invasive BCI systems can be rolled out into routine clinical and home use. Research into BCI for prosthetic control has, to date, almost exclusively focused on decoding only kinematic variables (position or velocity) from cortical signals. A BCI-controlled device unable to incorporate force information in reaching and grasping tasks is unlikely to be useful and almost impossible to integrate in a patient’s life for routine, everyday tasks. Thus far arguably the best clinical demonstration of BMI has still not been able to accurately manipulate the force level applied by a robotic hand [3], [11]. Consequently, the development of a BCI system able to decode force would significantly improve the current state-of-the-art of BCI-based prosthetic devices.

Another issue with BCI systems is the stability of neural interfaces over time. This remains limited to a few years at very best for devices directly detecting the discharge of cortical neurons [12]. Perhaps for this reason there has been a shift in interest to decode ECoG activity picked up by subdural grids, but these most often require craniotomies for their placement. In contrast, the LFP activity recorded from deep basal ganglia structures is known to be stable over many years [13], and electrode implantation involves a limited burr-hole and surgical techniques that have been honed through the routine use of this procedure for therapeutic purposes in Parkinsonian patients [14]. Accordingly, the surgery is relatively benign, and in epileptic patients in whom both temporary subdural grids and depth electrodes are often used there is evidence to suggest that the latter may be safer [15]. Here we demonstrate the decoding of force from the LFP recorded in the Subthalamic Nucleus (STN) of the Basal Ganglia. This could prove useful as a selection signal in communication aids, potentially providing a higher bit rate than has been currently demonstrated in locked-in patients [2], or as an adjunct to ECoG-based kinematic control of robotic devices.

The current study builds on our previous work [9], [16], [17] towards the development of a BCI system for continuous decoding of force employing depth electrodes. Our previous work [16] found that gripping force was encoded mostly in the beta (13-30 Hz) and gamma (55-90 Hz) bands, with beta band desynchronization during gripping initiation and gamma band synchronization during gripping. In the previous work, the magnitude of gamma band synchronization correlated with the amount of force. Considering real-time implementation constraints, the simplest possible decoder using a linear combination of features with lags (the Wiener filter) was initially investigated, for its simplicity, interpretability and ease of implementation. To capture any non-linearity, the Wiener filter was cascaded with a polynomial non-linearity (the Wiener-Cascade model) and any improvement determined. In addition, Kalman filter based algorithms (1-D and 2-D) were also investigated since they offer an interpretable framework allowing the option to decode both force and yank. Such algorithms represent the state-of-the-art in many successful BCI applications [18]. Lastly, a dynamic Neural Networks-based algorithm was also investigated to ensure that any non-linearity not captured by the former approaches could be modeled. Challenges posed by the demands of real-time implementation (feature extraction, feature selection, channel selection, window length and step length) were also investigated and appropriate strategies are recommended to address them. Data recorded from 5 patients with idiopathic Parkinson’s disease having bilateral depth electrode implantation were used in this study. Their electrodes were implanted for the purposes of therapeutic stimulation but used here for recording without stimulation.

II. Methods

This section will start by briefly describing the experimental paradigm and the dataset collected, the time-domain and frequency domain features extracted, and then explain the different decoding algorithms explored in this study: Wiener filter, Wiener-Cascade model (WC), Kalman filter (KC) and Dynamic Neural Networks. The processing pipeline from input to output is illustrated in Figure 1. The LFP from STN (input) is processed for feature extraction followed by a choice of a decoding algorithm to provide the decoded force (output). Each STN DBS lead provides 3 bipolar channels. Features can be extracted from either a single channel deemed to be the best in terms of localization or all channels thereby constructing a high-dimensional feature space. All the algorithms in this study were evaluated with a 5-fold cross validation, dividing the original LFP time-series into five continuous chunks and then training on 4 chunks of data while evaluating the performance on the remaining fifth chunk. All results reported are based on correlation between the test set and the original force of the hand (contralateral to the STN) as measured by the dynamometer.

Fig. 1.

Overview of the processing pipeline from input (LFP) to output (decoded force).

A. Dataset Collection

This work is based on data collected from 5 Parkinsonian patients providing 8 STNs who underwent surgery for bilateral implantation of deep brain stimulation (DBS) electrodes targeting the motor area (dorsolateral [19]) of the STN. These patients were externalized (DBS leads were accessible for recording) and kept in a hospital for about a week prior to a second surgery for implantation of a pulse generator under the skin while connecting the DBS electrode leads to the pulse generator. All the recordings in this study were undertaken prior to the second surgery (3-6 days after DBS electrode implantation) while the patients were on their normal anti-Parkinsonian (levodopa) medication. The experimental paradigm consisted of a visual cue presented on a screen (red Light Emitting Diode emitting for 3 seconds) and patients were instructed to grip a dynamometer (Biometrics Ltd.) at an effort level of their choosing ranging from very low to maximal effort in a random order for as long as the light appeared. The generated force and the LFPs from both sides of the STN of the patient were simultaneously recorded with a sampling frequency of 2048 Hz using a TMSI amplifier. Subsequent analysis was performed off-line in Matlab (R2015b, Mathworks, US [20]). This study was approved by the local ethics committee and all the patients provided informed consent for their data to be used for research purposes. Table 1 provides the demographic details of the patients including the duration that the patient had PD. The paradigm is explained in more detail in [16].

Table I.

List of Patients and the Sites That Were Selected for Continuous Force Decoding Based on Gamma ERS And Clinical Assessment

Gender	Age (years)	PDa duration (years)	STNs (L:Left, R:Right)
Male	49	13	L,R
Male	56	10	L,R
Male	56	10	R
Female	66	16	L,R
Male	52	7	L

^aPD: Parkinson's disease

^bERS: Event-Related Synchronization

B. Feature Extraction

A sliding window of a fixed length (with overlap between consecutive windows) was used, in order to extract various features from the LFP. The various features explored can conveniently be divided into time-domain and frequency-domain features depending on how they were extracted from the LFP (Figure 2 shows an overview).

Fig. 2.

Overview of various time-domain and frequency-domain features.

1). Time-Domain Features

The time-domain features included the mean LFP, and the Hjorth parameters (activity, mobility, and complexity) [21] computed for each sliding window.

2). Frequency-Domain Features

The Discrete Fourier Transform (DFT) with a Fast Fourier Transform (FFT) algorithm of a signal provides the amount of power in different frequency bands presuming local stationarity of the signal. The short-time Fourier transform (applying FFT on fixed, short length segments) and wavelet transform have been developed to deal with non-stationary signals.

Estimating power in specific frequency bands has been widely used in neuroscience [22]. For example, increases in power in the gamma band (55-95 Hz) and decreases in power in the beta band (13-30 Hz) at movement onset have been reported in several studies. These studies typically employ a wavelet transform, and then average time-frequency plots from several trials in order to investigate power changes with cue or movement onset [23]. The most common wavelet used in such studies is the Morlet wavelet (a sinusoid multiplied by a Gaussian) in order to capture ‘sinusoidal’ features. A distinctive advantage of wavelet transform is the ability to identify any arbitrary type of signal feature by using an appropriate wavelet, and a variable window size to adapt time-frequency resolution by using a longer segments for low frequency (for better frequency resolution) and shorter segments at higher frequency (for better time resolution) [22]. However, real-time applications have a number of practical limitations including a fixed buffer size leading to fixed window sizes, and a limited time for processing (to avoid potential packet loss). Consequently, a different approach has to be employed for estimating power in different frequency bands. The two fundamentally different approaches that can be used are either the short-time discrete Fourier transform using the FFT algorithm (a non-parametric approach) or Autoregressive modeling (AR), a parametric approach for spectral estimation. A potential problem with the FFT approach (while using short, finite segments of data) is the edge effect due to spectral leakage. This occurs due to the finite duration of signal and it can be addressed by multiplying the signal with a finite duration window. The choice of window depends on the application and there is a trade-off between frequency resolution (the frequency resolution improves with decreasing width of the primary lobe of the window function) and the amount of spectral leakage (spectral leakage decreases with smaller side-lobes). In this work, we explored the use of a rectangular window and a Hamming window. Compared to a Hamming window, a rectangular window provides better frequency resolution at the cost of increased spectral leakage.

A major limitation of using FFT-based approaches on short, finite, length segments is limited frequency resolution. In contrast, AR-based spectral estimation provides infinite resolution and has been found to be superior to FFT-based methods in a number of physiological signals [24], especially when long segments of data are not available. Nevertheless, it is challenging to select an appropriate model order for AR-based spectral estimation. An innovative approach, previously developed to address this challenge, is to generate a large number of AR models and then estimate the median spectrum. This method is explained in greater detail in [25], and was employed in this work.

The frequency bands used as features in this study were 5-7 Hz, 8-12 Hz, 13-20 Hz, 21-30 Hz, 31-45 Hz, 56-95 Hz, 106-200 Hz, 201-300 Hz, 301-349 Hz and 350-500 Hz. Some frequency bands were not included; very low frequency (0-4 Hz), as this may be confounded by movement artifacts, and 46-55 Hz and 96-105 Hz which may be significantly corrupted by powerline interference.

C. Wiener Filtering

A Wiener filter attempts to find the weights of a linear equation where the output is equal to a weighted sum of different predictors. This method can also be viewed as multiple linear regression where the dependent variable is the estimated force, and the explanatory variables are the time-domain and frequency-domain features at various lags. Equation (1) represents such a linear equation where x_i (n − j) is the i^th feature at a lag of j, a_ij is the corresponding weight to be determined with M features for up to N lags and F′(n) is the estimated force at time n. These weights are determined using an optimality criterion based on minimizing the sum of the squares of the differences between the estimated force (F′(n)) and the actual force (F(n)). Solution of this set of linear equations has a closed form solution (commonly known as the Normal equation, or the Wiener-Hopf equation [26]) given in equation (2) and can thus be determined very fast on any computing device. In this equation, X is a P × M(N + 1) matrix of all features at all lags, A is a M (N + 1)×1 matrix of a_ij, P is the total number of samples (corresponding to the total number of sliding windows) and F is the P × 1 vector of actual force. It is also straightforward to add a regularization term (A², ridge regression [27]) in order to deal with overfitting. Equation (3) show a modification of the normal equation to include regularization parametrized by λ.

$$F^{\prime }(n)={\displaystyle \sum _{j=0}^N{\displaystyle \sum _{i=0}^M{a}_{ij}{x}_i(n-j)}}$$ (1)

$$A={(X^{T}X)}^{-1}{X}^{T}F$$ (2)

$$A={(X^{T}X+\lambda I)}^{-1}{X}^{T}F$$ (3)

D. Wiener-Cascade (WC) Model

A limitation of Wiener filter is its inability to model any non-linear relationship between the input and output. An approach that has previously been employed in many applications [27] is to cascade a static non-linear unit to the output of the Wiener filter thereby enabling the modeling of non-linear systems. In this work, the output of the Wiener filter is adjusted with a third order polynomial to model any non-linearity in the system. The weights of the polynomial function are determined with least squares optimization.

E. Kalman Filtering

KF-based methods are one of the most widely used algorithms in BCI studies for prosthetic control [3] and cursor control [18]. KF provides an explicit model for system dynamics, and noise in the system. A KF is a special case of a recursive Bayesian filter with two probabilistic models: state estimation equation (a prior model) describing the evolution of the states, and a measurement model relating measurement to these states. The standard form of KF algorithm assumes that both the measurement and state models are linear. When the uncertainties associated with both the state model and measurement are Gaussian with zero-means and known covariations, the KF is an optimal estimator minimizing the mean square error of the estimated parameters. Equation (4) shows the measurement equation relating the neural activity, x_n ϵ R^M(N+1), with the state, s_n, at the current instant, n, through a linear measurement matrix, H ϵ R^M(N+1)xd, with noise, q_n. In this work, s_n = [force]_n for the 1-D case (d = 1) and $s_n={\left[forceyank\right]}_n^T$ for the 2-D case (d = 2). The noise q_n is assumed to be normally distributed with zero mean i.e. q_n ~ N (0, Q_n), Q_n ϵ R^{M(N+1)xM(N+1)}. The measurement vector, x_n, is a M(N + 1) dimension vector representing all the M features for up to N lags. Equation (5) shows the state evolution equation relating the state,s_n, at instant, n, with state, s_n−1, at instant, (n − 1), through a state transition matrix, A ϵ R^dxd, and noise, w_n, assumed to be normally distributed with zero mean i.e. w_n ~ N (0, W_n), W_n ϵ R^dxd. Consequently, the matrix A would be an identity matrix in the case of 1-D model ((d = 1) and it would be as shown in equation (6) in the case of 2-D model (d = 2) where dn is the time interval between successive iterative steps of the KF algorithm. The measurement matrix, H, and the noise covariance matrices (W and Q) are learnt with a least-squares optimization approach framing the problem as a system identification task similar to the approach developed in [28]. Once these parameters are estimated during the system identification stage, an iterative procedure is used to compute a new state value s_n at every time instant, n.

$${\mathit{\text{x}}}_n=H{\mathit{\text{s}}}_n+{\mathit{\text{q}}}_n$$ (4)

$${\mathit{\text{s}}}_n=A{\mathit{\text{s}}}_{n-1}+{\mathit{\text{w}}}_n$$ (5)

$$\mathit{\text{A}}=\lceil \begin{array}{cc}1& 0\\ dn& 1\end{array}\rceil$$ (6)

F. Dynamic Neural Networks

Neural Networks offer a powerful class of computational tools due to their ability to model complex, non-linear relationships between input and output. However, their uptake has been limited due to their susceptibility to overfitting and expensive computational requirements. With increasing computational capabilities and innovation in algorithms, these algorithms (especially deep learning) have been found to be increasingly useful as opposed to more classical approaches in applications with large amounts of data (e.g. Natural Language Processing, Speech Processing and Image Processing with several applications in healthcare [29]). Within the class of Neural Networks, dynamic neural networks are the most suitable for time-series analysis as opposed to static neural networks since the former have memory and can thus model temporal relationships. In this work, we explored the use of a (N − 10 − 1) neural network with N equal to the number of features, 10 hidden units and a single output. The input layer included a tapped delay line, with delays of up to 20 time steps. The parameters of the neural network were learnt with a gradient descent technique with the Levenberg-Marquardt algorithm (implemented in Matlab [20]) and early stopping on the validation set was used to avoid overfitting.

III. Results

Figure 3 illustrates a typical profile of force (shown in blue) and the corresponding force level estimated (shown in red) by using a decoding algorithm on LFP data recorded from a single patient. In this particular case, the decoding algorithm was based on the Wiener-Cascade model with a 256 milliseconds sliding window, using a step of 50 milliseconds, a lag of 30 (with a 2048 Hz sampling frequency) and a 5-fold cross validation. The decoded force shown in red is based on the test data and provided a correlation of 0.76. The reference force (measured by the dynamometer) was normalized by the maximum gripping force for each patient.

Fig. 3.

Actual force (blue) and estimated force (red) using a Wiener-Cascade model on a single patient (256 milliseconds, window, with a 50 milliseconds step) on test data

All methods were investigated at different lags (0, 10, 20, 30 and 40 time windows). The Kalman filter did not offer additional benefit after introducing a second dimension for modeling yank. With enough complexity, neural network-based algorithms can learn any transfer function and should therefore theoretically lead to the best performing algorithms. However, getting the optimal weights for neural network units is challenging and heavily dependent on getting an appropriate architecture for the problem at hand. For lags of more than 10, the WC model outperformed all the other methods and was consequently selected for subsequent analysis.

Apart from choosing an appropriate decoding algorithm, our ability to decode force from STN also depends on the accuracy of localization of the DBS electrodes. Ideally, these electrodes should be positioned to record from the motor areas of the STN (dorsolateral part [19]), also the preferred clinical target, but this can be difficult. A proxy measure of electrode localization is the increase in power of the low gamma band with movement onset. An approach, adopted previously in [16], is therefore to determine the average increase in power of the gamma band with gripping onset (termed as Event-Related Synchronization or ERS) across all movement trials for all bipolar channels and identify the channel with the most ERS to be the optimally placed channel. An alternative approach is based on feature extraction using all bipolar channels thereby creating a larger matrix for the decoding algorithm. From Figure 4, it can be seen that using all channels improves performance up to a lag of 20. This approach, however, is prone to overfitting and therefore requires more computational time to find the optimal regularization parameter. This explains the superior performance with a single channel as opposed to using all channels for lags greater than 30. Consequently, selecting a single bipolar channel based on ERS for feature selection is recommended.

Fig. 4.

Comparison of decoding performance using the WC model at various lags, with a single bipolar channel and with using all bipolar channels. Note that the WC model uses features from all windows, from the lag time (t = lag) to the time of decoding (t = 0)

This work also explored the optimal selection of window length and step length. A longer window length will allow more data to be utilized for feature extraction while the step length will determine the frequency of update. From Figure 5, it can be seen that the mean correlation is lowest for a step length of 20 milliseconds. The performance improves as the step length is increased from 20 milliseconds to 40 milliseconds. Any further increase in step length only marginally improves the mean correlation. Based on these results, a step length of greater than 40 milliseconds is suggested. Comparing the median values at step lengths of greater than 40 milliseconds, it can be seen that the median correlation performs better while using a window size of 256 milliseconds, as opposed to 512 milliseconds. While the mean marginally improves with longer windows, a window size of 256 milliseconds is more appropriate to ensure generalizability of these results (in order to avoid analyzing previous trial data to decode force for the current trial).

Fig. 5.

Comparison of performance of force decoding with various combinations of step length and window duration.

This work also investigated the importance of various features used in decoding force from LFP. Given 14 features, it would take (2¹⁴ = 16, 384) regression models in order to investigate every possible combination of features (assuming every other parameter is fixed) to determine the most important features. Such an exhaustive procedure is prohibitively computationally expensive. Consequently, there are alternative strategies for identifying the most important features in a classification or regression task. This includes sequential search (forward or backward [30]) and Least Absolute Shrinkage and Selection Operator (LASSO) [31]. LASSO is a regularization technique that works by modifying the cost function to include absolute sums of all weights that are to be determined. The amount of regularization is increased systematically, from no regularization to include all features, to heavy regularization to knock out all features. The features are knocked out (by getting a magnitude of 0) depending on the importance of the feature. Consequently, the LASSO technique can help us rank various features based on their importance. In this work, we applied the LASSO based feature selection and ranked all features based on how soon they were knocked out as regularization was increased. Table 1 provides a table of all the 14 features, sorted according to their mean ranking across all patients. From the table, it can be seen that the two most important features for force decoding are the low gamma band (56-95 Hz) and high beta band (21-30 Hz). The wide range of rankings does suggest that the rankings do vary from patient to patient and it is still advisable to use all the available features for force decoding. This is further corroborated by Figure 6 which shows that adding additional features on top of the top 2 identified here (21-30 Hz and 56-95 Hz) improves overall performance (based on analysis of 5 STNs that resulted in a correlation of at least 0.5). This figure also provides information on the performance detriment of using only features from frequencies of up to 95 Hz in order to reduce the sampling rate and thus reduce energy demands when the proposed algorithm is implemented in implantable devices.

Fig. 6.

Addition of different features on top of 'Reference'(top 2 identified via LASSO) lead to a better overall performance (“<100 Hz” corresponds to using all frequency-domain features up to 95 Hz, “<100 Hz+time” corresponds to the addition of time-domain features to frequency-domain features of up to 95 Hz, “all” corresponds to using all the 14 features).

This work also investigated any dependence of force decoding performance on the magnitude of gripping force. Figure 7 shows the procedure adopted to determine the force decoding error with respect to the maximum gripping force attained in each trial. A small window was chosen and placed around the maximum gripping force (identified by red crosses for each trial) in each trial (shown by the green margins in the figure). The maximum decoded force in that window was then identified (decoded force is shown in red in the figure). Figure 8 shows the regression line fitted for the single patient showing the estimated force and the actual force. The gradient of 0.74 suggests that, although force tended to be consistently under-estimated in this patient, estimates were no better or worse for any particular grip force.

Fig. 7.

A small window centered around the maximum gripping force for each trial was used to extract the maximum estimated force and the actual maximum force.

Fig. 8.

Actual maximum force and estimated maximum force for each trial for a single patient, along with fitted regression line and 95% confidence bounds.

Subsequently, we investigated the performance of force decoding with various force levels for all the patients. Figure 9 shows the results (similar to Figure 8) but for all the 8 STNs. At the group level there was the suggestion that force estimates were proportionately more variable at very low grip forces. This was confirmed by correlating estimate error per unit actual force with the actual force magnitude. Figure 10 shows that the mean absolute error per unit force magnitude does not depend on the magnitude of the actual force except for very low forces (~<0.1). The force decoding therefore performed poorly at these low forces, possibly due to the higher cognitive load associated with the performance of such fine force levels. Elsewhere it has been suggested that the STN codes for effort rather than force per se, although under most circumstances the two will be indistinguishable [32].

Fig. 9.

Actual maximum force and estimated maximum force for each trial for 8 STNs, along with fitted regression line and 95% confidence bounds.

Fig. 10.

Absolute difference between actual and estimated maximum force per unit force for each trial for 8 STNs.

Finally, Figure 11 shows the result of applying various feature extraction techniques ranging from using Periodogram with FFT (with a rectangular window and a Hamming window), and Autoregressive Modeling (using both a single order of 7, and a median spectrum-based approach). For comparison, the results of applying a wavelet-based technique (Morlet wavelet) are also shown. These results suggest that the FFT-based technique was superior to using AR modeling-based approach. While AR modeling results in a less noisy method and has been found to perform better in certain applications, attempting to find broad-band frequency features is better achieved using an FFT-based approach. A further challenge in AR modeling is selecting the right model order. This work attempted to use the median spectrum based technique suggested earlier in a different context [25] but overall, the FFT based method still outperformed AR based methods.

Fig. 11.

Comparison of force decoding performance using different spectral estimation techniques for feature generation.

IV. Discussion

In this work, we developed an optimal algorithmic approach for decoding force and explored various issues pertinent to the development of a BCI system, able to function in real-time to investigate the decoding potential of several time-domain and frequency-domain features (14 in total). A Wiener filter-based algorithm was found to be most suitable for force decoding. This algorithm has a closed form solution (both with and without regularization) and is therefore computationally efficient. Any non-linearity present in the system can be captured by cascading a static non-linearity to the Wiener filter (the WC model). The performance of force decoding did not improve with a Kalman filter, even if yank was added as an additional state. While some previous studies have reported improvement with KF algorithms [33], [34], there have also been studies reporting no additional benefit with using a KF algorithm [28], similar to our findings. KF-based algorithms are likely to lead to improvement in situations where there are neural features independently encoding the force yank. A previous study has proposed using unscented KF to model nonlinear relationships between input and output [35]. However, having explored non-linearity with a WC model and Dynamic Neural Networks, it was considered unlikely that unscented KF would lead to any further improvement over and above than shown with WC in the current study.

For spectral estimation of streaming data in real-time, either an FFT-based algorithm (Periodogram) or an AR-based algorithm could be used for extraction of frequency-domain features. However, even if the model order selection challenge in AR modeling were addressed, FFT-based algorithm may be more suitable due to the broadband nature of the features to be extracted. Other approaches, not investigated in this study, include techniques based on multitaper methods that aim to reduce the variance of power estimates, or hybrid methods (e.g. the approach proposed by Kim et al. [36] combining state-space and multitaper methods) which may lead to more stable estimates of frequency features.

It is advantageous to use short steps for sliding windows but the performance degrades with using a step length shorter than 40 milliseconds. Consequently, a step length of at least 40 milliseconds is recommended.

With conventional DBS electrodes targeting the STN, it is better to identify a single bipolar channel based on the gamma ERS and then extract features from that channel as opposed to extracting features from all available bipolar channels for use in a force decoding algorithm. This is partly due to susceptibility to overfitting with increasing dimensions (curse of dimensionality [30]) and partly due to the small size of the dorsolateral part of STN relative to the size of DBS electrode contacts making it less likely that multiple bipolar channels will have force information. This is, however, likely to change with innovations in DBS electrodes leading to better decoding performance when using multiple channels. This could come from having longer electrodes allowing acquisition of signals from multiple structures (e.g. SNr, STN and ventral thalamus,) or directional electrodes to be more robust and specific.

The error in decoding force per unit force was constant for all force magnitudes except very low force levels (<0.1). This may be because patients dealt with a higher cognitive load at very low force levels leading to increased activity in the STN [32]. It is recommended that potential applications other than prosthetic control (e.g. communication) should exclude the intention to create very low forces due to the disproportionate distribution of decoding error at very low forces.

Lastly, there was a wide variation in the relative importance of different features for force decoding. The low gamma band (55-95 Hz) and high beta band (21-30 Hz) had the most predictive power on average. However, the accuracy of force decoding still improved when all features are used, as opposed to when using a small subset of features.

Our investigation has some limitations. All the results in this study are based on an isometric gripping task only, and thus it is not possible to draw conclusions on force generation using a different kind of action. Furthermore, the dataset used in this study was acquired from patients with Parkinson's disease, and not patients (e.g. paralyzed or locked-in patients) for whom force decoding from STN is likely to be useful. It is expected that the target patients will have a fully functioning STN similar to healthy people. Nevertheless, patients in this study were on levodopa which will have helped approximate the basal ganglia physiology of the target population. In this study, the gamma ERS was used as a measure of the accuracy of localization of DBS electrodes to the motor area in the STN. This is, however, an indirect measure of localization and to date, there has been no clinical validation of this measure for localization. It is possible that some of the patients in the study did not exhibit much gamma ERS due to their disease. All the patients in this study had surgery for DBS implantation and were subsequently externalized. While the recordings were undertaken 3-6 days after surgery, it is possible that there was some stun effect affecting the LFP. It is important to highlight that the target population for the potential application of force decoding would be patients with locked-in syndrome or ALS (for communication), or patients requiring prosthetic/robotic arms who are likely to have normal STN activity and not need deep brain stimulation per se. This was not the case in the present patient cohort as both the underlying Parkinsonian pathology and the temporary stun effect caused by recent surgery may have deranged STN activity and led to an underestimate of the decoding performance to be expected in the chronically implanted target BCI patients. Lastly, patients were requested to sit at rest and only apply gripping force when cued. It is yet to be ascertained how the performance would be affected when the patient is not at rest (e.g. walking or speaking). It is anticipated that this will largely depend on the spatial specificity of the DBS electrodes. Ideally, the contact pair used for decoding should be recording from the contra-lateral hand region in the STN and with the advent of directional/multi-contact DBS electrodes offering higher spatial specificity than conventional DBS, we expect that force decoding with directional DBS electrodes would be more robust to behavioral interference.

The results from this study are unlikely to be influenced by movement artifacts because the gripping tasks were isometric thereby minimizing any potential movement. Furthermore, any movement artifact is expected to manifest as increased power in the low-frequency band and in this work, the very low-frequency band (0-4 Hz) was excluded from the feature list for decoding force. In addition, the performance of the decoding algorithm improved with lag suggesting that using LFP recordings prior to movement contributes to force decoding and hence decoding is unlikely to be due to the presence of movement artifacts. This study demonstrated the ability to decode force on a continuous basis, using the whole signal, both during the gripping task and during rest periods. Since a sliding window was used, and the features were extracted from the raw time-series signal in each sliding window, unlike previous studies with pre-processing across the dataset, the proposed algorithm lends itself to real-time implementation.

V. Conclusion

We can decode force from the LFP recorded via DBS electrodes targeting STN in the basal ganglia in real-time (with a correlation of up to 0.79). A WC model combined with a sliding window using the FFT provides a practical solution for real-time force decoding from DBS electrodes. Future work will involve the development of a BCI pipeline to demonstrate force decoding on similar patients in real-time. The use of the basal ganglia LFP signal in BCI-controlled devices has the advantages of providing force information using a signal that is stable over many years and which can be relatively safely recorded, leveraging a surgical technique refined over decades for deep brain stimulation. This study thus provides an important advance in the development of BCI-controlled prosthetic devices by offering the possibility of decoding force from the brain in real-time. This can potentially be combined with real-time decoding from other modalities (e.g. EEG, ECoG) to provide a robust and more natural control of prosthetic devices than has been demonstrated to date, or exploited as a selection signal for communication aids in locked-in patients.

Table II.

Ranking of Features Based on Using LASSO for Feature Selection. The 14 Features Are Ranked for Each Patient and the Table Lists Them According to the Mean Rank Across the 8 STNs (the Higher a Feature Is in the Table, the More Important it Is).

Feature	Mean (± σ)	Median (25-75 percentiles)
56-95 Hz	5.8 (± 6.2)	3.5 (1.0 – 9.5)
21-30 Hz	5.8 (± 3.8)	4.5 (3.0 – 8.0)
350-500 Hz	5.9 (± 3.0)	6.0 (3.5 – 7.5)
301-349 Hz	7.1 (± 6.3)	5.5 (1.5 – 12.5)
5-7 Hz	7.1 (± 2.5)	6.5 (6.0 – 9.0)
Mobility	7.5 (± 6.0)	5.5 (3.0 – 12.0)
201-300 Hz	8.6 (± 6.2)	6.5 (4.0 – 14.0)
13-20 Hz	9.3 (± 6.1)	10.0 (3.0 – 14.5)
31-45 Hz	9.4 (± 5.3)	8.0 (4.5 – 15.0)
Mean LFP	9.5 (± 6.5)	12.0 (2.5 – 14.0)
Activity	9.6 (± 4.8)	8.5 (7.0 – 13.5)
Complexity	10.0 (± 3.6)	10.5 (7.0 – 12.5)
8-12 Hz	10.8 (± 4.6)	11.0 (7.5 – 15.0)
106-200 Hz	11.4 (± 6.0)	13.0 (7.0 – 16.0)

Ranking of features based on using LASSO for feature selection. The 14 features are ranked for each patient and the table lists them according to the mean rank across the 8 STNs.