Brain-machine interactive neuromodulation research tool with edge AI computing

Abstract

Closed-loop neuromodulation with intelligence methods has shown great potentials in providing novel neuro-technology for treating neurological and psychiatric diseases. Development of brain-machine interactive neuromodulation strategies could lead to breakthroughs in precision and personalized electronic medicine. The neuromodulation research tool integrating artificial intelligent computing and performing neural sensing and stimulation in real-time could accelerate the development of closed-loop neuromodulation strategies and translational research into clinical application. In this study, we developed a brain-machine interactive neuromodulation research tool (BMINT), which has capabilities of neurophysiological signals sensing, computing with mainstream machine learning algorithms and delivering electrical stimulation pulse by pulse in real-time. The BMINT research tool achieved system time delay under 3 ms, and computing capabilities in feasible computation cost, efficient deployment of machine learning algorithms and acceleration process. Intelligent computing framework embedded in the BMINT enable real-time closed-loop neuromodulation developed with mainstream AI ecosystem resources. The BMINT could provide timely contribution to accelerate the translational research of intelligent neuromodulation by integrating neural sensing, edge AI computing and stimulation with AI ecosystems.

1. Introduction

Neuromodulation enhance or restore neural function by directly intervening in the neural activity of the brain. Deep brain stimulation has been widely used for the treatment of various diseases, such as Parkinson's disease (PD), epilepsy, and depression [1]. Currently, neuromodulation techniques are evolving from open-loop to closed-loop strategies, with closed-loop strategies enabling on-demand modulation, thereby improving therapeutic effects while reducing side effects [2]. Development of brain-machine interactive neuromodulation strategies could lead to breakthroughs in precision and personalized electronic medicine [3]. However, closed-loop neuromodulation still faces serials crucial challenges in both neural decoding and encoding in real-time. In terms of decoding, machine learning algorithms with large set of parameters may be required to decode patient's various pathological and physiological states based on complex and dynamic neural activity [4,5]. For encoding, highly specific precise and timely interventions of neural activity can be used to restore neural functions damaged by different diseases [6,7]. Moreover, mobile systems are needed so that patients can be evaluated while freely moving over long periods.

Decoding the neural states based on local field potentials or other neural activities could be the basis for closed-loop neuromodulation strategies. In Parkinson's disease, the beta activity of subthalamic local field potentials was found to be related to the severity of symptoms and could be improved by medication and deep brain stimulation, and beta-adaptive closed-loop deep brain stimulation led to better therapeutic efficacy and reduced side effects [8,9]. However, more sensitive and specific biomarkers of diseases or symptoms must be explored [10]. Recent studies based on dynamic synchronized neural states suggest that understanding the complex dynamic relationships among the multiple oscillatory components in neural signals may lead to better decoding of the pathological states related to specific symptoms [11,12]. Moreover, machine learning algorithms have shown great potential in advancing closed-loop brain stimulation strategies to decode and predict neural or behavioral states [13,14], especially as long-term recordings of deep brain local field potentials become more feasible [15,16]. Machine learning algorithms such as support vector machines (SVMs), convolutional neural networks (CNNs) and recurrent neural networks (RNNs) have been applied to develop closed-loop deep brain stimulation treatments for Parkinson's disease [4,13,17], essential tremors [[18], [19], [20]], and major depression [21].

Neural encoding with deep brain stimulation can influence local or circuit neural activity through direct interventions at the target site. Existing closed-loop neuromodulation techniques focus on amplitude modulation or on/off switching of specific frequencies for stimulation. These methods have modulatory effects on neural activity, e.g., 130 Hz stimulation suppresses beta activity in PD patients [22]. Moreover, modulation of brain states could be used for precise interventions [12,23,24]. Previous studies have shown that precisely timed stimulation can be used to encode neural activities, such as amplification or suppression [[25], [26], [27]]. More notably, the modulation of these neural activities leads to the expected behavioral changes [28]. These findings suggest that precise modulation of critical neural oscillations associated with disease may lead to improved therapeutic outcomes. More complex patterns encoded in the stimulation could be generated to modulate the brain states related to different diseases by modeling the neural activities and the biomarkers with real-time machine learning methods [13]. Therefore, it is highly demanded to develop new research tools for online neural signal recording, data streaming, signal processing and modeling in real time.

We developed a Brain-Machine Interactive Neuromodulation Research Tool (BMINT) to address these challenges by sensing neural activities, processing and modelling with machine learning algorithms and delivering electrical stimulation or controlling command in real-time. The brain-machine interactive neuromodulation involves bidirectional information transfer between the brain and the tool, and allows for neural activities sensing from the brain and delivery of information to the brain through electrical stimulation. We employed edge artificial intelligence (AI) computing with a heterogeneous computing (CPU + GPU) architecture to realize sufficient computing capability for real-time signal recording, processing and intelligent computing. In addition, we designed the intelligent computing framework to integrate mainstream machine learning frameworks and efficiently deploy offline trained models with the research tool.

2. Methods

2.1. System architecture

2.1.1. Hardware modules

The dimensions of BMINT are 24 × 12 × 10 cm. It is composed of three main hardware modules, including the recording, computing and stimulation modules (Fig. 1a). The tool is battery powered and the direct current to direct current converter (DC-DC) utilizes isolated power modules. The recording module includes 8 channels for recording neurophysiological signals with a 24-bit amplitude resolution and a 2000 Hz sampling frequency [29]. The recording module is equipped with build-in pre-amplifiers, enabling direct acquisition of neurophysiological signals from electrodes. The data are saved and streamed to the computing module through a serial peripheral interface (SPI). The computing module is a development kit equipped with a Jetson Nano (NVIDIA, USA) that can perform edge AI computing due to its compact form factor, low power consumption, and accelerated GPU functions (921 MHz, 472 GFLOPS) and CPU (1.43 GHz). The stimulation module delivers 2-channel constant current electrical stimulation, with pulse parameters such as the amplitude, frequency and pulse width adjusted in real time, and is controlled by a high-performance microprocessor (STM32H7, STMicroelectronics) with a 400 MHz dominant frequency. The stimulation module offers a broad range of constant current outputs, with a maximum current of up to ±10 mA, and the output is ±14 V compliant. For deep brain stimulation, the stimulation voltage and current were constrained to 5 V and 5 milli-amps at maximum, according to the safety curve area [30]. Biphasic pulses can be configured with a minimum pulse width of 15 μs. The 16-bit digital-to-analog converter (DAC) provides a minimum constant current output step of 0.15 μA. The stimulation module offers stable high-frequency electrical stimulation output, with change rate of biphasic pulses after 30 min of continuous stimulation being 0.012 % and 0.011 %, respectively (Section 1, Supplementary Material). The BMINT research tool also has various I/O ports to deliver control command to interface with other neuromodulation devices, i.e., transcranial magnetic stimulation (TMS) and ultrasonic stimulation.

Fig. 1.

The functional illustration of the BMINT. (a) The research tool consists of three modules: recording, computing, and stimulation. The recording and stimulation modules are connected to the computing module through SPI and UART, respectively. Machine learning algorithms are integrated into the research tool through the intelligent computing framework. (b) The intelligent computing framework integrates the AI ecosystem and supports the data streaming of hardware. The data from the recording module are sent to a real-time processing pipeline. The data are passed through processing algorithms, including preprocessing, neural decoding and stimulation encoding, to generate stimulation signals with specific parameters. The stimuli control commands are sent to the stimulation module and delivered to the brain; AI, artificial intelligence; SVM: support vector machines; CNN: convolutional neural networks; RNN: recurrent neural networks; PC, personal computer; HPC, High-Performance Computing; CCP, Cloud Computing Platform; CPU, central processing unit; GPU, graphics processing unit; LFP, local field potential; ECoG, electrocorticography; EEG, electroencephalography.

2.1.2. Intelligent computing framework

The intelligent computing framework, compatible with multiple programming languages (such as C/C++ and Python), is a software framework enabling real-time processing and integration with machine learning algorithms for closed-loop neuromodulation. As shown in Fig. 1b, the framework comprises three main units: a system management unit (programmed in C), a data processing unit (programmed in C++), and an AI-specific APIs unit (programmed in C++/Python).

The data processing unit is the key element in the research tool. The data processing unit performs signal preprocessing, neural decoding, and stimulation encoding algorithms in real time. This unit integrates the functions of AI ecosystems via AI-specific APIs, data streaming and saving, and system state monitoring. The unit can interface with the data acquisition and stimulation output. The real-time data processing, transmission, and storage capabilities are enhanced through the implementation of advanced system management techniques such as CPU isolation, memory sharing, and multi-threading processing, etc. Specifically, AI computation acceleration is performed with GPU to enhance the real-time processing of machine learning algorithms.

The AI ecosystem can be delineated into two components. The first component encompasses public pretrained AI model libraries available pre-trained AI model (e.g., NVIDIA NGC, TorchVision, and TensorFlow Hub) and custom trained AI model from cloud or external high-performance devices. These models undergo processing through the second component of the AI ecosystem, comprised of various optimization and acceleration tools (e.g., CUDA, cuDNN and TensorRT) integrated into research tool. Subsequently, the AI models are deployed into online closed-loop neuromodulation via the AI-specific APIs.

The SPI and UART drivers were developed to control the recording and stimulation hardware modules for real-time data acquisition and stimulation output. Additional functions could be realized using external devices that connect and communicate via supported input/output interfaces.

2.2. Evaluation of the real-time performance

We designed two experiments to evaluate the real-time performance of the BMNIT: the evaluation of the system time delay based on pulse detection and a closed-loop neuromodulation simulation based on gamma oscillation tracking.

2.2.1. Single pulse detection

A pulse with a width of 1 ms was generated using a signal generator 81160A (Keysight, USA). The signal was simultaneously recorded by an oscilloscope and the recording module. When the tool detected the pulse, it sent a command to the stimulation module to generate a stimulus and a synchronous marker signal through a General Purpose Input/Output (GPIO) pin. The system time delay, D, was calculated as the difference in time between the onset of the pulse signal and the stimulation recorded by the oscilloscope. The system time delay does not include specific algorithm processing time. Here, the pulse detection involves only the simplest numerical comparisons. The time cost for this calculation is virtually negligible on the 1.4 GHz CPU of the tool. Additionally, we further divide D into two parts: D1 represents the time delay between signal recording and stimulation command onset, while D2 represents the time delay between stimulation command onset and actual stimulus output.

2.2.2. Gamma oscillation tracking

The system real-time performance was further evaluated by tracking simulated gamma oscillations at 90 Hz. A sine wave with a fundamental frequency of 90 Hz and amplitude modulation of 1 Hz was generated to simulate gamma oscillations. We used the research tool to track the transient amplitude of gamma oscillations and generate a stimulus with the same peak amplitude when a peak was detected. The system time delay was then calculated as the time difference between the peak of the sine wave and the onset of the generated stimulus. The correlation analysis was carried out between the envelope of the gamma oscillation and the amplitude of stimulus to assess the real-time performance of the gamma oscillation tracking.

2.3. Evaluation of the computing capability

The computing capability of the tool was evaluated with Deep Residual Network (ResNet) of convolutional neural network (CNN), which is widely used for classification, target detection and semantic segmentation [31]. ResNet has rich network structures and parameters to evaluate the computing capability at different model complexities. In this study, we chose the ResNet18, ResNet50, and ResNet101 models, which have 18, 50 and 101 layers in the neural network, and 11.7, 25.6 and 44.5 million parameters, respectively. The models were developed from TorchVision (version 0.15).

Moreover, three levels of data sizes of 256 × 256, 512 × 512, and 1024 × 1024 were applied to evaluate the computing capability under combinations of model complexity and data size. The samples in each data size were randomly selected from a real EEG dataset (section 2.4.1). The models with different datasets were run on the GPU using TensorRT to accelerate the computation of the deep learning models. The models were also run on the CPU to evaluate the computing capability in a typical situation. The evaluation was repeated 100 times under each condition, and the average performance was obtained.

2.4. Closed-loop neuromodulation demonstration

To demonstrate the capability of the BMINT to deploy online neuromodulation models with machine learning algorithms, we applied three representative machine learning methods for seizure detection with the research tool. Additionally, the closed-loop neuromodulation capability was tested in simulated scenarios by triggering stimulation with online detection of seizures from single-channel EEG signals of patients with epilepsy.

2.4.1. Dataset

In this study, CHB-MIT Scalp EEG Database was used [32]. The EEG were recorded from 22 patients with epilepsy. For each patient, the EEG signals included 23 channels sampled at 256 Hz. The channels and periods with seizures were labeled. In this study, partial datasets of 10 patients (cases 1–10) were used to construct the detection model. A single-channel selection scheme for each patient was used according to a previous study [33]. Each channel in the signal was truncated to 1 s segments with labels. The data from each patient were used to construct the training and test sets at a ratio of 7:3. Detailed information for each patient is provided in Supplementary Table 5.

2.4.2. Algorithms

Three commonly used machine learning algorithms of SVM, CNN and RNN were developed and deployed into the research tool for seizure detection. The area under the receiver operating characteristic curve (AUC), accuracy, sensitivity and specificity were used to evaluate the performance of the detection. When using the SVM model, four features in the time domain (variance, energy, nonlinear energy, and Shannon entropy) were selected as the input features according to previous study [34].

While training the CNN model, each data segment was transformed into a 44 × 44 time-frequency spectrogram covering the frequency range of 1–45 Hz using the short-time Fourier transform (STFT). The time-frequency spectrographs were fed into a LeNet-5 network [35] for training. For the RNN model, the 1-s segments were fed directly into a long-short time memory (LSTM) network for training.

All three models were trained on a high-performance computing platform (NVIDIA 1080Ti × 2) using 5-fold cross validation. The SVM model was trained using Scikit-learn [36], and the CNN and RNN models were trained using PyTorch [37]. The SVM models were saved directly in Joblib format, as recommended by Scikit-learn. The CNN and RNN models were saved in Open Neural Network Exchange (ONNX) format. All of these files can be directly deployed for use in the proposed intelligent computing architecture. With the assistance of these frameworks and the AI-specific APIs in the research tool, the training and initial deployment of each model can be realized with a few lines of Python codes. Each model was deployed by simply replacing the model file and modifying the input and output. The code for training and deployment can be found at the address provided in the Code availability section.

2.4.3. Online simulation

An online simulation environment was set up to test the performance of the closed-loop neuromodulation. EEG signals were generated real-time by loading the EEG signals into an arbitrary signal generator 81160A (Keysight, USA). The output of the signal generator was rescaled to millivolt with resistors and connected to the input of the research tool and an oscilloscope. The output stimulation of the research tool was connected to the oscilloscope as well. Each 1-s data segment of single-channel EEG was recorded online and subjected to real-time classification using the CNN model, which determined whether the segment corresponded to a seizure event or not. The simulation was turned on or off in real-time according to the state of seizures. We employed two recording channels of the tool, one for recording of simulated single-channel EEG signal, and the other for recording the electrical stimulation generated by the tool. This allowed the tool to make synchronous recordings of the closed-loop neuromodulation process, which were subsequently utilized for analysis.

3. Results

3.1. Real-time performance of the tool

3.1.1. Single pulse detection

The system time delay D from the onset of the input pulse and output stimulus was 2.829 ± 0.057 ms (mean ± standard deviation), and mostly distributed between 2.8 and 3.0 ms with occasional occurrence around 2.6 ms among 100 repeated measures (Fig. 2b). The variation of the system time delay was less 0.5 ms. The time delay to deliver control command D1 was 2.216 ± 0.063 ms and the time delay to generate stimulation D2 was 0.613 ± 0.051 ms after receiving control command. D1 distributed between 2.0 and 2.4 ms, and D2 distributed between 0.45 and 0.75 ms. These measures demonstrated that the BMINT research tool was able to maintain a stable real-time performance at system level.

Fig. 2.

Real-time performance of the BMINT. (a) The system time delay was calculated between the onsets of input pulse and output stimulus. The time delay to generate command (D1, green) was calculated between the onsets of the input pulse and command marker, and time delay to generate stimulation (D2, yellow) was calculated between the onsets of the command marker and the output stimulus. System time delay(b) Distribution of the time delays D, D1 and D2 (grey) and system time delay estimated probability density distribution (red). (c) Amplitude-modulated gamma oscillations and synchronous stimulus (top). Demonstration of gamma oscillations and corresponding stimulus generated when peaks of the gamma oscillations were detected (bottom). (d) The envelope of gamma oscillations and the amplitude of stimulation (top) and their correlation (bottom). The largest correlation was close to 1.0 at 2.86 ms.

3.1.2. Gamma oscillation tracking

The system time delay was further evaluated with the continuous online gamma oscillation tracking experiment (Fig. 2c). The time delay between the peaks of the gamma oscillation and the output stimulus were 2.831 ± 0.054 ms. The correlation between the envelope of the gamma oscillation and the amplitude of stimulus was close to 1.0 with time delay at 2.86 ms.

3.2. Evaluation of the computing capability

The computing capability of the BMINT research tool was evaluated with different combinations of data size and models using GPU or CPU (100 trials for each combination) (Table 1). Increasing the data size and model complexity significantly increases the elapsed time to perform computation of each iteration using either GPU or CPU. In the case of the same data size, when using the CPU for computations, increasing the number of model parameters leads to a nearly proportional increase in cost. For example, with a data size of 512 × 512, the time costs of the three models are 1288, 2987, and 5504 ms. In contrast, when using the GPU, as the number of model parameters increases, the increase in the computation time becomes less significant, which are 76, 209 and 361 ms. For the 1024 × 1024 data size, the time costs of the three models are 273, 816, and 1434 ms using GPU. This indicates that the parallel computing capability of the GPU provides noticeable computational advantages as the data size and number of model parameters increase.

Table 1.

Computing cost under different combinations (ms).

model	processor	data size
		256 × 256	512 × 512	1024 × 1024
ResNet18	CPU	318 24	1288 76	5087 273
	GPU
ResNet50	CPU	736 67	2987 209	11950 816
	GPU
ResNet101	CPU	1407 106	5504 361	22484 1434
	GPU

When the same model was used, regardless of whether the CPU or GPU was used for computations, the increase in the cost is essentially consistent with the increase in data size. For instance, for ResNet18, the total data volume between any two data sizes differs by a factor of 4, resulting in an approximately fourfold increase in the time cost of the model (CPU: 318, 1288, 5087 ms; GPU: 24, 76, 273 ms). The acceleration result for each combination was calculated by using the ratio of CPU to GPU time cost. The mean GPU acceleration capability of the BMINT increase the computation efficiency by about 14.77 times comparing to CPU.

3.3. Implementation of closed-loop neuromodulation

3.3.1. Performance of the machine learning algorithms

The seizure detection from single channel EEG recordings were tested to illustrate the performance of using three machine learning algorithms with the BMINT. The AUC, accuracy, sensitivity and specificity of seizure detection were considered as the evaluation indicators. The results show that the CNN achieved the best performance (AUC: 97.59 % ± 3.5 %, accuracy: 97.02 % ± 2.59 %, sensitivity: 85.83 % ± 17.55 %, and specificity: 97.04 % ± 2.58 %). The RNN showed a slightly inferior performance (AUC: 95.60 % ± 3.87 % accuracy: 95.37 % ± 4.18 %, sensitivity: 84.55 % ± 12.17 %, and specificity: 95.39 % ± 4.16 %), and the SVM showed the worst performance (AUC: 92.71 % ± 8.08 %, accuracy: 90.53 % ± 7.09 %, sensitivity: 85.48 % ± 12.05 %, and specificity: 90.54 % ± 7.08 %) (Fig. 3a). For each patient's detail, refer to Tables 6–8 of Supplementary Material. In terms of individual patients, none of the methods completely outperformed the others in every metric (Tables 9–12, Supplementary Material). For example, the RNN model for patient 6 outperformed the other two methods in terms of sensitivity by at least 7 % but obtained the lowest sensitivity for patient 4 (61.34 %), and for patient 5, SVM obtained the best accuracy (95.8 %). These results emphasize the importance of exploring personalized treatment approaches for individuals. The average ROC curves of 5-fold cross-validation using different methods for each patient were shown in Fig. 1 of the Supplementary Material.

Fig. 3.

Implementation of online closed-loop neuromodulation for seizure adaptive stimulation in epilepsy. (a) The performance of seizure detection was evaluated with accuracy, sensitivity and specificity using SVM, CNN and RNN methods. (b) The implementation of the closed-loop neuromodulation responding to the seizure state detected with CNN model. The EEG of patient CHB10 was delivered to the BMINT research tool (upper), and the seizure state was detected using CNN model (middle) and the stimulation was generated according to the seizure state in real-time (bottom).

3.3.2. Evaluation of closed-loop neuromodulation

The closed-loop neuromodulation was evaluated with adaptive electrical stimulation responding to the real-time detection of seizures using CNN model from single-channel EEG. Fig. 3b showed 500 s simulated real-time closed-loop neuromodulation process. The state of seizures was determined according to the classification output of CNN model of each 1-s segment from signal-channel EEG in real time. If the state of seizure was detected, 130 Hz electrical stimulation was generated. The results showed that the algorithm obtained over 99 % accuracy in online detection and delivered instantaneous adaptive electrical stimulation. In addition to the closed-loop neuromodulation capability, the BMINT research tool recorded and saved intermittent variables of the signal processing and modelling processes accordingly.

4. Discussion

4.1. Real-time performance

The real-time performance is essential to the real-life experimental application of the adaptive or closed-loop neuromodulation, and the key factors influencing the real-time performance is the system time delay and computation capability. Regarding system time delay, a short and stable delay is of significant engineering and scientific value in many applications of neuromodulation. We illustrate two applications in this context. Firstly, it enables precise handling of stimulation artifacts. When the system time delay is accurately controlled, such as D2 delay mentioned in the paper, it represents the time from software sending a stimulation command to the actual output of the stimulation. This allows real-time artifact removal using various methods, such as template or blanking [[38], [39], [40]], applicable not only to regular stimulation (e.g., 130 Hz) but also to irregular stimulation. The latter has shown great potential in recent study of certain medical conditions [41,42]. Secondly, current methods requiring precise timing control, such as phase-dependent approaches, demonstrate significant potential for disease treatment [26,28,43]. Considering the cycles of important beta or gamma neural oscillations related to brain diseases [44,45], i.e., about 50 ms or 15 ms, the ideal system time delay should be stable and below these cycles so that cycle by cycle phase modulation could be possible[26,28,43,46]. A short and stable system time delay is essential to deliver stimulation at the specified phase of neural oscillations, potentially providing more effective strategies for neuromodulation. We further compared the performance of system time delay of our tool with others [[47], [48], [49], [50], [51]] in Table 12 of Supplementary Material, and our tool achieved the lowest system time delay (<3 ms). The lower system time delay provides further timing allowance for implementing algorithms.

In assessing computational capability, we demonstrate the time costs associated with various data sizes when using specific CNN models on our research tool. Various data sizes allow different lengths of data and features as inputs, such as the time-frequency representation of local field potentials over hundreds of milliseconds to seconds. It is worth noting that our research tool utilizes GPU acceleration technology, which achieved a speedup of approximately 14 times in this study. The 256 × 256 data size could correspond to 256 channels of 1-s EEG data sampled at 256 Hz, and the computation cost is less 400 ms in the largest model and only 76 ms in the small-scale model. When implementing closed-loop neuromodulation adaptive to seizure states using single-channel EEG data with a size of 44 × 44, the computation time of the LeNet-5 was approximately 8 ms. This computational capability, accommodating classification, prediction, and control scenarios with time delays ranging from tens of milliseconds to several hundred milliseconds, highlights the real-time performance of our tool.

4.2. Closed-loop neuromodulation with machine learning

There is significant trend that more and more machine learning algorithms are used in developing neuromodulation strategies [45,52,53]. Furthermore, the BMINT is compatible with mainstream AI ecosystems, including PyTorch, TensorFlow, and other AI frameworks, as well as various pre-trained model libraries and model optimization tools. This design enables researchers to efficiently apply state-of-the-art machine learning methods in closed-loop neuromodulation scenarios, machine learning models, such as transformers [53], reinforcement learning (Lu et al. [54]) and the machine learning methods used in this paper, rather than limiting machine learning models to offline performance evaluation stages.

Recent research has suggested that personalized neuromodulation approaches with machine learning may lead to better treatment outcomes, not only for individual patients [21] but also for specific targets of brain regions in each patient [55]. When conducting personalized closed-loop neuromodulation research, the efficient deployment capability provided by BMINT is particularly crucial. This significantly streamlines the steps required for online application of machine learning, particularly when verifying algorithms for multiple patients, as show in the case of closed-loop neuromodulation for seizure detection demonstrated in this paper. Utilizing the AI-specific APIs provided by the tool, deploying a new machine learning algorithm requires just a few lines of code (see Code availability). In instances where different patients employ the same machine learning method, only personalized model replacements are necessary.

In the simulated online demonstration, the sensitivity of the model was 96.16 % (patient 10, Fig. 3a). In the 500s online process, there are 6 false positive detections and the false positive rate was about 1.42 % (6 samples/423 samples). In epilepsy treatment with deep brain stimulation, the algorithms are generally tuned to achieve better sensitivity so that the seizure could be suppressed successfully. Optimal performance with both high sensitivity and specificity is highly desired to avoid unnecessary stimulation and deliver effective stimulation to suppress seizure.

4.3. Expanded applications of the tool

The BMINT has achieved the key functions of real-time processing, and implementation of signal processing and artificial intelligent algorithms, and it has been demonstrated the potential application for brain stimulation. The tool is possibly able to deliver control commands to interface with other neuromodulation devices via serial port control, and we achieved to deliver single pulse TMS stimulation with specific timing and intensity (MagPro X100, Magventure, Denmark). It could be used for other types of neuromodulation by incorporating with stimulation isolators, for instance, spinal cord stimulation or peripheral neuro-stimulation.

As for recording, the tool provides the capability to record neurophysiological signals from tens of micro-volts to hundreds of milli-volts with sampling rate till 2000 Hz. The recording module offers a voltage acquisition resolution of 0.54 μV. Such performance allows to record neurophysiological signals of EEG, ECoG and EMG, etc.

Anticipating future upgrade demands, the tool employs standard hardware interfaces (SPI, UART) and tailored software design (intelligent computing framework). This enables us to harness the continuously evolving Jetson series edge AI computing chips. For instance, the latest Jetson Orin offers 60 times higher computing performance than the Jetson Nano. This ensures the tool maintains advanced in terms of computational capabilities.

4.4. Limitation

The limitations of the study include the restricted sampling frequency, which imposes constraints on high frequency data acquisition. Additionally, the real-time performance of each strategy must be individually optimized to ensure efficient real-time operation. Furthermore, the stimulation output is capped at 5 V due to safety concerns associated with deep brain stimulation. For other forms of neuromodulation, such as spinal cord stimulation or peripheral neuro-stimulation, stimulator isolators are necessary to ensure safety and efficacy. Lastly, the study is constrained by a limited number of recording channels, which may impact the comprehensiveness of data collection and analysis. The tool will be tested and comply with the safety regulations of CE, CFDA or FDA when it is used for human in the future.

5. Conclusion

We developed a research tool for intelligent closed-loop neuromodulation. The BMINT research tool has capabilities of neural sensing of local field potentials, ECoG, EEG or EMGs, computing with mainstream machine learning algorithms and delivering electrical stimulation pulse by pulse in real-time. The research tool achieved system time delay less than 3 ms, and computing capabilities in feasible computation cost, efficient deployment of machine learning algorithms and acceleration process. Intelligent computing frameworks embedded in the BMINT research tool enable real-time closed-loop neuromodulation developed with mainstream AI ecosystem resources.

Data statement

The datasets of this study are available from the corresponding author on reasonable request.

Code availability

The code of machine learning methods can be found at: https://github.com/gaosiy/research-tool-seizure-detection.

Funding

For this work was provided by National Key Research and Development Program of China (No. 2022YFC2405100); STI 2030—Major Projects (No. 2021ZD0200407); STI 2030—Major Projects (No. 2022ZD0205300); National Key Research and Development Program of China (No.2021YFF1200600); Shanghai Municipal Science and Technology Major Project (No.2021SHZDZX) Shanghai Municipal Science and Technology Major Project (No.2018SHZDZX01), ZJ Lab, and Shanghai Center for Brain Science and Brain-Inspired Technology; Shanghai Municipal Science and Technology Major Project (No.2021SHZDZX0103); the 111 Project (No. B18015); National Natural Science Foundation of China (No. 82201400); China Postdoctoral Science Foundation (No. 2022TQ0071).

CRediT authorship contribution statement

Yan Li: Conceptualization, Formal analysis, Methodology, Software, Visualization, Writing – original draft, Writing – review & editing. Yingnan Nie: Methodology, Software. Zhaoyu Quan: Resources. Han Zhang: Software. Rui Song: Resources. Hao Feng: Software. Xi Cheng: Software. Wei Liu: Resources. Xinyi Geng: Writing – original draft. Xinwei Sun: Methodology. Yanwei Fu: Methodology. Shouyan Wang: Conceptualization, Methodology, Project administration, Supervision, Writing – review & editing.

Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Appendix A. Supplementary data

The following is the Supplementary data to this article: