The Neural Engine: A Reprogrammable Low Power Platform for Closed-loop Optogenetics

Abstract

—Brain-machine Interfaces (BMI) hold great potential for treating neurological disorders such as epilepsy. Technological progress is allowing for a shift from open-loop, pacemaker-class, intervention towards fully closed-loop neural control systems. Low power programmable processing systems are therefore required which can operate within the thermal window of 2o C for medical implants and maintain long battery life. In this work, we have developed a low power neural engine with an optimized set of algorithms which can operate under a power cycling domain. We have integrated our system with a custom-designed brain implant chip and demonstrated the operational applicability to the closed-loop modulating neural activities in in-vitro and in-vivo brain tissues: the local field potentials can be modulated at required central frequency ranges. Also, both a freely-moving non-human primate (24-hour) and a rodent (1-hour) in-vivo experiments were performed to show system reliable recording performance. The overall system consumes only 2.93mA during operation with a biological recording frequency 50Hz sampling rate (the lifespan is approximately 56 hours). A library of algorithms has been implemented in terms of detection, suppression and optical intervention to allow for exploratory applications in different neurological disorders. Thermal experiments demonstrated that operation creates minimal heating as well as battery performance exceeding 24 hours on a freely moving rodent. Therefore, this technology shows great capabilities for both neuroscience in-vitro/in-vivo applications and medical implantable processing units.

I. Introduction

THE very first implantable pacemaker was introduced by Senning and Elmqvist in 1958 [1] for use in cardiac synchronization. Later in 1963 Bekthereva [2] demonstrated the first chronic implant in the human brain, though the field of Deep Brain Stimulation (DBS) is largely attributed to the work of both the Benabid and Blond, and Sigfried groups in 1991 [1]. The architecture of early systems were largely analogue with simple open-loop oscillatory functions. Since then, modern commercial medical systems have developed digital control units with data logging and wireless communication capabilities. Nevertheless, such systems still largely implement open-loop oscillatory stimuli.

More recently, the development of the Reactive Neural Stimulator [3] demonstrates a direction of travel towards closed-loop systems which can provide targeted intervention based on specific neural activity. With the advent of human trials of optogenetics, initially in the retina [4][5], we could soon see the development of interference-free bimodal control with electrical recording and optical stimuli.

To achieve this, there needs to be a platform which can implement closed-loop stimuli within both the constraints of battery operation and thermal output. For the latter, the surface of the implant should not exceed 2o C to stay within the regulatory limits [6]. For the former, the largest rechargeable medical-grade batteries at the time of writing are between 300-350 mAhr (e.g. EaglePitcher Contego 325). As such, the target current draw needs to be an average of 7 mA or better (less) with a recharge cycle of 2 days or better. Furthermore, a similar figure can be attained if we assume batteries need to last at least five years over a recommended 1000 recharge cycles.

In order to reach human trials, proposed clinical architectures need to be verified in in-vitro/in-vivo neuroscience experiments. In particular, testing is required in freely moving non-human primates (NHP) and/or rodent models. Such experiments require a system with remarkably similar attributes to a clinical device. The physical size of a neural co-processor(embedded control system) needs to be 2 cm x 2 cm or better to mount in a head cradle (NHP) or backpack (rodent). The space available for the developed neural co-processor (embedded control system), in a medical control unit (chest unit) after the battery is similar. The battery pack is also perhaps surprisingly similar. The maximum capacity of clinical grade rechargeable batteries is similar to that of two CR2025 class watch batteries, which have a feasible weight (5.2 g) for both NHP and rat models. To achieve 48-hour recording at this capacity (minimum clinical target), a current consumption of around 7-8 mA or better is required.

other academic groups and industry have also been developing implantable control systems to meet these specifications. The technologies to achieve this can be classified into three groups: (i) Custom Application-specific Integrated Circuits (ASICs) with inbuilt digital processing (ii) Reconfigurable logic modules such as Field-programmable gate arrays (FPGAs) or Complex Programmable Logic Units. (iii) General purpose microcontroller (MCU) systems.

These technologies are complementary rather than mutually exclusive [7]. Dedicated ASIC implementations can be low power provided because they are implemented on a sufficiently advanced technology node. For example, Liu et al. [8], presented a bidirectional Brain-Machine Interface (BMI) ASIC for closed-loop neuroscience research which could perform neural feature extraction. They demonstrated operation with a total power requirement of 8 mW (2.16 mA). Similarly, Kassiri et al. [9] introduced an inductively-powered seizure-predicting ASIC based microsystem for monitoring and treatment of intractable epilepsy which operated at a total power of 2.78 mW (1.52 mA). However, these DSP functions cannot be reconfigured which means they are primarily useful when an intervention algorithm has been identified, and only operational variables need to be changed. For neuroscience research or immature clinical interventions, such a lack of flexibility is undesirable.

An alternative to having fixed algorithms in silicon is to configure multiple independent filters which can be called in sequence or utilize reconfigurable digital logic. Gagnon-Turcotte et al. [10] utilized option (ii), and developed a wireless headset for high resolution 32-channel electrical recording in tandem with optical stimulations. Their system utilized a low-power FPGA based DSP for a digital spike detector and a wavelet data compression module. Such systems can be efficient for high power parallel processing but need special tuning for use in low power systems due to high leakage power. At a spike rate of 45 s-1 conditions, the whole system consumes 119mW (32mA) when all channels are active. To provide like for like comparison, processing-only aspect of the FPGA consumed 47.2 mW (12.75 mA).

The third option is to utilize a general purpose microcontroller unit. These are very flexible with significant peripheral functionalities (e.g. SPI, ADC/DAC, timer). Such units form computation functions in software, thus taking multiple clock cycles. This allows for more complex computation such as recursive functionality and multitasking with peripherals, but at the cost of computational efficiency. However, typically such microcontrollers are available in advanced deep sub-micron technologies and have significant power cycling architectures. S. Zanos et al. [11] previously designed an MCU based system: The Neurochip-2 for freely-moving NHP experiments which could perform closed-loop electrical stimulation and recording. Their system utilized commercial PSoC (Prototyping System-on-Chip) development boards and therefore required a minimum power of 284 mW (78.8mA).

Another example is the Cortex M4 which we adopted in this work. This platform consumes a nominal 14 mA current at clock frequency 40 MHz under full CPU load. However, with advanced power management, it is possible to define power modes at each stage in the computational cycle so that full power is only utilized in short bursts. other operations can be performed operating in various levels from deep sleep to fully awake.

In this work, we have created a low-power flexible neural engine specifically for computing optogenetic closed-loop interfaces as shown in Fig. 1: It has one custom designed embedded Hardware/Software(H/W) processing system, and integrates with an ASIC based circuits for electrical recording and optical stimulations. Also, we have also incorporated communication protocols to make it compatible with both commercial data acquisition systems such as Intan as well as our previously published ASIC [12]. The remainder of this paper is structured as follows: Section II describes the hardware configuration; Section III and Section IV introduce the software architecture and BMI processing toolbox. The system processing has four stages: recording interrupt, closed-loop algorithm, optical converter and data storage. Sections V VI present the neuroscience in-vitro/in-vivo experimental configurations and results as well as power consumptions/thermal analysis. For power consumption comparisons with past systems, it can operate at an average of 2.93 mA under an optimised power cycling scheme. Finally, discussions and conclusions are included in Sections VII and VIII: the system has an architecture which could also progress towards clinical use, and great potentials towards animal behavioural control.

Fig. 1. A conceptual description of the developed neural co-processor.

It is aimed for a closed-loop processing with electrical recording and optical stimulation. The processing has four stages: recording interrupt, closed-loop algorithm, optical converter and data storage.

II. Hardware overview

The hardware architecture is shown in Fig. 2. This consists of two parts: one is an embedded microcontroller based neural co-processor, and the previously designed ASIC-based neural interface [12] integrated within a headstage board.

Fig. 2. The developed system hardware architecture: (on left) embedded control system; (on right) ASIC-based neural interface head-stage board.

The control system contains an ARM Cortex M4 device, a SD card and power management system. The red line indicates power system transmission and the black line represents data and commands transmission.

The embedded processor is an ARM Cortex-M4 microcontroller (MCU) with DSP function block. Specifically, we used an MK22FN512VLH12 MCU - 120 MHz with 512 KB flash memory, 128 KB RAM in a 64 pin LQFP package. This is employed to implement the closed-loop algorithms as well as communication with a head-stage chip which specifically performed the signal acquisition and driving. The MCU was implemented on a 25 x 22 mm printed circuit board (PCB) as shown in Fig. 3(a). The PCB board also contains a (TLv70233DBvR) low dropout (LDo) regulator to provide a consistent 3.3.V supply up to 330 mA from a 3.7 V lithium-ion portable battery. It also contains a (LTC3525ESC6-5#TRMPBF) synchronous boost converter which can provide a consistent 5 v supply boosted from the regulator. This latter supply is required for providing power sources to the optical stimulation circuits in the ASIC based headstage as defined in our prior work[13][14]. In addition, there is an indicator LED (CLV1A-FKB-CJ1M1F1BB7R4S3 full-colour LED) and a micro-SD card slot for long term data recording (the micro-SD can address up to 64 Gb). For external testing and MCU programming, a Multilink FX programmer with a JTAG connector was used. Additionally, there is a UART port for connecting with a computer for real-time debug. Finally, a 10-pin flexible PCB cable (FPC) connector has been used to provide a connecting link between the embedded controller and the head stage. This is adaptable to commercial systems, but in this case for use with our previously described neural interface chip[12].

Fig. 3. The developed system hardware architecture.

(a) shows system neural co-processing; (b) depicts an ASIC based head-stage configuration; (c) illustrates side views of the system, and (d) illustrates system mechanical configurations on a human head model.

The rodent head-stage board is shown in Fig. 3(b). The head stage has dimensions of 10 x 10 mm, with 8 pins for stimulation and recording. It should be noted that we originally tried a backback configuration. But if the weight is sufficiently low, a head mount is superior as the rodent cannot scratch it off.

Fig. 3 (c) shows the NHP head mount configuration. In this case, the system is mounted within a crown unit which is attached to the NHP head. The crown protects the electronics from the various vigorous activities of the NHP.

III. Software Architecture

The state machine operation of the software is shown in Fig. 4. The basic computing mechanism is as follows: after the chip initialization, a system is configured into a sleep mode, and a recording interrupt is reset. This indicates that most of the system peripherals are disabled except for a low power mode timer (LPT). The system is constantly at sleep mode until the LPT counter equals the biological recording periods. Then a recording interrupt is triggered to wake up the system into a normal run mode. The system will generate a recording request to an ASIC based head-stage via a serial peripheral interface (SPI). After a short recording delay (e.g. 45 μs) a 16-bit data will be received, that contains LFPs. Therefore, a recording is requested every time the microcontroller is interrupted. The interrupt occurs periodically, and this period equals the neural recording sampling frequency (e.g. 500 Hz). So effectively the neural recording is continuous with discrete time steps. A detection algorithm is employed to decide whether recorded LFPs belong to abnormal activities or not. A suppressor will be performed when abnormal activities are detected. The outputs of a suppressor will be translated into stimulation commands by using an optical converter. Then the head-stage will control light intensity based on received stimulation commands. In addition, the received data will be directly stored into an SD card or to a PC via UART (for debugging purposes). After that, the system is configured into sleep mode again to wait for the next interrupt. The SPI communication details are described in previous work [12]. Particularly, we designed a BMI based toolbox for three-stage processing: detection, suppression and optical simulation results are displayed in the Supplement S1 conversion. The details will be described in section IV. And the timing diagram of the power cycling technique is shown at the bottom of Fig. 4: a red dash line indicates a recording interrupt, sleep modes are in green areas, and normal run modes are in grey areas. The duration of a normal run mode depends on the computational loads, while the sleep mode is based on biological recording periods. The details of the power cycling approach can be seen at the Supplement S2.

Fig. 4. The developed system software architecture.

A green cycle represents a controller at sleep mode and a white cycle represents a controller at normal run model. The bottom shows timing diagram of system power cycling performances based on frames.

IV. BMI Processing Toolbox

The BMI processing toolbox has three stages: detection, suppressor and optical converter. Each stage has different algorithms as illustrated in Fig. 5. Also, the brief descriptions of each entity are presented in Table I, and the detailed results of each algorithm are described in Supplement S1.

Fig. 5. A BMI processing toolbox.

It has three calculation stages: detection, suppressor and optical conversion. A detection stage has a simple thresholding, line length and bandpass with integral three functions (a-c). A suppressor has a phase shift algorithm and a PID linear controller (d-e). And optical conversions has a max light intensity, a Pulse Amplitude Modulation (PAM) and a Pulse Width Modulation (PWM) three functions (f-h). The detailed simulation results are shown at texts and Supplement S1.

Table I. BMI Toolbox algorithms.

		Detection
(a)	Simple threshold	Use time domain signal amplitudes as a detection criteria.
(b)	Line length	Calculate the changing rates of data points to detect abrupt signals.
(c)	Bandpass with integral	Filtering signals with a bandpass filter and integral functions.
		Suppressor
(d)	Phase shift	Phase shift signals in a certain degree at a frequency bandwidth.
(e)	PID	Use PID controller to process signals.
		optical converter
(f)	Threshold-max	Delivery the maximum light intensities when signals are above thresholds.
(g)	Pulse amplitude modulation	Convert algorithm outputs into LED DAC values using pulse amplitude modulations.
(h)	Pulse width modulation	Convert algorithm outputs into LED DAC values using pulse width modulations.

The detection stage is a pre-processing stage to determine whether to intervene or not. We include three techniques: simple thresholding [15], line length [16] and bandpass with integral function. These specifically address on entities of signal amplitudes, signal change rates and frequency domains. A testing chirp signal is generated which contains blocks of V_pp = 20 mV, V_pp = 100mV and V_pp = 180mV. Each block is a chirp signal with frequency range in1 -5 Hz (Fig. S1 a(1)). This indicates that signals encode information changes both in time and frequency domain. For each stage, test signals were prestored into a waveform generator (Keysight 33500B).

A thresholding technique enjoys minimum computing resources and ultra-short latency and has been widely used in neuroscience [17][18] and neurotechnology applications [19]. This technique is an ideal candidate for a pre-processing stage or open-loop stimulus. one typical application is to compare recorded signals with the biological experimental noise level to enhance biological information processing [20]. The other model-based criteria such as visual threshold circular signals [21] and position [22] can also be considered as threshold factors. The example in Fig. 5(a) shows a comparison between the absolute threshold (red dash line) and neural signals, in which signals above the threshold values are considered as abnormal activities. The detailed example is illustrated in Fig. S1-1 (b).

The line length technique [16] is shown in Fig. 5(b). It is sensitive to the change rates of amplitudes in the time domain, and has already been widely applied on epilepsy detections [23][24]. It follows the relation in equation 1:

$$LL(n)=\frac{1}{K}{\displaystyle \sum }_{k=n-N}^{n}abs[x(k-1)-x(k)]=\frac{L(n)}{K}$$ (1)

where LL (n) is the normalized line length value at discrete time index n, L(n) is the mining sum of distances between successive points within the sliding window of size N, x(k) is the data sequence value at the k sample, K is the normalization constant, N is the sliding window length, and abs stands for absolute value. Window length and value K depends on signal behaviours and experimental protocols. The LL simulation results in Fig. S1-1 (c) demonstrate that algorithm output not only depends on the abrupt changes in amplitudes but also frequency variations.

Fig. 5(c) shows a bandpass filter with integral function. This calculates the power spectrum within a defined frequency range. The bandpass filter kernel is designed by convolving a windowed-sinc filter and its spectral reversal function. The equations are displayed at equation array (2–4):

$$u=\int h(\tau )g(\tau -t)d\tau$$ (2)

$$h[i]=K\frac{\sin (2\pi f_c(i-\frac{M}{2}))}{i-M/2}[0.42-0.5\cos (\frac{2\pi i}{M})+0.08(\frac{4\pi i}{M})]$$ (3)

$$g[i]=h[i]\times (-1{)}^i$$ (4)

where h[i] is a windowed-sinc filter and g[i] is a spectral reversal function; f_c is cut-off frequency and M is kernel length. Particularly, parameter M mainly decides the computing latency and filter transition bandwidth. Also, the Blackman window is introduced to improve signal continuity in the frequency response. After the bandpass filtering process, a standard integral function is implemented for threshold detection purposes. The simulation results in Fig. S1-1 (d) show the developed technology has reasonable frequency classification behaviour.

The second stage is the suppression functions. This includes a phase shift algorithm and a PID controller. The phase shift algorithm is described in Fig. 5(d). This algorithm determines the intensity of the signal at a given frequency and then returns a corresponding signal with a phase shift. The relation is given in equation (5) below:

$$y=e^{-kt}\cdot \cos (2\pi ft+\phi )$$ (5)

Where k is a decay constant, f is a central frequency and φ is the signal shift (in degrees). A case study of phase shift algorithm with central frequency 5Hz and 180-degree shift is given at Fig. S1-2(b).

The other suppressor technique, based on our previous work [25][7] and designed neural mass [26], is a PID control system. This is described in Fig5. (e) and described in equation (6) below:

$$u(t)=k_p\times e+k_i\times {{\displaystyle \int }}_0^{t}e(t)dt+k_d\times (e-\widehat{e})/dt$$ (6)

Where k_p, k_i and k_d are proportional, integral and derivative gain; e is an error signal calculated by reference signals and current outputs. $\widehat{e}$ is a previous error signal, and t is an integration step. Also, the developed control system can be further evolved into adaptive control schemes [27] for advanced operations. A system with the parameter setting k_p = 1, k_i = 10 and k_d = 0, results are shown at Fig. Sl-2(c).

The last stage is an optical conversion. This is to convert algorithm outputs to implantable LED control commands. The main purpose of this stage is to convert numerical outcomes from the suppressor into real-world optical intensities and waveforms. These need to take into account, light penetration [6] and Channelrhodopsin (ChR2) encoded cell performance [28]. Also, the efficiency of the LED drive circuit [12], LEDs and optrode electro-thermal-optical characteristics need to be taken into account[6].

Three essential converting methods are therefore implemented in the toolbox: 1) a maximum light intensity method with a threshold; 2) a Pulse Amplitude Modulation (PAM) and 3) a Pulse Width Modulation (PWM) technique. The maximum light intensity approach is described in Figure 5 (f). It simply provides a pulse of maximum intensity when the intervention is beyond a certain threshold.

The other two techniques show modulation variants in intensity and time domains which are presented in Figure 5 (g-h). A PAM is used to translate LFP values into LED DAC value linearly, and a PWM technique to translate LFP values into corresponding LED pulse width. Figure S1-3 (a-c) display the optical converter results based on the developed system.

V. Experimental setup

optogenetics is a relatively new tool that through genetic engineering, expresses light sensitive ion channels (e.g. channelrhodopsin) and/or pumps (e.g. halorhodopsin) on the cell membrane. It is therefore possible to use optical stimuli to activate or inhibit a neuron cell activity.

The recent Allergan-Retrosense trial (US Clinical Trial identifier: NCT02556736) demonstrates that channelrhodopsin-2 can be reverse engineered to human retina using Adeno Associated Viral (AAV) vectors. Ingusci[29] provides a review of alternative approaches with Lenti Virii. In this case, we used channelrhodopsin-2 delivered into animals via AAV.

A. In-vitro experimental setup

Rodent brain slices were cut using a 5100mz vibratome (Camden Instruments). The slices were later transferred incubated at room temperature in a brain tissue interface and holding chamber until later electrophysiological recordings.

During recordings, the slices were perfused with oxygenated ACSF (in mm: 126 NaCl, 24 NaHCO₃, 1.2 MgSO4, 1.2 CaCl₂, 10 glucose, 3 KCl, 1.25 and NaH₂PO₄) which also contained the compound 4-aminopyridine (4-AP; 200 μM) to induce epileptiform activity in rodent brain slices held at 32.5°C. As Fig. 6(a) depicts, all in-vivo electrophysiological recordings were performed using an interface recording chamber. Also, we utilized a commercial 16-channel linear multi-electrode array probe (NeuroNexus Technologies: A16x1-2mm-100-177 probes – shanks are 100 μm apart; recording site area on each shank). The simulation output of the developed ASIC based headstage was a 0-5 V voltage signal drove a blue LED light source (473 nm, M470F1; Thorlabs) coupled to a 200 μm diameter optical fibre (M89L01–200; Thorlabs). In the freely moving setup we recorded onto a SD card. The data transmitting latency is 20μs per cycle (Table II), which is significantly less than the real-time time constraint 2ms. Meanwhile, for this in-vitro case we transmitted the data directly to a PC via UART communication to allow for real time display and analysis.

Fig. 6. System experimental setup.

(a) In-vitro closed-loop processing platform. The recoding optrode is NeuroNexus 16-channel array probes and stimulation site is an optical fiber. (b) A rodent carries developed system in a freely moving environment (The holder is tailor designed by using a 3D printer). Fig. 6 (b4) shows system in-vivo experiment mechanical setup, a custom designed optrode(3mm) with electrodes can fully implanted in a rodent brain, the optrode has 8 pins which can integrates with the Neural Engine head-stage pinouts, the LED will be on the shaft. (c) A primate control system on the head for freely moving recording experiment. There are three components of the system as labelled: 1) the brain prototype; 2) a system holder with the neural engine; 3) a battery. The experimental overall picture is shown at (c) top right.

Table II. System Specifications.

Overview
Control unit(size)	25mm×22mm
Head-stage(size)	10mm×10mm
Control unit(weight)	4.1g
Head-stage(weight)	1g
Recording channel	4
Stimulus channel	8
Speed
Recording	45 μs
Stimulation	150 μs
Algorithm processing	3-270 μs
Data logger	19.5 μs
UART transmission	22 μs
Power
Control unit	0.63mA
Head-stage	1mA
SD card	1mA
PMU	0.3mA
Overall	2.93mA

The system recording frequency is setup at 50Hz, and clock frequency is at 40MHz.

B. Freely-moving rodent setup

Two male, 3 to 6 month old, Sprague Dawley rats were used in the electrophysiological recording experiments. The data presented in this study were acquired while the animals were placed in an enclosed home cage during a 2-3 h sleep session. The animals were given no tasks and naturally fell asleep on their own. The behavioural procedure and the electrophysiological recordings were performed under UK Home Office licenses and were approved by the Newcastle University Animal Welfare and Ethics Review Board. A custom designed system holder was mechanically fixed on a rodent head (as per Fig. 6(b). The developed system was connected to the implantable electrodes with 130k ohm [30], and is enclosed in this holder. Data was recorded using an SD card on the control board. Fig 6 (b4) shows an exemplar custom designed optrode with electrodes which can be fully implanted in a rodent brain. The optrode has a shaft length of 3mm and 2 electrodes, which can integrate with the Neural Engine head-stage pinouts.

Briefly, the fabrication of the custom probes was as follows: a silicon wafer was coated with 1 um of silicon dioxide via chemical vapour deposition (CVD). Ti/Au/Ti (20/200/20 nm) metallisation was deposited by evaporation and then patterned by photolithography followed by a three-stage wet etch: NH₄OH and H₂O₂ for the Ti and K₃Fe(CN)6, Na₂S₂O₃ and CS(NH₂)₂ for the Au. The metal patterns were then covered by a matching top layer of silicon dioxide insulation. Contact windows in the top oxide were opened by a Deep Reactive Ion Etch (DRIE) using a mixture of Ar and SF₆. The individual optrodes were then separated from the wafer using a deep reactive ion etch step to cut through the silicon. A gold wire was bonded into each recording site before the shank was coated in medical grade silicone encapsulation (MED6015), after which the gold wire was cut to open a conductive path through the passivation layer.

C. Freely moving NHP setup

Experiments were approved by the local ethics committee and performed under appropriate UK 112 Home Office licenses in accordance with the Animals (Scientific Procedures) Act 1986. Recordings were made from a female macaque with implanted micro-electrodes in the primary motor cortex. The animal was implanted with 116 custom arrays comprising 12 moveable 50 μm diameter tungsten micro-wires (of impedance 117 ~200 kΩ at 1 kHz) using the same technique as previously described[31]. There were six components of the system: (1) an ASIC recording head-stage; (2) a system holder; (3) a processing unit; (4) a recording pinout; (5) a reference pinout and (6) a battery connector. The developed system was integrated with a custom designed holder which was placed on the female macaque head as shown at Fig. 6(c). The data was recorded using the SD card.

D. In-vivo elosed-loop proeessing

As it is shown in Figure 7, Mice weighing 40 g were anaesthetized via inhalation of isoflurane (confirmed by the absence of the pedal withdrawal reflex). After a sufficient depth of anaesthesia was achieved, the animal was fixed in a stereotaxic frame (Kopf, Tujunga, CA, USA). A heating pad with feedback temperature control via a rectal probe (Harvard Apparatus, Holliston, MA, USA) maintained the core temperature of the mouse at 32°C. A skin incision was made in the scalp before the periosteum was retracted to expose the bregma. A craniotomy was drilled above the parietal cortex of the left hemisphere. Optrodes were implanted into the left hemisphere. Two custom designed optrodes (previously described) one for electrical recording and one for optical stimulation were placed in corresponding places. The data was transmitted via UART communication in real-time.

Fig. 7. In-vivo closed processing experimental setup.

Two oprodes are employed in this experiment: one is for electrical recordings and the other one is for optical stimulations. The Neural Engine selects UART data communications for real-time display. The implemented algorithm is a previously described phase shift algorithm. All components are well labelled in the figure.

VI. Results

A. Modulation local field potential activities in a closed-loop

A testing algorithm of phase shift-pulse amplitude modulation (PAM) was implemented as a case study. Specifically, we employed algorithms with different shifted phases from 0 to 315 degrees relative to a target frequency. The experiment was performed in-vitro as previously mentioned. Recorded signals were first filtered at a central frequency f = 6 Hz (experimental LFP central frequency), and shifted by pre-defined degrees. Then the outputs were generated by comparing with a threshold value V_th = 0 V. At the final stage they wee translated into LED driving voltages to control light sources via the PAM algorithm.

The optical simulation command was updated every 2 ms in our experiments. The amplitude depended on the implemented closed-loop algorithm. The algorithm processing time of each degree on/off period was 10 seconds. The results are shown in Fig. 8(A): (a) shows the LFPs recording results; (b) and (c) show the optical stimulation commands and phase condition lines respectively. (d) shows a wavelet transform power analysis performances. Based on the result, it can clearly seen that depending on the different phase shift, the LFP signal oscillates at specific frequency ranges between the 10-15Hz (indicated by the red dash arrows) range of interest. Particularly, three cases are displayed in detail in (e)-(g):(e) displays the LPF oscillations under shifted degree 45 condition; (f) shows the LPF oscillations under shifted degree 180 condition; and (g) gives a result of LPF oscillations under shifted degree 270 condition. These clearly illustrate that the developed system has a capacity to modulate LFP activities at certain frequency bandwidth in a closed-loop processing manner.

Fig. 8. (A): In-vitro closed-loop processing experiments.

(a) is the LFPs recording results; (b) and (c) are optical stimulation commands and phase conditiona lines; (d) is wavelet transform power analysis performances; and (e)-(g) are three different cases in details. (B): a rodent freely moving recording results. The top figure displays LFP recording activities in time domain; and the bottom one shows LFP data with PSD analysis. (C): the primate 24-hour freely moving LFP recording experiments, the top figure displays 24-hour LFP recording data in the time domain, and the bottom shows 24-hour data with PSD analysis. The primate LFP different oscillations are also labeled in the bottom figure.

B. Freely-moving rodent recording experiment

Recorded data captured during a freely moving rodent experiment is displayed at Fig 8(B): (top) LFP data in the time domain, and (bottom) frequency domain with Power Spectral Density (PSD) analysis. In general, there are three different types of signals: at the very beginning there have 50 Hz noises which are at experimental setting up stage: the recording pin was floating in the air. In between there are the other two types of signals: one is spindle oscillations (light sleep) between 10-12 Hz; the other one is Delta oscillation (deep sleep) between 0-5 Hz. These LFP oscillations are matched with the rodent sleep-awake behaviours in the experiment.

C. Freely-moving NHP recording experiment

A 24-hour non human primate freely moving recording is shown in Fig 8(C): the top figure displays 24-hour LFP recording data in the time domain, and the bottom figure shows data with PSD analysis. The data clearly shows the differences between the primate in asleep and in wake-up conditions. At the beginning of time period 11:00-13:00: there are lots of movement introduced large amplitude signals (over 1mV), which is due to the electrodes movements as well as connection wire issues. At time period 13:00-17:00, the data displays LFP oscillations with beta oscillations with broadband artefacts that indicates the primate was at the awake condition. At time period 17:00-05:00, The LFPs had regular delta oscillations (0-5Hz) and spindle oscillations (10-12Hz), which indicates that the primate was in light and deep sleep status. After 05:00, the LFP sleep patterns smoothly disappeared and came back to the beta oscillations (20Hz) in the awake condition. In addition, the movement introduced signal can be filtered out since it has clear difference characteristics (e.g. frequency range, amplitudes) with normal LFP oscillations.

D. In-vivo closed-loop processing

The LFPs and the closed-loop optical stimulation commands of the in-vivo experiment are presented in figure 9 (a). It shows the stimulation pattern defined by the closed-loop algorithm which overlay of stimulation onto filtered data. The central frequency of the algorithm was set to 10 Hz (with bandpass filter 5-15Hz). The bandpass range was chosen in such a way that the correlation between the input recorded data and the output stimulations. The algorithm processing time is ON for 5 seconds and OFF for 5 seconds. Each colour correlates to a different phase in the kernel of the algorithm (from 0 degree to 315 degree). This leads to eight discrete phases, which repeat in a cyclic manner. Figure 9(b) shows the effect of the stimulus in closed-loop processing. The LFPs data are displayed by using violin plots at each stimulus phase condition. By comparing with an Alg-off violin shape, there are clear modifications on neural patterns in all shifted degree domains since violin shapes are varied at each condition. This indicates that these shifts have considerable modifications in the closed-loop control manner.

Fig. 9. In-vivo closed processing results.

(a) is the overlay of stimulation onto filtered data. Different colour represents optical stimulation commands at different phases (from 0 deg to 315 deg), The algorithm processing time is oN for 5 seconds and oFF for 5 seconds. (b) are the effects of the stimulus in in-vivo closed loop processing. The LFPs are displayed by using violin plot at each stimulus condition. The mean value is labelled in black line and the medium value in labelled in red line.

E. System specifications

The system specification is shown in Table II. The head-stage is 10 mm × 10 mm and 1g, and a control unit is 25 mm × 22 mm and 4.1g. System recording and stimulation latency is 45 μs and 150 μs (per one request) respectively. Depending on the implemented algorithms, the latency of closed-loop processing varies from 3-270 μs, the latency of data logger (SD card) and the universal asynchronous receiver-transmitter (UART) is 19.5μs and 22μs, respectively.

The average power consumption of the control unit is 0.63 mA at the condition in which the recording frequency is 50 Hz, and the MCU clock frequency is set to 40 MHz. The head-stage approximately consumes 1 mA [12], the SD card and power management system consume 1 mA and 0.3 mA respectively. The maximum power that could be consumed by a single LED can be up to 5mA (at voltage supply 5V), though we typically consume much less to ensure no adverse thermal effect. However, the average power is very much dependent on the frequency of adverse activity in the tissue and the threshold for intervention. As such, the power budget for this would need to be calculated separately for each application.

The system power cycling performance is shown in Fig. 10(A): an embedded controller consumes 12 mA (peak currents at 14 mA) normal run mode and approximately only 0.3mA in sleep mode. The normal run period is 475 μs at this case, and the rest periods are all in sleep mode. The specific details of the power cycling approach are described in Supplement S2.

F. System thermal experiment

Previously discussed implantable electronics have been constrained by battery capacities and thermal issues [6]. This places significant limits on the processing and stimulus. Interesting, the constraints of implantable electronics are very similar to our work, such as 24-hour battery replacement. A thermal experiment was therefore carried out to investigate the thermal performance of the developed embedded system (at room temperature 25.1°C). For comparison purposes, systems with a normal run mode and with a power cycling mode were processed for an hour. The results are shown in Fig. 10(B): at the normal run condition, the device gradually raised up to 27°2C after 10 minutes, and then maintained this value. While at power cycling mode system was at standstill at a room temperature of 25.1°C. This indicates that the developed technology holds a promising solution for addressing thermal issues of implantable electronics.

Fig. 10.

(A): A system current dynamic performances at recording frequency 50Hz. The system at normal run and in sleep conditions are both labelled with red arrows and texts. At normal run mode, system consumes 12mA, while at sleep mode, system consumes 0.3mA. The data was captured by a DC power analyzer (Agilent Technologies N6705B). (B): The system power consumptions between recording frequency 1Hz and 2000Hz. Two benchmarks with the best and the worst computation load are implemented at case studies. (C): A thermal comparison between system at a normal run and at a power cycling mode. At normal run condition, the device gradually raised up to 27.2°C after 10 minutes, and then maintain this value. While at power cycling mode system always at room temperature 25.1°C. (D): The energy consumptions of each algorithm (a-i).

VII. Discussion

A. Power cycling technology

We investigated the neural engine power cycling performance at a biological recording frequency range between 1 Hz and 2000 Hz. In our developed BMI processing toolbox, two sets of algorithms are implemented to represent computationally light and computationally heavy loads (9 μs, 333 s). The light case is a simple threshold of the maximum intensity, and the heavy case is a bandpass-phase shift- PAM. The results are displayed in Fig. 10(C).

In general, as the biological recording frequencies increase, the current consumption increases up linearly. Specifically, the developed neural co-processor consumes less than 1 mA when recording frequency is below 100 Hz. This demonstrates that at low-frequency recording applications such as brain field signals (e.g. less than 50 Hz) [32], the developed technique has significant power reduction performance, which approximately equals the ASIC circuits level (e.g. less than 5 mW) [8]. However, when recording frequency is required over 100 Hz for fast signals such as action potentials, the developed technology shows similar power performances to the standard embedded hardware. Meanwhile, the system with different computational load displays significant power consumption variations (at recording frequency 1000 Hz, the best case consume 0.42 mA while the worst case is 2.6 mA). Also, Fig. 10(D) displays the energy consumptions of each algorithm. This can be further optimized by reducing window size and kernel taps, change the data-type from the floating-point to the integer, and use software in-line functions.

B. The comparison to the other works

We show several similar works developed recently in Table III. Liu et al. [8] and Kassiri et al. [9] develops an ultra-low power BMI system with elegant ASIC designs with 8 mW and 2.8 mW power consumptions, respectively. They both showed valid in-vivo data with AP detection along with phase synchronization performances. Regarding energy and physical dimension issues, these systems are very well developed. However, fixed CMOS circuits cannot be reprogrammable which may not ideal for medical practical applications. Zanos et al. [11] designed an MCU based system for primate freely-moving experiments. By taking advantages of a commercial development board PSoC, the algorithm could be updated. However, the overall consumption was significantly large at 284mW. Similarly, Gagnon-Turcotte et al. demonstrated a strong computing scalability system with 32 recording/stimulation channels with a comprehensive wavelet data compression technique [10]. The power consumption is still relatively high at 119mW. Therefore, translating these devices into clinical applications would require further power optimization efforts to avoid battery life and thermal issues. Meanwhile, [10] and our own effort employ optogenetic based stimulations [33], while the others use traditional electrical stimulation, which cannot be genetically targetted [34]. Regarding freely moving experiments, our system also shows both primates/rodents reliable LFP recordings. And for in-vitro experiments, the developed system demonstrates considerable closed-loop neuromodulation outcomes, which includes both stimulation and recording. Therefore, the technology presented in this paper is the only one with both ultra-low power and computing flexibility performances as well as valid freely-moving LFP recording data. And this embedded-ASIC hardware may inspire us for the next generation BMIs [35][36][37][38].

Table III. The comparison between the other works.

System	Recording/Stim channels	Computing flexibility	Weight/ size	Power consumption	Data transmission	Techniques	DSP functions	Optica] stimulus	Freely moving Experiments#
This work	2/8	YES	4.1g/25mm × 22mm 1g/10mm×10mm	2.9mA*(9.7mW)	UART/SD card	ASIC+MCU	BMI processing tool bos	YES	YES
Gagnon, 2017	32/32	NO	2.8g/17mm*18mm	32mA (119mW)	Wireless	AS1C+FPGA	Adaptive threshold & Wavelet compression	YES	NO
Liu, 2017	16/16	NO	18g/18mm*30mm	2.16mA (8mW)	Wireless	ASIC+MCU	AP detection and PID	NO	NO
Kassiri, 2017	24/24	NO	6g/20mm*20mm	1 52mA (2.8mW)	Wireless	ASIC+FPGA	Phase synchronization	NO	NO
Zanos, 2011	3/3	YES	NN/35mm*55mm	78.8mA (284mW)	SD card	ASIC+MCU	None	NO	NO

^*The Power consumption data tn this table doesn’t include implantable LEDs’.

^#The freely moving experiments are indicates both on primates/rodents.

Also it should be noted that we have utilized an SD card as as our biological teams found this more convenient in this instance.

Last but not least, there have been some the other techniques show various capabilities in optogenetic fields. Tae-il Kim[39] developed a multifunctional optrode with injected light sources, detectors, sensors and other components which can place into a precise location of the deep brain. This technique provides optical, thermal, and electrophysiological studies in a freely moving environment. Yu [40] provided a wirelessly controlled, implantable, micro-LEDs based optical neural face for behaving animals, which is interesting for animal behavior research in neuroscience.

C. The applications

The BMI processing toolbox is developed for neuroscience various applications. Regarding the three-stage algorithm configuration: for a detection stage, a simple threshold algorithm shows a low detection accuracy but short latency, which is suitable for noise detections or explicit time domain processing; a bandpass filter allows only interested signals pass into the next stage process with a considerable process delays (149 μs at 101 taps at sampling frequency 250Hz). Therefore, it suitable for frequency domain applications. A line length method can be applied either in time or frequency domain since it’s sensitive to signal abrupt variations. The phase shift algorithm could help us investigate several fundamental neuroscience mechanisms such as neural coherence [41][42] and synchronization [43][44]. A standard PID controller has been implemented for control/modification of abnormal activities – in particular, for epilepsy [45][46]. The final optical converter stage is determined by the required suppression/activation level as well as opsin expression, opto-electronics design and LED performance.

It should be noted that the parameter selection in each algorithm is experimentally determined. So there will need to be a long-term experimental follow up to explore these parameter ranges for different applications.

Two case studies have been employed to demonstrate system various applications. The power consumptions of benchmark 1(best case) and benchmark 2 (worst case) is 1.38mW and 8.5mW at recording frequency 500Hz (without extra averaging function), respectively. Therefore, the neural engine can be reconfigurable depends on experimental requirements such as closed-loop algorithms/parameters.

D. Future work of in-vivo closed loop processing

The current in-vitro and in-vivo is sufficient to demonstrate the platform is capable of electrical recording, closed loop processing and optical stimulation in a form factor that can be deployed in-vivo. For the near future in-vivo freely moving closed loop experiments, optrode LED bonding technique and encapsulation should be tailor designed for alleviating artefacts and leakage current issues. More importantly, system level approach is defined to evaluate animal behaviors corresponding with closed loop processing. Based on these, the developed platform BMI toolbox should be correspondingly modulated to maximum the system performances as well as mechanical setups (e.g. grounding, optrode locations).

VIII. Conclusion

In this work, we have developed a neural engine for closed-loop optogenetic processing. First, an in-vitro experiment is performed to demonstrate system closed-loop processing for modulating LFPs at specific frequencies. Both non-human primate and rodent in-vivo experiments have been shown with reliable system recordings. Also, a thermal experiment and a system current dynamic analysis has been carried out to illustrate system ultra-low-power computing performances for long lifetime.

Therefore, the major contributions are as below: 1) A novel embedded-ASIC hardware architecture has been developed for optogenetics closed-loop applications, which shows both ultra-low power performance and strong computing flexibility, we also demonstrate optical closed-loop neuromodulation performances; 2) An in-vitro and in-vivo closed-loop processing platform has been developed to meet various neuroscience/neurotechnology applications requirements. Particularly, tailor designed BMI toolbox is implemented which allows the system to has wider application scopes; 3) An in-vivo recording platform has been developed for both primates/rodents freely moving environments We have demonstrated the ability to record activity through electrical recording, modulate activity through optogenetic stimulation, and perform closed loop processing have all been demonstrated together in in-vitro preparation. We have also tested this same hardware in-vivo to show the form factor is appropriate. In the near future we will include in-vivo freely moving experiment and focus on the animal behaviors control research by taking advantages of developed closed-loop system.