NeuroGPU: Accelerating multi-compartment, biophysically detailed neuron simulations on GPUs

Abstract

Background:

The membrane potential of individual neurons depends on a large number of interacting biophysical processes operating on spatial-temporal scales spanning several orders of magnitude. The multi-scale nature of these processes dictates that accurate prediction of membrane potentials in specific neurons requires the utilization of detailed simulations. Unfortunately, constraining parameters within biologically detailed neuron models can be difficult, leading to poor model fits. This obstacle can be overcome partially by numerical optimization or detailed exploration of parameter space. However, these processes, which currently rely on central processing unit (CPU) computation, often incur orders of magnitude increases in computing time for marginal improvements in model behavior. As a result, model quality is often compromised to accommodate compute resources.

New Method:

Here, we present a simulation environment, NeuroGPU, that takes advantage of the inherent parallelized structure of the graphics processing unit (GPU) to accelerate neuronal simulation.

Results & comparison with existing methods:

NeuroGPU can simulate most biologically detailed models 10–200 times faster than NEURON simulation running on a single core and 5 times faster than GPU simulators (CoreNEURON). NeuroGPU is designed for model parameter tuning and best performs when the GPU is fully utilized by running multiple (> 100) instances of the same model with different parameters. When using multiple GPUs, NeuroGPU can reach to a speed-up of 800 fold compared to single core simulations, especially when simulating the same model morphology with different parameters. We demonstrate the power of NeuoGPU through large-scale parameter exploration to reveal the response landscape of a neuron. Finally, we accelerate numerical optimization of biophysically detailed neuron models to achieve highly accurate fitting of models to simulation and experimental data.

Conclusions:

Thus, NeuroGPU is the fastest available platform that enables rapid simulation of multi-compartment, biophysically detailed neuron models on commonly used computing systems accessible by many scientists.

1. Introduction

Electrical activity of single neurons is determined by the distribution of various ionic conductances arranged across complex morphologies (Mainen and Sejnowski, 1996; Hausser et al., 2000; London and Häusser, 2005; Spruston, 2008; Hay et al., 2013; Alonso and Marder, 2019). Our understanding of single neurons has long relied on the ability to construct biophysically rigorous models of how neuronal membrane potential [Vm], and, hence, action potentials (APs, spikes) are generated from currents [I] flowing across the membrane and through the cell (Fig. 1) (Hodgkin and Huxley, 1952). Wilfrid Rall described the biophysical theory of how membrane potential of a single neuronal segment (‘compartment’) depends on the conductance (e.g., gNa) and voltage dependent flow of specific ionic species [e.g., sodium (Na) and potassium (K)], as well as passive properties of the membrane (i.e., capacitance) (Fig. 1, top row) (Rall, 1962a). Using cable theory, Rall further described how to connect different compartments of a neuron, providing the foundation for modeling complex, spatially extended neuronal morphologies (Fig. 1A) (Rall, 1962b). Concomitantly, the membrane channels that mediate a specific ionic current exhibits large diversity of genetically defined conductances (e.g., gNav1.2, gNav1.6, etc.,), indicating that individual compartments are, in reality, quite complex (Fig. 1B). While just beginning to be appreciated at the time, we now know that ion channels have their own voltage dependent kinetics that are determined by the transition probabilities amongst various states of the channel subunits (Fig. 1C) (Hille, 1984; Colquhoun and Hawkes, 1995). Finally, simulations of realistic neural network models require faithfully capturing the complexity of the individual neurons (Einevoll, et al., 2019). While the physical theories required to link these vastly disparate spatial-temporal scales exists, our ability to utilize them for basic understanding and clinical translation is impeded by the computational burden of the required simulations (Fig. 1D and E).

Fig. 1.

Optimizing biophysical neuronal models increase with their complexity. Biophysical neuronal model complexity depends on three factors (A–C) which determine computational costs required to simulate and fit models to empirical data (optimization) (D and E). A: Morphological complexity: The abstraction of the neuronal morphology ranges from a point neuron, to a simplified morphology, and ultimately toward different methods to reconstruct neuronal morphology with high resolution. B: Compartmental complexity: Compartmental models vary in complexity both in number of compartments and the content of each compartment. Top: Single compartment with basic, conductances required for spiking and a resting membrane potential. Middle: Multi-compartmental model where ion-channels are aggregated under several conductances e.g. all different voltage gated potassium channels are represented in a slow and fast inactivating channel (Korngreen and Sakmann, 2000). Bottom: High resolution morphology with detailed representation to all channels sub-types in each compartment. C: Single-channel complexity: The overall conductance in compartmental models is formalized either with Hodgkin-Huxley equations or with Markov-based models. When fitting a model to empirical data, several parameters can be varied. Top: only the maximal conductance in a Hodgkin-Huxley formulation. Middle: coefficients added to Hodgkin-Huxley equations (time constants, voltage dependence). Bottom: In Markov-based channels the transition coefficients can be varied. D: Optimizing a model (fitting model to data) depends on the complexity of the model and number of free parameters. When complexity and number of free parameters require compute times that exceed a few days, it becomes impractical to use high resolution models. E: Developing tools to accelerate simulation and optimization time will enable us to use more complicated and biophysical relevant models.

Our predictive understanding of several neuronal cell classes has benefitted from the repeated refinement of compartmental models that describe their activity in health and disease. For example, neocortical pyramidal cells have been modeled extensively, providing insight into the effects of morphology on AP firing characteristics (Mainen and Sejnowski, 1996; Hay et al., 2011; Almog and Korngreen, 2014), how synaptic input patterns affect spike output (Maršálek et al., 1997; Dlesmann et al., 1999; Destexhe et al., 2003) how APs initiate and propagate within axons (Kole et al., 2007, 2008; Shu et al., 2007; Hu et al., 2009; Hallermann et al., 2012; Cohen et al., 2020), and how neuronal activity is affected by alterations in ion channel function induced by genetic variation (Zamponi et al., 2010; Allen et al., 2014; Ben-Shalom et al., 2017; Spratt et al., 2019). Similar intensive studies have focused on other cell classes, including hippocampal pyramidal cells (Mainen et al., 1996; Magee and Cook, 2000; Poirazi et al., 2003; Narayanan and Johnston, 2008; Milstein et al., 2015), cerebellar Purkinje cells (De Schutter and Bower, 1994; Miyasho et al., 2001; Roth and Häusser, 2001), and midbrain dopaminergic neurons (Canavier, 1999; Canavier and Landry, 2006; Kuznetsova et al., 2010). In parallel with these advances in modeling, there has recently been an enormous improvement in experimental approaches to better understand the diversity of neuronal classes and their activity patterns. Within the general group of neocortical pyramidal cells, for example, exists a wealth of diversity. This includes not only differences in morphology and activity across laminae (Smith and Häusser, 2010; Deitcher et al., 2017; Kanari et al., 2019), but also within laminae depending on genetic makeup or axonal projection targets (Dembrow et al., 2010; Gee et al., 2012; Clarkson et al., 2017), or even within what was thought to be a homogenous cell class within a single layer as one samples across a large region of cortex (Fletcher and Williams, 2019).

Given the enormous complexity and vast spatio-temporal scales described above, generating models that accurately recapitulate neuronal activity across the true range of diversity present in nature can be a daunting task. Model fitting often requires one to tune individual parameters to best match empirical observations. This process can be aided by iterative rounds of parameter exploration and optimization that aim to minimize the differences between empirical data targets and their associated models. These procedures can be computationally demanding (Fig. 1D and E). Indeed, each linear improvement in model accuracy requires an exponential increase in computational resources (Nocedal and Wright, 2006; Gurkiewicz and Korngreen, 2007). Thus, model optimization is often done on supercomputers that parallelize these computations across massive number of central processing unit (CPU) cores. Unfortunately, the cost of constructing and operating supercomputing centers is similarly massive. As such utilization of these resources are typically restricted to large consortia, such as the Blue Brain Project (BBP) (Markram et al., 2015) and the Allen Institute (Gouwens et al., 2018). For more restricted budgets, simulations must typically be compromised in scale, complexity, or accuracy, thus negatively impacting results (Almog and Korngreen, 2016).

In the past 10 years, graphics processing units (GPUs) have emerged as an alternative to CPU-based hardware that may offer comparable levels of performance at substantially reduced cost for some problems. GPUs utilize streaming multiprocessors with multiple simple cores that allow for distributed, parallelized computing for relatively small chunks of data. With software optimized for GPUs, GPU-based computing can often outperform CPU-based applications in processing speed and cost for some problems (Payne et al., 2010). Today, GPUs are being used in scientific simulations, including molecular dynamics (Go et al., 2012; Salomon-Ferrer et al., 2013) and climate modeling (Prein et al., 2015), and are the computational engine for most modern artificial intelligence applications (Schmidhuber, 2015). In neuroscience, GPUs are currently being used to accelerate complex imaging dataset processing (Eklund et al., 2013), spiking neural network analysis (Yavuz et al., 2016; Chou et al., 2018), clustering of activity from in vivo extracellular electrophysiological experiments (Pachitariu et al., 2016), and simulations of single ion channels (Ben-Shalom et al., 2012).

Recently, two platforms for neuronal biophysical simulations with GPU support were developed: Arbor (Akar et al., 2019) and CoreNeuron (Kumbhar et al., 2019). Both platforms focused on simulating large scale neuronal networks comprised of detailed multi-compartmental models. CoreNeuron supports previous NEURON models but is not implemented in CUDA, the fundamental operating language of NVIDIA’s GPUs. As such, its ability to accelerate model runtimes with GPUs may not exploit the full potential of GPU computing. Arbor, instead, is implemented in CUDA via an entirely new simulation environment. Thus, while it does harness the speedup potential of GPUs, it is not clear how existing models, such as those found in ModelDB and the BBP portal (McDougal et al., 2015; Ramaswamy et al., 2015), could be ported to Arbor, thus impeding utilization. Here, we describe NeuroGPU, a computational platform optimized to exploit GPU architecture to dramatically accelerate the simulation of multi-compartmental neuronal models. Our goal with NeuroGPU was different than that of CoreNeuron or Arbor. Rather than focusing on neuronal network simulations, NeuroGPU is designed to optimize fitting of models that best recapitulate empirical data derived from single neurons, and to study the parameter space of such models by iterating parameter values. To do so, we developed new approaches to parallelize compartmental models, utilizing the GPU-based programming language CUDA to optimize memory handling on GPUs manufactured by NVIDIA. This resulted in simulation speedups of up to 200-fold on a single GPU and up to 800-fold using a set of 4 GPUs. Similar optimizations may be made for other GPU manufactures, but this was not considered here, given the broad usage of NVIDIA hardware. We found that NeuroGPU performed faster than NEURON using (Message Passing Interface) MPI by up to 10-fold and CoreNeuron up to 4-fold. Building on our previous efforts (Ben-Shalom et al., 2013), we developed an intuitive user interface that can import most compartmental models implemented in NEURON (Carnevale and Hines, 2006) deposited at the ModelDB (McDougal et al., 2017) or BBP portal (Ramaswamy et al., 2015). Further, we provide methods to explore model parameter space to study how each parameter of the model contributes to its voltage output. The runtimes of NeuroGPU enables us to sample the parameter space in a very detailed manner, which were utilized here to reveal the response landscape of single neurons. Finally, we provide an interface to use NeuroGPU for fitting models to experimental data with evolutionary algorithms (DEAP and BluePyOpt) (Gagn, 2012; Van Geit et al., 2016). NeuroGPU implemented such model optimization algorithms in 2.5 h using a single GPU (see Fig. 5), compared to 26 h using 256 CPUs, which is the current standard (Nandi et al., 2020; Schneider-Mizell et al., 2020). This enables the use of more complex models that will better represent experimental data, as we demonstrate here in both simulated and experimental data. NeuroGPU therefore provides an open-source platform for neuronal simulation with increased speed and reduced cost, thus enabling the neuroscience community to perform high quality biophysically detailed simulations.

Fig. 5.

NeuroGPU fits BBP PC model to empirical data. A: Left: morphology of L5 thick-tufted pyramidal neuron from somatosensory cortex (Ramaswamy et al., 2015). Right: NeuroGPU fits (green traces) the L5 PC model to empirical data recorded from an L5 prefrontal cortex pyramidal neuron of a mouse (Black Traces) with different stimulus intensity 200–340pA (Spratt et al., 2019). B: Left: Prefrontal-cortex layer 5 pyramidal neuron morphology (Ascoli et al., 2007; Yin et al., 2018). The morphology of BBP PC model was modified to that of the prefrontal neuron and fitted to the same data as in A. C: Number of APs per 300 ms of step stimulation for different models of layer 5 pyramidal neuron. Note that both revised models overlap target data.

2. Methods

2.1. Hardware

NEURON and TitanXP-based simulations were run on a PC with Intel Core I7–7700 K 4.2 GHz with 16 GB of RAM. Tesla V100-based simulations were run using the NVIDIA PSG cluster. Here, each simulation was run on a single node with Haswell or Skylake CPU cores. For multi-GPU simulations, we used cluster nodes with NVLINK (Li et al., 2019) between the GPUs to enable memory peer-access. Fitting models to experimental data was done on the Cori GPU cluster from the National Energy Research Scientific Computing Cent (NERSC) at Lawrence Berkeley National Labs. Cori GPU nodes includes 8 NVIDIA Tesla V100 and 20 Skylake CPU cores with total of 384 GB memory.

2.2. Software

Simulations were performed in NEURON 7.6–8.0 and CUDA 10.1. All scripts were written in Python 3.7. All software is available at https://github.com/roybens/NeuroGPU.

2.3. Importing NEURON models

To ease installation, we separated processes for porting models (Translation Fig. S2C) and NeuroGPU execution (Execution Fig. S2D). After initial import, NeuroGPU models can run on a GPU machine independently from NEURON. The python script extractmodel.py exports NEURON models to NeuroGPU. This script reads all simulation details from runModel.hoc, which is populated using the GUI (Fig. S1). NEURON models are described using either hoc or python scripts. The scripts include a morphology that can either be called as a separate file or constructed within the script (Fig S1B). The user must input a file containing model stimulation, which includes temporal aspects of the model and command currents delivered at a prescribed location. Furthermore, all free parameters, such as channel properties, must be described. These import components are translated into CUDA code, termed kernels, that can run on the GPU via the python script “extractmodel.py” (Fig S1C). This script first takes runModel.hoc and loads it into NEURON, not to run simulations, but rather to query NEURON for model properties needed for subsequent porting to NeuroGPU, including compartment names and the tri-diagonal matrix (F-Matrix or Hines Matrix) which holds the differential system for the voltages of the dendritic tree. Then, the script iterates over the.mod files in the directory, parses them and creates relevant kernels for each mechanism described. CUDA kernels containing model mechanisms are generated by adding relevant CUDA keywords to C code generated when NEURON compiles mechanisms. Mechanism kernels are written to the AllModels. cu in similar structure as described previously (Hines and Carnevale, 2000; Carnevale and Hines, 2006), iterating over all compartments defined in the model. A new hoc file is created to register mechanism values, which are stored in AllParams.csv and inserted in each compartment. After model parameter maps are determined, they are cataloged as part of ParamMappings.txt for reference for future reiterations of the same model, eliminating the need to reload NEURON. Finally, the script writes code translated to CUDA in NeuroGPU.cu and packages the application to run on either Windows or Unix. After compiling the code, an executable is created that reads the AllParams. csv and the stimulation and runs the model on the GPU.

2.4. Translating mechanisms to CUDA and memory assignment

Mechanisms in NEURON are described by NMODL (.mod) files (Hines and Carnevale, 2000), which update the mechanism states every simulation time step. This is done using three different procedures within NEURON that initialize mechanisms (nrn_init), update currents that mechanisms affect (nrn_cur), and then update mechanism states (nrn_state) (Carnevale and Hines, 2006). In NeuroGPU, CUDA kernels are written for each of these procedures using.mod and.c files that are generated by NEURON when running nrnivmodl. When a NEURON model is exported to NeuroGPU, all.mod files and their corresponding.c files are parsed using custom code to extract variables and procedures defined in each specific.mod file. These are then translated to CUDA kernels written in AllModels.cu. Future iterations of NeuroGPU may make use of the nmodl Python library for parsing as this software evolves.

Several mechanisms are regulated by intracellular calcium. To support this, we created an array that holds internal calcium concentration within each compartment and calculates the reveral potential for calcium at each time step. We determined that calcium was the only ion whose intracellular concentration varied substantially during ongoing activity in ways that affect simulations, due to their effects on calcium-activated potassium channels. We note that most models lack detailed models of sodium and potassium pumps/transporters and generally do not model changes in their concentrations. We found that this was a reasonable approximation for NeuroGPU, as the presence or absence of Na/K ion concentration calculations did not affect simulation Vm.

CUDA is an extension of the C programming language that enables computation on the GPU (Nvidia, 2018). CUDA kernels are procedures running on the GPU that can be invoked from either the GPU or CPU. To invoke a kernel from the CPU, one must specify the number of parallel threads used. Threads, which allow for parallelization on the GPU, are organized into blocks, with each thread occupying a specific address within that block (idx.x) (Fig S2C). GPUs are structured to operate well when computing 32 parallel threads, a computing structure termed a warp (Nvidia, 2018). Therefore, we structured NeuroGPU to utilize 32 threads in the x dimension, corresponding to individual morphological segments within the model. For a given model with more than 32 segments, individual threads are responsible for calculating every 32nd segment. For example, thread #1 would calculate segments 1, 33, 65, … 32N + 1.

2.5. Extracting simulation properties from NEURON

NeuroGPU utilizes NEURON for simulation pre-processing, including mechanism translation which includes mathematical descriptions of various ion channels, calcium diffusion characteristics, and other elements of neuronal function, (Hines and Carnevale, 2000), a map for mechanism distribution across compartments (ParamMappings.txt), and exporting the tri-diagonal matrix using fmatrix(). These are stored in BasicConstSegP.csv. NEURON extracts all parameters for cable equations and mechanism values within each compartment to AllParams.csv (Fig. S1). External stimulation delivery location, intensity, and time-course are written in stim.csv. Resting membrane potential and number of time steps in the simulation are written in sim.csv.

2.6. Solving the tridiagonal matrix

Matrix solutions were performed using the branch-based parallelism approach as described previously (Ben-Shalom et al., 2013), with morphology analysis guiding iterative matrix computations. This analysis is done in extractmodel.py and the data structures to solve the tri-diagonal in parallel is stored in BasicConstSegP.csv.

2.7. Benchmarking

All benchmarking against a single CPU core running NEURON was done with NEURON 7.6, running in a single thread. The morphology was adjusted to have one segment per compartment in all platforms. Simulation runtimes were compared without hard drive read/write file steps, as these aspects depend more on hard drive properties than CPU/GPU comparisons. Benchmarking against the parallel version of NEURON was done using NEURON 8.0. In both cases, runtime compares time required to complete psolve() procedures. For NEURON-MPI, we used CoreNeuron settings with 32 MPI processors without GPUs (Kumbhar et al., 2019). For CoreNeuron, we used one MPI thread and one GPU (Kumbhar et al., 2019). Note, benchmarking of cases with 2^14 model instances is estimated based on linear extrapolation from the 2^13 case, as the time required to load model instances in the 2^14 case far exceeded the time allowed by policy queues at NSERC.

2.8. Multi-compartmental models

NeuroGPU performance was tested with 4 different models:

1. A passive model, utilizing passive channels described in NEURON distribution pas.mod file. These channels were distributed on both simple and complex morphologies (see Fig. S3A, D) (Mainen and Sejnowski, 1996). The simple morphology was based on the simple morphology described in Mainen and Sejnowski, with compartments reduced to 32, as this is the minimum number of compartments required for NeuroGPU-based simulations.

2. The Mainen and Sejnowski (1996) model, with channels distributed on the same complex and simple morphologies. Channels are distributed as in (Mainen and Sejnowski, 1996) (Fig. S4)

3. A pyramidal cell model from the Blue Brain Project portal (Ramaswamy et al., 2015) (Fig. 2). BBP PC refers to the model named L5_TTPC1_cADpyr232_1.

4. A chandelier cell model, termed BBP CC, referring to L5_ChC_dNAC222_1. For this model, the Kdshu2007.mod files were altered to run on NeuroGPU. Specifically, global variables were removed from the neuron block and instead placed in the assigned block (Carnevale and Hines, 2006) (Fig. 2).

Fig. 2.

NeuroGPU reduces simulation run-time of complex neurons by orders of magnitude without compromising accuracy.NeuroGPU reduces simulation run-time of complex neurons by orders of magnitude without compromising accuracy. A: Morphology of a BBP portal layer 5 neocortical pyramidal cell (Ramaswamy et al., 2015). Dendrite in black, axon in red. B: Top: injected current at the soma. Middle: NEURON voltage response as recorded at the soma. Cyan: NeuroGPU response as recorded at the soma. Bottom: difference in voltage between NEURON and NeuroGPU. C: Top: APs generated per current injection intensity in the soma. Middle, bottom: Peak and average voltage difference between the voltage response in NEURON and NeuroGPU. Red circles denote examples in B. D: Top: Comparing PC model runtimes for the different simulators: black – NEURON, grey MPI-Neuron (32 processors) green – CoreNeuron, blue – NeuroGPU. X-axis in log2 scale, Y-axis in log10 scale. Bottom: Speedup compared to NEURON. Note that the run time for 2¹⁴ neurons using CoreNeuron is extrapolated (see methods). E: Top: Runtime for the model on 1–4 GPUs (Tesla V100) on the same node. Bottom: Speedup compared to NEURON. F–J: Same as A-D, but for the chandelier cell model.

2.9. Optimization algorithms

Two different genetic algorithm versions were used in this study. For data related to Fig. 4, the eaMuPlusLambda algorithm from the DEAP package. eaMuPlusLambda stands for evolutionary algorithm where the next generation population is comprised from the existing population (Mu) and the offspring (Lambda). We modified the varOR procedure to call NeuroGPU (Rainville et al., 2012).”Optimization was performed on the BBP PC model. For each iteration, the algorithm began with a new population of parameters with values randomly chosen with the range specified in Supplemental Table 3. The model was modified to accept new values from the optimization algorithm (similar changes were necessary for parameter space exploration for Fig. 3). Target data were generated using the original parameters values described in Supplemental Table 3. Optimization was targeted to reduce error between target data and test data using both the interspike interval (ISI) and the root mean square (RMS) of the voltage as the error function. Error was reduced to a single variable by weighting these two variables as: 10*ISI + RMS.

Fig. 4.

NeuroGPU accelerates evolutionary optimization for fitting models to neuronal data. A: Voltage traces obtained from optimization (worst case from population of 100: red; best case from population of 10,000: cyan) compared to ground truth (black). B: Optimizations examples using DEAP with different sizes of populations. Four Optimizations with different random starting population over 50 generations. Y axis is the error from the target voltage as described in the methods section. Lower values denote less error from target data. C: Comparing runtimes for optimizations using NeuroGPU and NEURON (linearly extrapolated from 5 generations). Circles are color coded for population size as in A, and represent mean ± SEM. D: Best score in each optimization in A. Circles and error bars as in C.

Fig. 3.

NeuroGPU enables rapid exploration of parameter space in complex pyramidal neuron model. A: Each point in the grid represents the number of APs in the relevant model. Points on the axis represent the varied conductances of Na_v and K_v at the axon in the range of [0,10] and [0,20] S/cm², respectively. B: Example voltage responses for chosen models from A. Colors match to the corresponding model location.

For data related to Fig. 5, the BluePyOpt (Van Geit et al., 2016) implementation of Multiple Objective Optimization (MOO) was used. Experimental target data for these experiments were from whole-cell current-clamp recordings from layer 5b thick tufted pyramidal cells in acute slices from wild-type mouse prefrontal cortex (postnatal day 62) (Spratt et al., 2019). Optimization was targeted to minimize the root mean square voltage error at each time point between empirical data and model output as well as the following objectives, as defined in the electrophysiology feature extraction library (eFEL) from the BBP: voltage_base, AP_amplitude, voltage_after_stim, ISI_values, spike_half_width, and afterhyperpolarization_depth. Electrophysiological data were fitted in models with morphology from L5_TTPC1_cADpyr232 Fig. 5(A) or a reconstructed prefrontal cortex L5 thick tufted pyramidal neuron deposited at NeuroMorpho.Org (Ascoli et al., 2007; Yin et al., 2018). The model parameters that were varied for the S1 model and PFC model are described in Supplemental Tables 4 and 5, respectively.

2.10. Support

A series of tutorial notebooks that walk users through various approaches are available on Github, with corresponding video walkthroughs available at: https://www.youtube.com/playlist?list=PL-Amxh_lBdw99alE5L1yfnfwPuK2wLeel These tutorials describe: 1) the structure of the documentation, 2) standards for input data and file structure to run NeuroGPU, 3) porting of models from NEURON to NeuroGPU, 4) Parameter space exploration, and 5) Using DEAP optimization. Future developments will be documented in the NeuroGPU Github portal.

3. Results

Our goal for NeuroGPU was to develop user-friendly software for fitting compartmental models to experimental data, with improved speed, using relatively low-cost hardware. Furthermore, we sought to make NeuroGPU cross-compatible with NEURON, to allow one to import models available on public databases, including ModelDB and the BBP portal. Toward that end, we utilized the same basic structure as NEURON, including the use of hoc and mod files that define all aspects of the compartmental model (Fig. S1). To increase simulation speed, we focused primarily on parallelizing the most computationally intensive aspects of NEURON simulations in GPU architecture. NEURON calculates the voltages of each segment of the model by solving a system of differential equations that describes current flow in each compartment. Within NEURON, this system of differential equations is represented within a tri-diagonal matrix (Hines, 1984). Typically, matrix elements for neighboring compartments are solved in serial, as current flow in one compartment is interdependent on flow in neighboring compartments. We and others have previously developed methods to solve this tri-diagonal matrix in parallel across GPUs, despite the interdependence of current flow across compartments (Fig. S2) (Hines, 1984; Hines et al., 2008; Ben-Shalom et al., 2013). At that time, the method was implemented only for classic Hodgkin-Huxley models with 3 parameters (gNa, gK, gLeak) (Ben-Shalom et al., 2013). Here, we extended this method to support a wider range of models, including most models available in ModelDB and the Blue Brain Project (BBP) repository. We implemented this in Python and created an iPython Graphical User Interface (GUI) for easy use.

3.1. NeuroGPU Implementation

Traditionally, neuronal simulations diffusion matrixes are computed serially, since calculating the voltage at each section depends on voltage in neighboring sections. To leverage the Single Instruction Multiple Data (SIMD) architecture of GPUs, one needs to instead solve the diffusion matrix in parallel. This challenge has slowed adoption of GPUs for neuronal simulation (Kumbhar et al., 2019); however, in 2013, we described an algorithm that uses a parallel lower upper (LU) decomposition to solve the diffusion matrix, based on the work of Stone (Stone, 1973; Ben-Shalom et al., 2013). Our implementation relied on the fact that separate neuronal branches can be calculated in parallel before calculating regions where they merge at branch points, thus parallelizing computations. Each neuron can therefore be calculated using 32 parallel threads, with each thread calculating the voltage for n/32 compartments where n is the total number of segments in the model. A thread first calculates the contribution of all the mechanisms and then updates the mechanisms states, similar to implementation in NEURON (Carnevale and Hines, 2006). Given this structure, NeuroGPU outperforms NEURON only when the GPU is fully utilized when multiple model instances are simulated in parallel (Fig. 2). In such cases, several blocks of 32 threads each simulate a different model instance. To comply with SIMD requirements, these multiple models must have the same morphology to allow threads in different blocks to execute the same instructions. This approach is advantageous for techniques like parameter exploration, where multiple model instances with different conductance parameters built on the same morphology are compared. In this design, memory is optimized, since all the models share constant memory that holds mechanism details, morphological details, and the indices to solve the diffusion matrix in parallel. Individual models hold copies of the matrix diagonal, which changes every time step, and the parameters and states of the mechanisms that vary between model instances. Importantly, many of the optimizations made here can be leveraged to address other questions that are appropriate for GPU-based computing, or could be optimized better in future iterations. Current optimizations and future applications are discussed in Supplemental Table 1.

3.2. NeuroGPU Performance

To evaluate how NeuroGPU performs relative to NEURON, we benchmarked it for speed and accuracy across a range of models and hardware configurations. We first compared NeuroGPU performance with a single GPU to NEURON implemented on a single CPU core. To benchmark speed, we evaluated computing time for multiple instances of the same model. NeuroGPU was primarily evaluated on recently developed models from the Blue Brain Project (BBP) portal (Hay et al., 2011; Markram et al., 2015; Ramaswamy et al., 2015), but was also benchmarked on models with reduced morphology or reduced numbers of voltage-gated channels or ligand-gated receptors to determine how each of these aspects affects performance (Figs. S3–4). We used BBP models to benchmark different versions of NEURON (NEURON, MPI-Neuron, CoreNeuron). We focused on two specific models: a layer 5 pyramidal neuron (Fig. 2, top row: BBP PC, see Methods for specific model) and a layer 5 chandelier interneuron (Fig. 2, bottom row: BBP CC). Initially, models were interrogated with a range of stimulus intensities to determine relative differences between NeuroGPU and NEURON (Fig. 2).

We first confirmed the quality of the simulations. Overall, NeuroGPU was able to replicate NEURON simulations with high fidelity; however, small voltage differences were observed between the two platforms in all models tested. These were most commonly observed when voltage was changing rapidly between time steps (Fig. 2B–C, G–H), and were due to small differences in timing that likely arise from different approaches to number rounding in GPUs vs CPU architecture (Whitehead, 2011).

NEURON computation time scales linearly with the number of simulations, and, for relatively small numbers of model instances (< 8), outperforms NeuroGPU (Fig. 2, D,E,I,J). By contrast, models implemented on GPUs (NeuroGPU and CoreNEURON) scale linearly only after saturating all of the GPUs streaming multiprocessors (Nvidia, 2018) (Fig. 2D, E, I, J). Similarly, runtime scales linearly with number of mechanisms (e.g., distinct conductances), provided such mechanisms can be stored in the GPU’s constant memory (see NeuroGPU Implementation in the Results section). For example, in the Mainen model, 37 mechanisms could be simulated on a V100 GPU. This limitation is not an issue for most models, which typically use fewer mechanisms, and may be less of a bottleneck in future hardware with a larger constant memory allocation. Similarly, MPI-NEURON simulations start to scale linearly after all MPI processors (32) are occupied. NeuroGPU is 4.3x slower than MPI-NEURON when simulating only one neuron, but 5x faster when simulating 2^14 neurons Fig. 2D. For NeuroGPU and CoreNeuron, processing times are practicaly constant for any simulation incorporating fewer than 128 model instances, and begin to outpace NEURON simulations when > 8 simulations are run simultaneously. When computing large numbers of model instances, NeuroGPU outpaces both parallel versions of NEURON: NeuroGPU is 1.5–3.3x faster than CoreNEURON for the PC model and 1.9–4.1x faster for the CC model. This difference is likely due to differences in how memory transfers are implemented in the two programs. In NeuronGPU, memory transfers require little overhead, due to its use of CUDA, whereas CoreNeuron has significant overhead for similar transfers (Supplemental Table 2) (Anon, 2019).

Relative gains in processing time were noted when 8–16,384 model instances were run simultaneously. These gains were dependent on hardware. For example, implementing NeuroGPU on an NVIDIA TitanXP and Tesla V100 GPU resulted in 1.8–3.8x faster runtime when using the Tesla V100 GPUs (Fig. S3F). It is worth noting that TitanXP hardware is relatively low cost (< $1099) and a very similar card (NVIDIA GTX-1080) can currently be purchased for less than $500. As such, significant improvements in processing speed can be obtained even with modestly priced hardware. Additional returns can be gained from GPU tethering, as CUDA has been recently updated to allow for memory sharing across GPUs. To evaluate this, we connected up to 4 Tesla V100 GPUs together and measured speedup on both BBP models displayed in Fig. 2. As expected, adding more GPUs increased the overall processing capacity, and we noted shifts in the number of model instances that could be handled simultaneously before reaching maximum GPU utilization (Fig. 2E and J). Furthermore, speedup was almost 3 orders of magnitude faster relative to NEURON.

3.3. Exploring neuronal model parameter space

Parameter values (e.g. ion channels distributions) are correlated in a non-linear manner. This may lead to situations where vastly different parameter combinations nevertheless produce similar voltage outputs, at least for a limited set of stimuli (Prinz et al., 2004). The diversity of these parameter sets can be limited by constraining the range over which a parameter can vary before initiating model optimization, thus leading to more biologically realistic sets of parameters. NeuroGPU may be ideal for parameter exploration within such ranges, as these types of simulations require one to repeatedly model the same morphology with small differences in constituent parameter values, a process that lends itself well to parallelization within GPUs. Indeed, we predict that relative speedups would be identical to situations considered above (Fig. 2) and depend simply on the number of parameter sets used. To provide an example of parameter space exploration, we examined neuronal output (i.e., number of action potentials) in the BBP PC model when co-varying the density of the axonal fast inactivating sodium channel and axonal slow-inactivating potassium channel over a range of 0–10 and 0–20 S/cm², respectively. Single traces from with different sodium and potassium conductances are shown in Fig. 3A and total spike output as function of these two channel densities is shown in Fig. 3B. As expected, increasing sodium conductance allowed models to generate more APs until sodium conductance was so high that models entered depolarization block. Similarly, reducing potassium conductance produced comparable results. Interestingly, certain combinations of sodium and potassium conductance concentrations produced bursting phenotypes characterized by high-frequency APs riding atop long-duration depolarizations. These presumably reflect parameter ranges that then interact with other ion channels in the model (e.g., Ca_V3 of HCN channels) that promote such burst dynamics.

3.4. Fitting models to surrogate and empirical data

With the ability to rapidly sample parameter space, NeuroGPU may be ideally suited to accelerate model fitting to data, where a key constraint is the time needed to exhaustively sample possible solutions. To test this, we implemented two different genetic optimization algorithms within NeuroGPU. Initially we integrated the DEAP (Distributed Evolutionary Algorithms in Python) package (Gagn, 2012) (Fig. 4). Genetic algorithm success lies in the balance between exploration of the whole parameter space and the exploitation of specific areas that seem promising. For this, large sample populations are ideal, as this allows for effective and broad parameter space exploration. NeuroGPU is more efficient when many instances are running in parallel, allowing for more effective application of genetic algorithms.

Genetic optimization was tested here by fitting model-generated voltages to a single voltage epoch containing APs that was generated by the default values present in the BBP PC model. We focused first on such surrogate data, as the ground truth values for all parameters are already known. As such, we can easily compare how well NeuroGPU performs in arriving at similar values. Optimization began with different population sizes comprised of 100–10,000 individual parameter sets with random initial values (Fig. 4B). These populations were run in four independent trials, each for 50 generations, and the difference between the naïve model and ground-truth model was compressed to a single score value (see Methods). For these scores, lower values indicate less difference between the two cases. Scores improved for each of these populations, but the variance across trials and the overall score were markedly affected by the population size, with score decreasing in a near-linear fashion with each doubling of population size (Fig. 4C). These score improvements were paralleled by a decrease in total processing time. For example, optimization with 10,000 individual parameter sets ran 7.7× faster on NeuroGPU than NEURON (Fig. 4D; 10 vs 77 h, respectively). While these are significant improvements in simulation speed, they are relatively modest compared to those observed in other conditions (Fig. 2), likely since the version of NeuroGPU used here required NEURON to load the simulation and generate parameter values. In the next section we present how eliminating calls to NEURON greatly increases speedup.

Given these promising results fitting surrogate data, we next asked whether similar performance could be noted for empirical electrophysiological data. Therefore, we fitted the BBP PC model to whole-cell current-clamp recordings of action potential activity from neurons in acute mouse frontal cortex slices (Fig. 5A black traces). Models were implemented on two different pyramidal cell morphologies from somatosensory (original BBP PC morphology; Fig. 5A) and prefrontal cortex [from NeuroMorpho.Org; Fig. 5B (Ascoli et al., 2007; Yin et al., 2018)] to determine whether morphology differentially affects optimization. Here we used the BluePyOpt package, which is an extension DEAP that is specified for neuronal optimizations (Van Geit et al., 2016). We fitted the model to eight voltage responses from varying stimulations to verify that the model is robust across stimulation conditions (Fig. 5A, B). The optimization algorithm found values for the model’s parameters that resulted with good fits to both morphologies. Thus, models can be fitted to empirical data and new morphologies by combining NeuroGPU with BluePyOpt.

Unlike single trace stimulations (Fig. 4), optimization to 8 separate traces required more generations to obtain reasonable fits, which unfortunately increased processing time dramatically. To reduce runtimes, we revised NeuroGPU by adding two final features. First, we eliminated NEURON calls to the CPU entirely by generating a procedure that identifies the free parameters of the model and modifies NeuroGPU’s input, ‘AllParams.csv’, to new values without using NEURON. Second, we used CPU multithreading scoring, which reduced the overall computation time of each generation. With these improvements, runtimes were reduced for each iteration of the optimization algorithm. Simulation of all 8 traces with 10,000 putative solutions required 60 s of processing time. This was followed by a 103 s period required for scoring for each genetic algorithm generation. This latter aspect has been accelerated dramatically. On 8 CPUs, the same process requires 2891 s per generation. These speedups (48.2 fold in simulation time) are close to the speedups of single simulations (Fig. 2D). With these improvements, we were able to obtain models that fit well to experimental data using an 8 GPU node running NeuroGPU in just 4 h, a task that traditionally may require several days of computation using a large cluster (Hay et al., 2011; Hill et al., 2011; Almog and Korngreen, 2014; Nandi et al., 2020; Schneider-Mizell et al., 2020). As such, NeuroGPU allow one the ability to develop more complex models and fit them to empirical data in reasonable time frames (Fig. 1).

4. Discussion

Detailed models of neurons are critical to our understanding of neuronal functioning. However, the computational resources required of current software implementations of complex neuronal models are prohibitive, typically requiring supercomputers. At best, this limits the accuracy of results; at worst, it limits access to all but a select set of scientists. To address this gap, based on our previous efforts (Ben-Shalom et al., 2013), we designed a user-friendly environment that enables one to port multi-compartmental models for implementation with CUDA to run simulations on GPUs. By taking advantage of parallel processing inherent to GPUs, we were able to accelerate simulations dramatically—up to 2–3 orders of magnitude with multiple GPUs. NeuroGPU was developed to be interoperable with NEURON, thereby allowing anyone with expertise in the NEURON environment access to GPU-based acceleration. Towards this goal, we developed a platform to easily port NEURON models from either ModelDB or the BBP portal (Ramaswamy et al., 2015; McDougal et al., 2017) using a iPython notebook-based graphical user interface (GUI). We further developed GUIs for creating stimulation protocols, parameter exploration, and genetic optimization.

Leveraging on our parallel algorithm for solving the tri-diagonal matrix representing the dendritic tree (Ben-Shalom et al., 2013), we developed an NMODL importer to CUDA. This allowed us to translate any single neuron multi-compartmental model developed in NEURON to NeuroGPU. We implemented NEURON’s framework in CUDA and translated NMODL mechanisms to kernels for GPU execution. NeuroGPU has two independent processes:: First, models are translated into GPU-executable code (Fig S1C). Second, simulations of many instances of this model are executed on GPUs. Importantly, these two processes can be done on different machines, helping facilitate NeuroGPU installation on GPU cluters, as only CUDA and pythons are required for execution on such clusters (and not NEURON or other software).

The power of GPUs to speedup application comes from the single instruction multiple data (SIMD) architecture where one instruction can modify many different memory instances in parallel. In 2013 we showed for the first time that single neurons can be simulated efficiently in parallel utilizing SIMD architecture (Ben-Shalom et al., 2013). We suggested a novel algorithm to solve the tri-diagonal (Hines) matrix (Hines, 1984), which is the core of neuronal simulations and usually is the main calculation bottleneck. Since that time, other solutions for solving Hines matrices have been suggested, such as using Exact Domain Decomposition method (EDD) (Vooturi et al., 2018) and splitting the dendritic tree and solving each subtree asynchronously (Magalhães et al., 2019). Also, a tailored solver for the Hines tridiagonal matrices has been developed that produced major speedups (Valero-Lara et al., 2018). However, it is not clear how these algorithms are to be used by the neuroscience community. Conversely, NeuroGPU is a full CUDA implementation that builds on our tri-diagonal solver to accelerate existing NEURON models. As discussed in Ben-Shalom et al., 2013, we use the Stone algorithm (Stone, 1973) for parallelizing the tri-diagonal solver. While this might not be the optimal solver, NeuroGPU still outperforms other GPU based neuronal simulations. Implementing other solvers using our CUDA implementation might lead to better performance, and will be tested in future versions of NeruoGPU.

NeuroGPU addresses a major gap in currently implemented GPU-based simulation environments. Two other neuronal simulation environments for multi-compartmental models have been implemented using GPUs, CoreNeuron (Kumbhar et al., 2019) and Arbor (Akar et al., 2019). These environments are designed primarily to accelerate large scale network simulations. NeuroGPU, by contrast, is focused on greatly accelerating the simulation of single neurons with complex, multi-compartment morphologies, critical for exploring the parameter space of single models and optimizing such models to best fit empirical data.

Leveraging the acceleration of single-neuron simulations, NeuroGPU has expanded GUIs for parameter exploration, which allows for rapid assessment of how changes in ion channel density across compartments affects neuronal excitability (Fig. 3). This approach may be particularly useful to generate testable hypotheses regarding channel distribution with pharmacological manipulations (Keren et al., 2009; Almog and Korngreen, 2014; Mäki-Marttunen et al., 2018), modulation of ion channels (Byczkowicz et al., 2019), or in disease states where ion channel density is thought to be affected (Migliore and Migliore, 2012; Miceli et al., 2013; Ben-Shalom et al., 2017; Spratt et al., 2019). Furthermore, one could generate a range of cells with variable channel densities and confirm that their activity is physiologically realistic (e.g., Fig. 3, all cases before generating depolarization block). These conditions could then be used as building blocks for variable activity within neuronal networks (Prinz et al., 2003, 2004; Alonso and Marder, 2019). In addition to parameter exploration, NeuroGPU is designed for extensive model optimization (Gagn, 2012; Van Geit et al., 2016). Fitting complex models to empirical data is computationally expensive, often requiring days of compute time, even on large supercomputing systems (Hay et al., 2011; Almog and Korngreen, 2014). Here, we show that NeuroGPU accelerated model fitting runtime 7.7 fold (Fig. 4). While appreciable, these accelerations can be improved further by bypassing model export via NEURON and instead using the “mapping parameters” function. In this configuration, runtime was accelerated 48.2 fold. Of note, this runtime does not match that found for simulating single neuron instances (Fig. 2), since score calculation using the CPU imposes a bottleneck. Using more sophisticated evolutionary algorithms that can minimize CPU latency or implementing the score function calculation on the GPU may result in further improvements, scaling towards GPU acceleration of 800 fold observed with multiple GPUs (e.g., Fig. 2). Nevertheless, the current instantiation of NeuroGPU offers individual labs the opportunity to implement optimization algorithms with their own hardware. Furthermore, it opens the door to extremely high-speed model fitting, as NeuroGPU can be run easily on GPU supercomputing systems, which inherently have exponentially more computational resources compared than similarly kitted CPU systems. Finally, NeuroGPU is ideal for generating neuronal datasets with different configurations for exploration of, e.g., firing rate ‘phenotypes (e.g., Prinz et al., 2004), or for deep learning training (Ben-Shalom et al., 2019; Gonçalves et al., 2020).

Recent advances in genetic characterization and novel analysis methods have resulted in characterization of diverse neuronal types with respect to their morphologies, projections, and protein expression (Gouwens et al., 2019). However, these advancements lack detailed biophysical models that can describe and simulate these neurons. In the BBP portal there are ~200 models that each is comprised of an m-type, which describes the morphology, and one of 11 e-types, which is a set of conductances that describes the electrical properties of the cell. Currently, these e-types describe somewhat generalized activity patterns [e.g., neurons that fire at high frequency without accommodation, or neurons that have stuttering firing patterns (Markram et al., 2015)]. With NeuroGPU we can easily expand this repertoire of e-types, and even replace generalizable e-types with models that recapitulate the activity of data obtained from single neurons. This would improve network simulations, as single neurons would display biologically accurate diversity within and across neuronal subtypes (e.g., Fig. 5B).

NeuroGPU accelerates compartmental modeling through parallelization of matrix calculations. Solving the tridiagonal matrix is the most computationally demanding aspect of multi-compartmental model simulations (Hines, 1984; Hines et al., 2008; Ben-Shalom et al., 2013). Therefore, we took advantage of fast, on-GPU memory and controlled the timing of calculations and memory transfers to optimize the use of computational resources (Volkov and Demmel, 2008; Ben-Shalom et al., 2013; Nvidia, 2018). Resulting speedups depended primarily on neuronal morphology and the implemented conductance complexity (Fig. 2D&I, Fig. S3 S4). Overall, GPU utilization was limited by execution dependencies (profiling data not shown), where one aspect of GPU processing could not proceed until another aspect either transferred or processed its own memory. In the future, these dependencies may be further reduced through either dynamic parallelization (Zhang et al., 2015) or by increasing instruction level parallelism (ILP) (Volkov and Demmel, 2008). Nevertheless, the current version of NeuroGPU accelerates single neuron compartmental simulations by several orders of magnitude.

4.1. Future goals for NeuroGPU

Future iterations of NeuroGPU may expand on the strengths and address limitations in using GPUs for compartmental modeling. Ion channels are modeled typically with Markov-based kinetics, or a simpler Markov approximation based on Hodgkin-Huxley type equations. NeuroGPU currently supports Hodgkin-Huxley-based mechanisms only, as we previously found that implementation of full Markov-based mechanisms on GPUs requires too much shared memory and reduces performance drastically (Ben-Shalom et al., 2012). Furthermore, NeuroGPU is currently limited in the mechanisms that can be translated easily, including passive conductances and voltage-gated ion channels, and does not at present support point-processes, including synapses. Nevertheless, the current iteration of NeuroGPU can support most neuron models on ModelDB (70% of such models utilized Hodgkin-Huxley mechanisms), and the majority of models from the BBP (Ramaswamy et al., 2015) and the Allen Brain institute (Gouwens et al., 2018).

As with total GPU utilization, improvements in memory handling may improve these cases. Furthermore, GPUs work best when the same instructions are occurring simultaneously on multiple memory addresses. This makes them ideal for iterating through models with identical morphologies and different channel distributions, but less ideal for network models containing a diversity of neuron types. As an intermediate, one could address this limitation by modeling networks containing discrete sets of neurons. For example, a network could contain several compartmental morphology models that each support multiple instances with different channel parameters, similar to the Ring model applied by Arbor (Akar et al., 2019; Kumbhar et al., 2019).

NeuroGPU will help democratize multi-compartmental modeling. While NeuroGPU can support simulations in large, tightly integrated systems using UNIX-based mutli-GPU architectures, it also is ideal for individual laboratories running simulations on Windows-based workstations with GPUs, which are becoming increasingly common. Indeed, a workstation with total costs < $3000, when kitted with appropriate GPUs, can out-perform loosely inter connected CPU-based clusters (i.e., with commodity inter-node connection hardware). This could help broaden the use and utility of multi-compartmental modeling by bringing supercomputer-level processing power to a large range of research settings.