P-sort: an open-source software for cerebellar neurophysiology

Abstract

Analysis of electrophysiological data from Purkinje cells (P-cells) of the cerebellum presents unique challenges to spike sorting. Complex spikes have waveforms that vary significantly from one event to the next, raising the problem of misidentification. Even when complex spikes are detected correctly, the simple spikes may belong to a different P-cell, raising the danger of misattribution. To address these identification and attribution problems, we wrote an open-source, semiautomated software called P-sort, and then tested it by analyzing data from P-cells recorded in three species: marmosets, macaques, and mice. Like other sorting software, P-sort relies on nonlinear dimensionality reduction to cluster spikes. However, it also uses the statistical relationship between simple and complex spikes to merge disparate clusters and split a single cluster. In comparison with expert manual curation, occasionally P-sort identified significantly more complex spikes, as well as prevented misattribution of clusters. Three existing automatic sorters performed less well, particularly for identification of complex spikes. To improve the development of analysis tools for the cerebellum, we provide labeled data for 313 recording sessions, as well as statistical characteristics of waveforms and firing patterns of P-cells in three species.

NEW & NOTEWORTHY Algorithms that perform spike sorting depend on waveforms to cluster spikes. However, a cerebellar Purkinje-cell produces two types of spikes; simple and complex spikes. A complex spike coincides with the suppression of generating simple spikes. Here, we recorded neurophysiological data from three species and developed a spike analysis software named P-sort that relies on this statistical property to improve both the detection and the attribution of simple and complex spikes in the cerebellum.

INTRODUCTION

Recording neuronal activity from the cerebellum presents both opportunities and challenges. The principal cells of the cerebellum, Purkinje cells (P-cells), can be identified based on their unique electrophysiological properties. Among cells in the cerebellum only P-cells produce simple and complex spikes (1). This makes it possible to use statistical methods to identify a P-cell: generation of a complex spike should be followed by a pause in the simple spikes (2, 3). However, detection of complex spikes is difficult because these spikes are not only rare, their waveforms change dramatically within 25 microns of the P-cell soma. Whereas in the dendritic field of a P-cell the complex spike is a slow, negative waveform, as the electrode approaches the soma the slow waveform becomes positive, develops a sharp initial negative spike, and is then followed by a variable number of spikelets that alter the waveform from one spike to the next (4–6). Thus, it is common to detect the simple spikes but not the complex spikes, or alternatively, detect the complex spikes but later realize that they are not followed with simple spike suppression and therefore do not belong to the same P-cell. To address these issues, we developed a spike analysis software that aids detection of simple and complex spikes, as well as quantifies whether the two events are generated by a single P-cell.

Unlike simple spikes, complex spikes tend to have their greatest power in the low-frequency range (30–800 Hz). As a result, a typical complex spike can produce a “broad spike” in the low-pass filtered representation of the data (local field potential; LFP). Indeed, two recent developments in complex spike detection are novel algorithms that depend partly on the LFP waveform (7, 8). Once the simple and complex spikes are labeled, the final step is to determine whether the simple spikes have been suppressed after a complex spike. If so, then one may conclude that the two kinds of spikes were generated by a single P-cell. However, in some data sets complex spikes do not have an LFP signature. Moreover, even if the complex spikes are detected, some or all of the simple spikes may belong to a different P-cell, or even a non-P-cell.

As a result, the problem is twofold: in the identification step, we need to label the simple and complex spikes, whereas in the attribution step, we need to determine which group of complex spikes was generated by the P-cell that produced a particular cluster of simple spikes. To consider these challenges, we formed a collaboration that included laboratories that focused on marmosets, macaques, and mice. Our software was developed using a database of over 300 P-cells recorded in three species.

The diversity of species and recording electrodes helped us identify some of the critical issues that are present in cerebellar electrophysiology. The presence of experts from the various laboratories provided a diversity of opinions, helping us verify the algorithms, as well as highlight their limitations. Here, we report the results of this effort.

P-sort is an open-source, Python-based software that runs on Windows, MacOS, and Linux platforms. To cluster waveforms and identify simple and complex spikes, P-sort uses both a linear dimensionality reduction algorithm and a recently developed nonlinear algorithm called UMAP (Uniform Manifold Approximation and Projection; 9). Importantly, it quantifies the probabilistic interaction between complex and simple spikes, providing an objective measure that can split a single cluster, or merge two different clusters, despite similarities or differences in their waveforms. Thus, P-sort helps the user go beyond waveforms to improve labeling of spikes.

However, a limitation of P-sort is that it requires user interaction. To encourage development of automated algorithms for the cerebellum and to allow others to sort the same data sets and compare their results with P-sort, here we provide a large database of labeled spikes from all three species. P-sort’s source code is available at https://github.com/esedaghatnejad/psort. The labeled data are available at https://doi.org/10.17605/osf.io/gjdm4. Instructional videos that introduce the user to P-sort are available at https://www.youtube.com/channel/UCSifaCZt1HmnZJ3440AOf8g/videos.

METHODS

Subjects

Neurophysiological data were collected from two marmosets (Callithrix Jacchus, male and female, 350–370 g, subjects M and R, 4 yr old), six rhesus macaques (Macaca mulatta; males; 5.0–7.4 kg, subjects B, F, K, P, S, and W), and ten mice (Mus musculus; male C57BL/6J mice at least 12 wk of age, subjects N082, N083, N086, N089, T029, T052, T083, T101, T122, and T124).

The marmosets were born and raised in a colony that Prof Xiaoqin Wang has maintained at the Johns Hopkins School of Medicine since 1996. The procedures on the marmosets were evaluated and approved by the Johns Hopkins University Animal Care and Use Committee in compliance with the guidelines of the United States National Institutes of Health.

The procedures on the macaques were performed in accordance with the Guide for the Care and Use of Laboratory Animals (2010) and exceeded the minimal requirements recommended by the Institute of Laboratory Animal Resources and the Association for Assessment and Accreditation of Laboratory Animal Care International. The procedures were approved by the local Animal Care and Use Committee at the University of Washington.

The procedures on the mice were approved by the Baylor College of Medicine Institutional Animal Care and Use Committee based on the guidelines of the US National Institutes of Health. The experimental mice were singly housed on a reverse light/dark cycle (8:00 lights-off to 20:00 lights-on).

Marmoset Data Acquisition

Following recovery from head after implantation surgery, the new world monkeys were trained to make saccades to visual targets and were rewarded with a mixture of applesauce and laboratory diet (10). Using MRI and CT imaging data for each animal, we designed an alignment system that defined trajectories from the burr hole to various locations in the cerebellar vermis, including points in lobule VI and VII. We used a piezoelectric, high precision microdrive (0.5 μm resolution) with an integrated absolute encoder (M3-LA-3.4–15 Linear smart stage, New Scale Technologies) to advance the electrode.

We recorded from the cerebellum using three types of electrodes: quartz insulated 4 fiber (tetrode) or 7 fiber (heptode) metal core (platinum/tungsten 95/05) electrodes (Thomas Recording), and 64 contact high-density silicon probes (Cambridge Neurotech). We connected each electrode to a 32- or 64-channel head stage amplifier and digitizer (Intan Technologies), and then connected the head stage to a communication system (RHD2000, Intan Technologies). Data were sampled at 30 kHz and band-pass filtered (2.5–7.6k Hz). We computed a common average reference signal (median of all simultaneously recorded channels, computed at 30 kHz) and subtracted this signal from all channels. We used OpenEphys (11), an open-source extracellular electrophysiology data acquisition software, for interfacing with the RHD2000 system and recording of signals.

Macaque Data Acquisition

The data were collected during previous studies (12–14). Following recovery from surgery, the monkeys were trained to make saccades to visual targets and were rewarded with applesauce. A recording chamber was implanted on the midline of the cerebellum (14.5 mm posterior of the interaural axis and directed straight down), providing access to the oculomotor vermis (lobule VI and VII). Single-unit activity was recorded with homemade tungsten electrodes with an iron-particle coating (100 kΩ impedance at 1 kHz). Neurophysiology data were sampled at 50 kHz by a Power 1401 digitizer (Cambridge Electronic Design, Cambridge, UK) and subsequently band-pass filtered (30–10k Hz). Data were displayed in real-time on a computer monitor running the data acquisition and analysis software Spike2 (Cambridge Electronic Design, UK) and saved for offline analysis (15).

Mice Data Acquisition

Single-unit extracellular recording was performed as previously described (16). In brief, a 2- to 3-mm-diameter craniotomy was opened over the right side of the cerebellum (6.5 mm posterior and 2.0 mm lateral from bregma) to access lobule V and the eyeblink microzone, and the dura was protected by a layer of Kwik-Sil (WPI). A custom 3-D printed recording chamber and interlocking lid (NeuroNexus) was secured over the craniotomy with dental acrylic to provide additional protection. After 5 days of recovery, the mouse was fixed in place on a treadmill via a previously implanted headplate, and Purkinje cell simple spikes and complex spikes were isolated using a tetrode (Thomas Recording, AN000968) acutely driven into the cerebellar cortex with microdrives mounted on a stereotactic frame (Narishige MMO-220A and SMM-100). The voltage signal was acquired at a 24,414-Hz sampling rate, and band-pass filtered between 100 Hz and 10 kHz (AP channel) and between 2 and 300 Hz (LFP channel) using an integrated Tucker-Davis Technologies and MATLAB system (TDT RZ5, medusa, RPVdsEx) running custom code (https://github.com/blinklab/neuroblinks). The data include Purkinje cells from a previously collected data set (17, 18).

P-Sort Main Window

To allow P-sort to run on Windows, MacOS, and Linux, the code was written using Python-based libraries (19–22). The GUI was written using PyQt5 (The Qt Company and Riverbank Computing Ltd.) and PyQtGraph to provide a fast and intuitive interaction for the user. To facilitate further development of P-sort by the user community, we used object-oriented coding. P-sort’s source code is available for download at https://github.com/esedaghatnejad/psort and instructional videos are provided at https://www.youtube.com/channel/UCSifaCZt1HmnZJ3440AOf8g/videos.

A process starts by loading the data and dividing it into one or more periods of time (called slots). The slot framework helps the user to account for potential drift and fluctuation in spike quality and shape over time. After sorting one slot, the parameters and spike waveform templates will be copied to the next slot to facilitate the sorting, but the user can further change the parameters independently in each slot.

The sorting process starts by filtering the signal into two channels, action potential (AP) and local field potential (LFP). The default is a fourth-order Butterworth filter with the 50–5,000 and 10–200 Hz range for AP and LFP channels, respectively. However, these parameters can be modified using the GUI to better fit the specific conditions of the data. The default assumption is that simple spikes generate negative peaks in AP channel and complex spikes generate positive peaks in the LFP channel. However, this assumption can be changed via the GUI. Once the respective peaks are detected, the next question is what should be the threshold to reject or accept a peak as being a potential spike. P-sort computes the histogram of the peaks and fits a Gaussian Mixture Model (GMM) with two basis functions to the histogram for each channel. The lower bound Gaussian is considered the noise and the upper bound Gaussian is the signal of interest. GMM assigns query data points to the multivariate normal components that maximize the component posterior probability, given the data:

$$p\left(\mathit{\theta }|\mathit{x}\right)={\displaystyle \sum }_{i=1}^K{\tilde{\varphi }}_i\mathcal{N}\left({\tilde{\mathit{\mu }}}_i,{\tilde{\mathbf{\Sigma }}}_i\right),$$ (1)

where K = 2, and ith basis function is characterized by a normal distribution with weight $\tilde{\varphi }$, mean $\tilde{\mu }$, and covariance matrix $\tilde{\Sigma }$.

The lower bound Gaussian is considered the noise and the upper bound Gaussian is the signal of interest. Based on this assumption, the intersection of the two fitted Gaussians is used as the default threshold to prune the detected peaks. However, the user is provided with a GUI to manually change the simple spike (SS) and complex spike (CS) thresholds, either by using the interactive plots or directly by assigning their values.

The next question is how to relate a peak in the LFP channel, which may potentially be a complex spike to its waveform in the AP channel. In a typical recording, a CS waveform consists of an initial sharp negative peak and a broad positive bump. The peak in LFP happens due to the broad positive bump but its timing is variable. Thus, using the LFP peak to align the CS waveform is unreliable. P-sort provides three different methods to find the onset of the complex spikes: SS index, SS template, and CS template. Initially, it uses the detected sharp negative peak (SS index) to align CS waveforms. This provides a set of putative CS spike waveforms to calculate a CS template (average of the waveforms). However, due to variability in CS waveforms, not all CSs express the initial sharp negative peak. To address this problem, after forming a CS template, the user can redetect CS and use CS template to align them which is not sensitive to the lack of initial sharp peak.

For the template alignment, P-sort uses the LFP peak time point as the starting point. However, as mentioned in the previous paragraph, this time point is variable from one spike to the next. To account for variability in LPF peak time point, P-sort moves the CS template signal (3.5 ms in duration) along a 5-ms time window around the LFP peak (4 ms before and 1 ms after the LFP peak) on the AP signal and selects the point of time that results in maximum correlation between the two signals. Correlation between two signals is computed as:

$$C_{TS}\left(t\right){|}_{t=-4ms}^{t=+1ms}={\displaystyle \sum }_{t\text{′}=0}^{+3.5ms}T\left(t\text{′}\right).S\left(t+t\text{′}\right),$$ (2)

where t is time point relative to LFP peak, T is the template signal, and S is the AP channel signal. Optimum time (t^*) is when the C_TS(t) is maximum.

For the recordings in which the LFP peak is later than 4 ms after the sharp negative peak, this default value should be adjusted using the “Preferences” interface. Alignment of the simple spikes relies on the timing of the sharp peak value of the waveform.

P-sort ensures that a candidate spike is labeled as either a simple spike, or a complex spike, but not both. Moreover, due to the refractory period in a spiking neuron, two arbitrary spikes cannot happen closer than 0.5 ms with respect to each other. Based on these constraints, P-sort addresses potential conflicts between CS-CS, CS-SS, and SS-SS candid events. The default values for each scenario can be changed using the Preferences interface.

After resolving the potential conflicts, P-sort provides sets of potential simple and complex spikes. For each set of spikes, P-sort represents spike waveforms, instant-firing-rate distribution, peak distribution, conditional probabilities, and feature scatter plots. Numerous features can be used for clustering of these data, including UMAP, principal components, timing of the spikes, relative time with respect to next or previous spike, similarity to templates, peak value, and instantaneous firing rate. Using the interactive plots, the user can select subset of the spikes based on the waveform plot or feature scatter plot and further prune the data or even change their label from simple to complex spikes and vice versa. As these clusters are manipulated, P-sort provides immediate feedback on their statistical features, thus allowing the user to determine whether the simple and complex spikes are likely to have been generated by a single P-cell, and whether the latest manipulations of the clusters improved the probabilities.

Overall, P-sort’s main window aims to provide a balance between the ability to visualize each spike waveform, and the ability to cluster the spikes and visualize their interactions. From this main window P-sort branches into two additional windows: the “Cluster Module,” and the “Dissect Module.”

Cluster Module

The Cluster Module gives the user the ability to use various algorithms to form groups of spike waveforms, assign labels to each waveform cluster (or part of a cluster), and immediately assess the statistical interactions between the labeled groups. Cluster Module includes manual and automatic labeling tools to label data sets based on their features or waveforms. This is the key feature of P-sort that allows the user to go beyond spike waveforms and rely on statistical interactions between spikes to merge seemingly different clusters, or split a single cluster into two.

P-sort relies on various algorithms such as UMAP, principal component analysis, etc., to extract features from spike waveforms and reduce their dimensionality. UMAP uses a manifold learning approach for dimensionality reduction. The algorithm performs the reduction in two steps. First a weighted neighbor graph is formed based on the local distances of the samples in high dimensional space. The weights represent the probability of the two samples being connected in high-dimensional space. Then an optimization step finds a low-dimensional layout of data that minimizes the cross entropy between the two representations. For a complete description of the UMAP algorithm, please see Ref. 9.

One of the important aspects of UMAP optimization is the trade-off between preserving the global structure or forming local clusters. In contrast, principal component analysis preserves the pairwise distances among all samples. UMAP uses two hyperparameters to tune the aforementioned trade-off, number of nearest neighbors (n_neighbors) and minimum distance (min_dist). During the first step of forming the high dimensional graph, n_neighbors was used to determine the local distance based on the nth nearest neighbor of each data point. Hence, smaller n_neighbors values lead to a more local view of the data and loss of the global structure. On the other hand, min_dist determines the density of the points in the low-dimensional space. This hyperparameter affects the density of the clusters with higher values leading to denser representations and lower values resulting in more spread points. We used default values of min_dist = 0.1 and n_neighbors = 15 for our application. For further information regarding the comparison of UMAP algorithm with other dimensionality reduction techniques please visit https://umap-learn.readthedocs.io/en/latest/performance.html#.

Once these features are computed, P-sort provides various algorithms to automatically cluster and label the spikes. The statistical interaction between the labeled spike groups is the key information that helps the user confirm the identity of the simple and complex spikes, and determine whether the two belong to a single P-cell.

The clustering algorithms used by P-sort include: 1) Gaussian Mixture Models (GMM), which requires the user to specify the number of clusters and their initial centroids; 2) Iso-split algorithm (23, 24), which is automatic and determines the number of clusters based on bimodality tests; and 3) HDBSCAN (Hierarchical Density-Based Spatial Clustering of Applications with Noise) algorithm (25), which is also automatic and requires no user inputs. We implemented a postprocessing layer for HDBSCAN’s outputs and restricted the number of clusters to less than 10. We did this by setting extra clusters with least number of members as noise (assigned as label -1). In addition to automatic clustering algorithms, an outlier detection method was implemented based on Local Outlier Factor density (21) which receives the quantile threshold as input. All these algorithms use the selected elements of the feature scatter plot by default; however, the user can perform multidimensional clustering by selecting the multiple features from the feature list in the GUI.

To use the cluster module, the user will select the spikes of interest either from the feature scatter plot or the waveform plot and assign a label to the selection. Cluster module provides four interactive subpanels for representing 1) color-coded feature scatter plot of spikes, 2) spike waveforms of each label, 3) peak histogram of each label including the firing rates information, and 4) cross and auto correlograms of chosen labels. Once the user has labeled a group of putative simple spikes, the module provides the probability Pr(S(t)|S(0)), thus allowing the user to visualize the refractory period of the chosen spikes. When the user labels a group of putative complex spikes, the module provides the probability Pr(S(t)|C(0)), thus allowing the user to visualize the suppression period. Then, the user can address the attribution problem of potential simple spikes with the candidate complex spike by checking the correlogram plots.

A unique challenge in cerebellar neurophysiology is finding the simple and complex spike clusters that belong to a single P-cell. It is possible that on certain recordings, one or more neurons will contribute to the signals that are recorded by a single contact. For example, it is possible that the large amplitude simple spikes are not produced by the P-cell that has produced the complex spikes in the recording. Rather, the smaller amplitude simple spikes should be paired with the complex spikes.

Dissect Module

P-sort dissect module is designed to provide more tools for reviewing individual spikes. In some scenarios, looking at the individual spikes and their surrounding events provides a better insight than the average features. Dissect module provide a tool set to move between spike events and look at each one over time. This module also provides the tool to manually overwrite a spike or change its alignment.

Comparison with Expert Curation

Fifty sessions (34 in the macaque and 16 in mouse) were sorted using Spike2 (Cambridge Electronic Design, UK) and P-sort by different experts. For Spike2 sorted data, we ensured that a given spike was not labeled as both a CS and a SS and removed the overlapping labels. Next, we compared P-sort data with Spike2 data by finding shared complex/simple spikes that happened in a 0.5/5.0-ms window of each other. We used the window of time to account for the variability in alignments of Spike2 data due to lack of template matching. If a given spike was not shared between the two data sets, it was labeled as an exclusive spike. To compare the number of exclusive spikes over data sets, we normalized the number of the exclusive spikes by the total number of the spikes in each data set.

Comparison with Automatic Sorters

We quantified performance of various automatic sorting algorithms on three different cerebellar data, ranging from easy to medium to hard. The hard data set was named the “P-sort challenge.” Each data set included around 15 min of recording from the marmoset oculomotor vermis and contained some of the difficulties that are present in cerebellar neurophysiology, including diverse patterns of spikes from neighboring cells. Thus, sorting of the data required isolating different spike types, as well as addressing the CS-SS attribution problems. The same 0.5/5.0-ms window was used to detect shared and exclusive spikes between automatic sorters and P-sort data.

For automatic sorters, we asked the authors of some of the algorithms to sort the data. P. Yger sorted the data using SpyKING CIRCUS (26), and A. Markanday sorted the data using a neural network (7). In addition, we used Mountainsort4 (23) and KiloSort2 (27) to sort the same data. We postprocessed the outputs by merging and associating simple spikes, complex spikes, and multiunit activity (MUA) based on rates and cross correlograms to arrive at the best match. For Mountainsort4, we used the default parameters and then the output units were manually merged and labelled as SS1, SS2, CS1, CS2, and MUA. Similarly, for Kilosort2, we used the default parameters and manually merged and labeled the outputs using the Phy2-interface correlograms and rates.

We quantified performance of each algorithm by computing a spike “agreement score” (28). This score measured the percentage of spikes that were agreed upon between pairs of algorithms. This involved computing the number of matched spikes (using a 1-ms time window), divided by the total number of spikes reported by each sorter, minus the number of matches. Using the same score we quantified reproducibility of P-sort by having five experts sort the same data sets that were sorted by the automated algorithms.

RESULTS

To illustrate the variety of challenges that we face in cerebellar neurophysiology, consider the data shown in Fig. 1. Here, the LFP channel (10–200 Hz) is plotted in red and the AP channel (50–5000 Hz) is plotted in black. Occasionally, one is lucky enough to isolate a P-cell that exhibits easily identifiable complex and simple spikes (Fig. 1A). In this example, LFP shows a large positive peak for the complex spike. To confirm that the complex and simple spikes originate from the same cell, we compute the conditional probabilities Pr(S(t)|C(0)), and Pr(S(t)|S(0)), over a domain of ±50 ms. The term Pr(S(t)|C(0)) indicates the probability that a simple spike occurred at time t, given that a complex spike was generated at time zero. The term Pr(S(t)|S(0)) is the probability of a simple spike at time t, given that another simple spike was generated at time zero. Spikes that originate from a single cell produce a refractory period. Thus, Pr(S(t)|S(0)) exhibits a near zero probability period centered at time zero. On the other hand, a complex spike is followed by a pause in the simple spikes. As a result, Pr(S(t)|C(0)) is asymmetric, with a long period of near zero simple spike probability following the time point zero. The presence of simple spike suppression following a complex spike, as shown in Fig. 1A (right), confirms that these two groups of spikes are generated by the same P-cell.

Figure 1.

Examples of challenges in cerebellar spike sorting. In the left column, local field potential (LFP) channel (10–200 Hz) is plotted in red, and the AP channel (50–5,000 Hz) is plotted in black. The middle column displays the conditional probability of a simple spike (SS) at time t, given that a complex spike (CS) occurred at time zero, labeled as Pr(S(t)|C(0)). Note the asymmetric suppression following a complex spike. The conditional probability Pr(S(t)|S(0)) indicates the probability of a simple spike at time t, given that another simple spike occurred at time zero. The right column includes individual complex spike traces, as well as the average trace. A: in this recording, complex spikes have a positive LFP peak. B: here, complex spikes cannot be identified from their LFP waveform as they lack an LFP signature. C: in this recording, complex spikes have a negative LFP peak. D: here, complex spikes coincided with the suppression of one group of simple spikes (SS1), but not a second group (SS2). The probability pattern (right column) suggests that SS2 is likely not a P-cell. E: in this recording, complex spikes do not coincide with suppression of the simple spikes. Thus, the two groups of spikes are not generated by the same P-cell. Bin size is 1 ms for the probability plots.

Figure 1B presents a more challenging example. Here, the complex spikes do not have an LFP signature. However, analysis of the AP channel using UMAP (9) identifies potential complex spikes. The identified events are genuine complex spikes, as evidenced by their spikelets (more examples from the same cell are shown in Fig. 2A), and the fact that the simple spikes pause after a complex spike (Fig. 1B, right).

Figure 2.

Additional challenges in identification of complex spikes (CS). A: complex spike waveforms vary because of the timing and number of spikelets. B: complex spike waveforms vary because of the temporal proximity of a simple spike (SS). Here, a simple spike before onset of a complex spike distorts the complex spike waveform. Top and middle rows: simple spike occurs at 3 or 0.2 ms before a complex spike. Bottom row: red traces are complex spikes with deformed waveforms.

If the electrode is near the soma, the complex spike waveform can exhibit spikelets that are similar to simple spikes (6, 29–33). For example, the complex spike in Fig. 1B exhibits large spikelets. These spikelets are difficult to dissociate from ordinary simple spikes, as evidenced by the fact that Pr(S(t)|C(0)) shows a small non-zero probability between 0 and +10 ms (right, Fig. 1B), during the period in which we would expect a near complete suppression.

Another example of the diversity of complex spike waveforms is shown in Fig. 1C. In this case, the complex spike exhibits a negative LFP peak, which may indicate that the electrode is in the dendritic field of the P-cell (5). Nevertheless, once the complex spikes are correctly identified, the simple spikes show a period of suppression.

Although detection of complex spikes may be challenging, a more crucial problem is attribution: sometimes the prominent groups of simple and complex spikes are generated by different P-cells. An example of this problem is shown in Fig. 1D. Here, the LFP signal easily provides for the detection of complex spikes. However, there are two groups of simple spikes, SS1 and SS2. The spikes labeled SS2 are the larger amplitude events. Notably, the complex spikes do not produce suppression of SS2 spikes. Rather, the smaller amplitude events SS1 are the appropriate simple spikes that pair with the complex spikes.

A different form of the attribution problem is shown in Fig. 1E. In this case, the simple spikes are easily identified, and the complex spikes have a negative LFP. Remarkably, despite the excellent isolation, the simple spikes are not suppressed following the complex spikes. Rather, the two groups of spikes in this recording are generated by two different P-cells.

Identification of complex spikes suffers from additional problems. There are variable number of spikelets in the complex spike waveform (6, 29, 30, 34, 35), and thus template matching may have difficulty labeling all complex spikes. Examples of the variable spikelets are shown in Fig. 2A. Paradoxically, when the electrode is near the P-cell’s soma, the signal strength is high but there is also a large impact of waveform variations caused by spikelets.

Finally, the complex spike waveform can be distorted because of the proximity of a simple spike, as shown in Fig. 2B (7, 36). This is because simple and complex spikes are driven by different inputs to the P-cell. Simple spikes are produced spontaneously, as well as due to excitatory inputs from granule cells. On the other hand, complex spikes are produced because of the input from the climbing fiber. As a result, a simple spike can be generated up to a fraction of a millisecond before a complex spike (middle plot, Fig. 2B). Here, the complex spike that follows the simple spike at short latency lacks the sharp component that initiated more typical complex spikes. As a result, the complex spike waveform is distorted by the proximity of the simple spike (lower plot, Fig. 2B).

In summary, identification of complex spikes is difficult because they can lack an LFP signature, their waveforms can be distorted because of nearby simple spikes, or their waveforms can incorporate variable number of spikelets. After the simple and complex spikes are identified, one still faces the problem of finding the simple spike cluster that belongs to the P-cell that generated the complex spikes. We wrote the open-source tool P-sort to help with these identification and attribution problems.

Clustering Waveforms

The diversity of complex spike waveforms suggests that it may be difficult to find a single mathematical technique that could generalize to all situations. For example, in some cases it is possible to identify the complex spikes from the LFP channel (Fig. 1A), whereas in other cases it is necessary to search the AP channel (Fig. 1B). In the case where two types of simple spikes are present, often the larger amplitude simple spikes is suppressed after the complex spikes (Fig. 1C), but occasionally the smaller amplitude spikes are the correct choice (Fig. 1D). P-sort provides tools for clustering as well as hypothesis testing. The tools include traditional dimensionality reduction methods such as principal component analysis (PCA), as well as novel algorithms such as UMAP. As the user identifies putative groups of simple and complex spikes, the software provides statistical feedback regarding the probability that the spikes are from the same P-cell.

P-sort works best with the raw, broad-spectrum recording such as the data generated by Open Ephys, an open-source data acquisition software (11). However, P-sort can also import data in which the LFP and AP channels are acquired separately. Upon loading the data, the user is provided with a GUI to specify how the data should be chunked into “slots.” Each slot is a region of data that is analyzed in turn, but once a slot is analyzed, other subsequent slots inherit features such as spike templates and threshold. In case of broad-spectrum data, the user can specify the filter properties for the LFP and AP channels (Fig. 3A, part 1). P-sort automatically selects a threshold for each channel (Fig. 3, part 2, see methods) and displays the statistics of the resulting simple and complex spikes (Fig. 3, parts 3 and 4).

Figure 3.

Main window of P-sort. 1: specification of filter properties for the action potential (AP) and local field potential (LFP) channels. 2: threshold selection and the distribution of voltage peaks for events detected in the LFP and AP channels. 3: identification of simple spikes (SS). The windows include spike waveforms aligned to peak voltage, distribution of instantaneous firing frequency, conditional probability Pr(S(t)|S(0)) demonstrating a suppression period, and clustering using UMAP. 4: identification of complex spikes (CS). The windows include spike waveforms aligned to a learned template (yellow), distribution of instantaneous firing frequency, conditional probability Pr(S(t)|C(0)) demonstrating suppression of simple spikes, and clustering using UMAP. The red traces indicate complex spikes that have been distorted because of nearby simple spikes. UMAP, Uniform Manifold Approximation and Projection algorithm (9).

In the example shown in Fig. 3, P-sort automatically selected thresholds for LFP and AP channels and identified putative simple and complex spikes. The UMAP space indicated a single simple spike cluster (Fig. 3, part 3), which was confirmed by Pr(S(t)|S(0)), illustrating that the simple spikes had around 10 ms suppression period (centered at zero). However, the UMAP space was not homogeneous for the complex spikes (Fig. 3, part 4), meaning that there was variability in the waveforms. Regardless, Pr(S(t)|C(0)) exhibited a suppression period. Thus, there was statistical confirmation that the simple spikes were well isolated, and were suppressed after the complex spikes.

Of course, in most cases the data are not as easily sorted as the case shown in Fig. 3. Another frequent case is one in which the complex spikes do not have an LFP signature (Fig. 4A), and thus one must search the AP channel. In the recording shown in Fig. 4C, the simple and complex spike waveforms happen to be quite similar. As a result, in the PCA space the data present a single cluster (Fig. 4B, left subplot). Thus, if we were to rely on PCA alone, we might conclude that only simple spikes are present.

Figure 4.

UMAP dissociates simple and complex spikes (SS and CS, respectively). A: in this recording, complex spikes do not exhibit an local field potential (LFP) signature (red trace). B: clustering of the spikes in principal component analysis (PCA) space does not produce a clear separation. However, the two groups of spikes separate in the UMAP space. The complex spikes identified by UMAP are shown in red in the PCA space. C: the waveforms and average traces for the complex spike and simple spike clusters as identified by UMAP. The conditional probabilities demonstrate that the complex spike events coincide with suppression of the simple spikes, suggesting that the two groups of spikes are likely generated by the same P-cell. Error bars are standard deviation. UMAP, Uniform Manifold Approximation and Projection algorithm (9).

P-sort relies on a dimensionality reduction algorithm called UMAP (9). UMAP is a nonlinear technique that, in our experience, is particularly powerful for clustering waveforms and identifying complex spikes, as also shown by the work of Markanday et al. (7). Unlike PCA which tries to preserve the pairwise distances among all samples, that is, global structure, manifold learning algorithms such as tSNE and UMAP rely on local distances. UMAP first constructs a high-dimensional weighted neighbor graph and then aims at finding its low-dimensional layout with similar structure. The trade-off between global and local structure in UMAP space can be tuned using the n_neighbors hyper-parameter by constraining the local radius of connectedness in high-dimensional graph based on the nth neighbor of each point (default value is n_neighbors = 15). This leads to better dissociation of different clusters with local similarities, especially in unbalanced cluster sizes scenarios. For example, discrimination of simple and complex spike clusters faces the unbalanced cluster size issue because of the intrinsic difference in rates of these two kinds of spikes.

Indeed, in the case of the data in Fig. 4, projecting the waveforms onto the UMAP space unmasks two clusters (Fig. 4B, right subplot). Using the graphical user interface (GUI) we select the smaller group of spikes and tentatively label them as complex spikes. Immediately, P-sort updates the probability Pr(S(t)|C(0)) window, as shown in Fig. 4C, illustrating that these putative complex spikes were followed by a pause in the simple spikes.

Finding Onset of Complex Spikes

One of the issues in identifying complex spikes is that the waveform can be significantly distorted by a preceding simple spike. Indeed, a P-cell can produce a complex spike at a fraction of a millisecond following a simple spike, as shown in Fig. 2B. The simple spike proximity distorts the complex spike waveform. If the distortion is small, template matching can identify the onset of the complex spikes (Fig. 3, parts 4). However, as we will see below, template matching can sometimes produce an incorrect alignment of complex spikes. P-sort can solve the problem by first aligning the complex spikes using a simple spike template, and then realigning using a complex spike template.

An example of this two-step process is shown in Fig. 5A. Initially, the putative complex spikes are aligned by P-sort based on a simple spike based template (see methods). This alignment results in two groups of complex spikes (shown by black and red traces in Fig. 5A). Next, the user selects the correctly aligned complex spikes, and defines a new template based on their mean waveform (Fig. 5A, black traces). Finally, the user realigns all complex spikes using this template (Fig. 5B). This improves the complex spike alignment, but the problem of aligning the deformed complex spikes persists. Indeed, the UMAP space indicates presence of two groups of complex spikes (red and black dots in Fig. 5B). However, because both complex spike groups suppress the simple spikes, they are likely a single cluster that need to be merged. To correct the error, P-sort provides the Dissect module. The Dissect module is a semimanual platform to relabel the onset of the distorted complex spikes (Fig. 5C), and correctly identify the simple spikes that precede them. Following this relabeling, the complex spikes are correctly aligned (Fig. 5D).

Figure 5.

Correcting for misalignment of complex spikes (CS). A: P-sort initially aligns complex spikes based on the sodium/potassium peak which resembles simple spikes (SS) (see methods). This results in misalignment of some complex spikes because these complex spikes do not express sodium/potassium peak. The user can delete the misaligned spikes (red) and compute a complex spike template (yellow). B: alignment of the spikes to the complex spike template does not solve the problem for all complex spikes: some of the deformed complex spikes remain misaligned. C: P-sort provides a Dissect module with which the user can change the spike alignment to compensate for the deformation. D: final complex spike alignment. Note the deformation in the complex spikes caused by the proximity of the simple spikes (red traces).

Identifying Complex Spike Spikelets

A source of variability in complex spike waveforms is the stochastic nature of spikelets (33). In our experience, the waveforms of spikelets are not easily dissociable from simple spikes. For example, the complex spike waveforms in Fig. 6A suggest a well-isolated P-cell, but the simple spikes that follow a complex spike have a probability distribution Pr(S(t)|C(0)) that exhibits a sharp peak in the putative suppression period (Fig. 6C, top row, red arrow). This raises the possibility that spikelets are being mislabeled as simple spikes.

Figure 6.

Identification of spikelets in the complex spike (CS) waveform. A: example of complex spike waveforms and potential spikelets. B: to help identify whether the spikelets are part of the complex spike waveform or are regular simple spikes, P-sort allows the user to visualize the temporal regularity of the spikes. In this graph, the x-axis is the timing of each simple spike with respect to the immediately preceding complex spike and the y-axis is the peak voltage of that simple spike. There appears to be two clusters of spikes. The cluster indicated by the red arrow has higher temporal precision and smaller peak voltage. C: before spikelet identification, the probability Pr(S(t)|C(0)) shows a sharp peak at around 5 ms following onset of the complex spike (red arrow, top subfigure). Removal of these spikelets produces the probability in the bottom row.

A heuristic that can help identify some of the spikelets is their temporal precision with respect to CS onset. To check for the presence of this regularity, P-sort provides a tool that produces a raster plot in which the simple spikes are aligned with respect to the onset of each complex spike (Fig. 6B). To produce this plot, in the simple spike section of the main window (or in the cluster module) the user selects the drop-down menu, selects t_prev_CS for the x-axis, and selects spike peak for the y-axis. This will present the simple spikes in a format where the x-axis is the time of each simple spike with respect to the onset of the immediately preceding complex spike, and the y-axis is the voltage of that simple spike at the peak of its waveform.

If spikelets are present, their timing is likely to be highly regular, producing a pattern of spike alignment in the 4–10 ms after CS onset (Fig. 6B). In this example, the spikelets can be dissociated from simple spikes both because of their timing regularity and their smaller peak voltages. The user can select the putative spikelets and view their waveforms in the main window. In our experience the waveforms tend to have a somewhat smaller amplitude but slightly wider shape than regular simple spikes. However, the main heuristic is the strong temporal alignment of spikelets following the onset of the complex spike.

Merging Clusters or Splitting Them Using Statistical Interactions between Simple and Complex Spikes

A unique challenge in cerebellar neurophysiology is finding the simple and complex spike clusters that belong to a single P-cell. P-sort provides cluster labeling and statistical tools to help with this problem. However, the main strength of P-sort is to go beyond waveform clustering and use the statistical relationship between complex and simple spikes to merge clusters, or even split a single cluster.

To illustrate this, let us begin with a particularly challenging example (Fig. 7A). Here, visual inspection of the waveforms suggests presence of multiple neurons. Indeed, UMAP clustering produces numerous groups of simple (Fig. 7B) and complex spikes (Fig. 7C). The task is to find the clusters that are signal and not noise, and more importantly, determine which simple spike cluster(s) can be attributed to which complex spike cluster(s).

Figure 7.

Finding clusters of simple (SS) and complex spikes (CS) that are likely generated by the same P-cell. A: this recording includes at least four groups of spikes: two that appear to be complex spikes and two that are simple spikes. B: UMAP clustering of the simple spike space. The two major clusters are SS1 and SS2. Their waveforms are shown by SS1-1 and SS2-1 in D. The smaller clusters are distorted spikes that are due to the temporal proximity of these major spikes, as well as other spikes, as shown in D. A smaller cluster of spikes are labeled as spikelets of complex spikes. C: UMAP clustering of the complex spike space. The two major clusters are CS1 and CS2. Their waveforms are labeled as CS1-1 and CS2-1 in E. Their waveforms can be distorted by simple spikes, as shown by CS1-2 and CS2-2. D: waveforms of various clusters labeled in the simple spike space. E: waveforms of the four clusters labeled in the complex spike space. F: suppression period of SS1 and SS2 is quantified by the probability Pr(SS1(t)|SS1(0)) and Pr(SS2(t)|SS2(0)). The probability Pr(SS1(t)|CS1(0)) quantifies the suppression following CS1. Thus, CS1 coincides with suppression of SS1 but not SS2. CS2 coincides with suppression of SS2 but not SS1. Bin size is 1 ms. UMAP, Uniform Manifold Approximation and Projection algorithm (9).

The reason for the numerous clusters in the simple spike UMAP space (Fig. 7B) is because the electrode is picking up signals from multiple neurons, and sometimes spikes from one neuron can distort the spike from another neuron. For example, the cluster labeled SS1-1 in Fig. 7B is due to simple spikes from neuron 1 (Fig. 7D, SS1-1). The nearby cluster SS1-2 in Fig. 7B is due to simple spikes from neuron 1 that are in close temporal proximity with a spike from another neuron (Fig. 7D, SS1-2). There are also clusters associated with spikes from neuron 2 that occur in isolation (Fig. 7D, SS2-1), or in close temporal proximity with a spike from another neuron (Fig. 7D, SS2-2). Sometimes, SS1 and SS2 co-occur, resulting in a larger spike, as labeled by cluster SS1&2-1. Finally, there are spikelets in the complex spike waveform that can be mis-labeled as simple spikes. The spikelets are noted in Fig. 7D.

There are four clusters in the complex spike UMAP space (Fig. 7C). One large cluster is associated with complex spikes labeled CS1 (Fig. 7E, CS1-1), while the other large cluster is associated with CS2 (Fig. 7E, CS2-1). Near each of these large clusters there are smaller clusters, reflecting the variability in the complex spike waveform. A source of variability in the complex spike waveform is presence of simple spikes from multiple P-cells. Thus, the complex spike will coincide with suppression of one group of simple spikes, but not all groups. As a result, CS1 can be distorted by arrival of simple spike labeled SS2, and CS2 can be distorted by arrival of simple spikes labeled SS1 (Fig. 7E, CS1-2 and CS2-1).

P-sort provides tools to label the clusters, examine their statistical properties, and determine whether clusters should be merged or not. For example, both the conditional probabilities and the statistics of the firing rates suggest that SS1 and SS2 are two different simple spikes, as shown in Fig. 7F, first row. Furthermore, the complex spike cluster CS1 coincided with the suppression of simple spike cluster SS1, but not SS2 (Fig. 7F, second row). Similarly, the complex spike cluster CS2 coincided with the suppression of simple spike cluster SS2, but not SS1 (Fig. 7F, third row). As a result, in this recording we have two distinct P-cells.

In a second example, let us show that the statistical interactions between simple and complex spikes can provide evidence suggesting that a single cluster of spikes may in fact be composed of two different cells. This data set is shown in Fig. 8. In this recording, the complex spike waveforms produce a single cluster in the UMAP space (Fig. 8A). However, only a part of this complex spike cluster is followed by suppression of the simple spikes. The subcluster that preceded the suppression of the simple spikes is labeled as CS1, and its waveform is shown in Fig. 8B Notably, the subcluster CS2 has a waveform that is similar to CS1, but CS2 does not induce suppression of the simple spikes, as illustrated in Fig. 8C.

Figure 8.

Splitting a complex spike (CS) cluster based on its statistical properties with simple spikes. A: in this data set, the UMAP space indicates a single complex spike cluster. However, statistical considerations raise the possibility of two subgroups, CS1 and CS2. B: the waveforms for the CS1 and CS2 subgroups are similar. C: the yellow regions are the conditional probabilities of simple spike suppression following a CS1 at time zero, or CS2 at time zero. Note that CS1 coincides with suppression of the simple spikes, but CS2 does not. Thus, although it is difficult to split the complex spike cluster into two groups based on their waveforms, they exhibit very different statistical properties. UMAP, Uniform Manifold Approximation and Projection algorithm (9).

This example highlights the possibility that on occasion, the complex spike waveforms may suggest presence of a single cluster, but a consideration of the statistical interactions can reveal that the simple spikes have been suppressed by only a subgroup within that cluster. This interaction between simple and complex spikes is a crucial feature that allows the user to go beyond waveforms and identify and attribute spikes of a P-cell.

In summary, P-sort provides clustering tools to identify simple and complex spikes. It relies on template matching to find onset of spikes, but also provides tools to correct instances where template matching can fail. A critical feature of P-sort is to provide tools for labeling and pairing of simple and complex spikes, thus allowing the user to visualize the likelihood that specific groups of spikes are generated by a single P-cell. This can lead to merging of nearby complex spike clusters because both groups are followed by suppression of the simple spikes, or splitting of a single cluster because only a subgroup is followed by suppression.

Comparison of P-Sort with Manual Curation (Spike2)

A common method currently employed for sorting of cerebellar data is via manual curation by an expert user via Spike2. To validate P-sort results, we analyzed data from mice and macaques and then compared them with those generated by the experts in the two laboratories.

Figure 9A presents results from an example data set from the macaque cerebellum. Here, the expert and P-sort agreed on roughly 99% of the simple spikes, but around 92% of the complex spikes (the numbers are the average of percent spikes identified by one method, say P-sort, and confirmed by another method, i.e., the expert). The median difference between the two methods in determining the timing of the spikes was 0.06 ± 0.01 ms (median ± median absolute deviation, MAD) for simple spikes, and -0.12 ± 0.05 ms for complex spikes (Wilcoxon signed-ranks test, SS: Z = 0.0077, P = 0.99, CS: Z = −0.052, P = 0.96). P-sort labeled 74 (0.29%) simple spikes that were not identified by the expert. These are labeled as P-sort exclusive simple spikes. The expert labeled 382 (1.48%) simple spikes that were not identified by P-sort. These are labeled as expert exclusive simple spikes. P-sort’s rate of simple spikes was 45.5 Hz whereas that of the expert was 46.0 Hz.

Figure 9.

Comparison of P-sort with expert curation on mice and macaque data sets. A: data from a macaque recording session. P-sort picked out 74 simple spikes (SS) that were not identified by the expert (0.29% of total), labeled as P-sort exclusive. Expert picked 382 simple spikes that were not identified by P-sort (1.48% of total), labeled as expert exclusive. Complex spikes (CS) that were exclusive to P-sort and the expert are also plotted. B: summary statistics on the mice (n = 16 sessions) and macaque (n = 34 sessions) data sets. Percentage of exclusive simple and complex spikes are plotted for the expert and P-sort. The central mark indicates median of the distribution, and the bottom and top edges of the box indicate the 25th and 75th percentiles. The thin line indicates the range of the data excluding the outliers. C: difference between P-sort and expert in terms of firing rate. Right columns show the likelihoods, normalized to the baseline simple spike probability in each session, averaged over all recording sessions for each species (bin size is 1 ms). Error bars are means ± SE.

For complex spikes, there were 111 (16.7%) events picked by P-sort that were not picked by the expert, producing a complex spikes rate of 1.19 Hz for P-sort versus 1.0 Hz for the expert. Thus, P-sort identified 19% more complex spikes than the expert. The waveforms suggest that the P-sort exclusive complex spikes are likely valid. However, the 8 complex spikes picked by the expert and not P-sort are also likely valid, as indicated by their waveforms. The reason why P-sort missed these complex spikes is because in some cases, a simple spike was in temporal proximity and distorted the complex spike waveform. Thus, in this data set there was general agreement between the two methods, although P-sort identified 16.7% more complex spikes than the expert.

Across 34 recording session in the macaque data set, a median of 2.53% of the complex spikes were detected exclusively by the expert, and 2.79% of the complex spikes were detected exclusively by P-sort. For simple spikes, 0.12% (median) of the spikes were detected only by the expert, as compared to 0.35% for P-sort. The two methods converged in their estimates of simple and complex spike firing rates (Fig. 9C, firing rate of P-sort spikes minus expert, Wilcoxon signed-ranks test, SS: Z = −0.23, P = 0.82, CS: Z = −0.31, P = 0.76). The resulting conditional probabilities of the data sorted by P-sort and the expert were indistinguishable. Some of the complex spikes missed by P-sort were due to significant distortions that were present in the waveform. In general, agreement between P-sort and expert were higher for better isolated recording data.

In the mice data set (16 sessions), a median of 4.10% of the complex spikes were detected exclusively by the expert, and 3.53% of the complex spikes were detected exclusively by P-sort, as shown in Fig. 9B. For simple spikes, the agreement between the two methods was better: on average 0.86% (median) of the simple spikes were detected only by the expert, as compared to 0.857% for P-sort. Overall, there were no significant differences between P-sort and the expert in terms of rates of simple and complex spikes (Fig. 9C, firing rate of P-sort spikes minus expert, Wilcoxon Signed-Ranks Test, SS: Z = 0.25, P = 0.90, CS: Z = 0.21, P = 0.84).

Importantly, in one recording session P-sort highlighted the possibility that the expert paired the wrong sub-group of complex spikes with the simple spikes. In this data set (Fig. 8), the expert labeled a single complex spike cluster, which is of course reasonable because of the similarity of the waveforms. However, using P-sort we were able to split this cluster into two, labeling them as CS1 and CS2. Only the CS1 sub-group were the complex spikes that were generated by the P-cell that also produced the simple spikes.

Overall, a comparison of P-sort with expert manual curation suggested a general agreement: the rates of simple and complex spikes were generally similar. However, for a few sessions P-sort was able to identify more complex spikes (Fig. 9A), and correctly label spikelets. In one instance, P-sort prevented pairing of the wrong subgroup of complex spikes with the simple spikes (Fig. 8).

Reproducibility of P-Sort Results, and Comparisons with Automated Algorithms

A major limitation of P-sort is that it is relies on the user to select tools and explore their efficiency in identifying and attributing spikes. This may produce substantial differences in the results that are produced by different users. In comparison, automated software can identify spikes with little or no user intervention. To quantify reproducibility of P-sort results, we compared the sorts produced by 5 experienced users on three data sets. To compare P-sort results with automated algorithms, we sorted the same three data sets with several automated algorithms.

In these comparisons we sorted data sets of various difficulty, ranging from easy to hard. In the easy data set (Fig. 10A), complex spike waveforms were readily dissociable from simple spikes. In the medium difficulty data set (Fig. 10B), complex spikes waveforms were more similar to simple spikes. Finally, in the more difficult data set (Fig. 7), two different pairs of simple and complex spikes were present on a single electrode. Because the automated algorithms did not attribute pairs of simple and complex spikes, we manually performed this step following conclusion of the automated spike sorting.

Figure 10.

Comparison of P-sort with automated spike sorting algorithms on two data sets. A: easy data set. The simple and complex spike (SS and CS, respectively) waveforms are illustrated in the first row. The conditional probability for simple spikes Pr(S(t)|S(0)) is plotted in blue. The conditional probability for simple spike suppression following a complex spike Pr(S(t)|C(0)) is plotted in yellow. All algorithms identified the simple and complex spikes. B: more difficult data set. First row shows the spikes identified by each algorithm. Kilosort did not identify the complex spikes. Second row shows the complex spikes missed by each algorithm, with respect to P-sort. Third row shows the spikes that were identified by each algorithm but not P-sort. Fourth row is the conditional probabilities for the labeled spikes. Error bars are standard deviation.

The lack of ground truth made it difficult to validate our results. Our recourse was to rely on the waveform of each spike type, and the statistical interaction between them. In addition, we compared the P-sort labeled complex spikes to the results of a neural network trained specifically to find them (7).

We quantified performances of P-sort users and algorithms by computing a spike “agreement score” (28). This score measured the percentage of spikes that were agreed upon between pairs of users, pairs of algorithms, and pairs of users and algorithms. Briefly, we computed the number of matched spikes (using a 1-ms time window), divided by the total number of spikes reported by each sorter, minus the number of matches.

In the easy data set, the agreement score between P-sort users was 1.0 ± 0.0 (means ± SE) for the simple spikes, and 0.992 ± 0.003 for the complex spikes, demonstrating near unanimous agreement (Fig. 11B, Psort-Psort score). The agreement score among the automated algorithms was 0.99 ± 0.004 for the simple spikes, but only 0.750 ± 0.066 for the complex spikes. The agreement between P-sort and the automated algorithms was 0.99 ± 0.002 for the simple spikes, but 0.87 ± 0.028 for the complex spikes.

Figure 11.

Performances of the automated algorithms in comparison with each other, and with P-sort. A: performance on the difficult data set (data in Fig. 6). The plots show conditional probabilities for the simple and complex spikes (SS and CS, respectively) identified by the automated algorithms and P-sort. Left column is the SS1 and CS1 relationships. Right column is the SS2 and CS2 relationship. For simple spikes, the suppression period is particularly poor for Kilosort2 for both SS1 and SS2. For complex spikes, SpyKING CIRCUS and Kilosort2 produce little or no suppression of simple spikes SS2. Bin size is 1 ms. B: spike agreement scores for 5 P-sort users and 3 automated algorithms, measured on the easy, medium, and difficult data sets. The spike agreement score was computed separately for simple and complex spikes (SS and CS, respectively), for each pair of P-sort users (Psort-Psort column, blue dots), for each pair of automated algorithms (red dots), and finally each pair of algorithm-Psort users (green dots).

Kilosort2 (27) found 10452 simple spikes and 146 complex spikes (first row, Fig. 10A). MountainSort4 (23) found 10460 simple spikes and 127 complex spikes. These results compared well with 10,461 simple spikes and 147 complex spikes identified by P-sort. SpyKING CIRCUS (26) found 10,488 simple spikes and 198 complex spikes in the easy data set (Fig. 10A). Some (43/198) of the complex spikes found by SpyKING were likely invalid, as suggested by the relatively poor suppression period of simple spikes that followed.

In the medium difficult data set the agreement score between P-sort users was 0.985 ± 0.004 for the simple spikes, and 0.945 ± 0.01 for the complex spikes (Fig. 11B). The agreement score among the automated algorithms was 0.874 ± 0.05 for the simple spikes, but only 0.127 ± 0.127 for the complex spikes. As a result, the agreement between P-sort and the automated algorithms was 0.927 ± 0.198 for the simple spikes, and 0.383 ± 0.081 for the complex spikes.

Kilosort2 was unable to identify any complex spikes (first row, Fig. 10B), resulting in missed spikes shown in the second row of Fig. 10B. It also mislabeled many complex spikes as simple spikes (third row of Fig. 10B). To examine the labeling differences more closely, we plotted the waveforms for spikes that were labeled by P-sort but not the automated algorithms (“missed,” Fig. 10B), as well as the waveforms for spikes that were labeled by the automated algorithms but not P-sort (“extra,” Fig. 10B). MountainSort4 found 70,328 simple spikes and 996 complex spikes. For simple spikes the labeling was essentially identical with P-sort, but for complex spikes the labeling differed by 178 events (17.2%). There were 36 P-sort identified complex spikes that were not labeled by MountainSort4 (second row, Fig. 10B), and some simple spikes that were labeled as complex spikes (third row, Fig. 10B). The labeling produced by P-sort resulted in a clean suppression period (Fig. 10B, bottom row), suggesting that in the medium difficulty data set, MountainSort4 mislabeled or missed around 15% of the complex spikes. The ability of MountainSort4 to find the complex spikes, but not Kilosort2, may be because MountainSort4 uses a PCA branching algorithm that is better in discrimination of local differences between waveforms in comparison to PCA. SpyKING found 70381 simple spike and 796 complex spikes, missing 493 complex spikes (second row, Fig. 10B) that were found by P-sort, and mislabeled some complex spikes as simple spikes (third row, Fig. 10B).

In the medium difficulty data set, the complex spikes that were labeled by P-sort but missed by the automated algorithms had waveforms that were consisted with complex spikes, suggesting that they were erroneously missed by the automated algorithms. In contrast, the “extra” simple spikes found by Kilosort (as compared to P-sort) were likely to be complex spikes (Fig. 10B, “extra”). The extra simple spikes found by MountainSort were likely from a different cell because they had a smaller amplitude, and the extra complex spikes found by MountainSort were likely mislabeled simple spikes. Finally, the complex spikes that were missed by SpyKING were likely valid, as suggested by their waveforms (Fig. 10B, “missed”). The simple spikes that were labeled by SpyKING but not P-sort were likely mislabeled complex spikes, and the complex spikes that were labeled by SpyKING but not P-sort were likely mislabeled simple spikes.

We also tested the algorithms on a difficult data set that had simple and complex spikes from two different P-cells on a single contact, labeled as SS1, SS2, CS1, and CS2 in Fig. 7. Sorting this type of data can benefit significantly from statistical interactions between spike clusters. In this difficult data set the agreement score between P-sort users was 0.989 ± 0.0175 for SS1, 0.970 ± 0.004 for CS1, 0.92 ± 0.009 for SS2, and 0.965 ± 0.003 for CS2 (Fig. 11B). The agreement score among the automated algorithms was much lower, ranging from a low of 0.361 for CS2, to a high of 0.765 for SS1. The agreement between P-sort and the automated algorithms was 0.853 ± 0.012 for SS1, 0.745 ± 0.06 for CS1, 0.696 ± 0.032 for SS2, and 0.612 ± 0.082 for CS2.

MountainSort4 found 48478 simple spikes SS1, and 51,471 simple spikes SS2 (Supplemental Fig. S1A). This agreed with 98.6% of SS1 spikes labeled by P-sort, but only 67.4% of SS2 spikes. MountainSort4 found 1156 complex spikes CS1 and 892 complex spikes CS2. This agreed with 88.9% of CS1 and 85.3% of CS2 spikes labeled by P-sort. The complex spikes that were exclusively labeled by MountainSort4 or P-sort are shown on the right panel of Supplemental Fig. S1A. Some MountainSort4 CS1 spikes were labeled as CS2 by P-sort. Similarly, some MountainSort4 CS2 spikes were labeled as CS1 by P-sort. The conditional probabilities (Fig. 11A) provide a method to compare these results. For SS1 spikes, the probability Pr(S(t)|S(0)) for P-sort (blue lines, Fig. 11A, first row) exhibited a cleaner suppression period than MountainSort4. For the complex spikes, the probability Pr(S(t)|C(0)) for P-sort for SS1 by CS1, and SS2 by CS2, both showed a cleaner suppression for P-sort. Thus, the main disagreements in the two approaches were regarding the smaller amplitude simple spikes SS2, and memberships of complex spikes in CS1 and CS2.

Kilosort2 found 56,538 simple spikes SS1, and 117,750 simple spikes SS2 (Supplemental Fig. S1B). The smaller magnitude spikes SS2 formed a much larger group in Kilosort2 as compared to both MountainSort4 and P-sort. For complex spikes, Kilosort2 found only 354 CS1, missing roughly 70% of the CS1 complex spikes labeled by P-sort. In contrast, it found 1,117 CS2 events, agreeing with 94.3% of CS2 events found by P-sort. Many of the CS2 complex spikes labeled by Kilosort2 were labeled as CS1 by P-sort (right column, Supplemental Fig. S1B). In contrast, Kilosort2 missed many CS1 complex spikes labeled by P-sort. The conditional probabilities (Fig. 11A, third row) suggest a poor suppression period for both SS1 and SS2 simple spikes labeled by Kilosort2. The Kilosort2 CS1 complex spikes demonstrate an excellent suppression period, suggesting they were real. However, Kilosort2 missed a larger number of complex spikes that were labeled by P-sort and exhibited good simple spike suppression (Fig. 11A, first row).

SpyKING CIRCUS produced SS labels that agreed somewhat better with P-sort than other automated software. For simple spikes, 99.3% of the SS1 labels and 93.1% of SS2 labels in P-sort were also labeled by SpyKING CIRCUS. An important difference, however, were the coincidence simple spike events, that is, SS1 and SS2 spikes that occurred simultaneously. P-sort labeled 1640 events as coincidence of SS1 and SS2, but SpyKING CIRCUS labeled all of these as SS1 events. Another notable disagreement was among the complex spikes. While SpyKING CIRCUS found nearly the same CS1 events as P-sort (95.4% agreement), the two approaches disagreed entirely regarding the CS2 events (2.1% agreement). As the traces in the right column of Supplemental Fig. 1 C illustrate, the CS2 complex spikes labeled by SpyKING are likely to be simple spikes. This conjecture appears to be confirmed by a lack of suppression in the probability traces in Fig. 11A, right column, fourth row.

Because the main difference between P-sort and the automated algorithms was regarding identification of complex spikes, we thought to further verify P-sort’s performance specifically on complex spikes. We did this by comparing P-sort with a recently developed neural network (7) that was specifically trained on complex spike waveforms. Here we found near unanimous agreement between the two approaches (Supplemental Fig. S1D). The neural network labeled 1178 CS1 and 885 CS2 events. This corresponded to 99.0% of the CS1 and 93.0% of the CS2 events labeled by P-sort. The few disagreements in the labeled events are shown on the right column of Supplemental Fig. S1D. In almost all cases, the complex spikes were preceded by a temporally adjacent simple spike, thus producing waveform distortions, making the labeling process particularly challenging.

In summary, we compared results of P-sort with automated algorithms in three data sets and found that the main difference was in the identification of complex spikes. For example, in the medium difficultly data set in which complex and simple spikes had similar waveforms, Kilosort2 missed the complex spikes. Conditional probabilities suggested that in both the easy and the difficult data sets, performance of MountainSort4 was close to P-sort, though it also missed around 15% of the complex spikes. In the medium difficulty data set, performances of MountainSort4 and SpyKING were close to P-sort. In all three data sets, P-sort results appeared highly reproducible, resulting in a lower bound of 92% spike agreement among the users in the difficult data set. As compared to the automated algorithms, P-sort produced cleaner simple spike suppression periods, as well as more robust complex spike periods of simple spike suppression. Finally, there was near unanimous agreement between P-sort and a neural network trained specifically to identify complex spikes.

A Database for Testing and Development of Algorithms

Automated algorithms can analyze tens or hundreds of simultaneously recorded electrodes, making them essential for high-density probes. Unfortunately, current automated algorithms may not perform ideally in the cerebellum, as illustrated by the data here, and documented elsewhere (37). While we have successfully used P-sort on data sets that include 4–7 electrodes, we have found it very time consuming to use P-sort when the data set is 64 electrodes or more. Thus, there remains a clear need to develop automated algorithms for the cerebellum.

Development of these algorithms relies on training and validation of on large data sets. Unfortunately, to our knowledge none are currently available. Thus, we recorded and analyzed a database of over 300 P-cells from the marmosets, macaques, and mice cerebella. We then used P-sort to label and attribute the simple and complex spikes in each recording. Recordings in the primates were from the vermis, lobules VI and VII. Recordings from the mice were from the eye blink region of lobule V.

We found that on average, simple spike rate in the marmoset (Fig. 12A) was somewhat higher than in the macaque [63.8 ± 1.29 Hz vs. 55.9 ± 2.45 Hz, means ± SE, independent samples t test, t(285) = 2.75, P = 0.006]. In contrast, complex spike rate in the marmoset was somewhat lower than in the macaque [0.88 ± 0.013 Hz vs. 0.98 ± 0.027 Hz, t(285) = −3.6678, P = 0.0003]. For the simple spikes, the conditional probability Pr(S(t)|S(0)) was very similar in the two primate species (Fig. 12C), suggesting that the simple spike suppression periods are similar.

Figure 12.

Statistical properties of simple and complex spikes (SS and CS, respectively) in three species. A: distribution of average firing rates. B: waveform of simple and complex spikes. Simple and complex spikes of each P-cell were both normalized by setting to −1 the negative peak of the simple spike waveform. Error bars are standard deviation. C: suppression period of simple spikes (blue, SS|SS) and the suppression coincided with complex spikes (red, SS|CS). SS|SS indicates the rate of simple spikes at time t when another simple spike occurs at time zero. SS|CS indicates the rate of simple spikes at time t when a complex spike occurs at time zero. Simple and complex spike rates for each P-cell were normalized with respect to average simple spike firing rate. Error bars are standard deviation. D: suppression period of simple spikes following arrival of a complex spike. Suppression period for each P-cell was defined as the duration of time after a complex spike that was required before the simple spike rate recovered 63% of its precomplex spike value. The red line indicates mean. E: interspike interval (ISI) distribution for simple (blue) and complex spikes (red). ISI data for each spike type in each cell was normalized so that the average ISI, defined as the inverse of the average firing rate, was equal to one. Error bars are standard deviation.

The complex spikes in the macaque were followed by a somewhat longer period of simple spike suppression (Fig. 12C). To measure the duration of simple spike suppression that followed a complex spike, for each P-cell we computed the time T when the probability Pr(S(t)|C(0)) increased beyond a threshold of 63% of the baseline before onset of the complex spike (Fig. 12D). In the marmoset, P-cells had an average of 14.8 ± 0.26 ms suppression period, significantly less than the suppression duration of 20.4 ± 0.89 ms we quantified in the macaque [t(285)= −8.0212, P = 3e-14].

These differences are difficult to interpret because spike rates and suppression durations depend on the precise location of the P-cell. For example, P-cells located in zebrin negative regions display higher frequency simple spikes than those located in neighboring zebrin positive stripes (38). Furthermore, the simple spike suppression following a complex spike is longer in zebrin positive zones. In both marmoset and macaque, a fraction of a millimeter difference in the recording site along the medial-lateral direction can change the zebrin band characteristics, particularly in lobule VII of the vermis (39). Thus, the differences in rates and suppression duration between species may be due to small differences in sites of recordings.

In the mouse data the complex spike rate was somewhat higher than the marmoset, 1.42 ± 0.07 Hz [t(254)= 12.0275, P = 1e-26], and also somewhat higher than the macaque [t(81)= 7.1277, P = 4e-10]. Furthermore, the complex spikes in the mouse were followed by a shorter suppression period (12.96 ± 0.80 ms) than in the macaque [t(81) = −5.2055, P = 1e-6] and in the marmoset [t(254) = −2.2462, P = 0.026].

Finally, in all three species we observed a consistent pattern in the distribution of interspike intervals (ISI). To compute the ISI distribution, we first measured the average ISI for each type of spike for each P-cell and then normalized the distribution by setting the average ISI to one. The resulting simple and complex spike normalized ISIs were different from each other, but nearly identical in the three species (Fig. 12E). These patterns may be useful priors that can aid identification of these spikes in an automated software.

In summary, we labeled simple and complex spikes in over 300 recordings in three species and quantified their statistical properties. The raw data as well as the P-sort labels are available at https://doi.org/10.17605/osf.io/gjdm4.

DISCUSSION

Spike sorting in the cerebellum can be a joy, something akin to a treasure hunt: finding the correct clusters of waveforms produces a satisfying statistical pattern in which the complex spikes are followed with suppression of the simple spikes. However, identifying spikes that belong to a P-cell can be difficult both for those who prefer manual curation and those who employ automatic algorithms. While most complex spikes leave an LFP signature because of their broad spike (8), some complex spikes can lack this characteristic (Fig. 1B), possibly because of the location of the electrode with respect to the soma (5). Moreover, complex spike waveforms can differ from one event to the next because of their temporal proximity to simple spikes (7), because of spikelets (6, 29, 30, 34), or because of intrusion of spikes from neighboring neurons. Even after the spikes are identified based on their waveforms, the simple spikes may belong to one P-cell, while the complex spikes may belong to another (Fig. 1E).

To help with the identification and attribution problems, we organized a set of clustering and labeling tools in a GUI based cross-platform analysis software called P-sort. Like other sorting software, P-sort clusters spikes based on their waveform properties. However, P-sort emphasizes the statistical relationship between complex and simple spikes: there is a suppression of simple spikes following a complex spike. Although there may be rare cases in which this property is not present, usually this information can be used to justify merging of disparate clusters, for example when spike waveforms are distorted (Fig. 7), or justify splitting of a single cluster, for example when spikes from two different cells have similar waveforms (Fig. 8). Thus, P-sort provides the user with the means to go beyond waveform-based clustering by providing statistical information regarding how spikes interact with each other.

Our development of P-sort was aided by a diverse collection of data from marmosets, macaques, and mice cerebellums. To validate the results, we compared P-sort’s results with manual curation performed by the experts in the mice and macaque laboratories. In addition, we compared P-sort with automatic algorithms MountainSort4 (23), Kilosort2 (27), and SpyKING CIRCUS (26), as well as a neural network trained to identify complex spikes (7). On the easy data sets, the performances of these automated algorithms were generally good, but on medium and high difficulty data sets, they occasionally missed large groups of complex spikes, or mis-labeled them as simple spikes. A comparison of P-sort selected complex spikes with a neural network trained to identify complex spikes produced near unanimous agreement.

P-sort follows the traditional approach in which features of the spike waveforms are identified and then clustered. It relies on a recently developed algorithm called UMAP (9), a nonlinear dimensionality reduction technique that, in our experience, is particularly powerful for clustering waveforms. UMAP is related to another non-linear dimensionality reduction algorithms called t-SNE (t-distributed stochastic neighborhood embedding) (40). Unlike t-SNE, UMAP returns an invertible transform onto which new data can be projected without having to recompute the map. This has the unique advantage of allowing for cross validation, which can be employed by semisupervised learning methods in which the expert labels a subset of the data and leaves it to UMAP to make predictions on the unlabeled data set. Indeed, UMAP was recently adopted by Markanday et al. (7) to cluster complex spikes from the cerebellum, and by Lee et al. (41) to cluster spikes from the cerebral cortex.

However, UMAP has certain disadvantages. In UMAP space, cluster sizes and distances between them do not contain information regarding the waveform structures. This issue can be resolved by viewing clusters in the PCA space, which is also provided in P-sort. Moreover, UMAP relies on a stochastic optimization process that produces non-deterministic outcomes of different runs. To help with this problem, P-sort supports a GPU implementation of UMAP that provides results for a typical 5-min recording session in around 1 or 2 s. This rapid response makes it possible for the user to evaluate the same data set multiple times, perhaps with different waveform window sizes. In addition, in most cases UMAP can reproduce the same number of clusters with similar topological relationships. Thus, although UMAP is not deterministic, it can extract clusters that are reproducible. Regardless, there remains a possibility of overfitting in noisy recording scenarios. To help with this, the Cluster Module depicts waveforms of each cluster, as well as their statistical relationship to other spike clusters.

Once the clusters are identified, P-sort provides automated tools to find cluster boundaries. One way to improve this step is to use Louvain clustering, i.e., finding community of spikes that are highly inter-connected in the UMAP space. This approach was recently demonstrated by Lee et al. (41) in sorting of cortical spikes.

Finding cluster boundaries, however, is not the only method to sorting, as illustrated by SpyKING CIRCUS (26). That work uses a template to define the centroid of each cluster, not their precise borders. In our tests, this approach produced good results on the large amplitude simple and complex spikes, but poor results on the smaller amplitude spikes, as illustrated by the conditional probabilities (Fig. 11).

The development of high-density electrodes highlights the need for automatic sorters. These sorters have addressed many issues including cluster matching between different channels, as well as multi-unit sorting. Moreover, their software features GUI-based visualization and manual curation toolboxes, thus allowing the user to postprocess the results. However, the automated approaches currently lack GUIs for identification of dependent spikes, that is, simple and complex spikes. In contrast, P-sort was designed to efficiently illustrate and interactively handle the simultaneous sorting and attribution of complex and simple spikes. The addition of features like multi-unit sorting opens the way for automatic sorters to be used as an initial starting point, followed by clustering and attribution by P-sort.

The rapidly evolving silicon probe technology makes it essential that we encourage development of automated sorters for the cerebellum. Thus, we used P-sort to label spikes in recordings made from over 300 P-cells in various species, and provide this labeled database to help software developers test and improve their algorithms. P-sort software is available at https://github.com/esedaghatnejad/psort. The labeled neurophysiological data are available at https://doi.org/10.17605/osf.io/gjdm4.

We hope that the availability of this data set can help the cerebellar community by facilitating the development of automatic algorithms, which rely on the availability of large data sets for training and validation. Furthermore, because the labeled data reflect the results of P-sort, investigators can try their current sorters and compare their results, thus uncovering potential limitations of P-sort. Finally, the labeled data allows the novice P-sort user to compare their sorting results with the expert user, thus serving as a teaching tool.

SUPPLEMENTAL DATA

Supplemental Fig. S1 (https://doi.org/10.17605/OSF.IO/GJDM4).

ENDNOTE

At the request of the authors, readers are herein alerted to the fact that additional materials related to this manuscript may be found at https://github.com/esedaghatnejad/psort, https://doi.org/10.17605/osf.io/gjdm4, and https://www.youtube.com/channel/UCSifaCZt1HmnZJ3440AOf8g/videos. These materials are not a part of this manuscript and have not undergone peer review by the American Physiological Society (APS). APS and the journal editors take no responsibility for these materials, for the website address, or for any links to or from it.

GRANTS

The work was supported by the National Science Foundation Grant CNS-1714623, the National Institutes of Health (NIH) Grants R01-EB028156, R01-NS078311, R01-EY028902, R01-EY023277, P51-OD010425, R34-NS118445, R01-MH093727, RF1-MH114269, and R01-NS112917, and the Office of Naval Research Grant N00014-15-1-2312.

DISCLOSURES

No conflicts of interest, financial or otherwise, are declared by the authors.

AUTHOR CONTRIBUTIONS

E.S.-N. and R.S. conceived and designed research; E.S.-N., J.P., P.H., Y.K., R.S., S.O., and R.S. performed experiments; E.S.-N., M.F., J.P., P.H., Y.K., R.S., S.O., and R.S. analyzed data; E.S.-N., M.F., J.P., P.H., Y.K., R.S., S.O., J.F.M., and R.S. interpreted results of experiments; E.S., M.F., and R.S. prepared figures; E.S.-N., M.F., and R.S. drafted manuscript; E.S.-N., M.F., J.P., P.H., Y.K., R.S., S.O., J.F.M., and R.S. edited and revised manuscript; E.S.-N., M.F., J.P., P.H., Y.K., R.S., S.O., J.F.M., and R.S. approved final version of manuscript.