Measuring the quality of neuronal identification in ensemble recordings

Abstract

Technological advances in electrode construction and digital signal processing now allow recording simultaneous extracellular action potential discharges from many single neurons, with the potential to revolutionize understanding of the neural codes for sensory, motor and cognitive variables. Such studies have revealed the importance of ensemble neural codes, encoding information in the dynamic relationships amongst the action potential spike trains of multiple single neurons. Although the success of this research depends on the accurate classification of extracellular action potentials to individual neurons, there are no widely used quantitative methods for assessing the quality of the classifications. Here we describe information theoretic measures of action potential waveform isolation applicable to any dataset, that have an intuitive, universal interpretation, and that are not dependent on the methods or choice of parameters for single unit isolation, and that have been validated using a dataset of simultaneous intra- and extracellular neuronal recordings from Sprague-Dawley rats.

Introduction

As we begin decoding spikes to understand brain function, it is critical to reliably identify distinct action potential trains from many neurons simultaneously (Wilson and McNaughton 1993; Csicsvari et al., 2003; Buzsaki 2004; Hampson et al., 2001; Harris et al., 2003; Hill 2011). “Spike sorting” utilizes features of multiple simultaneously-recorded action potential waveforms to divide signals into isolated categories, each corresponding to the discharge of an individual neuron.

Spike sorting remains difficult, perhaps more art than science. Currently, the quality of waveform isolation in most studies is unknown, a matter of trusting a particular laboratory or user. Identification uncertainty is problematic because conclusions about information processing mechanisms in the brain depend on the quality of action potential isolation (Quirk and Wilson 1999; Schmitzer-Torbert et al., 2005; Fenton et al., 2008). Lack of a consistent method to determine single-unit isolation quality impedes progress in understanding how coordination of neuronal discharge produces behaviors and cognition, or how failure of coordination produces pathology. A quantitative method could validate experiments in different laboratories and would be useful for developing waveform-isolation algorithms.

During spike sorting, features of extracellular waveforms, such as peak amplitude and energy, are extracted and placed in feature vectors. An expert user then uses subjective notions of spike-waveform isolation to create boundaries around clusters in 2- or 3-dimensional (2-D or 3-D) scatter plots of these high-dimensional feature vectors. Cluster separation quality varies with operator expertise, choice of features, and visual assessment of cluster boundaries (Harris et al., 2000; Wood et al., 2004).

Although several measures of isolation quality were developed, they have not gained widespread use, generally only working on particular types of data sets, or favoring particular isolation methods and parameters used by one research group. For example, Isolation-distance (IsoD) and LRatio are two related measures of isolation quality (Schmitzer-Torbert et al., 2005). Although definable using any waveform parameter set, to make comparisons across datasets, it was necessary to use identical parameters for waveform isolation in each dataset. Therefore, that study proposed a standardization using only two waveform features: energy and the first principle component coefficient of the energy normalized waveform. In our experience, those two features alone rarely provide good waveform isolation, a process that generally requires a large number of features.

We sought a measure that can be interpreted consistently across data sets using feature spaces that differ in both the identities and number of features. We present an information theoretic approach to quantify the quality of extracellular waveform isolation. We provide a quality measure that can be compared across recordings and laboratories, without constraint as to the type of electrode, the data acquisition method, or the particular set of parameters used for waveform isolation. By providing a method allowing each study to utilize whatever set of parameters provides optimal isolation, we expect these measures can begin to provide a standard for ensemble recording studies. These quality measures will also provide the quantitative basis needed to evaluate and compare the effectiveness of automated waveform-isolation methods extant or in development.

Materials and Methods

Electrophysiology

The data described here, that we used for this study, was previously recorded from male, Long-Evans rats, from Taconic Farms. The methods have been described in detail (Fenton et al., 2008; Park et al., 2011). All procedures received IACUC approval and were in accordance with institutional and NIH guidelines. Briefly, tetrodes (O'Keefe and Recce 1993; Gray et al., 1995) made by twisting 4 strands of 25-μm nichrome, were used to aid unitary waveform discrimination. For the recordings from freely-moving rats (male, Taconic Farms), eight tetrodes were loaded into a custom machined microdrive assembly (Bio-Signal Group Corp, Brooklyn). The assembly was surgically implanted under pentobarbital anesthesia (50 mg/kg) to position the tetrodes above the pyramidal cell layer in the dorsal hippocampus (centered at AP: 3.8 mm, L: 2.5 mm, DV: 2 mm relative to bregma; Paxinos and Watson 2007). The rats were allowed to recover for at least one week before the tetrodes were individually advanced in < 50 μm steps during the course of 1-2 weeks. The goal was to position as many of the eight tetrodes in the CA1, CA3, or dentate gyrus principal cell layer at the same time. A data set of tetrode recordings from the hippocampus and medial prefrontal cortex of urethane-anesthetized Long-Evans rats (male, Taconic Farms) was also studied. These recordings were made using the methods described in Olypher et al., 2006.

Extracellular potentials were buffered by unitary gain amplifiers plugged into the microdrive connector on the rat's head. The buffered signals were transmitted to main amplifiers along wires. Action potential (300-10000 Hz) band-pass filtered signals were amplified (5000-10000 times), digitized (32 or 48 kHz) time-stamped (100 μs resolution) and 2 ms tetrode waveforms were recorded using custom software (AcX, A.A. Fenton) or a commercial system (dacqUSB, Axona Ltd., St. Albans, U.K.). Whenever an action potential voltage exceeded a threshold voltage the voltages on all 4 wires of the tetrode were collected. The recorded action potential waveforms were aligned with the voltage at the threshold crossing set to the sample at 250 μs. This alignment produced a time-axis origin for measuring time-dependent waveform features, such as the waveform voltage at a particular time point relative to the threshold crossing at t = 250 μs.

Single-unit waveform isolation was done manually using custom software (WClust A.A. Fenton, S.A. Neymotin) that allows the user to define unitary waveform parameter clusters by drawing convex polygon boundaries in 2-dimensional (2-D) slices of waveform parameter space, as described in the Results. The waveform parameters included the positive and negative peak amplitudes on each tetrode wire, determined from a cubic spline (Press et al. 1992) of the digitized waveform; the principal components computed either from the waveform on each tetrode wire separately, or computed from the concatenation of the four tetrode waveforms; the waveform energy between a user-selected pair of time points (defined in units of voltage as $\sqrt{{\sum }_{i=n1}^{n2}{x}_i^2}/(n2-n1+1)$, where x_i is the waveform value of sample i and n1, n2 are the starting, ending sample indices respectively); the waveform voltage or slope at a user-selected time point; the waveform voltage on each tetrode wire at the time of the peak voltage on the largest waveform in the tetrode event.

Nearest-neighbor estimates of probability with KD-Trees

The nearest neighbor Kullback-Leibler divergence (KLD; Kullback and Leibler 1951) estimator (Wang et al., 2006) requires finding the nearest neighbors of arbitrary feature vectors in a high-dimensional space. To reduce the search time to find nearest neighbors, we stored the feature vectors in KD-Trees (Bentley 1975; Cormen et al., 2001), which are a data-structure allowing for 0(log N) time lookup of nearest neighbors, where N is the size of a distribution. Before storing the feature vectors in the KD-Trees, we normalized the values of each dimension into the range of [0-1] by the following formula: $v^{\prime }=\frac{v_i-{\mathit{\text{min}}}_i}{{\mathit{\text{max}}}_i-{\mathit{\text{min}}}_i}$ Here v_i is element i of vector ν and min_i, max_i are the minimum, maximum values of dimension i. This normalization allows each dimension to play an equal role in calculating nearest-neighbor distances, regardless of magnitudes of values in the dimensions. One KD-tree was created for each cluster's feature distribution.

Simultaneous intra- and extracellular neuronal recordings

The waveform isolation quality measures we developed were validated using publicly available simultaneous recordings of intra- and extracellular voltages from hippocampal area CA1 of Sprague-Dawley rats (Harris et al., 2000; Henze et al., 2000). Each recording had a single intracellular channel and four extracellular channels. This allowed the identity of spikes on the extracellular channels to be correctly classified as belonging to the intracellularly recorded neuron.

To extract action potential waveforms from the intracellular recordings, we calculated the discrete derivative and then set a threshold for spike crossing at a threshold above the average discrete derivative value. To extract putative spikes from the extracellular recordings, we first performed a zero-phase distortion band-pass filter between 500 Hz-3 kHz. Any voltage level from the filtered signal that crossed a threshold indicated a spike. The portion of the signal surrounding the threshold crossing (including 0.3 ms before and 1.8 ms after) was extracted from each channel and stored. Next, commonly used features, including peak, energy, and principal components, were extracted from the extracellular waveforms on the four channels and used to form feature vectors. The extracellular waveforms were considered to originate from the intracellular neuron if the intra- and extracellular peaks occurred within 0.4 ms (Harris et al., 2000).

To assess correlation of our quality measures with false positives and negatives, we performed 28 increasing levels of corruption to the original intracellularly-identified clusters (IC). For each recording, to simulate false-positives, we calculated the IC cluster's center and added the waveform feature vectors from the background distribution that encroached on its borders. We defined an IC cluster border as two standard deviations above the mean from the center. To simulate false-negatives, we excluded feature vectors that were known to belong to the IC from its outermost borders, as measured by distance to center.

Isolation distance and LRatio

Isolation distance (IsoD) and LRatio are measures developed to assess the quality of spike feature clusters (Schmitzer-Torbert et al., 2005). Although these measures can operate on arbitrary features we used an 8-Dimensional space in our standard implementation. This space consisted of Energy and the 1^st principal component of the Energy-normalized waveforms, 2 dimensions for each of the 4 micro-wires of a tetrode. Both measures use a noise distribution for each cluster consisting of all feature vectors not in the cluster. To compute IsoD, we compute mean and covariance matrix for each cluster feature distribution and then calculate the squared Mahalanobis distance (Mahalanobis 1936) for each feature vector in the noise distribution from the cluster's mean. These distances are sorted in increasing order. IsoD is the n^th largest squared distance, where n is the number of spikes in the cluster. IsoD tends to be larger for clusters that are well separated from the background.

For each cluster C, L is defined as $L\left(C\right)={\sum }_{i\notin C}1-\mathit{\text{CDF}}{\chi }_{df}^2\left(D_{i,C}^2\right)$, where i ∉ C is the set of noise spikes not in cluster C, CDF ${\chi }_{df}^2$ is the cumulative distribution function of the χ² distribution with 8 degrees of freedom (Bulmer 1979), and $D_{i,C}^2$ is the squared Mahalanobis distance of i to cluster C. Noise spikes close to the center of C contribute more to this sum than noise spikes far from the center. A low value of L indicates C is well-separated from the surrounding noise. LRatio is then defined as $\frac{L(C)}{n}$, where n is the number of spikes in cluster C. As shown, IsoD and LRatio are highly correlated (r=0.91, p<0.001) and thus redundant. The redundancy is a result of both being functions of the squared Mahalanobis distances of the noise spikes from the cluster center.

Software

The software for calculating the different measures of single unit isolation quality and for extraction of waveform features was written in standard C++ and is available for download as a package at the International Neuroinformatics Coordinating Facility (INCF) software center (http://software.incf.org/) and at ModelDB https://senselab.med.yale.edu/ModelDB/ShowModel.asp?model=141061). The software was tested and compiled independently on Linux OS and also as part of custom single unit isolation software for the Windows platform, WClust.

Results

Isolation information

Intuitively, the quality of a cluster is greatest when the cluster is well separated from the background, separated from all neighboring clusters, and compact. To capture this intuition, we propose Isolation information (IsoI), which computes a symmetrized Kullback-Leibler divergence (KLD; Kullback and Leibler 1951) between two multi-dimensional probability distributions of action potential waveform features. In what follows, we primarily refer to this as a cluster or feature cluster. We refer to it as probability distribution or distribution when discussing KLD as a general measure. KLD measures the separation between two probability distributions, P and Q, with probability densities p and q:

$$\mathit{\text{KLD}}(P,Q)=\int p(x)log\frac{p(x)}{q(x)}dx$$ (eq. 1)

, where the integral is taken over the space of waveform features. Keeping with the information theory standard, logarithms in this paper are base 2, unless otherwise stated.

We sought a measure that increased monotonically as the uncertainty of waveform parameters decreased. KLD had the disadvantage of not being symmetrical -- distance from P to Q is not the same as distance from Q to P. We therefore defined our new measure, isolation information (IsoI), to be the resistor average (Johnson and Sinanovic 2001) of KLD(P, Q) and KLD(Q, P) to provide symmetry:

$$\mathit{\text{IsoI}}(P,Q)=\frac{\mathit{\text{KLD}}(P,Q)·\mathit{\text{KLD}}(Q,P)}{\mathit{\text{KLD}}(P,Q)+\mathit{\text{KLD}}(Q,P)}$$ (eq. 2)

Graphing IsoI against the individual entropies of P, Q divided by the entropy of the joint distribution of P, Q demonstrated an additional advantage of this measure: it provided an approximately linear association with entropy reduction, upon classification of waveforms to their respective clusters (data not shown).

Given a data set of classified action potential waveforms, we calculated two different IsoI values: IsoI(C, BG) (IsoI_BG, cluster vs. background) and IsoI(C, NN) (IsoI_NN, cluster vs. nearest neighbor). IsoI(C, BG) computes a distance measure between the waveform feature probability distribution of the isolated action potentials of a cluster of interest (C) and the waveform feature probability distribution of all the other recorded events in the background (BG). This quantifies how well isolated the cluster's feature vectors are from all of the other feature vectors as well as measuring the cluster's compactness, because high probabilities in the distribution result from tight clustering in feature-space.

IsoI(C, NN) computes the distinction between a cluster's waveform feature probability distribution and the nearest neighbor distribution (NN):

$$\mathit{\text{IsoI}}(C,NN)=min(\mathit{\text{IsoI}}(C,B)|B\ne C,B\ne BG)$$ (eq. 3)

IsoI(C, NN) is useful because a cluster may be far from a large proportion of the other feature vectors and hence have a large IsoI(C, BG), but be close to another cluster. In such situations, the IsoI(C, NN) value will be reduced, and may thereby be used to more sensitively quantify the uniqueness of the cluster.

These two measures quantify the isolation quality in bits (units of information) across arbitrarily-chosen waveform features, and quantify judgments of cluster isolation quality in terms of separation and uniqueness.

Calculating KLD

With high-dimensional feature spaces, accurately determining probabilities becomes difficult when the size of the data set is limited due to data points being sparse in the space. In particular, 0-probability events are often terms when calculating eq. 1, causing the KLD to be undefined. We therefore computed the KLD between probability distributions P and Q using a nearest neighbor divergence estimator that has been shown to converge to the exact value of KLD with the increase of the sample size (Wang et al., 2006). To calculate the value using this estimator, we iterate over all of the elements in P. We then find the distance to their nearest neighbor in P, and the distance to their nearest neighbor in Q. The log of the ratio of these distances is then summed:

$$\mathit{\text{KLD}}(P,Q)=\frac{d}{\left|P\right|}{\sum }_{P_i\in P}log\frac{{min}_{\text{j}}(\mathit{\text{dist}}(Q_j\in Q,P_i))}{{min}_{\text{j}}(\mathit{\text{dist}}(P_j\in P|i\ne j,P_i))}+log\frac{\left|Q\right|}{\left|P\right|-1}\prime$$ (eq. 4)

, where d is the number of dimensions, |P| signifies the number of elements in distribution P, min_j(dist(Q_j ∈ Q, P_i)) is the distance from element P_i to the nearest element in Q, and min_j (dist(P_j ∈ P | i ≠ j, P_i)) is the distance of element P_i to its nearest neighbor in distribution P. Euclidean distance was used: $\mathit{\text{dist}}(\mathbf{x},\mathbf{y})=\sqrt{{\sum }_{i=1}^n{(x_i-y_i)}^2}$, with n the dimensionality of the vectors. Eq. 4 indicates that when the elements of distribution P are well clustered, their nearest neighbors within P will be closer than their nearest neighbors in Q. In this case, the ratio of logs will contribute a positive value. If the nearest neighbor distance to distribution P is larger than that of Q, the ratio will contribute a negative value. This occurs for poorly defined clusters that are not well isolated and for points near the boundary in the case of two adjoining distributions. In summary, KLD is a summation over all elements of the ratio of likelihoods that an element from the distribution is from that distribution and not from outside of it.

Dimensionality reduction

To compare IsoI values across different feature spaces of arbitrary dimensionality, we performed a dimensional reduction. For each cluster, C, we found the 8 features out of 20-40 typically used which gave the highest pairwise two-dimensional (2-D) IsoI_BG, F_c. These features are then used to compute the final 8-D IsoI_BG and 8-D IsoI_NN. Note that, for calculating the IsoI_NN of a particular cluster C, we use F_c when calculating each IsoI(C, X), where X is any other cluster than C. The automatically chosen features in F_c maximize cluster isolation and compactness. Although the selected features typically differ somewhat for each cluster, this technique allows estimation of isolation quality using the features best suited for isolating each individual single-unit.

Although computing pair-wise 2-D divergence for each cluster comes at the cost of increased computation time, it allows for selecting dimensions that maximize both isolation and compactness. Using 1-D divergence to select for optimal dimensions doesn't allow selecting for compactness because 1-D divergence will only be maximal when the separation between the two clusters is high. The choice of 8-D for the final analysis was a conservative choice after testing assessment at different dimensions. In any case, the final 8-D calculation was not computationally expensive compared to the pair-wise 2-D comparisons.

Sensitivity of Isolation Information measures to isolation errors

To measure the sensitivity of IsoI to classification error, we assessed various forms of cluster degradations, starting with high-quality clusters defined by an expert user and verified by our inspection. These clusters were compact and had a large separation from each other, as well as from the background, resulting in large IsoI values, ranging from 9 to 14 bits (Fig. 1). Cluster 1 (red) and Cluster 3 (green) have the highest separation from background, and from their nearest-neighboring clusters. This can be seen in the larger distance of C1 and C3 from the other clusters and background spikes in Fig. 1b in slices 1, 2, and 4, as well as 3 for C1. The other clusters, C2 (blue), C4 (purple), and C5 (light blue) are well isolated in some slices, but not as well in others, particularly in slices 3 and 4. Although sometimes only a single slice is used to determine separation, separability in more dimensions does add to cluster quality. As a result, the IsoI values of C2, C4, and C5 are lower than those for C1 and C3. Degradations were tested on various clusters both from this data set and others. Degradation effects (Fig. 2) were assessed on the clusters from Fig. 1 after alterations to C1. Most of the alterations were performed in a single 2-D subspace (Fig. 1b3), which displays voltages at specific times on two different microwires. This 2-D slice contained the C1 boundary that contributed the most to isolating it from the background, as evidenced by it contributing the highest 2-D IsoI_BG of any 2-D feature slice. This shows an advantage of this method, with implications for future development of automated systems. By allowing the use of arbitrary features, higher isolation quality may be sought by keying in on peculiarities in waveform shape that are revealed through visual or automated inspection of the most influential slices. As a suggestion for best practice, we typically used standard features consisting of waveform peak amplitude, energy, principal components, and voltage level or derivative at different times.

Figure 1.

a. Extracellular action potential waveforms (negative up) of 5 well-defined clusters, C1-C5. Gray waveforms (C0) are un-clustered spikes. The 4 signals recorded from each wire of the tetrode (T1-T4) are drawn next to each other, separated by vertical black lines. Waveforms were stored 250 μs before the upward threshold crossing and 1.75 ms afterwards. b. Different 2-D slices of the clusters in feature-space. T1-Peak is defined as the peak voltage on wire 1 (see methods). T1-V(0.181) is a user-defined feature defined as the voltage on wire 1 at time 181 μs of the stored waveform. T3-V(0.307) is the voltage on wire 3 at 307 μs. These features were selected by an expert user to optimize waveform isolation, as reflected in the well-separated clusters. Only the 5 best clusters found are highlighted here. Voltage in units of mV. c. Top – IsoI_BG of the 5 clusters shown in b. Bottom – IsoI_NN of the 5 clusters.

Figure 2.

Sensitivity of IsoI measures to various cluster degradations. Left side: degradations are a high level of random intermingling, lower level of intermingling, expansion, and contraction. Scatter-plots of the clusters in the same 2-D projections after the various degradations of C1 are shown in a-d. Voltage in units of mV. Right side: IsoI values before (circle) and after (triangle) degradation. Square in a,b shows IsoI value for the additional new cluster (color coded). Faint gray arrows pointing downward emphasize where decreases in IsoI values occur. a. Random 50% reassignment produces substantial decrease in IsoI values for the 2 derived clusters. Note that oscillatory behavior about the exact value, e.g., 0 for C1′s new IsoI_NN here, during convergence is a general property of the nearest neighbor divergence estimator¹⁶. b. Approximate bisection of cluster C1 within a 2-D feature slice also reduces IsoI values, despite the clear border between derived clusters. c. Expansion: C1 is brought closer to C2,C3, so IsoI values of C2,C3 also decrease. C3, nearest-neighbor to C1, shows large decrease in IsoI_NN. C4,C5 are also slightly degraded. d. Contraction primarily degrades C1 IsoI_BG due to poor isolation from points that are now part of background.

Figure 3.

Correlation of IsoI measures with error rates. a. False-positives (right) and false-negatives (left) were incrementally added to the known intracellularly-identified (IC) feature distribution and the corresponding relative IsoI values were calculated. The cluster displayed has 833 feature vectors. b. Correlation of relative IsoI_BG and IsoI_NN values with false-positive and false-negative error rates were found to be large and significant (p < 0.001). Peak IsoI value occurs at zero error rate. Error rates were defined by the percentage of points added to (false positives) and subtracted (false negatives) from the IC cluster. At high error rates variability of relative IsoI values increased, but IsoI values remained non-negative. c. Correlation of relative IsoD and LRatio values with false-positive and false-negative error rates were found to be significant (p < 0.001). Note that LRatio increases with increasing error rate, indicating a drop in single unit isolation quality.

In the first degradation, we considered a common scenario where features of two clusters are intermingled. While a user would always avoid the extreme case of near complete intermingling (Fig. 2a), users commonly define what appears to be a single cluster in feature space and subsequently, upon close inspection of the waveforms, find an additional feature with potential to partially split the cluster in two. In this scenario, a user would first “clone” the original cluster boundaries and then attempt to find new distinguishing features. To simulate this, we duplicated C1 boundaries, splitting the feature vectors into two C1 clusters shown in pink and yellow (C1′,C1″ respectively). The two derived clusters are now each other's nearest-neighbor. Additionally, each derived cluster now has the other one as part of its background. As a result, the IsoI_BG, IsoI_NN values of C1′ and C1″ dropped significantly (note that oscillatory behavior about the exact value, 0 for IsoI_NN here, during convergence is a general property of the nearest neighbor divergence estimator from Wang et al., 2006). IsoI_BG values for all the other clusters remained unchanged. The IsoI_NN values were subject to minor change, since nearest-neighboring clusters can be changed by modifying a single cluster. As a result, C3 and C5 have lower IsoI_NN values than the originals.

The next degradation was a less extreme example of intermingling. Here, we contaminated C1 by creating a new cluster within its boundaries (Fig. 2b). In this instance, two clusters C1′ and C1″ were close in feature space, but were still fairly well-separated from the background distribution. As a result, IsoI_NN is lowered more significantly than IsoI_BG for these two clusters, compared to C1. The degradation again slightly lowered the IsoI_NN values of clusters C3 and C5.

Another cause of overlap between clusters occurs when a user has difficulty determining precise demarcations, typically due to a high density of feature vectors in the area between two or more clusters. To replicate this situation, we expanded several of C1′s 2-D boundaries by randomly reassigning a proportion of the points falling within this expanded boundary from their original clusters to C1 (Fig. 2c). C1 now encroached into the territory of the other clusters. This degradation lowered IsoI_BG for cluster C1, since it was now closer to the background distribution. Other clusters, with some of their points removed, were closer to the background distribution, resulting in reduced IsoI_BG values, most evident for C2 and C3. In addition, the IsoI_NN, of C1, C2, and C3 were all significantly lowered, since their nearest-neighboring clusters became closer. Since C3′s nearest-neighboring cluster is C1, its IsoI_NN value was lowered the most. Importantly, this demonstrates that our measures are sensitive to non-local changes. Here, IsoI_BG and IsoI_NN both show lower values for C2 and C3 as well as for C1 since these are clusters who have “given up” feature vectors to C1. A different problem occurs when the density of feature vectors around a cluster is low, making it difficult to determine where to place the border of a cluster. In this case, the user may exclude feature vectors that should belong to the cluster. To replicate this, we contracted one of C1′s 2-D boundaries, excluding feature vectors that lie outside the new boundary (Fig. 2d). These excluded vectors were added to the background, resulting in a lower IsoI_BG value. Contraction also produced an insignificant increase in C1′s IsoI_NN value since C1 is now farther from its nearest-neighboring cluster, C3, as a result of the contraction.

Validation using simultaneous intra- and extracellular recordings

We used publicly available simultaneous intra- and extracellular neuronal recordings from hippocampal area CA1 (see Methods) to validate the IsoI measures (Fig. 3). We also used the same recordings and methods to compare the performance of two previously developed measures, IsoD and LRatio (see Methods). In each recording, all extracellular feature vectors belonging to the intracellularly-recorded neuron were set to belong to the IC cluster. The remaining feature vectors belonged to the background distribution (BG). To calculate correlation of IsoI_NN with error rates, a nearest neighboring cluster was required for each IC. To estimate this, we performed k-means clustering on the background distribution and selected the cluster with smallest IsoI_NN values as IC's nearest-neighboring cluster. For IsoI_NN, false-positive and false-negative corruptions were performed using these two clusters (IC and NN).

We correlated each cluster quality measure with false-positives and false-negatives using 28 levels of corruption to the IC clusters (Fig. 3b). Error rates were defined by the percentage of points added to (false positives) and subtracted (false negatives) from the IC cluster. As an example of this procedure on the simultaneous intra- and extracellular dataset for one corruption level, Figure 3a shows a cluster (center) containing 833 spikes, with a high IsoI_BG value of 5.2 bits, and high values of IsoD (250.5) and LRatio (0.00836). We performed false-negative corruptions by removing up to 70% of the cluster's feature vectors (Fig. 3a left), bringing the new cluster closer to the background distribution. As a result, the IsoI_BG value dropped drastically, to 3.8 bits and the IsoD and LRatio values decreased to 45.8 and 0.87, respectively. We performed false-positive corruptions of the original cluster (Fig. 3a right), by adding 583 feature vectors from the background distribution (70% of the original 833 feature vectors). As a result the IsoI_BG value dropped to 2.8 bits and the IsoD and LRatio values worsened, changing to 59.8 and 0.56, respectively. All measures had their highest quality values at zero corruption, and dropped off sharply with the corruption, illustrating that all the measures were able to objectively quantify cluster quality.

To calculate error rates for the measures, we performed corruptions on the four recordings. At baseline, the (IsoI_BG, IsoI_NN, IsoD, LRatio) values of the IC clusters from the four recordings had the following values: (3.6, 3.3, 18.8, 0.20); (4.2, 2.0, 25.8, 0.05); (3.1, 2.0, 20.7, 0.20); and (5.2, 4.7, 42.5, 0.04). The clusters ranged in size from 172 to 833 feature vectors, which allowed us to make significant reductions in cluster size, when performing false-negative corruptions. For all recordings, the peak IsoI_BG (Fig. 3b left) and IsoI_NN (Fig. 3b right) values occurred when there was no corruption to the IC clusters, verifying the usefulness of our measures as an objective quantification of neuronal isolation quality. Similar results were obtained for the IsoD and LRatio measures (Fig. 3c).

We defined relative cluster quality values, for each of the cluster quality measures, as the cluster quality value at the given error rate divided by the uncorrupted IC cluster quality value. We found that increasing levels of corruption tended to monotonically decrease the relative quality indicated by IsoI_BG and, IsoI_NN. Similar results were obtained with IsoD and LRatio.

We calculated the Pearson correlation between false positive and false negative error rates with the relative cluster quality values, for each of the measures. The overall correlation levels between each of the measures and error rates were high and statistically significant at the p < 0.001 level for all recordings. The IsoI_BG correlation levels with false negatives ranged from −0.71 to −0.97 and had an average of −0.87. We found that IsoI_BG correlation with false positives had similarly high values, ranging from −0.82 to −0.99 with an average of −0.90. IsoI_NN correlation with false negatives ranged from −0.65 to −0.91 with an average of −0.78 and the IsoI_NN correlation with false positives ranged from −0.49 to −0.97 with an average of −0.71. The strong correlation levels of both IsoI measures with increasing error rates confirms that IsoI may be used to objectively validate the quality of single neuronal identification from extracellular recordings. Similar results were found for both IsoD and LRatio, which had (min,max,average) correlations with false negatives of (0.98,1.0, 0.99) and (0.94,0.99, 0.96) for IsoD and LRatio, respectively, and (min,max,average) correlations with false positives of (0.63, 0.89, 0.77) and (0.80, 0.99, 0.89), respectively. These results indicate the utility of these measures in quantifying single unit isolation quality as well.

To determine whether the different measures were symmetrically sensitive to both types of errors, we calculated the skew of the relative values of each of the measures, defined as: $\sum \frac{{(r_{fp}-r_{fn})}^3}{N}$, where r_fp, r_fn represent the relative value of the measure at the given false positive and false negative corruption level, respectively, and N is the number of corruption levels (28). We found that IsoI_BG and IsoI_NN had average skews of −0.0143 and −0.0095, respectively, across the four recordings. These low skew values suggest nearly symmetric sensitivity to both types of errors and may be particularly useful where it is not known what type of error will occur. Although IsoD had a low average skew of −0.015, LRatio had a high average skew of −49.738.

Comparison of IsoI to other measures

We performed further comparisons of the IsoI measures with two published measures of cluster quality: isolation distance (IsoD) and LRatio (see Materials and Methods), on a dataset of 350 clusters from over 45 different recordings. In these recordings, the number of clusters obtained ranged from 2 to 15. Detailed analysis of the comparisons is provided below.

The Pearson correlation coefficient was computed between each pair of cluster quality measures, between the measures and the number of spikes in the cluster, and between the measures and the number of clusters that were isolated from the tetrode recording (Table 1). Before computing the correlation coefficients on IsoD and LRatio, we computed their natural logarithms first (Schmitzer-Torbert et al., 2005).

Table 1.

Pearson correlations between different measures and the number of spikes in the cluster, and the number of Clusters in the recording. Significant correlations (p<0.05, n=350, two-tailed test) are in bold.

	IsoIBG	IsoINN	ln(IsoD)	ln(LRatio)	Clusters
IsoIBG	•	▪	▪	▪	0.28
IsoINN	0.55	•	▪	▪	−0.11
ln(IsoD)	0.5	0.47	•	▪	0.16
ln(LRatio)	−0.57	−0.46	−0.91	•	−0.13
Spikes	−0.02	0.21	0.37	−0.26	•

There is a high correlation between ln(IsoD) and ln(LRatio), showing significant redundancy. ln(IsoD) and ln(LRatio) also have high correlations with the number of spikes in the cluster. This property of these measures is undesirable since principal cells in hippocampus, striatum, entorhinal cortex, and other brain regions have low firing frequencies, while interneurons have high firing rates. Low values of these measures may wrongly exclude such cells from analysis. IsoI_BG is not correlated with cluster size.

There is a statistically significant, but only small correlation between each of the IsoI_BG, IsoD, and LRatio measures with the number of clusters in the recording, suggesting these measure perform well across a wide range of recording qualities. The correlation between IsoI_NN and the number of clusters in a recording is not significant, indicating enhanced robustness of this measure to the different recording conditions.

Importantly, IsoI_NN is not as strongly correlated with IsoI_BG as LRatio is with IsoD, demonstrating that IsoI_NN and IsoI_BG measure different aspects of cluster quality. This was of course our intention when designing these as complementary measures.

Comparison of IsoI_BG with IsoD

The standard implementations of IsoD and LRatio operate in the 8-dimensional feature-space consisting of the spike waveform's energy and first principal component of the energy-normalized waveform on each micro-wire of a tetrode. Since IsoI has no such constraint, an operator that uses other features to isolate a single-unit waveform can more accurately evaluate the quality of isolation using IsoI than with the standard implementation of IsoD or LRatio. The correlation of IsoI_BG vs. ln(IsoD) is significant and positive (r=0.5, p<0.001, n=350, two-tailed test), and the correlation with IsoI_BG vs. ln(LRatio) is significant and negative, though outliers are seen (Fig. 4a). We examined clusters where the measures agreed/differed to see the strengths/weaknesses of the measures. In exemplar 2D slices of feature space, the examined cluster is shown in red and its nearest neighbor is shown in blue. Since IsoD and LRatio are highly correlated (r = −0.91, p<0.001, n=350, two-tailed test) we focus on the IsoI_BG and IsoD comparisons.

Figure 4.

Comparison of IsoI_BG with ln(IsoD), ln(LRatio) on all clusters in the 350 cluster dataset along with several case studies. a. Plot of IsoI_BG vs. ln(IsoD) and ln(LRatio). Larger, labeled points represent clusters discussed in text. These clusters are shown in parts b-e. In b-e, the red cluster is the cluster of interest (highlighted point from part a) and the blue cluster is its nearest neighbor. The gray points are the remaining spikes. The left column of b-e shows the slices of the feature space used by IsoI_BG, and the right column shows slices of the feature space used by IsoD. Voltage in units of mV. b. A well-isolated cluster according to both measures. c. A poorly-isolated cluster according to both measures. d. A well-isolated cluster according to IsoI_BG, poorly-isolated cluster according to IsoD/LRatio. e. Poorly-isolated cluster according to IsoI_BG, well-isolated cluster according to IsoD/LRatio.

The first cluster examined had high ratings in both IsoI_BG (11.99 bits) and IsoD (203.98) (Fig. 4b). This cluster has good separation from the background distribution using both sets of features. The next cluster had low ratings in both of these measures: an IsoI_BG of 5.3 bits and IsoD of 2.82 (Fig. 4c). The cluster is poorly isolated from the background, and dispersed in both sets of features.

The next cluster examined had a low IsoD of 8.64 and a high IsoI_BG of 13.12 (Fig. 4d). The operator selected the boundaries of this cluster using the voltage at 2 different times in the action potential waveform, which allow for good discrimination of this cluster. These features are not measured by IsoD, and as a result, less than optimal features are used in calculating this cluster's quality. In addition, this cluster comprises less than 0.001% of the spikes in the recording. This makes it difficult for IsoD to give this cluster a good score, since IsoD is proportional to the cluster size (see Methods). IsoI_BG does not share these problems. This case illustrates how relying on IsoD may exclude valid clusters that require non-standard parameters for isolation, or exclude low frequency firing single-units from analysis.

The next cluster examined (Fig. 4e) had a relatively low IsoI_BG of 5.28 bits and a relatively high IsoD of 47.37. Although this cluster is not so well-isolated from the background distribution, it comprises about 19% of the spikes in the recording. As a result, the IsoD score is inflated, since a large distance from the center of the cluster must be traversed in order to accumulate as many noise spikes as are in the cluster (see Materials and Methods). IsoI_BG has no such problem as reflected in the low score of 5.28 bits. This case again illustrates the problem of using a quality measure that depends on the number of spikes in the cluster.

Comparison of IsoI_NN with IsoD

IsoI_NN measures the separation between nearest neighboring clusters, and IsoD measures the separation of clusters from the background distribution. The correlation between IsoI_NN and IsoD is significant and positive (r=0.47, p<0.001, n=350, two-tailed test), and the correlation between IsoI_NN and LRatio is significant and negative (r=−0.46, p < 0.001, n=350, two-tailed test), with outliers visible (Fig. 5a). We examined the highlighted cases in Fig. 5a to see where the measures agree and disagree. Again, we focus on comparing IsoI_NN and IsoD.

Figure 5.

Comparison of IsoI_NN and ln(IsoD), ln(LRatio) on all clusters in the dataset along with several case studies. a. Plot of IsoI_NN vs. ln(IsoD) and ln(LRatio). Larger, labeled points represent clusters discussed in text. These clusters are shown in parts b-e. b-e. The red cluster is the cluster of interest, and the blue is its nearest-neighboring cluster. The gray points are the remaining unassigned spikes. The left column of b-e shows the slices of the feature space used by IsoI_NN, and the right column shows slices of the feature-space used by IsoD. Voltage in units of mV. b. A well-isolated cluster according to both measures. c. A poorly-isolated cluster according to both measures. d. A well-isolated cluster from its nearest-neighbor cluster, but poorly-isolated from background. This example illustrates that neither IsoD nor LRatio is sensitive to separation between clusters. e. A cluster with poor separation from its nearest-neighbor cluster but good cluster-background separation. The overlap between the red and blue clusters indicates they may need to be merged into a single cluster or discarded.

The two measures agree when a cluster is well separated from both the background and from its nearest neighboring cluster (Fig. 5b). This is shown in Fig. 5b, where the cluster has a high IsoI_NN of 14.52 bits and a high IsoD of 531.66. The two measures also agree when a cluster is poorly separated from both the background and from its nearest neighboring cluster. This is shown in Fig. 5c, where the cluster has an IsoI_NN of 5.04 bits, and IsoD of 3.29. We next examined a cluster with good separation from the nearest-neighboring cluster, with an IsoI_NN of 14.47 bits, but poor separation from the background, with an IsoD of 7.73 (Fig. 5d). These measures indicate that the cluster is probably not a single unit, and needs to be re-clustered. This demonstrates that IsoI_NN provides further information on the quality of a good cluster: only when it is already known that the cluster is isolated from the background distribution, should IsoI_NN be used, providing a stricter criterion of cluster quality than either IsoD or IsoI_BG alone.

The next cluster examined (Fig. 5e) has poor separation from the nearest-neighboring cluster, with an IsoI_NN of 1.34 bits, but good cluster-background separation, with an IsoD of 53.62. The former can be seen in the overlap between the red and blue clusters; the latter in the relatively large distance of the red and blue clusters to the remaining feature vectors. This example demonstrates that a non-unique cluster, C, is only separated from the background (high IsoD or IsoI_BG), but not from its nearest neighboring cluster (low IsoI_NN), NN. This may suggest that C and NN should be merged into a single, larger cluster, which will have better isolation and uniqueness. This example illustrates that IsoD is not sensitive to uniqueness, and that IsoI_NN can help point out problems of over-splitting waveform feature clusters.

Comparison of isolation quality across electrode configurations

Although the IsoI measures were designed to work on arbitrary feature spaces, different laboratories utilize electrode configurations that vary in the number of microwires composing each electrode and these differences can have a substantial impact on single unit isolation quality (Gray et al., 1995). We therefore utilized a dataset of 31 recording sessions from hippocampal CA1, consisting of 437 single units, to measure how IsoI values change as a function of the number of electrode microwires used.

Initial clustering was performed by an expert, utilizing features available from all four microwires of the tetrode. We then used the feature space derived from all four-microwires to compute the baseline IsoI values. To allow for comparison across these feature spaces, RelIsoI(N) was defined as the IsoI value computed when using the best 8 features available from N (1 or 2) microwires, divided by the baseline IsoI value. We performed these calculations both for IsoI_BG and IsoI_NN. As expected, our calculations confirm that using only 1 or 2 microwires degrades the isolation quality significantly (Table 2). It is noteworthy that the degradation in quality had a larger impact on IsoI_NN, which confirms that a particular benefit of using multiple microwire electrodes is in reducing the contamination that can often occur between two or more clusters of action potential waveforms that are of sufficiently large amplitude to be distinct from the background activity.

Table 2.

Average ± standard error of the mean of the RelIsoI values calculated using 1 or 2 microwires.

RelIsoI(1)BG	0.698 ± 0.183
RelIsoI(2)BG	0.550 ± 0.020
RelIsoI(1)NN	0.165 ± 0.014
RelIsoI(2)NN	0.430 ± 0.027

Comparison of isolation quality across brain areas

Firing properties, including rates and action potential features, as well as the density of neurons, can differ significantly across different brain areas. These differences can impact the ability of experimenters to accurately identify single units from waveform feature distributions. We therefore tested IsoI sensitivity to differences in the extracellular waveform characteristics of different brain areas across areas, utilizing in-vivo recordings datasets from awake rats from multiple areas, including hippocampal CA1, CA3, and dentate gyrus (Table 3, using data from Fenton et al., 2008; Park et al., 2011). The average IsoI values had similar ranges from ∼6-11 bits, across regions. IsoI_BG values ranged from 3.7 – 14.6, while IsoI_NN values had a larger range from ∼0.3 – 16.7 bits.

Table 3.

IsoI values from different hippocampal regions, recorded from awake, freely moving rats. Ave, SEM, Min, Max, and N denote average, standard error of the mean, minimum, maximum, and number of recordings used, respectively. DG indicates dentate gyrus. Note that the Park et al., 2011 data displayed are obtained only from place cells.

Park et al. (2011)					Fenton et al. (2008)
		IsoIBG	IsoINN		IsoIBG	IsoINN
CA1	Ave	6.76	8.67	Ave	7.90	7.16
	SEM	0.19	0.31	SEM	0.11	0.19
	Min	4.11	4.12	Min	3.66	0.34
	Max	11.86	16.46	Max	14.63	16.03
	N	167	167	N	268	268
CA3	Ave	7.14	8.39
	SEM	0.21	0.39
	Min	4.48	4.19
	Max	10.55	15.8
	N	79	79
DG	Ave	7.04	8.97
	SEM	0.14	0.37
	Min	4.01	4.07
	Max	9.45	16.71
	N	104	104

We examined a recording of 56005 spikes from CA3, in detail (Fig. 6a). Approximately 21% of the spikes recorded were considered noise (Fig. 6a left, gray points). The remaining spikes formed six clusters, corresponding to putative single cells. These clusters were characterized as follows (color, number of spikes, IsoI_BG, IsoI_NN): red, 2358, 7.7, 6.9; blue, 2095, 5.4, 3.0; green, 33213, 4.8, 4.7; purple, 1809, 7.1, 4.4; orange, 3616, 4.3, 3.0; maroon, 1274, 4.5, 4.7. Cluster 1 (red) showed the best separation from both background (IsoI_BG of 7.7 bits) and its nearest neighboring cluster (IsoI_NN of 6.9 bits). Cluster 4 (purple) also showed strong background separation (IsoI_BG 7.1 bits) and separation to its nearest neighboring cluster (IsoI_NN of 4.4 bits). Although cluster 6 (maroon) appears to overlap cluster 3 (green), it has both substantial background separation (IsoI_BG of 4.5 bits) and nearest neighbor separation (IsoI_NN of 4.7 bits), which is due to good separation in other planes of feature-space. The other clusters had fairly high IsoI_BG values, all larger than 4 bits, while their IsoI_NN were usually only slightly less. Although firing activity of CA3 neurons can be fairly sparse, these substantial IsoI values demonstrate that with a sufficiently large dataset, it is possible to accurately identify single units from this brain region.

Figure 6.

Comparison of single-unit isolation quality in CA3 (a) and medial prefrontal cortex (mPFC, b). Left) Scatter-plots of feature vectors from six different clusters. Features displayed are microwire voltages normalized to be between 0 and 1. Gray points represent feature vectors in the noise distribution. Note that T1-V(0.407) is a user-defined feature defined as the voltage on wire 1 at time 407 μs of the stored waveform. Center and right panels display bar-plots of the IsoI values, color-coded to match the clusters in the scatterplots.

We evaluated IsoI from recordings made in medial prefrontal cortex (mPFC) and hippocampal CA1 from rats under urethane anesthesia (Table 4, data from manuscript under preparation). Both of these areas showed similar average IsoI_BG values of 7.6±0.3 and 7.7±0.3 for CA1 and mPFC, respectively. The IsoI_NN values were also similar, with averages of 10.4±0.4 for CA1 and with a slightly higher value of 11.3±0.6 for mPFC. Minimum and maximum values were similar to those from the awake, freely moving rats.

Table 4.

IsoI values from recordings from rats under urethane anesthesia. Ave, SEM, Min, Max, and N denote average, standard error of the mean, minimum, maximum, and number of recordings used, respectively. mPFC denotes medial prefrontal cortex.

		IsoIBG	IsoINN
CA1	Ave	7.64	10.41
	SEM	0.26	0.40
	Min	4.34	4.06
	Max	13.61	20.69
	N	67	67
mPFC	Ave	7.69	11.32
	SEM	0.27	0.59
	Min	4.44	4.2
	Max	15.18	25.05
	N	58	58

Next, we examined a recording of 283469 spikes recorded from mPFC, in detail (Fig. 6b). Approximately 95% of the spikes recorded were considered noise, due to low amplitude spikes on several channels (Fig. 6b left, gray points). The remaining ∼12,000 spikes were isolated into six clusters. These clusters were characterized as follows (color, number of spikes, IsoI_BG, IsoI_NN): red, 1449, 9.0, 15.2; blue, 504, 10.2, 14.5; green, 617, 9.3, 14.3; purple, 4370, 6.0, 9.5; orange, 3326, 7.4, 12.9; maroon, 1872, 5.6, 8.1. All of the clusters showed high separation from both background and their nearest neighboring clusters. Although each of the six clusters had high IsoI_BG values, ranging from 5.6 – 10.2 bits, each cluster had a higher IsoI_NN value, ranging from 8.1 – 15.2 bits. The higher IsoI_NN values found in these recordings were partially due to the higher level of noise present in the recording (95% of the spikes), as compared to a hippocampal recording. However, since the IsoI_BG values were typically in the same range as those from hippocampal recordings, the single unit isolation from background should be considered sufficiently high. The fact that this is so demonstrates that the IsoI measures are useful for quantifying single unit isolation in hippocampal as well as neocortical structures.

The minimum IsoI values across the datasets described was typically near ∼3 bits. We have adopted a best practice of rejecting clusters where the IsoI values are less than 4 bits. The similarity of IsoI values from these different brain regions demonstrates the utility of the methods in quantifying single unit isolation quality in different recording conditions.

Comparison of computation time

We utilized a dataset of over 400 single units from 31 different recordings from hippocampal CA1 to estimate computational costs for the different isolation quality measures. For IsoI calculations, we started with 12 feature vectors from the action potential Energy, Peak, and first principal component (PC1) on each of the four microwires. The feature selection stage of, which iterates over all $\frac{n(n-1)}{2}$ pairs of 2D slices, for each cluster, to find the best features for the final 8D IsoI calculation, was the largest bottleneck of the IsoI calculation. This stage had a computational cost ranging from 1.24 – 35.37 seconds, with an average of 7.28 ± 0.28 seconds (± standard error of the mean, N=440) per cluster. IsoI_BG had the next largest computational cost, ranging from 0.25 – 22.5 seconds and averaging 2.84 ± 0.13 seconds per cluster. IsoI_NN had a slightly lower computational cost, ranging from 0.01 – 27.85 seconds and averaging 1.43 ± 0.13 seconds per cluster. This lower cost for IsoI_NN was due to the smaller number of comparisons between feature vectors belonging to two distinct clusters, which are less than the large number of comparisons needed to compare feature vectors belonging to a cluster and its complement, as is done for IsoI_BG.

IsoD and LRatio both operate in an 8-dimensional space, consisting of Energy and the first principal component of the action potential waveform on each tetrode wire, and do not utilize feature selection. Their lower computational costs are partly a result of these differences. IsoD and LRatio had the following values for minimum – maximum, and average ± standard error of the mean (in milliseconds, per cluster): (5.6 – 46.4, 18.3 ± 0.5) and (10.0 – 82.2, 32.8 ± 0.8), respectively. Although the IsoI measures had a higher computational cost, judicious use of KD-Trees by caching nearest neighbor values may help reduce the cost of the calculations. In addition, parallelizing the 2D search across multiple threads would allow for additional time-savings. Nonetheless, use of IsoI amounts to spending approximately a minute extra to compute estimates of single unit isolation quality. This is an insignificantly short amount of time with enormous value, given that it can take an expert hours to select the waveform boundaries.

Discussion

We have developed and validated an information-theoretic measure for determining single-unit isolation quality in a standardized way that generalizes across brain regions and electrode configurations. The method, IsoI, calculates a symmetrized KLD between two sets of waveform-feature probability distributions in two complementary ways: between a cluster and the background (IsoI_BG), and between a cluster and the nearest-neighboring cluster (IsoI_NN). These two measures respectively quantify a cluster's isolation and uniqueness. We selected the features maximizing 2D IsoI_BG, on a per-cluster basis, to calculate a final IsoI. This approach permitted comparison of IsoI values across different feature-spaces. This is important for comparing quality across different users, who will choose different features given the same data set, and also for comparisons across laboratories. It is also important, as demonstrated, that the measures permit comparisons of single-unit isolation quality across recordings from different brain regions. In our data sets, single-unit isolation quality did not differ for recordings from the different hippocampal subfields, nor did isolation quality differ if recordings were made from mPFC or under anesthesia. This does not imply that it is just as easy to isolate single units in each of these areas and conditions, but rather that similar isolation quality standards were applied to the different data sets by the manual operators. Indeed, we have observed a reduction of isolation quality that was captured by a significant decrease of the IsoI measures in conditions when a larger proportion of the recordable cells were actively discharging (Fenton et al., 2008; Park et al., 2011). The IsoI measures were also lower for recordings that only utilized waveform information from one or two electrodes rather than from all four electrodes of a tetrode, which simply quantified the fact that multi-wire electrode configurations provide more information and power for single unit isolation. There is substantial merit in the use of a single-unit isolation metric that has general applicability (Hill et al., 2011) and we suggest that the IsoI measures described here are well suited to quantify the quality of single-unit isolation, regardless of where the data were recorded from, and regardless of how the units were recorded and isolated.

The isolation measures were applied to a database of clusters, and the values matched expert judgments of isolation quality. We simulated common clustering errors and demonstrated that IsoI reasonably quantified the degree of degradation whether changes were made locally (only a single cluster perturbed) or globally.

To objectively validate our IsoI measures, we used publicly available, simultaneous intra- and extracellular neuronal recordings to demonstrate that reductions in both of our IsoI measures correlated strongly with false-positive and false-negative error rates. The strong correlation levels indicate that our IsoI measures may be used to objectively validate the quality of single neuronal identification from extracellular recordings.

Our new measures generally correlated well with older measures (IsoD and LRatio), but showed additional significant advantages. One advantage is that IsoI is intuitive via its standard information theoretic scaling, providing a consistent interpretation even when varying features are used for isolation. Another advantage is insensitivity to cell firing rate, particularly important when both fast-firing interneurons and slower-firing principal cells are recorded. An additional advantage occurs with “over-splitting”, a common problem in single-unit isolation. Over-splitting occurs when waveforms from a single-unit are inaccurately split into two or more cluster. This occurs, for example, when higher amplitude spikes precede low- amplitude spikes in a complex-spike burst (Quirk and Wilson 1999). IsoI_NN directly detects over-splitting by identifying clusters that are excessively entangled with their nearest neighbor. Here, as in all cases, IsoI_BG remains a necessary complement, since IsoI_NN is only useful to quantify uniqueness once minimal isolation quality has been established with IsoI_BG. IsoI_NN can of course only be used when there are at least two single-units to compare.

Certifying students

Isolating single units from multiple extracellular electrodes is a time-sink for many neuroscience laboratories (Buzsaki 2004). It can take ∼2-4 hours for an expert to manually cluster a 10-minute ensemble recording. IsoI can speed up this process by identifying particular clusters that are not well defined. Such clusters can then be eliminated from consideration or examined using alternative feature sets. Since IsoI is agnostic to feature choice, it can be used as an aid in the choice of the most discriminative features that end up producing the best clusters. Spike clustering is even more frustrating for the student, who must also wonder when to stop refining clusters. Using the IsoI measures, the novice user can make several different clustering attempts and identify the best outcome. The measures also provide a laboratory with a criterion to use both while training new researchers in this skill, and subsequently certifying the student to perform these assessments independently.

Certifying projects

Manual spike clustering includes use of subjective judgments of cluster quality: different clusters look better or worse to different researchers depending on their levels of expertise (Wood et al., 2004). As a result, incorrect conclusions may be drawn where clustering quality is inconsistent, or lower than required for a particular investigation (Harris et al., 2000). For example, accurate characterization of the ms-scale temporal interactions amongst two or more neurons may require greater isolation quality than estimating the stability of the receptive fields (Quirk and Wilson 1999). A standardized measure will permit data quality to be assessed during paper review and after publication. The generality of the IsoI measures, allowing comparison across different choices of feature sets, is a key advantage in its use and potential adaptation by neurophysiologists using different electrode designs and amplifier settings, and looking at different species and different brain location, factors which require the use of different features to provide optimal discrimination.

Usefulness for long-term recordings

Until now, widespread long-term recording on the scale of days and months has not been practical due to a variety of technical difficulties (Emondi et al., 2004). IsoI can provide a robust method for investigating the consequences of these problems and for determining recording periods that are stable enough for detailed analysis. In this application, the user would first cluster a minimal N-minute period of the recording, defining cluster boundaries. The duration of this period would then be incrementally augmented, and the IsoI measures applied with these same cluster boundaries. With duration increase, the larger set of waveforms will blur the clusters, resulting in lower values of the IsoI measures. With experience, a threshold can be set that defines a distinction between “blurring” and cluster degradation, allowing identification of a time after which we can no longer be confident that the same units are being recorded. At this point, the user will need to re-cluster and find new cluster boundaries. This incremental “chunk-and-cluster” procedure will allow a quantified interpretation of whether it is reasonable to accept that individual units have been continuously isolated for periods of minutes or possibly hours (days or weeks seem unlikely with current technologies). Note that there will be nothing in this determination to distinguish whether unit appearance has changed or if new units have been recorded.

Automation

The IsoI measures represent an important step towards spike-sorting automation, which has previously been investigated by multiple groups (Lewicki 1998; Wheeler 1999; Shoham et al., 2003; Quiroga et al., 2004). IsoI automates quality control, allowing methods to be developed that use IsoI as a fitness function in machine learning optimization algorithms. IsoI can also be used to determine whether and when an algorithm's output approaches or exceeds expert efforts. Additional uses of IsoI include using the measures in a real-time system that assigns spikes and periodically prompts the user to adjust cluster boundaries (or re-cluster) when IsoI values fall below a threshold. Use of IsoI in such an automated fashion would require optimizations, such as caching of values in KD-Trees and a single calculation of optimal dimensions. Once the KD-Trees and best dimensions are found, nearest-neighbors can be found in logarithmic time, proportional to the number of elements in the KD-tree. Since a nearest-neighbor lookup will be required for each element in a cluster, the runtime will also scale in proportion to cluster size. The complexity of iterative optimization algorithms would also be proportional to the number of iterations run. Further development of partial or full automation methods will be a great time-savings for many labs, and also allow for inter-laboratory exchange of data, reducing error and bias.

Important for neuroscience

Improved understanding of physiological and pathophysiological brain states will require long-term recordings that can determine how individual neurons perform computations over time and across different experimental conditions (Thompson and Best 1990; Tolias et al., 2007; Dickey et al., 2009). Spike-sorting methods, already widely used for cortical recordings, will find increasing use in determining ensemble activity patterns in subcortical areas as well (Schmitzer-Torbert and Redish 2004; Bryant et al., 2009). Furthermore, usage can be expected to expand beyond the classical assessments of sensory and place-cell ensembles and begin to look at the pathophysiology of ensemble discoordination and hypercoordination seen in the dynamical brain disorders: epilepsy, Parkinson's disease, and schizophrenia (Lytton, 2008; Olypher et al., 2006).