Improving data quality in neuronal population recordings

Abstract

Understanding how the brain operates requires understanding how large sets of neurons function together. Modern recording technology makes it possible to simultaneously record the activity of hundreds of neurons, and technological developments will soon allow recording of thousands or tens of thousands. As with all experimental techniques, these methods are subject to confounds that complicate the interpretation of such recordings, and could lead to erroneous scientific conclusions. Here, we discuss methods for assessing and improving the quality of data from these techniques, and outline likely future directions in this field.

Introduction

The more powerful an experimental method, the more care must be taken to ensure its correct application. The two leading methods for measuring the activity of many neurons simultaneously – multichannel electrophysiology and population calcium imaging – are benefiting from an exploding range of technical innovations. The increasing complexity of these methods, however, requires increasingly sophisticated approaches to ensure the quality of the data recorded. Quality control is more essential than ever, to ensure the scientific conclusions based on data from these methods are correct, and not a result of experimental artifacts.

In this Review, we discuss some factors that can affect data quality in neuronal population recordings. Careful experiments, of course, are the foundation of high quality data. Experimental design involves inevitable tradeoffs, for example between the number of neurons that can be recorded and the error rates that are acceptable. It is also important that scientists use appropriate data processing techniques to reliably identify individual neurons, and detect when signals are likely to be corrupted. At present – as with many other techniques – the involvement of human operators plays an important role in these quality control procedures, and is currently unavoidable. While effective at catching or correcting data quality problems, input from human operators can also produce biases if not carefully applied.

There can be multiple types of experimental errors in neurophysiology, whose importance depends on the scientific question. False-positive errors – the assignment of spikes to a neuron that did not actually fire them – can lead to invalid conclusions about how information is encoded. False-negative errors – omission of spikes that a neuron genuinely fired – will have a potentially milder consequence of underestimating firing rate and reliability, but only if the errors occur at random: if the errors are themselves correlated with other factors (such as particular patterns of network activity, bursting, or movement), then invalid conclusions could again arise. In population recordings, false positive and false negative errors often arise together, resulting from the incorrect assignment of one cell's spikes to another, from the incorrect merging of multiple cells' spikes together, or from incorrect splitting of a single neuron's spikes into multiple detected cells. These correlated errors can lead to potentially invalid conclusions about population coding and correlation patterns. Furthermore, selection bias (a systematic failure to detect certain types of cells) can give an incorrect picture of how information is encoded at the population level.

Ultimately, a proper understanding of the limitations in current experimental techniques will only be achieved when sufficient “ground truth” data has been collected. In practice, “ground truth” refers to measurement of neural population activity simultaneously with a method such as on-cell electrophysiology, which offers nearly perfect detection of all spikes fired by a single neuron. Such data are presently rare. Nevertheless, the existing ground truth data, together with other approaches such as simulation, has helped the field develop an understanding of the types of confounds that can occur, and methods to identify or correct errors. Careful application of these approaches can help ensure scientific conclusions based on data from population activity measurements are robust.

Extracellular electrophysiology

Extracellular neuronal recordings are typically performed by inserting microelectrodes, insulated everywhere except one or more small recording sites¹. The signals from the electrodes are amplified and digitized with a sampling frequency in the range of 20–30KHz, a rate required to resolve extracellular action potentials waveforms ( “spikes”) of duration the order of 1-2 msec. In addition to spike waveforms, the extracellular voltage contains a higher-amplitude, lower frequency “local field potential” signal, which is typically separated from the spike signals by filtering, and used to provide an indirect measure of ongoing global activity patterns.

An electrode with a single recording site can detect the activity of multiple neurons, but to separate the activity of these cells requires appropriate computational analyses. Typically, spikes are detected as crossings of an amplitude threshold, and a waveform is extracted for each spike and temporally realigned to subsample resolution. Because the extracellular spike amplitudes and waveform shapes produced by different neurons at any one point in space can differ², the firing of individual neurons can be separated by classifying the waveforms into discrete groups, a process known as “spike sorting”^3–7. The peak amplitude of the spike decreases with the distance from the neuron to the recording site, and for neurons located closer than ~50 μm from the electrode tip, the spikes are large enough to be detected over background activity and it is often possible to separate them according to their shapes^{1, 8}. For neurons further away, up to about 150–200 micrometers from the electrode tip, spikes can be detected but the difference in their shapes is masked by the noise and they are grouped together as “multiunit activity”. Neurons further away cannot be detected and they contribute to the background activity in the recording.

Why is spike sorting necessary?

The importance of spike sorting is dramatically illustrated by intracranial recordings made in the human brain. Consider an example recording made with a intracranial microwire electrode, implanted for clinical reasons into the hippocampus of an epilepsy patient, who viewed pictures presented in random order (Figure 1)^{9, 10}. If one were to consider all detected spikes, without spike sorting, no obvious increase over baseline firing rate is visible for any of the stimuli. After separating the neurons based on spike shapes, however, a very different conclusion emerges: the spikes in the original signal reflected the mixed activity of multiple neurons, and these neurons are extremely selective to individual pictures: one neuron responded reliably to a picture of Vladimir Putin, and another responded to a picture of the Taj Majal. The spikes from these two neurons represent only about 4% and 1% of the total number of spikes recorded in this electrode, and this extreme selectivity could not have been detected without spike sorting. In general, the selectivity of single units in the human medial temporal lobe is higher than the selectivity observed for multiunits, where it is not possible to separate the contribution of the different units¹¹. Correct spike sorting is thus essential to understand the neural code employed by this brain circuit. Without it, one would not only miss very sparse responses, but also make the erroneous conclusion that the circuit employs a “dense code”, in which individual units conveyed information about multiple stimuli, rather than a “sparse code” in which single cells are exquisitely tuned for particular stimuli.

Figure 1.

Spike sorting is required to draw valid conclusions in extracellular electrophysiology. (a) Top, extracellular recording from a single microwire electrode in the hippocampus of a patient implanted with intracranial electrodes for clinical reasons. Bottom plots show overlapped all detected spikes (left), and the sorted spikes corresponding to 2 single units (clusters 3 and 5). (b) Responses to 5 pictures presented in an experimental session. Considering all the detected spikes together, no response can be observed in the raster plots. However, a clear response to Vladimir Putin appears when considering only the spikes corresponding to cluster 3, and a response to the Taj Majal appears when considering the spikes corresponding to cluster 5. Time zero corresponds to stimulus onset. For space reasons, only 2 of the 8 identified clusters and only 5 of the 14 presented pictures are shown, but there were no responses for these other clusters and pictures. Adapted from Ref.⁹.

Electrode design

The physical design of the electrodes used for extracellular electrophysiology makes a great difference to the type of signals recorded. Intracellular recordings represent a “gold standard” in neurophysiology, offering perfect spike detection, as well as the ability to measure and control membrane potentials, estimate synaptic conductances, control the cell's chemical environment, and stain for later anatomical reconstruction. Nevertheless, the difficulty of intracellular recording severely limits its use for large-scale studies. Electrodes placed directly outside the cell (juxtacellular or on-cell recordings) provide a somewhat easier way to obtain perfect isolation, but are still impractical to use at scale. Extracellular recordings – which rely on detection of electric fields tens of microns away from the recorded cell – are more straightforward, but also involve greater errors, which increase with the number of neurons recorded.

The quality of extracellular unit isolation depends on the amplitudes of extracellular spikes relative to background noise. This noise arises from two sources. The first is thermal noise, which increases with the resistance of the recording electrode (e.g. Ref.¹²), and can be ameliorated by coating with materials such as PEDOT¹³. The second noise source is the firing of the large number of neurons that are too far from the recording site to produce sortable spikes (>~50 μm), but superimpose to produce “hash” in the same frequency range^{1, 8}.

Different electrode designs offer complementary advantages. Small electrodes (with diameter < 5-10 μm and impedance > 1 MΩ) record from only a few nearby neurons¹⁴, but can show excellent unit isolation as they can be positioned very close to the neurons of interest; since spike amplitudes decay rapidly with distance^{2, 8}, this results in large amplitudes relative to background activity. Larger electrodes (diameter several tens of μm) have impedances typically below 1 MΩ, and record the activity of neurons in a larger area. For single sites, a smaller ratio of the amplitudes of nearby neurons to those of the distant cells contributing to the background activity leads to a lower signal-to-noise ratio (SNR). The optimal design of an electrode thus depends on the techniques that are used to process the data: with limited spike sorting algorithms, it might be preferable to have small electrodes with large SNR, whereas with more advanced algorithms and multisite probes, larger, low impedance electrodes can increase the yield of identified neurons^{15, 16}.

An important and understudied question concerns the degree to which electrode insertion damages the neural tissue that is being recorded. Calculations based on the decay of amplitudes with distance suggest that a single tetrode should be able to detect the activity of up to 100 single neurons in hippocampus, but in reality one typically finds an order of magnitude fewer¹. Although it is possible that the missing “dark matter” neurons are healthy but not firing, or firing but not identified by current spike sorting algorithms¹⁷, insertion of the electrode may also have damaged or killed a substantial fraction of them. Systematic investigation of how electrode geometries, materials, and insertion strategies affect tissue damage would greatly help optimal electrode design.

Multichannel electrophysiology

Understanding complex brain processes requires the analysis of large and simultaneously recorded neuronal populations^{1, 18–22}. Current multielectrode designs allow recording from hundreds of electrodes, and thus hundreds of neurons simultaneously^{4, 23–27}, and these improvements have been matched by increased capabilities of data acquisition systems.

There are two major approaches to the design of multielectrode arrays. The first approach – exemplified by microwire arrays that have been used for animal studies²⁸, or the microfabricated “Utah arrays” that have been implanted in human cortex to enable brain-computer interfaces²⁹ – consists of a large number of single-contact electrodes, with an inter-site distance of at least 100 μm. As any individual neuron can be detected by at most one of the recording sites, data processing for such electrodes (and the corresponding quality concerns) is the same as for single site electrodes. Thus, the recorded population size increases linearly with site count, although without independently movable contacts, many sites may not record neurons. Furthermore, isolation quality does not improve with increasing site count.

The second approach to multielectrode array design is to use dense arrays with an intersite distance of less than 50 micrometers. This approach employs either twisted microwire bundles such as “tetrodes” (4 microwires twisted together^{30, 31}), or micromachined silicon probes^{1, 32–35}. High density probes allow recording the spikes of a single neuron from multiple sites. This improves spike sorting performance, as the spikes of two neurons can frequently look identical on one channel, but differ on others^{31, 36}. With this method, unit isolation quality is expected to scale with site density, as confirmed by spatially subsampling of data from dense electrodes³⁷. Nevertheless, the number of sites achievable is limited by constraints of manufacturing technology, for example shaft diameter (probes that are too thick may damage brain tissue), or the increased noise found with small, high-impedance recording sites. Furthermore, the benefits of high density probes require using spike sorting algorithms that can combine information from different channels.

Spike sorting methods

Spike sorting is more complex for dense arrays than for single contact electrodes. For tetrodes, the traditional method of spike sorting – still commonly applied in many labs – is purely manual cluster cutting. In this procedure, a set of features are computed for each spike, such as the peak amplitude on each channel, or waveform features evaluated by principal component analysis. Using a graphical interface, an operator manually draws boundaries around the resulting “clusters”, which correspond to putative single neurons. The operator is guided in this process by a number of additional tools, such as the computation of auto- and cross-correlograms, which can help identify poorly isolated units by the presence of refractory period violations.

Today, the most commonly employed spike sorting method is “semi-automatic”. With this approach, spike detection and feature extraction proceeds as before, but an automatic cluster analysis algorithm is run on the spike data, and a human operator uses a graphical interface similar to those used for manual sorting to check its output, and correct mistakes the algorithm has made where necessary (some examples of such possible mistakes are described below). Compared to purely manual spike sorting, the semi-automatic approach has two major advantages. The first is time: it takes substantially less human time to check the output of an automatic algorithm, than to perform a fully manual sort. The second advantage is that this approach achieves substantially lower errors than purely manual sorting, as demonstrated using ground-truth tetrode data³⁶. These lower error rates occur because, the optimal boundary between clusters is a high-dimensional surface, which can be found by automatic algorithms but not drawn by human hand using a 2d computer interface³⁶.

The presence of a human operator in the data processing pipeline raises the potential for subjectivity and bias to occur. However, while a fully-automatic spike sorting system would clearly be desirable, it has to date not proved possible to implement algorithms which work robustly in real-world in extracellular recordings. Similar constraints are faced in several other fields of data-intensive biology, such as electron-microscopic connectomics, which also rely on manual operator curation³⁸. Thus, while substantial developments have occurred to reduce the amount of manual operator time required for the sorting process^{37, 39}, fully automatic systems are rarely applied in current practice. Fortunately, analyses comparing the decisions made by multiple expert operators have shown that their corrections of automatic cluster performance tend to be similar^{17, 37}.

Validation of spike sorting

One of the largest problems for the development of spike sorting methods is the paucity of “ground truth” data, in which extracellular arrays are combined with other methods providing unambiguous recording of firing times. While invertebrate and in vitro preparations have provided some data on which to test algorithms^40–42, the noise conditions and nonstationarity found in mammalian systems in vivo are substantially more challenging. The difficulty of obtaining ground truth data in vivo has made such data very rare^{8, 36, 43} (https://crcns.org/data-sets/hc/hc-1). The data that is available, however, suggests that error rates for semi-automatic clustering with tetrodes can be of the order 5-10%, but the error rates of purely manual cluster cutting may be substantially higher³⁶.

In the absence of suitable ground truth data, a number of methods have been used to generate “surrogate” ground truth to validate spike sorting algorithms. One approach consists of performing detailed biophysical simulations of extracellular activity^44–46. The computational expense of this method, combined with uncertainty in exactly how to model the challenges of real data has encouraged other authors to try a different method. In this “hybrid” approach, spikes isolated from one recording are digitally added at known times to a second recording, or to a simulation of background activity^{16, 47, 48}. This approach has been used to estimate the errors expected from semi-automatic analysis of high-count array data, again yielding an estimate of errors of the order 5-10%³⁷. An alternative approach relies on measuring the reliability of the spike sorting algorithm under perturbations of the data set⁴⁹.

Most current methods of spike sorting fail at times of high neuronal synchrony. In the hippocampus, for example, transient events of highly synchronous neural activity known as “sharp waves” are of great interest due to their proposed role in memory. Ground truth data suggests that error rates during sharp waves can be 5 times above average, reaching levels as high as 50% (Ref.³⁶). Such errors could come from two sources. First, synchronous activity might lead to temporally overlapping spikes, which cannot be resolved by traditional spike sorting algorithms; but might be resolvable by newer algorithms based on template matching^50–53. Alternatively, ripples might lead to the firing of otherwise silent cells with waveforms too similar to be distinguished accurately; consistent with this possibility, sharp waves are accompanied primarily by an increase in false positive, not false negative errors³⁶.

Common confounds in extracellular electrophysiology

Isolation quality

In even the best quality extracellular recordings, most spikes will come from neurons far from the probe, with amplitudes too low for effective isolation. It is therefore important to identify which clusters correspond to well-isolated single cells, and which represent mixtures of several neurons. The importance of such metrics is underscored by the fact that different operators – although they generally agree on which corrections to make in semi-automatic clustering – can have very different opinions on what constitutes a well-isolated unit³⁷.

To address this issue, several quantitative metrics of cluster quality have been proposed. As all neurons exhibit an absolute refractory period, any cluster that shows a large fraction of inter-spike intervals (ISIs) less than 1-2 milliseconds cannot be a well isolated unit. However, the converse does not apply: an apparently clear refractory period does not imply good quality isolation. Indeed, a cluster that contained the intermixed spikes of two different neurons that fired at separate times (for example hippocampal neurons with non-overlapping place fields) would have a completely clear refractory period. Furthermore, manual examination of autocorrelograms in the presence of bursting can lead to an erroneous impression of a clean refractory period even for very poorly isolated cells³⁶.

A second class of quality metrics measures how well the spikes of one cluster are separated from those of neighboring neurons^{12, 54–56}. It is important to note, however, that there is no single threshold value that objectively defines “good” isolation quality: the criterion must depend on the scientific application. For example, an analysis of the structure of complex spike bursts – which involves a progressive decrease of amplitude as the burst continues (e.g. Refs.⁵⁷, 58) – required a highly stringent criterion⁵⁶. Accurate isolation is also critical to study pairwise correlations of spike trains^59–61; for example, if the spikes of a single neuron are artificially divided into two clusters, these clusters will show a spurious negative correlation because their spikes will be always separated in time. Estimation of neuronal tuning and selectivity can be also highly sensitive to clustering errors, as demonstrated by the earlier example of human intracranial recordings. By contrast, for brain-machine-interface applications, the exact identity of the neuron generating each spike might not be crucial, and it may be advantageous to use the largest possible number of recording sites in an unsupervised way, even if not sorted at all^{62, 63}. Spike sorting may also be less critical where there is a topographic organization of responses – i.e. when nearby cells tend to fire to similar stimuli – compared to cases when nearby neurons fire to unrelated stimuli, as it has been described in the rodent⁶⁴ and the human hippocampus⁹ (Figure 1).

When using an isolation quality metric, how should a scientist decide what threshold value of isolation quality to require for a particular scientific question? A simple method is to consider how the quantity being measured depends on isolation quality. For example, the cross-validated spatial information encoded by putative pyramidal cells of rat CA1 drops substantially for values of isolation distance⁵⁴ less than 20, but reaches an asymptote above this value (Figure 2). This suggests that a threshold of 20 is suitable for analysis of spatial information coding in these cells.

Figure 2.

Quantitative measures of unit isolation in extracellular electrophysiology. Each point represents a single neuron recorded in CA1 of a rat exploring an environment, showing isolation distance (a measure of unit isolation quality⁵⁴) vs. estimated spatial information content (which can be negative as it is computed by cross-validation¹⁴²). The red curve represents a running median. The curve reaches an asymptote of ~1 bit/s for values of isolation distance greater than ~20, indicating that this is the true average for well-isolated cells. Data reanalyzed from Ref.¹⁴², with some points above the top y-axis value truncated for visualization purposes.

Selection bias

Simultaneous intra- and extracellular recordings suggest that there should be approximately 140 single neurons within the radius recordable by a single tetrode in the hippocampus^{1, 8}. However, actual tetrode recordings rarely detect more than a dozen neurons at a time. The reason for this disagreement has been attributed to the presence of silent neurons⁶⁵, electrical insulation⁶⁶, damage produced by electrode insertion⁶⁷ or a potential inability of spike sorting algorithms to deal with large numbers of neurons¹⁷.

Both manual and automatic spike sorting methods are likely to be biased against low-rate neurons: if a cell fires only a few spikes, these will not be sufficient to define a cluster and the cell will be missed. Given the preponderance of low-rate cells in brain circuits, a failure to account for this selection bias could lead to incorrect estimation of the firing distribution⁶⁸. In neocortex, where superficial-layer pyramidal cells fire with lower rates than deep-layer pyramids and fast-spiking (FS) interneurons of all layers^69–72, bias towards high-rate neurons has led most population electrophysiology to focused on deep cortical layers.

Historically, single neuron recordings have been performed by advancing the electrodes until neural activity is detected⁷³. This method can lead to a different form of selection bias: not only will recordings be made primarily from high-rate neurons (perhaps again leading to a bias toward FS interneurons), but also from cells responding to the specific stimuli or conditions present at the time of the recording. Without care, this could lead to a “confirmation bias”: an investigator would find an over-abundance of neurons that respond precisely to the stimulus or condition being investigated.

Sampling bias in extracellular electrophysiology can be ameliorated by performing non-stop chronic recordings, using fixed electrodes, over very long time periods: recording 24 hours/day for days or weeks can lead to sufficient spike numbers to define clusters even for cells of very low firing rate³⁹.

Operator bias

Because spike sorting has a manual curation step, the possibility of subconscious operator biases must be very carefully excluded. As with many other procedures, when comparing recordings of subjects in different conditions, it is essential that the operator performs manual curation blinded to the condition of each recording. This is particularly important when analyzing quantities such as stability of firing patterns, which can be easily altered by the manual curation step⁷⁴.

Another important concern regards the use of the neuron's firing correlates (e.g. sensory receptive fields or place fields) in the spike sorting process. While observing a similar firing pattern in two clusters does make it more likely that they represent the same cell, the use of receptive field information during clustering may bias results. Indeed, if neurons genuinely show receptive field plasticity, this will be underestimated if response stability is taken as a criterion for good isolation.

Electrode and waveform drift

Errors in spike sorting are of two types: the spikes of different cells can be erroneously merged together, or the spikes of a single cell can be erroneously separated into two or more clusters (“overclustering”), which often occurs when the spike waveforms of a particular neuron varies during the course of the experiment.

Waveform variability can occur for multiple reasons. The most common reason is “electrode drift”: the physical movement of the electrode relative to the brain. Due to the highly localized electric fields neurons produce^{2, 8, 16}, even a few microns' movement is enough to cause substantial variability in spike amplitudes. This variability is largest for high amplitude spikes, consistent with the sharply-peaked structure of extracellular electric fields. This problem is particularly severe in acute recordings, in which the electrode is fixed not to the skull, but to an external manipulator, meaning that small movements of the head, or relaxation of the brain after compression caused by probe insertion, will cause a movement of the cells relative to the electrodes. Physical electrode drifts are less serious in chronic recordings, where stable recordings have been observed for periods of days or weeks^{39, 75}.

Not all waveform variability is caused by physical movement. Extracellular spike amplitudes decrease during the course of complex-spike bursts, and also decrease after prolonged firing, even without bursting^{56, 58}. Moreover, extracellular waveform shapes can depend on cellular factors such as dendritic action potential backpropagation or electrogenesis, which can vary with firing history, inhibition and neuromodulation^{76, 77}. The difficulty of quantitatively modeling these phenomena is one reason fully automatic spike sorting has so far proved challenging; nevertheless, the fact that these effects can be caught during manual curation suggests that automatic systems may also be eventually possible.

Outlook

While today's silicon probes have at most a few hundred channels, probes with thousands of channels are currently under development. These probes will raise new challenges for data processing and quality control, the most important concerning manual spike sorting. Purely manual sorting is clearly impossible for this size of data, and even curation of semi-automatic sorting will become a serious burden. This burden can be dramatically reduced by the development of algorithms that minimize operator time, by directing attention to only those decisions that cannot be made automatically. Fully automatic spike sorting not only becomes more desirable with very high count probes, but it may also become more achievable. Electrode drift and the consequent spike shape changes presents one of the biggest barriers to fully automatic sorting; it is possible that large dense probes might sample the extracellular electric fields with enough spatial resolution to allow drift to be tracked and corrected in software.

A second challenge for spike sorting regards the long-term tracking of clusters, to study plasticity, for example, during learning experiments. With a few exceptions, scientists have been hesitant to use extracellular electrophysiology to study long-term plasticity of firing properties, as clusters may appear, disappear, merge or separate^78–80, and it is critical not to confuse changes in tuning of neuronal populations with changes in the recording conditions or small electrode movements. Three techniques may ameliorate these problems. The first is the gradual refinement of chronic recording methods, which can now ensure high stability of many cells over multiple days or even weeks of recordings^{39, 75, 81}. The second is the development of quantitative methods for unbiased assessment of cluster similarity, that may help identify neurons across multiple days^{78, 82–84}. The third is the use of 24-hour recording³⁹ which reduces the problem to a much easier one of tracking slow continuous changes, than tracking across sudden jumps between recording sessions.

Finally, and most important, there is a need for research into fundamental questions underpinning extracellular array recording. What electrode geometries, surface contact diameters, impedances, and materials provide the best data quality while avoiding tissue damage? How do these properties interact with the choice of sorting algorithms? The most critical experiments to solve this problem involve collection of ground truth data, to quantify the performance of different electrodes and sorting methods without relying on simulations.

Population recording via calcium imaging

Calcium imaging is a complementary technique for measuring the activity of neuronal populations. Depolarization during action potentials opens voltage-gated Ca²⁺ channels and results in a transient increase in intracellular [Ca²⁺], which can be detected optically using fluorescent reporters: calcium-sensitive dyes or proteins. Calcium imaging can be used to infer patterns of spiking activity across hundreds to thousands of identified cells in vivo^{85, 86}. Like any measurement, however, it demands careful application and analysis.

Calcium signals are correlated with neuronal spiking, but are an indirect reflection of it, and biophysical variations make the precise relationship between calcium signals and spiking variable⁸⁷. Calcium reporters can themselves limit the precision of spike inference: although synthetic calcium dyes can be used in a linear regime, they still exhibit nonlinear features including saturation; and genetically encoded calcium indicator (GECI) proteins are highly nonlinear due to cooperative Ca²⁺ binding⁸⁸.

The temporal resolution of the calcium signal is limited. Indicator kinetics are relatively slow (e.g., rise times of ~10 ms for single action potentials measured with synthetic dyes, and >50 ms for many GECIs). Furthermore, imaging methods that scan over space trade off recordable population size against sampling time. Recordings from large neuronal populations often require timesteps on the order of tens to hundreds of ms (Ref.⁸⁶).

As with electrophysiology, the ultimate check on recording quality is ground truth, typically obtained by simultaneous on-cell patch recordings. Such recordings are feasible with some, but not all calcium imaging instrumentation. However, even in the absence of ground truth, good experimental design and rigorous analysis can improve data quality. Below, we discuss these considerations. Our discussion primarily relates to two-photon laser scanning microscopy, but several points are relevant to one-photon imaging, including wide-field imaging through GRIN lenses⁸⁹, and light-sheet imaging in transparent specimens⁹⁰.

Experimental design and measurement noise

As with any technique, instrumentation, preparation, and recording parameters must be tailored to the demands of the specific scientific application. Some experiments require detecting single action potentials and/or resolving precise spike counts in each neuron with minimal uncertainty. For others, detecting qualitative increases and decreases in spike rate suffices. Experimental design is critical because most optimizations involve tradeoffs. For example, high zoom (many pixels per neuron) and high frame rate can provide faithful estimates of spiking based on calcium signals⁹¹, but also limit the number of neurons that can be sampled in each imaging frame. Low frame rate acquisition can be sufficient to map population responses when the stimulus changes slower than the acquisition rate: for example, Ohki and colleagues⁹² mapped the orientation-tuned responses of hundreds of neurons at 0.61 frames / s, with a stimulus that changed at 0.0625 Hz. By contrast, Dombeck and colleagues⁹³ acquired 15.6 frames/s over smaller fields-of-view, to capture the activity of hippocampal place fields with sub-second resolution.

Imaging quality depends primarily on recording a sufficient number of photons per pixel. Photon emission exhibits Poisson-like variability (“shot noise”), with SNR scaling as the square root of the photon count⁹⁴. Optimizing quality thus requires tuning parameters such as laser power and scan configuration, which includes parameters such as pixel dwell time, field-of-view, pixels per neuron, subject to the limitation that the overly-high laser power can cause tissue damage⁹⁵. Optimized scan patterns^96–98 can target sampling to individual neurons, but lack the spatial coverage required for posthoc motion correction. Photons per pixel can be estimated from parameters including the gain and offset of the photomultiplier tube (http://labrigger.com/blog/2010/07/30/measuring-the-gain-of-your-imaging-system/). Tradeoffs between photons per pixel, numbers of pixels, and SNR can quantified using signal detection theory, and compared and optimized for specific experiments^{99, 100,101}. For example, GCaMP6s imaging can provide single AP detection with nearly 100% detection of all spikes when imaged at a high frame rate over a small field-of-view (30 μm × 30 μm at 60 frames/s)⁹¹. However, larger field-of-view, population-level imaging (265 μm × 265 μm at 59.1 frames/s) yields spike rate estimates that correlate with the true spike rate at an average level of ~ 0.5 (Pearson's R, using a 50 ms spike rate binning window)⁸⁷. Quality is also affected by indicator properties and labeling intensity^{88, 94}. Typically, however, these quantities are not measured for in vivo preparations, and parameters yielding high SNR data are identified by trial-and-error (e.g., dye concentration, viral vector titer, number of days after transfection to image).

Signal contamination

In densely labeled tissue, structures adjacent to cell bodies can contribute contaminating signals¹⁰². In areas such as neocortex, somata are distributed sparsely enough to make contamination from adjacent cell bodies rare¹⁰³, permitting moderate-resolution imaging systems to accurately measure cellular-resolution dynamics^{92, 103, 104}. However, in structures with densely packed cell bodies, such as the hippocampus or cerebellar granular layer, cell-to-cell contamination cannot be ignored.

Calcium imaging of large populations typically leads to contamination from signals arising in the neuropil: the axons and dendrites of nearby cells, whose activity produces a substantial contamination of the signal recorded at any cell soma^{91, 105, 106}. Even with two-photon microscopy, axial resolution is often limited to several microns or more, although in-plane (lateral) resolution can be sub-micron (Fig. 3a). Furthermore, even high resolution (i.e. high numerical aperture) imaging systems are precluded from realizing their full resolution in practice, due to optical aberrations caused by brain tissue and the loss of marginal rays when imaging deep^{107, 108}. Imaging system aberrations can also limit resolution, particularly when imaging outside of the very center of the field-of-view^109,110. GRIN lenses, used to access deep structures, also suffer from significant aberrations^111,112. Adaptive optics can compensate for aberrations of the focused excitation light¹¹³, but the emitted fluorescence photons are still subjected to scattering and aberrations in the tissue and optical systems. Neuropil contamination can be somewhat mitigated by expressing GECIs in sparsely, for example using transgenic mice with Thy1 promoters, or conditional viral strategies^{91, 114–116}. However, neuropil contamination represents a serious concerns for calcium imaging. The neuropil signal reflects the summed activity of a large number of neurons, the majority of which will typically be located close to the imaged region, and can thus itself be tuned for similar stimuli or behavioral variables¹¹⁷. If not accounted for, neuropil contamination can lead to spurious conclusions about neuronal encoding. It is therefore essential that this contamination be understood and, to the extent possible, removed during analysis. Approaches will be discussed in “data processing”, below.

Figure 3.

Subtracting neuropil contamination from raw fluorescence time courses. (a) In two-photon imaging, the point spread function (PSF) is elongated in the axial dimension even in high numerical aperture systems. Pixels within the borders of cell bodies still contain signals from the surrounding neuropil. (b) The GECI GCaMP6s was expressed in mouse visual cortex neurons, resulting in brightly labeled cell bodies and neuropil. (c) Binary masks for cell body ROIs (black) were identified semi-automatically and neuropil regions were algorithmically constructed (avoiding pixels belonging to other potential cell bodies or black regions). (d) Raw traces for the fluorescence time courses of the selected cells. (e) Fluorescence time courses for the background regions for each selected cell. Note the high temporal correlation. (f) Fluorescence time courses after background subtraction. Note the reduced mean correlation¹⁴¹. All traces have been scaled to the same maximum height to better exhibit details in the time courses. Figure credit: J. N. Stirman, Y. Yu, S. L. Smith

Motion artifacts

When imaging in living subjects, heartbeat, breathing, and motor behavior can all contribute significant movement artifacts. While heartbeat-associated movements in the brain are typically on the order of 1 μm or less¹¹⁸, breathing and motor behavior can cause larger movements over 10 μm^{119, 120}. Movement amplitudes vary by brain area and can be reduced by appropriate surgical preparation¹²¹. During movements, cells can change their pixel location in the imaging plane (XY), or in and out of the plane of focus (Z). If temporal resolution is high enough to make frame-to-frame XY movement small, movement can be corrected through image registration, e.g. global cross-correlation or line-by-line alignment^{117, 122, 123}. Large and faster XY movements may require more complex model-based algorithms^{119, 124}. Motion in Z is more difficult to correct, since the neighboring planes are typically not recorded. Generally, point spread functions (PSFs) are extended in Z, making motion artifacts due to small Z movements less problematic^{125, 126}. In cases with large Z movement, multiplane imaging¹²⁷ and online motion correction^{120, 128} can help minimize artifacts.

Data processing

Before calcium data can be used for scientific analysis, it requires preprocessing, typically involving: image alignment/motion correction; segmentation to find regions of interest (ROIs) corresponding to imaged cells; time course extraction; neuropil compensation; and, optionally, spike train estimation. In some analysis algorithms, multiple steps are performed in concert.

Segmentation and time course extraction

The simplest way to define ROIs corresponding to individual neurons is with binary masks, which can be drawn manually or with varying degrees of automation⁸⁵. With binary masks, neuropil contamination must be estimated and subtracted out in a separate step, and can be estimated using either an average of the neuropil signal surrounding the cell of interest¹²⁹ (Fig. 3b–f), or by subtracting the first principal component of the contamination^{130, 131}.

More ambitious approaches describe each pixel's calcium signal as a superposition of signals from one or more cell bodies or processes, neuropil, and potentially other sources (e.g., instrumental noise). This is typically framed as a matrix factorization problem, where the spatial components (neuronal ROIs) and temporal components (neuronal time courses) are simultaneously learned. As with spike-sorting, the results of such automatic algorithms must be verified on a case-by-case basis. For example, methods based on PCA/ICA^{131, 132} can identify negative temporal signals, which cannot represent neuronal time courses; these methods can also produce ROIs corresponding multiple neurons or dendritic regions, since the algorithm has no prior information about the spatial extent of signal sources. Constrained nonnegative matrix factorization, optimized for the particular characteristics of calcium imaging data^{133, 134} provides promising results even in the presence of signal crosstalk¹⁰⁶, while a new combined clustering and factorization method can accurately process over 10,000 cells in approximately real time¹³⁵. As matrix factorization methods separate temporally distinct patterns of activity, they may fail on neighboring cells that exhibit highly synchronous activity; a supervised learning approach might avoid this shortcoming⁹⁶. Furthermore, any activity-dependent algorithm will be biased towards active cells; labeling cell nuclei with static (not calcium-dependent) fluorescent proteins can help identify cell bodies independent of activity¹¹⁷. Similarly, “dictionary” methods have been developed to detect cells based on average/resting fluorescence activity, thus finding a large number of inactive cells¹³⁶.

Manually verifying the results of automatic algorithms is time-consuming, but some simple automatic techniques can catch many artifacts. The zero-lag cross-correlation of ROI pairs can identify contamination of one cell's signal by another. These correlations should usually be close to zero, rarely above 0.5, and values > 0.8 typically indicate contamination. Minor contamination can result in smaller increases in correlation values, and so time courses from nearby cells should be checked particularly closely. Correlation due to contamination should be more stable in time than correlation due to neuronal firing, so rolling correlation analysis (with width substantially above that of a fluorescence transient) can help identify unusually stable correlations due to contamination. Excluding individual highly-contaminated pixels will result in less contaminated ROIs. As genuine neuronal correlations can be affected by brain state, this analysis is best performed on data obtained under conditions of relative desynchronization (e.g., stimulus-evoked activity rather than spontaneous activity under anesthesia). Finally, correlating the time-series of individual pixels to stimulus and behavioral variables can help diagnose artifacts.

Evaluation of segmentation approaches would benefit from a systematic comparison of algorithms against ground-truth. Ground truth for segmentation can be hard to obtain, or even define, but options include hand annotation, or co-expressing anatomical markers (e.g. fluorescent proteins that are confined to the nucleus) that report the presence or absence of a neuron soma at a location. The Neurofinder challenge provides several ground truth datasets and a web application for comparing algorithm results (http://neurofinder.codeneuro.org/), and ground truth data is also available at https://crcns.org/data-sets/methods/cai-1. Surprisingly, nearly all segmentation algorithms thus far have taken an unsupervised approach – trying to infer neurons directly from data, rather than by training a supervised model on existing annotations, as is common in object recognition, behavioral classification, and anatomical segmentation. Especially with the availability of annotated data, supervised methods could be a fruitful avenue of exploration.

Spike inference

The calcium signal is only an indirect reflection of spiking. Many analysis approaches aim to derive from the calcium fluorescence time course of each neuron an estimate of firing rates or exact spike times.

The calcium signal can be approximated as a temporally filtered version of the spike train. Although the decay kinetics of this filter are typically slow (half-decay times are hundreds to thousands of ms), its rise kinetics can be fast (times-to-peak are tens of ms). The simplest approach is thus deconvolution with an estimated unitary response, such as an instantaneous rise and exponential decay^{137, 138}. In practice, deconvolution performance is limited by several factors. The relationship between calcium signals and spiking is complex, nonlinear, and varies across neurons, especially during spike bursts. With GCaMP6f, for example, spike pairs can exhibit fluorescence transients whose size depends on inter-spike interval (personal communication from D. DiGregorio and S. Wang). As with segmentation, supervised learning methods using systematic ground truth data are a promising alternative to unsupervised deconvolution⁸⁷. With present technology, estimates of spike times from calcium imaging should always be treated as approximations, though this uncertainty can be propagated through stages of analysis (Fig. 4)¹³³.

Figure 4.

Probabilistic estimation of spike times from calcium signals. A model of calcium signal generation is inverted to yield a probability distribution of spike trains (blue raster), given a single observed calcium trace (black). (a), high signal-to-noise data leads to a highly certain estimate of spiking. (b), low signal-to-noise leads to an uncertain estimate, illustrated by the raster showing different spike trains that could have generated the calcium signal. Subsequent analysis can be performed using the entire spike train distribution. Figure credit: J. N. Stirman, Y. Yu, S. L. Smith.

Estimation of spike times is not necessary for many scientific questions. When neurons fire sparsely, for example, neuronal responses can be characterized by how the calcium response itself depends on stimulus or behavioral-related factors. The results of such analyses will not be numerically identical to analyses computed from actual counts (for example when computing correlations among neurons), but if interpreted correctly, this can avoid biases introduced by explicit spike estimation. This concept is explicitly formulated in hierarchical models that describe the calcium response as a function of spike times, which are in turn a function of the stimulus and/or behavior. The parameters of such an “encoding” function can be estimated directly without explicitly estimating spike times, instead treating them as latent factors (e.g. Refs.¹³⁹, 140). Finally, spike detection can be simply viewed as a nonlinear denoising step to remove spurious low-amplitude signals, rather than an explicit estimator of spike times; in this view, procedures based on template matching, which are more flexible and less computationally expensive, may be appealing¹¹⁷.

Ground truth from electrophysiology

Experiments verifying 2-photon calcium imaging with ground truth from simultaneous on-cell electrophysiology have indicated encouraging results. It is important to match the imaging parameters between ground truth data sets and actual experiments: highly zoomed-in imaging offers a “best-case scenario” about how faithfully an indicator might report spiking, but may vary markedly from those obtained when imaging is zoomed-out to increase the number of neurons imaged (which also decreases the SNR of recorded cells). Single action potentials can be resolved >80% of the time in optimized systems, and multi-spike bursts are even more reliably detected^{91, 141}. However, estimates of the number of spikes in multi-spike bursts is typically imprecise, and in practice larger fields-of-view decrease SNR and lead to overall correlation coefficients between 0.1 and 0.5 (Ref.⁸⁷), though in some studies this has ranged up to 0.8 (Refs.¹¹⁸, 141).

In some cases, it is impossible to perform calibration electrophysiology experiments. For example, the small working distance of GRIN lenses largely precludes correlative electrophysiology. Similarly, with air immersion objectives, simultaneous electrophysiology would so perturb the optical setup as to yield it largely irrelevant. In these cases – where ground truth is unobtainable – experiments should be designed to be insensitive to the expected imprecision of spike inference. To bracket the precision of the estimated spike trains, experimenters can compare their data to electrophysiological recordings under similar circumstances. For example, data from extracellular recordings in mouse visual cortex can provide a baseline for calcium imaging studies, setting both the expected spontaneous firing rates and expected distribution of maximal firing rates in mouse visual cortex in response to drifting grating visual stimuli⁷¹, bearing in mind the selection bias in electrophysiology towards active cells.

Summary

Calcium imaging and extracellular electrophysiology can both provide high fidelity readout of neuronal population activity. They have complementary advantages: electrophysiology allows detection of single action potentials with submillisecond timing, in deep structures and multiple brain areas; calcium imaging can provide a comprehensive and less biased view of a local population, and interfaces easily with the genetic toolkit required to identify neurons based on cell type or connectivity.

Both methods are subject to experimental confounds, most notably spike sorting errors for electrophysiology, and signal contamination for calcium imaging. While these confounds cannot be avoided, they can be mitigated through careful experimental design. Furthermore, a detailed understanding of causes and consequences of these confounds makes erroneous scientific conclusions unlikely if they are used carefully. When objective measures of data quality exist, it is important that these are used and documented together with the scientific conclusions drawn. To gain a truly quantitative understanding of the error rates likely to occur in population recordings, however, it is essential that substantial further effort be put into collecting “ground truth” data calibrating these techniques against reliable measures of neural activity.