Computational Methods and Challenges for Large-Scale Circuit Mapping

Summary

The connectivity architecture of neuronal circuits is essential to understand how brains work, yet our knowledge about the neuronal wiring diagrams remains limited and partial. Technical breakthroughs in labeling and imaging methods starting more than a century ago have advanced knowledge in the field. However, the volume of data associated with imaging a whole brain or a significant fraction thereof, with electron or light microscopy, has only recently become amenable to digital storage and analysis. A mouse brain imaged at light microscopic resolution is about a terabyte of data, and 1 mm³ of the brain at EM resolution is about half a petabyte. This has given rise to a new field of research, computational analysis of large scale neuroanatomical data sets, with goals that include reconstructions of the morphology of individual neurons as well as entire circuits. The problems encountered include large data management, segmentation and 3D reconstruction, computational geometry and workflow management allowing for hybrid approaches combining manual and algorithmic processing. Here we review this growing field of neuronal data analysis with emphasis on reconstructing neurons from EM data cubes.

Introduction

Neuroanatomical research has depended on large volumes of image data from its inception. Ramón y Cajal, working at the turn of the twentieth century, produced more than a thousand manual drawings of nerve cells [1–4] based on light microscopy (LM) of Golgi stained neurons, while the first full reconstruction of Caebirgavdutus elegans neuronal circuitry [5], initiated in the 1970s, involved already ~10,000 electron microscopic (EM) images. Contemporary initiatives to map local circuits using EM [6–13], or to map projection patterns at a whole brain level [14, 15] using (LM), have high data output rates that can be in the range of gigabytes per minute and are comparable to the data rates familiar in modern particle accelerators. At LM resolution a mouse brain produces ~1 TB of data and a human brain ~1petabyte, whereas just 1 mm³ of tissue in EM produces up to a petabyte of data.

Large volumes of data that have to be managed and analyzed pose significant hardware, software and algorithmic challenges. Similar challenges are being encountered in the commercial domain as well, as exemplified by Google Earth or Youtube data repositories; arguably the neuroanatomical data sets are smaller, but have to be managed and analyzed with a smaller economic footprint, thus giving rise to special challenges.

Image data annotation and quantification in neuroanatomy have been almost exclusively manual until recently [6,10–12], with an increasing use of computational tools and viewing interfaces to facilitate the human labor. While efficient machine-human interaction can substantially improve analysis throughput, this is however not an arbitrarily scalable future solution to the data analysis challenges posed by high-throughput neuroanatomy as will be required for large-scale circuit mapping. Reconstructing a single neuron at the light-microscopic level takes dozens of hours, while doing the same for EM data takes 100-fold longer [13]. In spite of the strong need for automation, algorithms have not yet succeeded in taking over the reconstruction task (although the challenge is recognized and is being worked on [16], cf. also the DIADEM challenge for light-microscopic reconstruction [17–19]). Quantification of cell bodies has fared somewhat better, as exemplified by the Allen Gene Expression Atlas of the mouse brain [20,21] that condenses hundreds of terabytes of raw image data into a co-registered, voxellated count of labeled cells. However, this automated analysis still falls short of classical stereological procedures for histological quantification (e.g., [22]) or manual cell body mapping [23].

The current absence of effective fully automated tools for high-throughput neuroanatomy indicates the need for two lines of algorithm and software development. First, a pragmatic hybrid approach involves the division of labor between machine and humans, utilizing the amplification of human abilities using efficient software tools. In this approach, as exemplified by software now being used to reconstruct neurons from EM data cubes [24], algorithms are used for low-level image processing (stitching, alignment, contrast adaptation), and humans contribute their unique ability to detect and trace neural processes in noisy data.

A second, more fully automated approach that requires minimal human intervention is being pursued as well (for example to count cell bodies in a volume of neuronal tissue, Mitra, unpublished). In this approach there is a temporal separation of the human and machine effort: the initial, human labor intensive stage involves prototyping the necessary algorithms, potentially attempting to replicate human performance, whereas the later stage is automated with minimal human intervention in the form of quality control procedures on the output.

This review is aimed at summarizing the available software for the analysis of large-scale neuroanatomical data sets with special focus on reconstruction of neurons from EM data (Table 1, Figure 2), paying attention to the detailed technical issues that arise in specific data gathering modalities. We briefly touch on the methods involved in LM data analysis to provide some contrasts with the EM related data challenges. We focus on those tools that have been productive in a concrete neurobiological setting.

Table 1.

Overview of reconstruction tools for the analysis of large-scale neuroanatomical data sets.

Reconstruction toolf	2D/3D navigationc	Volume annotation	Skeleton annotation	Maximum image data size	Registration, stitching	Semi-automation	Figure	www	Published results
Reconstruct [28•]	2De	✓		RAM	Stitching Registration			synapses.clm.utexas.edu/tools/reconstruct/reconstiruct.stm	[10•,29]b
TrakEM2 [12••]	2D	✓	✓	RAM/unlim h	Stitching Registration	✓	2a	t2.ini.uz.ch	[11•,12••,33–35]
CatMAID [53]	2D	✓	✓	Unlima				fly.mpi-cbg.de/~saalfeld/catmaid
Mishchenko, 2009 [38]	2D	✓		Unlima		✓			[10•]
Raveler	2Dd	✓		Unlima		✓			[54]
V3D/Vaa3D/Janelia3D [40••]	2D, 3Dg	✓	✓	RAM/unlima	Stitching Registration	✓[55]	2b	vaa3d.org	[15•,56 ,57]
Ilastik [42]	2D, 3D	✓[43]		RAM		✓		ilastik.org	[44]
Neurotrace [41]	2D, 3D	✓		RAM		✓		neurotrace.org
SSECRETT [41]	2D, 3D		✓	Unlima
KNOSSOS [24••]	3D		✓	Unlima			2c	knossostool.org	[6•,24••]

^aNavigation and annotation volumes not limited by RAM but by disk storage; for V3D the RAM limitation is waived by the map-view plugin.

^bMany original studies used this tool; see the cited reference far a review.

^cAll listed reconstruction tools aim at providing 3D objects. The distinction made here between 2D and 3D is however based on the mode of annotation, and depends on the anisotropy of the employed data sets (s. Text and Figure 1)

^dThe software provides orthogonal views and 3D object surfaces, but does not allow 3D skeleton annotation.

^eReconstruct provides 3D surfaces 'on-the-fly'.

^fThe table lists only published or publicly available tools. Further software is currently under development.

^gThe software in addition offers to navigate in color and time, coined '5D' by the authors.

^hRAM-limited for single tiles, virtually unlimited number of tiles and sections.

Figure 2.

Examples of successful reconstruction software for large-scale EM and LM data sets. (A) Snapshot of TrakEM2 [12], which is especially suited for ssTEM analysis (s. Text). (B) Reconstruction of 2 neurons in a fly brain using V3D [40], which was designed for whole-brain LM data. (C) Snapshot and neuron reconstruction (inset) using KNOSSOS [24], which was tailored to 3D EM data from SBEM or FIB/SEM experiments. See Table 1 for an overview of available reconstruction tools and their versatility. Images courtesy of A. Cardona (A) and H. Peng (B).

Analysis of serial section-based EM data

Most studies that have successfully used electron microscopic imaging for the analysis of neuronal connectivity [5,10,25–27] cut the tissue of interest into hundreds or thousands of very thin slices, at a thickness of typically 40–90 nm. Then, these slices are imaged using transmission electron microscopes (TEMs), which provide an in-plane resolution of usually 4–8 nm, because electrons of very high energy can be used to provide sample contrast.

An automated serial sectioning approach to EM was developed a few years ago [8]. This approach (AT(L)UM) attaches a tape-collection mechanism to a conventional ultramicrotome, which provides a more reliable method for picking up the many ultrathin slices produced in ssEM, and reduces the slice thickness to up to 30 nm or less. Since the conveyor-belt tape is not electron-transparent, the slices are imaged using scanning electron microscopy (SEM). This method yields a voxel size of typically 5–10 nm × 25–30 nm.

Thus, when using ssEM, data sets are usually highly anisotropic in resolution: very high in-plane resolution, but up to a factor of 10 less resolution across planes. This method and its associated anisotropy have several consequences for data analysis.

1. Sequential images have to be aligned to each other, since they are taken independently from many physical sections, and currently this typically requires manual interaction with the data;

2. Distortions due to folding, stretching and shearing have to be corrected, typically by warping algorithms involving non-affine registration;

3. Data browsing has two modes: one is laterally panning across a large single-slice image with the need to zoom in and out from a resolution of a few nanometers to several micrometers, the other is flipping through subsequent images.

4. Data annotation is almost exclusively in-plane, or two-dimensional, since the resolution is much higher in-plane. Typically, neurites are outlined as contours in one image, and then identified again in the subsequent properly aligned slice. This inference from one slice to the next turns the reconstruction into what one might call 2½ -dimensional data annotation (Figure 1A).

Figure 1.

Schematic of reconstruction modes for large-scale EM or LM data, depending on the degree of anisotropy of the image data. (A) When the in-plane resolution is substantially higher than that across planes, reconstruction is done in-plane, and structures are followed into the adjacent sections. This is the typical reconstruction mode for ssTEM, (s. Text), and amounts to a “2 1/2 D-” reconstruction.

(B) When voxel sizes are close to isotropic, neurites can be reconstructed in 3D. This is the typical reconstruction mode for SBEM, FIB-SEM, and some LM datasets, (s. Text). The most widely employed software for both types of reconstruction is listed in Table 1. Note that both reconstruction modes eventually aim at reconstructing three-dimensional objects, right panels.

Several software packages have been developed and successfully applied to ssEM data analysis (Table 1, Figure 2). The pioneering Reconstruct software by Fiala et al. [28] was used for numerous EM studies addressing synapse and spine geometry [10,29,30], and the reconstruction of neurite fragments [31]. It provides alignment and surface reconstruction routines, but is limited by the amount of data that can be loaded into the main computer memory.

A successful recent software package that overcomes these memory limitations is the TrakEM2 software by Cardona and et al. [12], which also incorporates good alignment and stitching routines [32]. It has been used in several recent studies addressing neuronal connectivity in fly larvae [12,33–35] and mouse cortex [11].

Methods that attempt to automate neurite reconstruction from anisotropic data or under the assumption of anisotropic neurites (running in one preferred direction) by imitating the contour detection-and-propagation process are under development [36–38], and are starting to be applied to neurobiological studies [10].

Analysis of blockface-imaging based EM data

Methods for the automated imaging of blocks of nervous tissue in the electron microscope have also been developed in recent years (cf. also the review by Briggman and Bock in the same issue). These methods require the en-bloc staining of the tissue, which is then transferred into the electron microscope, where the surface of the tissue block is imaged by a scanning electron beam. Next, the top of the tissue block is abrased using either a diamond knife (SBEM [6, 7]) or a focused ion beam (FIB-SEM [9]), and the newly exposed surface is imaged again.

These imaging methods yield a much more isotropic voxel size: resolution in z is mostly limited by the cutting thickness, which is currently 25 nm for SBEM [6] and typically 5–8nm for FIB-SEM [9]. The in-plane resolution is currently 12nm for SBEM, and up to 4nm for FIBSEM. Since the images are taken from the surface of the tissue block before the surface is cut off, images are usually very well aligned, and require much less post-processing, if any. However, since the field of view of the scanning electron beam is limited, image acquisition is typically tiled, requiring the post-hoc stitching of images in each imaging plane.

As a consequence, the analysis of such data requires:

1. Lateral alignment, usually translation-only, and stitching;, both can usually be done automatically without user interaction.

2. Fully 3D data navigation, either by displaying the data virtually sampled in three orthogonal planes, or by providing an oblique image plane.

3. 3D data annotation (s. Figure 1B). Data annotation can either be done by contouring the neurite walls; this resembles the analysis of ssEM data. However it is not trivial anymore to decide in which of the orthogonal image planes to best annotate a given structure. Alternatively, the reconstruction can be restricted to a center-line reconstruction, which is especially appropriate for the linearly shaped neuronal processes. This kind of annotation, also called skeletonization, has the advantage that each point marked along the center line can be placed in any of the imaging planes, making the annotation fully 3D.

The published non-commercial software tools dedicated to full-3D analysis of SBEM or FIBSEM data include ITK-SNAP[39], V3D [40], Ssecret [41], KNOSSOS [24], and Ilastik [42,43] (Table 1). Of these, KNOSSOS and Ssecret are fully independent of the size of the data set, since they load only the currently viewed segment of data into main memory, permitting usage on laptops (at least in the case of KNOSSOS). KNOSSOS is dedicated to skeleton reconstructions, but does not provide a volume annotation option. ITK-SNAP permits volume labeling only, and contains the snake algorithm for semi-automated reconstruction. V3D provides options for both volume and skeleton annotation, and Ilastik features a semi-automated labeling method for volume annotation based on the asymmetric watershed algorithm.

Circuit reconstruction from blockface EM data has recently been successfully applied to the direction-selectivity circuit in mouse retina [6], using KNOSSOS for neuron reconstruction, and ITK-snap for synapse labeling. Ilastik has been applied to synapse detection in mouse neocortical neuropil [44].

Independent of the employed methods for imaging and reconstruction, the reliability of the obtained results must be critically assessed. So far, experts have been mostly assumed to be able to correctly analyze anatomical data, if only enough time was spent on a given specimen. While this assumption is likely to be true for sparsely stained neurons, errors made even by experts become critical in dense large-scale manual reconstructions. This has only recently been fully recognized, and is being resolved by either proof reading [10,38,45,46], or by repetitively reconstructing the same neurons by different users, followed by a statistically justified consensus procedure [24].

Analysis of large-scale light-microscopy data

Complementary to the high-resolution but so far limited-volume EM-based reconstruction efforts, initiatives to map long-range neuronal connectivity using light-microscopic labeling of single neurons or small populations of neurons imaged in entire brains are also being pursued [14,47,48]. Strictly speaking, current LM based approaches (unless based on transsynaptic viruses, or combined with other methods) map the morphological properties of neurons, from which connectivity information is inferred.

The methodology and the data challenges depend on the size of the brain. For the small drosophila brain, individual neurons spanning a spatial extent comparable to the whole brain can be scanned using confocal microscopy or two-photon microscopy after rendering the brain transparent. A published LM-based atlas of a collection of neurons from the drosophila brain is now available [49], and algorithmic development for co-registration of individual fly brains to digital atlases is well under way [15].

Mapping connectivity in larger brains (e.g. mouse) poses significantly increased challenges [14,50]. Physical sectioning is currently required to visualize the whole brain, although imaging methods in which the brain is made transparent are also being currently developed [51]. Light microscopy is performed on a series of optical sections, and parallels can be found to the serial section EM and serial blockface EM methods.

The analog of the serial sectioning method is familiar from classical neuroanatomical work, and in this method thin tissue sections are cut with a cryomicrotome (~5–50 µm) and placed on glass slides. A tape transfer technique has also been developed [52] and is being used for high-throughput processing, although full automation remains as a future goal. The neurons may carry fluorescent label (produced using injections of fluorescent tracer substances or viruses, or suitable genetic constructs) or are suitable for brightfield imaging (in the case of tracer substances subjected to immunohistochemical processing). The slides are imaged using slide-scanning microscopes, which vary in scanning capability.

In this approach, the resolution is anisotropic (x-y resolution is 0.5 – 1 µm in the imaging plane, while the z-resolution is in practice set by the section thickness (~5–50 µm, with a thickness of 20–25 µm being used in the high-throughput projects). It is possible to gather multiple z-stacks within a physical section to improve z-resolution, but it is impractical to do this currently for whole brains. It is also possible to do confocal microscopy on the slides but this also remains difficult to do routinely on large scale. The anisotropy of the image voxels is thus comparable to that of ssEM data (s. above).

A second approach parallel to SBEM has also been developed for whole brain optical imaging, by removing tissue sections from a block using a vibratome, while performing imaging on the surface of the block (P. Osten, pers. communication, http://www.tissuevision.com/). Using two-photon microscopy, this approach leads to a more isotropic voxellation of the brain (e.g. 2 µm × 2 µm × 2 µm voxels), however this comes with a significant time penalty. In practice, the data sets being gathered by this method are also highly anisotropic, with comparable resolution to the serial sectioning approach described above. The blockface sectioning method reduces the distortions between subsequent sections and thus largely eliminates the section registration and morphing problem which needs to be addressed for the serial section method (although the latter is somewhat ameliorated using the tape transfer method for serial sectioning).

Another emerging area is the direct quantification of counts and densities of cells at the whole brain level. While this has traditionally been in the domain of stereological quantification using computer assisted manual techniques, the current drive in gathering whole brain data sets has led to the need for purely algorithmic approaches.

The analysis pipeline for whole-brain LM-based neuroanatomy projects ideally has a set of stages including:

1. Preprocessing for quality control purposes.

2. Registration of optical sections to each to assemble a whole brain.

3. Registration of whole brains to each other, or to a common reference atlas.

4. Segmentation and quantification of cell bodies, fragments of neural processes, or in special cases reconstruction of whole neurons.

5. Presentation of the processed data as well as raw images on the web through multi-resolution viewers, on database-backed web portals that serve experimental metadata.

6. Integration with other related online resources, including neuroanatomical databases and the published literature. In contrast with genomic data sets, the data volumes are too large to be downloaded over the internet for local processing so we might also see the advent of cloud-based collaborative processing of these data sets.

Conclusion and outlook

With the publication of large-scale EM reconstruction as well as major LM-based circuit mapping projects under way, neuroanatomy has entered a new, computationally driven and enhanced phase. In the EM case, almost all of the analysis published to date has been manual, amounting perhaps to ~10⁴ work hours. It is clear that automated analysis still requires further advances to substantially reduce the amount of manual labor involved. The data analysis challenges in computational neuroanatomy are not of the nature that a single algorithmic breakthrough will resolve all major issues; the underlying tasks are complex and multi-faceted, and we expect gradual performance increases. Efficient data annotation by optimized interaction between machines and humans can be expected to play a significant role in the near future.