Automation of 3D reconstruction of neural tissue from large volume of conventional Serial Section Transmission Electron Micrographs

Abstract

We describe an approach for automation of the process of reconstruction of neural tissue from serial section transmission electron micrographs. Such reconstructions require 3D segmentation of individual neuronal processes (axons and dendrites) performed in densely packed neuropil. We first detect neuronal cell profiles in each image in a stack of serial micrographs with multi-scale ridge detector. Short breaks in detected boundaries are interpolated using anisotropic contour completion formulated in fuzzy-logic framework. Detected profiles from adjacent sections are linked together based on cues such as shape similarity and image texture. Thus obtained 3D segmentation is validated by human operators in computer-guided proofreading process. Our approach makes possible reconstructions of neural tissue at final rate of about 5µm³/man-hour, as determined primarily by the speed of proofreading. To date we have applied this approach to reconstruct few blocks of neural tissue from different regions of rat brain totaling over 1000µm³, and used these to evaluate reconstruction speed, quality, error rates, and presence of ambiguous locations in neuropil ssTEM imaging data.

1. Introduction

Knowledge of map of synaptic connections between neurons in brain is important for understanding brain functions. Despite rapid advancements in the fields of fluorescent light microscopy and sub-diffraction-limited light microscopy, to date the only method proven to allow reconstructions of synaptic connectivity in dense population of neurons down to the level of individual synapses remains serial section electron microscopy (Briggman and Denk, 2006). The only complete wiring diagram of a neuronal circuit in existence is that of C. Elegans in (White et al, 1986; Durbin, 1987; Chen et al, 2006), produced using Serial Section Transmission Electron Microscopy (ssTEM). In ssTEM biological specimen to be imaged is sliced into ribbon of thin sections that are stained with heavy metal (usually uranyl acetate and lead citrate) and sequentially photographed in electron beam, thus producing 3D imaging data in the form of a stack of 2D images with nm resolution. Stain agent is selectively picked by different ultra-structures inside cells, such as lipid membranes, that makes external cell membranes and internal organelles visible in electron micrographs (Hayat, 2000).

For decades applications of ssTEM to reconstructions of neuronal circuits had been stalled by labor-intensity of the process of electron micrographs analysis. Reconstruction of a circuit of 300 neurons and fewer than 10,000 connections in C. Elegans, performed manually, took over a decade to complete (White et al, 1986). As of today, ssTEM data are processed manually with minor help from computerized tools such as Reconstruct, IMOD, Amira, etc., which rely on manual segmentation of images while keeping track of nomenclature and images registration. Although with such tools routine reconstructions of single neurons and sets of neuronal processes became possible, it also became evident that for large-scale neuropil reconstructions from ssTEM a capacity for computerized processing of images is necessary beyond what is currently available (Carlbom et al, 1994; Bertalmio et al, 1998; Vazquez et al, 1998; Fiala and Harris, 2001; Briggman and Denk, 2006).

Revived interest in acquisition of large scale EM imaging data for reconstruction of neuronal circuits had stimulated development of new computerized tools for its high-throughput analysis (Jurrus et al, 2006; Jain et al, 2007; Macke et al, 2008). These studies focused on automation of tracing of neuronal processes in stacks of images acquired with Serial Block-Face Scanning Electron Microscopy (SBFSEM) and stained using cell membrane targeted HRP-DAB reaction (Denk and Horstmann, 2004). High performance and substantial savings in reconstruction time had been reported (Macke et al, 2008). At the same time little effort had been devoted to development of such tools for conventional ssTEM even though substantially larger amount of data is available with higher resolution and richer ultra-structural contents (see though (Jurrus et al, 2008)).

In this paper we develop an approach for automation of reconstruction of neural tissue from conventional ssTEM. This problem is substantially different from analysis of images from SBFSEM: ssTEM images are acquired with much higher resolution 2–4nm/pixel, as opposed to 20–30nm/pixel in SBFSEM, that makes their analysis more straightforward. On the other hand, conventional ssTEM has disadvantage of high degree of edge clutter in images caused by staining of intracellular structures, and larger sections thickness of about 50nm, as opposed to 25–30nm in SBFSEM.

We organize the process of reconstruction as a sequence of steps (Supplemental Figure 1). First, we use modern image processing approaches to analyze individual images in ssTEM stacks and extract profiles of different neuronal processes. Second, we link together profiles that belong to the same neuron across different images in ssTEM stack. This process is fully automatic, can be easily scaled and requires little human intervention. Thus produced 3D segmentation data is presented for validation to human operators, who check validity of reconstructions of individual neurons and annotate them. With this approach we have segmented to date over 1000µm³ of neuropil samples from hippocampus and cortex of rat. Final reconstruction rate was about 5µm³/man-hour (i.e. ssTEM stack of (10µm)³ can be finalized in about two months (Mishchenko, preprint)).

The paper is organized as follows. The main text outlines our method with the emphasis on encountered problems and approaches to their solution, while details of specific implementations are placed in the appendix in supplementary materials. Section 2 discusses choice of architecture for automated reconstruction system, processing of individual EM images, extracting neuronal cell profiles, and associating extracted profiles across different serial sections into 3D models of neuronal processes such as axons and dendrites. Section 3 discusses errors characterization and errors propagation during reconstruction process, as well as data validation, proofreading, and frequency and impact of ambiguities in ssTEM data caused by thin axons running obliquely to section plane, as evaluated from our reconstructed samples of neuropil. Section 4 provides summary and discussion. Appendix A in supplementary materials provides details of our algorithm for detection of neuronal cell profiles in EM images using multi-scale ridge detector and anisotropic contour completion. Appendix B in supplementary materials provides details of grouping of neuronal cell profiles into 3D objects.

2. Materials and Methods

2.1. Overview of automated reconstruction process

ssTEM datasets have certain specific features making the problem of their analysis distinct from and more complex than existing medical imaging applications. These differences should be considered when deciding on the architecture for automated reconstruction system.

Typical state-of-the-art ssTEM are taken with 50nm thick serial sections and resolution of images (XY-resolution) of 1–5nm/pixel, thus featuring a very large aspect ratio. This results in asymmetry present in many aspects of the data: significantly more information is available from images of individual serial sections than across them (e.g. cell boundaries can be seen well in individual images but not in the re-sliced XZ images). In ssTEM serial sections are physically separate objects and suffer physical distortions during cutting and handling, that requires advanced registration to recover 3D continuity; also, occasionally physical damage may occur such as folds, burns, tears and lost sections resulting in total loss of information. The features carrying segmentation information in ssTEM are thin 10–20nm cell boundaries that are more vulnerable to noise than volumetric signal used in medical applications such as MRI. Furthermore, conventional sample preparation procedure stains cell boundaries indiscriminately complicating extraction of cell profiles by introducing heavy edge clutter from numerous stained internal structures. Finally, reconstruction process should span many orders of magnitude in scale as neuronal processes should be traced at resolution 5–10nm over 100–1000µm of their length.

Registration of ssTEM images is the first important part of reconstruction process, given the large amount of distortions introduced into EM sections during handling. In this work we used software Midas for registering ssTEM images, a part of IMOD tomography package publicly available from Boulder Laboratory for 3D Electron Microscopy of Cells (http://bio3d.colorado.edu/imod/). We found that capacities of this software were generally adequate except for occasional local distortions in the images, which required local fine-tuning (Chklovskii, private communication). In this work we do not focus on the issue of ssTEM images registration, and assume that the stack of aligned images has been previously prepared for 3D segmentation.

For segmentation, given large aspect ratio of the dataset and intrinsic disparity between XY and Z dimensions, we adopt a 2+1D architecture where we treat ssTEM dataset as a sequence of 2D frames rather than truly 3D volume. We begin analysis of a stack of ssTEM images by first processing them independently. During this step segmentation of each image into profiles of individual neuronal processes is produced. Segmentation of 2D images is based on local detection of external cell boundaries and the condition that cell boundaries should form closed contours. Thus found cell profiles are grouped across images to produce 3D models of neuronal processes. Each cell profile is viewed as a 2D slab-like elementary block from which 3D models are assembled as stacks of slabs. To group cell profiles into 3D objects, for every pair of overlapping cell profiles from adjacent images in ssTEM stack we calculate affinity based on set of features describing shape and image texture within the overlap.

Results of automatic 3D segmentation typically contain a number of errors and should be validated by human. For the purpose of validation of automatically segmented datasets we developed a specialized graphical user interface, which we call the Proofreading Tool (PRT). PRT facilitates inspection of automatically segmented volumes of neuropil by placing this process into structured and systematic framework. The user is guided by PRT in systematic manner through the list of objects that should be inspected. The list is prepared and ordered by PRT in a way that minimizes chances for committing significant errors. The user interface is optimized to streamline working with pre-segmented data, and perform corrections and annotations rapidly.

2.2. Segmentation of neuronal cell profiles in electron micrographs

Pre-processing of individual images in ssTEM stack is a natural first step given such features of ssTEM as large volume of data and large discrepancy between XY and Z dimensions. Two main goals here are taking advantage of high XY-resolution and reducing the volume of data by abstracting from voxel to segment-based representation (typically reducing volume of data by 3–4 orders of magnitude).

Segmentation of individual ssTEM images into profiles of neuronal processes is complicated by many factors. In particular, naïve application of generic segmentation methods such as thresholding, Markov Random Fields (MRF), graph cuts, active contours, or watershed typically fails to produce good results. Methods such as thresholding or MRF tend to identify sharp, well pronounced boundaries, but frequently fail to locate weak boundary fragments disrupting many cell profiles. Approaches such as watershed and active contours require initialization, but also get stuck on weak false boundaries inside cell profiles. Overall, generic methods tend to segment well large dendritic shafts but fail for larger number of smaller and less distinct axons. Thus, more specialized segmentation approach may be deemed necessary. We developed such specialized segmentation approach consisting of a sequence of steps as described in Supplemental Figure 2.

Extraction of profiles corresponding to cross-sections of neuronal processes in ssTEM images may be posed as the problem of finding boundaries of neuronal processes, which appear as dark ridges forming irregularly shaped loops (Figure 1, Figure 3 and Figure 5). Detection of ridges is the classic problem of machine vision with variety of approaches available ranging from simple linear filtering (Verndon, 1991), to energy minimizing active contours (Kass et al, 1988; Laptev et al, 2000), to complex boosted classification schemes on large number of features (Zheng et al, 2007). We experimented with few ridge detection schemes but found no clear evidence of more complex algorithms performing substantially better in ssTEM images than simpler ones. We use Gaussian smoothed Hessian (GSH) to identify ridges in ssTEM images

$$H_{\sigma }[I]=C(\sigma )\left[\begin{array}{cc}{\partial }_{xx}{G}_{\sigma }& {\partial }_{xy}{G}_{\sigma }\\ {\partial }_{yx}{G}_{\sigma }& {\partial }_{yy}{G}_{\sigma }\end{array}\right]*I.$$ (1)

Figure 1.

Typical image from conventional ssTEM dataset. Inset illustrates variable appearance of plasma membranes, such as width and contrast, due to varying orientation with respect to the electron beam. All images here and below are from 150µm³ ssTEM stack P21AA (Fiala, 2003). Indicated structures are cell boundaries (B), cytoplasm (C), synapses (S) and internal organelles such as vesicles and mitochondria (O).

Figure 3.

Illustration of detection of cell profiles in ssTEM images. A) A typical image from conventional ssTEM dataset. B) Map of best-scale Hessian eigenvalues difference. C) Map of learned boundary detection confidences from neural network based detector. D) Map of learned high-confidence ridges used as seeds for anisotropic contour completion. E) Map of boundary detection confidences after learned anisotropic contour completion from high-confidence ridges. F) Final map of closed contours corresponding to generated cell boundaries.

Figure 5.

Illustration of 3D grouping process. A) and C) are examples of two adjacent images from ssTEM stack P21AA. B) and D) are higher magnification sub-regions of these images (shown in A and C with red square). “*” and “x” show two contours that produce confusing overlap in F. E) Cell boundaries detected in images A and C overlaid on top of each other. Darker shade is for image A, and lighter shade is for image C. White corresponds to places where boundaries match. F) Location where boundaries from adjacent images do not match, thus producing an overlap that should not be used for grouping, indicated with “x”, as opposed to overlap indicated with “*”.

Here I(x⃗) stands for image grayscale intensity, G_σ ( x,y ) =( 2 πσ) ⁻¹ exp(− (x² + y²)/ 2σ² ) is the Gaussian smoothing kernel with scale σ, and * is the convolution operator (H * I)(x⃗)=∫ dx⃗′ H(x⃗ − x⃗′)I(x⃗′). C(σ ) is a scale-dependent constant to be discussed later. Ridges in images are commonly associated with points such that Hessian matrix has one eigenvalue large positive and the other small positive or negative, then strength of the ridge may be represented in terms of the difference of GSH eigenvalues L = λ_max − λ_min , where λ_min and λ_max are eigenvalues of GSH with smaller and larger absolute values, respectively.

Note that detection of ridges is sensitive to the choice of smoothing scale σ. In particular, ridges that are best detected are those that have width close to σ: ridges that are either too narrow or too wide will be either smoothed away or will produce weak response. At the same time, in ssTEM images cell boundaries often appear in different configurations with widely varying width (inset in Figure 1), which implies that smoothing scale of GSH should be adaptively tuned by the algorithm to the width of the local cell boundary at every point. For selection of the best scale the Maximal Scale Ridge (MSR) principle had been suggested in the past in the context of wavelets and scale space analysis (Fritsch et al, 1995; Lindeberg, 1998; Carey et al, 1999; Lindeberg, 1999). In MSR for each pixel and each scale the ridge response L is scale-normalized and the scale with the largest normalized response in absolute value sense is selected. Ridge strength should be scale-normalized because measurements performed at different scales may not be immediately comparable: because ∑ H_σ (Δx) = 0 (summation is over the kernel domain, and Dx is a point in the kernel domain), GSH admits different normalizations such as C( σ) = 1 (G_σ unity normalization), C(σ) = 2 / ∑| H_σ (Δx)| (bandpass normalization), C(σ) = σ^3/4 (scale space analysis of ridges (Lindeberg, 1999)), C(σ) = σ² (derivatives dimension (Lindeberg, 1993)), etc. In our algorithm we use simple band-pass normalization: we experimented with few choices for scale-normalization but did not find clear evidence of different prescriptions performing substantially and consistently better than the others.

To describe boundary detection confidence p(x) at point x, representing the posterior probability that a point in the image belongs to a cell boundary given best-scale GSH, we introduce function F which depends on GSH eigenvalues λ_min and λ_max at best scale and image intensity I in a 3×3 neighborhood 𝒩 of that point,

$$p(x)=F({\lambda }_{\text{max}}-{\lambda }_{\text{min}},{\lambda }_{\text{min}}/{\lambda }_{\text{max}},I).$$ (2)

Specific form of the function F is learned from the set of few annotated ssTEM images. This approach is similar to machine learning based boundary detection scheme proposed in (Martin et al, 2004; Zheng et al, 2007). Specifically, we represented function F with a single layer artificial neural network implementing logistic regression (Figure 2). This neural network was then trained using Matlab neural network toolbox to detect position and full width of cell boundaries on few manually annotated images. An example of these stages of detection of boundaries is illustrated in Figure 3A–C.

Figure 2.

Architecture diagram for neural network multi-scale ridge detector. A single layer neural network with sigmoid transfer function f operates on a set of image intensities I and max-scale ridge strengths L from N-neighborhood of a pixel 𝒩(x).

ssTEM images often contains many strong edges that are irrelevant for finding cell boundaries but result in strong responses from local ridge detector caused by stained internal structures (Figure 4). Many of such false responses may be eliminated by observing that true cell profiles should always be closed, and by retaining from each image only detected boundaries that form such closed contours. This procedure, however, is complicated by that detected boundaries often suffer from breaks due to uneven stain intensity and/or smoothing and discretization artifacts. Thus, we supplement local ridge detection with contour completion. Completion of partially recovered contours is an active area of machine vision research and many alternative approaches have been investigated in the literature such as anisotropic contour completion (Gil et al, 2003), perceptual grouping (Estrada and Jepson, 2004), energy minimizing paths (Cohen, 2001; Wang et al, 2005), extension fields (Guy and Medioni, 1996), stochastic completion fields (Williams and Jacobs, 1997a), Hidden Markov Models (Ren and Malik, 2002), Conditional Random Fields (Ren et al, 2005), etc. In our algorithm we use a variant of anisotropic contour completion from high-confidence ridges guided by image features and ridge headings.

Figure 4.

Examples of edge clutter in ssTEM image introduced by stained internal organelles. A) Vesicles in axonal terminal. B) Mitochondria inside axon. C) and D) Irregularly shaped internal structures.

Specifically, starting from strong ridges X such that |λ_mn |/|λ_max |≪ 1 we compute quantity D(x) representing the probability that pixel x may be connected to a strong ridge via path consisting entirely from pixels classified as cell boundary. This may be calculated as the largest pixel-wise product of boundary detection confidences over all paths C connecting pixel x to the set of strong ridges X.

$$D(x)=\underset{C(X\to x)}{\text{max}}{\displaystyle \prod _{x\prime \in C(X\to x)}p(x\prime ).}$$ (3)

D(x) may be viewed also as a generalized multiplicative distance function defined such that log D(x) = −T(x), where T(x) is the best propagation time from a strong ridge to point x over the field of velocities V(x) = −1/logp(x). To see why this is the case, rewrite Eq.(3) as

$$\begin{array}{l}D(x)=\underset{C(X\to x)}{\text{max}}{\displaystyle \prod _{x\prime \in C(X\to x)}p(x\prime )}=\underset{C(X\to x)}{\text{max}}\text{exp}\left({\displaystyle \sum _{x\prime \in C(X\to x)}\text{log}p(x\prime )}\right)\hfill \\ =\text{exp}\left(-\underset{C(X\to x)}{\text{min}}{\displaystyle \sum _{x\prime \in C(X\to x)}\frac{\mathrm{\Delta }x\prime }{V(x\prime )}}\right)=\text{exp}(-T(X\to x))\hfill \end{array}$$

where Δx′ = 1 is the displacement along the path C(X → x) in 4-connected topology, and V(x) is the velocity field introduced above.

Such generalized distance function may be computed by propagating wave-front away from the strong ridges X over the specified field of velocities using Fast Marching Method (Sethian, 1998) and recording wave-front arrival times at each point. We additionally introduce local anisotropy in such wave-front propagation by impeding propagation in the direction orthogonal to ridges and enhancing propagation along the direction of ridges. Thus, during each cycle of the wave-front propagation, D(x) is iteratively updated for all adjacent pixels x and x′ as in

$$D(x\prime )\mapsto \underset{<x,x\prime >}{\text{max}}D(x)\cdot f\left(L(x),L(x\prime ),\mathrm{\theta ⃗}(x),\mathrm{\theta ⃗}(x\prime ),{\mathrm{n⃗}}_{x\prime -x}\right).$$ (4)

Function f describes local delay in wave-front propagation from x to x′ and is set to depend on best-scale ridge strength L at points x and x′, collinearity of ridge orientations θ⃗ and θ⃗′, collinearity of ridge orientations with the propagation vector n⃗ from x to x′. The max value is taken over all neighbors < x, x′ > of pixel x′. The specific form of f is chosen to enforce given degree of anisotropy in the wave-front propagation, and is described in Appendix A in supplementary materials. In fact, function f may be learned from manually annotated data. Specifically, we represented f with single layer artificial neural network implementing logistic regression on vector of inputs $\left(L,L\prime ,\mathrm{\theta ⃗}\cdot \mathrm{n⃗},\mathrm{\theta ⃗}\cdot \mathrm{\theta ⃗}\prime ,I,I\prime \right)$ for a pair of adjacent pixels < x, x′ > (here f may also depend directly on image intensities I at x and x′). For each pair of adjacent pixels < x, x′ > such that pixel x was marked as cell boundary in the training dataset, the neural network was trained using Matlab neural network toolbox to predict whether pixel x′ was also cell boundary given input vector above. An example of the result of application of anisotropic contour completion is given in Figures 3D and 3E.

Although described algorithm performs local detection of cell boundaries well, we observed that heavy clutter due to stained internal structures significantly degrades its performance even after functions F and f had been learned from manually annotated dataset. For these reasons we found it critically important in conventional ssTEM to perform active detection of certain types of stereotypical organelles and remove them explicitly. Two most profound such cases are vesicles inside axonal terminals and mitochondria (Figure 4A–B). Vesicles are neurotransmitter carrying spherical organelles composed from single layer of lipid plasma membrane that are usually encountered in large numbers inside pre-synaptic axonal terminal boutons. In ssTEM images they appear as dense clusters of deformed circular shapes of varying contrast (Figure 4A, also Figure 1). Mitochondria are large tubular objects formed by a complex of intricately interlinked plasma membranes and generally are thought to be responsible for energy production in cells. In ssTEM images they appear as dark textured blobs of circular or oval shape (Figure 4B, also Figure 1). Stereotypical shape of vesicles allows their detection to be performed with good quality (Genkin, unpublished). Yet, because a substantial number of false detections typically is still observed on cell boundaries due to noise, to prevent these from breaking continuity of correct cell boundaries we only remove vesicles that are in clusters, such as inside axonal terminal boutons. Clusters are defined as groups of vesicles such that distances to three nearest neighbors for each vesicle in the group are below three respective thresholds. Thresholds usually may be readily identified by inspecting inter-vesicle distances in a single terminal bouton. At this time we do not perform active detection and removal of mitochondria.

The wide range of appearances of cell boundaries in ssTEM images requires relaxed acceptance criteria if critical weak boundaries are to be retained. At the same time, this leads to misclassification of large number of internal structures locally similar in appearance to weak cell boundaries. This suggests use of hysteresis approach where strict criteria are used to select high-confidence boundaries, while weak boundaries are retained if they are necessary to satisfy certain non-local conditions such as contour smoothness or continuation in 3D. Specifically, we divide all detected profiles in two sets of high-confidence and low-confidence profiles. High-confidence profiles are used to form 2D segmentation of each ssTEM image, while low-confidence profiles are retained if they are necessary to match a high-confidence profile in adjacent image. Finally, true cell profiles are selected as the boundaries that form closed contours (Figure 3F).

Using this algorithm we produce segmentation of each image in ssTEM stack into profiles of neuronal processes. We found that such scheme for detection of cell boundaries performs consistently well in ssTEM images – performance of different stages of boundaries detection in our algorithm is described in Table 2. Found profiles should be subsequently grouped into 3D models, as described in the next section. More details on the implementation of cell profiles extraction from ssTEM images can be found in Appendix A in supplementary materials.

Table 2.

Performance quantification for different stages of boundary detection algorithm. Different columns show rates of false positive and false negative pixel misclassification (defined as the fraction of all pixels in the image misclassified as boundary or interior, respectively); rate of merge errors (defined as the number of additional segments that should be created in 2D segmentation to match ground truth, per 100 segments of ground truth); rate of split errors (defined as the number of segments in 2D segmentation that should be merged to match ground truth, per 100 segments of ground truth); and over-segmentation factor (defined as the ratio of the total number of segments in automatic 2D segmentation, per 100 segments of ground truth). Also shown are rates of merge and split errors for 3D grouping algorithm, shown per 1000 contours of ground truth segmentation. Note that, because automatic segmentation often contains large number of extremely small and irrelevant “noise” segments, here we cleared all segments smaller than 5×5 pixels from automatic 2D segmentation before calculating the rates of merge and split errors, but not over-segmentation factor.

	Fp	Fn	Merge-errors	Split-errors	Over-segmentation factor
Hess simple	0.08	0.035	6.6/100	33.9/100	260/100
Hess+comphand	0.10	0.004	2.7/100	51/100	272/100
PNN	0.10	0.020	4.2/100	29.8/100	146/100
PNN+comphand	0.13	0.015	2.95/100	40.5/100	170/100
PNN+compNN	0.16	0.007	1.23/100	52/100	187/100
Final 3D	n/a	n/a	2.5/1000	63/1000	n/a

2.3. Linking of neuronal cell profiles across images in ssTEM stack

2D segmentation of different serial sections should be brought into correspondence by associating together profiles from different images that belong to the same neuronal process. We formulate this process as clustering problem on a weighted graph such that each node in the graph corresponds to single cell profile and each edge corresponds to adjacency of cell profiles either in XY or Z-directions. Weight of an edge, or affinity, corresponds to the likelihood that adjacent profiles belong to the same neuronal process.

Affinities are calculated by examining contact surfaces between adjacent cell profiles. In case two cell profiles are from the same image, the line separating them is the contact surface. Features such as histogram of grayscale intensity and of boundary detection confidence in the vicinity of that line may be used. In case two cell profiles are from adjacent images, the overlap area between two profiles is the contact surface. In this case we collect large feature vector v⃗ for the overlap of their interiors including features describing grayscale histogram, area of overlap, area of overlap for cytoplasm-like regions (clear bright areas inside cell profiles, see e.g. Figure 1), correlation of pixel intensities, etc., as well as features describing the shape of the overlap such as overlap’s width and Hausdorff distance. Width of a shape here is defined as the diameter of the largest circle that can be inscribed inside it, and the Hausdorff distance between parts of two profiles A and B forming the overlap is defined as

$$d_H(A,B)=\text{max}\left(\underset{x\in A}{\text{max}}\underset{y\in B}{\text{min}}|x-y|,\underset{y\in B}{\text{max}}\underset{x\in A}{\text{min}}|y-x|\right),$$ (5)

and measures the largest distance that a point from A (or B) needs to travel along the shortest path to reach a point in B (or A). In other words, d_H (A,B) measures the largest deformation that boundary A and B should suffer to match each other.

Hausdorff distance between two profiles is used to enforce condition that changes in the position of cell boundary caused by physical motion from one EM section into the next should be small, but also to address the possibility of “jitter” in cell boundary position due to minor discrepancies in alignment. Thus, shape-based features are helpful in evaluating cases where changes in cell boundary from one image to the next are either small or large. In the intermediate regime, features describing the image in the region enclosed by cell profiles in two subsequent sections become of greater importance. E.g., if such region really corresponds to a space swept by cell boundary moving obliquely from one section into the next, it should be dark and have specific texture.

We define probability in fuzzy-logic sense P(v⃗), depending on the set of features v⃗, that overlap between two cell profiles corresponds to oblique cell boundary and, thus, two cell profiles should be kept disjoint. Specific choice of function P and feature-vector v⃗ is described in detail in Appendix B in supplementary materials. Also, function P may be learned from manually segmented 3D dataset, similarly to learning functions F and f in Section 2.2, although we did not investigate this possibility in detail. Then, the probabilities P(v⃗) are thresholded and cell profiles are grouped into 3D objects by selecting thus defined connected components over the set of 2D cell profiles. To implement such clustering we used modification of Hoshen-Kopelman algorithm (Hoshen and Kopelman, 1976). Briefly, Hoshen-Kopelman algorithm crawls through the set of all 2D cell profiles in the graph and groups them into equivalence classes whenever two profiles are “connected” according to P(v⃗). Whenever two or more equivalence classes are found to share a common profile, a “merge event” is recorded. “Merge events” are resolved after forward pass has finished.

This process of grouping of 2D cell profiles is illustrated in Figure 5, where we give an example of associating profiles from two adjacent images in ssTEM stack. In each image cell profiles are first detected and then compared (Figure 5A, C and E). In Figure 5E such comparison is illustrated in detail: cell profiles from one image are drawn with darker shade of gray, while that from adjacent image are drawn with lighter shade of gray. Locations where boundaries match – such as boundary is present in both images – are indicated with white. As can be seen from Figure 5E, most of the boundaries from adjacent images match well, thus, allowing to map contours from one image into the next unambiguously. In some cases boundaries fail to match exactly, such as in Figure 5B, D and F. In this case the confidence P(v⃗) should be evaluated that overlap is an erroneous overlap between cell-profiles of nearby neuronal processes (indicated in Figure 5F with “X”) rather than a true overlap between interiors of two adjacent cell-profiles (indicated in Figure 5F with “*”). Once such confidences had been evaluated for each pair of overlapping profiles, associations between cell profiles from adjacent images are established. This process is then repeated for all adjacent images in the stack, and 3D models for neuronal processes are thus recovered.

Because our 3D grouping is based on the assumption of explicit 3D continuity of objects in ssTEM images, this approach may be rather sensitive to errors in alignment. The amount of misalignment between adjacent images that usually may be tolerated corresponds to the typical width of cell boundaries, such as in Figure 5E and 5F. This tolerance may be further improved if we are careful not to group contours that have overlaps with small Hausdorff distance d_H (A,B). Of course, such more conservative grouping criterion will also result in larger number of breaks in 3D traces of thin neuronal processes. In general, however, we did not find that misalignment errors were of the foremost concern for 3D segmentation, as relatively modest amount of final proofreading time spent on segmentation errors caused by misalignment demonstrates (column “alignment” in Table 1).

Table 1.

Proofreading time breakdown for ssTEM stack of 12 sections and total size of 50µm³ containing about 700 different objects. Different columns correspond to time associated with performing different tasks such as simply inspecting reconstruction (“none”), tracing through very difficult place in neuropil (“solve”), correcting errors caused by fault in images alignment (“alignment”), correcting over-segmentation caused by vesicles in axonal terminals (“vesicles”), correcting generic over-segmentation (“over-segmented”), re-drawing of cell boundaries (“drawing”) and correcting merged neuronal processes (“mergers”). Rows show the total number and percentage of cases, in terms of distinct fragments encountered, and mean, max and total time spent performing each task.

V=50µm3	None	solve	alignment	vesicles	over- segmented	drawing	mergers	total
Cases	407	51	6	88	124	23	10	709
Cases,%	57%	7%	0.8%	12%	18%	3%	1.4%	100%
Mean time per case	20s	111s	147s	66s	54s	48s	57s	41s
Max time per case	163s	315s	366s	275s	303s	94s	186s	366s
Total time	2.3h	1.6h	0.2h	1.6h	1.9h	0.3h	0.2h	8.1h
Total time,%	29%	20%	3%	20%	23%	4%	2%	100%

Described procedure suffices to recover majority of 3D structures from ssTEM dataset. Nonetheless, thus produced 3D segmentation often contains large number of errors due to image artifacts such as broken cell boundaries, misclassification of internal structures, or obscure locations in the images. 3D segmentation, therefore, should be subsequently validated by human operator, as described in next sections. More details on implementation of 3D clustering of cell profiles can be found in Appendix B in supplementary materials.

3. Results

3.1. Characterization of performance of automated reconstruction approach

Variety of performance measures may be introduced to describe quality of automated segmentation, however, we want such measures to reflect the final objectives of ssTEM reconstruction. In this light the measures of segmentation quality reflecting deviations in the contents of segmentation are of greater significance rather than such that are sensitive to the shapes of the objects or accuracy of classification of voxels. Two such measures are rates of split errors, i.e. one neuron erroneously split into two or more fragments, and rates of merge errors, i.e. two or more neuronal processes merged into single 3D object.

Although both types of errors have similar impact on final reconstruction, merge errors have substantially worse accumulation properties and are significantly more difficult to correct. Specifically, merge errors tend to only accumulate as more images are processed: i.e. it is sufficient to erroneously merge two cell profiles at single location to merge processes along their full lengths. Furthermore, because probability that a merge error would occur is approximately proportional to the number of neighbors of a given 3D object, and because when two objects merge such number of neighbors only grows, merge errors tend to accumulate exponentially fast with significant fraction of entire volume collapsing into single 3D cluster if certain percolation threshold is exceeded. The value of this threshold is as smaller as the bigger total volume is considered. Split errors, on the contrary, usually only dissipate as more images are processed, i.e., unless neuron is similarly over-segmented in a series of consecutive images such errors will usually disappear during 3D grouping. Split errors are also easier to identify and correct since location of each error usually is evident.

Given that, our automatic segmentation was set to operate at extremely low merge-error rate at the cost of higher over-segmentation. We try to keep rates of merge errors below 1 in 100–400 contours, while over-segmentation rate in single image may be as high as 10 to 1. Large fraction of such over-segmentation may be eventually removed during 3D grouping, resulting in final persistence length for typical axonal shafts of 5–10 serial sections. Note that this is in agreement with the persistence length expected from abundance of obliquely running thin axons which would be difficult to trace using local image cues that we utilize (see Section 3.3 for details).

When characterized by rates of erroneous split and merge errors (Table 2), our 2D segmentation performs at the level of 1–2 merge-errors per 100 contours of the ground truth, while over-segmentation rate is high – about 50 split-errors per 100 contours of the ground truth. Some of these errors are corrected automatically by considering adjacent images, either using hysteresis approach as described in Section 2.2 or during 3D grouping. Because of that, in the 3D reconstruction both rates of split and merge errors are typically lower. We found that the rate of merge-errors in the 3D automatic segmentation was 2–3 errors per 1000 contours of the 3D ground truth, and the rate of split-errors was approximately 60 per 1000 contours (Table 2). This is about 30–60 times higher than the error rates in reconstructions that were performed fully manually (see below).

Validation of automatically processed data, or proofreading, is an integral part of reconstruction process. Proofreading may be characterized by the time necessary to complete reconstruction of a given volume and a measure of the final reconstruction quality. We found that proofreading time scales linearly with the volume, or more precisely with the total number of neuronal cross-sections involved, and was approximately 5µm³/man-hour, or 1–2 hours per 1000 traced contours. To estimate quality of final reconstructions, we performed cross-validation with smaller sub-regions of our ssTEM datasets, totaling 200µm³, reconstructed by other operators independently fully manually using Reconstruct software (Mishchenko et al, preprint). We found that error rates (mostly errors at obscure locations in neuropil) could be described by the fraction of cell profiles that were incorrectly traced or linked in 3D. For our final reconstructions this rate was 0.2–0.4% or 1 error per 200–400 traced contours, and was close to that observed for reconstructions performed fully manually – 0.1–0.2% or 1 error per 500–1000 traced contours. Discrepancy in the shapes of traces was insignificant – on average within 1–2 pixels. Frequencies of errors in different types of processes showed clear correlation with processes smallest width: errors were more frequent in thin spine necks and sections of axons, while wide segments of axons and dendritic shafts typically contained no errors at all (Mishchenko et al, preprint).

Note that both the reconstruction speed and the reconstruction quality depend on neuropil characteristics (figures here are for a sample of hippocampal neuropil in rat) such as density of neuronal processes, distribution of their sizes and headings anisotropy. In particular, the reconstruction speed measured per 1000 traced contours may be more accurate proxy for the scale of necessary proofreading effort than the speed per µm³ of neuropil volume, if neuronal processes are substantially smaller than that in rat neuropil. In fact, final reconstruction speed and error rates should be independently evaluated each time parameters of neuropil sample might have changed substantially.

We also found that abundance of thin axons in ssTEM resulted in substantial number of configurations which were very difficult to interpret for human operators (column “solve” in Table 1). Although most of such thin axons could be traced eventually based on heading cues and the context of nearby neuropil, situations also occurred where such thin axons changed heading unexpectedly (Figure 6) or two thin axons came together obliquely and at the same time had similar heading. Such configurations were extremely difficult to reliably interpret even by an expert after inspecting images many times. Such configurations occurred in our ssTEM datasets with small but substantial frequency: one per 100–200µm³ or 1–2mm of traced axonal wire. Although two or more experts may in fact reach an agreement by consensus on the interpretation of such location, we argue that such situations cannot be solved reliably from ssTEM data and constitute true ambiguity. Consider, e.g. two thin oblique axons that come together with nearly identical heading. Such axons may not be reliably traced based on local cues due to their small size, but then also they may not be reliably traced based on heading cues due to extreme similarity of their headings (see Section 3.3 for more details). It is therefore likely that any interpretation of such location will be incorrect with high probability.

Figure 6.

A–D) Examples of a location in neuropil containing two axons, indicated with “*” and “x” (red contours), which was difficult to interpret by a human operator. The order of images is left to right, and up to down. All images are from ssTEM stack P21AA; traces are from complete reconstruction of this stack performed by the author with described approach. E) 3D reconstruction of these two axons, arrow shows direction normal to EM section plane.

3.2. Data validation and speed of neuropil reconstructions from ssTEM

Automatically segmented output often contains substantial number of errors and requires manual validation. Validation of automatic segmentation, or proofreading, is of paramount significance given long reach and profound impact of each such minor tracing error: if single error is committed in a trace of axon leading to its premature termination or confusion with nearby axon, hundreds of synapses downstream site of error may be potentially lost or misplaced.

Given large amount of data that can be prepared in short time with automatic segmentation algorithms, it is important to develop and use specialized software allowing inspection of data with highest speed and efficiency by utilizing advantages of working with pre-segmented dataset. For this purpose we developed specialized proofreading tool and protocols that allow operators to systematically inspect automatic segmentation of neuropil and correct it as necessary. In this tool computer selects a set of objects in automatic segmentation that are considered “significant”, and operator is instructed to inspect them in the order of decreasing significance. User interface in such tool is of foremost importance. The user interface is optimized to speedup working with pre-segmented data. Particularly, little drawing is involved so that operator typically only groups together 3D fragments of neuronal processes as necessary. Furthermore, user interface is focused and built around the typical operation involved in proofreading, in our case grouping of 3D fragments, and is designed to reduce to the minimum involvement with other tasks. Interface is optimized to facilitate browsing through the list of 3D objects, merge or split fragments of neurons. At the same time, naming, keeping track of changes, searching for relevant neuronal fragments and navigation, etc., are taken over by the computer, and automatically generated nomenclature is used so that operator spends no time for naming and renaming objects.

Proofreading of neuropil segmentation should be systematically organized. This is because even tiny block of neuropil contains thousands of different fragments and it is important to assist operator with organization of proofreading process. We instructed operators to examine entire contents of automatic segmentation in pre-defined order – from larger objects to smaller. Each shown neuronal process was inspected serially through the sequence of all sections in ssTEM stack at once. This is similar to approach used in manual tracing and, in our opinion, guarantees the highest possible quality of final reconstruction. However, it essentially amounts to checking every neuron in every section of ssTEM stack. This limits the speed of proofreading to the time necessary to at least visually inspect all neurons in all images of ssTEM stack. In particular, we observed from our reconstructions that typical number of cross-sections of neuronal processes in one section was 500 per a 100µm² section. Then, assuming that operator spends 1 second to check one profile visually, the time necessary to inspect 200 serial sections in a cube of neuropil of 1000µm³ should be at least 30 hours, or reconstruction speed should be at most ≈30µm³/man-hour. We also investigated another approach where operator was guided exclusively through the locations of errors in automatic segmentation and was not required to inspect neuronal processes in full. We found that in our case this alternative approach did not provide advantages relative to full volume inspection. This is likely because, in order to correct an error, operator often needs to browse through 5–7 adjacent images around location of the error to acquire necessary context information. If the errors in automatic segmentation are made often, the result of this will be that the operator will effectively have to examine the entire volume of reconstruction even if not explicitly asked to do so. Still, different automatic segmentation may favor alternative proofreading setup.

To evaluate bottle-necks of proofreading process we also performed benchmark measuring the time of particular tasks when working with actual data. A block of 12 consecutive serial sections measuring 9mm × 9 mm × 0.6 µm and containing fragments of about 700 neuronal processes was chosen from a larger series (Table 1). We found that nearly 30% of time was spent examining correct or nearly correct neuronal processes, consistent with the rate of 1–2 seconds per process per section. Furthermore, significant time was spent solving complex puzzles of obliquely running axons and spine necks such that involved substantial intellectual engagement from the operator: each requiring as much as 2–6 minutes for the stretch of mere 12 sections. These contributed 20% of proofreading time, although only 7% of all objects. Among the remaining time, the major fraction of effort (40–45%) was spent on axons over-segmented due to vesicles or other general inclusions, which in principle may be reduced with more advanced image processing.

3.3. Frequency of ambiguous configurations in ssTEM data and quality of neuropil reconstructions from ssTEM

Consider a thin axon that runs obliquely to EM section plane. If diameter of such axon is smaller than twice the thickness of the EM section, configurations become possible where there is no overlap between axon’s interiors in two consecutive images (Figure 8A). Rigorously speaking such configurations are indistinguishable in ssTEM from two axons such that one terminates and the other one begins (Figure 8B). For this reason we refer to such configurations as ssTEM ambiguities. Specifically, we call configuration of a neuronal process traversing EM section as ambiguous if there is no cytoplasmic overlap between its profiles in that section and one of the adjacent sections. An example of such configuration for a thin axon in actual ssTEM data is given in Figure 7C.

Figure 8.

A) Illustration of thin axon in configuration where there is no overlap between its interiors in two subsequent images. B) Rigorously speaking such configuration is indistinguishable in ssTEM images from one axon terminating and another starting at the same location.

Figure 7.

A–D) Example of thin oblique axon, indicated with “*” (red contours), which is difficult to trace from local image cues only; width of this axon is about 80nm. The order of images is left to right, and up to down. Despite its small size, this axon can be readily traced by human using context and heading cues. All images are from ssTEM stack P21AA; traces are from complete reconstruction of this stack performed by the author with described approach. E) 3D reconstruction of shown axon, arrow shows direction normal to EM section plane.

We further explain this notion in Figure 9. In this figure three such thin axons (“c”, “d” and “e”) are illustrated traversing two adjacent EM sections (“a”) at different angles. Axons whose heading is close to vertical (“c”) can be traced easily as their profiles in subsequent sections will clearly overlap. However, axons having large angles to section’s normal (“d”) will appear in configurations where there is no overlap between their interiors in subsequent images, such as in Figure 8A, and will cause ssTEM ambiguity. Furthermore, axons running nearly parallel to section’s plane will appear entirely enclosed within two adjacent sections (“e”), and the image of such axons in each section will consist entirely from the indistinct gray wall formed by grazed membrane (i.e. contain no cytoplasmic pixels at all). I.e. such axons even may be impossible to identify in the images. (Note that this only will happen with axons that are thinner than twice EM section thickness. Thicker axons will span at least three consecutive images and will be clearly seen in one of the middle images, where they can be traced.)

Figure 9.

3D illustration of ssTEM ambiguities. Two adjacent EM sections (“a”) are shown along with the cone of heading directions such that thin axons can be traced through (“b”) and the region of heading directions such that thin axons create ambiguous configurations (“f”). “c” illustrates an axon that can be traced, “d” illustrates an axon that is ambiguous, as in Figure 8A, and “e” illustrates thin axon lying nearly parallel to EM section’s plane and, thus, enclosed entirely within two consecutive EM sections.

In each case of such oblique neuronal process, ssTEM images will equally support two opposite hypothesis, i.e. Figure 8A and Figure 8B, unless some larger scale information such as prior and subsequent heading or packing of surrounding profiles is taken into account. Thus, we suggest here that these ssTEM ambiguities will present fundamental difficulty for any segmentation algorithm not utilizing such larger scale cues, i.e. any such algorithm may not be able to achieve error rates substantially lower than the frequency of occurrence of such ambiguities.

It is possible to estimate theoretically this frequency as a function of EM section thickness by using some simple geometric argument and the distribution of sizes of neuronal processes in neuropil that may be measured from our ssTEM reconstructions. Consider a thin axonal process running at certain angle to EM section plane (such as axon “d” in Figure 9). There will be no overlap between process’s interiors in subsequent images if such process runs at an oblique angle to section’s normal larger than certain threshold ϕ= sin⁻¹(d / 2h) (region “f” in Figure 9).Here d is the process diameter and h is the EM section thickness. The value of the threshold angle may be straightforwardly calculated from Figure 8A and the condition that the interiors of neuronal process in subsequent images should not overlap. (Note that for d>2h this formula breaks down since d/2h becomes larger than 1. This means that neuronal processes thicker than 2h can always be traced, as pointed out above.) Then, assuming that directions of neuronal processes as they traverse EM section are isotropic, the fraction of processes in one EM section in ambiguous configurations is described via the solid angle enclosed in the region “f” in Figure 9,

$$f_{ambiguous}(d)=\frac{1}{4\pi }{\displaystyle \underset{\varphi }{\overset{\pi -\varphi }{\int }}2\pi \text{sin}(\theta )d\theta =\text{cos}\left({\text{sin}}^{-1}(d/2h)\right).}$$ (6)

E.g., for h=50nm and d=75nm this fraction of neuronal processes in ambiguous configurations is about 70%.

The average fraction of neuronal processes in one EM section that will lead to ambiguity then may be estimated as

$$f_1(h)=\langle f_{ambiguous}(d)\rangle ={\displaystyle \underset{0}{\overset{2h}{\int }}\rho (d)f_{ambiguous}(d)\mathrm{d}d,}$$ (7)

where we take the average of the fraction of ambiguous profiles for given neuronal process size d over the distribution of the sizes of neuronal processes ρ(d). Such distribution ρ(d) for axons, dendrites and spine necks may be inferred from one of our ssTEM reconstructions (Figure 10A, measured from the reconstruction performed by the author of 150 µm³ ssTEM stack P21AA from CA1 S. radiatum region in hippocampus (Fiala, 2003), containing about 250 axons, 25 dendrites, and the total of 2mm of axonal wire). Then, using Eq.(6), Eq.(7) and Figure 10A, we calculate the fraction f₁ as function of the section thickness h (Figure 10B). We find that on average 5–15% of all axonal processes and 30–35% of all spine necks should appear in ambiguous configurations in 40–50nm EM sections, while with 20–30nm sections these fractions should be below 1–2% and 4–8%, respectively.

Figure 10.

A) Distribution of width of cross-sections of axons, dendrites and spine necks, measured by building 3D distance transform inward 3D reconstructions of neuronal processes and sampling distribution of its values along the centerline of each neuronal process. Distance transform assigns to each point in the volume the value of its distance to the surface of the nearest neuronal process; the centerline is defined as the set of points where distance transform is local maximum in at least two of three orthogonal directions (analogous to a ridge in 2D). Legend in A is also for B. B) Fraction of neuronal processes’ cross-sections in single image expected to appear in obscure configuration, shown as function of EM section thickness. C) Inverse frequency of expected true ambiguities in axons, shown as function of EM section thickness.

Although described ssTEM ambiguities may be next to impossible to interpret automatically using local cues, they often may be disambiguated by expert if neuron fragments before and after such location are clear and appear to run along the same direction. Nonetheless, we occasionally observe locations which are difficult to disambiguate by human expert even using such advanced cues (e.g. Figure 6). In some cases such configurations may be eventually unraveled, but in some cases we may speak of true ambiguity – configuration of neuropil that may not be reliably interpreted. Although definition of true ambiguity is inherently vague since it involves the condition of disagreement between two experts, we typically observed at least one such location in all our reconstructions with size of 50–100µm³ or above. This implies empirical rate of such true ambiguities of at least 1 per 100µm³.

We model such true ambiguities as instances of two adjacent sub-resolution neuronal processes lying obliquely to EM section plane and simultaneously having nearly the same heading direction. Because two processes in this case may not be reliably traced from information available locally in ssTEM images, and because similarity of their headings precludes their discrimination based on 3D cues, such ambiguities may not be reliably resolved even by experienced operators. In practice, empirically observed frequency of such events would indicate threshold of heading difference at which two sub-resolution axons may no longer be reliably disambiguated of about Φ≈5–10°.

The dependency of the frequency of true ambiguities on EM section thickness may be estimated theoretically given the above model and empirically observed rate of 1 true ambiguity per 100µm³ for h=50nm. Here we will consider this calculation for axons, since errors in traces of axons are the ones potentially capable of inflicting the worst damage to circuit reconstruction. This frequency then depends on the number of axons crossing with any other given axon per unit of its length and the probability that such incident axon will be sub-resolution and will have nearly identical heading. From reconstruction of P21AA neuropil we measure the number of incident axons per reference axon length n_a=4÷5µm⁻¹. The probability that the reference axon and the incident axon will simultaneously appear in ambiguous configurations is f₁(h)²; and the probability that headings of such reference axon and incident axon will differ by less than Ф is described by the solid angle enclosed in a cone around the reference axon’s heading with the opening angle Ф (similar to cone “b” in Figure 9), which may be calculated in the same way as f_ambiguous was calculated above (assuming that incident axons are isotropic),

$$p=\frac{1}{4\pi }2{\displaystyle \underset{0}{\overset{\mathrm{\Phi }}{\int }}2\pi \text{sin}(\theta )d\theta =1-\text{cos}(\mathrm{\Phi }).}$$

Then, approximate frequency of true ambiguities per micron of axonal wire as function of h is

$$f_2(h)=n_a\cdot f_1{(h)}^2\cdot (1-\text{cos}(\mathrm{\Phi }))\propto C\cdot f_1{(h)}^2.$$ (8)

It is more convenient to express f₂ as the typical length over which an axon may be traced without encountering true ambiguity as the function of EM section thickness h (Figure 10C). Then, for ssTEM with 40–50nm sections we find that true ambiguities should be expected every 1–10mm of reconstructed axonal wire length, and for ssTEM with 20–30 nm sections this is one ambiguity per 100–1000mm of axonal wire. (Compare this with 5–10mm of axonal wire present in a typical cube of mammalian neuropil of 1000µm³ and 20–40mm of axonal wire in a typical axonal arbor of single pyramidal neuron (Braitenberg and Schuz, 1998).)

Based on the nature of these ambiguities, we argue here that the frequency of true ambiguities will place fundamental limitations on any segmentation algorithm for ssTEM, either using larger scale 3D cues or not, so that reconstructions of neuropil volumes from ssTEM may not be performed with the error rate significantly lower than the frequency of true ambiguities estimated above. It should be noted, however, that this estimate depends directly on the distribution of sizes and headings of neuronal processes in neural tissue and so may change if tissue with substantially different distributions is been reconstructed.

4. Discussion

In this paper we describe automation for the process of neural tissue reconstruction from series of conventionally stained transmission electron micrographs. Given large anisotropy and physical discontinuity of serial sections in ssTEM, a natural approach for processing such data is by viewing it as a sequence of frames rather than truly 3D dataset. Such view motivates our choice of the architecture for reconstruction system: start with analysis of individual images, followed by analysis of the relations between 2D neuronal cell profiles across adjacent images, followed by profiles grouping into 3D objects, followed by computer guided data validation.

Although a variety of options are available both for image processing and subsequent clustering of neuronal profiles, we observe that simple choices work well in ssTEM. Multi-scale local ridge detection followed by analysis of adjacent profiles overlaps suffices to recover majority of the structures in 3D. Special care should be taken in the choice of operating point to avoid explosion of merge errors, while keeping over-segmentation under control. One of the main limitations of presented approach is abundance of thin obliquely running axons in neuropil that may appear in obscure configurations and cause local tracking algorithms to fail. This difficulty may be overcome in principle by considering neuronal processes in 3D and using a model for their expected behavior to fill information not available from local cues. Such approach is employed routinely by human operators reconstructing neuropil either using our software or fully manually. Unfortunately, formal definition of such model and its evaluation is a very nontrivial task.

With this approach we have already reconstructed over 1000µm³ of neuropil from different neural-tissue samples from rat. We used these reconstructions to evaluate our approach speed and accuracy. When characterized in terms of the data validation time, as that of ultimate interest for applications, our approach allows reconstructions at speed of about 5µm³/man-hour, or 1–2 hours per 1000 traced contours, which is 10–50 fold improvement over manual tracing (although admittedly it is difficult to estimate the time involved in tracing neuronal processes fully manually with estimates ranging from 0.1µm³/man-hour to 1µm³/man-hour). Accuracy of thus obtained final reconstructions is comparable with that obtained with fully manual approach and may be characterized by typical frequency of reconstruction contents errors such as trace of an axon or spine neck been lost, two nearby processes been confused and switched, etc. This frequency observed in our reconstructions was 1 error per 200–400 traced contours, as opposed to 1 error per 500–1000 contours for reconstructions produced fully manually.

We also used our first reconstructions to evaluate abundance and impact of ambiguities in ssTEM data. Whenever two very thin axons come together obliquely with the same heading, a true ambiguity may occur which may not be reliably resolved using ssTEM images. For conventional ssTEM such situations occur with low frequency of 5–10 true ambiguities per 1000µm³ or 10mm of axonal wire, which however is significant for the purpose of complete reconstructions of wiring diagram.

While some limitations of computerized reconstructions from ssTEM mentioned in this paper can be overcome by more advanced image processing, many of the problems necessitate improving the quality of conventional ssTEM data. In particular, thickness of the serial sections and low specificity of stain with respect to cell boundaries are major handicaps of conventional ssTEM. Thinner sections, such as 20–30nm, should be helpful in bringing the frequency of true ambiguities to one per 100–1000mm of axonal wire and reducing the frequency of obscure oblique axonal profiles. Use of stain specific to extra-cellular space or external cell membranes should reduce edge clutter caused by intracellular structures. Given that the degree of improvement along each of these directions depends on the improvement along the others, it is likely that the combination of all of these will be necessary to warrant reliable reconstructions of large volumes of neuropil.

Supplementary Material

01

Supplemental Figure 1: Flowchart of the overall 3D segmentation pipeline.

Supplemental Figure 2: Flowchart of the cell boundaries detection process, as described in Appendix A.

Supplemental Figure 3: A) An illustration of the shadow cast in electron beam by cell membrane obliquely traversing serial section. B) An illustration of the shadow intensity corresponding to the fragments of membrane in A: image intensities in two sections anti-correlate in the region enclosed by centerlines of respective membrane fragments (black arrow).