Serial section registration of axonal confocal microscopy datasets for long-range neural circuit reconstruction

Abstract

In the context of long-range digital neural circuit reconstruction, this paper investigates an approach for registering axons across histological serial sections. Tracing distinctly labeled axons over large distances allows neuroscientists to study very explicit relationships between the brain’s complex interconnects and, for example, diseases or aberrant development. Large scale histological analysis requires, however, that the tissue be cut into sections. In immunohistochemical studies thin sections are easily distorted due to the cutting, preparation, and slide mounting processes. In this work we target the registration of thin serial sections containing axons. Sections are first traced to extract axon centerlines, and these traces are used to define registration landmarks where they intersect section boundaries. The trace data also provides distinguishing information regarding an axon’s size and orientation within a section. We propose the use of these features when pairing axons across sections in addition to utilizing the spatial relationships among the landmarks. The global rotation and translation of an unregistered section are accounted for using a random sample consensus (RANSAC) based technique. An iterative nonrigid refinement process using B-spline warping is then used to reconnect axons and produce the sought after connectivity information.

1. Introduction

1.1. Motivation

Many neuroscientists are interested in microscopic-level brain connectivity and how variations in pathways that bridge functional networks influence mental capacity and behavior. Abnormalities in long-range pathways are thought to be directly linked to disorders such as autism (Belmonte et al., 2004) and schizophrenia (Lynall et al., 2010). Our long term goal is the examination of Williams syndrome, a genetic disorder characterized by impaired cognition and overly extroverted social tendencies (Dai et al., 2009). Specifically, we would like to map the limbic system pathways exhibiting the genetic defects to explicitly identify the affected brain systems. Our current preliminary work, however, focuses on reconstructing long-range neural circuits in macaque monkeys by means of fluorescence confocal microscopy. We are targeting neurons in a 12 mm-deep region of interest and work with approximately 30 μm-thick slices of tissue. The size of the complete dataset is an expected substantially large 400 sections. Reliably aligning structures across many microscope slides in the digital representation is one of the many challenges long-range connectivity studies must address. The objective of this paper is to register several moderately deformed sections consisting of predominantly axons in an automated fashion.

1.2. Serial Section Registration Overview

The building block of our datasets is a section, as depicted in Fig. 1. We refer to a section as a thin slab of tissue mounted on a microscope slide. In our case each section is ~30 μm thick. Sections are imaged using a confocal microscope, and we refer to an image from a given focal plane as an optical slice. A tile (or image stack) is the set of optical slices within one field of view of the microscope. Tiles are mosaicked together to recreate a section. For this work all of the tiles are mosaicked for each section prior to registration using the framework presented in Tasdizen et al. (2010).

Fig. 1.

Illustration of a section, tile, and optical slice. Each section is cut approximately 30 μm thick, immunohistochemically stained, and placed on its own microscope slide for fluorescence confocal imaging.

The histological techniques used to image neurons of interest in fluorescence confocal microscopy impose digital reconstruction challenges. Tracing centerlines accurately is non-trivial for structures suffering from low signal-to-noise ratio (SNR) and patchy staining. In applications where the image acquisition requirements are on the order of several cubic millimeters, such as our connectivity study, low magnification may be necessary to capture the area of interest in a reasonable time frame at the cost of making individual neuron differentiation even more difficult. Other data characteristics make section registration particularly challenging. Fig. 2 shows a single channel of an optical slice from the bottom of one section and the top optical slice from the following section. The first obvious feature of the images is that, generally speaking, only neurons are stained, so the images are primarily low intensity background (auto-fluorescence). A second noteworthy feature is that the overlap of an axon pair occurs at the point where both axons exit their respective section (ignoring any tissue losses). As a result, registration methods based on maximizing intensity correlation on section boundary images are unsuitable, unlike an application such as transmission electron microscopy (TEM) where there is abundant overlap of intensity data (Tasdizen et al., 2010). A third significant property is that because of tissue deformation in thin sections, a particular region of axons may align well under a rigid transformation but the neighboring areas may not. These arbitrary deformations introduced when cutting, staining, and mounting the tissue must be accounted for during the restoration of neuron continuity across sections.

Fig. 2.

Example 20× axonal confocal microscopy images from the bottom of one section (left) and the top of the adjacent section (center). The far right image shows a colored composite of the regions marked by the boxes. The x − y pixel spacings are 0.63 μm. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of the article.)

1.3. Related Work

Pursuits in digital neural circuit reconstruction have primarily focused on examining single sections of tissue (Lu, 2011). Analysis of fiber projections on the scale ultimately envisioned by neuroscientists entails reassembly of hundreds if not thousands of serial sections, making the development of an automated registration process necessary. A conceptually relevant work by Oberlaender et al. (2007) proposed a framework for reconstructing neural circuits across serial sections in brightfield microscopy. However, following coarse alignment using blood vessels, neurons in adjacent sections were relinked manually. Our work aims to augment a degree of automation to axonal section registration.

The characteristics of our datasets outlined in the previous section have led us to approach section alignment using landmarks as opposed to an intensity-driven method. For two-view microscopy registration, Al-Kofahi et al. (2003) used dendritic branch points from centerline traces as landmarks, and an extension of the widely used ImageJ tool called Fiji (Fiji Is Just ImageJ) has a plugin for registering multi-view microscopy datasets containing fluorescent bead landmarks (Preibisch et al., 2009). However, the end applications do not target serial section registration, which must account for arbitrary tissue deformations. As previously mentioned, Oberlaender et al. (2007) used blood vessels for registration. We do not use blood vessels to aid in coarse alignment because there is no explicit blood vessel channel in our datasets. We instead derive landmarks from traced axons due to the substantial research already invested in 3-D neurite tracing (DIADEM, in press; Donohue and Ascoli, 2011; Al-Kofahi et al., 2003, 2002; Meijering et al., 2004; Rodriguez et al., 2009; Wang et al., 2011; Chothani et al., 2011; Zhao et al., 2011; Turetken et al., 2011; Bas and Erdogmus, 2011), so our landmark acquisition approach is similar to Al-Kofahi et al. (2003). However, as opposed to using trace branch points as landmarks, we propose to use the location a trace intersects a section boundary for serial section registration.

Alternative methods attempt to simplify the registration problem using block-face acquisition procedures. Gerneke et al. (2007) embedded tissue blocks in wax or resin and stained the exposed top of the sample as it was repeatedly imaged and sliced. Roy et al. (2009) employed a cryo-imaging technique that also combined imaging with sectioning. These sectioning methods attempt to minimize distortion of sections by eliminating any handling that unnecessarily contributes to deformations. We do not utilize a block-face acquisition approach since it would be unsuitable for our immunohistochemical staining requirements.

Methods for serial thin section registration require correction for both a global rotation and translation of the sample across microscope slides as well as local distortions attributed to tissue deformation and loss. Correspondences between the axons of adjacent sections are initially unknown. For coarse (rigid) alignment of TEM data, Saalfeld et al. (2010) assigned correspondences to feature points across images and then used random sample consensus (RANSAC) (Fischler and Bolles, 1981) when finding appropriate global transformation parameters. This general methodology for identifying corresponding landmarks has also been used in stereo vision applications (Zhang and Negahdaripour, 2004). Another approach commonly used for rigid point registration is the iterative closest point (ICP) algorithm (Besl and McKay., 1992; Rusinkiewicz and Levoy, 2001). During each iteration point correspondences are calculated based on smallest Euclidean distances to points in the opposite set. These new correspondences dictate a transform update. In a pipeline for finding rigid transformation parameters for point sets both RANSAC and ICP are commonly applied. We opt to use a RANSAC-based technique since our correspondence assignments are expected to contain some errors (outliers), and we have no guarantee that the sections are free of being grossly misaligned at capture time.

A nonrigid registration framework is needed to correct for local deformations. This is not only necessary for visualization purposes but more importantly for updating axon connectivity assignments across sections. Al-Kofahi et al. (2003) present a variant of the ICP algorithm for their two-view microscopy application that incorporates the ability to fit an expanded set of models such as parabolic transformations. Nonrigid point registration methods such as thin-plate spline-robust point matching (TPS-RPM) (Chui and Rangarajan, 2003) and coherent point drift (CPD) (Myroneko et al., 2007) define energy functions that enforce smoothness in the transformations. We utilize the concept of motion coherence to register our datasets, meaning tissue should be warped in a similar manner locally. Our refinement process is most similar to the method outlined in Xie and Farin (2004), which combines coarse-to-fine basis spline (B-spline) warping with the ICP algorithm.

2. Methods

2.1. Methods Overview

The goal of our method is to align axon endpoints at section boundaries. Using an established neurite tracing algorithm to identify where axons exit a section, we are able to define point landmarks at the section boundaries. After calculating landmark similarities based on local landmark spatial configurations, we assign temporary correspondences between two sections using minimum weighted bipartite matching. In conjunction with RANSAC, the correspondences allow a suitable least-squares solution for the global rotation and translation parameters to be recovered. For refining the registration, i.e. aligning axon endpoints, we iteratively update the correspondences and apply nonrigid transformations with the intent of first correcting for large landmark displacements and shifting towards making more local corrections. When registering two sections we use the common nomenclature of reference and moving entities. Specifically, a moving section is transformed to align with a reference, or fixed, section.

2.2. Landmark Extraction

To extract axonal profiles, we traced our datasets using the freely available software package NeuronStudio (Rodriguez et al., 2006, 2008, 2009). Tracing with NeuronStudio is semi-automatic in that a user must manually place seed points, but the tracing itself does not require intervention. As discussed in Rodriguez et al. (2009), neurites are thresholded locally and centerline nodes are placed during a process called voxel scooping. The output is a 3-D trace file (swc) commonly used by neurocomputing applications, many of which are presented in Meijering (2010).

Tracing serves as our landmark detection step. The centerline endpoints near section boundaries indicate where axons exit a section, so point landmarks are set at these locations. Landmark features in addition to position are also available, since the axons in our datasets generally transition smoothly with respect to orientation across section boundaries. Fig. 3 depicts the features associated with a point landmark located at some x − y coordinate. These features include the angle at which each axon approaches a section boundary, ϕ, the traversal angle of each axon in the plane of the sample, θ, and the radius of each axon, r.

Fig. 3.

Landmark information for the same traced axon across a section boundary. The cyan line comes from the trace data, and the red line represents the vector used to calculate the angle information. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of the article.)

2.3. Coarse Alignment

Global rotation and translation between two sections are computed before connecting individual axons back together. This is accomplished by first pairing landmarks according to a simple criterion that considers the spatial relationships among the axons. The equation for the correspondence measure is,

$$C_{m,n}=\frac{1}{2}\sum _{i=1}^N\frac{{(h_m[i]-h_n[i])}^2}{h_m[i]+h_n[i]},$$ (1)

where m and n are point indices, h_m and h_n are histograms of distances to neighboring landmarks for points m and n, respectively, and N is the number of bins in h_m and h_n. If both h_m[i] and h_n[i] are zero, the summation term for the current i is set to zero. This equation may be recognized as the Chi-square measure for comparing two histograms (Belongie et al., 2001; Bradski and Kaehler, 2008). For each point a local histogram is constructed representing the distances to all neighboring axons from the current axon of interest within a specified distance threshold. The number of histogram bins must also be specified. The term serves to provide local spatial information regarding an axon’s neighborhood and helps determine, for example, whether or not the axon is in a sparse region. The angular distributions of neighbors are not yet included since the global rotation of the moving section is unknown.

Next we assign temporary correspondences to landmarks such that the lowest total sum of correspondence measures is produced, with the expectation that truly corresponding landmarks have very similar neighborhood configurations and hence the smallest c values. Assigning correspondences between the two datasets can be viewed as finding matches for the nodes of a bipartite graph. A bipartite graph, G = {U, V, E}, consists of two disjoint sets, U and V, whose edges, E, only bridge points belonging to different sets. Our two sets consist of the landmarks of the sections to be registered. A weighted graph is constructed by assigning the c values to the edges. Because the correspondence metric indicates the best match for a point m to another point n when it is smallest, the problem is that of determining the minimum weighted sum of edge links. This is a standard minimum weighted bipartite matching task, and we currently solve the matching problem using the Hungarian algorithm (Jungnickel, 2005).

Having correspondences across two point sets allows parameters of a rotation/translation model to be calculated. As shown in (2),

$${\mathbf{p}}_{\mathit{reference}}^{(i)}=\mathbf{R}{\mathbf{p}}_{\mathit{moving}}^{(i)}+\mathbf{T}+{\mathit{\eta }}^{(i)},\mathit{i}\in 1,\dots ,\mathit{M},$$ (2)

the ith point in a moving dataset, ${\mathbf{p}}_{\mathit{moving}}^{(i)}$, can be rotated by the rotation matrix, R, and translated by the vector, T, to align with its corresponding point in the reference dataset, ${\mathbf{p}}_{\mathit{reference}}^{(i)}$, with some mismatch dependent on the noise, η⁽ⁱ⁾.

A 3-D formulation of this problem is presented in Arun et al. (1987), although the concepts are directly applicable to our 2-D case. The solution for the rotation and translation of two point sets with known correspondences is determined in a least-squares sense. That is to say, we find the 2 × 2 rotation matrix, R̂ and the 2 × 1 translation vector, T̂ The rotation can be determined after evaluating (3),

$$\sum _{i=1}^M{\mathbf{q}}_{\mathit{moving}}^{(i)}{\mathbf{q}}_{\mathit{reference}}^{(i)}=\mathbf{H}=\mathrm{U\Lambda }{\mathrm{V}}^{\mathrm{T}},$$ (3)

where the q vectors for each of the datasets are simply the two point sets with the centroids, c_reference and c_moving, subtracted out. The q vectors are ordered so that corresponding points have the same indices. The UΛV^T term represents the singular value decomposition of H. Finally, a matrix X = VU^T is calculated. If the determinant of X is 1, then R̂ = X. If the points used to calculate the rotation are collinear or have large amounts of noise (e.g. do not really correspond), then det(X) = − 1 and may represent a reflection or is meaningless. For simplicity, both of these cases can be categorized as model fitting failures and, if possible, a different set of corresponding points can be tested. The translation,

$$\widehat{\mathbf{T}}={\mathbf{c}}_{\mathit{reference}}-\widehat{\mathbf{R}}{\mathbf{c}}_{\mathit{moving}},$$ (4)

is obtained by finding the difference between the centroid of the reference point set and the centroid of the moving dataset after rotation. The conceptual understanding is that the aligned point sets should have the same centroid. The reader is referred to Arun et al. (1987) for mathematical justifications for these solutions.

The previous discussion briefly reiterates a way to determine rotation and translation parameters from two sets of points with specified correspondences. We still face the problem, however, that although correct correspondences are identified after the minimum weighted bipartite matching procedure, they exist amidst numerous incorrect assignments. Ideally, global rotation and translation parameters should be calculated based on landmark correspondences that have been correctly assigned. Because perfect assignments cannot be guaranteed, however, we make use of RANSAC, a robust model fitting paradigm for data containing outliers (Fischler and Bolles, 1981). Outliers in this context refer to landmarks that have been incorrectly associated. Likewise, inliers refer to landmarks that have been correctly matched within a tolerance level.

In our application RANSAC first randomly selects a subset of points and attempts to fit a rotation/translation model based on their correspondences. A potential rotation and translation are calculated using the randomly sampled points and the least-squares solution previously described. The remaining points from the moving dataset are then transformed using these model parameters. The points that transform within a specified distance threshold, ε, with respect to their corresponding point are added to a consensus set. If the total number of inliers increases for a particular set of transformation parameters, the parameters are saved as the current best set. The process is repeated for a specified number of iterations or until the internal iteration limit, k, given in (5) is reached,

$$k=\frac{\mathit{log}(1-p)}{\mathit{log}(1-{\omega }^n)},$$ (5)

where p is the desired probability that at least one inlier is randomly selected for model fitting, ω is the actual probability of randomly selecting an inlier, and n is the number of points randomly selected for the model fitting (Fischler and Bolles, 1981). The ω term is updated each time a new best model has been found and is assigned based on the number of current inliers divided by the total number of points. Once the algorithm has terminated all of the points marked as inliers are used to form a final least-squares rotation and translation estimation. Fig. 4 shows an example of two coarsely aligned landmark sets taken from larger sections.

Fig. 4.

Example rough alignment of landmarks using the process described in Section 2.3. The os represent axon intersections at the bottom boundary of a section while the xs represent the axon intersections at the top boundary of the adjacent section. The size of the region is ~1290 μm × 1290 μm with isotropic x − y pixel spacings of 0.63 μm.

2.4. Refined Alignment

In a localized region, such as that illustrated in Fig. 4, we assume the inexact alignment of landmarks following coarse registration are attributable to a combination of factors such as small amounts of tissue loss, local shearing from the microtome, and section shrinkage/expansion. When examining the entire section at once, larger stretching distortions introduced during the sectioning and mounting are also apparent. In the interest of registering for connectivity analysis we currently disregard any minor tissue losses and just warp a moving section’s landmarks to coincide with the preceding reference section. Taking into consideration that the physical sections are very thin, we propagate corrections for the moving section throughout the entire section. In other words, each mosaicked optical slice is warped the same as the top mosaicked optical slice. The succeeding section is then registered to this transformed section.

The refinement process begins by recalculating landmark feature similarities using an updated metric. We now incorporate angular information with respect to the x − y plane since the approximate rotation of the moving section, R̂, is known from the coarse alignment. The updated correspondence criterion between points m and n is,

$$C_{m,n}=w\frac{P_{m,n}}{P_{\mathit{thresh}}}+(1-w)\frac{D_{m,n}}{D_{\mathit{thresh}}},$$ (6)

where w is a weight, P_m,n is a Chi-square histogram measure comparing the positional distributions of nearby points, D_m,n is the Euclidean distance between points m and n, P_thresh is a threshold for P_m,n, and D_thresh is a threshold placed on D_m,n. The P_m,n term is formed by dividing the region surrounding a point into bins in polar coordinates up to a specified distance. Within this region we use bins that are evenly spaced both radially and rotationally, such as is depicted in Fig. 5. The feature is a 2-D histogram, and comparison of two of these features is performed in the same manner as in (1). Similar to Belongie et al. (2001) interim correspondences are again assigned using minimum weighted bipartite matching, with additional considerations made to reduce erroneous landmark matchings discussed next.

Fig. 5.

Example bin layout in polar coordinates.

When assigning correspondence values for a given landmark, restrictions are used to limit the permissible point correspondences. An upper threshold, P_thresh, is placed on P_m,n to prohibit axons with largely different neighborhoods from being matched. To enforce local matching, a distance threshold, D_thresh, prevents a point from being paired with one outside of the specified radius. In other words, if either of the terms surpass their respective threshold, landmarks m and n are forced to be non-corresponding. Otherwise, the thresholds are used as normalization factors. The orientation of axons in the section and their size are also used to prevent insensible matchings. For example, if the difference in orientation in the x − y plane is larger than a threshold, θ_thresh, a potential match is disregarded. Analogously, maximum allowable differences are also placed on radii and boundary angles, r_thresh and ϕ_thresh, respectively.

Once correspondences have been reassigned, the arbitrary deformations are addressed using nonrigid transformations. We accomplish nonrigid warping using third order B-splines defined on a 2-D lattice of knot points (Unser, 1999). Control points are used to modify the shape of the underlying function, which in our case represents the deformation. The number of control points available is coupled with the number of B-spline knots. We iteratively reposition control points to pull regions of distorted tissue into alignment using the current correspondence assignments. Progressively increasing the density of the control point grid permits finer local refinement.

We adjust control points in a straightforward manner by taking advantage of the fact that, in general, our misaligned landmarks require approximately the same amount of correction in a given region as depicted in Fig. 6. Therefore, we determine the general direction of distortion for an area of the moving section and shift the regional control points to compensate. For a particular control point with indices p and q, we calculate its displacement vector, V_p,q, based on how landmarks are misaligned in the four surrounding grid cells. For each pair of corresponding landmarks in these grid cells we also define a displacement vector, v_m,n, originating at the moving points. From the v_m,ns we select the median x and y components as the motion for the control point. The median displacement vector components are chosen since it is possible for improper correspondences to still exist, and as long as half of the correspondences in the grid cells surrounding a control point are correct the region shifts in the general needed direction. If only one landmark exists within the grid cells around a control point, it will be drawn towards its corresponding point directly.

Fig. 6.

Illustration depicting approximate motion coherence for a local set of coarsely aligned landmarks. The os are the reference landmarks and the xs are the moving landmarks. The connecting lines indicate the current correspondences, which are used to determine the displacement vector, V_p,q, for the control point (black dot).

In a similar sense as the ICP algorithm and the method presented in Xie and Farin (2004), we iteratively update the correspondence assignments and transformations to progress the registration. The correspondences are assigned according to (6) and a stage of bipartite matching. The control point density increases at each iteration for improved local control. A specified control point spacing, S_D, determines when the correspondence assignments should completely favor D_m,n and effectively become nearest neighbor assignments. The number of iterations beyond the first required to reach this spacing is calculated as in (7) using the initial control point spacing, S_I, and its defined rate of change, S_RC.

$$I_D=\lceil {\log }_{S_{\mathit{RC}}}\left(\frac{S_D}{S_I}\right)\rceil$$ (7)

I_D is then used to perform a simple adjustment to the weight in (6). For the first iteration w is set to 1.0, and subsequent changes shift emphasis from matching similar clusters to nearest neighbors. After each transformation w is updated as,

$$w^{(i+1)}=w^{(i)}-\frac{1}{I_D},$$ (8)

where i is the current iteration number. Upon reaching the (I_D + 1)th iteration the restrictions imposed by θ_thresh, r_thresh, and ϕ_thresh are relieved to ensure that any trace inconsistencies do not prohibit true axon endpoints from being matched. Further iterations can be run for additional tweaking until a termination spacing, S_T, is reached. Lastly, D_thresh is automatically adjusted throughout the iterations so that it is never larger than the current control point grid spacing. An example warping progression is shown in Fig. 7.

Fig. 7.

Example landmark adjustment for several refinement iterations. The os represent axon intersections at the bottom boundary of a section while the xs represent the axon intersections at the top boundary of the adjacent section. The cyan lines indicate correspondences for the current iteration, which is labeled in each frame. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of the article.)

3. Results

3.1. Datasets

Confocal datasets used for this study are a subset of those acquired as part of a larger neuroanatomical investigation of the macaque brain. Imagery included here represents Neuropeptide antibody # AB1565 (Chemicon) expressing neuronal fiber projections immunohistochemically labeled with Alexa 568 (Invitrogen) secondary antibody. Fluorescence microscopy was performed on five consecutive tissue sections of the right hemisphere in the region of the basal forebrain and hypothalamus. A Nikon A1R confocal microscope equipped for resonant mode acquisition was used for 20× imaging yielding a x − y resolution of 0.63 μm/pixel. The Alexa 568 probe was excited using a 561 nm wavelength laser with 512 × 512 pixel emission image fields captured at ~3.7 frames/s. For each section a region was acquired comprising 13 × 17 fields (~4200 μm × 5500 μm) over 25 optical slices at 0.60 μm intervals.

3.2. Experiments

The registration is quantitatively assessed using landmarks derived from two sets of traces. The first collection of traces is obtained using NeuronStudio. Although the traces are attained semi-automatically, one drawback is that our dataset resolution limits NeuronStudio’s ability to differentiate intertwined or crossing axons that touch. As a result many axon traces become merged. For purposes of acquiring the locations where axons exit a section, however, these endpoints generally remain intact (with exceptions such as where an entwined axon pair exits a section). The second set of experimental traces is obtained by manually correcting problem areas in the NeuronStudio traces, such as false branch points and incorrectly merged axons. Manual editing is accomplished using Neuromantic (Myatt, in press). The purpose of using the two trace sources is to show that with respect to aligning axon endpoints at section boundaries, the result obtained using readily placed NeuronStudio traces is comparable to that based on traces which have undergone time-consuming manual editing.

The dataset used for quantitative evaluation is a ~1290 μm × 1290 μm subset of the five section series described in the previous section. The subset contains both a very sparse area as well as a region representative of the general axon density of the complete dataset. The average number of point landmarks at the section boundaries acquired from the five NeuronStudio-traced subsections is 286. For the manually adjusted traces the average is 229 landmarks. The difference is indicative of the elimination of spurious landmarks during manual editing.

Coarse registration results are presented in Table 1. The total possible correspondences are the number of correspondences assigned between sections following the initial bipartite matching. For computational savings a subset of the landmarks is used in the matching, accounting for the low number of possible correspondences compared to the average number of landmarks per section. The subset is chosen by limiting landmarks used in the matching to those whose associated axon approaches the section boundary greater than a given steepness. For the datasets examined in this work a threshold of approximately fifteen degrees reduces the possible matches to the values listed in the table. The inlier correspondence numbers in Table 1 indicate how many of the initial matches RANSAC uses to determine the final rotation and translation; i.e. these are the corresponding landmarks that fall within a given distance of each other after the moving section is transformed. These numbers are relatively low due to the limited nature of the correspondence criterion from which they are based, and an extended discussion is covered in Section 4.

Table 1.

Coarse alignment correspondences for the subsets.

Trace source	Sections	# total possible correspondences	# inlier correspondences
NS	1, 2	92	31
NS	2, 3	94	32
NS	3, 4	118	44
NS	4, 5	112	42
M	1, 2	88	51
M	2, 3	94	57
M	3, 4	94	58
M	4, 5	80	51

NS: NeuronStudio, M: manually edited.

The final correspondences assigned by the registration refinement stage are shown in Table 2, which translate into axon connectivity across sections. The table shows the comparison of the number of correspondences assigned manually and by the algorithm, as well as the breakdown of correspondences assigned by the algorithm. The algorithm correspondence breakdown contains subheadings that list landmark matchings as valid-to-valid, incorrect valid-to-valid, valid-to-spurious, and spurious-to-spurious. Valid and spurious landmarks are manually determined, with a valid landmark defined as having a corresponding landmark in the adjacent section. For example, the valid-to-spurious column represents the number of valid landmarks from both sections that have been incorrectly matched to a landmark without a corresponding point. The last column shows the number of valid landmarks incorrectly left unmatched.

Table 2.

Refined alignment correspondences for the subsets.

Trace source	Sections	# correspondences assigned		Breakdown of correspondences assigned by algorithm
		Manually	Algorithm	V↔V	Incorrect V↔V	V↔S	S↔S	V unmatched
NS	1, 2	141	158	126	1	9	22	19
NS	2, 3	145	167	135	1	5	26	13
NS	3, 4	169	179	151	1	13	14	21
NS	4, 5	158	142	142	2	9	19	19
M	1, 2	194	197	182	1	11	3	11
M	2, 3	199	202	184	2	13	3	13
M	3, 4	185	192	177	0	8	7	8
M	4, 5	182	182	177	0	4	1	5

NS: NeuronStudio, M: manually edited, V: valid landmark, S: spurious landmark.

A summary of Table 2 can be made in terms of precision and recall as,

$$\begin{array}{l}\mathit{precision}=\frac{\#\mathit{true\; positives}}{\#\mathit{true\; positives}+\#\mathit{false\; positives}}\\ \mathit{recall}=\frac{\#\mathit{true\; positives}}{\#\mathit{true\; positives}+\#\mathit{false\; negatives}}.\end{array}$$ (9)

The number of true positives refers to the number of correspondences the algorithm selects correctly with respect to the manual assignments. The false positives are the incorrect correspondences, and the false negatives are the unmatched valid landmarks. The perfect set of results would yield precision and recall values of 1.0. The values derived from our results are shown in Table 3. Precision indicates how many points are paired correctly with a penalty incurred for incorrect correspondence assignments. Spurious point matchings are the predominant cause for drawing the precision away from the maximum of one. Recall indicates how many true correspondences are missed.

Table 3.

Precision and recall measures.

Trace source	Sections	Precision	Recall
NS	1, 2	0.798	0.869
NS	2, 3	0.808	0.912
NS	3, 4	0.844	0.878
NS	4, 5	0.826	0.882
M	1, 2	0.924	0.943
M	2, 3	0.911	0.934
M	3, 4	0.922	0.957
M	4, 5	0.973	0.973

NS: NeuronStudio, M: manually edited.

In combination the precision and recall values suggest we match truly corresponding points reasonably well, but also include a non-negligible number of spurious landmarks. This applies more so to the case of the experiments using landmarks derived from unedited traces since there are more false landmarks available. The reason some false landmarks get paired is that in the final stages of the registration nearest neighbor assignments take hold under the assumption that region stretching has been corrected and endpoints should be reconnected.

The registrations have been carried out on a standard Linux desktop (Intel i7 3.2 GHz CPU with 8 GB RAM). Implementation is currently based in MATLAB (2011), with the nonrigid transformations performed using the Insight Toolkit (ITK, in press). The average time for calculating the initial feature correspondences was 1.36 s across all of the sections. RANSAC took an average of 0.044 s to find a suitable rotation and translation. There were a total of seven iterations of recalculating correspondences and warping landmarks. The average times for these two steps at each iteration were as follows: (1) 1.69, 0.79 s; (2) 1.69, 0.92 s; (3) 1.66, 1.36 s; (4) 1.58, 2.52 s; (5) 2.09, 6.44 s; (6) 1.64, 15.99 s; (7) 1.51, 46.81 s. The landmark warping times rise at each iteration due to increasing control point densities. The matching time for iteration 5 is slightly higher since it is the first iteration distances are the only contributing factors to correspondence assignments. Therefore, additional points are included in the matching. Generation of the transformed images was performed off-line since the process was much slower (on the order of minutes) than just manipulating landmarks, depending on the amount of parallelization used and the final control point density.

A visualization of what the registration aims to accomplish, i.e. align axon endpoints at the section boundaries, is shown in Fig. 8 with the aligned endpoints marked by circles. From this visual evaluation alone the axon exhibits enough continuity to recognize that the pieces comprise the same axon, although the landmark correspondences are what provide the connectivity information as to which segments should be considered the same axon. The stair-step pattern is caused by empty space along section boundaries that are not perfectly flat. We also applied the method to the full dataset containing approximately 900 landmarks per section. While quantitative results are only presented for the subregion, the same types of errors are observable in the full dataset, namely spurious points being matched and some missed correspondences. Isosurface renderings in Fig. 9 show examples of aligned axons, where sections are tagged by different colors to make the axon transitions from one section to the next easily identifiable.

Fig. 8.

Volume rendering of a single axon aligned. The red circles highlight the section boundaries. Breaks in the renderings (stair-step effect) are the result of section boundaries that are not perfectly flat. Two faint aligned axons can also be seen on either side of the marked axon. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of the article.)

Fig. 9.

Example regions of aligned axons. Colors indicate different sections: 1 – Green, 2 – Orange, 3 – Yellow, 4 – Red, 5 – Blue. (a) Isosurfaces of a zoomed-in region of the five section testing subset. (b) Isosurfaces of a zoomed-in region of the five section full dataset. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of the article.)

4. Discussion and Conclusions

Our registration task faces the common challenge of incomplete one-to-one landmark correspondences across datasets. The severity of spurious and missing landmark formulation is dependent on the tracing quality, as centerline endpoints at section boundaries are treated as landmarks. Faintly stained axons traced in one section but not the next, broken traces running along a section boundary, incorrectly merged axons, and false branches near section boundaries all generate erroneous landmarks. These problems are more pronounced in the landmarks extracted from the semi-automatically acquired traces, hence the higher rate of spurious landmark matchings in Table 2. The other problem is the merging of centerlines for touching axons as they weave through a section, since individual axon identities are lost. This issue is highlighted in Fig. 10a, which shows the semi-automatically obtained center-lines registered for the five section subset. The centerline colors represent axons connected across sections using the final correspondence assignments, and the problematic merging is depicted by the large bundle of green traces. In contrast, Fig. 10b presents registered traces that have undergone time-consuming manual editing prior to registration. The abundance of centerline colors in same region visually demonstrates that axon identities have been significantly better preserved.

Fig. 10.

Example registered centerlines for the five section testing subsets. Black lines represent segments with no connectivity to another section, and colors (randomly assigned) mark axons connected across sections. (a) Aligned centerlines obtained semi-automatically. (b) Aligned manually corrected centerlines. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of the article.)

A tradeoff exists between the capacity to follow individual axons over large distances and how readily results can be produced. While the semi-automatic traces are much faster to place than by hand, the resolution of our datasets makes it difficult to differentiate axons in moderate to heavily populated areas. Without correction, the ability to track any given axon is inhibited. However, manual corrections are very time-consuming, especially when considering hundreds of serial sections. Capturing sections at a higher magnification could potentially help in improving trace quality at the expense of substantially increased imaging times and both computational and storage demands in dealing with larger datasets. With the desire to maintain as much automation as possible, another alternative is to register sections with the semi-automatically obtained traces and track bundles of axons versus individual projections. Our registration results indicate this approach is plausible since truly corresponding landmarks are matched well. Because this approach strays from our goal of following individual axons, however, it is also worth investigating whether other automated tracing algorithms are better able to differentiate axons at our current data resolution (Wang et al., 2011; Chothani et al., 2011; Zhao et al., 2011; Turetken et al., 2011; Bas and Erdogmus, 2011).

Several user set parameters are required for the registration, though their selection is rather intuitive. Tables A.1 and A.2 in Appendix A show the parameters used to register all of the sections. The coarse registration requires three significant parameters. One parameter is the distance threshold for d, or the maximum distance for which to consider a landmark’s neighbors when creating the distance histograms. Our moderately sized threshold captures enough global context to identify the approximate region a point is located in the section. The number of bins in d accounts for variability in the locations of corresponding landmarks, since they will not align under a rigid transform alone. The third primary parameter is the distance tolerance, ϵ, for determining a valid correspondence in a given RANSAC iteration. The ϵ threshold is intentionally set to be lenient, implying that not all the inliers in Table 1 represent truly corresponding axon pairs. The reason is because the matching criterion for coarse alignment is solely based on distances to neighbors. Considering the presence of spurious landmarks and section stretching, it is difficult to match corresponding landmarks exactly based exclusively on the distance criterion. Nevertheless, in cases of mismatch the correctly corresponding axon is nearby, and the final least squares solution obtained provides an acceptable coarse alignment. The logic behind the use of the limited correspondence criterion is that under normal section distortions most local axon configurations remain intact, and we only need to match a portion of them correctly to establish a global transformation. The criterion also allows us to assume no a priori knowledge of alignment. The numbers of inlier correspondences in Table 1 associated with the manually corrected traces are higher since the distance-based correspondence assignments improve from the reduced number of spurious landmarks.

The remaining two parameters in Table A.1 are the number of points used to calculate a potential rotation and translation, n, and the desired probability that a pair of corresponding points is randomly selected for model fitting, p. Parameter n is kept small since we expect many of the initial correspondences to be incorrect as previously discussed, and we keep p near 1.0 so the iteration limit calculated in (5) does not cause premature termination.

The parameters for the registration refinement are listed in Table A.2. The logic behind selection of the polar histogram parameters is similar to the prior discussion regarding the distance-only histograms. However, the range threshold is smaller than that used for d since we want to focus on more local differences. The choices of radial and angular bin sizes are again moderately sized to account for variations in the alignment of corresponding neighbors. The P_thresh parameter prevents two points with very different neighborhoods from being matched. In other words, this threshold aims to keep spurious landmarks from being paired with one another. The smaller the threshold, the more similar the polar histograms need to be for two points to be eligible for correspondence. We allow some tolerance by setting P_thresh above the midpoint of its possible range.

The maximum radius a given point can be matched within is D_thresh, though this value becomes bounded by the control point spacing as the iterations progress. If there is too much distortion in a region, i.e. a stretch larger than D_thresh, it is possible for a region to not have any correspondences assigned. For a case like this the intent of starting with a sparse set of control points would be to pull this region of the section into closer alignment as a nearby region is corrected. The other thresholds that aim to restrict spurious matchings based on radii, x − y plane orientations, and boundary angles are r_thresh, θ_thresh, and ϕ_thresh, respectively. Axons are presumed to transition roughly linearly across section boundaries, so the value for θ_thresh assumes a factor of π has been subtracted from the moving section’s θ angles (see Fig. 3). Because of sporadic trace inaccuracies, all three thresholds are allotted some leeway to prevent truly corresponding landmarks from being disassociated.

The last series of parameters for the refined alignment relate to the control point spacing. The initial spacing is a function of a rough estimate of the amount of stretching present in the sections. The testing subregions cover a small area, so their initial spacing is set to only 176.4 μm. In contrast, the full sections require a larger initial spacing due to more prevalent section-wide stretching, so their spacing begins at 1134 μm. For the control point spacing rate of change, we use the heuristic of halving the spacing at each iteration. Too rapid of a decrease can lead to missed correspondences, since D_thresh is bounded by the spacing. The desired spacing at which weight w forces D_m,n to completely dominate the correspondence measure in (6) is 12.6 μm, and the desired final grid spacing is 3.15 μm. One clarification is that our grid spacing halves at each iteration, so the S_D and S_T spacings are not reached exactly. Instead, the first spacing reached that is smaller than the specified value is used. Lastly, the challenges in dealing with large, high density control point grids, such as memory consumption, can in part be addressed by utilizing a full hierarchical B-spline implementation (Forsey and Bartels, 1988; Xie and Farin, 2004). In a scheme like this only regions containing landmarks have increasing control point densities, so empty space is effectively ignored.

There are many opportunities for extensions to this work. For example, there are limitations to which datasets this method is entirely applicable, such as those containing severe deformations (perhaps as the result of a tear during mounting), small regions of overlap, and extremely dense regions of axons. Because we currently utilize spatial relationships among landmarks to aid in identifying matching pairs, substantial landmark detection errors or deformations so severe that large disparities exist between axon geometric configurations and orientations are problematic. For an immensely dense set of axons, we could potentially again lose the ability to identify unique local neighborhood configurations. Our datasets so far exhibit moderate axon densities and distortions. In addition, improvements are needed to reduce the number of spurious landmarks incorrectly included in correspondence assignments. The results based on the manually corrected traces confirm that fewer false landmarks boosts both the precision and recall values, supporting the efficacy of the method. However, the need still exists to reduce erroneous connections. Finally, with the end goal of analyzing long-range neural circuits, it is obvious that many more consecutive sections must be experimented on and statistics for the axon populations must be generated.

A methodology for registering axonal processes across serial sections has been presented in this work. We take advantage of the progress made in automated neurite tracing algorithms to extract centerlines and set point landmarks at section boundaries. The tracings also provide additional information for each landmark in terms of the representative axon’s radius and orientation in the section. We obtain a coarse registration without prior knowledge of how sections are initially misaligned and allow for nonrigid warping in response to moderate arbitrary deformations in the tissue. We have presented connectivity results for a region of axons and include visualizations of the final automated alignment. While extensions to the work presented are required to make it more applicable to broader datasets, we nevertheless show progress in providing neuroscientists with the ability to establish axonal connectivity in an automated fashion.

Appendix A. Parameters

See Tables A.1 and A.2.

Table A.1.

Coarse registration parameters.

Parameter	Value	Value in pixels
Distance threshold for d	409.5 μm	650
Number of radial bins for d	30
Distance error for RANSAC inlier: ε	47.25 μm	75
Desired probability that a RANSAC inlier is selected: p	0.99
Number of points used to determine a model for RANSAC: n	3

x − y pixel spacings are 0.63 μm.

Table A.2.

Refined registration parameters.

Parameter	Value	Value in pixels
Distance threshold for Pm,n	94.5 μm	150
Number of radial bins for Pm,n	10
Number of angular bins for Pm,n	10
Threshold for polar histogram similarities: Pthresh	0.6
Threshold for distance of potential matches: Dthresh	94.5 μm	150
Threshold for radii differences: rthresh	0.4 μm
Threshold for x − y plane orientation differences: θthresh	π/3 rad
Threshold for boundary angle differences: ϕthresh	π/3 rad
Initial control point grid spacing: SI	176.4 μm (1134 μm for full dataset)	280 (1800)
Control point spacing rate of change: SRC	0.5
Desired control point spacing at which Dm,n dominates: SD	12.6 μm	20
Desired termination control point spacing: ST	3.15 μm	5

x − y pixel spacings are 0.63 μm.

Appendix B. Supplementary Data

Supplementary data associated with this article can be found, in the online version, at doi:10.1016/j.jneumeth.2012.03.002.