Improving 3D Edge Detection for Visual Inspection of MRI Coregistration and Alignment

Abstract

Background

Visualizing edges is critical for neuroimaging. For example, edge maps enable quality assurance for the automatic alignment of an image from one modality (or individual) to another.

New Method

We suggest that using the second derivative (difference of Gaussian, or DoG) provides robust edge detection. This method is tuned by size (which is typically known in neuroimaging) rather than intensity (which is relative).

Results

We demonstrate that this method performs well across a broad range of imaging modalities. The edge contours produced consistently form closed surfaces, whereas alternative methods may generate disconnected lines, introducing potential ambiguity in contiguity.

Comparison with existing methods

Current methods for computing edges are based on either the first derivative of the image (FSL), or a variation of the Canny Edge detection method (AFNI). These methods suffer from two primary limitations. First, the crucial tuning parameter for each of these methods relates to the image intensity. Unfortunately, image intensity is relative for most neuroimaging modalities making the performance of these methods unreliable. Second, these existing approaches do not necessarily generate a closed edge/surface, which can reduce the ability to determine the correspondence between a represented edge and another image.

Conclusion

The second derivative is well suited for neuroimaging edge detection. We include this method as part of both the AFNI and FSL software packages, standalone code and online.

Introduction

Neuroimaging relies on the accurate coregistration of different images. Unfortunately, the automated coregistration tools do not always succeed, requiring visual inspection to detect errors and determine the appropriate intervention. A popular method for assessing the quality of spatial co-registration is superimposing the edges of one image onto another (refer to the Figures for illustrative examples). The motivation behind our current work stems from our experiences in training new users of the popular AFNI and FSL pipelines. A common critique raised by trainees pertains to the interpretability of the edge detection maps generated by these tools. Unfortunately, the edge detection algorithms often do not identify meaningful tissue boundaries. Furthermore, the edges often do not form closed contours, making it difficult to determine what anatomy the contour reflects. Our overarching aim is to demonstrate the second derivative (detected using difference of Gaussian, or DoG) provides a robust solution to this problem.

Most neuroimaging studies require the accurate spatial alignment of various images. This alignment serves several purposes, such as synchronizing the same modality over time (e.g., correcting motion in functional imaging time series), aligning images from different modalities (e.g., matching a low-resolution functional image featuring bright cerebrospinal fluid with a high-resolution T1-weighted anatomical scan where white matter is bright), and achieving alignment across different individuals (e.g., aligning high-resolution scans of different individuals into a common space).

Within our field, we heavily rely on automated co-registration algorithms. These algorithms iteratively manipulate the position of a source image to match a target image, aiming to minimize a cost function that estimates the alignment quality. Regrettably, these methods can encounter challenges, such as suboptimal cost functions or initial estimates, or complications arising from factors like lesions (Saad et al. 2009; Jenkinson et al. 2002; Brett et al. 2001). For these reasons, it is imperative to incorporate a quality assurance stage to verify the success of spatial co-registration (Saad et al. 2009). A common solution to this problem is to present the edges of one image as an overlay on the other image, allowing users to determine if anatomical features are aligned between images. By default, the popular SPM toolbox uses a different solution: with the coregistration validation showing each image in a different panel, with crosshairs yoked between images. This approach allows the user to interactively position the crosshair to ensure that anatomical features are aligned. While both approaches have merit, the edge approach is particularly useful for static displays, and it also maximizes image size efficiency by showing all images in a single panel. For completeness, it is worth noting that SPM users can manually display edge lines using Matlab’s parameter free ‘contour’ function.

As we elucidate below, the existing edge overlay images rely on determining a meaningful brightness threshold, which is often challenging, especially across different sequences as most modalities do not produce calibrated intensities. Another challenge is that the edge maps produced by AFNI and FSL seldom form closed contours, making it arduous for users to discern which anatomical structure a given edge corresponds to. In light of these issues, we propose that the second derivative, particularly the Difference of Gaussian (DoG) method, is well-suited for numerous medical imaging applications.

Creating edge maps can be particularly challenging for MRI modalities that are not optimized to maximize intensity contrast between different tissue types. The parameters of a structural T1-weighted scan are typically tuned to highlight contrast between gray matter (GM), white matter (WM) and CSF, and the dataset is typically acquired with high spatial precision (relative to the size of easily identified anatomical structures). In contrast, modalities such as functional magnetic resonance imaging (FMRI), diffusion tensor imaging (DTI) and arterial spin labeling (ASL) are designed to derive properties over time or multiple volumes (such as brain function, white matter bundle orientation, blood flow, respectively). The acquired images often exhibit low signal-to-noise (SNR), poor tissue contrast, poor intensity homogeneity, signal dropout and low spatial resolution, all of which pose unique challenges to edge-detection algorithms as will be discussed below. To improve alignment, previous efforts have altered the imaging acquisition protocol to enhance spatial contrast explicitly (Gonzalez-Castillo et al. 2013).

In this work we briefly describe three seminal edge detection methods: first derivative gradients, second derivative zero-crossing (which we refer to as Difference of a Gaussian, DoG) and the multi-stage Canny algorithm. We describe how popular neuroimaging tools have previously identified edges in 3D images, with FSL using the first derivative and AFNI having adapted a portion of the Canny algorithm.

We propose that the DoG may be particularly well suited to this application, with advantages over both other methods. First, the DoG generates closed loops, which can help identify the shape of 3D anatomical structures when observing 2D slices. Second, the DoG is rapid and easy to implement for 3D images. This contrasts, for example, to the latter stages of the full Canny filter (Bähnisch, Stelldinger, and Köthe 2009). Finally, all algorithms have some parameter dependence, but the primary setting for the DoG is based on the spatial size of features of interest, which is usually known for each imaging application. In contrast, the other methods typically require intensity-based thresholds, which vary by scanner, across volume and are generally not well-calibrated for most MRI modalities. We describe our DoG implementation(s), including new optimization steps. We provide examples for various popular imaging modalities, illustrating that the DoG provides a robust and easily interpretable solution. As developers for and in conjunction with several software packages, we have now implemented DoG in AFNI, FSL, Matlab/SPM, Python/nibabel, niimath and a new web-based viewing tool called NiiVue.

Background: Evolution of Edge Detection Operators

Historically, several computational methods have been devised for edge detection, originally for 2D images and then generalizing to 3D. We briefly describe three generations of methods.

The Roberts cross (Roberts 1963), Prewitt (Prewitt 1970) and Sobel (Sobel 1970) operators examine neighbors, where the first derivative zero-crossing (local brightness peak or valley) is used to infer edges. In 1980, Marr and Hildreth (Marr and Hildreth 1980) introduced a seminal edge-finding algorithm which was based on the second derivative of image intensity and built on earlier work of Wilson and Giese (Wilson and Giese 1977). These “Mexican hat operators” based on Laplacian of a Gaussian (LoG) or similar but computationally more efficient Difference of Gaussian (DoG) attenuates small features, which is robust against ringing artifacts (Gonzalez and Woods 2018). Edges from the original volume can be identified based on the zero crossings of the convolved image. Note that all the produced zero-crossings by definition form closed contours, analogous to contour lines on an elevation map: the edge is the transition between bright islands (blobs) and dark regions in the convolved image.

More recently, the Canny edge detector (Canny 1986) became the tool of choice for edge detection (Gonzalez and Woods 2018). This algorithm uses a five-stage approach to identify edges: blurring, first-derivative filtering, non-maximum suppression to identify and preserve local maxima, intensity thresholding to identify strong and weak edges, and edge tracking by hysteresis to only retain weak edges that improve the continuity of the stronger ones. Note how these stages require multiple parameters of scale, contrast or magnitude; thus, while it can be tuned to high accuracy for a given family of inputs, it can be challenging to retain high quality across a wide range of differing images.

First-Derivative Method (used in FSL)

When generating coregistration quality assurance images, FSL identifies edges based on the first derivative of image intensity (via “slicer”). The outline images created by FSL are computed on 2D images sampled from a 3D volume (e.g. sagittal, coronal and axial images are each computed in plane). Specifically, FSL estimates a voxel gradient using the 2D gradient (central tendency) method previously described. Subsequently, the 2nd and 98th percentile of all the gradients in the image are computed, and the gradient map is converted into a binary edge map using the mean of these two percentiles as a threshold. This formula will mask weak gradients that are present in many images (e.g. low contrast between tissues inside the brain relative to the strong gradients generated at the boundary between the brain and empty space which are created during FSL’s brain extraction).

Canny Method (used in AFNI)

AFNI’s “3dedge3” program adapts the Canny edge detection algorithm, which was originally described for 2D images, to 3D using recursive filters (Deriche 1987; Monga et al. 1991). It should be noted that the hysteresis stage of the Canny algorithm is especially complicated when applied in three dimensions (Bähnisch, Stelldinger, and Köthe 2009), and therefore popular 3D implementations do not typically use this step, including AFNI’s 3dedge3. A specific challenge faced when applying hysteresis thresholding to medical images is the issue of the varying contrast for edges, which makes automatically selecting the two thresholds in the Canny algorithm challenging. However, the edges estimated by 3dedge3 have still been useful for alignment purposes and are efficiently calculated, though perhaps containing more ‘noisy’ edges than the full algorithm would produce. We also note that, rather than generating a binary edge, 3dedge3 outputs a volume whose edge map has continuous intensity representing the magnitude (i.e. strength) of the edge, though it can also be displayed as a binary image (Saad et al. 2009). This approach makes it possible for users to manually adjust the gradient strength or manually threshold the output gradient image to highlight the most well-defined edges, if desired, in place of the hysteresis editing. However, again edges are not guaranteed to be closed loops, and the algorithm can be sensitive to underlying brightness patterns.

DoG: Second Derivative Zero-Crossing Method (proposed)

We suggest that the DoG (second derivative, zero-crossing) method described by Wilson and Giese (Wilson and Giese 1977) and Marr and Hildreth (Marr and Hildreth 1980) seems particularly well suited to the task of creating coregistration quality assurance images used by MRI researchers. Both the first derivative and Canny methods require threshold(s) based on gradient intensity. However, different tissue boundaries may have different relative contrasts, so a single threshold may be hard to define (as illustrated in the Figures described below). Likewise, due to effects like image inhomogeneity the relative tissue intensities can gradually change across the image, requiring the application of different thresholds at different image locations. Fundamentally, image intensity in most MRI sequences has relative arbitrary units (unlike CT, where tissue intensity is typically in known and calibrated Hounsfield units, which have physical relevance), making it difficult to tune brightness-dependent parameters, outside of using mean intensity (which can be more or less influenced by field of view size and other image features). Likewise, the inherently spatial nature of DoG allows it to seamlessly handle datasets with anisotropic voxels. For example, it is common for MRI scans to have much higher spatial resolution in plane than in the slice direction for clinical datasets (to reduce acquisition time) and for some ultra high-field modalities.

The primary adjustable parameter of the Marr and Hildreth (Marr and Hildreth 1980) algorithm is the ‘inner Gaussian’ sigma, which can be set according to the spatial size of the focal features within an image. This is a relatively convenient parameter to be able to set, particularly for visually assessing biological/anatomical alignment, since the minimal scale size of important features is typically known. For example, in human imaging the thickness of the gray matter that comprises the human cortex (and forms the de facto outline of the brain in two-dimensional axial, sagittal and coronal slices) is approximately 2–2.5mm (Fischl and Dale 2000). Pragmatically, in adult human neuroimaging features smaller than 2mm are unlikely to be of interest for coregistration, providing a useful “default” threshold for this algorithm in human neuroimaging to focus on GM while also suppressing extraneous, subscale features. When imaging of other species, an analogous scale size is similarly known. Additionally, users can also explicitly set this parameter to an appropriate length scale for any applications (e.g., high resolution subcortical scans, ultra-high field imaging that discriminates cortical layers, animal scans of smaller brains, etc.). In the Discussion, we note how image voxel dimensions often provide a useful proxy of spatial scale of interest for anatomical images, so a reasonable default for general, cross-species applications is possible.

Further, the closed contours formed by second derivative zero-crossing algorithms are ideally suited for matching the outlines of anatomical structures, especially when viewed slicewise. Specifically, when a user examines the edges of one image on top of the other, the shape of the edges generated by this operator can help identify what structure the edge refers to.

Methods

DoG algorithm and implementations

Here we briefly discuss the original edge algorithm of Wilson and Giese (Wilson and Giese 1977) and Marr and Hildreth (Marr and Hildreth 1980), which is shown schematically in Figure 1. Additionally, we discuss modern updates to features, which have computational and/or interpretational benefits. We note parameters within the algorithm, describing the effects of each and how defaults can be chosen. These methods have now been implemented in multiple open source neuroimaging software including the standalone niimath (which can be compiled to web assembly, for a live demo web page, see https://niivue.github.io/niivue-niimath/), Python, Matlab and 3dedgedog in AFNI (Cox 1996). There are slight differences among these implementations (defaults, optionality, etc.), and we note these when relevant.

Figure 1.

Illustration of the Difference of Gaussian edge detection. A) A schematic plot showing the narrower “inner” (σ = 1.0, red) and wider “outer” (σ = 1.6, blue) Gaussian kernel; the horizontal axis ticks are in units of σ, and the vertical axis shows intensity amplitude. The difference between these two kernels is shown in green. B) Initial T1-weighted MRI image. C) Image blurred with sigma_inner = 2.0. D) Image blurred with sigma_outer = 3.2mm Gaussian. E) Difference between images, C - D. F) Zero-crossings of image E superimposed in red.

In attempting to identify particularly beneficial edge-detection algorithms for 3D image alignment/coregistration and quality control/visualization, we initially examined the LoG- or DoG-based approaches. The LoG function is not separable, and therefore it is computationally expensive to apply to 3D images, as the number of underlying calculations increases with N³. Even with modern computing this is undesirable, and numerous strategies to accelerate this filter have been described (Chen, Huertas, and Medioni 1987). However, there are two popular, simple and computationally efficient methods that can approximate this function well, even for 3D images. The first method involves blurring the data with a Gaussian lowpass filter (which is a spatially separable procedure) and subsequently applying a Laplacian with a 3×3×3 kernel to the image. The second method, which is adopted in the programs implemented here, involves using the Difference of Gaussians (DoG) method, which had been previously described by Wilson and Giese (Wilson and Giese 1977) in the context of psychophysical studies. Here one applies a pair of “inner” and “outer” Gaussian blurs (which is highly efficient, due to separability and Gaussian additive properties) to the source image (see Figure 1A,C,D), with the result being the difference between these two images (Figure 1E). The value of the inner sigma (or “sigma_rad”) is the primary parameter to set in this algorithm, as above noted, which can be set based on the spatial scale of interest. It is important to note that the scale width of the inner Gaussian is two times the inner radius value; therefore, 2*sigma_rad is the appropriate quantity to make less than a minimal structural length.

The ratio of these two Gaussians (the “ratio_sigma”) is a secondary parameter that affects the amount of detail traced with edges. Wilson and Giese (Wilson and Giese 1977) suggested a value of ratio_sigma=1.5, and Marr and Hildreth (Marr and Hildreth 1980) suggested 1.6 in their work. Neither of the initially suggested values of ratio_sigma were definitively derived, and it is possible that this parameter could be application-dependent. It must be noted, though, that the similarity of DoG and LoG functions decreases as this ratio increases, so one would not expect a very large value---perhaps even values of 1.5–1.6 might be considered surprisingly large. Therefore, we investigated its effect here.

Having identified a quick and robust method for implementing DoG, the next step in the edge algorithm is to define locations of zero-crossings in the DoG image, which become the edges themselves (Figure 1F). There are several approaches for computing edge crossings, including those that offer sub-pixel accuracy (Huertas and Medioni 1986). One conceptually simple approach is to count a voxel as being an edge if the location is positive in the DoG and yet shares a neighbor with a negative DoG value (or vice versa, with a negative pixel neighboring a positive one). This would preferentially select the edge from the “bright” (or “dark”, respectively) region in the original image, which in some applications might be a usefully consistent choice. A related alternative that would not select one side consistently would be to define a zero crossing edge as a voxel that shares a neighbor with both opposite polarity and has greater magnitude/intensity. This is the strategy adopted in our niimath C program implementation. This method is fast in low level languages, but the reliance on conditionals and non-vectorized loops makes it difficult to implement efficiently in most higher level languages. On the other hand, our Matlab SPM, Python and AFNI (3dedgedog, also in C) programs use a three step approach to identify zero crossings. First, binarize the DoG image, so that voxels >=0 become 1 and the rest are zero. Then, perform the Euclidean Distance Transform (EDT) on this image, which calculates each voxel’s distance from the mask boundary; we utilize the highly efficient algorithm by Felzenszwalb and Huttenlocher (Felzenszwalb and Huttenlocher 2012) for this step. Finally, select voxels with a distance of one edge length (or some other chosen value) to define the boundary. This basic method provides consistent boundaries (e.g., all on one side of the contrast edge) of uniform thickness (which adapts to anisotropic voxels and, if needed, can be adjusted), while maintaining computational efficiency.

The above steps define the central DoG algorithm. We note a few options and variations that are also implemented in specific programs, for various use cases. The main algorithm finds edges defined in the 3D volume, but sometimes it can be useful to find only edges within a 2D plane (e.g., for slicewise viewing of a volume); the above algorithm translates directly to this, and both the niimath and AFNI implementations have options for calculating slicewise edges through any of the main image orientations. Additionally, the input maps can be “automasked”, to avoid showing edges of background/noise, which can be visually distracting. AFNI’s 3dedgedog has options for selecting a particular side of a boundary: positive (=higher contrast side), negative (=lower contrast side), both (=positive+negative bounds), and both-signed (= same as both, but higher and lower contrast sides have positive and negative values, respectively). 3dedgedog can also display an approximate magnitude of each boundary location, estimated as the standard deviation in that voxel’s immediate neighborhood. Finally, 3dedgedog allows the user to select values of ratio_sigma and sigma_rad, as well as to define the kernels anisotropically or in terms of the number of voxels.

Testing data

To demonstrate the properties of the both existing and proposed edge detection strategies, we utilize the following publicly available MRI datasets. Firstly, we use the standard T1-weighted (T1w) anatomical dataset from the AFNI Bootcamp¹, referring to this dataset as “sub-001”. This was acquired at 3T with standard resolution of near 1mm resolution (voxel = 1 × 0.938 × 0.938 mm³); this is particularly useful for examining DoG algorithm edges with changing parameters.

Additionally, we used a dataset containing multiple modalities from a single individual (“sub-002”), being representative of typical T1w and T2*-weighted images that would be acquired as part of a standard research neuroimaging protocol: T1w, arterial-spin labeling (ASL), diffusion weighted imaging (DWI) and functional MRI (fMRI). These images were acquired on a 3T Siemens Prisma Fit Scanner. The fMRI scan used a 90×90 matrix, 50 slices, 2.4mm isotropic voxels, TR=1650ms. The DWI sequence used a 140×140 matrix, 80 slices, 1.5mm isotropic voxels, TR=5700ms. The ASL (arterial-spin labeling) scan used a 70×70 matrix, 14 slices, 3.0×3.0×7.5mm voxels, TR=2500ms. The T1w sequence was acquired at high resolution with 0.8mm isotropic voxels.

We used AFNI’s 3dAllineate to rigidly align the first volume of each EPI dataset to the T1w volume, resampling to the latter’s grid. No corrections for EPI artifacts were applied, as our goal was to generate QA images that show regions with both good and poor edge alignment.

We are making our testing datasets publicly available (https://github.com/neurolabusc/DoG) allowing users to replicate our work and see the influence of tuning parameters interactively using any of our implementations (Python, Matlab, and compiled executables).

Results

DoG parameter investigation

We calculated DoG-derived edges for the standard T1w anatomical (approx. 1mm isotropic voxels) of sub-001, investigating part of the parameter space of the algorithm. The sigma_rad (= inner Gaussian sigma) took values from 0.6–2.6 mm, and ratio_sigma (= ratio of outer to inner Gaussian sigmas) took values from 1.2–2.0, using 3dedgedog. The results of these calculations are shown in Figure 2, for a zoomed-in portion of a sagittal slice that contains several scales of features and image contrast.

Figure 2.

A grid of 3dedgedog results showing the effects of two main parameter choices in the DoG algorithm: the sigma_rad (radius of inner Gaussian) and the ratio_sigma (ratio of outer to inner Gaussian sigmas). The underlaid dataset, which was the input, is a standard T1w anatomical (sagittal slice at x = 40R). Blue arrows highlight example features where results differ noticeably with sigma_rad: edges of finer features can be observed as this parameter decreases. This parameter should be smaller than the desired scale of structure to observe (here, <2–2.5mm, which is the typical GM thickness), though as the value gets smaller one might consider some of the observed features to be effectively noise. Green arrows highlight example features where results changed noticeably with ratio_sigma: smaller and finer features can “melt away” as this parameter increases. This likely reflects the fact that as ratio_sigma approaches unity, the DoG method better approximates the LoG. Interestingly, it is often difficult for the DoG approach to show both the gray-white boundary and the pial boundary (“outer” GM bound), with the latter typically observed with other large-contrast boundaries.

The effects of altering sigma_rad are observed by comparing results in a given column. As expected, edges of smaller features are visible as sigma_rad decreases; as sigma_rad increases, more boundaries are either missed or incorrectly merged across unrelated structures. Blue arrows highlight representative examples of features changing with this parameter. It should also be noted that fine, low-contrast features also appear at the low sigma_rad values, such as within-WM edges; these might be considered noisy, for judging alignment, and perhaps could be filtered with considerations of non-binarized boundary magnitude.

The effects of altering ratio_sigma can be observed across a row. These effects tend to be fairly subtle, but there are notable changes with this parameter. In particular, as ratio_sigma increases, finer features tend to decrease in size, notably at sharp curves or finer structures. Green arrows highlight representative examples of this.

Overall, for smaller sigma_rad and ratio_sigma, the edges appear to highlight a large number of the important edge features within the image. For assessing visual alignment of brains, one typically focuses on locations of tissue contrasts, such as the gray-white boundary and pial surfaces, and ventricle and other CSF boundaries. Sulcal and gyral structures are clearly visible here, though at these scales the edges typically show only one of the GM boundaries. For alignment purposes, the parameters used for the upper left panels of Figure 2 should provide the best representations (low ratio-sigma and low sigma_rad).

Comparison of edges across software and modalities

The results of estimating edges for different MRI modalities (which have very different tissue and image contrast) with various algorithms used in different software toolboxes for sub-002 are shown in Figures 3–5 (for EPI-derived fMRI, DWI and ASL acquisitions, respectively, each with T1w). In each figure, the left set of panels shows the EPI-derived sequence as underlay with the T1w edges overlaid, and the right set of panels shows the T1w data underlaid and the EPI-derived edges overlaid; the top row shows FSL’s first derivative method (via “slicer”), the middle row shows AFNI’s Canny method (via “3dedge3”), and the bottom row shows AFNI’s new DoG method (via “3dedgedog”; “niimath -dog” uses the same algorithm and produces nearly identical results).

Figure 3.

Example of rigid-body coregistration of a subject’s fMRI EPI to a T1w volume using different edge approaches (from top row, downward): FSL’s slicer (first derivative-based method), AFNI’s 3dedge3 (Canny-based method) and AFNI’s 3dedgedog (new DoG-based method, with sigma_rad=1.4 and ratio_sigma = 1.4). In the left panels, the EPI-based volume is underlaid beneath edges from the T1w volume, and in the right panels, the T1w is beneath EPI-derived edges. The sagittal, coronal and axial images are centered at (−7.9, 42.4, 20.8) in RAI-DICOM coordinates, and each image left = subject right.

Figure 5.

Example of rigid-body coregistration of a subject’s ASL volume to a T1w volume using different edge approaches (from top row, downward): FSL’s slicer (first derivative-based method), AFNI’s 3dedge3 (Canny-based method) and AFNI’s 3dedgedog (new DoG-based method, with sigma_rad=1.4 and ratio_sigma = 1.4). See Figure 3‘s caption for remaining details.

In each image, the largest image contrasts occur at CSF, the skull and other facial feature boundaries, and each algorithm shows these clearly. However, the sulcal and gyral features, which are most important for judging meaningful brain feature alignment, appear differently across algorithms. Notably, the top row in each case shows a sparse representation of features, missing many edges and producing difficult to interpret maps.

The middle (Canny) and bottom (DoG) panels of figure 5 show remarkably similar representations, with most features clearly present in each and therefore beneficial for judging coregistration across the volume. This allows one to verify locations of good alignment---for example, most posterior sulci and gyri across modalities---as well as to identify those of poor alignment---such as the anterior regions where EPI distortion is present, as well as in the DWI’s ventricles. In general, the DoG method produces finer edges, allowing for more detailed comparisons of boundaries; for example, comparison T1w edges at the ventricle boundaries. For all methods, the edges derived from the EPI methods are noisier, as expected; the Canny method (middle panel) shows less noise within WM and clearly delineates major boundaries, but the DoG method appears to highlight more and finer features of the sulcal and gyral patterns.

Additional edge estimation considerations

Both niimath and 3dedgedog provide the user with options to generate edges in a slicewise (2D) manner, which can be particularly useful and efficient for standard slicewise viewing of volumetric data. Figure 6 demonstrates this, using the sub-001’s anatomical dataset as input, using 3dedgedog. In the upper left panel, binary edges are estimated volumetrically using a standard sigma_rad=1.2 and ratio_sigma=1.2,. Note how some lines are particularly thick, representing out-of-plane structures that are tangential to the currently viewed slice. In the second row left panel, the “-only2D sag” option is used, showing cleaner lines within the slice; this facilitates views within the slice, and is particularly convenient for interactive displays and refreshing datasets (as the 2D estimation is faster to calculate). For static images, the 3D-derived edges may still be preferable.

Figure 6.

Additional considerations and features for edge estimation and visualization with 3dedgedog: 2D planar (slicewise) edges, boundary scaling and thresholding, and upsampling. The top-left panel shows standard, volumetric edges estimated for sub-001’s T1w dataset. The second row-left panel shows edges estimated only within the viewed sagittal plane: note that the thickest lines from the top panel, which appear when the slice is tangential to a structural boundary, do not appear, providing a clearer view of the given structures. The second column shows the edges when scaling (according to normalized local variation, red being highest) is applied, and in the third column this is translucently thresholded---thus, edges mostly likely to be “noise” are filtered. In the fourth column a smaller Gaussian is used to show more details, with both scaling and thresholding applied. The bottom two rows show the same types of edges, but first the input data has been upsampled by a factor of two in each dimension---the finer lines helpfully reduce image clutter, particularly when smaller sigma_rad is used.

In the second column of Figure 6, each boundary point is scaled by a measure of normalized local variation, ranging from 1 (yellow) to 100 (red). This information can be useful for removing or obscuring boundaries that are likely to be considered noise. Translucent thresholding of these scaled edges is applied in the AFNI viewer in the third column, leaving the main structural features primarily intact. In the fourth column, similarly scaled and thresholded edges are shown again, for the case of smaller sigma_rad=0.6. In this last column, both the gray-white and pial boundaries of the cortex are observable, while much of the “noise” of intra-WM variation is filtered out.

The bottom two rows of Figure 6 show edges created in the same manner as those in the top two rows, but with the difference that the anatomical dataset was first upsampled with 3dresample prior to edge estimation. The voxel edge length was halved in each dimension (so the output dataset has 8 times as many voxels), and nearest neighbor (NN) interpolation was used to avoid blurring the data. In this case, the edges can now be represented more finely, which is particularly important when viewing those estimated with sigma_rad=0.6 in the fourth column. In this case, the finer details can generally be appreciated more, and the image looks less cluttered. The computational cost is much higher, so this is less feasible for interactive updates, but still only takes about 10 seconds to estimate on a modern laptop (Apple MacBook with M2 CPU).

Additional modalities

Our online live demo (https://niivue.github.io/niivue-niimath/) allows users to replicate our own tests with additional modalities. This web page allows users to select both the parameters for the filter. The page embeds many modalities including T1-weighted, T2-weighted, fluid-attenuated inversion recovery T2 (FLAIR), arterial spin labeling (ASL), T2*-weighted blood-oxygenation level dependent functional MRI(BOLD fMRI), post-contrast enhanced T1-weighted imaging, balanced Proton-Density imaging (PD), raw and derived diffusion-weighted images, as well as CT scans with and without contrast. The demo also underscores the need to appropriately set the feature size: a CT scan from a small Iguana is provided which requires tuning of the parameters for the species. Example edges are shown in Figure 7. Finally, this demo allows users to drag and drop their own images (in NIfTI, DICOM, NRRD, and other voxel based formats) to explore and export results.

Figure 7.

Performance of FSL-style first-derivative (left 3 columns) and of the proposed Difference of Gaussian (right 3 columns) for 7T data with T1-weighted (T1), T2-weighted (T2) and BOLD T2*-weighted (T2*) weighted images. For each column, the white text refers to the modality of the grayscale underlay, and the red text refers to the modality of the red edges. The raw data is from the OpenNeuro “Large Single-Subject Functional MRI Datasets at 7T” repository. In this example, FSL’s FLIRT was used with default options to perform a 12-parameter (translation, rotation, rotation and zooms; each in 3 dimensions) to align the T2* and T2 to the T1. This was successful for the T2 but not the T2*. The edge computation can be computed for these images as well as other modalities using our live demo, allowing users to tune parameters interactively. For these images, the FSL-style solution is based on `-dehaze 3 -edge -thrp 50 -bin` while the DoG solution is based on `-dehaze 3 -dog 1.4 2`. Note that the relatively low-contrast for gray-and white matter as well as the intensity homogeneity fluctuations impact the sensitivity of the first-order derivatives.

Discussion

We described and tested a simple and robust method for identifying edge boundaries on MRI scans. The approach relies on the Difference of Gaussian (DoG) algorithm. While some existing methods do produce useful edges (such as the Canny method), we suggest that the DoG method has some particularly useful properties for neuroimaging, particularly for registration assessment. First, it generates closed loops, even in noisy images, that help observers inspecting 2D slices to recognize the shape of 3D structures (and therefore their anatomical identity) more easily than disjointed lines. Additionally, the major adjustment parameter of the DoG is based on the spatial scale of the features of interest (which is typically known for a given MRI scan) rather than on the image’s relative intensity (which may vary across different regions of the brain and from scan-to-scan). This makes it a good choice to check performance for various forms of affine as well as non-linear spatial correction including correcting for motion across time, geometric correction for eddy currents and echo planar imaging artifacts, coregistration between modalities, as well as spatial normalization between individuals.

The spatial resolution tuning provides useful consistency across images that can have differing brightness inhomogeneities across scanners and modalities. This does mean, however, that the adjustment parameter must be adjusted/tuned prior to application in different species, for example, the monkey or rat brain, where the features would often be much lower than the 2mm scale used for humans. Depending on the spatial resolution and feature size, the blur sigmas could be adjusted. While we particularly focus on the known feature size of anatomical structures, it is worth noting that the spatial resolution of many sequences is adjusted to preserve a reasonable contrast to noise ratio. In other words, modalities with inherently low contrast to noise are often acquired with low spatial resolution (sampling a larger volume of hydrogen). Likewise, since signal increases with field strength, spatial resolution is often correlated with this attribute. This suggests an alternative (and cross-species) method for using the DoG spatial filter, which is based on the size of the voxels for each image; this is possible in some of the implementations coded here. Both approaches leverage the fact that the DoG uses space rather than intensity as the fundamental tuning parameter. The move to ultra-high field MRI makes the deployment of DoG edge methods timely, as image inhomogeneity and resolution anisotropy are prominent with increasing field strength.

The DoG-based approach detailed in this paper is particularly well-suited to the construction of vivid edges that clearly illuminate the quality of inter-modality coregistration (e.g. alignment between T2*-weighted functional MRI scans and T1-weighted anatomical MRI scans acquired from the same). Integration of this approach in existing brain imaging preprocessing pipelines has the potential to improve early detection of erroneous coregistration that can be propagated to later stages. Moreover, the ability of users to tune the algorithm using a simple, easy to understand biologically calculable parameter (thickness of feature that needs to be highlighted), makes this technique flexible enough to be used by researchers interested in edge detection within systems where certain key properties are already known/consistent across samples.

While our approach is ideal for its proposed application, we would note that other visualization approaches may be more useful—or at least highly complementary—for later stages of image processing such as alignment to a template, where the primary goal is to warp (often nonlinearly) the shape of an individual’s brain to match a standard reference dataset. Standard templates often have known tissue probability maps (i.e. the probability of encountering of gray matter, white matter and cerebral spinal fluid at different locations is known), and the quality of coregistration can be improved by simultaneously segmenting the individual’s brain to these tissues (Ashburner and Friston 2005). For template alignment, the use of a priori tissue maps can provide nice edge maps for a ground-truth solution. The Conn toolbox (Whitfield-Gabrieli and Nieto-Castanon 2012) is one software program that benefits from this approach. Likewise, boundary-based registration (Greve and Fischl 2009) methods directly estimate tissue boundaries and edges based on these estimates are ideal for evaluating the performance of this method. Clearly, some edge-based methods are tailored for specific applications. We merely suggest that DoG is a robust edge finding method for 3D images that requires little a priori expectations. It may be easily added to other visualization and coregistration strategies.

The proposed DoG-based edge-detection method is easy to implement in currently available neuroimaging pipelines. Smoothing data with a Gaussian kernel is already a common step for most neuroimaging pipelines as it diminishes high-frequency noise, reduces the effective number of statistical comparisons and makes it possible for researchers to interrogate their data using standard parametric statistical tests that assume gaussian distributions (Worsley et al. 1996). Therefore, as most neuroimaging pipelines include optimized Gaussian smoothing, the implementation of DoG is straightforward.

Poldrack and colleagues (Poldrack, Gorgolewski, and Varoquaux 2019) note that SPM/MATLAB, FSL, AFNI and Python are heavily used in neuroimaging, with each mentioned in more than 1000 publications per year. We have provided open source DoG solutions for users of each of these tools. This allows users to leverage this edge finding method using the tools they are familiar with. Specifically, the AFNI tool ‘3dedgedog’ (https://github.com/afni/afni), the niimath `dog` functions (https://github.com/rordenlab/niimath), the FSL fslmath ‘-dog’ (https://fsl.fmrib.ox.ac.uk/fsl/fslwiki), the Matlab/SPM spmDoG script (https://github.com/neurolabusc/spmDoG), and the Python/nibabel PyDog script (https://github.com/neurolabusc/PyDog). Finally, we provide a web-assembly based implementation that can be used in any modern web capable device (phone, table, computer), embedded with our NiiVue visualization tool (this provides a zero foot print drag-and-drop solution that is compatible with any web enabled device, allowing users to calculate these images locally to ensure privacy, see https://niivue.github.io/niivue-niimath/ for a live demo).

Limitations

As noted, edge-detection visualization is a common quality assurance step for automated spatial alignment routines and properties of different methods will often be subjective. Indeed, the main purpose of these displays is to provide an objective opinion regarding whether the objective cost function used by the automated routines is appropriate. However, some argue that the “spaghetti effect” of the closed contours created by the DoG algorithm is undesirable for some applications (Gonzalez and Woods 2018). Furthermore, DoG does require setting a spatial parameter, which may differ based on the species or the feature of interest. For this reason, intensity-based thresholds may be preferable for modalities where the intensity is calibrated. Consider our live-demo where the teeth of humans and reptiles have similar intensity on CT scans, even though the spatial scale differs. However, it is worth noting that the Canny filter does also require a spatial size filter and that the first derivative method is susceptible to noise without a spatial filter such as a Gaussian blur. Therefore, while the DoG requires selecting a spatial filter, this is also best practice for competing methods.

Conclusion

In summary, the DoG-based approach is a computationally efficient method to estimate edges within biomedical and neuroimaging datasets. The edges themselves have the useful property of boundedness, which is typically appropriate and useful for medical imaging applications. As explored above, the primary parameters for controlling the outputs are size-based, rather than brightness-based, which is also typically convenient for the variety of images obtained across the field. Furthermore, having the approximate ability to report edges “down to a certain scale length” is often a useful way of examining an image. The method is also fairly flexible, with benefits of simple upsampling and edge-scaling shown to be useful in further balancing the dual goals of seeing more details and not being distracted by small, noisy features. This methodology has immediate benefits for coregistration validation, and we believe it will have other beneficial applications in other domains with 3D voxel-based datasets such as confocal mircroscopy.

Figure 4.

Example of rigid-body coregistration of a subject’s DWI (b=0 s/mm²) to a T1w volume using different edge approaches (from top row, downward): FSL’s slicer (first derivative-based method), AFNI’s 3dedge3 (Canny-based method) and AFNI’s 3dedgedog (new DoG-based method, with sigma_rad=1.4 and ratio_sigma = 1.4). See Figure 3‘s caption for remaining details.

Highlights.

• Converting images to edge maps allows users to overlay one image onto another.

• Edge maps help users ensure success of spatial coregistration and correction.

• Many tools define edge maps based on voxel intensity, which is rarely calibrated.

• Difference of Gaussian edges are based on spatial size, which is typically known.

• We provide Matlab, Python, Web and command line tools to compute DoG.