Joint Representation of Connectome-scale Structural and Functional Profiles for Identification of Consistent Cortical Landmarks in Macaque Brain

Abstract

Discovery and representation of common structural and functional cortical architectures has been a significant yet challenging problem for years. Due to the remarkable variability of structural and functional cortical architectures in human brain, it is challenging to jointly represent a common cortical architecture which can comprehensively encode both structure and function characteristics. In order to better understand this challenge and considering that macaque monkey brain has much less variability in structure and function compared with human brain, in this paper, we propose a novel computational framework to apply our DICCCOL (Dense Individualized and Common Connectivity-based Cortical Landmarks) and HAFNI (Holistic Atlases of Functional Networks and Interactions) frameworks on macaque brains, in order to jointly represent structural and functional connectome-scale profiles for identification of a set of consistent and common cortical landmarks across different macaque brains based on multimodal DTI and resting state fMRI (rsfMRI) data. Experimental results demonstrate that 100 consistent and common cortical landmarks are successfully identified via the proposed framework, each of which has reasonably accurate anatomical, structural fiber connection pattern, and functional correspondences across different macaque brains. This set of 100 landmarks offer novel insights into the structural and functional cortical architectures in macaque brains.

1. Introduction

Representation of structural and functional profiles for the establishment of a common structural and functional cortical architecture across individuals and populations has been of significant interest in the brain mapping field. Thanks to advanced multimodal neuroimaging techniques for quantitatively representing the whole-brain structural profiles (e.g., mapping the fiber connections using diffusion tensor imaging (DTI) (Mori and Zhang, 2006) ) or the functional profiles (e.g., mapping functional localizations using functional MRI (fMRI) (Logothetis, 2008) ) of the same brain, a variety of our recent studies have attempted to construct a connectome-scale, common representation of the human brain based on either structural or functional profiles (e.g., Zhu et al., 2011; Zhu et al., 2012; Yuan et al., 2013; Li et al., 2013; Jiang et al., 2014a; Jiang et al., 2015b; Lv et al., 2015a; Lv et al., 2015b; Lv et al., 2015c; Zhang et al., 2016). For example, for the structural profiles, our previous work (Zhu et al., 2012) successfully identified 358 consistent and common cortical landmarks across different human brains, each of which possesses group-wise consistent DTI-derived fiber connection patterns. This set of 358 landmarks was named as ‘Dense Individualized and Common Connectivity-based Cortical Landmarks (DICCCOL)’ (Zhu et al., 2012). Later on, more constraints (e.g., anatomy identity) were integrated into the landmark identification procedure and many other meaningful landmarks were identified and named as ‘anatomy-guided DICCCOL (A-DICCCOL)’ (Jiang et al., 2015b). These two sets of dense landmarks are complementary and they jointly represent the connectome-scale structural architecture of human brains. For the functional profiles, our recent works (Lv et al., 2015a; Lv et al., 2015b) aggregated all of hundreds of thousands of fMRI (either task fMRI or resting state fMRI) signals within the whole brain of one subject into a big data matrix, and decomposed the big signal matrix into an over-complete dictionary basis matrix and a sparse reference weight matrix via an efficient and effective online dictionary learning and sparse representation framework. It has been shown that connectome-scale well-characterized functional brain networks (including both task-evoked networks and intrinsic connectivity networks) can be effectively and robustly reconstructed via the computational framework (Lv et al., 2015a; Lv et al., 2015b; Zhang et al 2013). This novel strategy which aims to construct the connectome-scale functional architecture of the human brain was named as ‘Holistic Atlases of Functional Networks and Interactions (HAFNI)’ (Lv et al., 2015a). In addition, there have been intensive literature studies demonstrating the close relationship between the white matter (WM) structures and gray matter (GM) functions. To extensively study the relationship between brain structure and function, fusing DTI and fMRI data has received increasing interest recently. The major advantage of multimodal data fusion studies compared with the single modality study is to integrate the complimentary structural and functional information together to study the common characteristics of functional and structural profiles and to reveal the common structural and functional brain architecture. However, due to the remarkable variability of structural and functional architectures in human brain (Liu 2011), it is still challenging to jointly represent the connectome-scale structural (e.g., DICCCOL) and functional (e.g., HAFNI) profiles to establish a common cortical architecture which can comprehensively encode both structure and function characteristics in human brains.

Alternatively, the macaque brain has much less variability between structure and function across different individuals compared with human brain (e.g., Armstrong et al., 1991; Baaré et al., 2001; Chen et al., 2012; Zhang et al., 2012; Zhang et al., 2009; Zilles et al., 1988; Sereno and Tootell et al., 2005; Schoenemann et al., 2006; Dehaene 2005; Aboitiz et al., 1997), and it has been widely adopted as a critical nonhuman primate model to study brain structure and function (e.g., Van Essen et al., 2001; Van Essen 2004; Van Essen et al., 2011; Felleman and Van Essen 1991; Paxinos and Franklin 2004; Markov et al., 2011; Galletti et al. 1999; Preuss and Goldman-Rakic 1991; Lyon and Kaas 2002; Baylis et al., 1987; Kolster et al., 2009; Ferry et al., 2000). Moreover, a variety of recent studies have demonstrated the close relationship between DTI-derived fiber connections and fMRI-derived functions in macaque brains (e.g., Lee et al., 2003; Khachaturian et al., 2010; Passingham et al., 2009; Calabrese et al., 2015; Zhang et al., 2013; Zhang D et al., 2013; Li et al., 2011). These findings directly support the neuroscience theory that each brain’s cytoarchitectonic area has a unique set of extrinsic inputs and outputs, named as ‘connectional fingerprint’ concept, which largely determines the functions of each brain area (Passingham et al., 2002). Therefore, it might be suitable and feasible to jointly represent the structural and functional profiles with multimodal DTI/fMRI data to discover common structural and functional cortical architectures in macaque brains.

Based on the above rationale, the goal of this paper was to apply our DICCCOL and HAFNI frameworks to macaque brains to jointly represent the structural and functional profiles for identification of a set of consistent and common cortical landmarks with both reasonably good structural and functional correspondences across different macaque brains. The major contributions and novelty of this work, in comparison with prior DICCCOL and HAFNI frameworks, are as follows. First, we initialize the landmark locations in macaques with connectome-scale functional network peaks derived from HAFNI, instead of random initialization or manual labelling in previous studies (e.g., DICCCOL identification in human brains (Zhu et al., 2012; Jiang et al., 2015b)). The major advantage of initializing landmark locations with functional network peaks is that it enables and facilitates the joint representation of structural connectivity and function afterwards. Since HAFNI provides dense connectome-scale functional networks and the associated functional peaks which are consistent across different subjects, it is possible that those dense functional network peaks also exhibit consistent structural connection patterns in macaque brains. Second, during the group-wise landmark location optimization procedure based on the initialized locations, we integrate meaningful anatomical identity, functional information, DTI-derived structural fiber connection pattern, and spatial consistency constraints. These meaningful constraints ensure that the finalized landmarks with optimized locations have reasonable anatomical, structural fiber connection pattern, and functional correspondences across different macaque brains. Totally, we identified 100 consistent structural and functional landmarks and all of them have been publicly released online at: http://caid.cs.uga.edu/~szhang/dicccol.html. These dense landmarks encode the joint connectome-scale structural and functional profiles of macaque brains and shed novel insights into the regularity and variability of cortical architectures in the developing primate brain.

2. Materials and Methods

Our framework of joint representation of connectome-scale structural and functional profiles for identification of consistent landmarks includes three major steps (marked as 1–3 in Fig. 1): (1) representation of connectome-scale functional profiles based on resting state fMRI (rsfMRI) data for landmark location initialization (Section 2.2), (2) joint constraint of connectome-scale structural, functional, and anatomical profiles based on MRI/DTI data for landmark location optimization (Section 2.3), and (3) evaluation and determination of optimized landmark (Section 2.4). The materials and methods section is organized by following these 3 major steps, respectively.

Fig.1.

The pipeline of the proposed computational framework of joint representation of connectome-scale structural and functional profiles for landmark identification. The three major steps (representation of connectome-scale functional profiles based on resting state fMRI (rsfMRI) data for landmark location initialization (step 1), joint constraint of connectome-scale structural, functional, and anatomical profiles based on MRI and DTI data for landmark location optimization (step 2), and evaluation and determination of optimized landmark (step 3)) are labeled as 1–3, respectively. (a) Identified connectome-scale groupwise consistent functional networks across individual subjects. Axial slices of spatial maps of 3 example networks in the template space are shown for illustration. (b) Identified peak foci in the components of each functional network. The peak foci of the selected illustration networks are shown in red dots. (c) All identified peak foci are mapped to individual cortical surfaces as the initial locations of landmarks (represented as red bubbles). (d) Optimization of landmark locations on cortical surfaces based on group-wise consistency of anatomy identity (gyral/sulcal regions), structural fiber connection pattern, and spatial consistency constraints. (e) Finalized consistent and common cortical landmarks (shown as green bubbles) across individual macaque brains which encode joint connectome-scale structural and functional profiles.

2.1. Data Acquisition and Preprocessing

The subjects were rhesus monkeys (Macaca mulatta) living in the breeding colony maintained at the Yerkes National Primate Research Center (YNPRC), at Emory University (Lawrenceville, Georgia. Six 6-month-old macaques with multimodal T1-weighted MRI, DTI and rsfMRI scans were used in this study. These subjects represent typically developing, socially-housed rhesus monkeys included in a larger study (Howell et al, 2017; McCormack et al, 2015; Shi et al, 2017). They were raised with their mothers and families for the entire duration of the study in large social groups and they span all social hierarchy strata (high, medium and low ranking families). Standard high fiber, low fat monkey chow and seasonal fruits and vegetables were provided twice daily, in parallel to enrichment items. Water was available ad libitum. All studies were performed in accordance with the Animal Welfare Act and the U.S. Department of Health and Human Services “Guide for the Care and Use of Laboratory Animals”, and approved by the Emory University Institutional Animal Care and Use Committee (IACUC).

Images were acquired on a 3T Siemens Trio scanner (Malvern, PA) at the YNPRC Imaging Center using an 8-channel array, transmit and receive knee volume coil. The subjects were scanned supine under isoflurane anesthesia (standardized to the lowest possible −0.8–1% isoflurane, inhalation- to minimize effects on functional connectivity). A custom-made head holder with ear bars and a mouth piece was used to secure and prevent movement of the head in order to avoid motion artifacts. Animals were intubated, administered dextrose/NaCl (I.V.) for hydration, placed over an MRI-compatible heating pad to maintain temperature and physiological measures monitored during the scans. After each subject was scanned and had completely recovered from anesthesia, it was returned to its mother, and the mother-infant dyad returned to their social group.

The neuroimaging parameters are as follows. T1-weighted MRI data were acquired with a magnetization prepared rapid gradient echo (MPRAGE) sequence with repetition time(TR)/inversion time/echo time(TE) = 3000/950/3.31 msec, flip angle = 8°, matrix is 192×192×128, and resolution is 0.6×0.6×0.6 mm³ with 4 averages. DTI data were collected with a single-shot dual spin-echo EPI sequence with GRAPPA (R=3), b value = 1000 sec/mm², 62 directions of diffusion-weighting gradients, repetition time (TR)/echo time (TE) = 5000/90 msec, FOV (field of view) is 83.2×83.2 mm², matrix size is 64×64×43 covering the whole brain, and resolution is 1.3×1.3×1.3 mm³ with zero gap, and 12 averages. One image without diffusion weighting (b=0 sec/mm²) was acquired with matching imaging parameters for each average of diffusion-weighted images. The rsfMRI scans were acquired using an echo planar imaging (EPI) sequence, with TR/TE = 2060/25 msec, matrix = 85×104×65, resolution = 1.5×1.5×1.5 mm³, and 2×15 min scans with a total volume number (time points) per scan of 400.

The preprocessing of T1-weighted MRI data includes skull stripping, motion correction, tissue segmentation, and white matter surface reconstruction via FSL (http://fsl.fmrib.ox.ac.uk) and FreeSurfer (http://surfer.nmr.mgh.harvard.edu). For DTI data, skull stripping, eddy correction, and axonal pathway orientation estimation were performed via FSL-FDT. MedINRIA (https://med.inria.fr/) was then adopted to reconstruct DTI-derived whole-brain deterministic fibers. For rsfMRI data, skull stripping, motion correction, spatial smoothing, temporal pre-whitening, slice timing correction, global drift removal, and band-pass filtering similar as those in (Mantini et al., 2011) were performed using both publicly available FSL toolkits and in-house developed tools (Li et al, 2016). We used the INIA19 macaque brain atlas (Rohlfing et al., 2012) as the template space for anatomical reference and aligned all functional network patterns in individual fMRI spaces to this common atlas space via linear registration (using FSL-FLIRT) to identify consistent functional networks and associated peak foci. For each subject, those peak foci in the INIA19 template space were aligned and mapped onto the reconstructed cortical surface in individual MRI space via linear registration. In order to utilize the DTI-derived whole-brain deterministic fibers as structural profiles, we further aligned the functional peak foci and cortical surface in individual MRI space to individual DTI space for joint representation of the structural and functional profiles.

2.2. Representation of Functional Profiles for Landmark Initialization

There are two steps for the representation of connectome-scale functional profiles for landmark location initialization. First, we identified connectome-scale group-wise consistent functional networks across individual macaque brains via our recent HAFNI framework of dictionary learning and sparse coding of whole-brain rsfMRI data (Lv et al., 2015a; Lv et al., 2015b). Second, we identified the connectome-scale functional peak foci with the largest functional activity value in each component of each functional network. Those identified connectome-scale functional peak foci which have functional correspondences across different subjects are mapped to individual cortical surfaces as the initial locations of landmarks. The details of representation of the connectome-scale functional profiles are as follows.

Dictionary learning and sparse coding approaches have been successfully applied for brain fMRI time series analysis and functional brain network identification (e.g., Lee et al., 2011; Oikonomou et al., 2012; Abolghasemi et al., 2013; Zhao S et al., 2015; Zhang S et al., 2015; Jiang et al., 2014b; Jiang et al., 2015a; Jiang et al., 2015c; Lv et al., 2015a; Lv et al., 2015b). Based on our recent dictionary learning and sparse coding framework (Lv et al., 2015a; Lv et al., 2015b), the whole-brain fMRI signal matrix of a single subject can be represented as a dictionary matrix D and a sparse coefficient matrix α as illustrated in Fig. 2. Specifically, the rsfMRI signal in each voxel of whole-brain fMRI data in one subject is extracted and normalized to zero mean and standard deviation of 1 (Lv et al., 2015b). Then the wholebrain normalized signals are arranged into a matrix X∈R^t×n (Fig. 2a) with n columns containing n rsfMRI signals from n voxels. t is the rsfMRI time points. By using the publicly available online dictionary learning toolbox (Mairal et al., 2010), each column of X is decomposed and represented as sparse linear combination of dictionary atoms from a learned dictionary matrix D so that X=D×α, where D ∈ R^t×m is the learned dictionary matrix (m is the dictionary size) (Fig. 2b) and α∈R^m×n (Fig. 2c) is the sparse coefficient matrix. Each element of α represents the functional activity value. Each row of α can be mapped back to the brain volume as a functional brain network pattern (Fig. 2c). According to the experience from previous study (Lv et al., 2015b), we empirically set the same m=400 for the macaque brain rsfMRI data. Moreover, once we obtain m functional brain network patterns for each single subject, we align all patterns across all subjects to the INIA19 template space via linear registration and adopt the widely used k-means clustering method to cluster all patterns in order to obtain those group-wise consistent functional brain network patterns across different subjects. Specifically, we set the cluster number as 400 which is equal to the dictionary size. After discarding those obvious noise or artifact network patterns conformed by visual inspection, we perform k-means clustering to obtain 400 clusters based on the remaining network patterns. All functional network spatial patterns within each of the 400 clusters are double checked to ensure that they have similar spatial patterns. Then we discard specific clusters based on a relatively strict criterion requiring each retained cluster to contain at least one network pattern from each subject, since we aim to identify the consistent functional brain networks across all subjects. The retained clusters are further inspected by two groups of experts to discard possible noise patterns/artifacts clusters. Those finalized clusters which are agreed by all experts are considered meaningful network patterns. Finally, the group-wise consistent functional network is obtained by averaging all network spatial patterns within each of the finalized clusters.

Fig.2.

The computational framework of dictionary learning and sparse coding of whole-brain rsfMRI signals for identification of functional brain networks. (a) The aggregated whole-brain rsfMRI signals matrix X of one subject. (b) The obtained dictionary matrix D. Each column is a dictionary atom representing the temporal pattern. (c) The obtained sparse coefficient matrix α. Each element of α represents the functional activity value. Each row of α can be mapped back to brain volume to represent the spatial distribution pattern of a functional network.

After identifying the connectome-scale consistent functional brain networks in the INIA19 template space, we identified the functional peak foci (voxels) with the highest functional activity value in each component of each functional network identified. As illustrated in Figs. 1a–1b, we first automatically identify the functional components in each functional network by labeling the number of components of each functional network pattern using the widely adopted connected component labeling (CCL) algorithm implemented in FSL toolbox (http://fsl.fmrib.ox.ac.uk). The basic idea is that by searching the neighborhood of all voxels involved in a specific functional network, those connected voxels involved in the functional network are assigned to the same component. In this way, each functional network may have one or more components (e.g., the network 1 in Fig. 1a has two components). Note that in order to obtain meaningful stable and consistent functional components across different subjects, we only consider those components with more than 200 connected voxels. The other components with less than 200 connected voxels are viewed as noise and discarded. Second, we identified the peak voxel with the largest functional activity value in each retained component as the representative of the component. In this way, we obtained the connectome-scale functional peak foci based on the connectome-scale consistent functional brain networks. Finally, we transformed the connectome-scale functional peak foci from the INIA19 template space to each individual MRI space using linear registration, and mapped the connectome-scale functional peak foci onto the cortical surface as mesh vertices for each individual subject. Those mapped mesh vertices derived from identified connectome-scale consistent functional brain networks via representing the functional profiles have reasonable functional correspondences across different subjects and serve as the initial locations of cortical landmarks for the next step of optimization.

2.3. Joint Constraint of Structural and Functional Profiles for Landmark Optimization

In this paper, we represent the structural profile as the ‘trace-map’ of DTI-derived fiber bundles (Zhu et al., 2012; Chen et al., 2013; Jiang et al., 2015b). To be self-contained, here we briefly demonstrate the ‘trace-map’ representation and comparison of the DTI-derived structural fiber connection pattern. As illustrated in Fig. 3, for each landmark which is represented as a sphere centered at a mesh cortical vertex with a predefined radius (empirically defined as 5.5 mm in this paper) (Zhu et al., 2012; Jiang et al., 2015b) (Fig. 3a), we extract the DTI-derived fiber bundle passing through the sphere (Fig. 3b), which represents the structural fiber connection pattern of this landmark. In order to quantify the shape of the fiber bundle and compare the fiber bundles across different landmarks, we adopt our ‘trace-map’ model (Chen et al., 2013; Zhu et al., 2012) to represent the fiber bundle with a 48-dimensional vector. Specifically, the principal orientation of each fiber in the fiber bundle is firstly projected onto the standard surface of a unit sphere (Fig. 3c). The global shape information of the fiber bundle is represented as the points distributed on the unit sphere. Then, the surface of the unit sphere is segmented into 48 quasi-equal areas with a diamond shape (Fig. 3d) (Gorski et al., 2005). The number of points in each area are counted to calculate distribution density. A 48 dimensional histogram vector tr = [d₁, d₂.. d₄₈]containing the 48 point density values, namely ‘trace-map’ (Fig. 3e), is finally obtained as the structural profile of a landmark. As a result, the comparison between complicated shapes of fiber bundles is effectively converted to the comparison of the similarity of two 48 dimensional trace-map vectors. More details are referred to (Chen et al., 2013; Zhu et al., 2012). We will integrate the DTI-derived structural fiber connection profile similarity into the procedure landmark optimization as a constraint.

Fig.3.

The pipeline of ‘trace-map’ representation of the fiber bundle of the landmark for representation of structural profile. (a) An example cortical landmark (green bubble) and the whole-brain DTI-derived axonal fibers. (b) The extracted fiber bundle of the example landmark. (c) Points distribution by projection of the principal orientation of each fiber in the fiber bundle on the unit sphere. (d) Point density (d_i) for each of 48 quasi-equal areas. (e) The 48 dimensional ‘trace-map’ histogram vector containing the 48-points density values.

Based on the initial landmarks derived from identified connectome-scale consistent functional brain networks (Section 2.2), we optimize their locations via integrating meaningful anatomical, functional, structural fiber connection pattern, and spatial consistency constraints so that the optimized landmarks possess anatomical, structural fiber connection pattern, and functional correspondences across different macaque brains. Specifically, we search all possible combinations of candidate landmark locations (cortical mesh vertices) within their local morphological neighborhoods across different subjects, and seek the optimal solution of the combination of landmark locations across subjects under the following four constraints. First, as demonstrated in Section 2.2, since each set of corresponding landmarks across different subjects are initialized by the common functional network peaks, the functional activity values of the corresponding landmarks should not decrease much compared with the peak value after optimization to retain the functional consistency. Second, the corresponding landmarks across different subjects should have maximally similar DTI-derived fiber connection pattern after optimization to retain the structural consistency (Zhu et al., 2012; Jiang et al., 2015b). Third, the corresponding landmarks across different subjects should locate on the same gyral/sulcal regions after optimization to preserve the macro-anatomical identity (Jiang et al., 2015b). Fourth, the corresponding landmarks should move within a small size of the morphological neighborhood of their initial locations after optimization to preserve the globally spatial correspondence (Zhu et al., 2012; Jiang et al., 2015b). These constraints are jointly modeled as an energy minimization problem. Note that we perform landmark optimization for each corresponding landmark separately. In this paper, we argue that 4-ring neighbor size is suitable for the macaque brain in our study due to the following two main reasons. First, in our previous DICCCOL paper in human brain (Zhu et al., 2012), we used 5-ring neighbor size which comes up about 30 candidate vertices. Such searching area size was demonstrated to be guaranteed for the optimal landmark identification (Zhu et al., 2012). In this paper, considering the relatively smaller size of macaque brain compared with human brain and to maintain the anatomical consistency, we used 4-ring neighbor size which also comes up more than 30 candidate vertices, which guarantee the identification of optimal landmarks on macaque brains. It is not always better to enlarge the search area since large search area will lead to the inconsistent anatomical information of the landmark, i.e., the landmarks will locate on different gyral/sulcal areas. The second one is the computing cost. Since we search the optimal landmark location combination across different individual brains in the group level, the computing cost will increase exponentially as the number of rings of search area increases. In conclusion, 4-rings is suitable in this paper.

Specifically, we assume $v_{0i}^p$ is the initial location of landmark p in subject i (i=1..N), $v_i^p$ is the set of candidate locations within the morphological neighborhood $C_{v_{0i}^p}$ of $v_{0i}^p(v_i^p\in C_{v_{0i}^p})$, the functional activity value of $v_{0i}^p$ is $z_{v_{0i}^p}$ (peak value), and the functional activity value of $v_i^p$ is $z_{v_i^p}$. In this paper, we consider 4-ring neighbors of $v_{0i}^p$, i.e., about 30 mesh vertices as the candidate locations for optimization of landmark p in subject i. First, the landmark can only move within the neighborhood, i.e., $v_i^p\in C_{v_{0i}^p}$, and it is used as the spatial constraint. Second, in order to retain the functional consistency, the difference between $Z_{v_0{}_i^p}$ and $Z_{v_i^p}$ should be small, i.e., $(Z_{v_0{}_i^p}-Z_{v_i^p})/Z_{v_0{}_i^p}\le \lambda$. Third, the principal curvature value of $v_i^p$ is noted by $pcur{v}_{v_i^p}\{\begin{array}{l}\ge 0,p\in gyrus\\ <0,p\in sulcus\end{array}$, and $pcur{v}_{v_i^p}\times pcur{v}_{v_j^p}\ge 0$ of a corresponding landmark between subject i and j is used as the anatomical constraint. The rationale is that since we can obtain the gyral/sulcal information of a specific initialized landmark according to its location in the INIA19 template space (Section 2.2), the corresponding landmarks mapped to all subjects should preserve the same gyral/sulcal information during landmark optimization. Finally, we define the structural fiber connection pattern similarity constraint E p_s ( ) (Zhu et al., 2012; Jiang et al., 2015b) for landmark p as

$$E_s(p)=1-\frac{{\sum }_{i,j=1\dots k\dots N,i\ne j}corr\left(\mathit{tr}\left(v_i^p\right),\mathit{tr}\left(v_j^p\right)\right)}{N*(N-1)}$$ (1)

Where corr(.) is the Pearson’s correlation value between the ‘trace-map’ vectors of vertices $v_i^p$ and $v_j^p$ in subject i and j, respectively. N is the total number of subjects. The group-wise variance of the four jointly modeled constraints is then mathematically represented as the energy E:

$$\begin{gathered} E(p)=1-\frac{{\sum }_{i,j=1\dots k\dots N,i\ne j}corr\left(\mathit{tr}\left(v_i^p\right),\mathit{tr}\left(v_j^p\right)\right)}{N*(N-1)} \\ s.t.\frac{Z_{v_0{}_k^p}-Z_{v_k^p}}{Z_{v_0{}_k^p}}\le \lambda ,v_k^p\in {v_0}_k^p,pcur{v}_{v_i^p}\times pcur{v}_{v_j^p}\ge 0 \end{gathered}$$ (2)

Our aim is to minimize the energy E (p):

$$\underset{i,j\in C,i\ne j,pcur{v}_{v_i^p}\times pcur{v}_{v_j^{p\ge 0}}}{\min }1-\frac{{\sum }_{i,j=1\dots k\dots N,i\ne j}corr\left(\mathit{tr}\left(v_i^p\right),\mathit{tr}\left(v_j^p\right)\right)}{N*(N-1)}$$ (3)

Where the constraint $C=\{k|k=1..N,s.t.(Z_{v_0{}_k^p}-Z_{v_k^p})/Z_{v_0{}_k^p}\le \lambda ,v_k^p\in {v_0}_k^p\}$

The details of solving Eq. (3) are as follows. For each iteration, we search all possible combinations of candidate landmark locations across all subjects for landmark p, and find an optimal combination of landmark locations which has a minimum E (p). If the Euclidean distance between the landmark locations with minimum E (p) of two consecutive iterations across all subjects is less than or equal to a threshold ε, the iterations are stopped. We set ε =2mm since the Euclidean distance of two adjacent cortical mesh vertices is around 2 mm. We empirically set λ=50%. The whole procedure will stop once convergence. Note that since we perform landmark optimization for each corresponding landmark across different subjects separately, we check the Euclidean distance between two neighboring optimized landmarks in each single subject. If the distance is less than or equal to ε =2mm across all single subjects, these two landmarks are considered as merged and only one landmark is retained.

Fig. 4 illustrates the effectiveness of the proposed landmark optimization framework based on one example landmark. Fig. 4a shows the fiber connection patterns of one corresponding optimized landmark in two example subjects. Fig. 4b shows the locations of the same corresponding optimized landmarks on the surfaces of the two subjects. Figs. 4c-4d illustrate the fiber connection patterns and locations of the corresponding landmark of a third subject before and after optimization. We can see that after integrating the fiber connection pattern similarity constraint, the optimized landmark has better fiber connection pattern similarity than that before optimization (Fig. 4c) compared with Fig. 4a. We can also see that after integrating anatomical constraint, the optimized landmark moved from gyral region to sulcal region (Fig. 4d) which is identical to that of other subjects in Fig. 4b. Moreover, Fig. 4e shows the locations of the same corresponding landmark in the functional volume space before and after optimization. We can see that from visual inspection the functional activity value after optimization does not decrease much compared with the peak point of initialized landmark.

Fig.4.

Illustration of the effectiveness of the proposed landmark optimization framework based on one example landmark before and after optimization. (a) The fiber connection patterns of one corresponding optimized landmark (red bubble) in two example subjects. (b) The locations of the same corresponding optimized landmarks on the surfaces of the two subjects. (c) The fiber connection pattern of the corresponding landmark of a third subject before and after optimization. The left one is before the optimization and the other one is after the optimization. (d) The locations of the corresponding landmark of a third subject before (yellow bubble) and after (red bubble) optimization. (e) The locations of the same corresponding landmark in the functional volume space before (1) and after (2) optimization.

2.4. Consistent Landmarks Inspection and Determination

In order to identify those optimized landmarks which truly jointly encode the connectome-scale structural and functional profiles, we randomly separated all six available subjects into two groups (four individuals per group; two subjects are in both groups in order to examine the stability and reproducibility of landmark optimization, so six subjects and eight individuals in total), and perform the landmark optimization scheme in Section 2.3 for the two groups separately. In this way, two independent groups of optimized corresponding landmarks are obtained. Then, we determine those common and consistent landmarks which are reproducible across the two groups of subjects via both quantitative and qualitative measurements similar as those in (Zhu et al., 2012; Jiang et al., 2015b). Specifically, for each corresponding landmark, we calculated the functional activity value difference and ‘trace-map’ correlation as discussed in Section 2.3 across all subjects in both groups to check the functional and structural consistency. If the value is statistically different (two-sample t-test, p=0.05) between two groups, this landmark will be considered instable and discarded (Zhu et al., 2012; Jiang et al., 2015b). Moreover, we adopted in-house visualization tool (Li et al., 2012) to visually examine the anatomical identity and spatial consistency of corresponding landmarks across all subjects in the two groups by two independent groups of experts. In details, there are two major criteria for the visual examination of anatomical identity and spatial consistency. The first one is to check the consistency of spatial locations (within the same reasonable anatomical region) of the identified corresponding common landmarks across different subjects. The second one is to double check whether all the corresponding landmarks are located on the same gyri/sulci or not. These steps are checked by the experts as the final inspection. Those finally retained landmarks agreed by all experts have reasonably consistent anatomical, structural, and functional profiles across different subjects. The finalized landmarks reasonably encode the joint representation of connectome-scale structural and functional profiles.

3. Experimental Results

3.1. Connectome-scale Consistent Functional Networks

Based on the methods described in Section 2.2, we successfully identified 70 group-wise consistent and common functional brain networks across different subjects. Fig. 5 shows the spatial patterns of 34 examples of the identified group-wise consistent functional networks in the INIA19 template space. The spatial patterns of all 70 consistent functional networks are publicly released online at: http://hafni.cs.uga.edu/MonkeyNewICNs/MonkeyBrain_NewTemplateComponentsMap_Shu_presentatio n.html.

Fig.5.

The spatial patterns of forty examples of the identified group-wise consistent functional networks across macaque brains in the INIA19 template space. Each sub-figure (separated by dashed lines) shows one network averaged across subjects with three representative axial slices.

Fig. 6 shows the spatial patterns of 12 examples of the corresponding identified functional brain networks in the template space and in individuals. Another five examples are shown in Fig. 7a. All the consistent functional networks are released online at: http://hafni.cs.uga.edu/MonkeyNewICNs/MonkeyBrain_NewTemplateComponentsMap_Shu_presentation.html. The results are organized with 7 columns, in which templates are shown on the first column and the other six columns are representing the identified individual networks. By visual inspection, each functional network shows reasonably consistent spatial pattern across different subjects. Quantitatively, we measure the spatial pattern similarity as the spatial overlap rate R(S,G) between the functional network spatial pattern of a specific subject (S) and the corresponding group-averaged spatial pattern template (G):

$$R(S,G)=\frac{|S\cap G|}{\left|G\right|}$$ (4)

Fig.6.

12 examples of identified group-wise consistent and common functional brain network templates (T) and the corresponding networks in three individual subjects. For each network, its spatial pattern is shown in a representative axial slice.

Fig.7.

Identified consistent and common functional brain networks and the functional peak points used as initialized landmarks. (a) Five examples of functional brain networks. Each row shows the spatial pattern of one functional network template and the corresponding spatial patterns in three example individual brains. The identified functional peak points mapped onto individual surfaces are represented as red bubbles and highlighted by yellow circles. (b) All 107 initialized landmarks based on the 107 connectome-scale and consistent functional peak points in the six macaque brains.

Note that S and G are converted from continuous values to discrete labels (all values smaller than or equal to 0 are labeled as 0, and others are labeled as 1). The mean spatial overlap rate of all 70 functional networks across all subjects is as high as 0.336.

Based on the 70 connectome-scale functional network patterns, 107 connectome-scale and consistent functional peak points were identified and mapped to individual cortical surfaces as the initialized landmarks. Fig. 7a shows the identified functional peak points and the mapped landmarks derived from five example functional networks, respectively. We can see that the corresponding landmarks have rough correspondences on the cortical surfaces across different subjects. Fig. 7b shows all 107 initialized landmarks in the six macaque brains based on the 107 connectome-scale and consistent functional peak points. As a preliminary step, we interpret the 107 connectome-scale and consistent functional peak points using two publicly available parcellation maps: INIA19 NeuroMaps (Rohlfing et al., 2012) and CBCatel15 (Calabrese et al., 2015) since they provide relatively finer-scale brain parcellations (Rohlfing et al., 2012; Calabrese et al., 2015). Table 1 shows the interpretations of the functional peak points based on the five example functional networks in Fig. 7a.

Table 1.

Interpretation of the functional peak points in the five example functional networks in Fig. 7a using two parcellation maps (INIA19 NeuroMaps and CBCatel15).

Network	No. of landmarks	INIA19 NeuroMaps	CBCatel15
#1	2	l_occipital_white_matter; r_occipital_white_matter	parietal area PE; parietal area PEa
# 2	2	l_superior_temporal_gyrus; r_superior_temporal_gyrus	parietal area PF, opercular part; parietal area PF (cortex)
# 3	2	l_cerebral_white_matter; r_cerebral_white_matter	depth intraparietal area; dorsal parietal area
# 4	2	l_frontal_white_matter; r_frontal_white_matter	occipitoparietal area; dorsal parietal area
# 5	2	l_inferior_temporal_gyrus; r_inferior_temporal_gyrus	parietal area PG; parietal area PFG

3.2. Consistent Cortical Landmarks via Joint Representation of Connectome-scale Structural and Functional Profiles

We jointly represented the connectome-scale structural and functional profiles for identification of the consistent cortical landmarks as demonstrated in Sections 2.3 and 2.4. In total, we identified 100 consistent and common landmarks across subjects. The indices and locations of these 100 landmarks are shown in Fig. 8. Fig. 9a shows all 100 landmarks across all subjects in the two groups. We randomly selected eight example landmarks (Fig. 9b) and visualized their fiber connection patterns in Figs. 9c–9j, respectively. From visual inspection, we can see that the fiber connection pattern of the corresponding landmark is similar across different subjects. Quantitatively, the mean correlation value of ‘trace-map’ (Eq. (1)) across any pair of the eight individuals in two groups is 0.667, 0.677, 0.785, 0.768, 0.8041, 0.8247, 0.8151, and 0.6147 for the eight example landmarks, respectively. Fig. 10 shows another three example landmarks and their fiber connection patterns. The visualization of all 100 landmarks is released at: http://caid.cs.uga.edu/~szhang/dicccol.html. All fiber connection patterns of 100 landmarks across different subjects are at: http://caid.cs.uga.edu/~szhang/fiber.html. The mean correlation value of ‘tracemap’ across any pair of the eight individuals in two groups for all 100 landmarks in shown in Fig. 11(a). The overall mean correlation value of all 100 landmarks is 0.71. In conclusion, the 100 landmarks possess reasonably consistent structural profiles (DTI-derived fiber connection pattern) across different subjects. We will discuss the functional, anatomical, and the spatial consistency of the 100 landmarks in detail in the next section.

Fig.8.

Indices and locations of the identified 100 landmarks on one example brain. Some landmarks are hidden by the cortical surface and are shown at the locations by black/white arrows.

Fig.9.

Identified 100 consistent landmarks. (a) All 100 landmarks (blue bubbles) across different subjects. (b) Eight example landmarks (represented by eight different color bubbles) shown on one example surface. (c)-(j): The fiber connection patterns of each example landmark across all subjects, respectively.

Fig.10.

Another three examples of landmarks and their fiber connection patterns across different subjects in (a)-(c), respectively.

Fig.11.

The mean correlation value of structural fiber connection ‘trace-map’. (a) The mean correlation value of ‘trace-map’ across any pair of the individuals within two groups for all 100 landmarks after optimization. The horizontal axis represents the 100 landmarks and vertical axis represents the mean correlation value of ‘trace-map’ across any pair of the individuals for each landmark. (b) The mean correlation value of structural fiber connection ‘trace-map’ across any pair of the individuals for all 100 landmarks before optimization. The horizontal axis represents the 100 landmarks and vertical axis represents the mean correlation value of ‘trace-map’ across any pair of the individuals for each landmark.

3.3. Effectiveness of Joint Representation of Connectome-scale Structural and Functional Profiles

In this section, we quantitatively examine the effectiveness of the proposed joint representation of connectome-scale structural and functional profiles. To briefly illustrate the difference of identified areas, Fig. 12 co-visualizes all 100 finalized landmarks and 107 initialized landmarks on one example brain.

Fig.12.

The 100 finalized landmarks (green bubbles) and 107 initialized landmarks (red bubbles) co-visualized on one example subject brain in three views in (a)-(c), respectively.

Quantitatively, we first measure and compare the mean correlation value of structural fiber connection ‘trace-map’ of the 100 landmarks before and after optimization. As shown in Fig. 11(b), there are 63 out of 100 landmarks whose mean correlation value is significantly increased (p=0.05) after optimization. The rest of landmarks have comparable trace-map values before and after optimization. The averaged correlation of the landmarks is 0.71±0.1275 after optimization and 0.60±0.1625 before optimization.

Second, we calculate the percentage of functional activity value difference of each landmark before and after optimization compared with the functional peak value ($(z_{v_{0i}^p}-z_{v_i^p})/z_{v_{0i}^p}$ as demonstrated in Section 2.2). As reported in Table 2, the percentage is as high as ~80% across the two groups, indicating that the functional activity values are not changed much after landmark optimization to preserve the functional consistency.

Table 2.

The average percentage of functional activity value difference of each landmark before and after optimization compared with the peak value across the six subjects in the two groups. The value is represented as mean±SD.

Group 1 (subject ID)	1	2	3	4
value	0.9±0.168	0.88±0.188	0.85±0.213	0.85±0.216
Group 2 (subject ID)	2	4	5	6
value	0.87±0.2140	0.86±0.2103	0.87±0.19	0.87±0.19

Third, we calculate the Euclidean distance of spatial location movement of the 100 landmarks before and after optimization. The mean distance of all 100 landmarks is 1.466 mm, indicating that the 100 landmarks can achieve reasonable functional, structural, anatomical, and spatial consistency within a small range from the initialized locations. This finding also indicates that connectome-scale structural profile is reasonably consistent once the functional profile is consistent. This finding also proves the ‘fingerprint’ concept (Passingham et al., 2002) in connectome-scale, which premises that each brain’s cytoarchitectonic area has a unique set of extrinsic inputs and outputs that largely determines the functions that each brain area performs.

In conclusion, both quantitative and quantitative measurements demonstrate the effectiveness of the proposed joint representation of connectome-scale structural and functional profiles. The identified 100 landmarks effectively represent the joint anatomical, structural, and functional profiles across different macaque brains.

4. Discussion and Conclusions

In this study, we jointly represented the connectome-scale structural and functional profiles via a computational framework for identification of consistent cortical landmarks in macaques. We initialized the landmark locations with connectome-scale functional network peaks derived from representation of functional profiles via HAFNI, instead of random initialization or manual labelling in previous studies (e.g., DICCCOL identification in human brains). In this way, the initialized landmarks have functional correspondences and thus provide a foundation for the joint representation of connectome-scale structural and functional connectivity afterwards. During the landmark optimization procedure, we integrated four meaningful constraints: anatomical, structural fiber connection pattern, functional connectivity, and spatial information so that the identified landmarks comprehensively encode anatomical, structural and functional consistency. By applying the proposed computational framework, we have identified 100 consistent and common cortical landmarks in different macaque brains. This set of 100 landmarks has potential to represent common structural/functional macaque cortical architecture for advancements in the neuroscience and brain mapping fields.

The experimental results have demonstrated that the connectome-scale consistent functional brain network patterns across different subjects are successfully identified via our proposed framework, indicating that there exists functional regularity and consistency across different macaque brains. Based on these connectome-scale consistent functional network peaks as initialized landmark locations, the results have shown that a set of landmarks can achieve convergence of functional, structural fiber connection pattern, anatomical, and spatial consistency after the proposed landmark optimization. Our experimental results further demonstrate that there is reasonable regularity among brain function, structural fiber connection pattern, and anatomy within and between the individual macaques, which further proves the ‘fingerprint’ concept (Passingham et al., 2002) in connectome-scale. This is also the premise that we performed joint representation of connectome-scale structural and functional profiles in macaque brains.

In this study, we focused on the methodology development of joint representation of connectome-scale structural and resting state functional profiles. The potential applications in the neuroscience and brain mapping fields are open to future studies. For example, in this study, we applied the joint representation framework on six-month macaques and the results were promising. In the future, we can apply the proposed joint representation of connectome-scale structural and functional profiles to examine developmental changes in structural and/or functional connectivity after identifying the landmarks in the same individuals at different ages based on the availability of longitudinal, within-subject, multi-modal DTI and fMRI data. As a result, those consistent landmarks across different ages could potentially map the macaque brain regions with developmental-preserved joint representation of anatomical/structural/functional consistency. Conversely, those landmarks merely existing at specific ages could potentially map macaque brain regions with developmentally-related changes in joint representation of anatomical/structural/functional consistency. A more universal landmark map which can systematically encode both the developmentally preserved and changed joint representation of anatomical/structural/functional consistency of macaque brains can thus be obtained, and would be extremely important for neurodevelopmental studies in this animal model. Another application example includes to explore the regularity and variability of structural/functional connectivity/interaction among different cortical regions (e.g., the gyral/sulcal regions) based on these identified landmarks. Moreover, based on the landmarks which encode the joint connectome-scale structural and functional profiles, we can identify possible connectome-scale structural/functional connectivity alterations in specific cortical regions between groups of macaques with typical/normative developmental experiences (“control groups”) and groups with specific experimental treatments or pathologies (e.g., early life stress (Howell et al., 2013; Howell et al, 2016), and use those connectivity alterations as potential biomarkers for identification of different macaque groups.