Longitudinal Infant Functional Connectivity Prediction via Conditional Intensive Triplet Network

Abstract

Longitudinal infant brain functional connectivity (FC) constructed from resting-state functional MRI (rs-fMRI) has increasingly become a pivotal tool in studying the dynamics of early brain development. However, due to various reasons including high acquisition cost, strong motion artifact, and subject dropout, there has been an extreme shortage of usable longitudinal infant rs-fMRI scans to construct longitudinal FCs, which hinders comprehensive understanding and modeling of brain functional development at early ages. To address this issue, in this paper, we propose a novel conditional intensive triplet network (CITN) for longitudinal prediction of the dynamic development of infant FC, which can traverse FCs within a long duration and predict the target FC at any specific age during infancy. Targeting at accurately modeling of the progression pattern of FC, while maintaining the individual functional uniqueness, our model effectively disentangles the intrinsically mixed age-related and identity-related information from the source FC and predicts the target FC by fusing well-disentangled identity-related information with the specific age-related information. Specifically, we introduce an intensive triplet auto-encoder for effective disentanglement of age-related and identity-related information and an identity conditional module to mix identity-related information with designated age-related information. We train the proposed model in a self-supervised way and design downstream tasks to help robustly disentangle age-related and identity-related features. Experiments on 464 longitudinal infant fMRI scans show the superior performance of the proposed method in longitudinal FC prediction in comparison with state-of-the-art approaches.

1. Introduction

Functional connectivity (FC) constructed from resting-state fMRI is one of the most prevalent data for revealing brain functional organization [1, 2]. Typically, brain functional connectivity is represented as an undirected graph encoded in a symmetric connectivity matrix, where each element is the Pearson’s correlation coefficient (PCC) of the average blood-oxygen-level-dependent (BOLD) signal between a pair of regions of interest (ROIs) [3]. Being demonstrated as an effective way to represent a comprehensive mapping of neural activities, FC has been widely used in many applications, such as mutual prediction between FC and structural connectivity (SC) [4, 5], brain development and aging studies [6–8], and early detection of brain diseases [9–12]. To answer questions raised in brain research for early ages with extremely dynamic neurodevelopment, it is critical to obtain sufficient longitudinal FC data. However, the usable longitudinal infant FC data are still scarce due to many reasons, e.g., strong motion effect, subject dropout, high acquisition costs, few volunteers, and long-time acquisition time [13]. As a result, predicting infant FC development at specific ages from limited samples becomes extremely challenging in neuroscience study, though it is of great importance in understanding normal brain development and early diagnosing neurodevelopmental abnormalities [14–16]. Some learning-based methods have been proposed for the longitudinal prediction of missing structural images and features [17–22], but these methods generally fail in predicting functional connectivity due to the large heterogeneity of FCs between individuals and developmental stages. Besides, the multi-view brain graph synthesis methods designed for cortical morphological connectomes usually cannot perform well in FC prediction [23, 24]. To the best of our knowledge, computational techniques for predicting the longitudinal development of FCs in infants remain unexplored.

To address this issue, in this paper, we propose a novel conditional intensive triplet network (CITN) for longitudinal prediction of the dynamic development of infant FC, which can traverse FCs within a long duration and predict the target FC at any specific age during infancy. Specifically, 1) we introduce an intensive triplet auto-encoder for effective disentanglement of age-related and identity-related information, which accurately disentangles brain development-related and identity-related features; 2) we design an identity conditional module to fuse identity-related information with designated age-related information, which effectively models the progression process of FC, while maintaining the individualized intrinsic brain functional patterns; 3) our method enables FC prediction at any time period during infancy by concatenating the trained identity extractor and identity conditional module. We validate our proposed CITN on the Baby Connectome Project (BCP) [25] dataset, including 464 longitudinal scans from 119 subjects, and achieve better FC prediction than state-of-the-art methods.

2. Methods

2.1. Model Overview

The framework of our proposed conditional intensive triplet network is shown in Fig. 1 and detailed below. Specifically, the whole framework is divided into two parts: the training stage and the testing stage.

Fig. 1.

The framework of the proposed conditional intensive triplet network (CITN). In the training stage, the model has a structure in the form of a triplet network. Each branch consists of six components: 1) Encoder, mapping the source FC to a mixed latent representation; 2) ID Extractor, extracting ID-related features from mixed latent features; 3) Age Info Extractor, extracting age-related features from mixed latent features; 4) Age prediction, utilizing age-related features for age prediction to enforce the age-specific information in age-related features; 5) ID Conditional Module, learning ID conditional age progression patterns; 6) Decoder, reconstructing the source functional connectivity.

The goal is to predict infantile functional connectivity at any age by capturing the FC progression pattern during early brain development. In the training stage, a triplet network is the backbone with three parallel branches and identical modules, such as encoder, ID (identify) extractor, decoder, etc. Specifically, the proposed model is composed of six parts in each branch: 1) learning the latent representations of individual FC by an encoder; 2) disentangling ID-related and age-related features by corresponding ID extractor and age Info (information) extractors; 3) strengthening the similarity/difference between the FCs acquired from the same/different individuals by delicately designed triplet loss of ID-related features; 4) using age-related information for a downstream task, i.e., age prediction; 5) learning ID conditional age features by ID conditional module; 6) reconstructing the input functional connectivity by the decoder. In the prediction stage, the trained model takes an available FC as input to extract ID-related features for predicting the target FC at any age during infancy.

2.2. Functional Connectivity Prediction Loss Design

The age-related features change over time because of brain development, but the ID-related features should be approximately time-invariant. Therefore, one critical step in the proposed model is to effectively disentangle the mixed age-related and ID-related information from the source FC. For this purpose, we first formulate the training samples in triplet units $\left(x_i^{t_1},x_i^{t_2},x_j^{t_n}\right)$, where the first two are FCs from the same individual i but at different ages t₁ and t₂, and the last one is an FC from another individual j at any age t_n. The triplet units employ a multilayer perceptron neural network, denoted as E, as their shared encoder. The outputs of the encoder are the latent representations of the input functional connectivity, which are denoted as $z_i^{t_1}$, $z_i^{t_2}$, and $z_j^{t_n}$. Time indices t₁, t₂, and t_n will be omitted for convenience unless otherwise specified. The latent representation z can be disentangled into two parts via a shared ID extractor and a shared age Info extractor module: ID(z) and Age(z), representing the ID-related and age-related information, respectively. Besides, to predict FCs at different ages, the model needs to learn identity conditional progression patterns, for which we design the identity conditional module (ICM) that will be detailed in the next section. Finally, a shared decoder G recovers the source input. The similarity between the source FC x and recovered $\text{FC\hspace{0.17em}}\widehat{x}=G(ICM(ID(z),t))$ is maximized to make sure a high-quality reconstruction of the input functional connectivity. Here we adopt the mean absolute error (MAE) and Pearson’s correlation coefficient (PCC) to evaluate the reconstruction [21, 33].

The PCC loss of the reconstruction is defined as:

$${ℒ}_{\mathit{\text{recon }}}^{PCC}={\sum }_{x=x_i^{t_1},x_i^{t_2},x_j^{t_n}}{\mathbb{E}}_x\mathit{\text{corr}}(x,G(ICM(ID(E(x)),t)))$$ (1)

where t ∈ {t₁, t₂, t_n}, corr represents the Pearson’s correlation and $\mathbb{E}$ represents the expectation. The MAE loss of the reconstruction is defined as:

$${ℒ}_{\mathit{\text{recon }}}^{MAE}={\sum }_{x=x_i^{t_1},x_i^{t_2},x_j^{t_n}}{\mathbb{E}}_x\left|x-G(ICM(ID(E(x)),t))\right|$$ (2)

The whole reconstruction loss is defined as a weighted sum of the reconstruction PCC and MAE loss:

$${ℒ}_{\mathit{\text{recon }}}=\lambda {ℒ}_{\mathit{\text{recon }}}^{MAE}-\beta {ℒ}_{\mathit{\text{recon }}}^{PCC}$$ (3)

where λ and β control the balance of different loss terms.

The disentanglement of the ID-related features is an essential step to the success of our proposed model. The disentangled ID-related features from the same subject should be as close as possible while being distinct from other subjects. Therefore, we introduced the intensive triplet loss to distinguish ID-related features from different subjects. We borrow the concept from the triplet loss models, where we consider the anchor and positive as the ID-related features of $ID\left(E\left(x_i^{t_1}\right)\right)$ and $ID\left(E\left(x_i^{t_2}\right)\right)$, while the negative is the ID-related features of $ID\left(E\left(x_j^{t_n}\right)\right)$. The distance between anchor and positive is minimized, while the distances between the anchor/positive and negative are maximized in the latent space. However, the original triplet loss only considers the relative distance between (Anchor, Positive) and (Anchor, Negative) pairs. Since (Positive, Negative) is also a pair of distinct labels, the relative distance between (Anchor, Positive) and (Positive, Negative) is measured as an additional constraint to increase the inter-subject dissimilarity [33]. Therefore, we define a new intensive triplet loss as follows:

$${ℒ}_{I-\mathit{\text{tri}}}={ℒ}_{\mathit{\text{tri}}}+{ℒ}_I$$ (4)

$${ℒ}_{\mathit{\text{tri}}}=\mathit{\text{corr}}\left(ID\left(E\left(x_i^{t_1}\right)\right),ID\left(E\left(x_j^{t_n}\right)\right)\right)-\mathit{\text{corr}}\left(ID\left(E\left(x_i^{t_1}\right)\right),ID\left(E\left(x_i^{t_2}\right)\right)\right)$$ (5)

$${ℒ}_I=\mathit{\text{corr}}\left(ID\left(E\left(x_i^{t_2}\right)\right),ID\left(E\left(x_j^{t_n}\right)\right)\right)-\mathit{\text{corr}}\left(ID\left(E\left(x_i^{t_1}\right)\right),ID\left(E\left(x_i^{t_2}\right)\right)\right)$$ (6)

To encourage the proposed model to disentangle the latent representation robustly, we use a downstream task, i.e., age estimation, to supervise the feature disentanglement. The output of age Info extractor module Age(z) is further fed into the age predictor module P for age estimation. Its loss function is defined as:

$${ℒ}_{\mathit{\text{age }}}={\sum }_{x=x_i^{t_1},x_i^{t_2},x_j^{t_n}}{\mathbb{E}}_x\left|{\mathit{\text{y}}}_x-P(\mathit{\text{Age}}(E(x)))\right|$$ (7)

where y_x is the real age corresponding to FC x.

On the whole, the FC prediction loss is formulated as

$${ℒ}_{\mathit{\text{overall }}}={ℒ}_{\mathit{\text{recon }}}+\alpha {ℒ}_{\mathit{\text{age }}}+\delta {ℒ}_{I-\mathit{\text{ tri }}}$$ (8)

where α and δ controls the influence of age estimation loss and intensive triplet loss, respectively.

2.3. Identity Conditional Module

The identity conditional module (ICM) is built by cascading several identity conditional blocks (ICBs) to improve the age smoothness of synthesized FCs. The detailed structure of ICBs is shown in Fig. 2.

Fig. 2.

The architecture of the identity conditional blocks (ICBs). The proposed ICBs aim to learn the general FC progression pattern with the conditionality of identity. It takes the ID-related feature from the ID extractor as input to learn an identity-level progression pattern. Here the target age is set to be the period of 101–200 days for illustration.

In our work, the number of ICBs is set as six after multiple empirical tries. Specifically, the designed ICB takes the identity-related feature from the ID extractor as input to learn the general FC progression pattern with the condition of identity. The ICBs shown in Fig. 2 contains a fully connected feed-forward network (FFN), which consists of two linear projections with a ReLU activation in between.

$$FFN(v)=\text{mask}\left(\text{mask}\left(\text{max}\left(0,v{W}_1+b_1\right),t_n\right)W_2+b_2,t_n\right)$$ (9)

where v represents the input vector. While the linear projections are the same across the ICBs, they use different parameters from layer to layer. The mask() function is to filter out the features of the target age group, which is the ID-Age features as shown in Fig. 2. Then, with a dedicated designed weights-sharing strategy, the age smoothness of synthesized FCs is enforced by sharing part of ID-Age features across adjacent age groups. A hyper-parameter p is further introduced to adjust the percentage of the ID-Age features shared between two adjacent age groups, which is empirically set to 0.1. In the training process, the ICM synthesizes the corresponding ID-Age features under the guidance of the age label of source functional connectivity. In the prediction process, the ICM fuses the target age label and ID-related features together to synthesize the corresponding target ID-Age features.

3. Experiments

3.1. Dataset

In this work, we use 464 longitudinal resting-state fMRI scans of 119 typically developing infants before 600 days of age from the Baby Connectome Project (BCP) dataset [25]. Since the individuals have two different types of scan orders, anterior to posterior (AP) and posterior to anterior (PA), we divide the dataset into two separate groups, AP and PA, for independent validation, where each group has 232 scans. After a detailed investigation of the dataset, we uniformly divide the dataset into six partitions forming a relatively balanced data distribution. The six temporal partitions are [1, 100], [101, 200], …, [501, 600], corresponding to six ages. Note that none of the individuals has complete scans of six ages. For structural and functional MRI processing, we follow the strategies in [28, 29], resulting in average fMRI time series in each of 35 cortical regions for each hemisphere. We construct the functional connectivity by calculating the Pearson’s correlation coefficient between time series of each pair of regions and performing Fishers r-to-z transformation. The flattened vector of the upper triangle of FC is taken as the input of our model. Of note, we use the absolute value of FCs, which means we narrow the range of FCs from [−1, 1] to [0, 1], since we focus more on predicting the strength of the connection [30] in this study.

3.2. Results and Visualization

The encoder E and decoder G were designed as a two-layer perceptron neural network. The ID extractor and age Info extractor both consist of densely connected layers of dimension (1500, 1000) and (1500, 500), respectively, with ReLU as the activation function. The age predictor comprises 3 densely connected layers of dimension (256, 32, 1) with a nonlinear activation function of ReLU. There are six identical ICBs in ICM, where each ICB project ID-related features to a high dimension (1000, ((φ − (φ − 1) × p) × 1500)), in which φ is the number of age groups and corresponding to the number of partitions 6 in our experiments. The ID-Age features that match the target age were then correspondingly selected out as shown in Fig. 2.

We show the predicted results of two representative individuals that have three FCs at differentages. Each individual takes one of the three FCs as the source input and the other two as target FC to predict. To evaluate the stability and effectiveness of the proposed method, we display the FCs of ground truth and predictions in Fig. 3. For each subject, there are two target predictions at two ages. The prediction results demonstrate that our method can maintain the stability of the FC prediction at any specific age regardless of different source FCs, where the average correlation between the predicted FCs obtained from different source FC achieves above 0.90.

Fig. 3.

Visual comparison between the ground truth and the predicted FCs of two representative individuals. Each column represents a specific age, and each row includes the FCs in three different ages. The first two rows are the predicted FCs, and the last row is the ground truth FCs. For each predicted FC, the text below it describes the MAE and PCC between the predicted and ground truth FCs.

3.3. Evaluation and Comparison

To quantitatively evaluate the performance, we compare our conditional intensive triplet network with two well-recognized approaches. The first one is the MLP-type network [31], which incorporates source FC with the target age information by one-hot encoding. The other one is multi-marginal Wasserstein GAN (MWGAN) [32], which defines a generator for target FC prediction at each age. We display the comparison results of the prediction of two representative individuals at two time periods in Fig. 4, where the first three columns represent the predicted results and the last column is the ground truth, while the rows represent the ages. MLP deteriorates the performance rapidly when the FC pattern changes a lot along the time, while the target results of WMGAN are not stable when input source FCs from different ages. We also tried different filter-sharing strategies by setting the hyper-parameter p with a range of values. Table 1 shows the group-level performance comparison between the proposed method and two baselines, where p0, p1, and p2 mean the hyper-parameter p set to be 0.0, 0.1, 0.2. From the comparisons in Fig. 4 and Table 1, we can see our proposed model achieves better performance in both AP and PA groups and under different settings.

Fig. 4.

FC prediction obtained by proposed CITN and two competing methods. The first three columns are the mutually predicted results between two ages by MWGAN, MLP and CITN. The last column is the ground truth. The two individuals shown in this figure have two FCs at two different ages.

Table 1.

Group-level prediction performance comparison.

	Method	MAE p0	PCC p0	MAE p1	PCC p1	MAE p2	PCC p2
AP	CITN	.12 ± .03	.77 ± .07	.12 ± .03	.77 ± .07	.12 ± .03	.76 ± .07
	MLP	.18 ± .04	.62 ± .06	N/A	N/A	N/A	N/A
	MWGAN	.15 ± .06	.75 ± .11	N/A	N/A	N/A	N/A
PA	CITN	.12 ± .04	.76 ± .07	.13 ± .04	.75 ± .08	.12 ± .04	.76 ± .08
	MLP	.18 ± .09	.60 ± .04	N/A	N/A	N/A	N/A
	MWGAN	.17 ± .06	.73 ± .08	N/A	N/A	N/A	N/A

4. Conclusion

In this work, we proposed a conditional intensive triplet deep learning network, which is a novel approach for predicting longitudinal infant functional connectivity from severely irregularly-distributed data. To this end, we formulate triplet units to augment training samples, design identity and age information extractors to disentangle the identity-related and age-related features from the latent representations, and devise identity conditional module to fuse the identity-related features with the target age information, thus enabling generate functional connectivity at any specific age. The promising results on the BCP dataset demonstrate the practicality and feasibility of our model in the application for infant functional connectivity prediction.