Employing Connectome-Based Models to Predict Working Memory in Multiple Sclerosis

Abstract

Introduction:

Individuals with multiple sclerosis (MS) are vulnerable to deficits in working memory (WM), but the search for neural correlates of WM within circumscribed areas has been inconclusive. Given the widespread neural alterations observed in MS, predictive modeling approaches that capitalize on whole-brain connectivity may better capture individual differences in WM.

Materials and Methods:

We applied connectome-based predictive modeling to functional magnetic resonance imaging data from WM tasks in two independent samples with relapsing-remitting MS. In the internal sample (n_internal = 36), cross-validation was used to train a model to predict accuracy on the Paced Visual Serial Addition Test from functional connectivity. We hypothesized that this MS-specific model would successfully predict performance on the N-back task in the validation cohort (n_validation = 36). In addition, we assessed the generalizability of existing WM networks derived in healthy young adults to these samples, and we explored anatomical differences between the healthy and MS networks.

Results:

We successfully derived an MS-specific predictive model of WM in the internal sample (full: r_s = 0.47, permuted p = 0.011), but the predictions were not significant in the validation cohort (r_s = −0.047; p = 0.78, mean squared error [MSE] = 0.006, R² = −2.21%). In contrast, the healthy networks successfully predicted WM in both MS samples (internal: r_s = 0.33 p = 0.049, MSE = 0.009, R² = 13.4%; validation cohort: r_s = 0.46, p = 0.005, MSE = 0.005, R² = 16.9%), demonstrating their translational potential.

Discussion:

Functional networks identified in a large sample of healthy individuals predicted significant variance in WM in MS. Networks derived in small samples of people with MS may have limited generalizability, potentially due to disease-related heterogeneity. The robustness of models derived in large clinical samples warrants further investigation. ClinicalTrials.gov ID: NCT03244696.

Impact statement

Working memory deficits in people with multiple sclerosis have important consequences for employment, leisure, and activities of daily living. Identifying a functional connectivity-based marker that accurately captures individual differences in working memory may offer a useful target for cognitive rehabilitation. We demonstrate that machine learning can be applied to whole-brain functional connectivity data to identify networks that predict individual-level working memory in people with multiple sclerosis. However, existing network-based models of working memory derived in healthy adults outperform those identified in multiple sclerosis, suggesting translational potential of brain networks derived in large, healthy samples for predicting cognition in multiple sclerosis.

Introduction

Multiple sclerosis (MS) is the most common demyelinating disease of the central nervous system, driven by chronic inflammatory, neurodegenerative processes. The pathological hallmark of MS is damage to white matter in the form of demyelinating focal lesions that present in heterogeneous locations and combinations in the brain and spinal cord (Filippi et al., 2018). The confluence of diffuse demyelination, axonal injury/loss, and neurodegeneration reflected in gray matter atrophy is inherently heterogenous, thus manifesting in variable clinical presentations. Nearly one million individuals in the United States and 2.5 million individuals worldwide are living with the disease (Wallin et al., 2019). At least half of these people will exhibit cognitive deficits during their clinical course (Chiaravalloti and DeLuca, 2008).

Among the cognitive sequelae that intensify the personal and economic burden of the disease, working memory (WM) dysfunction is notable (Macías Islas & Ciampi, 2019). WM is the brain's ability to temporarily store, prioritize, and actively manipulate transitory information. In the constant competition between relevant and irrelevant information, deficits in maintaining target information can result in slower or inaccurate processing, thereby causing demonstrable difficulties in daily tasks that rely on WM (Cabeza et al., 2016). WM deficits in people with MS (PwMS) are associated with functional ramifications, including lower work engagement and unemployment (Macías Islas and Ciampi, 2019; Nicholas et al., 2019). The diffuse impact of MS pathology (i.e., lesions and neurodegeneration) has prompted neuroimaging investigations on ways the disease impacts widespread neural processes supporting WM.

Existing studies of WM in MS have predominantly focused on a priori regions and/or networks of interest, yielding evidence limited to select brain areas. Some functional magnetic resonance imaging (fMRI) studies show that during WM tasks, PwMS demonstrate greater activity of prefrontal regions than healthy controls (Forn et al., 2006, 2007; Hillary et al., 2003). However, other research indicates that activity patterns depend on individual WM capacity; relative to healthy controls, unimpaired PwMS show similar activity patterns, whereas WM-impaired individuals exhibit greater recruitment of frontal and parietal regions (Chiaravalloti et al., 2005).

Neural recruitment also appears to depend on task demand. Under low WM demand, PwMS activate prefrontal and parietal areas to a greater degree than healthy controls (Colorado et al., 2012; Mainero et al., 2004; Sweet et al., 2006), suggesting a compensatory response. In contrast, when faced with high WM demand, PwMS exhibit a pattern that is thought to be maladaptive (Giorgio et al., 2015; Vacchi et al., 2017). During high WM demand conditions, PwMS exhibit reduced recruitment of core prefrontal and parietal regions and greater activation of areas outside of typical WM circuitry (Sweet et al., 2006; Vacchi et al., 2017; Wishart et al., 2004).

Recent fMRI studies on the relationship between WM and communication among intrinsic functional networks reveal diffuse connectivity changes in MS. Relative to healthy controls, PwMS show increased functional connectivity in sensorimotor and cognitive control networks at rest (Giorgio et al., 2015). In addition, there is evidence that disease progression may be associated with more widespread alterations in communication between disparate brain areas. Compared with healthy controls and unimpaired PwMS, cognitively impaired PwMS demonstrate increased connectivity between both the default mode and the frontoparietal networks with other brain networks (Meijer et al., 2017). However, across studies, differences in regional activity or connectivity between select networks do not adequately distinguish individuals with worse clinical course (Vacchi et al., 2017). This may be due to the heterogeneous disease presentation, whereby pathology and neural dysfunction impacts a wide swath of cortical and subcortical regions (Rovaris et al., 2000). Evidence for the widespread effects of MS pathology is evident in diffusion-based studies, demonstrating white matter degradation patterns that relate to different symptom presentations, including WM (Walsh et al., 2011; Welton et al., 2015).

The predominant use of correlational methods for discovering associations between brain activity and WM have proven insufficient for converging upon a single reliable signature of WM in MS. Previous studies are limited in that they: (1) are restricted to specific areas/networks of the brain; (2) rely on group-level analysis, yielding sample-specific findings; (3) ignore rich individual-level variability; and (4) do not test the generalizability of findings in independent samples. Predictive modeling methods that capitalize on whole-brain functional connectivity offer promising tools for identifying a WM neural signature in MS. In fact, an increasing number of studies indicate that individual differences in distributed patterns of functional connectivity can be harnessed to identify neuromarkers that predict cognition (Barron et al., 2019; Greene et al., 2018; Rosenberg et al., 2018), including graph-theoretic approaches that show reliable associations with WM performance in MS (Rocca et al., 2016; Welton et al., 2020). Given the heterogeneous and widespread neural alterations observed in MS, broadening the search field for brain–behavior relationships may help identify a comprehensive network-based model of WM in MS.

Connectome-based predictive modeling (CPM) is a data-driven technique for identifying whole-brain functional networks that can be used to make predictions about cognitive performance in previously unseen individuals (Shen et al., 2017). CPM has been used to successfully predict a wide range of cognitive measures, including sustained attention (Rosenberg et al., 2016a), reaction time and executive control (Rosenberg et al., 2018), fluid intelligence (Finn et al., 2015; Greene et al., 2018), and WM (Avery et al., 2020) in healthy young adults. Individual differences in age-related and disease-related decline in cognitive functioning have also been successfully predicted through CPM techniques (Avery et al., 2020; Fountain-Zaragoza et al., 2019; Lin et al., 2018), suggesting successful clinical translation.

Relevant to the current study, Avery et al. (2020) used CPM to identify functional networks during a WM task that predicted performance in healthy adults. This working memory CPM (wmCPM) successfully predicted memory performance in an independent, heterogeneous sample of older adults with preserved cognition, amnestic mild cognitive impairment, and Alzheimer's disease. The demonstrated ability of CPM to yield network-based models that predict individual differences in WM across different tasks and to generalize those differences to clinical populations suggests the utility of this approach for predicting WM in PwMS.

In addition, a growing body of CPM research demonstrates the applicability of this methodology in yielding brain-based signatures that generalize across tasks. For example, sustained attention CPM networks not only predicted attentional performance in the gradual-onset Continuous Performance Task used for derivation (Rosenberg et al., 2016a) but also generalized to predict several other domains of attention, measured by a stop-signal task (Rosenberg et al., 2016b) and Attention Network Task (Rosenberg et al., 2018). Further, these networks also generalized to predict attention deficit hyperactivity disorder symptom severity in adolescents (Yoo et al., 2018) and Stroop task performance in older adults (Fountain-Zaragoza et al., 2019), showing further clinical application of these networks to attentional tasks.

Based on this previous evidence of the generalizability of networks derived using CPM, the primary aims of this study were to: (1) derive a connectome-based predictive model of WM in MS (MS-wmCPM), and (2) test the generalizability of this model to an independent sample of PwMS and an alternative task of WM. A secondary aim was to apply an existing wmCPM derived in healthy young adults (Avery et al., 2020) to both samples of PwMS to assess its translational potential. In addition, we explored the differences between the wmCPM and MS-wmCPM in topology and in the involvement of canonical networks.

Materials and Methods

Participants

Two independent samples were used for deriving the WM model (internal sample) and testing prediction in an independent group (validation cohort). Individuals with relapsing remitting MS were recruited from Columbus, Ohio and the surrounding areas between 2014–2016 (internal sample) and 2017–2021 (validation cohort). Eligibility requirements for both studies included: normal or corrected vision ≥20/40, no cognitive impairment as indicated by a score >23 on the Mini-Mental Status Exam (Folstein et al., 1975; range_internal = 25–30, range_validation = 24–30), no significant depressive symptoms as indicated by a Beck Depression Inventory score ≤19 (Beck et al., 1996), right-handedness, the ability to walk without aid for 100 m as indicated by a self-reported Expanded Disability Status Scale score between 0 and 5.5 (Kurtzke, 1983), the absence of comorbid neurologic or psychiatric disorders, and the absence of corticosteroid use or relapse in the previous 30 days.

Data for the internal sample were initially acquired as part of a cross-sectional study assessing the relationship between physical activity and graph theoretical metrics. Data for the validation cohort were drawn from baseline (before randomization) assessments of a randomized controlled trial (Clinicaltrials.gov: NCT03244696) comparing the effects of a physical activity tracking intervention with a water consumption tracking intervention on cognition. In both samples, participants were excluded from further analysis if they demonstrated excessive head motion (see “Motion controls” in the Supplementary Data) or were deemed by the collaborating neurologist to have incidental findings on structural imaging atypical of MS pathology. Ultimately, the internal sample included 36 participants for model derivation (29 women, aged 30–58 years, M_age = 45.5, M_{disease duration} = 10.8 years) and the validation cohort included 36 participants (28 women, ages 31–58, M_age = 45.3, M_{disease duration} = 9.99 years) for testing model generalizability. For both studies, informed consent was collected in accordance with The Ohio State University Institutional Review Board. Healthy WM networks were generated on 502 adult participants of the Human Connectome Project (274 women, aged 22–35 years, M_age = 28, SD = 3.6 years; as reported in Avery et al., 2020). This cohort was imaged on various scanners as specified in the Human Connectome Project protocol (Van Essen et al., 2013).

Neuropsychological testing

The internal sample completed the Paced Visual Serial Addition Test (PVSAT; Fos et al., 2000) during scanning. Participants were asked to add two numbers presented either consecutively (serial condition) or simultaneously (math condition) and indicate by button press whether the sum was > or <10. The task was divided into 4 alternating serial and math blocks, with 15 trials per block for a total duration of ∼14 min. The dependent variable of interest was the average accuracy across all serial blocks only, as this condition is known to tax WM.

The validation cohort completed the N-back task from the Human Connectome Project, described in detail elsewhere (Barch et al., 2013; Van Essen et al., 2012), during scanning. Participants were presented with pictures of places, tools, faces, and body parts and asked to indicate by button press whether each subsequent image matched the target (0-back condition), or the image two previous to the current image (2-back condition). The task included two runs, each with 8 alternating blocks (four 0-back, four 2-back), and 10 trials per block for a total duration of ∼15 min. Accuracy in the 2-back condition served as the dependent variable of interest.

Two different tasks of WM were selected for network identification and validation as this study aimed at identifying a domain-specific, but task-agnostic network of WM. Prior evidence of correspondence between the auditory analogue of PVSAT and the N-back task in MS has shown adequate behavioral overlap between the two measures of WM (Parmenter et al., 2006). Neuroimaging evidence also shows that both tasks recruit overlapping frontoparietal areas (Au Duong et al., 2005; Colorado et al., 2012; Staffen et al., 2002), and training in one task (N-back) reduces brain activity during completion of the other (i.e., PASAT) (Miró-Padilla et al., 2020), indicating near transfer of training from one WM task to another. Based on the effectiveness of previous sustained attention CPM-based networks to predict attentional performance across tasks and populations (Rosenberg et al., 2016a,b, 2018; Yoo et al., 2018), we aimed to assess the generalizability of WM networks across contexts (tasks and samples).

Network construction

Data for both studies were collected at The Ohio State University Center for Cognitive and Behavioral Brain Imaging on a 3T Siemens scanner with a 32-channel head coil. Acquisition parameters for both studies (Supplementary Table S1) and additional processing details are provided in the Supplementary Data.

Whole-brain functional connectivity

Preprocessing steps included brain extraction and lesion filling of the T1w image, motion and distortion correction, slice timing correction for nonmultiband data, brain extraction, spatial smoothing of the 4D data, and registration of 4D data to the participant's T1w lesion-filled image. Nuisance variables (average white matter, gray matter, cerebrospinal fluid, and global signal) were regressed from the 4D data, and the residual data were advanced to the Graph Theory General Linear Model (GTG) toolbox (Spielberg et al., 2015) for estimation of a whole-brain functional connectivity matrix for each participant, which comprised of edges (i.e., functional connections) between all nodes (i.e., brain regions).

Brains were parcellated by using the 268-node Shen functional atlas spanning cortical, subcortical, and cerebellar regions (Shen et al., 2013). Mean activity in each node was calculated as the average blood-oxygen level-dependent signal across time within all constituent voxels. Functional connectivity was indexed by the Pearson correlation between the mean timecourse of every pair of nodes. All values were then Fisher r-to-z transformed. Sixteen nodes missing sufficient coverage (<5 voxels) in at least one participant (seven in each right and left cerebellum, one in the right brainstem, and one in the left temporal lobe) were excluded from internal analysis, resulting in a 252 × 252 symmetrical connectivity matrix for each participant.

Internal model fitting

CPM (Shen et al., 2017) was implemented by using custom MATLAB scripts to identify functional networks that were predictive of PVSAT scores in the internal sample. Within each of the 36 rounds of leave-one-out cross validation (LOOCV), steps included feature selection, model building, and model-based prediction. Feature selection involved conducting a Spearman's rank correlation between each edge and PVSAT performance across the training set. The correlation threshold was set at p < 0.01 and divided into edges that were positively associated with performance (high-WM network) and negatively associated with performance (low-WM network). Network strengths were computed as the average connectivity among edges included in the high- and low-WM networks. Combined WM network strength was computed as the difference between strengths in the high- and low-WM networks.

Model building involved conducting three linear regressions to separately fit network strengths (high, low, and combined) to WM scores in the training set. Lastly, network strengths for the left-out test participant were computed and entered into each model to generate predicted WM scores based on the low, high, and combined models. These steps were repeated until all participants had served as the left-out sample once. Model performance was assessed as the Spearman's rank correlation between predicted and observed WM scores. To construct final high- and low-WM models (MS-wmCPM), edges found in every round of cross validation were retained, and a general linear model was built to fit strength in those edges to PVSAT scores in the full internal sample. In addition to model derivation using linear regression, we built a separate model using ridge regression. This technique includes a regularization $\lambda$ function, which tends to shrink regression coefficients (Hoerl and Kennard, 2000) to account for multicollinearity among edges, counter overfitting in small samples, and improve model generalizability to unseen samples (Cui and Gong, 2018). Permutation testing, involving random shuffling of brain–behavior relationships, was conducted across 1000 iterations to generate permuted p-values.

Validation

As the gold-standard assessment of predictive power (Scheinost et al., 2019), the MS-wmCPM models were applied to the previously unseen validation cohort. We calculated whole-brain functional connectivity during the N-back task for each participant and then calculated high, low, and combined MS-wmCPM network strengths for each participant. Model performance was defined as the correspondence between network strength and observed N-back scores. We expected a positive correlation between high network strength and WM scores (such that those with greater connectivity in the network that predicts better performance would show higher WM scores) and a negative correlation between low network strength and WM scores (such that those with higher connectivity in the network that predicts worse performance would show lower scores). The “Statistical Analysis” section outlines the multiple measures of predictive power used.

Comparing predictive ability

The predictive abilities of the MS-wmCPM and WM networks derived in healthy young adults (wmCPM; Avery et al., 2020) were compared. For both samples of PwMS, each participant's functional connectivity in the included wmCPM edges was averaged to compute high, low, and combined network strengths. All correlations were Fisher r-to-z transformed and the Dunn and Clark's z test (1971) was computed by using the statistical comparison package cocor (Diedenhofen and Musch, 2015), which compares values from the same (i.e., dependent) sample with overlapping (i.e., observed WM) scores.

Functional anatomy of WM networks

To account for the size of each macroscale network, including medial frontal, frontoparietal, default mode, motor, visual I, visual II, visual association, salience, subcortical, and cerebellar, an index of contribution was calculated by using the formula from Greene et al. (2018). Each edge between nodes i and j was assigned to the respective pair of canonical networks, A and B. In this contribution index, $m_{A,B}$ denotes the number of edges between those canonical networks in the model, and $m_{tot}$ is the total number of edges in the MS-wmCPM. $E_{A,B}$ is the number of edges between networks A and B in the whole brain, and $E_{tot}$ is the total number of edges in the whole brain. A relative contribution value >1 indicated a disproportionately greater contribution of that network pair to the model.

$$Contributio{n}_{A,B}=\frac{m_{A,B}∕m_{tot}}{E_{A,B}∕E_{tot}}$$

Statistical analysis

Statistical analyses were performed by using SPSS 26.0 (SPSS, Chicago, IL) and MATLAB 2019a (MathWorks, Natick, MA). The Shapiro–Wilk and χ² tests were used to assess normality of all continuous and categorical variables, respectively. Independent samples t-tests and Mann–Whitney U tests assessed the differences between the samples on demographic and clinical characteristics. Two-tailed statistical tests with an alpha of .05 were used to determine significance. As predicted and observed values are not expected to lie on the same scale and only correspond in relative terms, model performance was assessed using Spearman's rank correlation (r_s) between observed WM scores and the summary strength of the high, low, and combined networks. As recommended by Scheinost et al. (2019), additional indicators of model performance included the mean squared error (MSE) and prediction/adjusted R² (hereinafter referred to as R²), calculated as 1 minus the normalized MSE between observed and predicted values.

Data availability

The study data and code can be made available on request from the corresponding author.

Results

Demographics, clinical characteristics, and behavioral performance

Table 1 summarizes demographic, clinical, and behavioral data for all three samples. There were no significant differences between the MS samples in any demographic or clinical variable (all p's > 0.38); however, there was a significant difference in total lesion volume (U = 308, p < 0.001).

Table 1.

Demographics, Clinical Characteristics, and Behavioral Performance of Training and Test Samples

	Internal sample, n = 36		Validation cohort, n = 36
	Mean	SD	Mean	SD
Demographics
Age (years)	45.5	7.67	45.3	7.28
Education (years)	16.4	2.34	16.5	1.98
% female	80.6		77.8
Race
Caucasian/White	31		29
African-American or Black	2		4
American Indian or Alaska Native	0		0
Asian	0		0
Native Hawaiian or Pacific Islander	0		0
Other	2		3
Prefer not to answer	1		0
Clinical characteristics
Disease duration (years)	10.8	7.71	11.0	6.01
EDSS	3.71	1.15	3.94	0.72
Total lesion volume (mL)	9.62	6.92	4.48	4.93
Behavioral performance
Math	0.95	0.069	—	—
PVSAT	0.90	0.10	—	—
0-back	—	—	0.91	0.083
2-back	—	—	0.86	0.077

EDSS, Expanded Disability Status Scale; HCP, Human Connectome Project sample used by Avery et al. (2020) to derive working memory networks in healthy adults; PVSAT, Paced Visual Serial Addition Test; SD, standard deviation; —, not applicable data.

Network construction

Internal model fitting

The results of network derivation using LOOCV revealed significant associations between observed performance and predictions based on the high (r_s = 0.47, permuted p = 0.011), low (r_s = 0.50, permuted p = 0.008), and full MS-wmCPM networks (r_s = 0.47, permuted p = 0.008; Table 2). Figure 1 displays the correlations between predicted and observed scores for each network and the corresponding permuted null distributions used for significance testing. It should be noted that prediction accuracy indices of MSE and R² are not reported for internal analysis, as these metrics overestimate model fit when using cross-validation (Poldrack et al., 2019).

Table 2.

Model Performance of Each Network

Network model	Sample	Network	rs	p	MSE	Prediction R2
MS-wmCPM	Internal	High	0.47	0.011a,*
		Low	0.50	0.008a,**
		Full	0.47	0.008a,**
MS-wmCPM	Validation	High	0.026	0.88	0.006	−2.93%
		Low	0.084	0.63	0.006	−0.90%
		Full	−0.047	0.78	0.006	−2.21%
wmCPM	Internal	High	0.20	0.24	0.010	3.39%
		Low	−0.33	0.046*	0.009	16.8%
		Full	0.33	0.049*	0.009	13.4%
wmCPM	Validation	High	0.41	0.013*	0.005	12.7%
		Low	−0.51	0.002**	0.005	17.8%
		Full	0.46	0.005**	0.005	16.9%

The Spearman rank correlations between predicted network strength and observed scores are presented here. For the MS-wmCPM internal sample, correlations are reported between predicted and observed WM scores. The “Network Model” indicates which model was applied to which “Sample” data using which “Network.”

^a Indicates permuted p^* = p < 0.05, ^** = p < 0.01.

MS, multiple sclerosis; MSE, mean squared error; MS-wmCPM = WM connectome-based predictive model derived in the internal sample of individuals with multiple sclerosis using linear regression and leave-one-out cross-validation; WM = Working Memory; wmCPM = WM connectome-based predictive model derived in healthy young adults by Avery et al. (2020).

FIG. 1.

Internal model fit with permutation testing in the derivation sample. On the left are observed PVSAT Z-scores plotted against predicted PVSAT Z-scores from the: (A) high MS-wmCPM, (B) low MS-wmCPM, and (C) full MS-wmCPM networks. Z-scores presented for visualization purposes. Annotations represent Spearman correlation coefficients between predicted and observed scores. Each scatterplot is paired with its respective null distribution of 1000 correlations between predicted and observed PVSAT scores from the CPM procedure with randomly shuffled brain–behavior pairings. The gray bar depicts the Spearman's rank correlation between predicted and observed scores for the true model with actual brain–behavior pairings. Scatterplots and histograms were created by using the python matplotlib package. CPM, connectome-based predictive modeling; MS, multiple sclerosis; MS-wmCPM, connectome-based predictive model of WM in MS; PVSAT, Paced Visual Serial Addition Test; WM, working memory. Color images are available online.

Using ninefold cross-validation with linear regression over 1000 iterations also resulted in a significant correlation between observed and predicted scores for the full network (median r_s = 0.47, median permuted p = 0.004). A single round of LOOCV with ridge regression resulted in significant prediction from the full model (r_s = 0.38, p = 0.022). The model with 100 permutations of ninefold CV using ridge regression was marginally significant (median r_s = 0.31, median permuted p = 0.065), suggesting that the linear model with LOOCV was overfitting to the training sample. As motion was associated with PVSAT accuracy (r_s = −0.42, p = 0.010), in the Supplementary Data we include analyses controlling for the effects of motion on model performance. All models using linear regression and controlling for motion were significant; models constructed using ridge regression were not significant.

Validation

The final networks, containing edges that appeared in every round of cross-validation, contained 189 edges in the high-WM network and 214 edges in the low-WM network (Fig. 2). When testing the generalizability to the validation cohort, the MS-wmCPM did not successfully predict N-back performance in novel PwMS (Table 2). This model did not account for significant variance in observed N-back performance based on the high (r_s = 0.026; p = 0.88, MSE = 0.006, R² = −2.93%), low (r_s = 0.084; p = 0.63, MSE = 0.006, R² = −0.90%), or full MS-wmCPM network (r_s = −0.047; p = 0.78, MSE = 0.006, R² = −2.21%).

FIG. 2.

Distribution of the MS-wmCPM Networks. Panels (A) and (C) present the anatomical distribution of the high network, and Panels (B) and (D) present the anatomical distribution of the low network. Nodes (spheres) represent functionally defined regions, and edges (lines) represent functional connections between nodes. Node size and edge color is proportionate to degree (i.e., number of edges). Ring plots depict the distribution of edges across macroscale regions of the brain. Darker lines in the ring plots indicate a higher degree between macroscale brain regions and match the color scale. Brain images and ring plots were created by using the Yale BioImage Suite Connectivity Viewer Version 1.0.0. LH, left hemisphere; RH, right hemisphere; wmCPM, working memory CPM. Color images are available online.

Comparing predictive ability

The healthy wmCPM model (Avery et al., 2020) successfully predicted WM performance in both samples of PwMS. In the internal sample, models predicted significant variance from the low (r_s = −0.33, p = 0.046, MSE = 0.009, R² = 16.8%) and full networks (r_s = 0.33 p = 0.049, MSE = 0.009, R² = 13.4%) but not the high network (r_s = 0.20, p = 0.24, MSE = 0.010, R² = 3.39%). In the validation cohort, successful predictions of WM were found from all three wmCPM networks: high (r_s = 0.41, p = 0.013, MSE = 0.005, R² = 12.7%), low (r_s = −0.51, p = 0.002, MSE = 0.005, R² = 17.8%), and full (r_s = 0.46, p = 0.005, MSE = 0.005, R² = 16.9%). Comparing model performance between the wmCPM and the MS-wmCPM when applied to the validation cohort, we found a statistically significant difference between the high (z = 2.14, p = 0.032), low (z = 2.28, p = 0.023), and full networks (z = 2.42, p = 0.015), such that the healthy wmCPM outperformed all three of the MS-wmCPM models. Model performance was not compared between the healthy and MS networks for the internal sample, as in-sample model fit estimates likely inflate prediction accuracy.

Functional anatomy of WM networks

Comparing the MS-wmCPM and wmCPM networks, the first striking difference is the smaller number of edges in the MS than in healthy networks: 189 edges in high-MS versus 1674 edges in high-healthy, 214 edges in low-MS versus 1203 edges in low-healthy. It is plausible that successful validation of only the wmCPM may in part be due to the larger raw number of edges comprising these networks. In addition, we found minimal overlap between the networks. Only 17 edges (8.99%) of the high MS-wmCPM were shared with the high wmCPM, and only 6 edges (2.80%) overlapped between the low MS-wmCPM and low wmCPM networks.

We next examined the involvement of 10 functionally defined canonical networks in each network, although interpreted with caution given the smaller size of the MS networks (Fig. 3). We used a contribution index that accounts for network size and the total possible number of connections. Both high and low WM networks in MS showed greater involvement of visual nodes as compared with the healthy network. Examined separately, the high and low, and MS versus healthy networks illustrate observable differences involving the frontoparietal and default mode canonical networks (Fig. 3).

FIG. 3.

Contribution of canonical networks to wmCPMs. Contribution of canonical networks to the high wmCPM (top panel) and low wmCPM (bottom panel). (A) Includes networks derived in individuals with MS (MS-wmCPM); (B) includes networks derived in Avery et al. (2020) using the HCP dataset (wmCPM); and (C) represents differences between the MS-Healthy. Degree of contribution is represented by color opacity; for the third column representing difference, positive (red or blue) values indicate greater contribution in MS networks and negative (gray) values indicate greater contribution in healthy networks. Diagonals represent intranetwork contribution, and the bottom row depicts the average internetwork contribution. The overall average internetwork contribution is presented as the last value (at the intersection of Inter-Inter in the matrices). Value >1 indicates a disproportionately large contribution. Matrices were generated by using the seaborn python package, and ring plots were constructed by using Flourish software. CB, cerebellar; DMN, default mode network; FP, frontoparietal; HCP; Human Connectome Project; INTER, Internetwork; MF, medial frontal; MOT, motor; MS, multiple sclerosis; MS-Healthy, MS relative to healthy networks; SAL, salience; SC, subcortical; VI, Visual I ; VII, Visual II; VA, Visual Association. Color images are available online.

In general, the expected pattern of intranetwork and internetwork involvement was evident in the healthy networks. Greater intranetwork contribution of the frontoparietal network was prominent in the high network (related to better performance) whereas greater internetwork connections between the default mode network and the rest of the brain were evident in the low network (related to poorer performance). In contrast, the MS networks showed a reverse pattern. Compared with the high healthy network, the high MS network involved greater inter-frontoparietal network edges, suggesting that to perform better on a demanding WM task, PwMS may rely on greater integration between frontoparietal nodes and the rest of the brain. In addition, intranetwork connectivity within the default mode was also greater in the high MS relative to the high healthy network. In a similar fashion, the low MS networks involved greater internetwork contribution of the default mode, suggesting that reduced segregation of the default mode network may be related to poorer WM performance. Taken together, these comparisons illustrate that for better WM performance, compared with healthy individuals, PwMS may rely on greater intra-default mode network connectivity and inter-frontoparietal network connectivity with diffuse functional networks.

Discussion

This study aimed to apply a validated predictive modeling approach to identify a functional network predictive of WM in PwMS. Using task-fMRI, we identified a predictive network of WM and tested its generalizability to an independent sample of PwMS during completion of an alternative WM task. Although the WM model generated using linear regression yielded significant predictions in the internal sample, it did not successfully generalize to a novel sample. In addition, the WM model built using ridge regression was not significant, suggesting that the linear regression procedure may have been overfit to the training sample. In contrast, WM networks derived in healthy young adults (Avery et al., 2020) successfully predicted WM performance in both samples of PwMS. These findings complement Avery et al.'s (2020) external validation of these healthy WM networks to clinical populations with varying degrees of cognitive impairment. The healthy and MS networks were found to have minimal overlap in edges and significant differences in involvement of canonical networks. These findings suggest that WM networks derived in healthy adults may demonstrate more robust, generalizable predictions of WM across heterogeneous clinical samples.

The challenge of identifying a reliable neuromarker of cognitive dysfunction in MS has persisted for decades against a backdrop of growing evidence that cognitive function in MS depends on a diffuse functional tapestry (Manca et al., 2018). The WM network derived in MS in this study, spanning all 10 canonical networks, suggests that WM is indeed an orchestration of widespread areas of the brain. The distribution of this network across disparate parts of the brain aligns with previous studies in healthy adults, suggesting that WM calls on distributed patterns of functional circuitry (Breukelaar et al., 2018; D'Esposito, 2007; Eryilmaz et al., 2020; Gazzaley et al., 2004). Further, this expands on correlational research in MS demonstrating that there are associations between WM performance and greater recruitment of broad regions during task-based MRI relative to healthy controls (Mainero et al., 2004; Wishart et al., 2004).

Contrary to our hypothesis, the MS networks did not accurately predict WM function in a novel sample of PwMS. CPM has been successful in predicting many different measures across clinical populations with heterogeneous presentations and symptoms. Specifically, CPM has been used to predict symptom improvement after pharmacotherapy in major depressive disorder (Ju et al., 2020), waist circumference and fasting insulin in overweight and obese individuals (Farruggia et al., 2020), global cognition in samples of individuals with mild cognitive impairment and Alzheimer's disease (Lin et al., 2018), and self-reported executive dysfunction and memory in patients with breast cancer (Henneghan et al., 2020). Of note, although some of these studies included repeated, longitudinal assessment of networks (Henneghan et al., 2020; Ju et al., 2020), none of these studies in clinical populations included independent samples to test the generalizability of identified networks. Thus, successful prediction of WM in our derivation sample is consistent with prior clinical work and our results illuminate potential limitations to generalizability.

Given the success of WM networks derived in young adults to generalize to clinical samples of older adults with varying levels of memory impairment (Avery et al., 2020), we also believe that failure of the WM model to generalize to an independent dataset of MS is not attributable to some central limitation of the method used. However, alternative methods to ordinary least squares (i.e., ridge/lasso regression) may be better suited for small clinical samples as they account for multicollinearity in data with a large set of features (e.g., edges) and counter overfitting to the training data. Our findings support the use of independent samples and penalizing machine-learning methods for stringent tests of generalizability, particularly when aiming to identify robust networks for use as targets in intervention.

Another explanation for the unsuccessful generalization may be related to the inherent structural damage observed in MS. There is evidence that as early as one year from a single neurological episode, white matter damage precedes functional restructuring at both global and modular levels (Koubiyr et al., 2019). Some modeling research posits that functional connectivity changes follow an inverted-U curve such that white matter damage initially leads to higher local functional connectivity, followed by a subsequent decrease, suggesting that the topology and timing of structural damage are important determinants of functional connectivity patterns in MS (Tewarie et al., 2018). Relatedly, as the derivation sample in the present study had high variability in disease duration (M = 10.8, SD = 7.71 years), it may have been comprised of individuals at variable disease stages, and thus with heterogeneous structural and consequent functional connectivity profiles. Given that the final network includes functional connections that were strongly related to behavior in every individual in the derivation sample, it is plausible that heterogeneous reorganization of functional connections at the individual level excluded edges that were relevant in only some individuals. As such, structural disconnection and consequent functional reorganization may have been too variable across individuals for identification of a reliable signature of WM.

Interestingly, WM networks derived in healthy young adults demonstrated robust generalization to both datasets of PwMS. In general, both healthy and MS-derived high-WM networks were characterized by greater contributions of intranetwork connectivity, suggesting that greater functional integrity of canonical networks is predictive of better WM performance. In contrast, the low-WM networks consisted of greater contributions of internetwork connections, suggesting that decreased segregation between canonical communities predicts worse WM performance.

Comparing the relative involvement of canonical networks in the MS versus healthy networks, we found several interesting, yet complementary differences. The high MS network involved greater connections between the frontoparietal network and other networks, suggesting that functional integration may support better WM performance. Our results also show that the high-WM network in MS involved greater connections within the default mode network whereas the low-WM network had a greater contribution of connections between the default mode network and other network nodes. This is consistent with evidence that even in early stages of disease, PwMS show increased functional connectivity in the default mode network—posited as a compensatory mechanism to maintain cognitive efficiency in the presence of structural damage (Louapre et al., 2014). Interestingly, these patterns are the reverse of those observed in the healthy WM networks where greater segregation of frontoparietal networks was related to better performance and poorer segregation of the default mode network was linked to worse performance. These patterns illustrate that although the frontoparietal and default mode networks are of relevance, the segregation and integration of these networks may have differing contributions to performance in healthy individuals and PwMS.

Several important limitations require consideration. First, our sample sizes may have been underpowered to identify a robust neuromarker of WM. Recent studies suggest that samples on the order of ≥100 participants may be necessary for deriving reliable neural signatures of behavior (Poldrack et al., 2019). Thus, size differences between our internal sample and the Human Connectome Project sample could be an alternative explanation for the observed results. As the samples examined here were used as part of parent investigations of physical activity, our study was also restricted to individuals with relapsing remitting MS with mild disease severity and minimal cognitive dysfunction. There was also a significant difference in lesion volume between the two MS samples, which may have contributed to differences in functional network topology and poor generalizability. Another potential limitation of this study was the use of two different measures of WM. It is plausible that, with smaller clinical samples that may demonstrate greater variability in brain–behavior relationships, the behavioral task may impact how well the network predicts WM function in independent samples. Future research may rely on a single gold-standard measure for initial validation of functional networks. This study may also have been limited by the use of ordinary least squares regression for WM model derivation, which was selected a priori to match the CPM procedure used in the healthy sample (Avery et al., 2020). In an exploratory analysis, we were unable to derive a significant model using ridge regression, suggesting that our MS-wmCPM may have been overfit to the internal sample and thus failed to generalize to the validation cohort of MS. Future clinical studies should leverage machine-learning methods (i.e., ridge/lasso regression) to counter overfitting to training data and identify models with greater generalizability. To further enhance the predictive power and generalizability to independent samples of MS, researchers should seek to incorporate structural connectivity and/or lesion pathology data. Finally, a future direction may be to derive and compare models of WM derived in PwMS and healthy controls matched in age, gender, and education, to control for these factors and disentangle differences between healthy and clinical connectomes.

Conclusion

This study demonstrates the utility of CPMs in predicting WM function in PwMS. Models derived in healthy individuals outperformed models derived in MS, suggesting translational potential for use in clinical populations. Future well-powered investigations could build upon these findings by carefully sampling individuals with MS along important parameters, including disease duration, disease subtypes, and levels of cognitive preservation/impairment. Moreover, this literature would benefit from the continued use of predictive and whole-brain search light approaches to understand the potential brain mechanisms of cognitive functions in MS. Joint structure–function connectomes from larger samples provide promising avenues to better understand the complex neural systems supporting cognition and to better identify reliable targets for intervention in MS.

Authors' Contributions

H.R.M. contributed to conceptualization, data curation, formal analysis, investigation, methodology, software, writing—original draft, visualization, project administration. S.F.-Z. contributed to validation, formal analysis, conceptualization, methodology, software, writing—review and editing, visualization. A.S. contributed to data curation, software, writing—review and editing. J.A.N. contributed to conceptualization, supervision, writing—review and editing. R.S.P. contributed to conceptualization, supervision, funding acquisition, resources, writing—review and editing.

Author Disclosure Statement

H.R.M., S.F.-Z., and A.S. report no competing interests. J.A.N. has received research grants from Biogen Idec, Genzyme, Novartis, PCORI, ADAMAS, and Alexion. She has received consulting fees and honoraria from Biogen, Genentech, GW Pharmaceuticals, EMD Serono, Bristol Myers Squib, Novartis, Alexion, Viela Bio, and the American Academy of Neurology. R.S.P. has received speaking honoraria from Sanofi Genzyme.

Funding Information

This research was funded by a grant from the National Multiple Sclerosis Society (No. RG 1602-07744) awarded to R.S.P.

Citation Diversity Statement

Papers by women and other racial and ethnic minorities are under-cited in prominent neuroimaging journals and in science broadly (Caplar et al., 2017; Chakravartty et al., 2018; Dion et al., 2018; Dworkin et al., 2020a,b; Maliniak et al., 2013; Mitchell et al., 2013). In this article, we proactively chose references that reflected diversity in thought. In addition, to support equitable practices in neuroimaging research, Dworkin et al. (2020a) recommended inclusion of a citation diversity statement that reports the proportions of women and other underrepresented scholars that comprise first/last author pairs in manuscript citations. Following their described methodology, we used the open source Gender Diversity code on Zenodo (Zhou et al., 2020) to perform an automatic binary classification procedure. Conducted on both the main text and the Supplementary Data, this diversity analysis classified gender and race based on the first names of the first and last authors (Ambekar et al., 2009; Dworkin et al., 2020b; Sood and Laohaprapanon, 2018; Zhou et al., 2020). Excluding self-citations to the first and last authors of the current study, our references cite 11.47% woman (first)/woman (last), 13.99% man/woman, 21.97% woman/man, and 52.57% man/man. In addition, our references cite 12.37% author of color (first)/author of color (last), 13.31% white author/author of color, 17.93% author of color/white author, and 56.39% white author/white author. These percentages may be biased as the names, pronouns, and social media profiles may not accurately reflect the gender or racial/ethnic identity of the authors. Further, the database is also limited by its false dichotomization of gender into the labels of female and male, excluding intersex, nonbinary, and transgender people. Despite these unfortunate limitations, we believe that highlighting the lack of representation of women and scholars of color in peer-reviewed publications is an important first step toward equity in science.

Supplementary Material

Supplementary Data

Supplementary Table S1