Systematic validation of structural brain networks in cerebral small vessel disease

Abstract

Cerebral small vessel disease (SVD) is considered a disconnection syndrome, which can be quantified using structural brain network analysis obtained from diffusion MRI. Network analysis is a demanding analysis approach and the added benefit over simpler diffusion MRI analysis is largely unexplored in SVD. In this pre-registered study, we assessed the clinical and technical validity of network analysis in two non-overlapping samples of SVD patients from the RUN DMC study (n = 52 for exploration and longitudinal analysis and n = 105 for validation). We compared two connectome pipelines utilizing single-shell or multi-shell diffusion MRI, while also systematically comparing different node and edge definitions. For clinical validation, we assessed the added benefit of network analysis in explaining processing speed and in detecting short-term disease progression. For technical validation, we determined test-retest repeatability.

Our findings in clinical validation show that structural brain networks provide only a small added benefit over simpler global white matter diffusion metrics and do not capture short-term disease progression. Test-retest reliability was excellent for most brain networks. Our findings question the added value of brain network analysis in clinical applications in SVD and highlight the utility of simpler diffusion MRI based markers.

Introduction

Cerebral small vessel disease (SVD) is a leading cause of vascular cognitive impairment and loss of independence in the elderly. Sporadic SVD, related to increased age and arterial hypertension, is particularly common with a prevalence up to 50% in individuals over the age of 70.^1,2 While neuroimaging features currently used in clinical routine are typically based on visible lesions – such as white matter hyperintensities, lacunes, and microbleeds ³ – there is a move towards quantitative markers for measuring disease burden and progression. Measures based on diffusion MRI, such as diffusion tensor imaging, have shown high potential as quantitative markers. They allow detecting subtle white matter changes, are strongly associated with clinical deficits and provide excellent reliability.^4,5

Diffusion MRI analysis approaches differ substantially in their complexity, both in terms of data acquisition and subsequent processing. ⁶ Because SVD is considered a disconnection syndrome, ⁷ brain networks based on tractography and graph theoretical analysis of network structure are regarded as a compelling approach for quantifying clinically relevant brain network alterations in SVD. These structural brain networks are based on fiber tractography and pre-defined regions-of-interest, i.e. the nodes, which are connected via white matter tracts, i.e. the edges. The corresponding network architecture is quantified with graph-informed measures, such as global efficiency, ⁸ which has proven to be the most sensitive graph measure to capture brain alterations in SVD. ⁹ Several studies suggest a high potential of structural network analysis for characterizing SVD burden, for exploring the underpinnings of symptoms or for predicting the disease course.^10–12

There are, however, several critical knowledge gaps that are considered major roadblocks for further application in research and clinical routine. Network analysis is a highly demanding diffusion MRI analysis approach, and the added benefit over simpler diffusion MRI analysis has so far not been systematically assessed. Also, connectome pipelines depend on arbitrary choices, especially in terms of tractography algorithm and the definition of nodes and edges. Of particular interest are more elaborate tractography algorithms, which better model the complex fiber architecture of the brain, but typically rely on a more demanding data acquisition, such as multi-shell diffusion imaging and high-angular resolution. ¹³ These different choices have so far not been systematically compared in SVD. Previous studies suggest that most graph metrics capturing structural network architecture show good to excellent test-retest reliability in healthy young volunteers. ¹⁴ Importantly, the reliability of the network analysis approach in SVD is largely unknown, but is a crucial factor for clinical application.

The goal of this pre-registered study was a systematic clinical and technical validation of structural brain network analysis in SVD. We applied two different connectome pipelines, utilizing single-shell or multi-shell diffusion MRI data, while also systematically comparing different node and edge definitions. For exploration and independent validation, we used two non-overlapping patient samples with state-of-the-art diffusion MRI from the RUN DMC study.^1,15

For clinical validation, we assessed the added benefit of structural brain network analysis in explaining processing speed deficits, the main cognitive deficit in SVD, and in detecting disease progression over time. Our pre-specified hypotheses were that i) compared with simpler global white matter diffusion metrics, brain network analysis better explains processing speed deficits and better captures disease progression over time, and ii) a more elaborate connectome pipeline using multi-shell data and constrained-spherical deconvolution-based tractography outperforms a simpler deterministic connectome pipeline using single-shell data. For technical validation, we assessed test-retest repeatability in a high-frequency serial imaging longitudinal dataset.

Material and methods

Our study design, analysis plan, and hypotheses were pre-registered and are available at https://aspredicted.org/382ha.pdf.

Participants

We included data from SVD patients participating in the RUN DMC study. ¹⁵ For exploration, we used data from the RUN DMC – InTENse sub-study (Radboud University Nijmegen Diffusion tensor and Magnetic resonance imaging Cohort – Investigating The origin and EvolutioN of cerebral small vessel disease). ¹ In this sub-study, 54 patients from the main study were invited to a baseline MRI assessment, used for the cross-sectional analysis, and a total of 9 monthly follow-up MRI scans, used for the longitudinal analysis. For the cross-sectional exploratory analysis, two patients were excluded because of confounding neuropsychological test results, ⁵ which resulted in a final sample of 52 sporadic SVD patients.

For independent validation of the cross-sectional results, we used a non-overlapping sample from the 3rd follow-up visit of the RUN DMC main study (n = 183). Some patients had to be excluded due to missing DWI data (n = 2), non-SVD infarcts (n = 6), MRI protocol violation (n = 1), missing neuropsychological data (n = 16), or insufficient image quality (n = 5). To ensure that results were not driven by outlier observations, we excluded five patients with Trail Making Test (TMT) compound scores qualifying as outlier according to the interquartile range criterion (i.e., scores outside the range defined by the cut-points of the first and third quartile plus 1.5 times the interquartile range above and below). Since the InTENse sub-study deliberately included a subset of patients with higher lesion load, we restricted the main study sample to SVD patients above 70 years of age to keep disease severity roughly similar across both samples (Table 1). This resulted in a final sample of 105 SVD patients for validation.

Table 1.

Sample characteristics.

	RUN DMC – InTENse sub-study n = 52	RUN DMC main study n = 105	p-value
Demographic characteristics
Age [years], median (IQR)	68.50 (8.25)	77.15 (8.19)	<0.0001
Female, n (%)	18 (35)	48 (46)	0.2484
Vascular risk factors, n (%)
Hypertension	43 (83)	75 (72)a	0.2102
Hypercholesterolemia	25 (50)	62 (60)a	0.3318
Diabetes	6 (12)	16 (15)a	0.6843
Current or past smoking	37 (71)	72 (69)	0.8835
Clinical scores, median (IQR)
Processing speed z-score	−0.15 (1.16)	−0.18 (1.55)	0.6265
Barthel scale score	100 (5)	100 (5)a	0.8930
SVD imaging markers, median (IQR)
WMH volumeb [%]	0.35 (0.59)	0.31 (0.74)	0.7984
Lacune count	0 (0)	0 (1)	0.3648
Microbleed count	0 (1)	0 (1)	0.8816
Brain volumeb [%]	77.73 (5.35)	72.51 (4.79)	<0.0001

IQR= interquartile range; WMH = white matter hyperintensity.

^aBased on n = 104 due to missing data for one patient.

^bNormalized by intracranial volume.

For the longitudinal analysis, we split the sample from the RUN DMC – InTENse sub-study into an exploration (n = 27) and validation group (n = 26) while accounting for a similar number of visits and disease severity (i.e. WMH volume) across groups. A few visits had to be excluded from the longitudinal analysis due to insufficient data quality, of which some only became apparent during tractography (n = 7). Only patients with at least 3 MRI visits were included for the longitudinal analysis, rendering the sample size to 25 patients for exploration and 26 patients for validation with a median of 9 (range 3–10) MRIs per participant.

Study protocols were in accordance with the declaration of Helsinki and approved by the medical research ethics committee (CMO Arnhem-Nijmegen). Written informed consent was obtained from all participants prior to the start of the study.

Neuropsychological testing

Neuropsychological testing was performed following identical protocols in both samples. We pre-specified to focus on the core deficit in SVD, i.e. processing speed, which was assessed by the time to complete Trail Making Test matrix A and B. We derived age- and education-corrected z-scores for matrix A and B separately as based on healthy subjects^4,5,16 and next calculated the average to derive an established compound score. Patients were further characterized with respect to vascular risk factors (arterial hypertension, hypercholesterolemia, diabetes, smoking status) and activities of daily living (Barthel scale score).

MRI acquisition and conventional SVD imaging markers

MRI scans were performed on a single 3 Tesla scanner (Magnetom Prisma with 32-channel head coil; Siemens Healthineers, Erlangen, Germany). Imaging protocols in both studies were largely similar and included 3D T1-weighted, 3D fluid-attenuated inversion recovery (3D-FLAIR), and 3D gradient echo (T2*-weighted) sequences. The diffusion MRI protocol was identical in both samples and comprised a multi-band echo planar imaging multi-shell diffusion-weighted imaging sequence (repetition time 3220 ms, echo time 74 ms, diffusion-encoding directions 30 × b = 1000 s/mm² and 60 × b = 3000 s/mm², 10 × b = 0 images, multi-band factor 3). One b = 0 image with inverted phase-encoding direction was acquired for correction of susceptibility-induced distortions during preprocessing. A complete description of all sequence parameters can be found in Table e1.

Conventional SVD imaging markers (white matter hyperintensity [WMH] volume, lacune count, microbleed count, brain volume) were quantified according to the STRIVE consensus criteria. ³ All volumes were normalized to the intracranial volume. Details on the calculation of conventional SVD imaging markers have been described previously. ¹⁷

Diffusion MRI preprocessing

Preprocessing steps included visual quality control, Marchenko-Pastur principal component analysis-based denoising, Gibbs artefact removal, and correction for susceptibility-induced distortions, eddy current-induced distortions, as well as head motion. This was done using tools from MRtrix3 (www.mrtrix.org/, version 3.0.0, dwidenoise,^18–21 mrdegibbs^21,22) and the Functional Magnetic Resonance Imaging of the Brain (FMRIB) Software Library (FSL; version 5.0.10 (RUN DMC – InTENse), version 6.0.1 (RUN DMC main), topup^23,24 eddy ²⁵ ).

Skeleton-based, global white matter diffusion markers

To assess the added benefit of structural brain network analysis, we included measures of global white matter integrity as reference, that were based on diffusion tensor imaging (DTI) and diffusion kurtosis imaging (DKI) metrics. DTI metrics were calculated using ‘dtifit’ in FSL (using only b = 0 and b = 1000 s/mm² images) and DKI metrics using the Diffusional Kurtosis Estimator (www.nitrc.org/projects/dke). ²⁶

For DTI, we included the most commonly used diffusion metrics fractional anisotropy and mean diffusivity. ²⁷ For DKI, we included mean kurtosis and radial kurtosis as reference metrics, since these showed the highest association with processing speed in a previous study. ⁵

Global white matter measures of these metrics were derived as average over a skeleton of the major white matter tracts, as implemented in the tract-based spatial statistics (TBSS) pipeline in FSL. ²⁷ The TBSS-based registration to standard space and projections onto the white matter skeleton was estimated from fractional anisotropy images and then applied to all other diffusion metrics. Prior to averaging, we applied a custom mask to remove all areas from the skeleton that are typically susceptible to cerebrospinal fluid partial volume effects in SVD patients. ⁴ The resulting global white matter diffusion metrics will be referred to as ‘skeleton-based’ diffusion markers.

Overview of the structural brain network analysis

We applied two brain network pipelines (described in detail in the following sections), using either single-shell or multi-shell data as starting point. The key difference between these two brain network pipelines was the tractography approach. In the single-shell pipeline (Figure 1, left), streamlines were tracked by following the main direction of the diffusion tensor per voxel. The multi-shell data (Figure 1, right) enabled a more advanced tractography approach based on constrained-spherical deconvolution (CSD), which reconstructs complex fiber orientation distributions of multiple fiber populations within a voxel. As such, CSD-based tractography allows to track crossing fibers, which occur in most white matter regions. ²⁸ In addition to CSD, we improved the biological accuracy of the streamline reconstruction by introducing anatomical constraints (i.e. streamlines followed white matter fiber orientation distributions, were terminated when entering cortical grey matter and rejected when entering fluid-filled regions; for full algorithm details see original publication). ²⁹

Figure 1.

Overview of the two connectome pipelines (single-shell left, multi-shell right). The single-shell pipeline relies on diffusion tensor imaging and tractography using the FACT algorithm. The multi-shell pipeline relies on MSMT-CSD and anatomically constrained tractography. For both pipelines, we applied the node definition according to the AAL or Brainnetome atlas. After network reconstruction, each structural brain network was summarized by the global efficiency metric (E). N is the set of all nodes in the network, and n is the number of nodes, whereas d(i,j) is the weighted distance between nodes i and j,(i,j ∈ N).

AAL: automated anatomical labelling; ACT: anatomically constrained tractography; BN: Brainnetome; DTI: diffusion tensor imaging; FACT: fiber assignment through continuous tracking; mFA: mean of fractional anisotropy of streamlines; mMK: mean of mean kurtosis of streamlines; MSMT-CSD: multi-shell multi-tissue constrained spherical deconvolution; nSL: number of streamlines; T1w: T1-weighted image; 5TT: five tissue-type image.

We then applied a brain parcellation to the reconstructed streamlines to form structural networks, which are defined by a set of nodes (i.e., brain regions) and edges connecting these nodes. Across both pipelines, we defined the nodes based on the AAL atlas (automated anatomical labelling, 90 ROIs in total, after excluding cerebellar regions) and the Brainnetome atlas with a higher number of smaller regions (246 ROIs in total). Several commonly used edge definitions were used to calculate edge weights (details are given below for each pipeline).

Finally, we derived the global efficiency as a well-established network marker of SVD burden from each structural network (Figure 1, center bottom). Efficiency between two regions is expressed as the inverse of the shortest path length between two regions, ⁸ where the length of each possible path is equal to the sum of the lengths of all edges in that path. Global efficiency of the network is then defined as the average efficiency across all node pairs. To assess whether some nodes within the global network disconnect faster than others over time, we also calculated local efficiency for each node of the structural network that showed the highest clinical and technical validity in the longitudinal dataset (i.e. highest effect sizes in regression analyses, highest added benefit in random forest regression and highest intraclass correlation coefficient [ICC]).

Single-shell pipeline

For single-shell networks, streamlines were reconstructed based on the diffusion tensor using the fiber assignment by continuous tracking (FACT) algorithm (‘dti_tracker’, Diffusion Toolkit, version 0.6.4.1). In short, the algorithm started at the center of all voxels with fractional anisotropy >0.2 and terminated if the streamlines left the brain mask, encountered voxels with fractional anisotropy <0.2 or when the turning angle exceeded 45°. Reconstructed streamlines were filtered and smoothed requiring a step length of 1 voxel (‘spline_filter’, Diffusion Toolkit). Two nodes were considered connected if the endpoints of the reconstructed streamlines lay within both nodes. Atlas parcellations were registered to diffusion space, applying a series of linear and non-linear registrations, leading from the MNI template space, through T1-weighted and FLAIR space, to the diffusion space. All registrations were estimated and concatenated with the Advanced Normalization Tools (ANTs). ³⁰ For the longitudinal data, as previously described in detail, ³¹ baseline T1-weighted images were indirectly normalized into MNI space via an intermediate custom template, and follow-up scans were normalized by concatenating this normalization with the registration between baseline and follow-up T1-weighted images.

Five commonly used edge weights were applied in the single-shell pipeline: number of streamlines (nSL), number of streamlines weighted by the mean length (mLen), number of streamlines weighted by the inverse of the streamline length (invLen), ³² mean fractional anisotropy over all streamlines (mFA), number of streamlines weighted by the mean fractional anisotropy (wFA). Each edge was further normalized by the average volume of the connected nodes to correct for the different volumes of the nodes and for different brain sizes.

For each of the resulting 10 (2 node definitions × 5 edge definitions) weighted undirected networks, we calculated the global efficiency using the Brain Connectivity Toolbox. ⁸

Multi-shell pipeline

For the advanced multi-shell networks, streamlines were reconstructed using a multi-shell multi-tissue constrained spherical deconvolution tractography pipeline using tools from MRtrix3. ²¹ To limit false positives and improve biological accuracy, ²⁵ tractography was anatomically-constrained to white matter by using a 5-tissue-type image generated from T1-weighted images. WMH masks were set as the fifth volume (i.e., pathological tissue) of the 5-tissue-type image to allow tracking within these regions, which are often is misclassified as grey matter, leading the tracking algorithm to terminate prematurely. Importantly, WMH segmentations were performed on each time point in the longitudinal dataset. The remaining steps were: response function estimation (‘dhollander’ algorithm), estimation of the fiber orientation distributions, ³³ multi-tissue informed log-domain intensity normalization, modelling 10 million streamlines using anatomically constrained streamlines tractography, dynamic seeding and cropping at the GMWM-interface, as well as SIFT2 filtering of the streamlines. ³⁴

Consistent with the single-shell pipeline, nodes were defined either according to the AAL atlas or Brainnetome atlas.

Seven different edges were applied in the multi-shell pipeline: number of streamlines (nSL), number of streamlines weighted by the length of each streamline (mLen), number of streamlines weighted by the inverse length of each streamline (invLen), ³⁵ mean of the mean kurtosis of streamlines (mMK), number of streamlines weighted by the mean of the mean kurtosis (wMK), mean fractional anisotropy of streamlines (mFA), number of streamlines weighted by the mean fractional anistropy (wFA). While fractional anisotropy maps were calculated on single-shell diffusion data, edges based on fractional anisotropy were also included here to allow a more direct comparison of the two pipelines. Again, we calculated global efficiency for the 14 (2 node definitions x 7 edge definitions) networks.

Statistical analysis

All statistical analyses were performed in R (version 3.6.1). ³⁶ The statistical

significance level was set at α < 0.05. Since we mainly focused on the effect sizes when interpreting the results, we did not correct for multiple comparisons.

To compare sample characteristics between RUN DMC – InTENse and RUN DMC main, we used chi-squared (χ²) tests (for categorical variables) and non-parametric Wilcoxon rank sum tests (for numeric variables), as appropriate.

Subsequent analyses were conducted completely independently for both samples. The processing speed compound scores were power-transformed using the Yeo-Johnson transformation to approximate a normal distribution. ³⁷

Four main analyses were conducted to examine clinical and technical validation of structural brain network analysis in SVD.

First, we performed simple linear regression analyses between global efficiency of structural brain networks (independent variable) and processing speed performance (i.e., TMT compound score, dependent variable). We used the adjusted R² to quantify and compare associations. Second, we assessed the added benefit for each marker on top of conventional SVD imaging markers (i.e., normalized WMH volume, lacune count, microbleeds and normalized brain volume). We used multivariable random forest regression with conditional inference trees (R package ‘party’, version 1.3.3) to overcome the problem of multicollinearity, which is a critical aspect since SVD imaging markers and diffusion markers are intercorrelated. We constructed one random forest regression model with conventional SVD imaging markers only (number of trees = 1501, mtry = 3), and additional models adding respectively one of the diffusion-based markers. Prediction accuracy was calculated for each random forest regression model as the root-mean-square error (RMSE) between observed and predicted values using leave-one-out cross-validation. The added benefit of each diffusion-based marker for prediction of processing speed was quantified by the difference of the RMSE between the model with and without that diffusion-based marker. ⁵ We repeated random forest regression for each model (with cross-validation) 100 times to determine a point estimate and 95% confidence interval for the RMSE.

Third, to assess the ability of structural brain networks to capture change over time, we calculated linear mixed effects models in the split exploration and validation longitudinal samples. Brain network and skeleton-based diffusion markers were first normalized individually for each patient to the baseline score and then centered and scaled (by subtracting the mean and dividing by the standard deviation). Time of MRI visits (relative to baseline visit) was modelled as fixed effect. To account for patient-specific variation, we included a random intercept for each patient in the model architecture, but models with random slopes per patient did not converge. The fixed effect reflects the mean change in the structural brain network over time, marginal R² reflects the explained variance by the fixed effect. The following R packages were used for estimation of linear mixed models: ‘lme4' (version 1.1.21), ³⁸ ‘lmerTest' (version 3.1.2), ³⁹ ‘boot' (version 1.3.22), ⁴⁰ ‘MuMIn’ (version 1.43.15). ⁴¹

For technical validation, we assessed the test-retest reliability of structural brain networks as our fourth analysis within the same exploration and validation sample used for the longitudinal analysis. Intraclass correlation coefficients (1,1) ⁴² were calculated with the R package ‘psych’ (version 1.9.12.31). ³¹

Deviations from the pre-registered analysis protocol

For parts of this investigation, we had to follow an unplanned path for valid reasons. First, for the independent validation sample of the cross-sectional analysis, we originally planned to randomly sample one hundred subjects from a subset of the UK Biobank with matching range of the WMH volume and age of the RUN DMC – InTENse subjects. However, when calculating the established TMT compound score, we noticed implausibly low z-scores in many UK Biobank subjects due to extremely low reaction times. This might have resulted from a key difference in task administration, since in the UK Biobank study the TMT was performed using a computer mouse, whereas the norm data was based on the paper-pencil version. ¹⁶ Since the TMT data was pre-specified as the clinical endpoint for the cross-sectional analyses, we did not want to deviate from this aspect, but instead chose a different validation sample.

Second, we pre-registered to normalize global efficiency values by the global efficiency of random networks using the Brain Connectivity Toolbox. However, after revisiting the literature, we noticed that in most brain network studies in SVD - if not all - global efficiency values were not normalized. This might result in important consequences of the interpretation of these global efficiency values per se (see discussion), however, to ensure comparability with previous studies, we chose to follow the established procedure for brain networks in SVD.

Third, we planned to compare a random forest regression model with skeleton-based DTI/DKI metrics only as a comparison model, but since we were interested in the added benefit of brain networks compared with skeleton-based diffusion markers – and not in the added benefit of brain networks on top of skeleton-based diffusion markers – we revised our random forest regression approach accordingly.

Data availability

Anonymized data will be made available upon request to the corresponding author.

Results

Sample characteristics are presented in Table 1. SVD patients of the RUN DMC – InTENse sub-study were younger and presented with higher brain volumes compared to patients from the RUN DMC main study (p < 0.0001).

Clinical validation: Associations with processing speed performance

Associations with processing speed performance as assessed by simple regression greatly varied depending on node, edge, and tractography pipeline (Figure 2(a); Table e2).

Figure 2.

Associations between diffusion MRI markers (skeleton- or network-based) and processing speed. Analyses were performed in an exploration (RUN DMC – InTENse) and validation sample (RUN DMC main study). (a) Simple linear regression between each diffusion marker and processing speed. Color and circle size depict explained variance (adjusted R²). (b) Multivariable random forest regression assessing the added benefit of each diffusion marker on top of conventional SVD markers. Plots indicate point estimate and 95% confidence interval for the change in model accuracy as assessed by the RMSE decrease.

AAL: automated anatomical labelling; BN: Brainnetome; FA: fractional anisotropy; invLen: number of streamlines weighted by the inverse length of each streamline; MD: mean diffusivity; mFA: mean of fractional anisotropy of streamlines; MK: mean kurtosis; mMK: mean of mean kurtosis of streamlines; mLen: mean length of streamlines; nSL: number of streamlines; RK : radial kurtosis; RMSE: root mean squared error; wFA: number of streamlines weighted by fractional anisotropy; wMK: number of streamlines weighted by mean kurtosis.

In the exploratory sample, brain networks defined by nSL/wFA edges and AAL nodes explained the highest variance of processing speed deficits (R²=11%) in the single-shell pipeline. Thus, this combination performed slightly better than the simple, skeleton-based diffusion marker mean diffusivity (R²=8%). Explained variance was overall higher for the multi-shell pipeline. Brain networks defined by the wFA edge and Brainnetome nodes best explained processing speed deficits (R²=20%), followed by skeleton-based mean kurtosis (R²=18%) and radial kurtosis (R²=15%).

In line with the findings in the exploration sample, brain networks defined by nSL/wFA edges and AAL nodes yielded the strongest associations with processing speed deficits (up to R²=16%) in the validation sample, explaining more variance than the best-performing skeleton-based diffusion marker fractional anisotropy (R²=13%). In contrast with the exploratory sample, explained variance was barely higher for the multi-shell pipeline. Only brain networks defined by the mLen edge performed as well as the skeleton-based diffusion marker radial kurtosis (R²=14%).

To assess an added benefit in explaining processing speed deficits on top of conventional SVD imaging markers (normalized WMH volume, lacune count, microbleed count, normalized brain volume), we performed multivariable random forest regression analyses (Figure 2(b)). In the exploratory sample, brain networks based on the nSL/wFA edges and AAL nodes showed the highest added benefit (RMSE decrease 5%), whereas the skeleton-based diffusion markers only showed a small added benefit (RMSE decrease 2.5%), or even no benefit (i.e., skeleton-based fractional anisotropy). In the multi-shell pipeline, skeleton-based radial kurtosis added the highest benefit in explaining processing speed deficits (RMSE decrease 7%), followed by brain networks based on the wMK edge (RMSE decrease 6%).

In the validation sample, skeleton-based fractional anisotropy added the highest benefit on top of conventional SVD imaging markers (RMSE decrease 4%), but brain networks with the wFA/mFA edge and the Brainnetome nodes showed a similar benefit. Results from the multi-shell pipeline showed a mixed pattern in the validation sample, with no consistent difference between skeleton-based diffusion markers and most structural brain networks.

Clinical validation: Tracking short-term disease progression in serial MRIs

To assess the ability of brain networks to monitor short-term disease progression, we used data from high-frequency serial imaging and linear mixed effects models (Figure 3(a) and (b); Table e3). In the exploratory sample and using the single-shell pipeline, the brain network based on the mFA edge was the only network parameter demonstrating a significant change over time. A change over time was more evident for skeleton-based diffusion markers (all p < 0.05), as indicated by substantially larger marginal R². Most brain networks derived from the multi-shell pipeline showed a change over time. However, also in this pipeline, the simpler, skeleton-based diffusion markers outperformed brain networks in the ability to capture short-term disease progression.

Figure 3.

Short-term disease progression analysis using linear mixed effects models. (a) Single subject data from the exploration sample. Skeleton-based RK (top) and structural brain networks with AAL node and nSL edge definition (bottom) plotted against time as examples. For better visibility, five subjects are depicted in black and the fixed effect of time is depicted in red. (b) Marginal R² (variance explained by time) from the linear mixed-effects models in the exploration and (c) validation sample.

AAL: automated anatomical labelling; BN: brainnetome; FA: fractional anisotropy; invLen: number of streamlines weighted by the inverse length of each streamline; MD: mean diffusivity; mFA: mean of fractional anisotropy of streamlines; MK: mean kurtosis; mLen: mean length of streamlines; mMK: mean of mean kurtosis of streamlines; nSL: number of streamlines; RK: radial kurtosis; wFA: number of streamlines weighted by fractional anisotropy; wMK: number of streamlines weighted by mean kurtosis.

This pattern was replicated in the validation sample (Figure 3(c)).

To assess regional changes over time, we calculated local efficiency of each node of the structural brain network weighted by the mFA and defined by the AAL atlas, since this was the network with the largest change over time among global networks. Only four out of 90 nodes showed a significant change over time, but effect sizes were extremely small (fixed effects <0.002, marginal R² < 0.02).

Technical validation: Test-retest repeatability

For technical validation, we used intraclass correlation coefficients to estimate test-retest repeatability in the serial MRI dataset. Most networks from the single-shell pipeline showed excellent test-retest repeatability with intraclass correlation coefficients higher than 93% in the exploratory sample (Figure 4(a) and (b); Table e4). Only brain networks based on the AAL node and mLen edge definition were less reliable (ICC < 89%). Still, skeleton-based diffusion markers demonstrated a better test-retest repeatability (ICC > 98%). Some brain networks derived from the multi-shell pipeline showed also high test-retest repeatability, in particular those based on a mMK or mFA edge definition were in the range of the skeleton-based diffusion markers (ICC > 97%). The remaining brain networks of the multi-shell pipeline were less reliable. Especially the invLen edge showed intraclass correlation coefficients below 60%, indicating the worst technical validity of all assessed brain networks. Again, this pattern was replicated in the validation sample (Figure 4(c)).

Figure 4.

Test-retest repeatability of diffusion markers. (a) Scatterplots showing the consistency of diffusion markers illustrated using the first two visits (time points t1 and t2) for skeleton-based MD (top) and wFA structural brain networks (bottom) as examples. In case of perfect test-retest repeatability, all points would lie on the diagonal. (b) Intraclass correlation coefficients of diffusion markers assessed in the exploration and (c) validation sample.

AAL: automated anatomical labelling; BN: Brainnetome; FA: fractional anisotropy; ICC: intraclass correlation coefficient; invLen: number of streamlines weighted by the inverse length of each streamline; MD: mean diffusivity; mFA: mean of fractional anisotropy of streamlines; MK: mean kurtosis; mMK: mean of mean kurtosis of streamlines; mLen: mean length of streamlines; nSL: number of streamlines; RK: radial kurtosis; wFA: number of streamlines weighted by fractional anisotropy; wMK: number of streamlines weighted by mean kurtosis.

Discussion

We systematically assessed the clinical and technical validity of brain network analysis as a marker for SVD. Our main findings are that i) for explaining processing speed, structural brain networks provide only a small added benefit over simpler, global white matter diffusion markers; ii) structural brain networks do not capture short-term disease progression over time; iii) multi-shell imaging does not improve the clinical validity of structural networks; iv) most structural brain networks show excellent test-retest reliability and thus a high technical validity and v) node and edge definitions have a substantial effect on brain network analysis results, highlighting the need for standardization to facilitate comparisons between studies.

Markers from diffusion MRI are well-established quantitative markers for SVD, both for cross-sectional and longitudinal use.^4,5 While global white matter markers, e.g. obtained as average over the entire fiber tract skeleton, offer a straightforward implementation, they do not account for the complex network structure of the human brain. Brain network analysis leverages this complex information for additional mechanistic insight, but its implementation is more demanding and subject to arbitrary decisions, such as node and edge definitions. In our pre-registered analysis and systematic comparison, we did not find a substantial advantage of brain network analysis over simpler, skeleton-based diffusion MRI markers. Importantly, for capturing change over time in the longitudinal dataset, the simple markers were clearly superior. Thus, our findings question the added value of brain network analysis over simpler methods for clinical applications in SVD.

Diffusion MRI excels especially for longitudinal studies of SVD due to excellent test-retest reliability and high sensitivity to subtle white matter changes. Previous studies have shown that diffusion MRI markers (i.e. global skeleton-based markers) yield the smallest sample size estimates for assessing treatment effects over time, thus offering great potential to facilitate phase II randomized controlled trials.^4,43 While brain networks were also very reliable in terms of test-retest repeatability, they failed to capture short-term disease progression over time. However, brain topology changes might only become visible throughout long-term disease progression reflecting secondary degeneration. ⁴⁴ Still, tracking short-term progression is of particular interest for clinical trials with typically limited study duration. Thus, skeleton-based markers can be considered the first choice for application in clinical trials. Nonetheless, network analysis might still be useful for gaining pathomechanistic insights. As such, previous work has shown that especially connections between rich club nodes (i.e., nodes that are highly interconnected) decline in SVD. ⁴⁵ The development of targeted intervention strategies might benefit from such mechanistic insights. Also, while global diffusion markers are mostly determined by SVD – not Alzheimer’s disease pathology ⁴⁶ – in memory clinical patients, regional brain network analysis might offer the possibility to disentangle the contribution of different pathologies.

Given that the brain’s white matter contains more than 80% of crossing fibers, ²⁸ we expected the more elaborate connectome pipeline with modelling of multiple fiber populations within a voxel via constrained spherical deconvolution to better depict SVD burden. Surprisingly, networks based on the multi-shell pipeline did not show an advantage over those based on the single-shell pipeline, although the latter method completely neglects the issue of crossing fibers.

Overall, we can only speculate why there was no clear added benefit of structural network analysis over simpler, skeleton-based diffusion markers. A potential explanation might be that the complex algorithms needed to construct the networks were developed on brains of young, healthy volunteers. The marked alterations of white matter microstructure ² in SVD might interfere with or violate model assumptions, e.g. of tractography algorithms. Furthermore, small vessel disease is now recognized as a global brain disease and thus, a simple global marker, such as the skeleton-based markers, might be sufficient to capture disease burden. Similar to the tractography algorithm, the more elaborate Brainnetome atlas with a finer parcellation and better representation of the functional organization of the cortex was not superior to the AAL parcellation, which has a rather coarse, purely anatomically based parcellation. In conclusion, the simpler connectome pipeline, which was used in most previous brain network studies in SVD, performed best among the different combinations. Along the same theme that simpler measures perform better than more complex methods, and contrary to our previous study, ⁵ we did not observe a benefit of the diffusion kurtosis model in the validation sample.

In line with previous work on brain networks in SVD,^10,11 we did not normalize the brain networks by the global efficiency of random networks. We also did not threshold the corresponding networks to a certain density and refer to previous work suggesting that – at least in the multi-shell pipeline – it might not be necessary to do so. ⁴⁷ In addition, others have already reported the effect of density thresholding and concluded that networks weighted by fractional anisotropy might be less prone to network density effects. ⁴⁸ Still, global efficiency measures might be influenced by the density of the structural network and should thus not be understood as the “efficiency” of the brain network, but rather be interpreted as a global diffusion marker of the brain network. However, to not add another level of complexity, we decided a priori against trying out different arbitrary density thresholds.

Several limitations of our study need to be discussed. First, we only focused on global efficiency as the core graph metric of structural networks in SVD, even though other graph metrics might have been suitable as well. However, others have reported global efficiency to be the most sensitive graph measure of cognitive impairment in SVD, ⁹ which is why we decided a priori on using this graph measure to reduce the complexity of the study. Also, we did not normalize the global efficiency measure against null-models. Consequently, global efficiency might be heavily influenced by the density of the structural network. However, to facilitate a comparison with previous work, ¹¹ we decided against this normalization step. Second, while the exploration and validation sample were non-overlapping in terms of study participants, they were collected at the same center with identical protocols and might therefore not be considered as fully independent. Accordingly, we cannot estimate how our results would generalize to a dataset with a different acquisition protocol. On the other hand, this can also be regarded as a strength, since observed differences between results in the exploration and validation sample are unlikely to originate from technical differences. Third, to identify the optimal imaging marker, we did not test for significance between different markers. However, since this is methodologically non-straightforward, we decided to focus on the comparison of effect sizes and included two non-overlapping samples for replication of findings and to further facilitate the interpretation of results. A main strength of the study is its pre-registration. To our knowledge, this is the first brain network study in SVD using a fully pre-specified analysis plan. As demonstrated here, results highly depend on arbitrary choices during analysis, which is why pre-specifying the analysis plan is of great importance to improve the transparency and quality of research on network analysis in SVD.

New diffusion analysis techniques are constantly emerging. Evaluating the benefit of a novel method over established techniques as well as technical validation are indispensable for evaluating the utility in research and clinical use. ⁴⁹ The structural network approach is compelling as it captures the complex network structure of the human brain. But when in need of disease burden or progression markers, network analysis did not show an advantage over the simpler skeletonized approach. Because skeleton-based markers are more straightforward to implement, even with fully automated processing, we conclude that skeleton-based diffusion markers are currently better suited for clinical research, trials and potentially also routine use.