Reply: Lesion network mapping predicts post-stroke behavioural deficits and improves localization

In their letter to the editor, Cohen and colleagues¹ commented on our study² in which we compared different structural and functional MRI methods to predict behavioural deficits in a large cohort of subacute stroke patients.³ Specifically, we estimated behavioural deficits in different domains (language, vision, motor, attention, and memory) using: (i) lesion location and size; (ii) structural disconnection of white matter pathways estimated in a healthy control atlas (SDC); (iii) functional disconnection of brain networks (FDC) estimated in healthy controls using an approach similar, but not identical to lesion-network mapping (LNM)⁴; and, finally, (iv) direct measures of functional MRI connectivity in a subset of patients. The prediction of behavioural impairments at 2 weeks post-stroke used ridge regression, a machine learning technique that provides out-of-sample estimates while minimizing the possibility of overfitting.⁵

Our results showed higher behavioural prediction for lesion and SDC, and functional connectivity, but a weak prediction for FDC. We concluded that FDC helps to localize functional network abnormalities after focal damage, but not to predict behaviour reliably, nor as a substitute for direct functional MRI-FC (functional connectivity).

We wish to thank Cohen and colleagues for their interest and because their critical analysis further stimulates a scientific discussion on indirect methods for measuring network dysfunction based on healthy control connectomes. They performed new analyses using LNM maps on the same dataset to directly challenge our conclusions. LNM maps are voxel-wise t-maps of the temporal correlation between the mean signal blood oxygen level-dependent (BOLD) time course (averaged over voxels) within the lesion seed and all other voxels outside of the lesion (voxel-wise) in a dataset of 1000 healthy control subjects.⁶ From this LNM they first generated a univariate t-map correlating behavioural factor scores and voxel-wise functional connectivity leaving one subject out. Next, by summing the positive (only) T-values across the voxels falling within the left-out subject’s lesion mask, they computed a ‘network damage score’. They repeated these steps each time, leaving out a different subject. The distribution of network damage scores obtained across all lesions was then correlated with the behavioural scores using a Pearson’s correlation R then squared to R² to estimate the accuracy of the behavioural prediction (i.e. shared variance between the sum of the T-value and patients’ behavioural outcome). Cohen et al.¹ compared these R² values with those computed from ridge regression in our paper and showed the behavioural prediction based on network damage scores higher than FDC. They concluded that the ‘lesion network mapping predicts post-stroke behavioural deficits and improves localization’ and that their strategy to obtain behavioural prediction from indirect resting state functional MRI (rs-fMRI) connectomes is more appropriate.

An essential step for the reader is to understand the many differences between our methods and those of Cohen et al.¹ We will then consider whether each difference matters. Our goal is to compare ‘apples with apples’ even though this will require more ad hoc work.

Normative resting state functional MRI datasets used for SDC and FDC maps differs. Our SDC and FDC maps came from computing measures of structural and functional disconnection in a group of n = 170 subjects from the Human Connectome Project (http://www.humanconnectome.org/study/hcp-young-adult/) scanned on a 7 T MRI scanner, while Cohen et al. used the 1000 subject functional connectome at 3 T.⁶ It is not clear at this point whether a smaller sample size (n = 170 versus 1000) or field strength (7 T versus 3 T scanners) matters. However, we note that the smaller sample size in the 7 T dataset comes with longer acquisition times (64 versus 6 min), which should improve signal-to-noise, and smaller slice thickness (1.6 mm versus 3 mm isotropic), which should improve anatomical resolution.

After visually comparing the results of the functional disconnection maps obtained with the two procedures, it is apparent that LNM maps have lower spatial resolution and involve more of the white matter (Fig. 1). This point is relevant to the recent reply⁷ to Boes.⁸ However, LNM maps are more dissimilar than FDC maps across different lesions, which may explain why LNM maps yield a higher number of principal components to describe the data variability (Table 1 in Cohen et al.¹). Higher spatial dimensionality will improve behavioural prediction.

Figure 1.

Lesions, and comparison of structural disconnection (SDC), functional disconnection (FDC) and lesion network (LNM) maps for a subset of stroke lesions from Salvalaggio et al. ²

The second most important difference is the method used to estimate the accuracy of the prediction of the patients’ behaviour, i.e. the computation of the R² values. We used a state-of-the-art machine learning method to develop an out-of-sample estimate of model versus real behavioural scores based on the principal components that account for 95% of the structural or functional disconnection variance across subjects. Cohen et al. computed Pearson r-values between behavioural and out-of-sample network damage scores, which were then squared. Like apples and oranges, the two measures are not comparable.

To examine the FDC and LNM methods more rigorously, Cohen et al.¹ kindly provided their LNM maps and we ran ridge regression estimates in several combinations (Table 1). Across different behavioural domains (language, vision, motor, etc.) LNM performed comparably or better than FDC, but always lower than estimates based on lesions. Also, LNM always performed worse than SDC, except for the right visual factor. These analyses confirm that indirect functional disconnection (both FDC and LNM) performs worse than lesions and indirect structural disconnection in predicting acute impairment post-stroke.

Table 1.

Ridge regression analysis: R² values

	Lesion (R2)	SDC (R2)	FDC (R2)	FDC t-maps (R2)	LNM t-maps (R2)	LNM t-positive (R2)	Lesion + LNM t-maps (R2)	Lesion + LNM t-positive (R2)
Language	0.48	0.41	0.06	0.19	0.25	0.38	0.40	0.39
Motor left (right lesion)	0.35	0.37	0.12	0.11	0.13	0.14	0.26	0.19
Motor right (left lesion)	0.28	0.42	0.08	0.09	0.12	0.11	0.17	0.11
Visual left (right lesion)	0.40	0.23	0.18	0.11	0.21	0.23	0.35	o.f
Visual right (left lesion)	0.58	0.33	0.38	0.33	0.40	0.38	0.63	o.f.
Verbal memory	0.06	0.05	–	0.03	–	–	–	–
Spatial memory	0.19	0.19	0.04	0.03	0.15	0.16	0.15	0.16
Attention (visual field bias)	0.18	0.16	0.10	0.14	0.04	0.11	0.17	0.17
Attention (IC visual field bias)a	0.09	0.18	0.08	0.07	0.06	0.08	0.06	0.08

FDC = functional disconnection; IC = ipsi-contra correction; LNM = lesion network mapping; o.f. = model overfitting; SDC = structural disconnection.

^aAttention (IC visual field bias) score takes in account the ipsi-contra correction of visual field bias.

Another difference comes from the mathematical computation of the indirect functional disconnection map. The original FDC maps were based on average Pearson correlation, while LNM maps are t-maps. We used R-values to take into account the strength of the signal temporal correlation across subjects, while t-values emphasize consistent correlation values across subjects, both small and large. When we ran ridge regression on t-maps from our dataset, the results were mostly similar to the original except for language (Table 1).

Another critical issue is whether both positive and negative correlation values should be kept in FDC or focused on positive correlations only as in LNM. We agree with Cohen and colleagues that negative correlations are more challenging to interpret, yet we believe that both positive and negative correlations provide information on network abnormalities. In any case, ridge regression on LNM maps considering both positive and negative t-values, or only positive values yielded comparable results (Table 1).

Lastly, LNM and FDC differences may be due to data preprocessing differences such as brain normalization and template registration in the two healthy control datasets. This check will require more systematic investigations.

In summary, our new analyses support Salvalaggio et al.,² that indirect functional disconnection maps have lower explanatory power than lesions or structural disconnection for behavioural impairments.

However, why did the LNM maps from the analysis of Cohen et al.¹ perform better than ours? We note that Cohen et al’s method for behavioural correlations relies, first, on a univariate regression between LNM and behavioural score (t-maps), then, a correlation between ‘network damage score’ (i.e. the sum of positive t-values within each subject’s lesion mask) and the behavioural score. The resulting Pearson r is essentially a measure of the intersection between the lesion location (size) and the functional connectivity-to-behaviour regression t-maps. We also note that the function linking network damage scores to behavioural scores should be estimated through cross-validated regression, because correlation does not speak to out-of-sample prediction accuracy and it is descriptive at best. Moreover, squaring the r-value is not considered an adequate measure in predictive modelling.⁹

We think that the label ‘network damage score’ is slightly misleading because the score reflects the overlap of the behaviourally significant LNM map with the lesion. Perhaps a more accurate term would be ‘network-to-lesion damage score’. However, more critically, this score does not tell us whether the LNM maps are behaviourally relevant, rather whether spatial information in the LNM map overlapping with the lesion is relevant. Network spatial information and spatial lesion information are thus confounded.

First, even though lesion size was controlled in the computation of t-maps, it was not factored out in the correlation between ‘network damage score’ and behaviour (note that the number of summed t-values is critically dependent on the number of lesioned voxels). The simple correlation between lesion size and behaviour was significant and comparable to that obtained with LNM (Table 2).

Second, to assess the influence of the lesion location’s influence, we first computed the univariate regression between the lesion and behavioural score (lesion t-maps) (Supplementary material in Salvalaggio et al.²); then, we computed the correlation between the sum of positive t-values within each lesion mask and the behavioural score (Table 2). The behavioural correlation was comparable to that obtained with LNM maps indicating that LNM maps tap into information already present in the lesion itself. To verify this assertion, we checked whether the amount of explained behavioural variability increases by adding LNM information to the lesion. In all domains, but the right visual, the lesion alone performed better than lesion plus LNM (Table 1).

Table 2.

Correlation analysis: r-values

	LNM from Cohen et al.1 r (absolute values)	Lesion from Salvalaggio et al.2 r (absolute values)	Correlation score-lesion size r (absolute values)
Language	0.64	0.71	0.71
Motor left (right lesion)	0.63	0.64	0.55
Motor right (left lesion)	0.47	0.56	0.49
Visual left (right lesion)	0.57	0.68	0.59
Visual right (left lesion)	0.62	0.66	0.33
Verbal memory	0.17	0.48	0.44
Spatial memory	0.42	0.49	0.46
Attention (visual field bias)		0.11	0.08
Attention (IC visual field bias)a	0.42	0.41	0.13
Total mean	0.49	0.58	0.46

IC = ipsi-contra correction.

^aAttention (IC visual field bias) score takes in account the ipsi-contra correction of visual field bias.

In summary we believe that the LNM maps from the Cohen et al.¹ analysis performed better than the FDC maps in predicting behavioural impairments because the network damage score correlation with behaviour is partially explained by variance explained by lesion location and size.

We hope that this debate will clarify the pros and cons of LNM. In our opinion, indirect estimates of functional disconnection are useful in mapping the brain circuitries associated with focal lesions and possibly other disorders.⁴ However, we remain sceptical of this approach to predict behaviour or recovery of function. For such precision-medicine applications, direct measurements of functional connectivity are bound to be more precise. Finally, as discussed in our paper² and in a recent letter by Aron Boes⁸ and scientific commentary by Umarova and Tomalla,¹⁰ other factors limit LNM, including lower signal-to-noise in the white matter and possibly mixing of signal time courses when averaging in a large lesion. Some of these issues are worth exploring in future work because the method is beautifully simple, straightforward, and relatively easy to use.

 

Data availability

The data are available from the corresponding author, upon reasonable request.