How covariate control can bias our insights into brain architecture and pathology

Abstract

Inferential analysis of normal or pathological brain imaging data—as in brain mapping or the identification of neurological imaging markers—is often controlled for secondary variables. However, a rationale for covariate control is rarely given and formal criteria to identify appropriate covariates in such complex data are lacking. We investigated the impact and adequacy of covariate control in large-scale imaging data using the example of stroke lesion-deficit mapping. In 183 stroke patients, we evaluated control for age, sex, hypertension, or lesion volume when mapping real or simulated deficits. We examined (i) the impact of covariate control when mapping cognitive poststroke deficits, (ii) the association of covariates and brain pathology (iii) the impact of covariate control when mapping simulated deficits, and (iv) the impact of covariate control under a verified null hypothesis. We found that the impact of covariate control varies across covariates and deficits in both real and simulated data. However, simulations showed that covariate control does not necessarily improve the precision of lesion–deficit inference. Instead, it systematically reshapes statistical maps according to the associations between covariates and lesion anatomy. As a result, covariate control can bias statistical inference and, under a verified null hypothesis, may even generate spurious associations. These findings suggest that the widespread use of covariate control in clinical brain imaging—and likely other biological high-dimensional data—should be reconsidered, as it may introduce substantial analytical flexibility without necessarily improving inference.

Supplementary Information

The online version contains supplementary material available at 10.1038/s41598-026-47122-4.

Introduction

“Trust is good, control is better” could serve as a credo for many of the life sciences, including neuroscience: statistical control appears to be omnipresent, generally desirable and necessary. In imaging studies on human brain function or pathology, a plethora of variables is commonly controlled for as covariates, while only a minority of studies include no covariates¹. Easily accessible covariates such as age and sex are very common¹. Education level is a covariate in studies on cognitive and even non-cognitive outcomes^2–4. The relationship between stroke lesion imaging and cognitive deficits or clinical outcomes is often controlled for lesion size^5–8. When mapping the neural correlates of different semantic error types in aphasia, taxonomic and thematic errors are controlled for against each other⁹. The association of small vessel disease markers and neurological health is investigated with 12 covariates, including diabetes, smoking, and serum total cholesterol⁴. The association of cortical thickness and the Big Five personality factors is controlled for intelligence, and each personality factor is controlled for the remaining four¹⁰.

Even given this small number of examples, the question arises as to what criteria, aside from mere conventions of each field, should be used to select covariates. What covariates are included markedly varies even across similar studies, and brain mapping studies using anywhere from no covariates to more than a dozen are publishable¹. To our knowledge, a rationale for why a particular covariate is considered and why others are not is rarely stated. However, the inclusion of covariates can have different and strong effects¹. Hence, a clear, objective rationale for the selection of covariates seems necessary. Moreover, control can have negative effects by overshadowing true associations or creating statistical artefacts. For example, a humorous illustration of the abuse of researchers’ degrees of freedom¹¹ suggested that, after the inclusion of a covariate, listening to a song biologically rejuvenates listeners. In stroke lesion-deficit mapping, control for lesion size or secondary deficits can create spurious associations^12,13.

The literature on causal inference cannot only provide a theoretical background as to when and why to apply statistical control, but it also shows awareness of the potential dangers of ill-applied covariate control^14–16. When we want to infer causal effects from observational data, the strategy to identify a covariate requires making assumptions about the causal relationships between all variables. This strategy also identifies variables that should not be controlled for because they potentially mask true causal relationships or introduce spurious correlations. These causal assumptions are commonly visualised in causal graphs (for an accessible introduction, see^15,16.

With the present manuscript’s focus on methods in human neuroscience, we should now realise the methodological abyss that has just opened up before us: in neurological or biological big data, statistical analyses often include thousands of variables. The current study investigates the impact of covariate control on the example of stroke lesion-deficit mapping. This method investigates the relationship between the location of brain lesions and behavioural or cognitive deficits, and is used to explore the functional architecture of the human brain and to understand the causes of post-stroke deficits. This brain mapping method on clinical data serves as a strong example for the analysis of observational high-dimensional data and carries a history of methodological discussions about the appropriateness of covariate control (e.g.^7,12. The many imaging variables—like the lesion status of voxels or brain regions—are not independent but are intertwined due to the underlying complex anatomy of the cerebral vasculature¹⁷, where an occlusion within the hierarchy of the arterial tree causes pathology in many imaging features. Post-stroke deficits are caused by damage to certain brain structures and networks—but the severity of a deficit may vary across the location and extent of damage to relevant structures and may depend on varying interactions of multiple structures¹⁸. Next, covariates can have a complex relationship with many variables. For example, consider sex: stroke outcome is linked to sex-specific lesion patterns¹⁹, sex is associated with stroke outcomes and interacts with age^20,21, and animal models and theories on the impact of sex hormones suggest a causal effect of sex on stroke outcomes^21–23. Last, a causal relationship between sex and a clinical or cognitive target variable may exist through sociological, psychological or biological effects. In summary, it is difficult, if not impossible, to make reasonable assumptions about the causal relationships among all variables in brain mapping with tens to millions of variables. Hence, the requirement to evaluate the appropriateness of covariate control in brain mapping can hardly be fulfilled.

In the current study, we examined the impact and adequacy of common demographic or clinical covariates when mapping neurological and cognitive deficits. We focussed on the method of lesion-deficit mapping in stroke²⁴, as it is a popular method to infer functional brain architecture and as it can be objectively evaluated in silico through an established simulation approach²⁵. To gauge the effects of covariate control, we investigated (i) how covariates affect real lesion-deficit mapping results, (ii) how these covariates relate to lesion anatomy, (iii) how covariates affect lesion-deficit mapping of in silico variables, i.e. simulated variables for which the neural correlates are perfectly known, and (iv) the impact of covariates alone under the null hypothesis, i.e. when the mapped variables are not related to lesions of any brain area.

Methods

Patients and clinical assessment

We analysed data from a prospective study on post-stroke cognitive impairment²⁶(ClinicalTrials.gov Identifier: NCT05653141). This study recruited acute ischemic stroke patients admitted to the Stroke Centre of the University Hospital Bern. Detailed inclusion and exclusion criteria are reported elsewhere²⁶. In short, patients with a first-ever anterior circulation ischemic stroke confirmed by MRI and stroke onset ≤ 10 days were included. Patients with aphasia were excluded to avoid the influence of language deficits on the neuropsychological test results. Hence, due to the use of data from this stroke study, the final sample contained more right-sided (56.3%) than left-sided (36.1%) or bihemispheric (7.7%) strokes. Clinical and demographic information was retrieved from the clinical documentation, including the diagnosis of arterial hypertension. We received written free and informed consent from all participants. The study was approved by the local ethics committee (Kantonale Ethikkommission Bern KEK 2020–02273) and conducted in accordance with the Declaration of Helsinki.

A trained psychologist performed a multi-domain cognitive assessment during the hospital stay within 10 days after stroke onset. The present study focussed on the impact of covariates such as age or sex. Hence, we used raw performance scores and, contrary to common clinical practice, not scores that already account for such variables by age- or sex-specific norms. For the current study, we investigated three different neuropsychological measures: (i) Visuoconstructive ability as included in the CERAD test battery²⁷. Patients were asked to copy four line drawings of increasing complexity and their performance was rated on an 11-point scale according to 2–4 pre-defined evaluation criteria for each shape. (ii) Spatial selective attention using the paper-and-pencil Bells Test²⁸. Patients were asked to circle bell-shaped black icons placed among distractor items. The final score was the total number of omitted targets. (iii) Short-term memory measured by digit span forward performance included in the WAIS-IV²⁹. Patients were asked to repeat between 2 and 9 digits in the same order as read aloud by the examiner. The test was terminated after two false digit sequences in succession with the same length. The final measure was the maximum number of correctly repeated digits.

Brain imaging and lesion delineation

We identified lesioned tissue using diffusion-weighted MRI acquired 24 h after stroke onset with MATLAB and SPM12 (https://www.fil.ion.ucl.ac.uk/spm/software/spm12/). The lesion area was delineated by a semiautomatic algorithm under supervision of a neurologist with > 10 years of experience in lesion mapping (author R.M.U.) as done previously³⁰. Lesion masks were normalised to the Montreal Neurological Institute space at 2 × 2 × 2 mm³ resolution with normalisation parameters from the co-registered T1 scans. These procedures resulted in a binary mask for each patient indicating lesioned vs. intact voxels in a standardized coordinate system that allows spatial comparison of the lesion area between patients. Lesion volume was assessed as the volumetric size of each patient’s normalised binary mask.

Lesion-deficit mapping

We mapped lesion-deficit associations using well-established methods²⁴ that statistically map the association between lesion status (lesioned vs. intact) and a deficit for each voxel in the brain. Analyses were done either with frequentist or Bayesian general linear models. Frequentist statistical mapping was performed in NiiStat (https://www.nitrc.org/projects/niistat) in MATLAB R2023a with control for multiple comparisons (i.e. control for α-error accumulation due to repeated testing in each voxel) by maximum statistic permutation, a family-wise error correction procedure³¹ with 4000 permutations and a one-sided α = 0.05. Covariate control was implemented by the Freedman-Lane procedure³², which allows for covariate control in a permutation design. Bayesian lesion-deficit inference (BLDI) was performed with the BLDI toolkit¹³ in R software 4.2.1 using Bayesian general linear models in the BayesFactor package v0.9.12³³. Covariates were included as an additional independent variable. Resulting Bayes Factors indicate the ratio of evidence for the alternative hypothesis H1 (a lesion-deficit association exists, two-sided) against the null hypothesis H0 (a lesion-deficit association does not exist) and were interpreted according to common conventions³⁴. All analyses were restricted to voxels that were damaged in at least 4 patients.

As default, we utilised frequentist statistical mapping, which is commonly considered a comparatively precise gold standard of mass-univariate voxel-wise mapping³⁵. We instead used BLDI whenever the focus of an experiment was the null hypothesis as Bayesian statistics, contrary to frequentist statistics, can provide evidence for the null hypothesis. We also used BLDI if real-world deficits yielded no significant results with the frequentist approach, as it can detect subtle effects¹³.

Study design and experiments

Experiment 1: impact of covariate control on real-world lesion-deficit inference

We mapped the real-world post-stroke neuropsychological measures of visuoconstructive ability, selective attention, and short-term memory. We did so either without any covariate control, or while controlling for one of the following variables: age, sex, presence of hypertension, or lesion volume. We first aimed to descriptively explore the impact of covariate control in such a setting. Quantitatively, the similarity of controlled and uncontrolled maps was assessed by the Dice similarity coefficient. For each analysis, patients with missing neuropsychological measures were excluded (not more than 2 patients per variable).

Experiment 2: associations of covariates and lesion status

Here, we focussed on the four covariates that were controlled for in the previous experiment—age, sex, hypertension, and lesion volume. We explored if these variables and the voxel-wise lesion status are associated. For each voxel in the brain, the Spearman rank correlation coefficient between lesion status (0 - intact; 1 - lesioned) and the variable of interest was computed.

Experiment 3: impact of covariate control in a simulation with known ground truth

In this experiment, we aimed to evaluate covariate control in a perfectly transparent setting with a known ground truth, which allows us to evaluate the precision of brain mapping. We followed a simulation strategy that has been used previously to evaluate the validity and precision of lesion-deficit inference^25,35. The basic concept is accessibly described in the Supplementary. Importantly, such simulation allows us to evaluate the impact of control of real covariates with real imaging data in a well-known system.

In the present experiment, we randomly placed a sphere with a 1 cm radius in the tested brain space as a ground truth region. In the first condition, every patient whose lesion overlapped with this sphere was considered to suffer a deficit in the simulated variable. Hence, lesion location, independent of covariates, was the only cause for the simulated deficit. Patients with the deficit received a random normal score of 0 ± 1; patients without the deficit a score of 1.5 ± 1. The effect size was derived in piloting simulations to generate statistical maps with a size that neither extremely over- nor underestimates the size of the ground truth region. In the second condition, we simulated an impact of both damage to the ground truth region and the covariate. Again, scores of 0 ± 1 were simulated for patients with damage to the ground truth region, and else 1.5 ± 1. However, the mean value was also influenced by the relevant covariate. For sex or hypertension, the mean score was further decreased by 0.5 points, i.e., more pathological, for male patients or with hypertension. For age, we computed the z-standardisation and subtracted z(age)*0.5 points from the score, i.e., higher age led to a more pathological performance. We did not include the covariate lesion volume in the main analyses of the second condition of experiment 3. This was done because the causal relationship between lesion volume and a deficit is complex, with varying and partially strong associations between lesion volume and a deficit even without implicitly modelling such in a simulation³⁶. Explicitly including an effect of lesion volume would create a situation where the covariate is related to the target variable in various and not fully traceable ways. However, we still report such a simulation with the covariate lesion volume in the supplementary. For simplicity and interpretability, we always controlled only for one of the covariates. We repeated the simulation with 30 different ground truth spheres for each condition.

For the final analysis of the simulation experiment, we first compared several parameters between brain mapping with and without covariate control: total number of significant voxels, positive predictive value (true positives/[true positives+false positives]), number of false negatives, and the Dice similarity coefficient comparing the uncontrolled statistical results with the controlled results. Variables were compared for each covariate separately with Wilcoxon-signed-rank tests with Bonferroni correction for multiple comparisons. In the second step, we investigated the influence of covariate control at the topographic level. This analysis focussed on the Z-statistic in the general linear model that is mapped across the brain. For each simulation, we computed a ΔZ map by subtraction of the map without covariate control from the map with covariate control. Averaging this map across all simulations shows us if and how the impact of covariate control is specific to certain regions. Further, with this map, we examined the relationship between the change in the mapped statistic after covariate control and the voxel-wise associations of covariates and lesion status obtained in experiment 2.

Experiment 4: impact of covariate control under the null hypothesis

This experiment evaluated the impact of covariates under the null hypothesis, i.e., when associations between a behavioural measure and lesion location do not exist. We ensured that the null hypothesis was true by design using random data, which were generated based on the real neuropsychological data for visuoconstructive ability. For each simulation, we shuffled the original scores to create a measure with the distribution of a real neuropsychological measure, but for which no lesion-deficit association exists. We mapped lesion-deficit associations with BLDI and computed the extent of evidence for the null hypothesis H0 (i.e., voxels with Bayes factors ≤ 1/3³⁴ and for the alternative hypothesis H1 (i.e., voxels with Bayes factors ≥ 3). We performed 30 simulations of a behavioural measure and mapped each measure either without a covariate or one of the four potential covariates.

Results

We analysed data from 183 acute stroke patients. Information on sample demographics and study variables is shown in Supplementary Table 1. The correlation of neuropsychological measures and potential covariates varied (Table 1). Visuoconstructive ability correlated with all potential covariates, selective attention with age and lesion volume, and short-term memory only with lesion volume. The correlation strength varied from weak to strong (Spearman’s ρ = 0.55). We also found correlations between some of the covariates: sex and age correlated weakly, with men being younger, and sex and hypertension correlated strongly, with men suffering more often from hypertension. We performed statistical mapping for 31,206 voxels that were lesioned in at least 4 patients (see Fig. 1).

Table 1.

Correlations between neuropsychological measures and deficits.

Visuoconstructive ability	Age	Sex (1 = male)	Hypertension (1 = present)	Lesion volume
	− 0.307***	0.182*	0.152*	0.465***
Selective attention	− 0.181*	0.136	0.051	0.551***
Short-term memory	− 0.042	0.058	0.087	0.181*
Age	1	− 0.216**	− 0.067	− 0.095
Sex		1	0.523***	− 0.109
Hypertension			1	− 0.022
Lesion volume				1

Spearman’s rank correlation between study variables. Binary variables were coded by 0/1. Asteriks indicate significance at * p < 0.05; ** p < 0.01; *** p < 0.001.

Fig. 1.

Lesion topography. (A) Lesion overlap topography of the total sample of 183 patients. (B) The same topography thresholded at n ≥ 4, i.e. the area that was included in brain mapping analyses. Images are shown in neurological convention.

Experiment 1: impact of covariate control on real-world lesion-deficit inference

The impact of covariate control on real-world deficits varied and resulted in minor to substantial changes in the statistical results (Dice coefficient range between uncontrolled and controlled results: 0.003–0.76; Fig. 2). For visuoconstructive ability, the impact of control for age, sex, or hypertension was mostly quantitative. The clusters of voxels implicated without control were also present with covariate control, but, depending on the covariate and localisation, were enlarged or reduced in size (range Dice coefficients 0.63–0.76). Occasionally, this change in the extent of clusters meant that some brain regions were additionally or not anymore implicated. The results after control for lesion volume stood out with small clusters that did hardly overlap with the uncontrolled results (Dice coefficient 0.003). For selective attention, control for age, sex, or hypertension modified the extent of most clusters and even removed some small clusters entirely (range Dice coefficients 0.56–0.65). With control for lesion size, the analysis found almost no results anymore, except for a tiny, non-interpretable cluster of voxels in the white matter. Notably, after lesion size control, the frequentist mapping of both visuoconstructive ability and selective attention also found significant results that only appeared when testing the inverse association, i.e., the hypothesis that a lesion is linked to better and less pathological performance. For short-term memory, frequentist analysis generated null results under all conditions. We hence re-analysed the measure with Bayesian lesion-deficit inference (BLDI). The uncontrolled analysis found evidence for the null hypothesis in many brain regions, while evidence for the alternative hypothesis, i.e. lesion-deficit associations, was limited to inferior and middle temporal regions. Controlling for age, sex, or hypertension shifted the results toward the alternative hypothesis. The number of voxels showing evidence for the alternative hypothesis increased from 456 without covariate control to 1296, 1038, and 1181 voxels when controlling for age, sex, and hypertension, respectively. Evidence for the null hypothesis decreased and additional clusters showing lesion-deficit associations emerged, for example in frontal areas and the white matter. In contrast, controlling for lesion volume had the opposite effect: larger areas showing evidence for the null hypothesis were observed, while evidence for the alternative hypothesis decreased to only 83 voxels.

Fig. 2.

Results of Experiment 1. Lesion-deficit mapping of real-world post-stroke neuropsychological measures. For short-term memory, no statistically significant results were found with frequentist mapping under any condition and results of Bayesian mapping are shown instead. For frequentist analyses (left and middle column), Venn diagrams show the spatial overlap of the map without covariate control and each map with control, with the diagrams representing the total volume of significant results, according to which the area of each circle has been scaled. Numbers below show the Dice similarity coefficient. For Bayesian analyses, pie charts show the proportion of tested voxels within each evidence category. For analyses with the covariate lesion volume, frequentist analyses found areas with inversed lesion-deficit associations (i.e. areas that, when damaged, were associated with lower impairments), which are shown in blue.

Experiment 2: associations of covariates and lesion imaging features

For all variables that we controlled for in the previous experiment, we observed at least weak correlations (Spearman’s |ρ|≥0.1) with the voxel-wise lesion status in both the negative and positive direction (Fig. 3). The lowest peak correlations were found for sex (|ρ|=0.23) and the highest for lesion volume (ρ = 0.52). For age, sex, and hypertension, several clusters of voxels with at least weak correlations were found scattered across the brain. For lesion volume, correlations were higher, and the lesion status of almost all voxels in the brain correlated with lesion volume. In summary, the variables we included as covariates in Experiment 1 are associated with certain lesion locations, whether by chance or true associations between lesion anatomy and the variables.

Fig. 3.

Results of Experiment 2. Voxel-wise correlation between lesion status and the variables controlled for in other experiments. Covariates were found to be associated with the lesion status of several brain regions. The upper panels show all voxels with a Spearman’s rank correlation of at least |ρ| ≥ 0.1. Colour scales represent the range from |ρ|=0.1 to the maximum |ρ| scores. The lower panels show the distribution of ρ values. The voxel-wise lesion status was coded as intact (0) or lesioned (1), sex as female (0) or male (1), and hypertension as absent (0) or present (1).

Experiment 3: impact of covariate control in a simulation with known ground truth

The first condition of Experiment 3 evaluated the impact of covariate control, when no direct effect of the covariate on the target variable was present. Hence, only a direct causal impact of lesion location, but not the covariate was simulated. For the covariates age, sex, and hypertension, covariate control had no significant effect on any of the parameters—number of significant voxels, positive predictive value, number of false negatives, and Dice coefficient. With control for lesion volume, the number of significant voxels decreased strongly (Fig. 4A) from a median amount of 584 voxels without control to 11 voxels with control. The number of false negatives decreased accordingly, but the positive predictive value remained unchanged, while the Dice coefficient decreased. For detailed statistics, including the number of significant voxels, positive predictive value, number of false negatives, and Dice coefficients, as well as results of statistical tests, see Supplementary Table 2. In line with the absence of effects, the maps of the average statistical changes through covariate control were very small and close to zero, except for lesion volume control (Fig. 4B). The average statistical changes with lesion volume control correlated strongly with the map of lesion - lesion volume associations generated in Experiment 2 (ρ(31206)=-0.65; p < 0.0001), meaning that the effect of lesion volume control was region-specific and depended on the focal relationship between lesion status and lesion size.

Fig. 4.

Results of Experiment 3—condition with simulated sole impact of the lesion but not the covariate. Covariate control for lesion volume had a strong impact, while the impact of other covariates was negligible. (A) Example results for a single simulation run. (B) The average change in the voxel-wise Z-statistic after covariate control across 30 simulation runs. The indicated scaling of the values should be paid attention to; values were very small and indicated only a minimal impact of covariate control in this condition. The values were coded so that warm colours (red-yellow) show areas for which statistics are pushed towards suggesting a lesion-deficit association. To put these Z scores into reference, the average threshold to consider a statistical result significant in the uncontrolled analyses was |Z| = 4.15 ± 0.14.

The second condition of Experiment 3 evaluated the impact of covariate control when the covariate directly affects the target variable. Hence, in this simulation, both a direct causal impact of lesion location and the covariate were included. Here, the total number of significant voxels increased when any covariate was controlled for. The number of false negatives minimally, but significantly increased, but the positive predictive value was unchanged. The Dice coefficient indicated an improvement with the covariate age, but not for sex or hypertension. For detailed statistics, as well as results of statistical tests, see Supplementary Table 3). The maps of the average statistical changes through covariate control now included much larger values (Fig. 5C). For the covariate age, the average change in statistical maps ranged from ΔZ=− 1.06 to 1.65, which correlated almost perfectly (ρ(31206)=− 0.95; p < 0.0001) with the association map generated in experiment 2. For sex, the average change in statistical maps ranged from ΔZ=− 0.49 to 0.76, likewise with an almost perfect correlation with the map from experiment 2 (ρ(31206)=− 0.99; p < 0.0001). For hypertension, the average change in statistical maps ranged from ΔZ=− 0.46 to 0.92 and again correlated almost perfectly (ρ(31206)=− 0.99; p < 0.0001). To put these Z scores into reference, the permutation-based threshold |Z| score to consider a statistical result significant in the uncontrolled analyses without any impact of the covariate was Z = 4.15 ± 0.14. In an additional analysis, the impact of lesion volume under the same conditions has been tested (see Supplementary results).

Fig. 5.

Results of Experiment 3—condition with simulated impact of both lesion status and covariate. The impact of covariate control was found to be region-specific. (A , B) Example results for a single simulation run with different simulated lesion location. (C) The average change in the voxel-wise Z-statistic after covariate control across 30 simulation runs. The indicated scaling of the values should be paid attention to. The values were coded so that warm colours (red-yellow) show areas for which statistics are pushed towards suggesting a lesion-deficit association.

In summary, (i) covariate control changed statistical results, but not systematically for the better, and (ii) lesion-covariate associations as shown in Experiment 2 (Fig. 3) closely described the average effect of covariate control in lesion-deficit mapping. In brain regions for which a covariate was anticorrelated with the lesion status, covariate control shifted results towards assuming a lesion-deficit association. Vice versa, in regions for which the covariate was correlated with the lesion status, covariate control shifted results towards the null hypothesis.

Experiment 4: impact of covariate control under the null hypothesis

Experiment 4 evaluated the impact of covariate control if the null hypothesis is true, i.e. if no association between lesion location and target variable exists. In data for which the null hypothesis is true by definition, we mapped lesion-deficit associations by BLDI. Under this condition, Bayes factors should be as small as possible and smaller than 1/3, i.e., should provide at least moderate evidence in favour of the null hypothesis H0 over the alternative hypothesis H1 (following existing conventions³⁴.

Accordingly, the majority of Bayes factors were smaller than 1/3 across all simulations and independent of any covariate (Fig. 6A). However, the inclusion of a covariate generally increased Bayes factors, meaning that they shifted evidence from being in favour of H0 towards being in favour of H1: the number of voxels with at least moderate evidence for H0 decreased under any covariate control (Bonferroni-corrected Wilcoxon-signed-rank tests, all p < 0.0001). The number of voxels for which Bayes factors incorrectly indicated at least moderate evidence in favour of H1 (Bayes factors > 3) increased (all corrected p < 0.0045). Notably, lesion volume stood out among all covariates with the nominally highest impact.

Fig. 6.

Results of Experiment 4. Covariate control falsely decreased the number of voxels indicating evidence in favour of the null hypothesis. (A) Results of the simulation experiment under the null hypothesis. Each dot stands for one simulation, i.e. the statistical mapping of one simulated score. Bars show the median. Asterisks indicate a significant difference after Bonferroni correction. The total number of analysed voxels was ~ 31,200. (B) Example results for a single simulation run.

Discussion

In multiple experiments, we evaluated the impact and adequacy of covariate control when mapping diseased brains in lesion-deficit mapping. In Experiment 1, we found that the effects of control for age, sex, presence of hypertension, and stroke lesion volume varied by covariate and post-stroke deficit. While in some cases the effects were only minor, in other cases, they were more substantial and suggested partially different neural correlates. This leads us to the obvious question of which result—if any—corresponds to the truth. Without a precise rule for the selection of appropriate covariates, we face the serious epistemological problem that numerous contradictory results can exist. In Experiment 2, we have shown that the covariates are associated with the imaging data of brain pathology. Hence, in real data, associations may indeed be as complex as indicated in the introduction. Brain imaging variables and secondary variables that we control should not generally be seen as two factors that independently impact the genesis of behaviour or pathology, but as closely and closely and intricately intertwined.

We have shown the consequences of this complexity in the next experiments. In Experiment 3, we evaluated the impact of covariate control in a well-controlled artificial setting. Covariate control had little to no impact when our simulations did not include a direct causal relationship of the covariate on the target variable. With such a causal relationship included, covariate control had an impact. Of note, we believe that such a direct causal effect likely represents the most common idea that researchers have about the impact of a covariate variable on behaviour or pathology. And, arguably, such an effect alone—a simple linear effect independent of any imaging feature and without any interactions—should provide a relatively simple benchmark of what covariate control may accomplish. However, even though covariate control affected statistical results, we found no systematic improvement. Instead, what we found to be systematic was that covariate control changed statistical results following inverse associations of covariates and lesion imaging features uncovered in Experiment 2. In other words: covariate control impacted results, but instead of improving them, it rather invertedly imposed the covariate’s association with the lesion anatomy onto statistical results. A researcher who, for example, controls stroke lesion-deficit mapping for the variable sex might do so to “control for the effect of sex”. This does not necessarily mean that the analysis better captures the neural correlates of a deficit. Instead, given the associations we found between sex and lesion status in Experiment 2, it means that we effectively amplify statistical signal in frontal and temporal areas and attenuate signal in parietal areas. In Experiment 4, we found that covariate control can work against a confirmed null effect. In other words, in the absence of a true association between imaging features and a target variable, covariate control can erroneously shift results towards the alternative hypothesis and introduce spurious results. Hence, at least in the context of Bayesian statistics, the selection of inappropriate covariates can thus create statistical effects out of nothing.

In our experience, covariate control in neuroimaging studies often adheres to established conventions rather than requiring extensive justification for each analysis. Conversely, deviations from commonly used covariate sets, such as the omission of a control variable that is usually included, may be criticised during peer review without consideration of its appropriateness. Apart from conventions, covariates can often be included arbitrarily and in large numbers. Given the broader incentive structures in academia, this asymmetry may create challenges for scientific progress. In particular, the choice of covariates can introduce analytical flexibility that may influence study outcomes, sometimes unintentionally. While related concerns have been discussed for many years¹¹, they may become especially pronounced in large-scale datasets such as in clinical neuroimaging. Such datasets are characterized not only by a large number of imaging-derived variables (e.g., voxel-wise features), but also by extensive clinical documentation that provides numerous potential covariates.

Among the four potential covariates that we evaluated in the current study, lesion volume stood out. The distinctive feature of this variable could be the high correlation with imaging data. Experiment 2 found correlations ranging up to strong levels between lesion volume and voxel-wise lesion status for almost the entire brain. Another feature could be the close and not clearly comprehensible coupling of the two variables: both lesion volume and lesion location are inseparably intertwined and it appears impossible to attribute the direct cause of a deficit to lesion volume alone³⁶. Covariates for which a causal relationship with other variables cannot be clearly defined are often encountered in neuroscience. We probably cannot clearly define a causal relationship between one personality trait and another. Then what is the neural correlate of a personality trait controlled for several other personality traits¹⁰? What is a subscore in a stroke severity assessment battery controlled for the total test score³⁷? What is focal cortical thickness controlled for average brain-wide cortical thickness³⁸? Even if such covariates situationally improve statistical analysis, we still face the issue that causal relationships become even more difficult to grasp than they already are in large-scale neurobiological data. And whatever covariance control does will only become more complex and more difficult to comprehend and to communicate.

What variables should we control for in biological or neurological large-scale data? Our current work is unable to provide guidelines to answer this question. Hence, a first step to address the challenges identified in the current study is to make researchers aware that covariate control is not a gold standard solution, but that it carries potential dangers. The strategy to identify covariates just based on the correlation with a target variable is flawed, which is already well-known in the literature on causal inference^14,16. The field of causal inference provides well-developed guidance on the selection of appropriate covariates by various concepts that describe assumed causal relationships between variables, such as confounders, mediators, and colliders, as well as more high-level concepts such as the back-door criterion¹⁴. However, as explained above, high-dimensional data further complicate the situation. The true causal structure may be difficult to specify or justify in high-dimensional imaging data, or they may also vary across different imaging variables. While there may be no perfect solution here, we propose several practical strategies that could improve research methodology. First, methodological decisions about covariate control should be justified. While such justifications may be controversial, they at least offer the opportunity to transparently evaluate and discuss methodological decisions. Second, simplified representations of causal relationships may be used to guide methodological decisions. Although these representations will rarely capture the full complexity of high-dimensional imaging data, they can serve as a useful heuristic for identifying plausible confounders and avoiding problematic adjustments. Third, both controlled and uncontrolled results should be reported, and marked discrepancies may be discussed.

Our data raise further crucial questions: Even if covariate control is appropriate, it might be difficult to select among strongly correlating potential covariates, such as age and hypertension in the current study. In this case, both variables could also be controlled, which leads us to the main limitation of the current study: we aimed to keep our simulations and analyses as simple as possible. We limited simulations to simple effects and only ever controlled for one covariate at once. And yet the complexity of the data was sufficient to produce undesirable changes in the results. But what if we step up the complexity and control for a dozen covariates, as done in actual studies¹? The issues identified in our study will probably not only persist in more complex situations but be severely amplified. Likewise, our simulations only looked at relatively simple causal relationships between a covariate and a target variable. Our study does not allow any strong conclusions to be drawn on what covariate control does in the face of complex interactions between covariates and imaging variables. Here too, we assume that the problems mentioned will only be exacerbated.

Our study has further, more general limitations. Firstly, the impact of covariate control in lesion-deficit mapping is likely to vary depending on the sample parameters, such as the distribution of the covariate itself and the lesion distribution. Stroke samples often vary greatly in terms of population parameters, and hence the impact of covariate control may differ between samples. For example, the current study was characterized by comparatively mild strokes on average as a result of high rates of timely reperfusion therapy²⁶, hence, the impact of control may differ quantitatively in a sample with more severe strokes. Secondly, there is not just one valid approach to controlling for a covariate. While the current approach uses established methods, others exist. For example, there are specific methods for controlling for lesion volume in lesion-deficit mapping⁷. In general, various approaches to covariate control exist. Most commonly, covariates of no interest are included in statistical models. Other approaches include, e.g., residualisation or stratification. Our study cannot determine whether our findings apply equally to all types of covariate control. Thirdly, the accuracy of statistical results was generally low in our simulation experiments with ground truth regions. However, this is a generally known issue of lesion-deficit mapping, where statistical results are to some degree misplaced along the brain vasculature²⁵.

In conclusion, the widespread use of covariate control in the statistical analysis of clinical brain imaging data—and, likely, other biological high-dimensional data as well—may not generally improve statistical results, but may change them in partially unintentional ways. Therefore, covariate control constitutes a problematic degree of freedom in the analysis of brain imaging data. The current modus operandi of selecting statistical covariates by correlation with the target variable, according to conventions, or plainly by availability, needs revision. However, given the high complexity of neurobiological data, the creation of objective rules to identify appropriate covariates might be difficult. In any case, covariate control should be limited, and uncontrolled statistical results should also be considered in the interpretation of a study.

Supplementary Information

Below is the link to the electronic supplementary material.

Abbreviation

BLDI

Bayesian lesion-deficit inference

Author contributions

C.S.: Conceptualization, Methodology, Formal analysis, Writing—Original Draft. L.G.: Investigation, Data Curation, Writing—Review & Editing. M.A.: Resources, Investigation. R.U.: Investigation, Data Curation, Resources, Supervision, Writing—Review & Editing.

Funding

This work is funded by the Dementia Research Switzerland – Synapsis Foundation (Grant number 2019-PI05), the Heidi Seiler Foundation, and the Swiss National Science Foundation (SNSF, project number 10003589).

Data availability

Statistical maps and scripts are available at https://data.mendeley.com/datasets/gvn3k8cj9r/1. Original clinical data are not publicly available, but qualified researchers may request access to anonymised data. Proposals need to be approved by the local ethics commission.

Declarations

Competing interests

The authors declare no competing interests.