Using Precision Approaches to Improve Brain-Behavior Prediction

Abstract

Predicting individual behavioral traits from brain idiosyncrasies has broad practical implications, yet predictions vary widely. This constraint may be driven by a combination of signal and noise in both brain and behavioral variables. Here, we expand on this idea, highlighting the potential of extended sampling ‘precision’ studies. First, we discuss their relevance to improving the reliability of individualized estimates by minimizing measurement noise. Second, we review how targeted within-subject experiments, when combined with individualized analysis or modeling frameworks, can maximize signal. These improvements in signal-to-noise facilitated by precision designs can help boost prediction studies. We close by discussing the integration of precision approaches with large-sample consortia studies to leverage the advantages of both.

Expanding Sample Size to Reliably Predict Behavior from Brain Measures

Elucidating links between the brain and behavioral measures can unravel the neural underpinnings of behavioral phenotypes. One approach in this domain has been to focus on how variations across people in brain structure or function link to individual differences in cognitive or mental health metrics, an approach coined as “brain-wide association studies” (BWAS [1]) (Figure 1A). BWAS, especially those that aim to predict outcomes in unseen data using machine learning and cross-validation (Figure 1C), offer a powerful tool for using brain imaging to make diagnoses [2–5] and prognoses of psychiatric disorders [6] and to predict future performance and outcomes in education and human resources [6–9].

Figure 1. Linking brain features to behavioral variables.

A) Associations between brain and behavior can be measured across individuals, an approach referred to as BWAS. As the example suggests, individual differences in brain organization may be correlated to variation in personality, skills, cognition, psychopathology, or demographic information. B) Links between brain and behavior can also be examined at the individual level through within-subject designs, for example, by measuring relationships as a single participant undergoes multiple conditions. Notice the subtle changes in brain network assignments (indicated by black arrows) across experimental conditions. C) One powerful approach to linking brain and behavior uses prediction models. In predictive modeling approaches, a subset of data (i.e., training data) is used to build a statistical model to estimate a behavioral variable from brain features. Machine learning techniques are often used to build models from multiple brain features simultaneously (e.g., whole-brain functional connectivity) to account for the multivariate nature of brain function. The model then makes predictions from independent data not used in training (i.e., test data). The successes and generalizability of the model are evaluated by assessing the similarity between the predicted and the observed measures of the test data. While many studies may share these common procedures, details in methodological decisions differ as much as variations in prediction performance (for an in-depth discussion, we refer the reader to the following works that examine different methodological factors to optimize prediction performance: Brain parcellation [51,52,53,17,50], functional connectivity [112,53,17,50], and machine learning classifiers [19,59,17,50].

However, in the past decade, increasing attention has been drawn to issues of poor replicability in BWAS, which has limited the practicality of BWAS and cognitive neuroscience more broadly to achieve translational goals. One major issue with past work has been the historical reliance on small sample studies [10–12]. To address these issues, there has been a growth in consortium datasets with large numbers of participants, including the Human Connectome Project (HCP [13]), the Adolescent Brain Cognitive Development study (ABCD [14]), and the UK Biobank [15] which collectively gather data from thousands to tens of thousands of participants. These studies have been instrumental in demonstrating that replicable BWAS results, when using measures common to the field, primarily consist of small effect sizes (univariate effect sizes ranging from 0 to 0.16 at the maximum [1,16]). Given this finding, standard small sample studies will be dramatically underpowered, limiting their ability to identify replicable effects [1]. However, even with more powerful multivariate prediction approaches using consortium datasets, prediction accuracy varies across measures, with demographic information, such as age (r ≈ 0.58 [17]), often more strongly predicted than cognitive task performance (Vocabulary [picture matching] ≈ 0.39 [18–19]), which in turn exhibits stronger prediction than self-reported surveys (Openness [NEO] ≈ 0.26). Notable variation also exists across cognitive tasks, such as between crystallized knowledge (Vocabulary task above) and inhibitory control (flanker task, which shows prediction below r = 0.1 [18]). This is concerning because inhibitory control and self-reported clinical symptom measures are among the most relevant to translational applications, such as guiding diagnosis or treatment plans for individuals.

Our goal here is to review factors that limit current BWAS, particularly focusing on variables of high clinical interest that exhibit low prediction performance. We highlight two potential constraints: large measurement noise and small signal. BWAS using consortium datasets and the typical smaller sample studies that have historically been present in the field often suffer from issues in gathering sufficient behavioral and fMRI data per individual. With insufficient data, it is difficult to accurately and reliably characterize individuals [20]. One solution is to focus further on improving individual-level estimates [20–23], such as through precision approaches. Precision approaches (also referred to as “deep”, “dense”, or “high sampling”) refer to a class of methods that collect extensive per-participant data, often across multiple contexts and days, with careful attention in analysis to alignment, bias, and sources of variability [24–26]. While many precision studies are often based on small participant numbers, they are notably distinct from typical small-sample studies, with their focus on substantially expanded quantities of data to enhance the reliability, validity, and analysis potential of individual participant measures [24–26]. We highlight these issues through a review of recent papers with a particular focus on inhibitory control and fMRI measures of brain networks. These studies show that obtaining precise individual measures of behavior [27] and brain function [20,21,24] can reduce measurement noise and increase the size of BWAS effects, especially when united with targeted experimental manipulations and modeling frameworks. We close by weighing the relative merits of consortium and precision-design studies and discuss opportunities for uniting these two approaches to capitalize on the advantages of each.

Successes and Limitations of Current BWAS

Consortia have offered an opportunity to assess the successes and limitations of BWAS (Figure 2). For example, functional brain measures generally lead to better predictions of behavior than structural measures [1,28]. Task fMRI also leads to better predictions than task-free resting-state functional connectivity [29–32] (note, however, that the strengthening of correlation with task fMRI may be linked to task-specific effects in the behavioral variables, confounding the association [1,33]). What seems particularly effective in improving prediction is the use of multivariate machine learning approaches that combine information from a range of brain features [1,32,34–36].

Figure 2. BWAS approaches leading to more successful prediction.

A) Functional measures, such as resting-state functional connectivity between time-series of distinct brain regions, yield better predictions than anatomical measures, such as cortical thickness [1]. B) Multivariate analyses generally provide better predictions of behavior with lower chances of false negatives than univariate analyses; multivariate analyses integrate brain signals by assigning weights to each parcel, whereas univariate approaches examine associations for voxels or parcels individually. C) Cognitive measures, based on computerized tests quantified via reaction time or accuracy, are typically better predicted than questionnaires collecting self-report evaluations of a measure. D) Research employing individualized approaches, for example for the definition of brain regions, improves brain-behavior predictions relative to more standardized approaches, such as those based on group parcellations. As we will review in the remaining portions of the paper, ‘precision’ approaches with individual-focused analyses garnered through deep phenotyping and repeated measurements have substantial promise in furthering our understanding of brain-behavior associations.

BWAS using multivariate prediction approaches have generally shown that cognitive test scores are better predicted than self-report questionnaires [1,18,28,37]. Interestingly, however, there is variation among cognitive tasks in their predictive performance: measures of crystallized intelligence, such as vocabulary (picture matching) and reading (pronunciation) tests, typically exhibit the highest predictions among the 58 behavioral measures from the HCP dataset, whereas inhibitory control (flanker task) displays one of the poorest [18,38]. As we will explore in more detail later, some of this variation may be attributable to task designs that limit the precision of individual cognitive measures. For instance, improving task designs to more accurately assess individual cognitive abilities and extending task duration (from less than five minutes to over 60 minutes) can facilitate predictive performance [39].

Finally, prediction is better with individualized approaches. The structural organization and functional connectivity of the brain vary uniquely across individuals [20,40–43]. Thus, rather than assuming group-level correspondence, modeling individual-specific patterns of brain organization can yield more precise measures and facilitate behavioral predictions. For instance, ‘hyper-aligning’ fine-grained features of functional connectivity markedly improved the prediction of general intelligence compared to typical region-based approaches [44]. HCP behavioral phenotypes were better predicted when functional connectivity was derived from individual-specific parcellations instead of group-level parcellations [18]. Removing common neural signals across individuals [45] or global artifacts across the brain [37] has also been suggested to facilitate individual-specific mappings of the brain. With mounting evidence that predictive models may fail to generalize across sex, age, or ethnicity [46–49], transitioning to individually powered analyses presents a promising approach, where distinct models can be tailored to subsets within the population.

Overall, recent analytic advancement has been successful in improving prediction performance [17,50–53]. However, even when pipelines are optimized, some variables may still suffer from close to zero prediction accuracy. Inhibitory control serves as a primary example [18]. Inhibitory control involves inhibiting distractions and resolving interference, and deficits in this ability have been related to various psychiatric disorders (e.g., depression [54]). Inhibitory control is assessed through tasks like the flanker or Stroop task, which gauge an individual’s ability to respond to a target in the face of congruent or incongruent distractors. However, flanker task performance shows the lowest BWAS prediction accuracy in the HCP data [18], likely due to measurement error: inhibitory control measures often exhibit high trial-level variability, resulting in noisy estimates when based on only few trials (e.g., 40 in the HCP data) [55]. Thus, improvements may be seen by attending to precision and statistical power in these measures.

Improving Prediction using Precision Approaches

In a prior editorial, we suggested two major paths to improve searches for brain-behavior associations by using experimental approaches that maximize the signal and minimize noise [21]. Here, we expand on these ideas, reviewing recent evidence that precision approaches can increase statistical power by: (1) strengthening the reliability of measures to minimize noise and (2) improving the validity of measures to maximize signal (Figure 3, Key Figure).

Figure 3. Maximizing signal-to-noise to enhance the detection of robust BWAS effects.

Single dots represent individual participants, positioned based on estimates of their brain features and behavior. The gray dashed line represents a perfect association between the two measures, and the blue solid line represents the strength of the relationship estimated from a given sample. The closer the blue line is to the gray line, the stronger the BWAS effect. A) The first panel shows a cartoon of current brain-behavior associations in many studies: individual measures are noisy (large error bars of the dots), leading effect sizes to be imprecise (large error bars of the line) and small (low slope). In B and C, we display how precision approaches may modify these relationships and enhance the detection of effects in BWAS. B) First, precision approaches allow researchers to strengthen reliability by minimizing noise. Reducing measurement error in brain or behavioral measures yields more precise individual-level estimates (reducing the error bars of the dots). This allows more precise detection of the brain-behavior association (reducing the error bars of the line). Minimizing noise also increases the maximum possible associations between two measures (higher line slope), because measurement error attenuates correlations [55,65] (see also Fig. 4C). C) Second, precision approaches can enhance BWAS signal by improving the validity of measures. For example, the repeated testing design of precision approaches lend themselves well to controlled within-subject experiments and modeling analyses [77,78], which can both improve the validity of underlying measures. With more valid, higher signal measures, the strength of BWAS effects becomes larger and easier to detect (the blue line better overlapping with the gray line). Note that without sufficient sampling to reduce noise, however, these manipulations may still have high imprecision (large error bars of the line) and an inability to estimate individual-level results very accurately (large error bars of the dots).

Minimize Noise

Considerable literature has emphasized the importance of increasing participant numbers [12,56–57] to enhance the reproducibility of BWAS [1,58–59]. However, recent research has highlighted that the amount of data collected from each participant is equally crucial [20–27]. This is particularly relevant given the trade-off between increasing participant numbers and lengthening testing duration [60].

For individual-level precision, more than 20–30 minutes of fMRI data is required [20,61]. Further, extending the duration of cognitive tasks (e.g., from five-minute to 60 minutes for fluid intelligence tests) can improve predictive abilities [39]. These findings suggest that BWAS may benefit from longer data acquisition. Without sufficient testing, individual-level measures have large measurement errors. This noise affects measures of both within- and between-subject variability, and, if uncontrolled, they can be confounded. Hence, controlling for high individual-level noise is critical as (1) the noise makes it hard to reliably estimate individual-level effects, which are often the target of BWAS and (2) it makes estimates of between-subject variability inaccurate (both biasing measures and, downstream, leading to a potential focus on the wrong measures). These jointly will fundamentally distort BWAS efforts, as noise attenuates the correlation between measures [55] and diminishes the prediction accuracy of machine learning classification algorithms [62].

To illustrate these issues, one can consider the case of experimental measures of inhibitory control, introduced earlier. Our group recently investigated the reliability of inhibitory control tasks using a precision behavioral dataset [27]. We took an extensive repeated measures approach, collecting more than 5,000 trials for each participant in four different inhibitory control paradigms across 36 days of testing. We examined how the precision of individual estimates of inhibitory control influenced estimates of within- and between-subject variability (Figure 4).

Figure 4. Importance of minimizing individual-level noise.

This schematic depicts findings from our recent study on precision approaches for measuring inhibitory control [27]. However, these results likely apply to a broader range of brain and behavioral measures. A) Small numbers of trials yield high within-subject variability due to measurement error. For individual-level precision to reach a stabilizing point, such that within-subject variability is small and collecting more trials becomes redundant, over 1,000 trials need to be collected for each participant. In contrast, the NIH Toolbox, commonly used in consortia and smaller sample datasets in the field, estimates inhibitory control in the flanker task across 40 trials. B) High within-subject variability inflates between-subject variability estimates. Not only do small trial numbers impair individual-level estimation, but they also bias estimates of individual differences relevant to BWAS. C) A consequence of these effects is that individual-level noise attenuates correlations. Our simulation (replicating [55]) shows the effects of increasing within-subject variability in brain and behavioral measures on correlations (see Text box 1). The dots represent simulated subjects, and the error bars indicate within-subject variability. Larger within-subject variability is associated with larger variability in the estimated brain-behavior relationship. Further, as within-subject variability inflates the estimated between-subject variability (Fig. 4B)—and since the denominator in the correlation coefficient calculation is the standard deviation of the two correlating variables— correlations can be attenuated even when true correlations are high. D) To avoid issues depicted in A-C, increasing the precision of individual-level estimates with longer testing can be effective. This heatmap (using data from [27] and [79]) shows the consistency between the true and observed inhibitory control estimates across people, calculated with ICC, when varying the numbers of trials and participants. The ICC is high (0.89) for precision approaches that collect large numbers of trials even from small numbers of participants, while the ICC is low for studies based on smaller trial numbers, both for consortia (0.38) and standard small N designs (0.26).

Individual-level estimates of inhibitory control vary widely with short amounts of testing (Figure 4A). This variability can be mitigated by collecting more extensive data from each participant. Less intuitively, insufficient per-participant data can also bias between-subject variability [27,55,63–64] as high within-subject variability inflates estimates of between-subject variability (Figure 4B). This is problematic in BWAS because inflated between-subject variability attenuates the correlation between behavioral and brain measures (Figure 4C; see also [55,65]). The same applies to brain-behavior predictions using machine learning, as measurement error in behavioral variables attenuates prediction performance [62].

Notably, these issues caused by high within-subject variability can be resolved by increasing trial numbers, but not by increasing participant numbers [27]. Whereas extended precision approaches (>1,000 trials per participant) yielded high consistency between true and observed estimates of inhibitory control (Intraclass Correlation Coefficient: ICC=0.89), the consortium approach of collecting thousands of participants with limited individual-level data still resulted in relatively low consistency (ICC=0.38) (Figure 4D). The number of participants is indeed important in BWAS [60], as it improves generalization and reduces biases from overfitting on small participant numbers [66]. However, collecting sufficient per-participant data is critical especially for measures with high within-subject variability, because if an estimate has been contaminated by within-subject variability, collecting more participants cannot resolve the issue. Accordingly, the precision of individual-level data is crucial for estimating how measures differ across individuals.

Indeed, a number of studies have pointed to limitations in reliably measuring individual-level fMRI data with small amounts of data [20,22,24]. With only 10 minutes of data or less, the reliability of various brain-network measures from both resting-state [20] and task-based fMRI [61] was poor. For example, in 25 studies using fMRI-based functional connectivity, the average ICC was 0.29 [67].

With more individual-level data, fMRI reliability and replicability can markedly improve [20,68–69]. Detailed individual-level estimates of functional and structural brain connectivity were characterized [68] using data accumulated over more than 80 sessions of testing in an individual [70]. Similar approaches may also improve the reliability of brain morphometry measures [71]. Characteristics of individual brains from 10 participants demonstrated that resting-state functional connectivity and task fMRI become highly reliable with more data [20]. The exact amount of time needed for precise estimates will likely differ based on the brain region (e.g., cortical vs. subcortical [72–74]), measurement (e.g., connectome, graph metric [20]), and fMRI pulse sequence characteristics (e.g., single echo vs. multi-echo [75]). However, for common measures of cortical connectomes, more than 30 minutes of low motion data are needed to achieve test-retest reliability of r>0.8.

In conclusion, individual-level precision in both fMRI and behavior requires more data than what is commonly acquired, and this lack of precision will fundamentally distort between-subject variability estimates. For both reasons, a lack of precision limits the ability to detect accurate and reproducible effects in BWAS, regardless of participant number.

Maximize Signal

The second way to increase statistical power of BWAS is to maximize signal. Most currently replicable BWAS effects are small [1,76], and one consideration is whether the behavioral or brain measures may have limited validity in measuring constructs that are related to one another. The measures used in large consortium and other BWAS may be selected because they are common in the field and relatively robust, but not necessarily because they have precise links to brain function. Therefore, to improve BWAS, one possibility is to search for cognitive/behavioral features that are related to particular aspects of brain function, and vice versa.

Consider the case of current consortia, which often take general measures of personality, mental health, or cognition. BWAS then relate these to measures of brain function. This is akin to asking, for the visual system, whether someone has difficulty seeing with the aid of corrective lenses. While this metric may be robust and provide a signal for the need for additional care, it is too broad to link with a specific neurobiological phenomenon. Instead, a series of diagnostic tests (e.g., comparing sight from the two eyes, determining contrast, color, and motion sensitivity, etc.) is needed to produce a clearer picture of the underlying biological deficit. Without this step, it would be hard to determine strong links between specific brain features and visual deficits. Likewise, it is likely that more controlled measures are needed to improve our search for specific links between the brain and a range of cognitive and clinical conditions.

We highlight three approaches that could increase the validity of BWAS measures: (a) controlled within-subject experiments, (b) individually focused measures, and (c) neurobiologically informed computational models. Precision approaches are well suited to implementing these approaches, because of their extended, repeated sampling, which can incorporate multiple experimental conditions within the same participant [20,27,70,77], their inherent focus on individual measures [20,70], and their ability to extract data from multiple conditions to facilitate computational modeling [78,27]. Below, we provide examples to support the efficacy of such designs.

Experimental manipulations isolate a process of interest to improve links with specific neural measures. For example, reaction time is a common behavioral measure in experimental tasks, but it is influenced by a range of neural processes, from sensory perception to motor execution. Therefore, while overall reaction time can be reliable, measuring an individual’s reaction time in a single experimental condition (e.g., incongruent trial response in the flanker task) may not isolate a targeted process (e.g., inhibitory control). Instead, contrasting two conditions within individuals can increase process specificity (e.g., subtracting performance on congruent from incongruent trials to measure inhibitory control).

These subtraction-based experimental paradigms, however, increase measurement error [79–81], reducing their use in studies of individual differences [63,55]. As we reviewed in the prior section, this issue may be resolved straightforwardly by increasing trial numbers per participant [27]. Other experimental manipulation approaches may also help, such as increasing the salience of distracting information or combining tasks to enhance the underlying process (e.g., a combined Stroop and Simon task) [82]. These manipulations isolate the process of interest and help identify its association with brain measures.

Controlled experiments are also beneficial in fMRI. Several studies have suggested that task fMRI can produce larger BWAS effects than resting-state [29–31,83], possibly because they amplify underlying processes linked to behavioral measures [29] (but see [33,1]). Beyond BWAS-style cross-subject associations, within-subject brain-behavior links based on task fMRI have been extensively replicated without the need for thousands of participants; these include task contrasts probing specific sensory/motor mapping (retinotopic maps in visual cortex) [84], somatotopic maps in motor cortex [85], attentional enhancement of visual responses [86–87], deactivations in the default mode network [88–89], and activity related to working memory load [90–91]. Precision fMRI studies with resting-state have also shown evidence of large brain-behavior links using within-subject designs. For example, comparing before, during, and after applying a cast to the dominant arm demonstrated dramatic and replicable alteration in somatomotor functional connectivity [77].

These results suggest that focusing on individual-level measures in within-subject designs enhances the specificity, and thereby the signal and validity, of BWAS. Recent studies have underscored the high variation in the layout of brain regions and networks even among healthy neurotypical adults [83,42,40,20,92]. Thus, targeting individual-specific brain locations, such as through functional localizers, can uncover neural features that might be overlooked in group-averaged analyses [20,92]. For example, individually defined motor networks more accurately map motor activation than group network maps [20]. Individually defined default mode sub-networks can dissociate task responses related to episodic projection and theory of mind [93] in a manner not seen in group averages. Individual-specific localizers have also successfully identified adjacent but distinct regions for language [94] and multiple-demand in the frontal cortex [95]. These individual-focused approaches can initially identify neural responses related to behavioral variables within individuals and further expand to support comparisons across individuals.

A prior study from our lab demonstrated that individual-focused precision approaches also improve the prediction accuracy of fMRI machine learning [96]. Using a precision dataset, different task states were predicted with fMRI functional connectivity. Models were trained on data from a single participant and tested on independent data from either the same or different participants. The models successfully predicted task states of new participants, performing equivalently to more classic leave-one-subject out machine learning methods. However, performance showed significant improvement (~30% accuracy increase) in within-subject classification. Similar results have been demonstrated in the domain of evoked task fMRI responses [97]. These findings indicate that machine learning prediction studies can benefit from approaches that learn individual-specific characteristics, potentially unveiling detailed idiosyncratic features within individuals that are necessary to meaningfully link brain and behavior.

Our final suggestion is to expand the use of neurobiologically meaningful modeling. As mentioned, behavioral metrics reflect a series of processing steps ranging from sensory perception to motor execution. Modeling approaches are useful in this context to establish quantitative links between observed outcomes and unobserved variables critical to experimental manipulations (e.g., inhibitory control in the flanker task). For example, drift rate in the diffusion decision model can quantify the quality of processed information [98]. Thus, rather than using reaction time alone, drift rate can help isolate the process of interest [99–100]. Modeling latent experimental factors across multiple tasks, such as with factor analysis, can also help extract key processes assumed to be shared across related tasks (e.g., inhibitory control in Stroop and stop-signal tasks [101]) and to separate variance of interest from error variance. Notably, our study [27] demonstrated that for these modeling approaches, as with other analysis methods (e.g., Bayesian hierarchical modeling [63]), precision measures are important to avoid erroneous or misleading modeling results.

Studies seeking to unite cognitive neuroscience findings with artificial intelligence approaches also demonstrate benefits when using precision designs [78]. For example, a precision fMRI dataset was collected from eight participants over the course of a year as they viewed over 9,000 natural scenes, known as the ‘Natural Scenes Dataset’. This dataset was used to power a deep neural network that predicted brain data with improved accuracy relative to other computer vision modeling approaches. Individuals also showed a much broader range of brain-based recognition memory effects to repeated presentations of scenes, relative to group averages.

Indeed, each of our suggestions to strengthen effects in BWAS benefits from extensive individual-level data. Precision datasets lend themselves naturally to collecting controlled experimental tasks, to individualized designs, and to advanced modeling methods based on multiple task measures. In each case, attention to the reliability of individual measures increases the robustness of results. These advantages, while notable, do not come without their limitations --- for example, the possibility for habituation or learning effects, as well as the relatively restricted sample sizes that are often practically possible in precision fMRI studies (see Outstanding Questions). We are optimistic that as precision studies continue to grow in popularity in the field, these limitations can be addressed and overcome. For example, precision fMRI studies, while susceptible to learning effects, also offer the opportunity to study these temporal phenomena in greater depth [102] (phenomena that we note are likely relevant to real-life cognition). One particularly promising avenue for addressing these limitations is to combine precision methods with the broader, but lower signal-to-noise ratio measures, obtained from consortium datasets.

Joining the Two Paths toward Reliability

In our prior editorial [21], we suggested that experimenters could choose between two paths to reliably correlate brain and behavior. One is to conduct consortium studies with large numbers of participants. Having many participants is necessary to measure population-level variation with good replicability and to enhance the generalizability beyond specific samples [56–57,1,70]. With typical small sample sizes (well below 100), replicability estimates were modest for task-based [56] and resting-state fMRI [1]. Statistical power issues in small-sample studies have only become evident with increasing participant numbers [1]. However, the challenge is the high cost and low flexibility of such designs. It is less likely for researchers to adopt innovative tasks and measures, such as those discussed in the previous section, in consortium studies. Furthermore, to secure many participants in various measures, individual-level precision is often compromised by allotting a short duration for each measure (e.g., the UK Biobank collects ~6 minutes of resting-state data).

The other path involves conducting smaller but focused studies with precision approaches. The key lies not in focusing on smaller samples but collecting extensive per-participant data across multiple conditions. Without sufficient individual data, one likely encounters the same power issues as those seen in generic small-sample studies. We discussed in this review how precision approaches improve BWAS that focus on measures with large measurement error. Furthermore, as consortium studies generally center on measures that have been extensively tested, focused studies are needed to examine novel task paradigms that can improve the validity of designs. Focused studies also facilitate within-subject investigations of brain-behavior associations across time or states and provide high-quality data for advanced modeling methods. However, smaller precision approaches cannot fully replace large consortium approaches, as they will continue to need methods to generalize to larger samples (see Outstanding Questions). Given these observations, it is natural to consider how consortium and focused studies can be jointly adopted.

One strategy is to start with focused studies to identify brain and behavioral associations within a small sample of participants and test their replicability and generalizability in larger samples. Precision datasets can provide a means to identify features that would be obscured in larger cohorts with noisy individual-level data. Once identified, these features can be investigated through targeted searches, such as by creating templates. For example, in our prior work [40,103], methods to identify individual differences in brain organization were first developed based on a precision fMRI dataset (Midnight Scan Club dataset [20]) where their measurement properties, such as reliability and association with task fMRI measures, could be investigated in detail. These methodologies were then extended to the larger HCP dataset to examine population-level variation.

Similarly, precision fMRI methods were initially used to map individual-specific organization of the motor cortex within a small sample of individuals [104]. In these robust datasets, a previously unknown network was identified that was interdigitated with the classic motor homunculus, termed the Somato-Cognitive Action Network (SCAN). The precision data allowed the investigators to determine the repeatability of these findings and connect them with a deep phenotyping of motor-related task measures. Once identified, SCAN was also found in large group-averages, including the HCP, ABCD, and UK Biobank, where the representation of SCAN was previously harder to separate from noise.

In a more clinically based example, precision data identified a consistent feature in individuals with major depressive disorder: an expanded salience network size [105]. Again, this pattern had not been previously highlighted by prior large-scale studies [106–108], but, once identified using precision fMRI, the authors developed an analysis procedure to replicate this effect in the ABCD.

Another strategy for integrating precision and consortium datasets is to initially use consortium studies to identify reliable (if small) cross-subject links between brain and behavior and test how the links may fluctuate within individuals using focused studies. Consortium datasets provide a way to focus on associations found across a large population, while individually centered designs provide a means to test mechanistic and experimental models of these associations [109]. For example, researchers could use consortium datasets to identify brain network connections whose scale correlates with executive function (e.g., [110]). This information could guide a subsequent precision fMRI study, involving extensive sampling of different executive function factors, such as updating, shifting, and inhibition [101], to investigate mechanistic relationships within participants after maximizing reliability in each measure, including inhibitory control (requiring >1,000 trials according to our prior work [27]). This approach could then be used to examine daily variation and training effects in executive function and to causally test the import of the locations and their suitability as treatment targets, such as with TMS [111]. These steps may be necessary to effectively translate consortium results to the clinical domain.

Concluding Remarks

Despite significant advances in BWAS, one major limitation of current designs is the low individual-level precision of the brain and behavioral measures. Precision approaches provide a means to reduce measurement noise and enhance signals by focusing on more valid measures of brain function and behavior. Jointly, we suggest calls for more attention to the amount and form of data collected per participant, in addition to the total participant number, in efforts to increase BWAS effects. Future studies may gain additional traction by combining such precision studies with consortium studies. With the promise of using more individual-focused approaches, we have put together important questions for future studies (see Outstanding Questions).

Text box 1. Simulation methodology.

We conducted a simulation to illustrate how increasing within-subject variability attenuates correlations estimated in BWAS. Assuming that brain and behavioral measures are linked, we generated correlated samples with a perfect true correlation (r=1) by constructing a multivariate probability distribution via Matlab’s copulas function. From this distribution, we simulated brain and behavioral measures in 100 subjects. Across iterations, we varied the size of the within-subject variance in brain and behavioral measures at four different levels. Then, the Pearson correlation between the sampled brain and behavioral measures was computed for each level of within-subject variability. This process was repeated 100 times to show the mean correlation coefficient across tests. The error bars are the 95% confidence interval of the mean.

HIGHLIGHTS.

• Recent work has highlighted that a large number of participants are needed to reproducibly predict individual behavior traits based on characteristics in brain structure or function. However, having enough data per participant is also critical for prediction.

• Here, we review recent evidence that the limited performance of current brain-behavior prediction models is driven by two major causes, which are noisy data and small effects. We offer a framework to tackle these challenges through ‘precision’ brain and behavioral approaches that collect more per-participant data and implement within-subject experimental designs.

• We discuss how integrating precision approaches with consortium studies may provide improved brain-behavior predictions necessary to achieve more powerful clinical applications.

GLOSSARY

Adolescent Brain Cognitive Development (ABCD) data

a longitudinal consortium dataset comprising approximately 11,800 children, starting at ages 9–10. It measures a variety of behavioral and MRI measures aimed at tracking cognitive, behavioral, and biological development through adolescence

Between-subject variability

differences observed among individuals in a given measure. In BWAS, the focus is on examining how between-subject variability in behavioral and brain measures correlate across individuals

Consortium data

large-scale data collected from many participants using the same scanning protocols to address a common research goal, often across multiple sites

Functional connectivity

temporal correlations between brain regions in ongoing fluctuating blood-oxygen-level dependent signals used to define the functional architecture of the brain (e.g., brain networks)

Human Connectome Project (HCP) data

a consortium dataset of structural and functional MRI and behavioral measures of cognition, emotion, and personality from 1,200 healthy young adults (ages 22–35)

Intraclass Correlation Coefficient (ICC)

the ratio of the variance due to differences between individuals to the total variance, which is the sum of between- and within-subject variances. ICC ranges from 0 to 1; measures of ICC of close to 1 are interpreted as having low within-subject variability (relative to the amount of between-subject variability). As such, ICC is often used as a measure of agreement or consistency

Measurement error

the difference between a measured estimate and the true value of a parameter that can be caused by randomness or systematic variance

Midnight Scan Club dataset

a precision fMRI dataset of 10 participants containing five hours of resting-state and six hours of task fMRI over 10 sessions with the goal of acquiring precise individualized measures

NIH Toolbox

a set of cognitive, emotional, motor, and sensory measures developed to be a norm-referenced test across a range of studies

Precision approach

an approach based on collecting a high-volume of data per participant (often across multiple days and states) to reduce noise, resulting in precise individual-level estimates

Resting-state fMRI

fMRI data collected in the absence of an explicit task to examine intrinsic brain activity. Participants may be asked to keep their eyes open on a fixation or to keep their eyes closed

(Test-retest) Reliability

a measure of consistency when administering tests repeatedly. High reliability indicates that similar outcomes are obtained when repeated tests are conducted on the same participants under the same conditions. Reproducibility, in contrast, measures the extent to which other researchers can replicate findings using the same or equivalent procedures

UK Biobank Imaging Dataset

a consortium dataset of over 60,000 adults containing multiple structural MRI scans, four minutes of task-fMRI, and six minutes of resting-state fMRI with the goal of identifying lifestyle and genetic phenotypes and how they affect disease risk

Validity

the extent to which an assessment accurately measures the intended construct. A high construct validity can legitimize the inferences made from a measure

Within-subject variability

the extent to which an individual’s measure varies. It can be influenced by factors that are unsystematic (random noise) or systematic (differences across conditions or changes over time [e.g., learning, habituation])