A multi-sample evaluation of the measurement structure and function of the modified monetary incentive delay task in adolescents

Abstract

Interpreting the neural response elicited during task functional magnetic resonance imaging (fMRI) remains a challenge in neurodevelopmental research. The monetary incentive delay (MID) task is an fMRI reward processing task that is extensively used in the literature. However, modern psychometric tools have not been used to evaluate measurement properties of the MID task fMRI data. The current study uses data for a similar task design across three adolescent samples (N = 346 [Age_mean 12.0; 44 % Female]; N = 97 [19.3; 58 %]; N = 112 [20.2; 38 %]) to evaluate multiple measurement properties of fMRI responses on the MID task. Confirmatory factor analysis (CFA) is used to evaluate an a priori theoretical model for the task and its measurement invariance across three samples. Exploratory factor analysis (EFA) is used to identify the data-driven measurement structure across the samples. CFA results suggest that the a priori model is a poor representation of these MID task fMRI data. Across the samples, the data-driven EFA models consistently identify a six-to-seven factor structure with run and bilateral brain region factors. This factor structure is moderately-to-highly congruent across the samples. Altogether, these findings demonstrate a need to evaluate theoretical frameworks for popular fMRI task designs to improve our understanding and interpretation of brain-behavior associations.

1. Introduction

The way in which phenomena are numerically represented in a study plays a critical role in the interpretations that a researcher can make. For over half a century, psychology has grappled with measurement issues to improve the reliability and interpretation of results (Association American Educational Research et al., 2014, Borsboom, 2006, Cronbach and Meehl, 1955, Flake and Fried, 2020, Lilienfeld and Strother, 2020). While not a core concern in the initial work using functional magnetic resonance imaging (fMRI), measurement issues such as reliability (i.e., precision of measurement) have become an increasing concern in the neuroimaging community (Bennett and Miller, 2010, Elliott et al., 2020, Nikolaidis et al., 2022, Noble et al., 2019). However, the validity of measurements (i.e., the degree to which the measurement accurately reflects the target construct) has largely remained at the periphery.

Since the inception of the blood oxygen level dependent (BOLD) contrast (Kwong et al., 1992, Ogawa et al., 1990), neurodevelopmental researchers have adopted several experimental tasks to measure the neural processes associated with latent constructs of cognitive functioning. This includes the measurement of reward processing (Richards et al., 2013, Silverman et al., 2015) and cognitive control (Luna, 2009, Luna et al., 2015) to better link changes in brain and behavior across the developmental period from childhood to adulthood (i.e., adolescence [Sawyer et al., 2018]). To date, neurodevelopmental work has contributed to progress in understanding the developing brain (Bethlehem et al., 2022, Mills et al., 2016) and has defined several neurodevelopmental frameworks (Casey et al., 2019, Ernst, 2014, Steinberg, 2010). Despite this progress, the generalizability of neurodevelopmental findings (Sherman et al., 2018), predictive ability of the frameworks (Ernst et al., 2022) and reliability issues (Elliott et al., 2020, Noble et al., 2019) in brain-behavior studies using fMRI remain a barrier to its ability to inform theory, clinical practice and public policy.

To confidently interpret a brain-behavior association, it is crucial to understand how the neural response elicited during fMRI tasks reflects the specific mental process the researcher intended to measure (Francken et al., 2022). The monetary incentive delay (MID) task is a popular fMRI reward processing task that is extensively used in the adult and adolescent literature. Despite its popularity in brain-behavior research, the measurement properties of the task remain unclear. The purpose of the present study is to provide clarity about the measurement properties of the MID task by evaluating the degree to which a nearly identical MID task design elicits a neural response that can be and is measured in the same way across multiple adolescent samples. Holding preprocessing and analytic pipelines constant, both confirmatory (restricted, hypothesis-driven) and exploratory (unrestricted, data-driven) factor analyses are used to determine whether a range of definitions of the reward processing constructs (i.e., approach and avoidance) are related and interchangeable across samples. Results from these analyses will be valuable in understanding the generalizability of neural findings within and across MID task contrasts and adolescent samples.

1.1. Task FMRI in correlational research: methodological and conceptual challenges

A primary goal of contemporary neurodevelopmental frameworks is to explain brain-behavior associations across samples and conditions (Ernst et al., 2022, Rosenberg et al., 2018). The range of operationalizations of a given construct, such as reward, in fMRI pose methodological and conceptual challenges, as different operationalizations of variables (e.g., contrasts from task fMRI) may not always map onto postulated construct(s).

One methodological challenge is designing fMRI tasks that can truly isolate specific mental processes (Poldrack and Yarkoni, 2016) such as sensitivity towards rewards (i.e., researcher’s construct of interest). In the simplest form, BOLD signals obtained during two experimentally manipulated conditions (e.g., low-probability/high-reward vs. high-probability/low-reward) of a task are compared to obtain scores that measure between-person differences in how a presumed phenomenon, such as motivation to approach rewarding stimuli, is manifested. The assumption is that subtraction of the estimated signals between trial types is a valid numerical representation of a specific interindividual difference in the construct of interest: arousal and allure of high reward stimuli that drives approach behaviors. However, this assumption is often violated because a contrast score does not always produce a measurement that accurately captures the latent mental process (Hanson, 2022, Poldrack, 2010).

One conceptual challenge is the under-discussed gap between the experimental and correlational psychology frameworks (Cronbach, 1957). Whereas experimental psychologists aim to control their conditions to find ordered structure in the behavior of their subjects/participants through experimental designs (Skinner, 1956), correlational psychologists take advantage of naturally occurring individual differences between participants to infer associations, or networks of associations, among variables. In the case of neurodevelopmental research that leverages both frameworks, ensuring the consistency of the interpretation(s) across brain-behavior studies requires confidence that (a) the fMRI task is a valid measure to isolate the mental process of interest1 and (b) there is converging evidence across different measures of a given construct reflecting the direction2 of associations that are consistent with the theoretical framework. While a mental process from an experimental task can be demonstrated to occur across administrations (Caplan, 2007, Henson, 2005), it is crucial for neurodevelopmental researchers using or developing tasks for correlation brain-behavior studies to validate that the structure and function of the construct aligns with the theory (Cronbach and Meehl, 1955, Flake et al., 2022, Lilienfeld and Strother, 2020). Failing to undertake this challenging work may lead to misinterpretation of the resulting brain-behavior findings across studies. Engaging in these efforts may also help standardize the verbal descriptions and language of the measured, complex mental processes.

In the neurodevelopmental literature, consistency of measurement is often established by citing work for a particular construct, such as reward (as author MD here has done in [Demidenko et al., 2020]) or motivational processing during the MID task (as in [Bjork et al., 2010]), through reference to studies that use a range of fMRI task designs and contrasts. In principle, this is a reasonable approach to justify and form a testable hypothesis about the association among constructs based on coherence across the literature. There are indeed a wide range of tasks (Richards et al., 2013, Sherman et al., 2018) and contrasts (Tervo-Clemmens et al., 2020) being used in the neurodevelopmental literature. The diversity in tasks and contrasts is both a strength and limitation. Measurement diversity is a strength because it provides the potential for convergent evidence across different reward types, such as instrumental-reward tasks (e.g., MID task) or decision-making tasks (e.g., Wheel of Fortune) (Richards et al., 2013), contributing inferences about the overarching construct. Measurement diversity is a limitation because each study typically measures the construct of interest using a single task or set of contrasts, so the relations among the different operational measures of what is assumed to be the same construct are often unclear. In practice, the postulated mental process reflected in the specific study is accepted, ‘loosely connected’ (Haeffel, 2022, p. 2) and rarely confirmed empirically, which is problematic when testing theories (Hull, 1952, Uttal, 2003).

1.2. Tasks and contrasts: uncertainty about the measured process

Motivational processing tasks are common tools used to study brain-behavior associations in the neurodevelopmental literature. One hypothesis is that an individuals’ motivational processing manifests as activation of distinct brain regions comprising approach and avoidance systems. Approach systems are thought to manifest in striatal and orbitofrontal cortex regions, whereby the function of these systems is to track the appetitiveness, incentive salience and motivation to approach stimuli. Avoidance systems are thought to manifest in amygdalar, hippocampal and insular regions, whereby the function of these systems is to track aversiveness of stimuli, negative valence, and motivation to avoid stimuli (Richards et al., 2013). Both systems are developmentally important in that to they are hypothesized to have implications for when and how substance use and risk-taking may emerge and progress over time (Doremus-Fitzwater et al., 2010, Ernst, 2014, Ernst and Luciana, 2015, Luciana et al., 2012). While there are distinct questions that can be formed based on the motivational processing literature, there are several challenges in interpreting and comparing functional activation across the neurodevelopmental studies.

Take for instance, the proposition that scores obtained from reward tasks used in fMRI can numerically represent a component of motivational processing (Schultz, 2015) that is important in neurodevelopmental research, such as reward anticipation (Silverman et al., 2015). Studies evaluating brain-behavior associations using the MID task have used a diverse set of contrasts to measure motivational processing and test their brain-behavior hypotheses in the context of reward anticipation. For example, in a late-adolescent sample, individual differences in activity in the nucleus accumbens (NAcc) during combined Large and Small Gain versus Neutral contrast were related to individual differences in marijuana use (Martz et al., 2016) whereas NAcc activity during the Large Gain versus Neutral stimuli contrast was not related to differences in a broad measure of substance use (Demidenko et al., 2020). In mid-adolescents, activity in the NAcc during the Large Gain versus Small Gain contrast was related to differences in alcohol use only among high sensation seekers in one study (Büchel et al., 2017) and only tobacco use in another study (Karoly et al., 2015). Some may argue that the same construct – approach motivation – was measured in all these studies, however, without empirical tests of measurement structure of the fMRI task these differences in brain-behavior relations becomes difficult to interpret.

While meta-analyses of reward tasks have provided comprehensive reviews of brain regions involved in different components of reward processing (Bartra et al., 2013, Chen et al., 2022, Clithero and Rangel, 2014, Diekhof et al., 2012, Dugré et al., 2018, Flannery et al., 2020, Lutz and Widmer, 2014, Oldham et al., 2018, Silverman et al., 2015, Wilson et al., 2018), there are few studies to date that evaluate within- and between-study measurement properties of brain-brain associations in these tasks. This, in part, is because most of the studies using reward fMRI tasks to measure motivational processing have obtained data from small samples using a wide variety of study designs, making the assessment of measurement properties difficult. However, some tasks, such as the MID, have a long history in the reward literature and have been administered in numerous studies, making them a good target for investigation of measurement properties.

The MID task (Knutson et al., 2000) was designed to elicit robust activation of reward regions during reward anticipation and reward consumption (Knutson and Heinz, 2015). The task has been cited as, ‘the most widely used probe of human incentive motivational processing by nondrug rewards’ (Bjork, 2020, p. 2). The task has been reported to relate to affective ratings in adult and adolescent samples (Bjork et al., 2004, Knutson et al., 2001, Wu et al., 2014) and has been utilized extensively throughout the neurodevelopmental and substance use literature (Balodis and Potenza, 2015, Bjork, 2020). Different variants of the task are currently used in large consortium (Casey et al., 2018, Schumann et al., 2010) and longitudinal studies (Murray et al., 2020, Yau et al., 2012). In the context of neurodevelopmental research, the MID task is used to evaluate changes in reward-related activity during adolescence and how these changes in neural activity are associated with outcomes including risk-taking and substance use (Casey et al., 2018, Bjork, 2020).

One hypothesized theoretical framework of the MID task that has been proposed decomposes the anticipatory phase into two distinct constructs. In short, cues in the task elicit approach and avoidance signals (Knutson et al., 2014, Knutson and Greer, 2008), whereby levels of approach are presumed to manifest as activity in the NAcc whereas levels of avoidance are presumed to manifest as activity in the anterior insula (AI). During the anticipation phase, anticipatory gain and loss cues often differentially activate the NAcc, AI, medial premotor and dorsal striatal regions (Bjork et al., 2004, Knutson et al., 2003). Meta-analyses, based mostly on adult samples, have provided some evidence for the proposed hypothesis: activity in the bilateral AI is higher during anticipatory loss versus gain conditions and the activity in the right AI and bilateral NAcc are higher for anticipatory gain cues than for neutral cues (Knutson and Greer, 2008). However, this theoretical framework has not been empirically tested using the tools of modern psychometrics. In fact, it remains unclear whether these are discernible components of reward observed in the between-subject neural activity during the anticipatory phase and how consistent this measurement property is across adolescent samples.

Understanding the measurement properties of the MID task is critical to making valid interpretations about neural changes and their association with behavior across adolescence. In human and animal models, there are significant biological and behavioral changes with the onset of pubertal development during adolescence (Dahl et al., 2018, Forbes and Dahl, 2010, Spear, 2004). In particular, puberty is marked by changes in dopamine receptors (Spear, 2000) in reward-relevant regions that are important to motivational processing and risk-taking behaviors (Luciana et al., 2012, Pfeifer and Berkman, 2018, Silverman et al., 2015). While the MID task has repeatedly been shown to activate brain regions hypothesized to be meaningful to approach and avoidance systems (Chen et al., 2022, Oldham et al., 2018), there is little empirical evidence confirming that the hypothesized approach and avoidance systems are observed in the MID task consistently across adolescent samples.

Given the popularity of the MID task (Bjork, 2020), evaluating psychometric properties of the task would be valuable to researchers asking developmental and brain-behavior questions. If the between-subject covariation between manifest variables from the MID task is not consistent across adolescent and adult groups, the differences observed between different age groups may be attributed to a construct (i.e., motivational processing) when in fact the difference may be due to differential item or test functioning (Association American Educational Research et al., 2014). Second, if the between-subject covariation between the manifest variables from the MID task demonstrates a different number of factors and different structure across the samples, this may indicate that the studies are measuring meaningfully different constructs. In both cases, the bias in the measure may be inadvertently attributed to differences observed between groups, samples or brain-behavior associations (Borsboom, 2006).

2. Current study

To date, meta-analyses and systematic reviews of reward processing have spanned diverse sets of tasks, task contrasts (Sherman et al., 2018, Silverman et al., 2015, Tervo-Clemmens et al., 2020) and different software, preprocessing and analytic pipelines have the potential to impact results in adolescent and adult samples (Botvinik-Nezer et al., 2020, Carp, 2012). The neurodevelopmental cognitive neuroscience literature would benefit from understanding the extent to which measurement properties generalize across multiple adolescent samples that collected data using the same reward task. The Adolescent Brain Cognitive Development (ABCD; [Casey et al., 2018]), Michigan Longitudinal Study (MLS; [Yau et al., 2012]) and Adolescent Health Risk Behavior (AHRB; [Demidenko et al., 2020]) studies collected data using a nearly identical MID task. These MID task data allow for an empirical test of the measurement properties of the task and its invariance across studies. This would be an important first step in understanding how task results can and can’t be interpreted within and between studies using this version of the design. Here, the theoretical framework proposed in Knutson et al. (2014) is put to a severe test and data-driven measurement properties are evaluated for the MID task across three adolescent fMRI samples.

First, the degree to which a primary theoretical framework of the MID task is empirically supported across samples is evaluated using a restrictive and semi-restrictive model. Aim 1a evaluates the degree to which task contrasts reflect the hypothesized latent factors, approach and avoidance. This is achieved by fitting a confirmatory (restricted) factor (CFA) model to six anticipation task contrast measures for four brain regions of interest (ROIs). The model articulates the hypothesis that contrasts at these hypothesized regions (e.g., NAcc and AI) load onto two factors, approach and avoidance, and the hypothesis that the same structure is present in each adolescent sample. The hypothesized structure is based on the theoretical framework proposed in Knutson et al. (2014) and is represented by the diagram in Fig. 1B. In Aim 1b, given the likelihood of non-zero cross-loadings across brain regions a less restrictive CFA is fit using the exploratory structural equation model (ESEM). Like CFA, ESEM fits an a priori factor model. Unlike the specific CFA model in Aim 1a, ESEM allows for empirically driven non-zero cross-loadings, providing a semi-restrictive test. Like Aim 1a, Aim 1b fits the ESEM model to examine invariance across the three samples of participants. The hypothesized two factor model is consistent with Fig. 1B, except for adding data-driven non-zero loadings.

Fig. 1.

Brain Regions (A), Conceptual Figure for Confirmatory Factor Analysis (CFA) and Exploratory Structural Equation Model (B). Harvard-Oxford Bilateral Nucleus Accumbens (NAcc; Yellow) and Anterior Insula (Red) regions imposed on MNI152 2 mm brain. R = Right; L = Left; A = Anterior; P = Posterior.

Second, because the hypothesized factor structure in Aim 1 may be an incorrect representation of the estimated neural activity, a data-driven factor structure of MID task contrasts is evaluated and compared across samples. In Aim 2, exploratory (unrestricted) factor (EFA) is used to evaluate the structure of the measures using a data-driven approach across the three adolescent samples. This allows for the flexibility in the number of factors and the item loadings. Due to the exploratory nature, no specific hypotheses are made but EFA is run on each study separately and compared qualitatively and quantitatively.

Pubertal development coincides with changes in motivational processing during adolescence (Forbes and Dahl, 2010). Therefore, in Aim 3, the effects of pubertal development are considered. Local SEM (sliding window) is used to examine how the factor loadings change across the self-reported pubertal developmental scale in the ABCD study, to evaluate whether measurement properties from Aim 2 stabilizes as participants’ self-reported pubertal maturity increases. Due to the unclear evidence of how pubertal development may impact the measurement properties of the MID task, no hypotheses are specified.

The introduction, hypotheses, methods, and analytic code (fMRI analyses using python, and CFA, ESEM, EFA and LSEM models using R) were pre-registered and are openly available online (https://osf.io/a6t8k; Demidenko et al., 2023).

3. Methods

3.1. Participants

3.1.1. Michigan Longitudinal Study (MLS)

The MLS is a longitudinal community study that is designed to characterize neural differences between participants from families with (2/3) and without a parental history of alcohol use disorder. The MLS study includes two cohorts, a Child Risk and Neuropsychological Risk cohorts. The focus here is on the first available session of neuroimaging for the Neuropsychological Risk cohort (Martz et al., 2016). For a full description of the study purpose see Zucker et al. (2000). MLS had a total of 177 participants (Age 18 - 23 years old) in the Neuropsychological Risk cohort that completed the imaging protocol. Participants who have the necessary structural, functional, and behavioral data and pass quality control steps are retained for the final analyses.

3.1.2. Adolescent Health Risk Behavior (AHRB) study

The AHRB study is a longitudinal multi-phase study that is designed to characterize neural differences of risk-taking behaviors (Demidenko et al., 2020). The focus here is on the neuroimaging phase (Phase 2). For Phase 2 Wave 1, participants were recruited based on Phase 1 Wave 1 self-reported risk behaviors. AHRB recruited N = 115 high and average/low risk-taking adolescents (Age 17 - 21 years old) to complete the neuroimaging protocol. Of the 115 participants eligible for inclusion, seven participants could not undergo magnetic resonance imaging (MRI) so 108 completed the imaging protocol. Participants who have the necessary structural, functional, and behavioral data and pass quality control steps are retained for the final analyses.

3.1.3. Adolescent Brain Cognitive Development (ABCD) study

The ABCD study is a longitudinal national study that is designed to study the ways socioeconomics and ethnicity affect substance use, mental health, and health outcomes across development (Volkow et al., 2018). The ABCD study recruited 11,878 adolescents across 21 sites (Casey et al., 2018). The focus here is on the 4.0 brain imaging data that is released by the ABCD-BIDS Community Collection (ABCC; Feczko [ et al., 2021]), which as of January 2023 (following stage 1 in principal acceptance) contains year 2 imaging data from approximately 7000 participants (Age 11 - 13 years old). Given the reported effects of motion on reliability of task fMRI measures in the younger ABCD sample (Kennedy et al., 2022) participants that have a reported mean framewise displacement value for the MID that is > .90 are excluded from our analysis. The ABCD sample (N = 11,878) is much larger than the MLS and AHRB samples, which poses an imbalance issue when testing for differences in factor structure across samples. To minimize this, a subsample of 1000 participants are randomly selected across sites (to alleviate site effects) and low motion subjects are excluded. Independent of the first 1000 randomly selected participants, another subsample of 1000 participants are randomly selected to use as a held-out sample to evaluate the performance of the EFA and LSEM in the ABCD data. Thus, a total of 2000 adolescents are subsampled from the full ABCD sample data and participants who have the necessary structural, functional, and behavioral data and pass quality control steps are retained for the final analyses.

3.2. FMRI task & regions of interest

3.2.1. Monetary Incentive Delay (MID) task

Across the ABCD, AHRB and MLS studies, reward processing is measured using comparable versions of the modified Monetary Incentive Delay (MID) task. The task and their nuanced differences are described below, and schematics provided in Fig. 2.

Fig. 2.

Motivational Processing: Schematics of the Monetary Incentive Delay (MID) task from the ABCD, AHRB and MLS samples.

The MID task (Knutson et al., 2003) is used to model neural signatures of the anticipation and receipt of monetary gains or losses. The modified MID task from ABCD and AHRB study (Casey et al., 2018) are identical in design. Apart from some minor differences in fixation/probe/feedback durations, the version of the MID task in the MLS (Yau et al., 2012) is nearly identical to the ABCD and AHRB studies. Consistent across the ABCD, AHRB and MLS studies, during the MID task each trial starts with a cue type and consists of three phases: anticipation, probe, and outcome (that is, feedback).

Details about the tasks across the three studies are summarized in supplemental Table S1. In the ABCD (Casey et al., 2018) and AHRB (Demidenko et al., 2020) versions of the modified MID task, the cues are differentiated by color (“Win” cue = Pink; “Dont Lose” cue = Yellow; “No Money at Stake” cue = Teal). There are twelve trial orders of the task, consisting of 50 contiguous trials and 10 trial types per run (5:42 min long). In the MLS version of the MID task (Martz et al., 2016, Yau et al., 2012), the cues are differentiated by color in grayscale (“Win” cue = Red; “Dont Lose” cue = Blue; “No Money at Stake” cue = White). Trials are presented in pseudorandom order, there are 50 contiguous trials and 100 trials total (lasting approximately 5 min long). In all three studies the task difficulty is individualized by first obtaining a mean response time during a practice trial of the task outside of the scanner. The task is individualized to reach a 60% accuracy rate by adjusting the difficulty (i.e., probe duration). For example, in the ABCD/AHRB study, the difficulty is adjusted every third incentivized trial based on the overall accuracy for the last six trials.

In the ABCD, AHRB and MLS study, participants are compensated with a fixed amount of money to participate in the study and receive additional money in accordance with their performance during the MID task. The average during the baseline data in the ABCD sample is $21.43 (Casey et al., 2018) and the maximum in the AHRB sample is $30 (Demidenko et al., 2020), specific details for average or maximum compensation in the MLS sample have not been reported. Compared to other approaches, such as rewarding “points” that can be exchanged for candy (Cao et al., 2019), a monetary incentive is vital in motivational processing during the task if measuring arousal and valence. Several contrasts are traditionally modeled in the MID task, for more details see Oldham et al. (2018).

3.2.2. Regions of interest (ROI)

The focus of this study is to evaluate the static mapping (Knutson et al., 2014) of the MID task. Following the pre-specified theoretical structure (Fig. 1), we limit the problem space to two bilateral brain regions: the left and right nucleus accumbens (NAcc) and the left and right anterior insula (AI). The regions are defined using the Harvard-Oxford Cortical and Subcortical atlas (50thr). In the Harvard-Oxford Cortical atlas, the insula is not constrained to the AI. Therefore, the term ‘Anterior Insula’ is used to identify the probability map on neurosynth.org on October 8th, 2022. The associated AI map (.nii.gz) is downloaded, thresholded at ∼50 % (> 8) and used to mask the left and right ROI from the Harvard-Oxford Cortical atlas to the anterior portions. The resulting left/right NAcc and left/right AI ROIs are used to extract the mean signal intensity values (from the contrast beta maps described below) for the subsequent analyses.

The NAcc and AI regions selected here are consistent with brain regions commonly described in the reward circuitry (Bissonette et al., 2014, Gentry et al., 2019, Haber, 2017), those indicated as meaningful to the anticipatory phase of the MID task (Knutson et al., 2014, Oldham et al., 2018) and the regions associated with the terms ‘reward anticipation’, ‘approach behaviors’ and ‘avoidance behaviors’ using the NeuroQuery tool (Dockès et al., 2021).

3.3. MRI acquisition, preprocessing and analysis

3.3.1. Acquisition

The acquisition details for the ABCD, AHRB and MLS datasets are summarized in supplemental Table S2.

3.3.2. Preprocessing anatomical and functional data

Structural and functional MRI preprocessing is performed using fMRIPrep 23.1.0 (Esteban et al., 2019, Esteban et al., 2022; RRID:SCR_016216), which is based on Nipype 1.8.3 ((Gorgolewski et al., 2011, Gorgolewski et al., 2018); RRID:SCR_002502). Preprocessing between the ABCD, AHRB and MLS datasets are held constant except for two differences. The MLS study does not collect fieldmaps and uses a spin-echo acquisition. Therefore, fMRIPrep’s fieldmap-less distortion correction is not possible to perform on the MLS BOLD data. For all ABCD and AHRB data, fieldmap distortion correction is used when the participants do have fieldmap data. In addition, the time-repetition (TR) in MLS is 2000 ms and ABCD/AHRB is 800 ms, due to the longer repetition-time in the MLS sample slice-time correction is applied in MLS and but not in ABCD/AHRB. Outside of these exceptions, the preprocessing of the BIDS data use comparable pipelines.

Preprocessing of anatomical data. T1-weighted images are corrected for intensity non-uniformity (INU) with N4BiasFieldCorrection (Tustison et al., 2010), distributed with ANTs 2.3.3 (Avants et al., 2008, RRID:SCR_004757) and used as T1w-reference throughout the fMRIPrep workflow. The T1w-reference is then skull-stripped with a Nipype implementation of the antsBrainExtraction.sh workflow (from ANTs), using OASIS30ANTs as target template. Brain tissue segmentation of cerebrospinal fluid (CSF), white-matter (WM) and gray-matter (GM) is performed on the brain-extracted T1w using fast (FSL 6.0.5.1:57b01774, RRID:SCR_002823, Zhang, [ et al., 2001]). Brain surfaces are reconstructed using recon-all (FreeSurfer 7.2.0, RRID:SCR_001847, Dale, ] et al., 1999]), and the brain mask estimated previously is refined with a custom variation of the method to reconcile ANTs-derived and FreeSurfer-derived segmentations of the cortical gray-matter of Mindboggle (RRID:SCR_002438, Klein, [ et al., 2017]). Volume-based spatial normalization to one standard space (MNI152NLin2009cAsym) is performed through nonlinear registration with antsRegistration (ANTs 2.3.3), using brain-extracted versions of both T1w reference and the T1w template. The following template is selected for spatial normalization: ICBM 152 Nonlinear Asymmetrical template version 2009c (Fonov [ et al., 2009], RRID:SCR_008796; TemplateFlow ID: MNI152NLin2009cAsym).

Preprocessing of functional data. For each of the 2 BOLD functional runs, the following preprocessing is performed. First, a reference volume and its skull-stripped version are generated using a custom methodology of fMRIPrep. Head-motion parameters with respect to the BOLD reference (transformation matrices, and six corresponding rotation and translation parameters) are estimated before any spatiotemporal filtering using mcflirt (FSL 6.0.5.1:57b01774, Jenkinson, [ et al., 2002]). The estimated fieldmap [in ABCD and AHRB] is then aligned with rigid-registration to the target EPI (echo-planar imaging) reference run. The field coefficients are mapped on to the reference EPI using the transform. In the MLS data, BOLD runs are slice-time corrected to 0.966 s (0.5 of slice acquisition range 0 s-1.93 s) using 3dTshift from AFNI (Cox and Hyde [, 1997], RRID:SCR_005927). The BOLD time-series (including slice-timing correction when applied) are resampled onto their original, native space by applying the transforms to correct for head-motion. The BOLD reference is then co-registered to the T1w reference using bbregister (FreeSurfer) which implements boundary-based registration (Greve and Fischl, 2009). Co-registration is configured with six degrees of freedom. Framewise displacement (FD) is computed using two formulations following Power (absolute sum of relative motions, Power [ et al., 2014]) and Jenkinson (relative root mean square displacement between affines, Jenkinson [ et al., 2002]). FD and DVARS are calculated for each functional run, both using their implementations in Nipype (following the definitions by Power [ et al., 2014]). Principal components are estimated after high-pass filtering the preprocessed BOLD time-series (using a discrete cosine filter with 128 s cut-off) for anatomical CompCor (aCompCor). For aCompCor, three probabilistic masks (CSF, WM and combined CSF+WM) are generated in anatomical space. The head motion (translations and rotations) estimates are expanded with the inclusion of temporal derivatives and quadratic terms for each (Satterthwaite et al., 2013) and are included with the aCompCor components and cosine estimates as nuisance regressors in the first level design matrix. The BOLD time-series are resampled into standard space, generating a preprocessed BOLD run in MNI152NLin2009cAsym space.

3.3.3. Analyses

First level, second level and region of interest analyses are performed using Python 3.9.7 and Nilearn version 0.9.2 (Abraham et al., 2014).

In the first level analysis, a design matrix is specified for each participant. The design matrix includes the 15 task-relevant regressors (see task schematic in Fig. 2) and a number of nuisance regressors that are calculated by fMRIPrep. The task relevant regressors include the onset and duration of the five anticipatory cue types (e.g., Large Gain or No Money at Stake) and ten feedback cue types (e.g., You won $5 or You did not win $5), convolved with glover hemodynamic function (HRF). The nuisance regressors include estimated translation and rotation (+ derivatives & squared derivatives) of head motion, the first eight aCompCor noise components and the corresponding cosine regressors for high pass filtering (with a cutoff of 128 s) that are calculated by fMRIPrep (see preprocessing of functional data). Individual BOLD timeseries are prewhitened using an ‘ar1’ noise model and spatially smoothed using a 5 mm full-width half-maximum Gaussian kernel. A GLM is fit to estimate the brain masked BOLD timeseries by task-relevant conditions in the design matrix. For each individual subject and run, six contrasts are computed using the estimated beta (supplemental Fig. S1):

• 1.Anticipation Big Gain versus Neutral
• 2.Anticipation Big Gain & Small Gain versus Neutral
• 3.Anticipation Big Gain versus Big Loss
• 4.Anticipation Big Loss versus Neutral
• 5.Anticipation Big & Small Loss versus Neutral
• 6.Anticipation Big Loss versus Big Gain

For each of the six contrasts and two runs, four ROI mean signal intensity values are estimated. For each of the four ROIs, voxels within the ROI mask for a given MNI contrast beta map are extracted and averaged. If a participant’s beta map contains < 50 % of the voxels in an ROI, the participant is excluded from the analyses. Across the three samples, this produces a dataset of Six Contrasts x Four ROIs for each run across N participants.

For each study (i.e., ABCD, AHRB, MLS), a second level model is used to estimate whole brain activation maps. For each participant, contrast estimates from the two MIDs runs are averaged using a weighted fixed effects model and a group level mixed effect model averaged the within-subject estimates across subjects. These maps are used as measures of average activation patterns during the MID tasks in each of the studies across contrast types. The Big Gain versus Neutral contrast for each study is presented in-text, the remaining contrast types are provided in supplemental materials and publicly on NeuroVault.org. Due to the influence of sample size on the t-statistic, each whole brain contrast map across samples is converted to a Cohen’s d effect size using the formula: $\frac{t-\mathit{statistic}}{\sqrt{n}}$. Cohen’s d, t-/z-statistical maps are shared on NeuroVault. Since no inferences are meant to be drawn from these maps, and the data are primarily used for qualitative comparison of the task activation across the samples, no cluster correction is applied to resulting statistical maps.

3.4. Data analysis

3.4.1. Restricted, semi-restricted and unrestricted factor analysis

All subsequent Confirmatory Factor Analysis (CFA, Restricted), Exploratory Structural Equation Modeling (ESEM, Semi-restricted), Exploratory Factor Analysis (EFA, Unrestricted), and Local Structural Equation Modeling (LSEM) are conducted in R version 4.2.1 (R R Core Team, 2022). These analyses are performed on the extracted mean signal intensity values for the four ROIs across six contrast types (in each of the two runs) for participants from the ABCD, AHRB and MLS samples for data that pass quality control. Before running the structural models, the associations among all ROIs across contrast types are evaluated using product-moment correlations. To reduce the number of redundant parameters in the models, left and right ROIs whose correlations exceed r = .85 are constrained to be equal in the model.

CFA and multi-group CFA are performed as implemented in the lavaan package (Hirschfeld and Brachel, 2014, Rosseel, 2012, Rosseel et al., 2021) and used to describe how task contrast variables reflect the hypothesized factors and to test the invariance of that description across the three adolescent samples (Aim 1a). An a priori model is specified using lavaan syntax depicting the item level indicators across contrasts, runs and ROIs (see Fig. 1 for specified model) as indicators of two exogenous latent variables, approach, and avoidance, that may be correlated. Models were identified by fixing one of the factor loadings for each latent factor to = ‘1’ and estimated using the maximum likelihood estimator (ML). The CFA model comprising all three samples specified the same factor structure across the combined three samples and so is used as the strict invariance model comparison (i.e., all constraints are equal). In the second step, configural invariance (factor structure) is invoked by first including ‘group’ (i.e., ABCD, AHRB, MLS) in the CFA model to evaluate structural invariance across the three samples. If the models show adequate fit, in the third step, metric (item loadings) invariance is invoked by adding a group equality constraint of ‘loadings’ (e.g., weak invariance), with invariance of the factor loadings across groups tested via nested model comparison of model fits, using χ² difference. Given the lack of measurement models of fMRI ROI mean signal intensity data, we do not plan to test strong invariance, such as scalar invariances (D’Urso et al., 2022, Putnick and Bornstein, 2016). Individual and multigroup model parameter estimates are reported and a table with the χ² statistic, degrees of freedom, p-value, root mean squared error of approximation (RMSEA), comparative fit index (CFI), Tucker-Lewis index (TLI) and standardized root mean square residual (SRMR) is provided. While traditional dynamic fit index cutoffs are debatable (McNeish and Wolf, 2021), model fit cuts-offs are used to determine whether a model has ‘excellent fit’ (i.e., RMSEA ≤.05, CFI ≥.95, TLI ≥.95, and SRMR ≤.05). These cut-offs are used in the fMRI connectivity literature (Beltz and Gates, 2017) and are recommended cut-offs for CFA (Brown, 2006). Models exhibiting poor fit (i.e., RMSEA ≥.10, CFI/TLI ≤.90, and SRMR ≥.10) will not be interpreted. Change in model fit indices in CFI (≤.005), RMSEA (≥.01) or SRMR (≥.025), and AIC/BIC differences via model comparisons (i.e., anova) are used as indicators whether models are worse fits to the data (Chen, 2007; Meade et al., 2008). These indicators are used to provide converging evidence about improvements in model fit and disagreements are contextualized in text.

ESEM is performed as implemented in the esemComp package (Silvestrin and de Beer, 2022). ESEM is used to test a less restrictive model (Aim 1b). As previously described (Marsh et al., 2014, Zyl and Klooster, 2022), measurement error and method covariance may contribute to non-zero cross-loadings that may negatively impact model fit. ESEM complements EFA with target rotation towards the CFA model, but allowing for cross-loadings of items that increase fit to the data. Like the CFA model, a two-factor ESEM model is fit to all the item level indicators across contrasts, runs and ROIs, with target rotation specified as the hypothesized structure in Fig. 1. The model is fit to the data using a ‘ML’ estimator with the variances of the latent variables fixed to ‘1’ for identification.

EFA is performed as implemented in the factanal R package. This data-driven approach is used to estimate an unrestricted (data-driven) factor structure in the data for each study separately (Aim 2). First, a scree-plot, Horn’s parallel analysis (Dinno, 2009) as implemented using the paran package (Dinno, 2018) and BIC estimate are used to converge on the number of factors in the data that is most parsimonious. Using an oblique, ‘promax’ rotation (not imposing orthogonalization on the data), item loadings are estimated for the specified number of factors. The EFA is performed on the individual samples (i.e., ABCD, AHRB and MLS). The results are reported using heatmaps to indicate item loadings onto the exogenous variables. The EFA models are quantitatively compared between datasets using the Tucker index of factor congruence (Lorenzo-Seva and ten Berge, 2006) as implemented using the factor.congruence() function in the psych package.

3.4.2. Effect of pubertal developmental and sex at birth on EFA results

The ABCD sample is composed of youth in the early-to-middle pubertal developmental stages. This may impact the factor structure of neural mechanisms derived from reward contrasts. Using the held-out sample (N = 1000), the EFA results from the above are invoked as a CFA structure for the ABCD data to evaluate potential impact of pubertal development on the factor structure (Olaru et al., 2020). To evaluate how the factor structure differs across pubertal development (Aim 3), local SEM is performed as implemented in the sirt package (Robitzsch, 2022). The pubertal developmental scale (range 1 – 5) is introduced as a moderator into the CFA to examine if and how the factor loadings differ/change across the levels of pubertal development. Given that there may be differences between child and parent reports of pubertal development in the ABCD sample (Demidenko, Kelly et al., 2022), a separate model is run for the youth self-reported pubertal development and parent-reported pubertal development measures. To provide supplemental information to the local SEM results, we evaluate measurement invariance of the CFA structure across the self-reported sex at birth, male and female, in the ABCD data.

3.4.3. Sensitivity analysis: site and sampling effect

Sensitivity analyses are performed to evaluate the effect of ABCD sampling and the effect of University of Michigan as a site on the CFA and EFA models.

First, the AHRB and MLS studies are both collected on 3T GE scanners at the University of Michigan by the MRI Laboratory technicians. One of the 21 ABCD sites includes the University of Michigan (‘site_id’ = 13). To check the extent to which scanner site contributes to differences in factor structure, the CFA and EFA are rerun using only the data from the University of Michigan sites (i.e., holding scanner type and location constant). These results are reported in the supplemental materials.

Second, the CFA a priori (Fig. 1) model is bootstrapped (N = 300) using the ABCD data and model fit statistics are extracted (as described above). Then, the EFA model is bootstrapped (N = 300) using the ABCD data to estimate the variability in the number of factors that best explain the data. The extracted statistics (i.e., model fit and number of factors) are then reported in supplemental materials to visualize stability in these CFA/EFA estimates.

4. Results

4.1. Deviations from the stage 1 registered report

The analyses below are consistent with the analyses proposed in the Stage 1 Registered Report with three minor and one major exception (https://osf.io/a6t8k). First, in the Stage 1 submission the fMRIPrep preprocessing pipeline proposed to perform slice-time correction on all three samples. However, the ABCC data does not contain the necessary slice-timing information in the associated json files. The fast temporal sampling enabled by the multi-band sequence in the ABCD and AHRB fMRI data reduces the importance of slice-timing corrections in the short TR data. Thus, slice-timing correction is used in the MLS preprocessing pipeline but is not used in ABCD and AHRB preprocessing pipelines. Second, fieldmap-less distortion correction could not be applied on the MLS data because it is collected using spiral acquisition. As the AHRB data do not contain missing fieldmaps and the ABCC data selects a single fieldmap within a session to apply on all the functional runs, fieldmap-less distortion correction is not used in the preprocessing pipeline. Third, in the Stage 1 submission we proposed using the maximum-likelihood robust (MLR) estimator when estimating the lavaan models. However, when applied to the data, this estimator had a high rate of singularity issues as compared to the standard maximum-likelihood estimator, specifically for the ESEM. Since the proposed EFA analyses use the maximum-likelihood (ML) estimator, the ML estimator is also used in the lavaan models. Finally, one major protocol deviation is the exclusion of GE scanner sites in the ABCC data. As of October 2023, the GE data contain fieldmap errors, which made the functional data not usable based on the exclusion criteria. Hence, all ABCD data from GE scanner sites are excluded and post hoc sensitivity analyses are performed to evaluate the effect of this deviation across different intervals of N in the non-GE ABCD data. The final Stage 2 code & analyses are available on Github3.

4.2. Sample selection: ABCD, AHRB and MLS

For the ABCD sample, the NDA contains N = 10,414 Year 2 follow-up subjects (NDA Data Package #1209276). From these data, two subsamples of N = 48 subjects for each of the 21 study sites are subsampled (First: N = 1008, Second: N = 1008). From these three subsamples, subjects are selected if they contain an a) anatomy, b) field map and c) MID data via ABCC as of January 2023. This results in N = 599 (Primary Sample), N = 595 (Held-out sample) and N = 157 (site-13) that have usable MRI data.

From the initial ABCD, AHRB and MLS samples, participants are excluded based on five metrics to arrive at the final analytic sample. These metrics are described in greater detail in supplemental Section 2.1.2. For the ABCD sample, Primary N = 253, Held-out N = 248 and Site-13 N = 157 are excluded resulting in the final three ABCD samples of N = 346, N = 347 & N = 0, respectively. For the AHRB sample, out of the N = 108 participants that have MRI data, 11 participants are excluded resulting in a final sample of N = 97. For the MLS sample, out of the N = 177 participants that have MRI data, 65 participants are excluded resulting in a final sample of N = 112. For more specific details for exclusion reasons, see supplemental Table S3.

4.3. Descriptive statistics and main effects

The demographic information for each study is reported in Section 2.1.3, Table S4. The ABCD, AHRB and MLS samples differ in their probe hit response accuracy (%) and probe response times (ms). The average probe hit accuracy during the MID task in the ABCD, AHRB and MLS are 57 % (SD = .04), 57 % (SD = .04) and 71 % (SD = .13), respectively. The average probe mean response times during the MID task in the ABCD, AHRB and MLS are 269 ms (SD = 44 ms), 298 ms (SD = 21 ms) and 205 ms (SD = 28 ms), respectively. For more details about the performance and meaningful differences on the MID task across the three samples, see supplemental Section 2.1.4, Fig. S4.

For each of the three samples, the Cohen’s d effect size maps for the Big Win > Neutral anticipatory contrast are reported in Fig. 3. The activation in the ABCD, AHRB and MLS samples is consistent with reported activation likelihood estimates in meta-analyses for MID task contrasts (Oldham et al., 2018). Group-level activity is observed in the bilateral dorsal and ventral striatum, AI, supplementary motor cortex and occipital regions. The magnitude of the effect in the dorsal and ventral striatum is largest in the MLS sample, followed by the ABCD and AHRB samples.

Fig. 3.

Group Cohen’s d activation maps for (A) ABCD, (B) AHRB and (C) MLS sample. Threshold: d min = .20 & max = 1.5. Map overlaid on MNI152NLin2009cAsym brain. Cross placed on MNI coordinate associated with Right NAcc [12, 8, –10]. Note: The ‘Win’ cue type in the ABCD/AHRB sample is Pink and in the MLS it is Red.

Cohen's d maps for the other five contrasts (ABCD, AHRB, MLS), and each of the ABCD scanners and sites are provided in supplemental Section 2.1.5, Fig. S5, Fig. S6 and Fig. S7. The associated t-/z-statistic and Cohen’s d maps are available at the NeuroVault Collection #15642 (https://identifiers.org/neurovault.collection:15649).

4.4. Aim 1: evaluating the a priori CFA model of the MID task

The goal of Aim 1 is to evaluate whether the proposed theoretical framework (Fig. 1B) for the MID task is empirically supported by task fMRI MID data using a restrictive (Aim 1a) and semi-restrictive model (Aim 1b). The proposed hypothesis is that the measurement structure, as visually represented in Fig. 1B, is empirically represented in the data and consistent across the three samples. The product-moment correlation matrix between the 28 manifest variables across the three samples is reported in Fig. 4.

Fig. 4.

Product-moment correlations between the mean signal intensity for the [L]eft/[R]ight Nucleus Accumbens and Anterior Insula across six contrast types for Run 1 and Run 2 in ABCD, AHRB and MLS samples. Note: Manifest variable [1–28] labels specified in Fig. 5.

4.4.1. Aim 1a: evaluating the confirmatory factor analysis model across the ABCD, AHRB and MLS samples

In Aim 1a, an a priori model is specified depicting the item-level indicators across runs, ROIs and contrasts representing the two exogenous latent variables, approach and avoidance. Using model fit statistics, the CFA (restrictive) model is used to evaluate whether the model is empirically supported and consistent across the three samples.

The strict model (as specified in Fig. 1B) is fit to the data, by fitting the a priori model to the combined mean signal intensity data (ABCD, AHRB and MLS). Based on the registered cut-off criteria (i.e., RMSEA ≥.10, CFI/TLI ≤ = 0.90, and SRMR ≥.10), the model fit statistics, χ2(354) = 23,971.2 p < .001; CFI = 0.12, TLI = .06, RMSEA = 0.35, SRMR = 0.20, indicate that the a priori strict model is a poor fit to these data. Next, configural structural invariance is evaluated across the three samples (or group structure) by only imposing the constraint that the factor structure is equivalent across the three samples (i.e., item loadings, intercepts and residual variances are permitted to vary across samples). The model fit statistics for the configural invariance model, χ2(1072) = 24,733.9, p < .001; CFI = 0.12, TLI = .07, RMSEA = 0.35, SRMR = 0.21, indicate that the configural invariance model is also a poor fit to these data (Table 1). The fits of the strict (BIC: –7595) and the configural invariance models (BIC: –7132) are not significantly different (χ2[718] = 762.8, p = .12). As the configural invariance model had poor fit, metric invariance with further equivalence constraints on item loadings is simply included as a reference in Table 1 which contains the complete model fit statistics.

Table 1.

Model Fits Statistics Across Confirmatory Factor Analysis (CFA) and Exploratory Structural Equation Modeling (ESEM).

	χ2	DF	p-value	RMSEA	SRMR	TLI	CFI	AIC	BIC
Strict CFA(Reference)	23,971.2	354	<.001	0.35	0.2	0.06	0.12	–7940	–7595
Config MG-CFA	24,733.9	1072	<.001	0.35	0.21	0.07	0.12	–8125	–7132
Metric MG-CFA	24,856.7	1104	<.001	0.34	0.21	0.1	0.12	–8066	–7211
Overall ESEM	23,358.4	323	<.001	0.36	0.18	0	0.14	–8491	–8011

Config = Configural; MG = Multi-group; CFA = Confirmatory Factor Analysis; DF = Degrees of Freedom; RMSEA = root mean squared error of approximation; CFI = comparative fit index, SRMR = standardized root mean squared residual.

Based on the parameters in the strict and configural invariance models, as specified in Fig. 1B, the manifest variables do not form the latent constructs of approach or avoidance. For the strict CFA model (supplemental Section 2.2.1, Table S5 and Fig. S8A), 9 of 16 (56 %) manifest variables that are hypothesized to form the latent approach construct have βs ≤ .30. The pattern of low loadings is similar for the latent Avoidance construct, also in the strict model. These results indicate that the covariance structure in these data does not align with the a priori theoretical structure for an approach and avoidance measurement model.

4.4.2. Aim 1b: evaluating the exploratory structural equation model across the ABCD, AHRB and MLS samples

In Aim 1b, ESEM is used to fit an a priori factor model. Unlike the CFA model in Aim 1a, the ESEM allows for empirically driven non-zero cross-loadings. This is a less-restrictive model than the CFA, and so, the cross-loadings may improve the model misspecifications in this semi-restrictive framework. Like Aim 1a, model fit statistics are used to evaluate whether the model was empirically supported and consistent across the three samples. In the combined mean signal intensity data, as indicated by the fit statistics, χ2(323) = 23,358.4, p < .001, CFI = 0.14, TLI = 0.00, RMSEA = 0.36, SRMR = 0.18, the strict ESEM is a poor fit to the data. Also, the ESEM (BIC = –8011) is a poorer fit to the data than the strict CFA model (BIC = –7595), χ2(31) = 612.8, p < .001. Taken together, the CFA and ESEM provide empirical evidence that the a priori model, as specified in Fig. 1B, is a poor representation of the covariance structure for the twenty-eight manifest variables from the MID task fMRI data across the three samples. Thus, the hypothesis that the mean signal intensities from bilateral NAcc and AI from the two BOLD runs across the contrasts would load onto two factors, approach and avoidance, is not supported.

4.5. Aim 2: evaluating the data-driven specified structure of the MID task

Aim 2 uses an EFA (unrestricted) model to estimate the number of factors and the item loadings in a measurement model for the contrasts and brain regions obtained from the MID task across the ABCD, AHRB and MLS samples.

The heatmap in Fig. 4 represents the product-moment correlations among the mean signal intensity values for the bilateral NAcc and AI from the two BOLD runs across the six contrast types in the ABCD, AHRB and MLS samples. A visually comparable structure across the three samples can be observed. Based on parallel analyses and scree plots, a parsimonious six factors best represent the covariance matrices for the twenty-eight manifest variables in the ABCD and AHRB data, and seven factors best represent the covariance matrix in the MLS data. Results from the EFA are shown in Fig. 5; factor structure and loadings in Fig. 5A and the Φ factor intercorrelation matrix in Fig. 5B. (for example of structure and loadings diagram for ABCD, see supplemental Fig. S8B). Across the three samples, separate factors are indicated by the Run 1 and Run 2 measures of the contrasts for the bilateral NAcc. Similarly, the Run 1 and Run 2 measures of the contrasts for the bilateral AI are indicators of different factors.

Fig. 5.

(A) Exploratory Factor Analysis loadings for the ABCD, AHRB and MLS samples, loadings < .20 are removed from the EFA results; and (B) Factor Intercorrelation matrix (Φ). AD = ABCD; AB = AHRB; MS = MLS; R1 = Run 1, R2 = Run 2. L/R-Nacc = Left/Right Nucleus Accumbens; L/R-Ins = Left/Right Anterior Insula.

The quantitative similarity of the factors obtained from the EFA models in Aim 2 are reported in Fig. 6. Fig. 6A consists of factor labels as output by the data-driven process implemented in fa(). Fig. 6B, is expanded on in detail here, consists of the relabeled factors based on qualitative/quantitative similarity. The data-driven factors identified for the ABCD and AHRB data are highly similar, as indicated by the high congruence of the pattern of factor loadings. For example, both ABCD and AHRB contain factors that are highly congruent (see Fig. 6B for heatmap and supplemental Section 2.2.2, Table S7 for Congruence matrix) which are indicated by Run 1 NAcc Reward (AD4 ∼ AB4, r = .95), Run 2 NAcc Reward (AD3∼AB3, r = .96), Run 1 AI Reward versus Neutral (AD2 ∼ AB2, r = .96), Run 2 AI Reward versus Neutral (AD1 ∼ AB1, r = .97), and negative/positive loadings of Reward and Loss combinations for Run 1 (AD5 ∼ AB5, r = .94) and Run 2 (AD6 ∼ AB6, r = .95). Likewise, there is factor congruence between the MLS data and ABCD/AHRB factors as well. Specifically, Run 2 NAcc Reward (AD3 ∼ MS3, r = .98; AB3 ∼ MS3, r = .97), Run 2 AI Reward versus Neutral (AD1 ∼ MS1, r = .92; AB1 ∼ MS1, r = .91). However, the structure and loadings for the MLS results do differ from ABCD/AHRB. For example, the simple, single factor indicated by the Run 1 AI Reward in ABCD/AHRB (AD2, AB3), is distributed across two factors in the MLS data MS2 and MS5 (AD2 ∼ MS2, r = .79; AD2 ∼ MS5, r = .51; AB2 ∼ MS2, r = .90; AB2 ∼ MS5, r = .45). Consistent with findings from Aim 1, across the three samples, the EFA results suggest that the factor structure underlying the multidimensional associations among NAcc and AI regions across the six contrasts for the two runs in these data is not the a priori theoretical structure that is hypothesized in Fig. 1B. Rather, mean signal intensity in these regions and contrasts appear to be differentiated more by ROI (NAcc versus AI) and run (Run 1 versus Run 2) than approach and avoidance latent constructs.

Fig. 6.

Factor Congruence across (A) data-driven factors for ABCD (AD), AHRB (AB) and MLS (MS) samples and (B) relabeled data-driven factors based on multidimensional and qualitatively similarity.

4.6. Aim 3: effects of pubertal development on the ABCD EFA results

The ABCD sample contains participants that are at different stages of pubertal development. Since pubertal development coincides with changes in motivational processing during adolescence (Forbes and Dahl, 2010), LSEM is used to evaluate whether the factor loadings change across the self-reported pubertal development in the ABCD sample.

The ABCD EFA structure and loadings results from Aim 2 are used to inform the a priori model for the held-out sample in the LSEM analyses. A parsimonious four factor model is used that reflects an (1) Run 1 Avoidance AI, (2) Run 2 Avoidance AI, (3) Run 1 Approach NAcc and (4) Run 2 Approach NAcc. Based on the highly correlated residual variance within runs for several contrasts for the EFA results from Aim 2, in four cases for the NAcc and two cases of the AI, the a priori model included correlated residuals. While not having an ’excellent’ model fit based on the cut-offs, the held-out sample model fit statistics are meaningfully improved over the Aim 1 models, χ2(240) = 2117.1, p < .001, CFI = 0.77, TLI = 0.73, RMSEA = 0.15, SRMR = 0.10 (∆CFI = 0.65, ∆TLI = 0.67, ∆RMSEA = –0.20, ∆SRMR = –0.10). As we did not pre-register additional model fit improvement procedures, extra steps are not taken to prevent overfitting of the data in a sample size that may be insufficient to estimate all parameters (e.g., loadings, variance and residual covariances).

Based on the LSEM model, there is little consistent evidence that the parent or youth reported pubertal development moderates the factor loadings. In the parent LSEM permutation model (bootstraps = 1000), although the loading of the Right Big Win versus Big Lose Insula for Run 2 onto the NAcc Run 2 factor decreases across levels of PDS (p = .04), all other estimates are not significant (p > .05). Then, in the youth LSEM permutation model (bootstraps = 1000), while Right Big Win versus Big Lose Insula parameter loading does not significantly decrease across levels of PDS, the loading of Left/Right All Lose versus Neutral Insula for Run 2 onto the Insula Run 2 factor decreases significantly across levels of the PDS, all other estimates are not significant (p > .05). The global fit statistics are reported in supplemental Section 2.2.3, Fig. S9.

4.7. Sensitivity analyses

Several sensitivity analyses were pre-registered. First, to complement the LSEM analyses of whether pubertal stage moderated the factor loadings, a sensitivity analysis is performed to check for measurement invariance of ABCD EFA structure across sex (Aim 3). Second, with multiple samples being tested at the University of Michigan site, the CFA and EFA results were re-run for only the University of Michigan site. Third, the effect of sampling on the CFA fit statistics and the EFA number of factors in the ABCD data is evaluated.

4.7.1. Measurement invariance in ABCD for males/females

Overall, the factor structure is comparable across males and females. Fit statistics indicated comparable model fit to the data in the configural invariance model across sex, χ2(484) = 2444.7, p < .001, CFI = 0.76, TLI = 0.73, RMSEA = 0.15, SRMR = 0.11, to the strict CFA model for the held-out data, χ2(240) = 2133.0, p < .001, CFI = 0.77, TLI = 0.73, RMSEA = 0.15, SRMR = 0.10. Based on ANOVA comparisons, the sex configural model (BIC = –.7253.4) fits the data better than the strict held-out model (BIC = 7603.4), χ2(244) = 311.6, p = .002.

4.7.2. Effect of University of Michigan site on CFA and EFA results

The pattern of findings are consistent when only analyzing AHRB/MLS data from the University of Michigan (UM) site. The fit of the a priori model to the empirical data for the CFA model is similarly poor. For the CFA model, the strict and configural CFA models are a poor fit to the data (Strict: χ2(354) = 8843.5, p < .001, CFI = 0.13, TLI = 0.07, RMSEA = 0.34, SRMR = 0.20; Configural: χ2(713) = 9229.9, p < .001, CFI = 0.13, TLI = 0.08, RMSEA = 0.34, SRMR = 0.21). For the EFA analysis, the ABCC data did not have usable data for University of Michigan site (site 13), so the EFA analyses were not re-ran for ABCD UM only site. Thus, the comparison between UM sites is the same as is obtained for AHRB and MLS samples (see Fig. 4 and 5), in that the EFA identified the same 6-factor structure as found with the larger data.

4.7.3. Effect of sampling on CFA model fit and EFA factors

Sampling within the held-out ABCD sample, there is little effect on the CFA model fit statistics or the recommended factors for the EFA analysis. Specifically, across the bootstrapped datasets (N = 300) the RMSEA (M =.36, SD =.01) & SRMR (M =.21, SD =.01) had minimal variability. There is wider variability in the AIC (M = –4473.9, SD = 352.9) and BIC statistics (M = –4165.9, SD = 352.9). Finally, across the bootstrapped datasets the minimum and maximum recommended number of factors for the EFA are 6 and 7 (Mean: 6.2), respectively. The distribution of these estimates are reported in supplemental Section 2.3, Fig. S10.

4.8. Post hoc analyses

In the Aim 1 and Aim 2 results, the factor structure and covariance matrices indicate that the a priori ROI and contrast variables are more congruent within runs than across runs. To better understand this finding, run specific (i.e., Run 1 and Run 2) mean-level group maps for the Big Win > Neutral anticipation contrast are estimated for each of the three samples using the run level estimates in a group level mixed effect model. The difference [Run 1 - Run 2] is calculated between the resulting maps. Fig. 7A-C reports the difference between the group-level run maps for each sample. Across the three samples, the magnitude of the activity decreases from Run 1 and Run 2. To evaluate the voxelwise similarity of Cohen's d estimates between Run 1 and Run 2, pairwise_similarity() is used as implemented in the PyReliMRI package (Demidenko and Poldrack, 2023). The Spearman’s rank (rho) correlation coefficient between the run maps is reasonably high (Fig. 7D), ABCD Run 1 ∼ Run 2 rho = .82, AHRB Run 1 ∼ Run 2 rho = .72, and MLS Run 1 ∼ Run 2 rho = .80. This demonstrates that, on average, the group-level activation is relatively stable across samples and runs. However, the change in between-subject scores across runs may have substantially reduced the association between factors for each run.

Fig. 7.

Differences in estimated Cohen’s d group activity between Runs [Run 1 - Run 2] for (A) ABCD, (B) AHRB and (C) MLS samples, and (D) pairwise similarity (Spearman’s rho) between run Cohen’s d maps.

4.8.1. Protocol deviation for GE scanners

The Stage 2 analyses did not include the ABCD GE scanner data. The ABCC data contained an error that resulted in faulty fieldmap data for all GE scanners. As of October 2023, the fix is not implemented and so the data are not usable. To ensure the difference in sampling and the sample size in ABCD data does not impact the above CFA and EFA results, a sensitivity analysis is performed. The primary (N = 346) and held-out (N = 347) ABCD data were combined into a larger dataset (N = 693). Then, the combined ABCD data were randomly sampled from N = 50 to N = 1000 (intervals of N = 10). CFA fit statistics and the recommended number of factors were calculated at each sample size. Besides the AIC/BIC fit statistics which linearly scale with sample size, all other fit statistics did not meaningfully differ and the recommended number of factors in the ABCD data are stable at six across the range of sample sizes (see supplemental Section 2.3.2, Fig. S11).

5. Discussion

The theoretical dimensions of the approach and avoidance systems in the MID task (Knutson et al., 2001) have been extensively discussed (Knutson et al., 2014, Knutson and Greer, 2008, Knutson and Srirangarajan, 2019) and characterized in meta-analyses of group-level maps (Balodis and Potenza, 2015, Chen et al., 2022, Oldham et al., 2018). However, it is not yet understood whether the between-subject covariance structure supports this framework. The current empirical study uses a measurement approach to evaluate and understand the structure of the MID task across three independent adolescent samples. The [three] key findings are: (1) during the anticipation of the probe for gain and loss trials, the MID task consistently activates the bilateral AI and dorsal/ventral striatal regions in group-level maps; (2) across the a priori ROIs and contrasts the model fit statistics from CFA do not support the multidimensional structure for approach and avoidance system in the data; however, (3) there is a consistent structure in the data that is run and region specific. The results and interpretations are largely consistent in the three independent adolescent samples with comparable fMRI tasks across multi-band (ABCD & AHRB) and single-band (MLS) MRI acquisition protocols.

The MID task activates regions that are consistent with the prior literature (Chen et al., 2022, Oldham et al., 2018) but effects are attenuated in multi-band data. Specifically, across the ABCD, AHRB and MLS samples, activation is observed in the bilateral ventral striatum, AI, supplementary motor cortex and occipital regions during the Big Win versus Neutral contrast. However, Cohen's d estimates for neural activity are larger in the MLS (single-band) than the ABCD/ABCD samples (multi-band). While the observed effect size for neural activity for the Big Win > Neutral contrasts is largest in the MLS sample, the decrease in magnitude of the effect from Run 1 to Run 2 is also largest in the MLS data. The higher effect size in single- versus multi-band data is consistent with prior work reporting that increased multi-band factors may reduce the effect size in subcortical regions due to increased noise (see Fig. 2 in Risk et al., 2021, p. 7) and this may impact the neural activity during the MID task (Srirangarajan et al., 2021). To an extent, our results confirm that the estimated neural effects in subcortical regions may be lower in multi-band acquisitions. Nevertheless, the proportion of the difference attributable to the multi-band factor is unclear in these data. Specifically, the MLS sample (single-band) had larger inherent smoothness in the data, a different trial randomized scheme that holds constant across runs (as opposed to trials randomized for each run as in ABCD/AHRB) and higher probe accuracy/faster mean response times during the performance of the MID task. As a result, higher smoothness in the data may produce a larger effect as a result of the increased signal-to-noise ratio (Mikl et al., 2008) but come with lower spatial specificity (Blazejewska et al., 2019). It is unclear, however, how task randomization and performance may increase or decrease group-level activity across runs in the MID task design. This highlights the nuanced differences in group-level estimates of neural activity across samples that needs to be better understood as a function of MRI acquisition and task administration.

When evaluating whether the between-subject covariation in the neural data supports the approach and avoidance systems multidimensional structure as proposed in Knutson et al. (2014), the models demonstrate a poor fit to these data. The measurement model includes the a priori bilateral NAcc and AI brain regions from six contrasts across two separate MID task runs (as specified in the diagram in Fig. 1) from the ABCD, AHRB and MLS sampled BOLD data. Neither the strict model nor the configural model (only constraining the factor structure) fit the data well. Sensitivity analyses evaluating the model solely at the University of Michigan site (to keep facility/scanners constant) also demonstrate that neither the strict nor the configural model fit the data well. While the group-level neural activity is consistent with reported meta-analytic results for the MID task (Oldham et al., 2018), the covariance structure of the data combined from three samples is inconsistent with the theoretical framework (Knutson et al., 2014). This suggests that the pre-specified multidimensional structure of the approach and avoidance manifest variables is not an adequate explanation of the phenomenon evoked during this modified version of the MID task in these three adolescent samples.

The lack of evidence for the multidimensional structure of the approach and avoidance framework may be due to both methodological and conceptual challenges. Methodologically, the modified version of the MID task in the current analyses cannot delineate anticipatory and action preparation/probe preparation processes (Richards et al., 2013). This is a limitation observed in many MID tasks that use short (e.g., 250 ms; Cao et al., 2019; Knutson et al., 2003) or long cue designs (e.g., 2000 ms (Casey et al., 2018, Yau et al., 2012). In other words, after observing the trial type, participants either immediately or after a brief delay prepare to respond. While the current study models ‘anticipation’ as the entirety of the Cue and Fixation period, there may be modeling decisions that may improve the delineation of the incentive salience and action preparation. Conceptually, conjunction analyses of published BOLD data have reported that there is meaningful overlap between positive and negative rewards in the AI and striatal regions (Bartra et al., 2013). Overlap in the AI may be due to its role in executive control pertaining to attention and cognitive control (Molnar-Szakacs and Uddin, 2022) and overlap in the NAcc may reflect the fact that there are context specific responses in appetitive and avoidant processes (Berridge, 2019). These challenges may reduce the precision with which small effects of approach and avoidance processes may be disentangled in the data.

Despite the lack of support for theoretical framework in the data, the EFA demonstrates consistent structure that is modestly-to-extremely congruent across the samples. Across the three adolescent samples, the covariance structure of the twenty-eight variables of the mean signal intensity values is best represented by six (in ABCD & AHRB) or seven factors (in MLS). These data-driven factors delineate runs and brain regions. Specifically, the Left and Right NAcc or AI mean signal intensity values load onto independent factors for Run 1 and Run 2. This finding is consistent across the ABCD, AHRB and MLS samples. In fact, factor congruence demonstrates that the run and region-specific factors have a high degree of structural similarity between the ABCD and AHRB samples (r = .94 –.97) and moderate-to-high degree of structural similarity between the ABCD/AHRB and MLS samples (r = .51 –.98). However, across each of the three samples there is little structural similarity between manifest variables for Run 1 and Run 2. It is unsurprising that the within run data are highly correlated as there is a high degree of overlap in stimuli used in the a priori contrasts (i.e., Big Win) and evidence from the multivariate pattern analysis literature reports that there is a high degree of similarity between stimuli within runs (Mumford et al., 2014). The weak covariance between runs is surprising but consistent with the literature reporting low test-retest reliability in univariate multi-session and multi-run task fMRI data (Elliott et al., 2020). The preserved structure but lack of run congruence poses two conceptual questions: How does averaging weakly related run variables modify the covariance structure of the data; and are there analytic decisions that may increase reliability of between-person differences across runs while preserving the factor structure?

The factor structures identified in the current study are consistent across pubertal development, site effects and sampling variability. First, the loadings in the run and region-specific factors are largely consistent across the levels of pubertal development, for both child and parent reports, in the held-out ABCD data. Second, as briefly mentioned above, when the CFA/EFA and EFA analyses are constrained to only the University of Michigan site for the AHRB and MLS samples, the pattern of results are consistent. Finally, when randomly subsampling the ABCD data, the fit statistics for the CFA and the recommended number of factors for the EFA are largely consistent. This demonstrates that the CFA and EFA results for the current data are insensitive to these parameters. However, the analyses are based on data collected in adolescence. Complimentary work should also evaluate whether and how the factor structure manifests in adulthood and other phases across the lifespan.

The lack of between-subjects covariation between the bilateral NAcc and AI mean signal intensity values is surprising given the consistent activation in these brain regions at the group-level. One explanation may be that the increased noise in the multi-band data (Risk et al., 2021) may have modified the structure of covariances among cortical and subcortical regions. However, the covariance pattern in the a priori brain regions is consistent in the single-band (MLS) and multi-band (ABCD/AHRB) MID task fMRI data. A second explanation for the lack of support for theoretical framework may be that the covariation may be present at a more granular level, specifically, at the voxel or ROI time-series data. However, in an independent analysis of the AHRB MID data (Demidenko, Huntley et al., 2022), no evidence is reported for group or subgroup paths in combined and run specific time-series that demonstrate model specified (group iterative multiple model estimation) covariation between the bilateral insula and ventral striatum ROIs.

Evidence from CFA and EFA model results obtained here do not provide sufficient support to accept the approach and avoidance framework defined in Knutson et al. (2014) to explain the between-subject differences in univariate neural activity. While this does not undermine the experimental effect of the anticipatory phase in the MID task across the adolescent samples, the results do demonstrate that the measurements diverge from the theoretical framework. As discussed in the introduction, the scores obtained from a univariate analysis during the MID task are a manifestation of between-person differences in what is a presumed phenomenon of motivation to approach rewarding stimuli. While subjecting biological process measures to modern psychometric procedures may not always be well-reasoned, if brain-behavior researchers elect to use neural activity elicited during the MID task in correlational research it is prudent to provide a sufficient description of the measured mental process (Francken et al., 2022) and leverage tools from the psychological testing literature to determine how the proxy score for ‘motivational processing’ derived from neural activity during the MID task supports an interpretation for different populations and testing conditions (Association American Educational Research et al., 2014, Slaney, 2017). Without a well delineated framework, the approaches used to explain why activation in one group and a given contrast may differ from another group may lack theoretical grounding. Despite the lack of support for the theoretical framework, there is evidence of a relatively consistent and converging structure from the MID contrast data. Refining the interpretability of the underlying BOLD signal and the resulting theoretical framework for the MID task may be helpful as it continues to be used in large studies focused on individual differences research (e.g., IMAGEN [Schumann et al., 2010] and ABCD study [Casey et al., 2018]). Future research may explore what phenotypes these features are consistently related to and how they explain pre- and/or post-scan affective ratings across developmental samples. Further probing the intercorrelations among the data-driven factors, self-report and behavioral data would help elaborate how these factors contribute to between-subject differences.

5.1. Study limitations

The analyses here are based on pre-specified data preprocessing and analytic decisions. It is well documented that there are a wide-range of pipeline decisions which can impact the resulting estimates (Botvinik-Nezer et al., 2020, Bowring et al., 2022, Churchill et al., 2015, Li et al., 2021). As a result, decisions at the preprocessing and/or model fitting approach may have meaningfully impacted some of the results and interpretations. However, a strength of our study is that it leverages registered reports (Chambers et al., 2014), whereby the conceptual and analytic framework is reviewed prior to the access to the data and subsequent analyses. Nevertheless, future research that uses alternative preprocessing and modeling decisions would help confirm the reproducibility of these results and interpretations.

The modified version of the MID task used in the current study maintains limitations from earlier designs (Knutson et al., 2003) that do not allow for temporally separating the BOLD response between trials. Specifically, the MID task is a rapid event-related design whereby the interstimulus interval is shorter than the hemodynamic response function (Soares et al., 2016). This is confounded by the lack of an intertrial interval between the current trials (t) feedback cue (i.e., participants read the feedback of whether they are successful in responding to the probe and the amount they did/didn’t win/lose) and the onset of the subsequent trial (t + 1). Given the slow properties of the BOLD signal (Buxton, 2012), this poses challenges in deconvolving the signal in traditional GLM models. As a result, this may have contaminated the properties of the “anticipatory” signal and thus the validity of the interference.

Third, a subset of the MID contrasts and specific selection of ROIs from Harvard-Oxford subcortical brain atlas are used in the current analyses. The MID task includes a range of contrasts and modeling decisions. Likewise, the use of the Harvard-Oxford subcortical brain atlas to extract the bilateral NAcc and AI alternative to a different atlas may have a meaningful impact on the resulting estimates (Bryce et al., 2021). While the typical contrasts and a standard deterministic atlas are used in these analyses, perhaps the data may be represented differently in an EFA model if these decisions were varied and/or the variable expanded. However, expanding the number of manifest variables for EFA may increase the uncertainty around the estimates at small samples.

Finally, GE scanner data are not included for these analyses which may impact some of the estimates. While CFA fit statistics and EFA numbers of factors are stable across sample sizes, it is feasible that the structure of loadings may slightly differ in the GE data as these data may include different types of participants, sites and noise structure. Furthermore, the LSEM results may also be impacted because of the reduced sample size. While this may not impact the interpretations/conclusions in the current manuscript, the estimates should be considered in the context of this limitation. Nevertheless, this major deviation is fully documented, and sensitivity analyses were performed to evaluate the robustness of the conclusions (Chambers and Tzavella, 2022).

6. Conclusion

Modern psychometric tools have not been used to evaluate measurement properties of the fMRI signals obtained during participants’ performance of the MID task. The current study uses a comparable MID task design across three adolescent samples to evaluate multiple theoretical and data-driven measurement properties of fMRI for twenty-eight fMRI-based manifest variables. Across the three samples, group-level neural activity during the MID anticipatory phase activated expected regions: bilateral NAcc and AI. Evaluation of an a priori multidimensional approach and avoidance systems framework using CFA provided no support for the a priori framework based on twenty-eight manifest variables across three adolescent samples. However, data-driven EFA models identified a six/seven factor structure of between-person covariance that is congruent across samples, where the twenty-eight manifest variables are indicators of run and bilateral brain region specific factors. Future research should evaluate the phenotypes that best characterize the covariance structure in the MID task during fMRI to enhance the interpretability of brain-behavioral findings within and between adolescent samples.

Data & code availability statement

The ABCD-BIDS input data can be accessed through the ABCD-BIDS Community Collection with an established Data Use Agreement (see https://abcdstudy.org/). The BIDS version of the MLS and AHRB data can be obtained with the approval of Dr. Mary Heitzeg and/or Dr. Daniel Keating, respectively.

R code

The .html file containing the description of the function, analyses, and associated code is named ‘ABCD_Analysis_CFA-ESEM-EFA-LSEM.html’ located within the Stage 2 Github folder (associated .rmd is also provided). If using R version > 4.1 the script should run from start to finish.

Python code

The .py script to run the first level analysis with Nilearn’s implementation on the MID data is named ‘mid_firstlevel.py’ and is located in the Stage 2 Github folder. The .py script to (1) extract region of interest (ROI) mean signal intensity estimates from the specified contrast beta maps and (2) calculate the voxel overlap between the NAcc ROI and participants' brain mask is named ‘betamap_roi.py’ and is located in the Stage 2 Github folder. (Note: steps for acquiring ROI parcellations is not included in the script but a text file named ‘ROI_create.txt’ is provided in the Stage 1 Github folder). The extracted values for a given subject, run, contrast and ROI, as implemented using Nilearn’s functions, were compared for 3 random files using fslmeants -i {beta_map.nii.gz} -m {roi.nii.gz} to confirm values were identical. The Second Level fixed effect model (estimates across runs) and group level model (estimates across subjects) is implemented using Nilearn’s functions in the script 'mid_fixedeff.py' and ‘mid_sgroup.py’, respectively, and is located in the Stage 2 Github folder.

CRediT authorship contribution statement

Mumford Jeanette A.: Methodology, Writing – original draft, Writing – review & editing. Demidenko Michael: Conceptualization, Formal analysis, Methodology, Project administration, Visualization, Writing – original draft, Writing – review & editing. Ram Nilam: Methodology, Writing – original draft, Writing – review & editing. Poldrack Russell A.: Conceptualization, Methodology, Supervision, Writing – original draft, Writing – review & editing.

Declaration of Competing Interest

The authors declare that they have no conflicts of interest.

Appendix A. Supplementary material

Supplementary material.

.