Limits to the generalizability of resting-state functional magnetic resonance imaging studies of youth: An examination of ABCD Study® baseline data

Abstract

This study examined how resting-state functional magnetic resonance imaging (rs-fMRI) data quality and availability relate to clinical and sociodemographic variables within the Adolescent Brain Cognitive Development Study. A sample of participants with an adequate sample of quality baseline rs-fMRI data containing low average motion (framewise displacement ≤ 0.15; low-noise; n = 4,356) was compared to a sample of participants without an adequate sample of quality data and/or containing high average motion (higher-noise; n = 7,437) using Chi-squared analyses and t-tests. A linear mixed model examined relationships between clinical and sociodemographic characteristics and average head motion in the sample with low-noise data. Relative to the sample with higher-noise data, the low-noise sample included more females, youth identified by parents as non-Hispanic white, and youth with married parents, higher parent education, and greater household incomes (ORs = 1.32–1.42). Youth in the low-noise sample were also older and had higher neurocognitive skills, lower BMIs, and fewer externalizing and neurodevelopmental problems (ds = 0.12–0.30). Within the low-noise sample, several clinical and demographic characteristics related to motion. Thus, participants with low-noise rs-fMRI data may be less representative of the general population and motion may remain a confound in this sample. Future rs-fMRI studies of youth should consider these limitations in the design and analysis stages in order to optimize the representativeness and clinical relevance of analyses and results.

Introduction

Head motion presents significant challenges for resting-state functional magnetic resonance imaging (rs-fMRI) studies, especially those involving youth (Engelhardt et al., 2017), and may relate to participant characteristics. Previous studies have demonstrated increased motion in individuals with neurodevelopmental disorders (Tyszka et al., 2014), males, and those with higher body mass indices (BMIs) and externalizing symptoms (Ekhtiari et al., 2019). Because of this, motion-correction methods may inadvertently bias samples and results (i.e., if motion relates to an outcome of interest). Indeed, previous rs-fMRI findings, such as children having decreased connectivity in the default mode network compared to young adults, have been linked to motion artifact (Power et al., 2015). These issues are especially relevant for large studies like the Adolescent Brain Cognitive Development (ABCD) Study that aim to include participant samples with sufficient clinical and demographic variability to support identification of relationships that are meaningful and generalizable. If motion relates to sociodemographic characteristics and/or health conditions, this may provide an additional obstacle for accomplishing these goals. It is especially important to recognize that health disparities driven by systemic social, economic, and environmental disadvantage are disproportionately prevalent in marginalized populations (HHS, 2011), potentially raising concerns for the representativeness of rs-fMRI data selected for low noise (i.e., selection bias) and how such issues could lead to inappropriate conclusions.

The longitudinal ABCD Study® is collecting psychosocial, neurocognitive, and neuroimaging data from nearly 12,000 youth throughout adolescence and has a primary focus on the development of substance use and mental health (Volkow et al., 2018). The imaging protocol includes either three or four 5 min rs-fMRI scans per subject, which are processed for curated data releases using a standardized pipeline (see Hagler et al., 2019 for details). All scanners are 3 Tesla and include Siemens Prisma, General Electric 750, and Phillips models. There may be discrepancies in the amount of motion across study sites, however, due to only those with Siemens scanners implementing real-time motion monitoring (Hagler et al., 2019).

With the ABCD Study data release 3.0, a recommended sample for rs-fMRI analyses was identified by the Data Analysis, Informatics, & Resource Center (DAIRC) using a binary variable (‘imgincl_rsfmri_include’) indicating whether or not participants had an adequate amount (i.e., >375 frames) of quality rs-fMRI data (i.e., no serious clinical findings, passed automatic and manual quality control) that has undergone filtering for motion-related distortions (i.e., B0 Unwarp; ABCD Human Subjects Study, 2020). This variable identifies a sample of 9,387 with varying rates of average motion. Thus, researchers may also wish to apply an average motion threshold. However, it has not been examined whether 1) there are clinical or sociodemographic differences in participants with low-noise data (i.e., recommended for analysis with low average motion; LN) versus those without an adequate sample of quality data and/or containing high average motion (i.e., higher-noise data; HN) and 2) motion remains a confound relating to participant characteristics in LN. The present study involved an initial exploration of these issues using baseline ABCD Study data to determine the representativeness of these samples, with a particular focus on characteristics that may be driven by systemic inequities.

Methods

Study design

The design of the larger ABCD Study has been described elsewhere (see Casey et al., 2018 for a description of the neuroimaging protocol). For the present study, participants were between the ages of 9.0 and 10.9 years. See Table 1 for additional sample characteristics. Of the recommended sample, only participants with low average motion indexed by a commonly used motion measure, mean framewise displacement (FD; Power et al., 2015), were included in the LN sample (FD ≤ 0.15 mm; n = 4,356). Thus, participants who were not recommended for analysis or had FD > 0.15 mm were included in the HN sample (n = 7,437; see Supplemental Fig. 1 for an overview of sample creation). Notably, the FD threshold was chosen to ensure motion effects had limited impact on LN data. The analyses in the present report were also conducted in a sample of participants with a less conservative threshold (FD ≤ 0.20 mm). Results were very similar to those herein and can be found in Supplemental Tables 3 and 4.

Table 1.

Demographic and physical/mental health characteristics of the low-noise sample (LN; n = 4,356) and higher-noise sample (HN; n = 7,437) in the ABCD Study

	LN % (Z)	HN % (Z)	X2 (df)	p	OR 95% CI
Sex at birth			80.71 (1)	< 0.001	1.24 - 1.40
Female	53 (5.17†)	45 (−3.96†)
Male	47 (−4.94†)	55 (3.78†)
Race/Ethnicity			130.84 (4)	< 0.001	1.34 - 1.51
Non-Hispanic White	58 (5.17†)	49 (−3.96†)
Non-Hispanic Black	11 (−6.92†)	17 (5.30†)
Hispanic	19 (−2.63#)	21 (2.01*)
Non-Hispanic Asian	2 (−0.70)	2 (0.53)
Other	11 (0.68)	10 (−0.52)
Parent education			130.47 (4)	< 0.001	1.34 - 1.51
Less than HS	4 (−4.23†)	6 (3.24#)
HS diploma/GED	8 (−4.05†)	11 (3.10#)
Some college	23 (−3.88†)	28 (2.97#)
Bachelor’s degree	27 (2.03*)	24 (−1.56)
Graduate degree	39 (5.37†)	31 (−4.11†)
Parent marital status			92.86 (1)	< 0.001	1.27 - 1.43
Married	73 (4.34†)	65 (−3.33†)
Not married	27 (−6.32†)	35 (4.84†)
Annual household income			94.55 (2)	< 0.001	1.28 - 1.45
<$50 K	24 (−6.07†)	33 (4.69†)
≥$50 K - <$100 K	29 (0.47)	28 (−0.37)
≥$100 K	47 (4.70†)	39 (−3.64†)
	M (SD)	M (SD)	t (df)	p	Cohen’s d 95% CI
Age in months	120 (8)	118 (7)	12.63 (11,791)	< 0.001	0.20 - 0.28
Neurocognitive component scores
General cognition	102.81 (14.80)	98.42 (14.84)	15.00 (10,963)	< 0.001	0.26 - 0.34
Executive function	102.45 (14.11)	98.63 (15.27)	13.32 (9110)	< 0.001	0.22 - 0.30
Learning and memory	102.42 (14.63)	98.64 (15.02)	12.84 (10,963)	< 0.001	0.22 - 0.29
CBCL dimensional scale scores
Internalizing symptoms	50.09 (10.01)	49.95 (10.01)	1.32 (11,785)	0.186	−0.01 - 0.06
Somatoform symptoms	50.14 (9.96)	49.93 (10.04)	2.57 (11,785)	0.010	0.01 - 0.09
Detachment symptoms	49.63 (9.65)	50.22 (10.21)	−2.87 (11,785)	0.004	−0.09 - −0.02
Neurodevelopmental symptoms	48.78 (9.16)	50.71 (10.40)	−10.52 (9323)	< 0.001	−0.24 - −0.16
Externalizing symptoms	49.21 (9.29)	50.45 (10.35)	−6.49 (11,784)	< 0.001	−0.16 - −0.09
BMI	18.14 (3.57)	19.11 (4.15)	−12.68 (9504)	< 0.001	−0.28 - −0.20
Sleep problems	36.22 (7.99)	36.73 (8.36)	−3.23 (11,759)	0.001	−0.10 - −0.03
FD	0.09 (0.03)	0.42 (0.33)	−141.55 (11,167)	< 0.001	−2.67 - −2.58

^*p < 0.05

^#p < 0.01

^†p < 0.001

Note. Z, standardized (z-scored) residuals; HS, high school; CBCL, Child Behavior Checklist; BMI, body mass index; FD, average framewise displacement. The “Other” Race/Ethnicity category includes those who were identified by their parent as multi-racial, American Indian/Native American, Alaska Native, Native Hawaiian, Guamanian, Samoan, Other Pacific Islander, Asian Indian, Chinese, Filipino, Japanese, Korean, Vietnamese, Other Asian, or Other Race. Neurocognitive component scores (from Thompson et al., 2019) were scaled to have a mean of 100 and SD of 15 to improve interpretability. CBCL scales were derived from a factor analysis of baseline ABCD data (Michelini et al., 2019) and were created by summing the scores for each item that loaded onto the factor. Scores were scaled to a T-score metric (i.e., mean = 50, SD = 10) to aid with interpretability. Prior to analysis, CBCL scores, BMI, and sleep problems were log-transformed

Variables of interest for the present study included those previously implicated in studies of motion in rs-fMRI and/or commonly used as covariates in ABCD Study analyses. Demographic variables were age; sex; parent-reported race/ethnicity; and parental education, marital status, and income. Categories in the race and ethnicity variable (i.e., ‘race_ethnicity’ in the ABCD Study data release) were based on the US Office of Management and Budget’s Revisions to the Standards for the Classification of Federal Data on Race and Ethnicity and consisted of the following categories, reflecting a combined ethno-racial construct: non-Hispanic white, non-Hispanic Black, His-panic/Latinx, non-Hispanic Asian, and a final category labeled ‘Other’ that included multi-racial participants and/or participants in additional categories that were not further disaggregated due to small sample sizes. Physical/mental health variables included body mass index (BMI), total sleep problems (from The Sleep Disturbance Scale for Children; Bruni et al., 1996), three orthogonal, neurocognitive component scores (general cognition, executive functioning, learning and memory; Thompson et al., 2019), and five, dimensional psychopathology scales (from the Child Behavior Checklist; internalizing, somatoform, detachment, neurodevelopmental, and externalizing symptoms; Michelini et al., 2019). BMI, sleep problems, and psychopathology scale scores were right-skewed (skewness values listed in Supplemental Table 1) and were log-transformed prior to analyses (via ‘optLog’; https://github.com/kforthman/optLog).

Analyses

Analyses were conducted using R statistical programming. Chi-squared analyses (via ‘chisq.test’) with follow-up pairwise comparisons (via ‘CrossTable’) and t-tests (via ‘t.test’) were used to assess for sociodemographic differences between LN and HN (Bonferroni-corrected p = 0.05/16 = 0.003). A linear mixed model (LME) was run (via ‘lmer’) with sociodemographic variables as independent variables and FD as the dependent variable to examine relationships between participant characteristics and motion within LN. Random intercepts were included for study site and family (i.e., having a sibling in the study) nested within site.

Results

Sample comparisons

The chi-squared analyses revealed significant differences in the sociodemographic compositions of the samples (see Table 1). Follow-up pairwise comparisons indicated LN contains significantly more females (z = 5.17, p < 0.001); non-Hispanic whites (z = 5.17, p < 0.001); and youth with married parents (z = 4.34, p < 0.001), parents with graduate degrees (z = 5.37, p < 0.001), and household incomes ≥$100,000 (z = 4.70, p < 0.001) than expected. LN participants were significantly older (p < 0.001, Cohen’s d = 0.24) with fewer mental and physical health problems including lower BMIs (p < 0.001, Cohen’s d = −0.24); fewer sleep problems (p = 0.001, Cohen’s d = −0.06); and fewer neurodevelopmental (p < 0.001, Cohen’s d = −0.20) and externalizing symptoms (p < 0.001, Cohen’s d = −0.12) than HN. LN also had significantly higher general cognition (p < 0.001, Cohen’s d = 0.30), executive functioning (p < 0.001, Cohen’s d = 0.26), and learning and memory scores (p < 0.001, Cohen’s d = 0.25). No significant differences were found for internalizing, detachment, or somatoform symptoms (ps > 0.004). Table 1 includes all sample comparison results.

Motion associations in LN

Within LN, several variables exhibited significant relationships with average motion. The most robust of these was BMI, which was positively associated with FD (p < 0.001, Cohen’s d = 0.27). Being female (p = 0.001, Cohen’s d = −0.11), older (p < 0.001, Cohen’s d = −0.16), and having a household income ≥$100,000 (p = 0.03, Cohen’s d = −0.08) were associated with significantly less motion. Somatoform symptoms (p = 0.001, Cohen’s d = −0.11) and executive functioning (p = 0.002, Cohen’s d = −0.10) were also negatively associated with FD. The full results of the LME can be found in Supplemental Table 2, and effect sizes of significant findings are depicted in Fig. 1.

Fig. 1.

Cohen’s ds of the significant predictors from the linear mixed model examining relationships between demographic and physical/mental health characteristics and average framewise displacement (FD) in the low-noise sample (LN; n = 4,356). Blue indicates the variable has a negative relationship with motion, and red indicates a positive relationship. Somatoform Symptoms = dimensional psychopathology scale derived from the Child Behavior Checklist (Michelini et al., 2019). Sex: Male = biological sex is male (‘sex_at_birth’). Household Income: ≥$100 K = annual household income is greater than or equal to $100,000 (‘household.income’). Executive Function = neurocognitive component score (‘neurocog_pc2.bl’; Thompson et al., 2019). BMI = body mass index (‘anthro_bmi_calc’). Age = age in months at baseline visit (‘interview_age’)

Discussion

ABCD Study participants with low-noise rs-fMRI data (i.e., recommended for analysis with low average motion) are more socioeconomically privileged, are less diverse, exhibit higher neurocognitive assessment scores, and report better physical/mental health than those with higher-noise and/or an inadequate sample of rs-fMRI data. Moreover, within LN, average motion significantly related to BMI, sex, age, household income, somatoform symptoms, and executive function (but not race/ethnicity). It is crucial to recognize that the demographic variables identified across these analyses (e.g., BMI, race/ethnicity, household income) are not causal variables influencing the quality of fMRI data. Instead, current results must be considered in historical and socio-cultural context, recognizing there are numerous factors that lead to disparities for disadvantaged and minoritized populations and these disparities, in addition to systemic biases (i.e., systemic racism), contribute to the identified relationships.

These findings are cross-sectional and have relatively small effect sizes. However, they reveal issues of consideration for researchers working with the ABCD Study or other youth samples - particularly given that many relationships between clinical variables and neural measures may also be reflected by small effect sizes (Dick et al., 2021; Paulus & Thompson, 2019). First, there are likely limits to the generalizability of outcomes from rs-fMRI studies of youth. Second, covarying for average motion in analyses, which is a common motion-correction method (Power et al., 2015), may remove variance related to outcomes of interest, even in samples with relatively good quality data. This is highly relevant for the ABCD Study, as some factors that related to motion have also been associated with adolescent substance use (e.g., executive functioning; Gustavson et al., 2017).

Moving forward, rs-fMRI studies of youth should, at the very least, acknowledge the issues discussed herein when determining how to balance data quality with generalizability. Current protocols do not eliminate the potential systematic bias introduced by procedures from data acquisition to analysis. Examples of factors that may introduce disproportionate motion along sociodemographic characteristics of participants may include demographics of study staff (Does et al., 2018), potential mistrust of research institutions, and/or health disparities experienced by disadvantaged or minoritized populations (HHS, 2011). For instance, youth from disadvantaged or minoritized populations may not have as much exposure to research or healthcare settings, which could lead to increased discomfort during MRI scans. Findings of relationships between BMI and head motion replicate prior work (Beyer et al., 2020) and could also relate to discomfort in the MRI scanning environment. Although exhaustive consideration of such factors is beyond the scope of the current work, these examples highlight the importance of considering and measuring the contextual factors underlying sociodemographic effects on fMRI data quality (Simmons et al., 2021).

There may be methodological changes that can help mitigate potential bias, such as oversampling from certain sociodemographic groups, developing and using wider bore scanners, and providing feedback on motion during scans. Indeed, previous research by Greene et al. (2018) indicates both real-time feedback and movie watching reduce average motion during rs-fMRI scans in children under age 11, although movie watching may alter functional connectivity measures. Additionally, weighting based on population-based estimates could be used in analyses of large datasets like the ABCD Study to improve representativeness (Heeringa & Berglund, 2020), though this strategy for addressing biased data will not remedy the issue of having biased samples. Lastly, researchers should determine whether head motion during rs-fMRI relates to their outcomes of interest. In these situations, rather than covarying for motion, novel analysis methods should be explored (e.g., propensity matching groups based on average motion) and findings should always be appropriately contextualized and interpreted in light of potential limitations to generalizability.

Conclusions

Data quality in rs-fMRI studies of youth relates to sociodemographic and health factors, which may lead to biased results. Future research should continue to explore other variables that may relate to data quality in rs-fMRI studies of children and adolescents, interactions between such variables, and how these relationships change over time. There also remains opportunity for developing methodological approaches to promote equity and mitigate biases in rs-fMRI studies from the design to data analysis stages. In sum, it is crucial that issues of bias and generalizability continue to be evaluated and discussed to prevent them from becoming the status quo.