Behavioral and neural stability of attention bias to threat in healthy adolescents

Abstract

Considerable translational research on anxiety examines attention bias to threat and the efficacy of attention training in reducing symptoms. Imaging research on the stability of brain functions engaged by attention bias tasks could inform such research. Perturbed fronto-amygdala function consistently arises in attention bias research on adolescent anxiety. The current report examines the stability of the activation and functional connectivity of these regions on the dot-probe task. Functional magnetic resonance imaging (fMRI) activation and connectivity data were acquired with the dot-probe task in 39 healthy youth (f =18, Mean Age = 13.71 years, SD = 2.31) at two time points, separated by approximately nine weeks. Intraclass-correlations demonstrate good reliability in both neural activation for the ventrolateral PFC and task-specific connectivity for fronto-amygdala circuitry. Behavioral measures showed generally poor test-retest reliability. These findings suggest potential avenues for future brain imaging work by highlighting brain circuitry manifesting stable functioning on the dot-probe attention bias task.

Introduction

An emerging literature examines test-retest reliability of functional magnetic resonance imaging (fMRI) measures, which crucially informs understandings on the neural underpinnings of psychopathology (Sauder, Hajcak, Angstadt, & Phan, 2013). Specifically, finding reliable neural correlates of specific behaviors vitally guides attempts to identify biomarkers, which may guide treatment and prevention research (Zuo & Xing, 2014). However, few such studies document temporal stability of functional connectivity or examine tasks that probe behaviors strongly linked to psychopathology. To address these gaps, the current study examines reliability of both activation and functional connectivity during the dot-probe task, which consistently elicits behavioral differences between patients with anxiety and healthy subjects.

An expanding literature uses the dot-probe task to assess attention allocation to threat. In this task (Fig 1), threat and neutral stimuli are presented simultaneously, followed by a probe. Individual differences in anxiety relate to differences in reaction-time (RT) on trials where the probe appears in the same, vs. different, spatial location as the threat. This RT-difference measure is referred to as an “attention bias”. Increased attention bias to threat in anxious relative to healthy children and adults is one of the most consistently observed associations in research on anxiety (for review see Bar-Haim et al., 2007) and provides a target for novel therapies (Beard, 2011; Hakamata et al., 2010; Linetzky, Pergamin-Hight, Pine, & Bar-Haim, 2015).

Fig 1.

The three right columns depict the three main trial conditions of the dot-probe task. The column on the left displays the duration of each trial event.

Recent work demonstrates poor stability for at least some RT-based measures of attention bias (e.g., Brown et al., 2014), which could reflect limitations in particular RT-based measures (Salthouse & Hedden, 2002). For example, an RT-based measure of variability in attention bias across trials (Iacoviello et al., 2014; Naim et al., 2015) shows better stability than the standard RT-based attention bias measure (Price et al., 2015). However, even the most stable RT-based measure is likely to incompletely reflect particular neural function. This is because RT-based measures of attention bias reduce multiple, dissociable neuro-behavioral processes, such as attention orienting, response selection, and motor execution, to a single value. Some of these neuro-behavioral processes may be more stable than others, and RT-based measures could be unduly influenced by particularly unreliable processes. Such unreliable processes could vary across task administration, as opposed to other processes, which are expressed more stably. Brain-based measures may detect these more stable processes (Britton et al., 2013), which may quantify more stable underlying functions that sustain threat-processing patterns within an individual. Given widespread use of the dot-probe task in clinical research, finding reliable brain functions engaged by the task could advance attempts to adapt the task to create a biomarker in translational neuroscience research on anxiety.

To date, fMRI research using the dot-probe task yields generally consistent findings (Britton, 2012; Fani et al., 2012; Monk et al., 2006, 2008; Telzer et al., 2008). On this task, individual differences in anxiety relate to perturbed function in brain regions supporting emotional processing (e.g., amygdala) and attentional control (e.g., ventrolateral prefrontal cortex, vlPFC; dorsolateral prefrontal cortex, dlPFC). Considerable imaging research examines individuals between the ages of 10 and 17, which creates a particular need for reliability in subjects within this age range. Moreover, in these imaging studies, some findings emerge for contrasts comparing angry incongruent (different spatial location) and angry congruent (same spatial location) trials (i.e., the Attention Bias to Threat contrast). These studies find that anxiety is associated with greater amygdala (Monk et al., 2008) and greater lateral PFC activation (e.g., Britton et al., 2012; Telzer et al., 2008). Other findings emerge when examining neural activation to any threatening facial expression, collapsing across congruency, compared to neutral facial expressions on the dot-probe task (Monk et al., 2006). Perhaps the most consistent finding in adults (Hardee et al., 2013) and youths (Monk et al., 2008) links individual differences in anxiety to negative coupling of frontal and amygdala regions (i.e., fronto-amygdala connectivity). This suggests dysfunctional recruitment of attentional control processes following threat exposure, as has been found on other attention-emotion tasks (Gold, Morey, & McCarthy, 2014; McClure et al., 2007; Payer, Baicy, Lieberman, & London, 2012).

Such perturbed fronto-amygdala connectivity is associated with anxiety and threat processing beyond the dot-probe task (e.g., Sylvester et al., 2012; McClure et al., 2007); therefore, fronto-amygdala connectivity may represent a biomarker to be pursued in translational neuroscience research and in reliability studies. However, to date, no study examines test-retest reliability of task-based fronto-amygdala connectivity. Moreover, across various other task-based indices of activation, divergent reliability estimates emerge in youth (e.g., Koolschijn, Schel, Rooij, Rombouts, & Crone, 2011) and adults (e.g Sauder et al., 2013). For example, intraclass correlation coefficients (ICCs) for PFC activation vary widely across children and adolescents (e.g., ICCs: 0.07 – 0.62, Koolschijn et al., 2011; ICCs: 0.15 – 0.43, Ordaz, Foran, Velanova, & Luna, 2013; ICCs: 0.17 – 0.56, van den Bulk et al., 2013) and adults (e.g., ICCs: 0.37 – 0.66, Koolschijn et al., 2011; ICCs: 0.15 – 0.55, Plichta et al., 2012) with few estimates falling in the “good” (ICCs of 0.6 – 0.74) or “excellent” (ICCs > 0.75) range (Cicchetti, 2001). Estimates of amygdala stability are also quite variable. Some studies report poor reliability (ICCs < .4) in samples of adults (Sauder, et al., 2013) and youth (van den Bulk et al., 2013), while other studies have reported good amygdala reliability (ICCs > 0.6) in adults (Plichta et al., 2012 ). Of note, the one stability study for the dot-probe task generated reasonable amygdala and PFC reliability in children and adolescents (ICCs: 0.73 – 0.83), but the study is limited by a small sample size (n = 12; Britton et al., 2013).

The current study examines reliability using the dot-probe task in healthy children and adolescents. There is a particular need for such research in youth. As noted above, considerable fMRI research uses the dot-probe task in this age group. Moreover, behavioral data in youth link attention bias to anxiety vulnerability (Pérez-Edgar et al., 2011; Shechner et al., 2012; White et al., in press), which creates a further need. Finally, in the few fMRI reliability studies conducted in children and adolescents, reliability estimates appear to be significantly lower in youth than adults (Koolschijn et al., 2011; Ordaz et al., 2013). Thus, understanding the reliability of neural activation associated with this behavioral vulnerability factor in child and adolescent populations is important. Of note, no prior studies examine reliability for task-specific functional connectivity, where between-group differences most consistently emerge in studies of anxiety. To address this issue, the current study reports a method for examining stability of connectivity and applies this method to the dot-probe task. Moreover, reliability work often compares task conditions to an implicit baseline (e.g., response to faces vs. fixation cross). To identify clinically-relevant targets, research is needed on contrasts typically used for group level analyses (e.g., threat incongruent trials vs. threat congruent trials, threat faces vs. neutral faces). The current study considers both such types of contrasts across a nine-week period. This interval between visits was chosen to parallel the frame when effects of treatment on brain function and dot-probe behavioral performance have been examined in anxious youth.

Methods

Participants

Thirty-nine healthy children and adolescents (18 female; M = 13.71 years, SD = 2.31, range: 10.25 – 17.42) completed the current study. Beyond these 39 individuals, an additional seven participants were enrolled but excluded due to technical issues during fMRI data acquisition (n = 2) or poor task performance (accuracy < 75%; n = 5). Participants were free of psychopathology as determined by a structured interview (K-SADS-PL; Kaufman et al., 1997). All participants had an IQ > 70 as assessed by the Wechsler Abbreviated Scale of Intelligence (WASI; Wechsler, 1999). The current study procedures were approved by the National Institute of Mental Health Institutional Review Board. Prior to participation, parents signed consent forms and youth signed assent forms. None of the participants included in the current sample were participants in a previous reliability study by Britton and colleagues (Britton et al., 2013).

Dot-probe Task

Participants completed a dot-probe task in the MRI scanner at two different sessions approximately nine weeks apart (M = 9.36 weeks, SD = 2.09). The current version of the dot-probe task was provided by Tel-Aviv University/NIMH Attention Bias Modification Treatment Initiative (http://people.socsci.tau.ac.il/mu/anxietytrauma/research/). In this task (Fig 1), participants viewed a fixation cross, then a display of two facial expressions (angry-neutral or neutral-neutral), from the same actor, were presented vertically (above and below the fixation cross). Following the face display, a probe appeared on the screen (i.e., < or >) in the location of a previously viewed angry face (i.e., congruent trials) or neutral face (i.e., incongruent trials for angry-neutral trials). Using a button-press, participants were asked to respond to the direction of the probe as quickly and accurately as possible.

The task was administered using E-Prime (Psychology Software Tools, USA) across two functional runs, with a total of 80 congruent trials, 80 incongruent trials, and 80 neutral trials. There were also 80 fixation-only trials. Trials began with a 500-ms fixation cross in the center of the computer screen followed by a 500-ms presentation of the face display. Immediately following the face display, the probe was presented for 1000 ms. The angry face and probe locations were counterbalanced throughout the experiment.

Attention bias to threat scores were calculated from reaction times (RT) on the threat-neutral face display trials, by subtracting congruent trial RTs from incongruent trial RTs. Positive attention bias to threat scores reflect a bias toward threat; negative scores reflect a bias away from threat. Attention bias variability (ABV) scores were calculated according to Naim et al., (2015). A moving average algorithm calculated bias scores on bins of 10 successive congruent and 10 succesive incongruent trials across the entire task, with overlapping bins. Then the standard deviation of all bias scores was divided by the subject’s overall RT to produce the final ABV score. Additionally, since much of the dot-probe neuroimaging work also examines the contrast comparing all trials that include a threat face, collapsing across congruency, to trials where only neutral faces are presented (e.g., Hardee et al., 2013; Monk et al., 2008), a RT score was created to be consistent with this literature. Thus, the current study also examined the test-retest reliability of this contrast (i.e., the difference between RTs on all threat trials relative to neutral RTs). This RT difference score is referred to here as a Threat Distraction score. Positive scores on this Threat Distraction score reflect slower RTs in the presence of threat; negative scores reflect faster RTs in the presence of threat.

Prior to bias calculations, incorrect trials and trials in which RTs were <150 ms or >2000 ms were removed from analyses. Additionally, for each participant at each visit, trials with >2.5 standard deviations of the mean RT for each condition (congruent, incongruent, neutral) were also removed (e.g., Britton et al., 2014; Naim et al., 2015). Average accuracy on the task was 96.8% (SD = 3.1%) at visit 1 and 96.5% (3.5%) at visit 2.

Anxiety Questionnaire

The Screen for Childhood Anxiety Related Emotional Disorders (SCARED; Birmaher et al., 1997) was administered to assess anxiety levels of the participants. The SCARED is a 41-item questionnaire designed to assess childhood anxiety symptoms in terms of the Diagnostic and Statistical Manual of Mental Disorders (DSM-IV). Each item is endorsed using a 3-point scale: 0 “never true”, 1 “somewhat or sometimes true”, and 2 “very true or often true”. At each visit, children and their parents completed the questionnaires. Scores from each informant were summed to create two separate total anxiety scores at each visit.

fMRI data acquisition

Neuroimaging data was acquired with a 3T General Electric (GE) MRI scanner (Waukesha, WI, USA) with an 8-channel head coil. For each subject, functional image volumes with an in-plane resolution of 2.5 mm x 2.5 mm were acquired using a T2° weighted gradient echo pulse sequence (TR/TE = 2300/25 ms, flip angle = 50°, FOV = 24 cm, matrix = 96 × 96, 41 contiguous 3 mm interleaved axial slices). For use in co-registration and normalization procedures, a high resolution 3D MPRAGE spin echo sequence (NEX=1, TE/TI = min/725 ms, FOV = 22 cm, matrix = 256 × 192, bandwidth 31.25HTz per 256 voxels) was collected for all subjects at each assessment.

fMRI data processing

fMRI data was processed and analyzed using AFNI (Cox, 1996). Standard preprocessing procedures included slice timing correction, co-registration, and normalization into Talaraich space. At each visit, the EPI data were registered to the anatomical scan collected from that visit. Of note, while within-subject median templates reduce variability in longitudinal work on brain structure (e.g. Reuter et al, Neuroimage 2012), individual scans from each site were used in the current methods, since this median-template approach has not been used in prior fMRI reliability research. The data were smoothed (6 mm full-width half-maximum) and scaled resulting in 2.5 mm isotropic voxels. To correct for motion, TR pairs with a Euclidean Norm motion derivative > 2 mm were censored prior to individual-level analyses. Participants were excluded if 20% or more TRs were censored; no participant met this exclusionary criteria. An average of 2.3% (SD = 2.9%) of TRs were censored at Visit 1, and 2.4% (SD = 3.9%) at Visit 2.

BOLD Neural Activation

For first-level analyses, a general linear model (GLM) was created for each participant at each visit (Time 1, Time 2). Separate regressors were created for each of the three stimulus event types (threat congruent, threat incongruent, and neutral-neutral trials). The beginning of an event was classified as the onset of the face stimuli. Incorrect trials, including incorrect responses and trials in which the RTs were excluded during the behavioral processing, were also modeled. Additional regressors were included to model baseline drift and motion (i.e., rotational movement of roll, pitch, yaw and motion displacement in the x, y, and z axes).

gPPI

A generalized psychophysiological interaction (gPPI) set of analyses was used to examine functional connectivity (http://afni.nimh.nih.gov/sscc/gangc/CD-CorrAna.html). This set of analyses modeled differences in connectivity between a “seed” region and other brain regions across the different psychological events (trial types) in the task. Based on prior PPI studies with the dot-probe task (Hardee et al., 2013; Monk et al., 2008), the right and left amygdala, anatomically defined by the Talairach atlas, were used as seeds. Separate GLMs were created for each seed. PPI terms for each of the trial types were the product of the detrended and demeaned seed regressor and the psychophysiological event (i.e., threat congruent, threat incongruent, or neutral-neutral). Individual-level GLMs were created using the same regressors as in the analysis of task-related changes in BOLD signal, the eigenvariate time series for the “seed”, and psychophysiological interaction terms for three trial types.

Test-Retest Reliability Analyses

Single-measure intra-class correlations (ICC) were used to assess reliability across time. One set of analyses considered the test-retest reliability of behavioral and anxiety measures. First, for the dot-probe test-retest reliability, ICCs of overall RTs, Attention Bias to Threat scores, and Threat Distraction scores were examined. Then, ICCs of the test-retest reliability of participants’ parent- and self-reported anxiety scores were examined. ICCs were modeled with a two-way mixed model with absolute agreement (i.e., ICC(2,1)). ICC results for the behavioral measures were conducted using IBM SPSS software package.

Another set of reliability analyses considered fMRI data. We performed three fMRI test-retest reliability analyses. The first fMRI ICC analysis examined test-retest reliability of BOLD activation evoked for all task events compared to a null-event baseline. This test provides a reference for the overall neural stability during dot-probe task performance. Examining stability of contrasts that use a baseline reference is the most frequently reported type of ICC analysis in prior studies of fMRI reliability (e.g., Ordaz et al., 2013); however, these contrasts are often not behaviorally relevant. Therefore, the second and third fMRI stability analyses focused on the test-retest reliability of two task-relevant contrasts that have neural and behavioral associations with psychopathology. For these task-relevant analyses, ICCs were examined for both BOLD neural activation and functional connectivity. Specifically, the second set of analyses examined ICCs of the Attention Bias to Threat contrast, comparing threat incongruent to threat congruent events. Consistent with prior fMRI research using the dot-probe (Hardee et al., 2013; Monk et al., 2008), the third set of analyses examined ICCs of the Threat Distraction contrast, comparing threat trials, collapsed across congruency, with neutral trials. The Threat Distraction contrast probes more general attention processing in the face of threat compared to the attention bias contrast.

The fMRI ICC (2,1) analyses for individual level contrasts (i.e., all trials vs. baseline, Attention Bias to Threat contrast, and Threat Distraction contrast) were conducted in AFNI using subject and visit as random variables in the LME model (Chen, 2010). Individual level contrasts from the BOLD and PPI GLMs were used as input in the ICC analyses. A single prefrontal cortex (PFC) mask was used for the ICC analysis because most previous dot-probe work implicates the PFC, and connectivity between the amygdala and PFC, in attention bias (Hardee et a., 2013; Monk et al., 2008). The PFC-specific mask encompassed grey matter voxels anterior to a plane drawn at y = 0 perpendicular to the AC/PC line.

The initial ICC threshold was set to 0.41, corresponding to an uncorrected p < .005 threshold, with 38 degrees of freedom (Bartko, 1966). Next, the PFC cluster threshold was set to identify significant regions while correcting for multiple comparisons across the PFC. This threshold was determined using 1000 Monte Carlo simulations in the 3dClustsim tool included in AFNI for a one-sided ICC test, with mean spatial correlations derived from the data collected for this study. These data had 9.24, 9.13, 8.41 mm FWHM (in the respective x, y, and z dimensions) and it was determined that 33 contiguous voxels in the PFC were needed to achieve a threshold of p < .005, with an overall family-wise error rate of α < .05. Exploratory whole-brain ICC analyses were also conducted. The 3dClustsim tool produced a whole-brain gray matter corrected threshold of 50 voxels; clusters that survived this whole-brain gray matter correction are presented in Supplementary Table 1.

To eliminate effects of outliers, scatter plots were inspected for each significant cluster, and analyses were re-run after removing participants with individual values greater than three standard deviations from the group mean contrast score (e.g., Attention Bias to Threat, Threat Distraction). Only regions that remained significant after removal of outliers are presented. Attention bias variability scores were log transformed to correct for normality prior to ICC analyses.

To complement the reliability analyses, group-level activation patterns and functional connectivity results were examined at each visit for each of the contrasts of interests (Supplemental Material). Each individual level contrast of interest (i.e., all trials vs. baseline, Attention Bias to Threat contrast, and Threat Distraction contrast) was submitted to a one-sample t-test. Compared to the test-retest reliability analyses, these analyses provide information on the regions that survive group-level statistical thresholding at each time point. These contrasts were masked using the PFC mask. Areas surviving the PFC-cluster threshold are presented in Supplementary Table S.2. Additionally, average activation and connectivity for the contrasts of interest (Attention Bias, Threat Distraction) are presented at a liberal threshold (p < .005, uncorrected; Supplementary Figures S.1–S.2).

Several additional post-hoc analyses were conducted to further explore reliability of the dot-probe task. Using a region-of-interest (ROI) approach, post-hoc analyses were conducted on amygdala activation on the two main contrasts of interest: Attention Bias to Threat and Threat Distraction. Relevant functional activation of the contrast estimates were extracted from the right and left anatomically-defined amygdala mask (i.e., the amygdala seed used in the PPI analyses). An identical approach used to estimate the ICCs to the behavioral data (i.e., ICC(2,1) in SPSS) was employed to calculate the ICCs for the amygdala. Second, ICCs were recalculated on fMRI data, restricting analyses to only the data acquired in the first half of the task. Identical procedures were followed to those of the main imaging reliability analyses described above; however, individual level GLMs only included data on the first half of the trials. Given that the participants could have become fatigued or bored towards the end of the task, restricting the reliability analyses to the first half of the tasks may exclude extra noise introduced towards the end of the task. These results are reported in the Supplementary Material (see Table S.3). Third, anxiety, age, behavioral measures (i.e., attention bias to threat scores, threat detection scores, and ABV) and time-between scan interval effects of the neural reliability of the primary imaging contrasts of interest (Attention Bias to Threat Contrast, Threat Distraction Contrast) were also examined. For each variable, the participants were divided into two groups using a median split on the measure from visit 1 (median SCARED = 1, age = 13.75, attention bias to threat = 1.4 ms, threat detection =1.9 ms, attention bias variability = .68). Post hoc ICC analyses were conducted in SPSS on the significant regions for the above groups; regions that were non-significant for a given group are reported in Supplementary Material. Lastly, correlations between activation and connectivity patterns of significantly reliable regions, anxiety, and behavioral measures on the dot-probe task were examined at each time point. Correlations are reported in the Supplementary Material. Due to the number of correlations conducted and the low level of anxiety scores in the sample, results of the correlation analyses are not discussed in the main body of the paper.

Results

Reliability of Anxiety and Dot-Probe Behavioral Reaction Time Measures

Means and standard deviations of anxiety scores and behavioral data from the dot-probe task are presented in Table 1. For task performance, participant’s RTs on the task, collapsed across all conditions, were stable across time (ICC = 0.57). However, Attention Bias to Threat scores (ICC = −0.16) and Threat Distraction scores (ICC = −0.12) showed poor test-retest reliability. ABV scores also showed low reliability, but did have a higher ICC values than the other behavioral measures (ICC = 0.36). It should be noted that the overall mean RT was significantly faster at the second visit compared to the first visit, t(38)=3.45, p = .001. This may reflect general aspects of learning across time, as indicated by faster RT, that could affect the reliability of the contrast Task vs Baseline. Importantly, however, mean values for the three behavioral measures relevant to the current study, Attention Bias to Threat, Threat Detection, and ABV, did not differ across the two visits (ts < 1.1, ps > .3). This lack of difference coupled with the fact that these measures are relative (difference scores) suggests that general learning or practice effects were not likely to impact the contrasts of interest. As for the reliability of anxiety scores, results also revealed high stability in self-reported (ICC = 0.88) and parent-reported (ICC = 0.80) anxiety scores.

Table 1.

Means and SDs of Reaction Times (RTs) and Anxiety Scores

	Visit 1	Visit 2
Congruent RTs (ms)	534 (60)	508 (48)
Incongruent RTs (ms)	534 (58)	511 (43)
Neutral RTs (ms)	533 (57)	509 (45)
Attention Bias to Threat Score (ms)	−0.28 (12.68)	3.13 (13.65)
Attention Bias Variability Score	0.07 (0.02)	0.07 (0.03)
Threat Distraction Score (ms)	1.45 (10.64)	0.32 (11.08)
Anxiety Score, Parent Report	3.71 (5.31)	4.16 (5.09)
Anxiety Score, Self-Report	7.15 (6.51)	6.42 (6.69)

Attention Bias to Threat Score = Incongruent RTs – Congruent RTs; Threat Distraction Score = Average of Incongruent RTs and Congruent RTs – Neutral RTs

fMRI Reliability of Overall Task Effects

Neural Activation

Areas of stable neural activation across the task, collapsed over task conditions and compared to baseline, are shown in Table 2. Subjects’ activation patterns were quite stable across many regions of the PFC, including the middle frontal gyrus and the inferior frontal gyrus (Fig. 2).

Table 2.

Neural Stability of Prefrontal Cortex on Dot Probe Task

	Peak Talairach coordinates			Cluster Size	ICC value at peak	Peak TLRC Location Area	Brodmann
	x	y	z
Task vs. Baseline Neural Activation
	−21	4	61	416	0.74	L. Middle Frontal Gyrus	L. BA6
	−29	24	31	209	0.67	L. Middle Frontal Gyrus	L. BA9
	34	16	21	167	0.76	R. Middle Frontal Gyrus	R. BA44
	−46	1	34	156	0.68	L. Precentral Gyrus	L. BA6
	46	−1	24	96	0.69	R. Inferior Frontal Gyrus	R. BA6
Attention Bias to Threat Contrast (Incongruent vs. Congruent trials) Neural Activation
	−24	24	1	42	0.55	L. Claustrum/L. Insula/ L. Inferior Frontal Gyrus	L. BA45
Functional Connectivity
L. Amygdala Seed
	−26	56	9	53	0.59	L. Middle Frontal Gyrus	L. BA10
	−29	1	54	52	0.57	L. Middle Frontal Gyrus	L. BA6
	−9	31	36	42	0.64	L. Medial Frontal Gyrus	L. BA8
R. Amygdala Seed
	-	-	-	-	-	No significant regions
Threat Distraction Contrast (Threat vs. Neutral trials) Neural Activation
	-	-	-	-	-	No significant regions
Functional Connectivity
L. Amygdala Seed
	41	34	21	49	0.58	R. Middle Frontal Gyrus	R. BA9
	56	11	9	42	0.58	R. Precentral Gyrus	R . B A44
	−34	24	34	40	0.61	L. Middle Frontal Gyrus	L. BA9
R. Amygdala Seed
	−41	31	14	53	0.72	L. Middle Frontal Gyrus	L. BA46

Fig. 2.

Stability of event-related middle frontal gyrus activation for the contrast comparing task effects to baseline. Images above show several stable regions of activation. Graphs to the right of the image plot BOLD (% signal change) values at each visit from the associated activation cluster. Correlation of BOLD values at each visit are listed below. Images show stable activation in two regions of the left middle frontal gyrus [a. 416 voxels, r(39) = .32, p = .005, BA 6; b. 209 voxels, r(38) = .38, p = .02, BA 9] and the right middle fontal gyrus [c. 167 voxels, r(39) = .59, p < .001, BA 44].

fMRI Reliability of Attention Bias to Threat (Incongruent vs. Congruent)

Areas of stable event-related activations and functional connectivity for the Attention Bias to Threat contrast comparing incongruent and congruent threat trials are shown in Table 2. Neural Activation: The ICCs revealed that the vlPFC (BA 45) showed stable neural response on the Attention Bias to Threat contrast across the two time points [Fig 3. (−24, 24, 1), 42 voxels, ICC peak activation = 0.55, BA 45].

Fig. 3.

Stability of event-related ventrolateral prefrontal cortex (vlPFC, 42 voxels, BA 45) activation for the Attention Bias to Threat contrast (Threat Incongruent vs Threat Congruent). The graph to the right of the image plots BOLD (% signal change) values at each visit from the vlPFC activation cluster. Correlation of activation across visits was r(37) = .46, p = .004.

Functional Connectivity: The ICCs revealed stable connectivity between amygdala and several middle frontal gyrus regions [Fig 4a. (−26, 56, 9), 53 voxels, ICC peak activation = 0.59, BA 10; Fig 4b. (−29, 1, 54), 52 voxels, ICC peak activation = 0.57, BA 6], as well as a region in the medial frontal area [Fig 4c. (−9, 31, 36), 42 voxels, ICC peak activation = 0.64, BA 8].

Fig. 4.

Stability of fronto-amygdala connectivity for the Attention Bias to Threat contrast (Threat Incongruent vs Threat Congruent). Graphs to the right of the image plot PPI values at each visit from the associated activation cluster. Correlation of PPI values at the two visits are listed below. Results show stable connectivity between the amygdala and several regions in the middle frontal gyrus [a. 53 voxels, r(38) = .73, p < .001, BA 10; b. 52 voxels, r(37) = .53, p < .001, BA 6; c. 42 voxels, r(38) = .51, p = .001, BA 8].

fMRI Reliability of Threat Distraction Contrast (Threat vs Neutral)

These analyses examined both the stability of event-related activations and functional connectivity for the Threat Distraction contrast, comparing all threat trials [threat incongruent trials and threat congruent trials] to neutral trials. Areas of stable neural activation and connectivity are reported in Table 2.

Neural Activation

Examination of the ICCs of the BOLD activation revealed no stable areas of activation in the prefrontal cortex.

Functional Connectivity

The ICC analyses revealed stable amygdala-dlPFC connectivity. Specifically, significant reliability occurred between the right amygdala and left dlPFC [Fig 5a. (−41, 31, 14), 53 voxels, ICC peak activation = 0.72, BA 46] and between the left amygdala and right dlPFC [Fig 5b. (41, 34, 21), 49 voxels, ICC peak activation = 0.58, BA 9; Fig 5c. (56, 11, 9), 42 voxels, ICC peak activation = 0.58, BA 44]. Functional connectivity between the left amygdala and left dlPFC was also stable [Fig 5d. (−34, 24, 34), 40 voxels, ICC peak activation = 0.61, BA 9].

Fig. 5.

Stability of fronto-amygdala connectivity for thr Threat Distraction contrast (Threat vs Neutral). Graphs to the right of the image plot PPI values at each visit from the associated activation cluster. Correlation of PI values at the two visits are listed below. Results show stable amygdala-dlPFC connectivity [a. 53 voxels, r(38) = .45, p = .005, BA 46; b. 49 voxels, r(39) = .49, p = .002, BA 9; c. 42 voxels, r(38) = .41, p = .01, BA 44; d. 40 voxels, r(39) = .61, p < .001, BA 9].

Other analyses

One set of analyses used an ROI approach to examine the reliability of amygdala activation on the Attention Bias to Threat and the Threat Distraction contrasts. The results revealed poor reliability (Attention Bias to Threat: ICC _L.Amygdala = 0.00, ICC _R.Amygdala = 0.00; Threat Distraction: ICC _L.Amygdala = 0.12, ICC _R.Amygdala = 0.00). A second set of analyses compared stability when using all data from the task, relative to using only data from the first half of the task. As presented in the Supplemental Material, these results generally demonstrated stronger reliability with the full task. The third to fifth sets of analyses examined reliability as a function of anxiety, age, and time interval between scans. These results revealed few relations between any of these factors and reliability, as also summarized in Supplemental Material. Finally, analyses examined the relations between score on anxiety rating scales or RT-based measures and brain activation on the task. These results did suggest a consistent association between activation and anxiety, but not with behavior on the task (see Supplemental Material, p. 8).

Discussion

The current study examined test-retest reliability of brain regions engaged by the dot-probe task in healthy youth. Results suggest the potential utility of continuing to explore attention tasks as anxiety-related biomarkers. Specifically, the current findings demonstrate good test-retest reliability in key PFC regions highlighted in prior dot-probe research. The findings also revealed good stability in fronto-amygdala connectivity. In fact, more regions were found to be stable for the functional connectivity analyses than for the activation results. Interestingly, areas of activation and connectivity that are stable in the current study have also been shown to change with treatment in anxious patients (e.g., Goldin et al., 2013; Voss & Paller, 2012). Thus, the current results set the stage for studies comparing healthy individuals and patients over time, possibly before and after the patients receive treatment. This could inform attempts to use biomarkers in translational neuroscience research on anxiety.

Good stability in fronto-amygdala connectivity was established across two behaviorally relevant contrasts. On incongruent trials relative to congruent trials (i.e., the attention bias to threat contrast), connectivity between the amygdala and mid-frontal regions showed good reliability. On threat relative to neutral trials (i.e., the threat distraction contrast), amygdala-dlPFC connectivity showed good test-retest reliability. Given that prior dot-probe research finds consistent differences in fronto-amygdala connectivity between anxious and non-anxious youth (Hardee et al., 2013; Monk et al., 2008), perturbed fronto-amygdala connectivity may represent a promising biomarker. Thus, establishing adequate reliability is important and may justify intensified clinically-focused research on fronto-amygdala connectivity.

Overall, the results showed reliable neural patterns in key regions identified in prior dot-probe imaging work (e.g., dlPFC, vlPFC); however, somewhat different ICC patterns emerged for the attention bias to threat and threat distraction contrasts. The vlPFC showed reliable activation patterns on the attention bias contrast; however, on the threat distraction contrast, no PFC region demonstrated reliable activation. As for fronto-amygdala connectivity, amygdala-dlPFC connectivity showed good reliability on the threat distraction contrast. On the attention bias to threat contrast, however, good reliability was detected for amygdala connectivity with other regions of the middle frontal gyrus. Although we did not compare the two contrasts directly, results demonstrate different reliability profiles for each measure. As such, both contrasts appear worthy of investigation and may capture the neural correlates of different, but important, threat-related processes. For example, the attention bias contrast might isolate processes engaged by competition between neutral and threat information for attention-orienting resources. In contrast, the threat distraction contrast may capture other effects on attention, such as effects of heightened arousal in response to threats. Future work should examine how neural stability on these specific contrasts relates to the stability of the underlying attention processes that the contrasts are thought to test.

Despite the reliability of fronto-amygdala connectivity across time, amygdala activation did not show adequate reliability on either contrast of interest. The dot-probe may be a poor task for eliciting reliable amygdala activation; stronger amygdala activation reliability has been detected on different emotion tasks (e.g., Gee et al., 2015; Plichta et al., 2012). However, poor reliability in the current study and elsewhere (e.g., Sauder, et al., 2013; Van Den Bulk et al., 2013) suggests amygdala activation, outside the context of functional connectivity analyses, may represent a problematic biomarker.

Interestingly, regions showing the greatest reliability in activation and connectivity overlapped minimally with regions showing the highest mean level of activation and connectivity. Consistent with this, data points for stable regions depicted in Figures 3–5 are equally distributed around the y-axis. Previous reliability studies have also found discrepancies between reliability and task-level activations or connectivity in specific regions (Bennet & Miller, 2010; Caceras et al., 2009). This discrepancy impacts future research on individual differences in brain function. That is, when attempting to link function in brain networks to individual differences, an overly narrow focus on regions most robustly engaged by a task may obscure regions where stable individual differences emerge. This subtle point may significantly inform research on biomarkers associated with psychopathology. Regions showing reliable activation, even without particularly strong activations in one or another group of subjects, may critically inform attempts to develop biomarkers. As this point could be lost in cross-sectional research, continued fMRI reliability work and other longitudinal approaches to biomarker discovery are needed.

Previous fMRI reliability work has documented lower reliability in younger children compared to adolescents and adults (e.g. Koolschijn et al., 2011; Ordaz et al., 2013). In the current study, post-hoc analyses were conducted to examine whether there was adequate reliability in the younger children. Overall, reliability did not tend to differ with age (see Supplemental Material). However, there were several age differences to note. The younger group did not show adequate reliability of vlPFC activation on the attention bias to threat contrast or on one of the fronto-amygdala connectivity findings on the threat distraction contrast. On the attention bias to threat contrast, however, older children did not show adequate reliability on one of the connectivity findings. These differences could reflect developmental differences in fronto-amygdala connectivity (e.g., Gabard-Durnam et al., 2014). However, despite these few differences, the overall pattern of results held for both children and adolescents. Future work with larger sample sizes across age groups should continue to investigate possible age differences in reliability.

Another important endeavor when conducting test-retest reliability work is to understand the number of trials needed to calculate reliability. Many psychological tasks, including the dot-probe task, can be long and tedious for participants. An important question concerns whether the same pattern of reliability can be achieved, or even improved, with fewer data points. Therefore, ICC results of the neuroimaging data using only data acquired during the first half of the task at each visit were examined. Although there is some overlap between the reliability findings of the whole task and first half of the task, using data across the whole task yielded stronger reliability. This difference could reflect increased signal to noise ratio associated with a greater number of trials.

Similar to previous reports (Brown et al. 2014, Price et al., 2015), poor reliability was found for the standard behavioral RT-difference attention bias measure on the dot-probe task. Although we found higher reliability for the behavioral attention bias variability scores than for the other behavioral measures, the reliability was still low. Stability of RT-based difference scores can be difficult to detect (Salthouse & Hedden, 2002) and may be obscured by artifacts that plague RT-based cognition-emotion tasks (Brown et al., 2014). The poor reliability of the RT-based bias scores could reflect the complexity of task-related processes, coupled with nonspecific influences on measures of RT. Brain imaging studies suggest that a single trial of the dot-probe task engages multiple, complex, potentially dissociable processes (Britton et al., 2012). Thus, RT-based measures may reflect many of these processes, including the less stable processes. Neuroimaging, on the other hand, may selectively detect more stable processes (Britton et al., 2012).

The present findings should be considered in light of several limitations. First, the study was conducted in a group of non-anxious youth. Stability was examined over a two-month period, which mirrors the time frame when effects of treatment on brain function and dot-probe behavioral performance have been examined. There is major interest in stability among patients with anxiety disorders, which was not examined here. It is unclear if analyses of stability in anxious youth would yield similar ICC results. Unfortunately, conducting such a study in anxious youth is difficult and faces ethical complications. It would be important on ethical grounds to provide affected patients with treatment during the time separating the two administrations of the task; however, such treatment could alter activation. It is possible that future work could implement a wait-list control in a patient group and assess reliability during that time frame; however, the time frame would be relatively short (e.g., 1 month). The current study is interested in relatively longer stability. The second limitation of the current study surrounds the generalizability of the current results to other versions of the dot-probe task. The current version of the task was chosen to maximize the trial count of the task conditions and parallel the version of the task commonly used in ABMT research. However, it remains unclear the extent to which task parameters, such as stimulus presentation duration (i.e., 500-ms face display) or stimulus orientation (i.e., vertical face and probe display), influence stability. Moreover, the current study only used angry and neutral faces. Thus, the findings cannot speak to the reliability of attention biases to other types of emotional faces (e.g., fearful, disgust, happy) that have been used in other versions of the dot-probe task (Britton et al., 2013; Hardee et al., 2013; Price et al., 2014). Another limitation to note is that the analyses exploring how factors such as age or anxiety influenced the reliability results may have been underpowered due to sample size and a limited range of anxiety scores in the current sample. As such, future work with larger sample sizes and greater variable in anxiety scores should continue to examine possible differences in dot-probe reliability as a function of age and anxiety. Next, although the current results suggest that neural markers of attention bias are stable across time, similar to previous reports (Brown et al., 2014; Price et al., 2015), the standard behavioral measure of attention bias did not show adequate reliability across time. Thus, future dot-probe work may want to incorporate brain-based measures or other behavioral measures (e.g., attention bias variability) and analyses recommendations, like the ones discussed in Price et al., 2015 (e.g., attention bias scores based only on trials where the probe appears at the bottom location) that show better reliability. Lastly, there may be additional image processing procedures, such as using a within subject average template before normalization to standard space, that could improve fMRI reliability measures in future reliability studies.

The current results add to the growing body of fMRI stability research by assessing the test-retest reliability of neural activation and functional connectivity in the context of behaviorally relevant contrasts of the dot-probe task. Importantly, the current findings demonstrate that healthy youth showed good reliability in vlPFC and fronto-amygdala connectivity, regions found to be associated with anxiety in prior dot-probe imaging work. The current findings highlight several important brain regions (e.g., the vlPFC and dlPFC) associated with stable threat processing that may inform target brain regions for future treatment research. Finally, the adequate reliability found in the neural correlates on the dot-probe task, despite poor reliability of behavioral measures, suggests that brain-based measures may be useful in research on individual differences studied in psychology and psychiatry. Future work on attention bias in anxious youth may be bolstered by the use of neuroimaging techniques.

Highlights.

• Reliable neural correlates of specific behaviors guides attempts to identify biomarkers of psychiatric disorders.

• On a dot-probe task, healthy youth showed good reliability in vlPFC and frontoamygdala connectivity, regions found to be associated with anxiety in prior dotprobe imaging work.

• Behavioral measures on the dot-probe task showed generally low test-retest reliability.

• The reliable neural activation and functional connectivity found in the current study may inform attempts to use biomarkers in translational neuroscience research on anxiety.