Test-Retest Reliability of Amygdala Response to Emotional Faces

Abstract

In the current study we evaluated the test-retest reliability of amygdala response using an emotional face-matching task that has been widely usedto examine pathophysiology and treatment mechanisms in psychiatric populations. Activation within the fusiform face area (FFA) was also examined. Twenty-sevenhealthy volunteers completed a variation of the face-matching paradigm developed by Hariri et al. (2000) at two time-points approximately 90 days apart. Estimates of test-retest reliability of amygdala response to fearful faces were moderate, whereas angry and happy showed poor reliability. Test-retest reliability of the FFA was moderate to strong, regardless of facial affect. Collectively, these findings indicate that the reliability of the BOLD MR signal in the amygdala varies substantiallyby facial affect. Efforts to improve measurement precision, enlarge sample sizes, or increase the number of assessment occasions seem warranted.

Over the past decade, there has been increasing emphasis on research explicating the neural basis of psychopathology. This trend is consistent with major funding initiatives such as the Research Domain Criteria (RDoC) project, part of NIMH’s strategic plan to develop a neuroscience-based system of classification for psychiatric disorders (Insel et al., 2010). Likewise, several researchers have suggested that the evaluation of neural markers is a necessary step toward accurate diagnosis and subsequent treatment(Beauchaine, Neuhaus, Brenner, & Gatzke-Kopp, 2008; Cicchetti & Toth, 2009; Sroufe & Rutter, 1984). This focus on translational neuroscience is evident in the growing literature that uses functional magnetic resonance imaging (fMRI) to better understand psychopathology, as researchers increasingly use this neuroimaging method to both identify neural markers of disease-related traits and characterize treatment response (Matthews, Honey, & Bullmore, 2006).

Although the field has progressed in identifying some of the neural mechanisms of psychopathology and related processes, research on the psychometric properties (i.e., reliability and validity) of fMRI measures is sparse. This is in stark contrast to other domains of research, in which validation of a measure is a prerequisite for its clinical utility. For example, the internal consistency and test-retest reliability of the Beck Depression Inventory-II (Beck, Steer, & Brown, 1996) have been repeatedly demonstrated prior to the release of the measure (Beck, Steer, Ball, & Ranieri, 1996) and over the course of its use (Dozois, Dobson, & Ahnberg, 1998; Sprinkle et al., 2002; Storch, Roberti, & Roth, 2004). Despite nearly two decades of fMRI research examining the neural response to emotional stimuli, evidence supporting the reliability of this measurement is sparse, and the data is mixed.

Classic statistical theory suggests that reliability places an upper limit on the predictive validity of any measure, and that the product of the reliability coefficients of any two measures limits the correlation between them (Shrout, 1998). Low measure reliability also reduces power, particularly to detect relationships between measures (e.g. the ability to correlate brain response and self-report measures). Thus, sufficient data on measure reliability is a prerequisite for both study design and interpretation, as reliability estimates directly impact the number of participants or scans needed (as a function of power) while limiting potential outcomes (i.e., a measure cannot correlate better with another measure than it correlates with itself).

Adequate measure reliability is especially important in intervention studies, which typically assess dependent measures multiple times over the course of several weeks or months. Clinical trials for pharmacotherapy and psychotherapy, as well as psychopharmacological challenge studies, increasingly includepre- and post-treatment fMRI assessments. However, the concurrent and predictive validity of fMRI-based measures of neural activityare necessarily limited by reliability. In the current study, we focus specifically on the test-retest reliability of the amygdala response to emotional faces. As part of the limbic system, the amygdala has been implicated in vigilance, fear processing, pain processing, and the identification and regulation of emotions (Davis, 1992; Davis & Whalen, 2001; Neugebauer, Li, Bird, & Han, 2004). A large corpus of studies have consistently reported augmented amygdala response to emotional faces, regardless of valence (Costafreda, Brammer, David, & Fu, 2008; Phan, Wager, Taylor, & Liberzon, 2002; Sergerie, Chochol, & Armony, 2008).

Moreover, substantial translational research has focused on amygdala activation in response to emotional faces in relation to psychopathology. Altered amygdala response to emotional faces has been implicated in several psychiatric disorders including depression, anxiety disorders, borderline personality disorder (BPD), antisocial personality disorder (ASPD) and bipolar disorder (Etkin & Wager, 2007; Leppänen, 2006; Lis, Greenfield, Henry, Guilé, & Dougherty, 2007; McCloskey, Phan, & Coccaro, 2005; Rauch, Shin, & Wright, 2003; Strakowski, DelBello, & Adler, 2004). Across these studies, abnormal amygdala response to salient emotional stimuli is taken to reflect trait-like abnormalities in emotion processing or its regulation. As a result, several researchers have proposed that amygdala response to faces, in particular exaggerated response to emotional faces, may be a predictive neural marker of disorder pathogenesis (Benes, 2007; Goodman, New, Triebwasser, Collins, & Siever, 2010; Hasler & Northoff, 2011; Rauch et al., 2003).

The utility of this putative biomarker of psychiatric diseases is dependent upon the assumption that amygdala response to emotional faces is trait-like; that is, that this measure is reliable over time. To date, four studies have examined the reliability of amygdala response to emotional faces (Johnstone et al., 2005; Manuck, Brown, Forbes, & Hariri, 2007; Plichta et al., 2012; van den Bulk et al., 2013), with mixed results. The first study byJohnstoneet al. suggested moderate to strong reliability for fearful faces in comparison to fixation across three sessions that were zero, two, and 8 weeks apart. Reliability estimates varied significantly according to the contrast examined (fear vs. fixation or fear vs. neutral), area of interest (functional vs. anatomically defined amygdala), test-retest period (2 weeks vs. 8 weeks) and hemisphere. Reliability was highest forthe functionally-defined left amygdala when calculated across all three sessions (ICC_(*,1) = 0.66), but much lower for the anatomical amygdala (ICC_(*,1) = 0.28). Across longer test-retest periods, the left amygdala response was reliable (ICC_(*,1) = 0.63), whereas the right amygdala showed poor reliability (ICC_(*,1) = 0.27).¹In comparison, Manuck and colleagues found moderate test-retest reliability over a one year period for threatening faces (i.e., fearful and angry faces combined) in the right amygdala only (ICC_(*,*) = 0.59), despite significant bilateral activation at both time-points. However, a recent study by Plichta et al. suggested strong reliability (ICC_(3,1) > 0.60), bilaterally to fearful and angry faces across a two-week retest period. Finally, van den Bulk and colleagues examined amygdala reliability in an adolescent sample, but found poor amygdala reliability in response to fearful, happy and neutral faces (combined) that was inconsistent across three time points approximately three months apart (ICC_(3,1) ranging from −0.02 to 0.34, depending on time period assessed, ROI definition, and hemisphere).

Both the Manuck and Plichta papers used variants of a popular task in which participants match faces based on expression (Hariri, Bookheimer, & Mazziotta, 2000)—and these studies demonstrate higher reliability estimates than the other studies. Moreover, higher reliability using this task was apparent at shorter (e.g., Plichta et al) as compared to longer (e.g., Manuck et al.) test-retest intervals, although the latter studyincluded only 13 participants. It is difficult to directly compare these studies however, as the Manuck et al. study failed to find significant reliability for the functionally defined left amygdala, and did not report reliability associated with anatomical regions of interest. Thus, there is a need to further assess the stability of the amygdala using larger samples and over longer test-retest periods.

Additionally, extant studies have either combined reliability results across multiple emotional expressions, or examined fearful faces in isolation. Although fearful faces are likely the most widely utilized facial expression, the use of angry, disgusted, and happy faces is not uncommon in clinical neuroscience research(Leppänen, 2006; Monk et al., 2008a; Monk et al., 2008b; Somerville, Kim, Johnstone, Alexander, & Whalen, 2004; Weng et al., 2011). Thus it is difficult to ascertain whether findings from previous studies might generalize to alternative facial expressions.

To address these gaps in the literature, the current study evaluated the test-retest reliability of the amygdala response to fearful, happy, and angry faces using a variant of the emotional face-matching paradigm over a period of approximately 90 days (Hariri et al., 2000; Phan et al., 2008). This face-matching task has been utilized across a variety of disorders including autism spectrum disorders, schizophrenia, depression, bipolar disorder and Williams-Beuren syndrome (Fakra, Salgado-Pineda, Delaveau, Hariri, & Blin, 2008; Matthews, Strigo, Simmons, Yang, & Paulus, 2008; Meyer-Lindenberg et al., 2005; Robinson et al., 2008; Wang, Dapretto, Hariri, Sigman, & Bookheimer, 2004); it has been linked to serotonin transporter genetic variation (Hariri et al., 2002a), and has been used in both treatment (Phan et al., 2012) and psychopharmacological challenge studies (Arce et al., 2008; Paulus et al., 2005; Phan et al., 2008). This task has also been associated with the best reliability estimates in previous studies(Manuck et al., 2007; Plichta et al., 2012). Using a larger sample and longer test-retest period than previous studies, we sought to evaluate the stability of amygdala response over a 90-day period and determine whether it differed by emotion expression. We predicted superior reliability for fearful and angry faces relative to happy faces, which is consistent with previous research indicating greater specificity of amygdala response to threatening stimuli (Costafreda et al., 2008; Phan et al., 2002).

As a point of comparison, we concurrently evaluated the reliability of the fusiform face area (FFA), an area of the brain critical to face processing that has been shown to respond differentially to affective faces in comparison to neutral faces (Kanwisher, McDermott, & Chun, 1997; Vuilleumier & Pourtois, 2007). The FFA is both anatomically and functionally linked to the amygdala (Amaral & Price, 1984; George, Driver, & Dolan, 2001; Sabatinelli, Bradley, Fitzsimmons, & Lang, 2005), and lesion studies have demonstrated that amygdala damage impairs the processing of affective faces within the FFA(Vuilleumier, Richardson, Armony, Driver, & Dolan, 2004). In concordance with previous research demonstrating that areas of the visual cortex respond consistently to affective faces regardless of valence (Phan et al., 2002), we predicted that the reliability of MR response in the FFA would not vary across facial expression.

Materials and Method

Participants

The current study consisted of 27 right handed subjects (14 males, 13 females; age range 18–46, M = 27.5, SD = 8.3) without a history of head injury who were recruited as part of a healthy control group in two larger treatment outcome studies. All participants were given the research version of the Structured Clinical Interview for DSM Disorders, non-patient edition (First et al., 2002) to rule-out individuals with previous or current psychiatric illness, substance abuse or learning disability. Participants also underwent a health screening by a physician to rule out neurologic or major medical problems, and completed a urine drug test immediately prior to scanning and all were free of psychoactive/psychotropic medications (lifetime) or drugs at time of scanning. Consistent with local Institutional Review Board procedures, written consent was acquired from all individuals after explanation of the experimental protocol.

Design and Procedure

Participants underwent fMRI scanning at two separate time-points approximately 90 days apart (M = 88.9, SD = 6.4). During both visits participants completed a previously described face-matching task (Hariri et al., 2000) that has been shown to robustly activate the amygdala(Klumpp, Angstadt, & Phan, 2012; Labuschagne et al., 2010; Sripada, Angstadt, McNamara, King, & Phan, 2011). In this task participants match one of two cues (presented at the bottom of the screen) to a corresponding target (centered at the top of the screen) via a left or right button press using their dominant hand. In face-matching blocks, participants indicated which of the two facial cues (one of which is always neutral) matched the emotion of the target face (always happy, angry, or fearful). In the control condition participants matched simple geometric shapes. The current study paradigm consisted of three 20 second blocks for each angry, happy, and fearful face-matching conditions interleaved with shape-matching control blocks. The entire task lasted approximately 6 minutes.

fMRI Acquisition

Imaging data were acquired using a 3 Tesla GE Signa System (General Electric; Milwaukee, WI, USA). A T2*-weighted reverse spiral sequence was used for all participants with temporal resolution (TR) of 2000ms, although there was some variation in acquisition across different study control group participants that were combined in the current sample. For most participants, temporal echo (TE) = 25ms, flip angle = 77°, field of view (FOV) = 240×240mm, slice thickness = 5mm, 64×64 matrix with an inplane resolution of 3.75×3.75mm and a total of 30 axially acquired slices. Nine of the participants had slightly different parameters, TE = 30ms, flip angle = 90°, FOV = 220×220mm, slice thickness = 3mm, 64×64 matrix with a 3.44×3.44mm in plane resolution and a total of 32 slices. A reverse-spiral acquisition sequence specifically designed to reduce susceptibility artifacts(Stenger, Boada, & Noll, 2000, 2002, 2003)was used for all participants. Remaining scanning parameters were selected to be consistent with the work of Noll and colleagues. To ensure that the small differences in scan procedures within a small subgroup of our sample did not unfairly bias the results, scan subgroup was initially included as a covariate in second level analyses; however, the covariate had no significant effect on BOLD signal, and thus was dropped from the final model.

fMRIData Analysis

Data from the face-matching task were analyzed using Statistical Parametric Mapping software, version 8 (SPM8; http://www.fil.ion.ucl.ac.uk/spm/software/spm8). Data preprocessing for both scanning sessions followed SPM standard procedures, including motion and field warping correction, spatial normalization to the echo-planer imaging (EPI) template and 8mm Gaussian smoothing. A fixed-effects general linear model (GLM) was then created for the first and second scanning session of each individual participant separately to compare face-matching blocks to baseline condition (shape-matching blocks). Contrast images for the first scanning session only were then entered into a mixed-effects second-level GLM with faces minus shapes as a fixed effect and subject as a random effect, and a one-sample t-test of the face-shape contrast as created.

The resulting parametric t map was analyzed using a region-of-interest (ROI) approach. Briefly, areas of functional activation for faces in contrast to shapes were extracted from the amygdala and fusiform gyrus, bilaterally, as defined by the Automated Anatomical Labeling (AAL) atlas (Tzourio-Mazoyer et al., 2002)in conjunction with the Wake Forest University PickAtlas(Maldjian, Laurienti, Kraft, & Burdette, 2003). Parametric maps were corrected for multiple comparisons using a false-discovery rate (FDR) correction, thresholded at p < .05, and areas of significant activation within the anatomical labels for the amygdala and fusiformwere used to create the functional ROIs. Consistent with previous studies, functional ROIs were created using data from the first time point only, collapsed across emotional expression. These ROIs did not differ substantially from regions of activation for individualemotional expressions, at both time points (i.e., the area activated within the amygdala by angry faces at T1 & T2 showed >90% overlap with seed ROI). Finally, anatomical ROIs were created using anatomical labels from the AAL atlas.

Functional and anatomical ROIs were then analyzed using MARSBAR toolbox (Brett, Anton, Valabregue, & Poline, 2002), which estimates neural response for each participant individually, averaging across all voxels within the functional/anatomical ROI. Contrast estimates were extracted separately for fearful, angry and happy face-matching blocks in contrast to shape matching blocks for both session one and two. To assess differences in signal quality, signal to noise ratio (SNR), which is defined as the mean activation across time divided by the standard deviation, was calculated for each functional ROI at both time points.

To evaluate test-retest reliability, intraclass correlations (ICCs) were calculated using extracted contrast estimates from anatomical and functionally defined ROIs for all participants across both scan sessions. Consistent with the most recent study of reliability (Plichta et al., 2012), we calculated measures of absolute agreement, ICC_(2,1) (which reflects both measurement consistency and mean level differences), as well as consistency, ICC_(3,1) (which does not take mean level differences into account). Qualitative descriptions of ICC values are based on standards in psychometric research(Cicchetti and Sparrow, 1981; McGraw and Wong, 1996), and are consistent with recent studies of amygdala reliability(Plichta et al., 2012; van den Bulk et al., 2013): < 0.4 = poor, 0.41–0.59 = fair, 0.60–0.74 = good, and > 0.75 = excellent. ICCs were calculated in SPSS using a random effects two-way model, consistent with the work of McGraw and Wong (1996).

Results

Fearful, happy and angry faces all produced significantly greater activation in contrast to shapes within both the amygdala and FFA, bilaterally, at both time points (see Table 1). Moreover, patterns of activation were consistent across both time points, with cluster overlap ranging from 78–94% (see Figure 1). To assess potential differences in activation across hemisphere, time, and facial expression, a repeated measures ANOVA was performed for the amygdala and FFA, separately (see Figure 2). For the amygdala, there was significant main effect of expression, F(2,52) = 3.55, p< .05, with angry faces producing more amygdala activation than both fearful faces, t(26) = 2.27, p < .05, and happy faces, t(26) = 2.13, p < .05; and no difference in activation between fearful faces and happy faces, t(26) < 1. Amygdala activation was comparable across hemispheres, F(1,26) = < 1; though amygdala activation was greater at the first compared to second testing session overall, F(1,26) = 7.62, p < .05. There were no significant interaction effects (all p-values >.14).

Table 1.

Areas of significant activation within the anatomical boundaries of the fusiform gyrus and amygdala, bi-laterally.

Region	Coordinates	Z-score	Cluster Vol.	Overlap
Left FFA
T1	−33 −60 −15	7.04	8640	78%
T2	−34 −62 −16	6.00	7236
Right FFA
T1	33 −59 −15	7.02	10449	82%
T2	34 −60 −15	7.20	8991
Left Amygdala
T1	−23 −2 −16	5.75	1350	87%
T2	−24 −2 −18	5.58	1782
Right Amygdala
T1	26 −1 −17	6.93	1566	94%
T2	27 0 −19	3.52	1971

Notes. T1 = first assessment, T2 = second assessment. Coordinates are in Montreal Neurologic Institute (MNI) coordinate space, and reflect the center mass of each cluster of activation. Cluster volume is in mm³, and Overlap reflects the percentage of active voxels at T1 that are also active at T2.

Figure 1.

Figure 2.

Overall, the FFA response was equivalent across facial expressions, F(2,52) = < 1; however, this was qualified by a significant hemisphere X expression interaction, F(2,52) = 4.52, p < .05. Within the right FFA, angry faces produced greater activation than both fearful, t(26) = 2.30, p <.05, and happy faces, t(26) = 2.09, p < .05, with no difference in activation between fearful and happy faces t(26) = < 1; in the left FFA, angry faces produced significantly greater activation than happy faces, t(26) = 2.13, p<.05, but not fearful faces, t(26) = 1.79, p >.05, while activation for fearful and happy faces was comparable, t(26) = 1.37, p > .15. Like the amygdala, FFA activation was greater at the first time point, F(1,26) = 13.00, p < .01. There was no overall difference in activation across hemispheres, F(1,26) < 1. All other two-way and three-way interaction effects were non-significant (p >0.12).

A repeated measures ANOVA was also calculated to assess differences in SNR across region, time and hemisphere. In addition, to account for the potential impact of differences in scanning procedure among a subset of the sample, scan-type was included as a covariate. There was a significant main effect of region, such that the amygdala had significantly lower SNR (M = 160.31, SD = 10.54) than the FFA (M = 186.36, SD = 8.34), F(1,25) = 4.35, p< .05. There was no main effect of hemisphere or time, and interaction effects on SNR did not reach significance (all p-values >.30). Likewise, differences in scanning procedures had no significant impact on SNR (p > .15).

Estimates of the test-retest reliability of both the FFA and amygdala are presented in Table 2. For the functionally defined ROI, amygdala response to fearful faces was more reliable than response to happy or angry faces. Only fearful faces produced reliability estimates that were statistically different from zero for both the right and left hemisphere. Happy faces were less reliable, with significant ICC values for the left amygdala only. Reliability estimates for angry faces did not differ from zero in either hemisphere (see Table 2). Incontrast, activation within the FFA was much more consistent (see Figure 3). FFA response to faces was reliable over time regardless of facial affect or hemisphere, and reliability estimates in the FFA were generally higher as compared to the amygdala².

Table 2.

Intraclass correlation coefficients for the functionally-defined left and right FFA and amygdala, and the anatomically defined amygdala.

		Left			Right
		Fearful	Angry	Happy	Fearful	Angry	Happy
Amyg-FX	ICC(2,1)	0.32*	−0.12	0.35*	0.40*	0.11	0.23
	ICC(3,1)	0.34*	−0.12	0.37*	0.43*	0.11	0.24
Amyg-Anat	ICC(2,1)	0.42*	−0.11	0.19	0.36*	−0.24	0.21
	ICC(3,1)	0.41*	−0.10	0.20	0.38*	−0.24	0.21
FFA-FX	ICC(2,1)	0.50*	0.46*	0.62*	0.50*	0.37*	0.49*
	ICC(3,1)	0.63*	0.58*	0.68*	0.50*	0.42*	0.53*

Notes. ICC_(2,1)refers to absolute agreement measures of reliability, while ICC_(3,1) coefficients reflect consistency measures. FX refers to the functionally defined ROI, while Anat refers to anatomical definition.

^*Denotes ICC values that are significantly different from zero.

Figure 3.

In line with the work of Johnstone and colleagues (2005), we also evaluated the reliability of response within the anatomically defined amygdala. Results were largely consistent, suggesting that response within the amygdala is reliable for fearful faces, while neither happy nor angry expressions produced significant ICC estimates.

Discussion

In the current study we assessed longer term test-retest reliability of MR response to emotional faces within the amygdala. Results indicate moderate reliability over a 90-day interval, but only in response to fearful faces. Although angry faces produced stronger activation than both fearful and happy faces in the amygdala, this response was not reliable over time. These results were consistent for the left and right amygdala, and regardless of whether the amygdala was defined functionally or anatomically. Neural response to happy faces was reliable within the left (but not right) amygdala, and only when functionally defined.

As a point of contrast, the current study also evaluated reliability within the fusiform gyrus, an area of the brain with strong functional and anatomical connectivity with the amygdala that has been shown to respond consistently to facial stimuli (George et al., 2001; Kanwisher et al., 1997). Results suggest that MR signal from this region shows good reliability irrespective of facial affect. Specifically, reliability estimates for the FFA ranged from fair to good for all facial expressions, and this was true across both left and right hemispheres.

Studies of amygdala reliability typically evaluate reliability using functional ROIs, a subset of voxels within the amygdala that exceed a certain statistical threshold (i.e., a subset of the anatomical region). While this approach may lead to greater reliability estimates, it may result in less generalization of study findings to other emotional paradigms, which may not activate the same subset of voxels within the amygdala. To date, only one other study has reported reliability for the whole, atlas-defined amygdala (Johnstone et al., 2005). In that study, reliability was much lower for the anatomically defined amygdala than the functionally defined amygdala. However, reliability estimates in the current study were largely the same, whether evaluated for the whole anatomical amygdala or smaller regions based on functional activation.

Interestingly, estimates of BOLD signal in the amygdala were greater during the initial assessment as compared to follow-up, despite the fact that clusters of activation were larger at the second time point. While these results may simply reflect attenuation of amygdala signal as a result of habituation (Breiter et al., 1996; Fischer et al., 2003; Wright et al., 2001), it is also important to note that, consistent with previous studies of amygdala reliability, functional ROI definition was dependent on the first testing session only. Thus, despite substantial overlap in active voxels across testing sessions (see Figure 1; Table 1), attenuated signal seen at the second time-point may reflect the inclusion of non-significant voxels. A similar effect was also seen within the FFA, with greater BOLD signal seen at the first assessment. While the same habituation effects are not typically seen within the FFA, numerous studies indicate a functional link between the FFA and amygdala, with some studies suggesting that amygdala response directly modulates FFA response, and vice-versa(George et al., 2001; Pujol et al., 2009; Robinson, Laird, Glahn, Lovallo, & Fox, 2010; Sabatinelli et al., 2005). Finally, the similarity of reliability estimates taken from the functional ROI and anatomical amygdala suggest that these potential confounds have little to no effect on reported reliability.

Results of the current study suggest that amygdala response to emotional faces may be less reliable than response in other regions such as the FFA. This finding is consistent with previous studies that have found greater consistency in whole brain and regional activation as compared to amygdala response(Plichta et al., 2012; van den Bulk et al., 2013). In addition, we found significantly lower SNR within the amygdala compared to the FFA. Low SNR limits measurement precision, can lead to an underestimation of BOLD response(LaBar, Gitelman, Mesulam, & Parrish, 2001), and negatively impacts estimates of fMRI reliability (Bennett and Miller, 2010). Thus, lower reliability estimates for amygdala response to emotional faces could reflect lower SNR within this region.

Several studies have evaluated reliability of fearful faces, either alone or in conjunction with angry or both angry and happy faces. The first of these studies demonstrated good reliability of amygdala response to passively-viewed fearful faces across three scanning sessions over an 8 week period (Johnstone et al., 2005). When reliability was assessed across sessions at zero, two, and 8 weeks, estimates were higher than in the current study; however, estimates taken across the first and third scanning session were much more similar to the current study. Thus, the number of assessment points may be less important that test-retest interval. For instance, the recent study by Plichta and colleagues (2012) reported good to excellent reliability over a two-week period. Likewise, estimates of reliability over two weeks in the Johnstoneet al. (2005) paper were also strong. In both studies, reliability over a two week test-retest period was higher than over the longer test-retest period used in the current study. Attenuation of reliability estimates over long intervals may in part reflect the context-dependent nature of amygdala activation. For example, previous research suggests that amygdala response is vulnerable to the effects of state anxiety(Somerville et al., 2004), life stress(Lemogne et al., 2011), sleep and perceived stress(Prather et al., 2013), and reappraisal capacity(Drabant et al., 2009). Thus, increased temporal delay between testing sessions may result in greater variation in environmental context – an effect that may differentially impact amygdala reactivity and reliability.

The current study used very similar task procedures as the recent paper by Plichta and colleagues (2012). While the Plichta study suggests that reliability of amygdala response to threatening faces (i.e., angry and fearful) over a short test-retest period is strong, our results suggest more modest estimates over a much longer period of time. Reliability estimates in the current study may be particularly relevant to intervention researchers who seek to include fMRI measures of amygdala response as an index of treatment outcome, as the typical duration to test therapeutic effects in pharmacotherapy and psychotherapy clinical trials is approximately 90 days.

In the current study fearful, happy and angry faces all produced large areas of significant activation within the FFA and amygdala, at both time points. However, estimates of amygdala reliability for happy and angry faces were generally poor. One exception was that the left functionally-defined amygdala response to happy faces was reliable, and on par with that observed for fearful faces. In contrast, FFA reliability estimates were largely equivalent across facial affect and stronger, ranging from fair to good. Thus, facial expression appears to have a substantial impact on amygdala, but not FFA reliability.

These findings have important implications in applied neuroimaging research, as many studies have linked psychopathology to abnormalities of amygdala response. Psychopathology is trait-like and generally stable over time (Krueger, Caspi, Moffitt, & Silva, 1998)—and neural correlates and biomarkers of psychopathology should themselves have high stability and test-retest reliability (i.e., if a disease is trait-like, then a biomarker of that disease should be trait-like as well). However, results from the current study suggest that neither happy nor angry faces produce consistent activation of the amygdala over time, and that reliability in response to fearful faces is at best fair. If amygdala response demonstrates poor reliabilityover time, it is not clear how this neural measure could reflect trait-like differences that underlie psychopathology.

In the current study, measures of amygdala consistency, ICC_(3,1), did not exceed 0.41 for the amygdala, and ranged from 0.42 – 0.68 for the FFA. While this may be considered insufficient reliability in psychometric research(Nunnally, 1978), especially in the case of the amygdala, studies of fMRI reliability typically find ICCs in the 0.33 – 0.66 range (Bennett & Miller, 2010). However, as pointed out by Bennett and Miller, there is no consensus value of what constitutes an acceptable level of reliability in fMRI research. In other words, it is difficult to determine what constitutes a “good” fMRI measure. In the current study we attempted to address this question by concurrently evaluating the test-retest reliability of more basic visual processing of facial stimuli in the FFA. Reliability estimates were much higher within this region, suggesting that the maximum long-term reliability of fMRI measures of facial processing in the widely used face-matching task (Hariri et al., 2000) may exceed 0.60, which in behavioral research is considered “good” (McGraw and Wong, 1996). In comparison, amygdala activation is less reliable, indicating that amygdala response to emotional faces is more varied and perhaps subject to greater sources of measurement error.

Although the current study suggests that amygdala responsemay not be as stable and trait-like as previously believed, there remains significant evidence that the amygdala response to threatening stimuli is an index of psychopathology and intervention. The results of this study do, however, necessitate caution in the interpretation of studies that examine amygdala response over time. For example, previous studies clearly indicate that amygdala response is sensitive to psychopharmacological agents (Arce et al., 2008; Paulus et al., 2005; Phan et al., 2012). However, treatment studies have not consistently demonstrated intervention-related changes in amygdala activation (Fu, Steiner, &Costafreda, 2012), which may be partially due to low reliability of amygdala response over time – amygdala response cannot correlate with another measure (e.g. symptoms) better than it correlates with itself.

Just as reliability of measurement affects the necessary sample size to detect differences, the accurate assessment of the true reliability of a measure is dependent on the number of participants included in the study(Walter et al., 1998). The current study is the largest to date (N=27), allowing us to more accurately assess the reliability of amygdala response to emotional faces. The inclusion of both happy and angry faces in addition to fearful faces also allowed us to contrast the stability of amygdala response across expressions, and further confirmed that fearful faces most reliably and accurately recruit the amygdala (Costafreda et al., 2008; Phan et al., 2002). Further, the task used in the current study has shown consistent findings in many populations and has been used in other recent studies of amygdala reliability, allowing for a more direct comparison of study results.

However, there are several limitations in the current study. The inclusion of three task conditions necessarily reduced the number of assessments of neural response to these stimuli, and there is evidence to suggest that reliability of fMRI data may be reduced as a result (Bennett and Miller, 2010). Likewise, several factors including magnet strength, length of MR acquisition, type of stimuli, MR parameters and preprocessing methodology all influence the stability and strength of amygdala response(Bennett and Miller, 2010; Hariri et al., 2002; LaBar et al., 2001; Robinson, Windischberger, Rauscher, & Moser, 2004), and will differ substantially across studies. Although scan acquisition parameters were specifically designed to reduce signal artifact within the amygdala (Stenger et al., 2000, 2002, 2003), previous research has demonstrated that small variations in parameters such as temporal echo (TE), voxel size, slice orientation, Z-shim and spiral sequence can all have significant effects on BOLD signal contrast and susceptibility artifacts and SNR (Chen, Dickey, Yoo, Guttmann, & Panych, 2003; Du, Dalwani, Wylie, Claus, & Tregellas, 2007; Glover and Law, 2001; Gorno-Tempini et al., 2002; Merboldt et al., 2001; Morawetz et al., 2008). In the current study, we did not assess how variations in these parameters may impact reliability estimates of BOLD response in the amygdala and FFA. As such, the results of the current study may not apply equally to other fMRI studies of amygdala response.

The current study includes only healthy individuals and does not necessarily reflect expected reliability estimates in clinical groups, which may be more variable than healthy individuals (Krueger et al., 1998). Furthermore, although the current study is the largest such study to date, our sample still included less than 30 individuals. Reliability studies typically include hundreds of participants, thus the current study may not be accurately assessing the limits of amygdala reliability. Future studies of amygdala reliability that include more individuals are needed, allowing us to determine more specifically the effects of task conditions, scanning methodology and sample characteristics.

There is an increasing emphasis on research that seeks to identify biomarkers of psychiatric illness, and neural markers evaluated using fMRI are of particular interest (Insel et al., 2010; Singh & Rose, 2009). However, the utility of amygdala response as a potential neuralmarker of psychiatric illness is by necessity linked to its reliability. Low reliability reduces power, and in turn requires increases in scanning acquisition time and number of participants, thus increasing utilization cost(Perkins, Wyatt, & Bartko, 2000). More importantly, low reliability limits the validity of a measure. A measure cannot correlate better with another measure than it correlates with itself, and measurement reliability is a necessary precondition for measurement validity (Shrout, 1998). The current study suggests poor to fair reliability of amygdala response to fearful faces over test-retest periods of several months. Although within the range expected based on reviews of studies of fMRI reliability (Bennett and Miller, 2010), clinical utility of this measure may require multiple assessments, shorter test-retest periods, or longer scanning sessions. The degree to which an individual activates their amygdala in response to fearful faces is somewhat stable over time – but not nearly as stable as other individual difference measures (e.g., self-report measures of anxiety or depression). The current results suggest caution in the trait-like interpretation of amygdala response to fearful faces, and even moresowith regard to happy and angry faces.

Conclusions

Findings from the current study suggest that the reliability of single assessment of amygdala response to fearful faces is at best fair over a 12-week period. In contrast, FFA reliability was fair to good, irrespective of facial affect. Future studies should evaluate potential differences in amygdala consistency within clinical populations and assess reliability of amygdala response to other types of stimuli frequently utilized in affective and clinical neuroscience. Meanwhile, further assessment of the impact of scan acquisition parameters on reliability within the amygdala and other regions of the brain is necessary.