Better together: A systematic review of studies combining Magnetic Resonance Imaging with Ecological Momentary Assessment

Abstract

Social neuroscientists often use magnetic resonance imaging (MRI) to understand the relationship between social experiences and their neural substrates. Although MRI is a powerful method, it has several limitations in the study of social experiences, first and foremost its low ecological validity. To address this limitation, researchers conduct multimethod studies combining MRI with Ecological Momentary Assessment (EMA). However, there are no existing recommendations for best practices for conducting and reporting such studies. To address the absence of standards in the field, we conducted a systematic review of papers that combined the methods. A systematic search of peer-reviewed papers resulted in a pool of 11,558 articles. Inclusion criteria were studies in which participants completed (a) Structural or functional MRI and (b) an EMA protocol that included self-report. Seventy-one papers met inclusion criteria. The following review compares these studies based on several key parameters (e.g., sample size) with the aim of determining feasibility and current standards for design and reporting in the field. The review concludes with recommendations for future research. A special focus is given to the ways in which the two methods are combined analytically and suggestions for novel computational methods that could further advance the field of social neuroscience.

The field of social neuroscience aims at understanding the interplay between neurobiology and social experiences. A major challenge in the field, particularly when using structural or functional Magnetic Resonance Imaging (s/fMRI) to assess neural processes, is low ecological (external) validity (e.g., Poldrack, 2011; Shamay-Tsoory & Mendehlson, 2019). To increase internal validity and experimental control, the measurement of neural activation in s/fMRI is typically done under conditions that are fundamentally different from social experience, i.e., when individuals are mostly passive, practically immobile, and socially isolated, and with carefully contrived, typically static, stimuli (McGowan et al., 2023; Osborne-Crowley, 2020; Shamay-Tsoory & Mendehlson, 2019). Moreover, s/fMRI scans are typically conducted once, whereas human behavior is highly situation-dependent (Mischel & Shoda, 1995). Finally, fMRI measures typically suffer from low internal and test-retest reliability, thus limiting their ability to identify individual or developmental differences (e.g., Elliott et al., 2020; Kennedy et al., 2022) that are often of interest for social neuroscientists.

To address the challenge of limited ecological validity of s/fMRI, researchers have increasingly supplemented s/fMRI with ecological momentary assessment (EMA). EMA involves repeated assessments of thoughts, emotions, and behavior in everyday life, typically by having participants complete brief self-report questionnaires at least once a day over several weeks. The combination of s/fMRI and EMA helps generalize s/fMRI findings to real-world situations and to different states, important characteristics of ecological validity (McGowan et al., 2023; Osborne-Crowley, 2020). In addition to its high ecological validity (Bolger & Laurenceu, 2013), EMA has several strengths that make it a good counterpart to s/fMRI. First, EMA allows studying both state and trait characteristics, via simultaneous examination of between-individual and within-individual processes, (e.g., asking not only who experiences certain emotions, but also when). Teasing apart state and trait facets and examining their interactions is of particular importance in social neuroscience, as it can help researchers understand not only who behaves in a particular way, but also when they are more likely to do so. Second, EMA can be utilized to assess psychological processes in indirect ways that do not rely on participant insight. For example, instead of asking individuals whether their motives for certain behaviors are interpersonal, EMA can be used to detect whether, for example, interpersonal affect (e.g., loneliness) changes before and after the behavior of interest (e.g., Snir et al., 2015). Third, EMA can capture the dynamic aspects of variables – the pattern of change over time (e.g., emotional instability). Moreover, it can be argued that human experience comprises of psychological, biological and environmental layers (McGowan et al., 2022). While s/fMRI can assess biological and psychological aspects, EMA can assess psychological and environmental aspects. Thus, together they give us a more complete understanding of human experience.

Along with its advantages, EMA has some limitations. First, EMA’s high ecological validity comes hand-in-hand with noise. Individuals are typically not exposed to the same events; thus, many alternative explanations are possible. Statistical methods such as mean-centering can help to alleviate some of these issues, but they do not correct them completely (Bolger & Laurenceu, 2013). Second, EMA is based on self-report, therefore influenced by biases, although the frequent assessments reduce memory biases, and repeated measurements reduce noise (Russell & Gajos, 2020). Finally, although statistical methods can help to determine directionality and reduce third-variable explanations, the results from EMA cannot be used to infer causality (Bolger & Laurenceau, 2013).

The present study

The current systematic review’s goals are (1) To summarize the current state of research combining s/fMRI and EMA. (2) To make recommendations and provide social scientists guidelines when planning this type of study. (3) Offer new directions in data analysis.

Method

We carried out a systematic review following the recommendations of the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement (Liberatti et al., 2009).

Study selection criteria.

Studies were included if they met the following criteria: Participants’ data included (1) structural or functional MRI scans and (2) EMA, defined as repeated assessments, at least once a day, for at least three consecutive days; (3) EMA included a self-report component. Studies including only passive sensing (e.g., GPS) were excluded (see Camacho et al., [2021] for a review of studies combining MRI with passive sensing); (4) Participants were neurologically intact (5) papers were written in English; and were (6) published in peer-reviewed journals.

Search strategy, study selection and data extraction.

We searched Google Scholar using the following search string: (EMA OR ESM OR “ecological momentary assessment” OR “experience sampling method”) AND (fMRI OR neuroimag* OR “functional MRI” OR “functional magnetic resonance imaging”). We searched by year for the years 1996–2020 (including). Search results from each year were downloaded into CSV files and inspected manually. The search was completed in June 2021. We also performed backward and forward searches on included papers. Figure 1 presents the PRISMA flow diagram. Table S1 in supplementary materials includes full-text articles excluded with reasons. Articles were coded by two coders (one of whom was the first author), and discrepancies were resolved by discussion. Table S2 includes the full list of papers and characteristics that were coded.

Figure 1.

PRISMA Flow Diagram

Results

Seventy-one published papers¹ were included in this review, summarized in Table 1. As can be seen in Figure 2, most papers (40 papers; 56%) were published between 2016–2020, indicating a trend of increased use of this multimethod design.

Table 1.

Summary of 71 Studies that Combine MRI/fMRI and EMA, 2007–2020, in chronological order by year of publication.

Studya	Study populationb	Sample Sizec	EMA assessments (Prompts/Days)d	EMA constructs	MRI measures/constructs	Method order
Eisenberger et al. (2007)	Non-clinical; adults, age=20.7e	30	80/10	Social interactions, affect	Social	EMA before fMRI
Eisenberger et al. (2007)	Non-clinical; adults, age=18–36	30	80/10	Social interactions, affect	Social	EMA before fMRI
Forbes et al. (2009)	Clinical depression; adolescents, age=8–17	43	12/4	Social interactions, affect	Reward processing	EMA before fMRI
Holm et al. (2009)	Non-clinical; adolescents, age=11–13	58	NA/4i	Sleep	Reward processing	Varied
Forbes et al. (2010)	Non-clinical; adolescents, age=11–13	77	12/4	Social interactions, affect	Reward processing	Varied
Hooker et al. (2010)	Non-clinical; adults in committed relationships; age=21e	27	21/21	Social interactions, affect, substance use	Social	-
Berkman et al. (2011)	Heavy cigarette smokers; adults, age=28–69	27	168/21	Affect, substance use, craving	Regulation/inhibition	fMRI before EMA
Forbes et al. (2011)	Non-clinical; adolescents, age=11–13	76	12/4	Affect	Emotion processing	Varied
Habets et al. (2012)	Psychotic disorders; adolescents and adults, age=16–55	89	60/6	Social interactions, affect	Structural/DTI	EMA before sMRI
Hasler et al. (2012)	Non-clinical; adolescents, age=11–13	56	12/4	Affect	Reward processing	Varied
Masten et al. (2012)	Non-clinical; adolescents, mean age=17.77	21	14/14	Social interactions	Social	EMA before fMRI
Rameson et al. (2012)	Non-clinical; adults, age=19i	32	14/14	Social interactions	Social	EMA before fMRI
Bourne et al. (2013)	Non-clinical; adults, age=22e	22	6.82/7	Psychopathology symptoms	Emotion processing	fMRI before EMA
Collip et al. (2013)	Psychotic disorders; adolescents and adults, age=16–55	89	60/6	Social interactions, affect	Structural/DTI	EMA before sMRI
Winkleman et al. (2013)	Primary insomnia; adults, age=18–60	76	14/14	Sleep	Structural/DTI	EMA before sMRI
Chester & DeWall (2014)	Non-clinical; adults; mean age=18.92e	37	7/7	Social interactions, affect, substance use, self-regulation	Social, regulation/inhibition	EMA before fMRI
Dodell-Feder et al. (2014)	At-risk for schizophrenia; adolescents and adults, age=15–32	32	28/28	Social interactions, affect	Social	fMRI before EMA
Hooker et al. (2014)	Non-clinical; adults, age=19–52	30	21/21	Affect, psychopathology symptoms	Social	fMRI before EMA
Kashdan et al. (2014)	Non-clinical; adults, age=20.9e	25	21/21	Social interactions, affect, self-esteem	Social	EMA before fMRI
Lopez et al. (2014)	Non-clinical; adults, age=18–28	28	49/7	Craving	Regulation/inhibition	fMRI before EMA
Morelli et al. (2014)	Non-clinical; adults, age=19.9e	32	14/14	Social interactions	Social	fMRI before EMA
Olino et al. (2014)	At-risk for depression; children and adolescents, age=8–17	26	12/4	Affect	Reward processing	-
Rahdar & Galvan (2014)	Non-clinical; adolescents and adults, age=15–30	45	42/14	Affect	Regulation/inhibition	fMRI during EMA period
Tully et al. (2014)	Schizophrenia; adults, age=18–65	47	21/21	Affect, psychopathology symptoms	Emotion processing	fMRI before EMA
Giuliani et al. (2015)	Non-clinical (food restriction intervention)f; adults, age=18–30	46	56/14	Craving	Regulation/inhibition	fMRI before EMA
Heller et al. (2015)	Non-clinical; adults, age=18–24	40	250/10	Affect, reward processing	Reward processing	EMA before fMRI
Telzer et al. (2015)	Non-clinical; adolescents, age=14–17	46	28/28	Social interactions	Risk-taking	EMA before fMRI
Dodell-Feder et al. (2016)	Non-clinical; adults in committed relationships, age=18–35	22	21/21	Social interactions, affect, self-esteem	Social	fMRI before EMA
Lopez et al. (2016)	Non-clinical (chronic dieters)f; adults, age=18–23	69	49/7	Craving	Regulation/inhibition	fMRI before EMA
Powers et al. (2016)	Non-clinical; adults, age=18–21	33	49/7	Social interactions	Social, emotion processing	fMRI before EMA
Price et al. (2016)	Clinical anxiety; children and adolescents, age=9–14	78	14/5	Affect, psychopathology symptoms	Emotion processing	fMRI before EMA
Fischer et al. (2017)	Bulimia nervosa and EDNOS; adults, age=18–40	16	98/14	Affect, craving, psychopathology symptoms	Regulation/inhibition	fMRI before EMA
Lee et al. (2017)	Non-clinical; adolescents and caregivers, age=13–57	53	14/14	Sleep	Resting state	-
Lee et al. (2017)	Non-clinical; adolescents and caregivers, age=13–57	53	14/14	Affect	Resting state	fMRI before EMA
Lopez et al. (2017)	Non-clinical (chronic dieters)f; adults, age=18–23	69	49/7	Craving	Regulation/inhibition	fMRI before EMA
Pace-Schott et al. (2017)	Primary insomnia; adults, age=mid-30si	26	14/14	Sleep	Resting state	EMA before fMRI
Servaas et al. (2017)	Remitted clinical depression; adults, age=mid-50si	69	60/6	Social interactions, affect, self-esteem	Resting state	EMA before fMRI
Uy & Galvan (2017)	Non-clinical; adolescents and adults, age=15–30	45	42/14	Affect	Risk-taking	fMRI during EMA period
Uy & Galvan (2017)	Non-clinical; adolescents and adults, age=15–30	44	42/14	Affect	Risk-taking	fMRI during EMA period
Waugh et al. (2017)	Non-clinical; adults, age=21–65	29	7/7	Affect	Social	fMRI before EMA
Wonderlich et al. (2017)	Bulimia nervosa; adults, age=18–40	16	98/14	Affect, craving, psychopathology symptoms	Regulation/inhibition	fMRI before EMA
Bastiaansen et al. (2018)	Non-clinicalg; adults, age=18–25	69	70/14	Affect	Emotion processing	EMA before fMRI
Chen et al. (2018)	Non-clinical (chronic dieters)f; adults, age=19e	36	98/14n	Craving	Regulation/inhibition	fMRI before EMAj
Flores et al. (2018)	Non-clinical; adolescents, age=14–18	33	28/10	Social interactions, affect	Reward processing, social reward	EMA before fMRI
Ismaylova et al. (2018)	Non-clinical; adults, age=32–36	42	5/5	Affect	Emotion processing, structural/DTI, resting state	f/sMRI before EMA
Kluge et al. (2018)	Schizophrenia; adults, age=32i	18	40/4	Social interactions, affect	Reward processing	EMA before fMRI
Kronke et al. (2018)	Non-clinical; adults, age=20–26	118	56/7	Social interactions, substance use, self-regulation, craving	Regulation/inhibition	fMRI before EMA
Nook et al. (2018)	At-risk for schizophrenia; adolescents and adults, age=15–32	36	28/28	Social interactions, affect, psychopathology symptoms	Social	fMRI before EMA
Provenzano et al. (2018)	Non-clinicalg; adults, age=18–25	65	70/14	Affect	Social	EMA before fMRI
Seidel et al. (2018)	Anorexia nervosa; adolescents and adults, age=12–29	70	84/14	Affect, psychopathology symptoms	Emotion processing	fMRI before EMA
Wonderlich et al. (2018)	Bulimia nervosa; adults, age=18–40	16	98/14	Affect, psychopathology symptoms	Regulation/inhibition, emotion processing	fMRI before EMA
Bakker et al. (2019)	Clinical depression; adolescents and adults, age=16–25	87	150/15	Social interactions, affect, reward processing	Reward processing	fMRI before EMA
Butterfield et al. (2019)(5)	Clinical anxiety; children and adolescents, age=9–14	120	14/5	Affect	Emotion processing	EMA before fMRI
Grosse-Rueschkamp et al. (2019)	Non-clinical; adults, age=18–25	63	60/10	Affect	Emotion processing	Varied
Kaiser et al. (2019)	Clinical depression; adolescents, age=13–19	28	14/14	Affect	Emotion processing	fMRI before EMA
Moran et al. (2019)	Schizophrenia; adults, age=mid-30si	58	28/7	Affect, reward processing	Reward processing	EMA before fMRI
Rattel et al. (2019)	Non-clinical; adults, age=22.9e	53	NA/4i	Affect, psychopathology symptoms	Emotion processing	fMRI before EMA
Scholz et al. (2019)	Non-clinicalh; adults, age=19–33	52	90/30	Social interactions, affect, substance use	Regulation/inhibition	fMRI before EMA
Schwartz et al. (2019)	Clinical depression; adults, age=18–35	64	105/21	Social interactions, affect, psychopathology symptoms	Resting state	fMRI before EMA
Tost et al. (2019)	Non-clinical; adults, age=18–28	52	87/7	Affect	Emotion processing	EMA before fMRI
Alarcon et al. (2020)	Non-clinical; adolescents, age=14–18	45	28/10	Social interactions, affect	Social reward	fMRI before EMA
Blaine et al. (2020)	Alcohol use disorder; adults, age=21–60	69	14/14	Substance use	Regulation/inhibition	fMRI before EMA
Culbreth et al. (2020)	Schizophrenia; adults, age=37i	28	28/7	Reward processing	Reward processing	-
Heller et al. (2020)	Non-clinical; adults, age=18–31	58	22–78/3–4	Affect, sleep	Resting state	EMA before fMRI
Hua et al. (2020)	Mood disorders; adults, age=25e	26	100/11–16	Affect	Emotion processing	fMRI before EMA
Kronke et al. (2020)	Non-clinical; adults, age=19–27	294	56/7	Self-regulation	Resting state	-
Martin-Soelch et al. (2020)	At-risk for depression; adults, age=18–40	32	28/7	Affect	Reward processing, emotion processing	fMRI before EMA
Oppenheimer et al. (2020)	Clinical anxiety; adolescents, age=11–16	36	28/10	Affect	Social	EMA before fMRI
Uy & Galvan (2020)	Non-clinical; adolescents and adults, age=15–30	38	42/14	Affect	Social	fMRIs during EMA period
Vekaria et al. (2020)	Non-clinical; adults, age=21–55	48	14/14	Social interactions	Social	fMRI before EMA
Woods et al. (2020)	Non-clinical; adolescents, age=14–18	48	28/10	Affect	Social	EMA before fMRI

Note. Information that was not reported as part of the publications is marked with a single dash. For example, the presence of a dashed line under “Compensation” only indicates that information regarding participant compensation was not reported in the publication; it is not representative of authors explicitly stating that participants were not compensated.

^aIncludes the first author and the year of publication

^bDescribes the population of interest and the age range of participants in years. For clinical populations, the diagnostic group of interest is listed.

^cSize of the sample that completed both EMA and fMRI, and were included in analyses for each paper

^dNumber of overall number of prompts that participants received over (/) total number of EMA days. “NA” denotes that overall number of prompts is unspecified. Some studies reported average or range of prompts completed instead of overall number of prompts; in those cases the average or range are reported the right side of the dash (/).

^eSome papers reported the mean age of participants whose data was included in analyses (in years) rather than the age range of participants

^fNon-clinical study populations with specific criteria are described in parentheses, such as chronic dieters (Lopez et al. 2016; Lopez et al. 2017; Chen et al. 2018) and non-clinical participants that were assigned to a food restriction intervention as part of the study (Giuliani et al. 2015)

^gStudies by Bastiaansen et al. (2018) and Provenzano et al. (2018) used non-clinical study populations, though they selected participants such that they would have a normal distribution of neuroticism within their samples

^hScholz et al. (2019) selected young adult participants who reported drinking a standard drink containing alcohol at least 2–3 times per month or more for the last 12 months

ⁱStudies by Holm et al. (2009) and Rattel et al. (2019) did not specify the overall number of prompts sent to participants throughout the EMA period. The study by Rattell et al. (2019) used an event-based design and the number of overall EMA prompts differed between participants

^jIn Chen et al. (2018), participants completed two fMRI scans and two sets of EMA. They completed one fMRI scan, followed by a 14-day EMA. This sequence was then repeated. The overall number of EMA prompts reported represents one 14-day EMA period

Figure 2.

Number of Identified Papers Published Each Year, 2007–2020

Practical considerations

Participants

Sample size varied between papers, with an average number of 50.94 participants (SD=37.02, Median=45, range=16–294). Sample size justification was not provided in most of the papers (N=60; 84.5%). Of those that justified sample size, six papers argued that their sample size is within standards from previous literature. Five papers conducted a power analysis based on one of the methods (EMA or s/fMRI). None of the papers reported the amount of time it took to recruit the samples.

Measurement

S/FMRI.

Participants typically completed one s/fMRI scanning session. Five papers (7%) scanned participants twice. Most of the papers reported data from a task-based fMRI (N=60; 85%), seven papers (10%) reported resting-state fMRI, three papers (4%) used structural MRI or diffusion tensor imaging, and one paper (1%) used a combination of structural, resting state, and task-based fMRI scans.

EMA.

On average, papers reported data from a 13.24 days-long EMA protocol (SD=12.78, Median=14, range 4–105) with a mean of 4.62 daily assessments (SD=4.18, Median=3, range 1–25), generating an average total of 46.82 separate assessments per paper (SD=42.95, Median=28, range 5–250)².

Thematic issues

Study population.

As can be seen in Table 1, the populations studied are diverse: the age range of participants spanned from 8 to 69 years, with 27 papers (38.0%) including participants younger than 18 years. Thirty-three papers (46.48%) focused on individuals who were either diagnosed with or were at risk for a psychiatric disorder, or with psychopathology-related traits (e.g., chronic dieters). Finally, two papers (3%) examined romantic couples. In other words, most of the papers included in the review (N=51, 72%) focused on populations that are more difficult to recruit, namely clinical and/or developmental populations or romantic couples

Psychological constructs

EMA-measured constructs.

The most frequently assessed constructs were negative or positive affect and stress (N=51, 71.8%), followed by social interactions (N=26; 36.6%). Two additional frequent themes were self-regulation/craving/consumption of substances (e.g., alcohol; N=14; 19.7%), and psychopathology symptoms (N=11; 16%). Additional constructs were sleep-quality (N=5; 7%), reward-processing (N=4; 6%), and self-esteem (N=3; 4%).

S/FMRI constructs.

Using task-based fMRI (N=60), the constructs assessed were social (N=21, 30%) and emotional³ (N=16, 22%), inhibition/self-regulation (N=14, 20%), reward-processing⁴ (N=13, 18%), and risk-taking (N=3, 4%).

Combining s/fMRI and EMA

Order of measurement.

We identified four research designs: (a) s/fMRI first, followed by EMA. This order was used by 33 of the papers (46%). (b) EMA first, followed by s/fMRI. This sequence was used by 24 of the papers (34%). (c) s/fMRI scans during EMA period. In four studies (6%), participants completed a baseline period of EMA to assess mean stress levels and were invited for two MRI scans within the EMA period – one scan on a day with lower-than-average stress, and a second scan on a day with higher-than-average stress (e.g., Uy & Galván, 2017a). The four papers originated from the same research group. (d) In five papers (7%), participants completed the two methods in varying orders. One of them (Grosse Rueschkamp et al., 2019) counterbalanced the order of the s/fMRI and EMA. Finally, five papers (7%) did not report any information regarding the order in which participants completed the methods.

Similarities in measured constructs.

Constructs were similar in both methods in 86% of the papers that used fMRI to assess inhibition/regulation (N=12), 88% of the papers that used fMRI to assess emotion processing (N=15), 71% of the studies that used fMRI to assess social constructs (N=15), 31% of the studies that used fMRI to assess reward processing (N=4), and none of the studies that used fMRI to assess risk-taking.

Increasing the ecological validity of fMRI tasks

Using individualized stimuli.

One option for increasing ecological validity of fMRI is using meaningful, personalized stimuli (e.g., pictures of loved ones) in addition to the EMA. Five papers (7%) adopted this strategy.

Bringing the social context into the lab.

Another option for increasing ecological validity is scanning participants’ close others. Four papers included in the current review (6%) adopted this strategy. None of these studies scanned dyad members while interacting with each other (i.e., hyperscanning).

Analytic methods

Brain-EMA correlations.

One way of combining EMA data with s/fMRI data is averaging measures across the EMA period and computing a correlation with activation extracted from a Region of Interest (ROI). Twenty-four of the papers in this review used this as a primary or secondary method. For example, Forbes et al. (2009) examined associations between mean positive affect across the EMA period and activation in striatal clusters in which the response to reward was different for participants with and without depression.

Brain-derived variables as mediators.

Four papers (6%) computed averages from across the EMA period and tested whether brain activation or connectivity could serve as a mediator between the averaged EMA variable and an additional variable (or vice versa). For example, Price et al. (2016) reported that amygdala-dlPFC connectivity mediated the association between attentional vigilance during a behavioral task and distraction during the EMA.

Dynamic indices.

EMA can be used to capture the dynamic aspects of processes that are difficult to capture otherwise. Twenty-one papers (30%) extracted these types of dynamic indices from the EMA, typically emotional dynamics (e.g., affective instability; Servaas et al 2017). Another analytic choice included in this category is computing the correlations between two variables in the EMA, such as social distress and social disconnection (Eisenberger et al., 2007), and then examining whether these associations were related to neural activity.

Brain as a moderator of momentary processes.

The most common method (N=22; 31%) for combining the s/fMRI and EMA was examining whether neural activation or brain structure moderates the associations between momentary variables. For example, Hooker et al. (2010) examined whether neural activation while watching a romantic partner’s negative (vs. positive) emotional expressions moderated the associations between daily interpersonal conflict and mood.

EMA as a moderator of neural processes.

Five papers (7%) analyzed EMA as a moderator of neural responses. For example, Bourne et al. (2013) had participants watch potentially traumatic videos during fMRI, and then complete an EMA every time they experienced a flashback from that video. The fMRI contrast compared videos that were classified as flashback-provoking with videos that were not.

Dimensionality reduction across methods.

One study (Giuliani et al., 2015) combined assessments across methodologies. This study examined reactivity and regulation of food craving, and measured these constructs using self-report, EMA, and fMRI. Assessments from each method were extracted and fed into a principal component analysis (PCA). Then, results from the factor analysis were entered into a regression analysis to predict food consumption.

Discussion

The current systematic review provides an organized account of common research practices and reporting standards for papers combining structural or functional MRI (s/fMRI) and ecological momentary assessment (EMA), a combination which is particularly relevant for the field of social neuroscience. Below we describe what such studies typically include, limitations, and recommendations for future research, along with a section about future analytical directions.

1. Current practices in s/fMRI-EMA studies

Who are the participants?

The current review demonstrates that the combination of s/fMRI and EMA is feasible and worthwhile even for populations that are difficult to recruit, such as developmental and psychiatric populations.

What are the main constructs?

The constructs assessed most frequently were social interactions and affect. Not all studies assessed the construct of interest using both methods. From a multimodal perspective, using different methods to capture similar processes can facilitate finding meaningful shared variance across modalities. The relative ease of assessing additional constructs via EMA can be leveraged in future studies to determine the convergent and divergent validity of fMRI by examining which EMA constructs are associated with neural activation and which ones are not, which is pertinent to the problem of reverse inference (Poldrack, 2006).

Analytical choices.

Many of the studies used neural activation or structure as a moderator of daily processes, thus adopting a person-by-situation approach. Conversely, several studies extracted variables from EMA that capture the dynamic nature of psychological processes (e.g., emotion instability) and examined their neural correlates. Only one paper used a multivariate approach to reduce the dimensionality of the data across methods (Giuliani et al., 2015). Almost all the papers reported at least one significant finding. Significant associations between levels of analysis are not trivial, considering that coherence across levels can be weak (e.g., Mauss & Robinson, 2009). Because none of the included papers were pre-registered, it is unknown how many comparisons were made between the methods. We discuss this topic more in the next section.

Advancing the literature.

The studies reviewed contributed to the literature in numerous ways. In this section we demonstrate several ways by which the contribution was related to the multimethod approach they used. This is by no means an exhaustive list, but it hopefully demonstrates the benefits of combining s/fMRI and EMA. One way in which studies that combine f/sMRI and EMA can contribute to the literature is by helping translate f/sMRI findings from the neural level to the psychological level (i.e., showing to which psychological outcomes neural structure or function is associated with). In the process of this interpretation, the combination can facilitate understanding f/sMRI findings in a more fine-tuned way. With task-based fMRI, for example, multiple regions are activated. When the fMRI tasks assess more complex, social processes, it is often difficult to find a good experimental control that helps interpret what the activation is related to, thus limiting both internal and external validity. EMA can, in these cases, contribute to interpretation of the fMRI results. For example, Woods et al. (2020) had adolescent participants watch videos of their close friends and videos of unfamiliar peers while in the scanner, and then report positive affect and interactions with friends using EMA. The fMRI showed that adolescents had greater activation in several regions related to reward processing and emotion regulation (e.g., orbitofrontal cortex, dorsomedial prefrontal cortex, anterior insula) when viewing their friend (vs. stranger). The sources of the difference in activation might have been related to the different degrees of familiarity between the conditions, the different emotions evoked by seeing a friend vs. stranger, or other aspects of the interaction viewed (as they could not be completely identical). The combination of the methods allowed the researchers to find that of all the brain regions identified, only the anterior insula was associated with increases in positive affect following encounters with friends in daily life. Thus, using EMA in addition to fMRI helped better understand the activation. This higher specificity in interpretation of brain activity may be particularly important when using (as Woods et al., 2020 did) an ecologically valid, personalized stimuli, to which it is difficult to find a perfect control.

A second way by which studies combining f/sMRI with EMA can advance the literature is by showing connections between different layers of human experience, thus advancing a more complex and complete understanding. A good example of this can be seen in the paper by Heller et al. (2020), who used EMA to follow geographical locations and affect, followed by a resting-state fMRI. The EMA showed that exposure to more diverse geographical locations was associated with higher positive affect. They used the strength of the association between location diversity and positive affect to guide their connectivity fMRI analysis, and found that the connectivity between the hippocampus (which processes locations) and the nucleus accumbens (which is part of the reward system) was stronger for individuals who experienced more positive affect with increased location diversity. These findings allowed them to suggest a neural mechanism by which diversity in spatial environment can be subjectively rewarding, thus leading to a better understanding of both neural and psychological levels, in addition to the association between them.

A third way by which the f/sMRI-EMA combination can advance the literature is by using EMA to invite participants to be scanned, thus capturing neural activation under different environmental conditions. For example, Uy and Galván (2017a) had participants complete a risk decision-making task during fMRI once on a day in which they experienced lower stress than they typically do, and once on a day in which they experienced higher stress than they typically do; Stress levels were determined using the EMA. They found that adolescent boys (but not girls or adult men or women) showed stress-related decreases in prefrontal activation when making risky decisions. Furthermore, the authors explain how using EMA to determine stress levels is superior to manipulating stress during the fMRI scan because traditional stressors may affect adolescents and adults differently, whereas EMA-assessed stress reflects developmentally-appropriate daily stressors (Uy & Galván, 2017a).

2. Limitations & recommendations.

Our review helped identify several limitations of the existing literature. A common limitation across the included papers is that sample sizes were rarely justified. The inclusion of power analyses is necessary for the field to advance. Relatedly, pre-registration of research hypotheses was not done. This is a notable issue because both s/fMRI and EMA produce rich datasets. Together with the typically limited sample sizes in studies of this kind, this raises questions regarding robustness and reproducibility. Pre-registered, hypotheses-driven analyses are one way to address this concern. Additionally, we propose that researchers should consider multimodal, computational methods. Such approaches include (but are not limited to) multimodal modeling and analyses, pattern recognition, and machine learning. In the next section we present several recommendations for analytic approaches that can help uncover patterns that emerge from the integration of these two modalities.

Another limitation of the literature described is that fMRI has extremely low internal and test-retest reliability (Elliott et al., 2020; Kennedy et al., 2022; cf. structural MRI – Hedges et al., 2022), thus limiting researchers’ ability to find a valid, reproducible connection between the two modalities. Indeed, most of the reviewed studies focused on individual differences, developmental, or clinical effects – that was, we presume, the main motivation for adding the EMA to the s/fMRI. As such, we strongly urge researchers who contemplate the undertaking of an s/fMRI-EMA study to consult the literature and adopt procedures that increase MRI reliability (Botvinik-Nezer & Wager, 2023; Elliott et al., 2020; Elliott et al., 2021; Hedges et al., 2022; Kennedy et al., 2022; Kragel et al., 2021). Though not all recommendations are applicable in every situation, they include (but are not limited to): (1) Using structural MRI measures more often (they were rarely used in the papers we review), as they are highly reliable, (2) Because movement in the scanner decreases reliability, adopting procedures that are known to help reduce motion during the scan (e.g., mock scanner practice prior to fMRI) or dealing with motion during processing (e.g., censoring), (3) Obtaining longer runs of data, (4) Using more ecologically valid or even personalized stimuli, as some of the reviewed papers have done, (5) Using multivariate measures (vs. examining specific regions). The last recommendation can be applied by adopting machine-learning methods such as we suggest in the next section. A summary of our recommendations can be found in Table 2.

Table 2.

Recommendations for researchers planning an f/sMRI-EMA study

Research aspect	Recommendations
Design	◾ Order of methods should reflect research question direction; this is particularly important in cases when neural activation/structure is the proposed mediator ◾ With task-based fMRI, make sure that the same construct assessed in the fMRI is assessed using EMA ◾ When planning EMA, consider assessing several constructs so that discriminant validity of s/fMRI can be assessed ◾ Due to reliability issues with fMRI, increase focus on sMRI and adopt methods that enhance fMRI reliability
Transparency and openness	◾ Use power analysis for at least one of the methods (preferably for both), and aim for larger sample size compared to the norm in fMRI literature ◾ Pre-register design, hypotheses, inclusion and exclusion ◾ Keep a publicly-available list of all manuscripts that use the dataset generated by the project
Reporting	◾ Provide detailed account of the procedure, including reimbursement structure, number of days lapsing between use of each method, etc.
Analysis	◾ Use multivariate methods that enhance reliability of fMRI data ◾ Complement hypotheses-driven analysis with machine-learning, particularly unsupervised methods that are data-driven

3. New Directions for Data Analysis

Supervised methods.

Most researchers use parametric models, such as regression or classification, trained on labeled data to find a statistical relation between a set of independent variables and a dependent variable. These methods often rely on a-priori hypotheses regarding the statistical model (mostly linear; James et al., 2013). This category also includes non-linear methods such as logistic regression and artificial neural networks. The common thread of these methods, dubbed supervised methods, is the fact that they aim to learn a set of model parameters to characterize a relation between inputs and outputs given training samples on which to fit the model and data set aside for testing the performance of the model on unseen data.

For multimodal data, one modality can take the part of the input variable used to predict a feature taken from the other modality, which takes the part of the outcome variable. For example, to test the correlation between specific EMA features (e.g., affect variability) and activity in a specific brain region, EMA data is often used to predict the activation of that brain region. This type of analysis is straightforward and effective, and indeed, most of the analytic methods (cf., factor analysis) used in the papers included in this review fall under this definition. However, supervised methods require prior knowledge or assumptions regarding which features in one modality (EMA) are likely to be related to which features in the other modality (brain region).

An additional issue is that although linear models provide a straightforward formulation for the importance of each predictor, they may be too simplistic to capture the relationship between the modalities. If the researcher suspects that the relationship between f/sMRI data EMA data is non-linear, supervised non-linear methods should be used. For example, to use EMA features to train an artificial neural network that predicts the activity of a specific brain region. Notably, for non-linear methods such as artificial neural networks, the trained model is a “black box” and there is no way to infer the importance of each feature from the parameters of the model (Everaert, Benisty et al., 2022). This lack of interpretability may be particularly challenging given the large number of features in EMA and s/fMRI studies. To deal with the interpretability challenge, various methods for feature selection assess which input predictors are the most important (Park & Casella, 2008; Kohavi & John, 1997; Zou & Hastie, 2005; Yamada et al., 2020). Specifically, to test which brain regions are more relevant to stress reactivity (expressed by the EMA), one can take task-related fMRI data and use it to predict the stress-related EMA data. In this case, feature selection methods would provide a quantitative score indicating which predictors (i.e., brain regions) were most relevant for prediction.

Another merit of incorporating feature selection in the analysis is regularization; in many cases the number of features is large relative to the sample size, which may cause overfitting and hinder the ability of the trained model to generalize to unseen examples.

Unsupervised methods.

In some cases, such as pilot studies, researchers have no clear hypothesis regarding the statistical relations between EMA and s/fMRI data, preventing them from applying supervised methods. In these situations, unsupervised methods provide an effective tool to find intrinsic patterns in the data, with no external knowledge. For example, k-means analysis of s/fMRI activation patterns might indicate a structure of several clusters of participants, which can then be further explored by looking at EMA and demographic data to see if the clusters based on s/fMRI correspond to these other modalities. This type of data-driven analysis can help better understand fMRI activation patterns or s/fMRI characteristics and corresponding EMA data.

Another challenge related to both EMA and s/fMRI being high-dimensional data modalities is that it is challenging to extract and visualize data patterns across multiple dimensions. Some features or combinations of features may exhibit a dominant pattern across the population, while others are superfluous. Unsupervised dimensionality reduction methods find a low-dimensional representation capturing the major trends or latent patterns within and across each modality, without an external variable guiding it. These can then facilitate data exploration and data visualization for hypothesis generation, as well as training and application of subsequent pattern recognition and machine learning algorithms, such as K-means or Gaussian Mixture Models (GMM) or supervised techniques discussed above, for establishing a statistical relation between the detected patterns across participants between the two modalities. Applying these techniques in the reduced dimensions has better computational complexity (it is faster to run these algorithms in smaller dimensions) as well as improved performance due to the suppression of insignificant features. Common approaches for dimensionality reduction include linear methods such as Principal Component Analysis (PCA) or nonlinear kernel-based methods (Belkin & Niyogi, 2003; Coifman & Lafon, 2006) or data visualization such as t-distributed stochastic neighbor embedding (t-SNE, Van der Maaten et al., 2008). Namely, for fMRI analysis, dimensionality reduction methods can be used to quantify brain dynamics by projecting the fMRI time-series onto a low-dimensional space (Allen et al., 2014; Monti et al., 2017; Shine et al., 2016; Shine et al., 2019, Gao et al. 2021). Alternatively, the participants in the study can be mapped to a low-dimensional space to identify communities or re-organize the population based on psychological conditions.

Multimodal methods.

So far, we have described methods for linking EMA and s/fMRI data by modeling one modality using data from the other. A more advanced approach for deciphering the relationship between these two modalities is multiview (also called multimodal) analyses such as Canonical Component Analysis (CCA; Hardoon et al., 2004). The multiview paradigm requires a joint dimension, such as participants, or time for a longitudinal study, that is assessed through different “views” (modalities). Each view yields different features, and multiview analysis reveals information—a latent factor—that is shared by the two (or more) views. Thus, instead of predicting one modality from another, or reducing dimensionality for each view separately, multiview analysis finds a shared representation for the two modalities simultaneously.

CCA, for example, calculates a linear transformation of the features from each modality to a low-dimensional joint space that reveals the shared organization of the data (participants or time) according to the two modalities. For instance, if one assumes the participants can be organized into groups, CCA will reveal such an organization that both modalities agree with, and modality-specific groups will be suppressed. Similarly, CCA can be used to uncover shared temporal trends in a longitudinal study that are evident according to both modalities. Since CCA is linear, it also provides an interpretable transformation indicating which features of each modality are most correlated with the joint space, and thus with each other. In some cases, linear approaches fail to capture the intrinsic relations between the two modalities. In these cases, using non-linear variates, such as kernel CCA, may be better at capturing structural properties of the data (Schölkopf et al., 1998; Michaeli et al., 2016; Lindenbaum et al., 2020). However, such methods would not provide the same interpretability capabilities linear approaches provide.

Better together: Supplementing EMA for MRI studies can advance social neuroscience

This review shows that s/fMRI and EMA are an excellent combination as they address each other’s limitations (Berkman et al., 2013; Wilson et al., 2014). S/FMRI (brain structure or the BOLD signal) are objective measures, thus balancing the fact that EMA is reliant on subjective self-report. Functional MRI (like any task-based method) typically relies on standardized stimuli which increases internal reliability – thus complementing the unstandardized, idiosyncratic occurrences captured by EMA. Supplementing EMA to s/fMRI can help examine brain-behavior links and translate s/fMRI findings to everyday life and particularly the social context. We hope that this review will encourage researchers to combine these methods and will give them a roadmap to designing such studies.

4.5. Summary

The current review describes common research practices, advantages, and limitations of studies that combine s/fMRI and EMA. This combination increases the ecological and predictive validities of MRI studies, along with potential clinical utility. Although the combination of these methods is demanding, both for research teams and study participants, this review demonstrates the feasibility of conducting such studies on diverse developmental and clinical populations. This combination will greatly advance the field of social neuroscience by helping us situate neural processes in the everyday social context.

Figure 3.

Four Study Designs Used to Combine EMA with MRI/fMRI.