A functional neuroimaging dataset acquired during naturalistic movie watching and narrated recall of a series of short cinematic films

Abstract

Whole-brain functional magnetic resonance imaging (fMRI) data from twenty healthy human participants were collected during naturalistic movie watching and free spoken recall tasks. Participants watched ten short (approximately 2 – 8 min) audiovisual movies and then verbally described what they remembered about the movies in their own words. Participants’ verbal responses were audio recorded using an MR-compatible microphone. The audio recordings were transcribed and timestamped by independent coders. The neural and behavioral data were organized in the Brain Imaging Data Structure (BIDS) format and made publicly available via OpenNeuro.org. The dataset can be used to explore the neural bases of naturalistic memory and other cognitive functions including but not limited to visual/auditory perception, language comprehension, and speech generation.

Specifications Table

Subject	Neuroscience: Cognitive
Specific subject area	Naturalistic memory encoding and recall
Type of data	Brain images in the NIFTI format and text files containing transcribed verbal responses
How the data were acquired	Brain images were acquired using a 3T Siemens Prisma MRI scanner with a 64-channel head/neck coil. Verbal responses were audio recorded using an MR-compatible recording system (FOMRI II; Optoacoustics Ltd.) and then manually transcribed.
Data format	Raw
Description of data collection	Twenty human participants participated in an fMRI experiment. Participants watched ten short movies and then verbally recounted what they remembered about the movies in their own words.
Data source location	Institution: Princeton Neuroscience Institute City: Princeton Country: United States
Data accessibility	Repository name: OpenNeuro Data identification number: 10.18112/openneuro.ds004042.v1.0.0 Direct URL to data: https://openneuro.org/datasets/ds004042/versions/1.0.0
Related research article	(1)H. Lee, J. Chen, Predicting memory from the network structure of naturalistic events, Nat. Commun. 13 (2022) 4235. (2)H. Lee, J. Chen, A generalized cortical activity pattern at internally generated mental context boundaries during unguided narrative recall, eLife. 11 (2022) e73693.

Value of the Data

• •Whole-brain functional neuroimaging data collected during continuous naturalistic tasks, especially unguided spoken recall, are relatively rare, and provide opportunities for exploring the neural bases of real-world cognition.
• •Although the primary aim of the data collection was to study memory, brain responses during naturalistic movie watching and free spoken recall are also relevant to other fields of psychology and neuroscience including but not limited to visual/auditory perception, language perception, and speech generation.
• •The brain images can be reanalyzed focusing on any brain areas and functional characteristics not explored in the dataset's related published articles [1,2]; for example, functional connectivity between various cortical and subcortical structures.
• •The behavioral data (i.e., transcripts of spoken recall) can be reanalyzed on their own; for example, focusing on stimulus-related factors that affect various behavioral characteristics of narrated memory recall.

1. Data Description

The current dataset consists of raw brain MRI data and transcribed verbal responses of twenty human participants collected during naturalistic movie-watching and free spoken recall tasks (results published in [1,2]). The dataset is organized in the BIDS format [3] and publicly available via OpenNeuro [4]. Each participant's brain data consists of a high-resolution anatomical image, fieldmap magnitude and phase difference images, and functional images in the Neuroimaging Informatics Technology Initiative (NIfTI) format. Participants’ verbal responses collected during free spoken recall are provided as text files in the tab-separated values (TSV) format. Each row of the TSV file includes the manual transcription of the participant's utterance, the onset time of the utterance relative to the onset of the scanning run in seconds, and the duration of the utterance in seconds. The onset times and durations of movie stimuli presented during the experiment are also provided as TSV files. MRI scanning parameters required for data analysis are provided as JavaScript Object Notation (JSON) files. Participants’ age, biological sex, and handedness information are provided in a TSV file. Further descriptions of the individual files and folders in the dataset are provided in Table 1.

Table 1.

List of individual files and folders in the dataset.

File or folder name	Description
dataset_description.json	General information about the dataset
participants.tsv	Participants’ age, sex, and handedness
phasediff.json	Parameters used to obtain fieldmap images
task-movie_bold.json	Parameters used to obtain the movie watching phase functional images
task_movie_run-01_events.tsv	Onsets and durations of the movies presented in movie watching run 1
task_movie_run-02_events.tsv	Onsets and durations of the movies presented in movie watching run 2
task-recall_bold.json	Parameters used to obtain the recall phase functional images
sub-xx	Folder containing data from a single subject
anat	Folder containing an anatomical image
sub-xx_T1w.nii.gz	High-resolution anatomical image
fmap	Folder containing fieldmap images
sub-xx_magnitude1.nii.gz	Fieldmap magnitude image 1
sub-xx_magnitude2.nii.gz	Fieldmap magnitude image 2
sub-xx_phasediff.nii.gz	Phase difference image
func	Folder containing functional images
sub-xx_task-movie_run-01_bold.nii.gz	Functional images from movie watching run 1
sub-xx_task-movie_run-02_bold.nii.gz	Functional images from movie watching run 2
sub-xx_task-recall_run-01_bold.nii.gz	Functional images from recall run 1
sub-xx_task-recall_run-02_bold.nii.gz	Functional images from recall run 2 (if exists)
sub-xx_task-recall_run-01_events.tsv	Recall transcript from recall run 1
sub-xx_task-recall_run-02_events.tsv	Recall transcript from recall run 2 (if exists)

2. Experimental Design, Materials and Methods

2.1. Participants

Twenty right-handed native English speakers from the Princeton community participated for monetary compensation (20 USD per hour). Eleven participants were female and nine participants were male. Participants were aged between 20 and 33 (M = 26.25 years, SD = 3.92 years). All participants were in good health and reported no history of neurological disorders. All participants reported normal hearing and normal or corrected-to-normal vision. One additional participant's data was collected but not included in the dataset as the participant had difficulty hearing audio during movie-watching.

2.2. Audiovisual Movie Stimuli

Participants watched 10 short (ranged 2.15–7.75 min) audiovisual movies that varied in content and narrative structure. See Table 2 for detailed information on the movie stimuli. Half of the movies contained speech (English language). Subtitles were not provided in any of the movies. Three of the movies (A Single Life, Keith Reynolds Can't Make It Tonight, and An Object at Rest) were animations, and the others were live-action movies. Two of the movies were clips edited from longer full movies; we used the first 5.77 min of Catch Me If You Can and the first 4.3 min of Arrival. The first five movies listed in Table 2 were presented in the first movie-watching scanning run, and the remaining five were presented in the second run. Within each scanning run, the movie clips were presented consecutively except that we inserted a 6 s-long title scene at the beginning of each movie; the title of a movie (shown in white letters) gradually appeared and disappeared on the black screen. We also played an additional 39 s audiovisual cartoon (Let's All Go to the Lobby) at the beginning of each movie-watching scanning run (immediately before the first movie of the run).

Table 2.

List of audiovisual movie stimuli (adapted from Supplementary Table 1 in [1]).

Presentation order	Title (*contains speech)	Duration (min:sec)	Release year
1	Catch Me If You Can*	5:46	2002
2	A Single Life	2:12	2014
3	High Maintenance*	7:45	2006
4	How They Get There	2:09	1997
5	Keith Reynolds Cannot Make It Tonight*	5:48	2008
6	An Object at Rest	5:25	2015
7	Arrival (First episode of the TV series “The Prisoner”)*	4:20	1967
8	The Black Hole	2:22	2008
9	Post-It Love	2:41	2009
10	Stray Dogs*	6:54	2015

2.3. Experimental Design and Procedures

The current fMRI dataset was acquired during an experiment consisting of two phases: movie-watching and free spoken recall. The movie-watching phase consisted of two consecutive scanning runs. In each run, participants watched five audiovisual movies in a row. The presentation order of movies was fixed across participants. The movie stimuli were visually presented on a rear-projection screen in the magnet bore using an LCD projector, and were viewed with an angled mirror. Audio tracks were delivered via in-ear headphones. Participants were instructed to pay attention to the movies as they would normally do in real life. Participants were also informed that there would be a memory recall task following movie-watching. No verbal or manual responses were required during the movie-watching scans. The movie-watching phase lasted approximately 50 min in total (first run video duration = 24.9 min, second run video duration = 22.9 min).

The free spoken recall phase immediately followed the movie-watching phase, with no intervening task or break between the phases. Participants were instructed to verbally describe what they remembered about the movies in their own words, in as much detail as possible, regardless of the presentation order of the movies. Participants were asked to speak for at least ten minutes. Participants indicated that they were finished by saying “I am done”. Participants’ verbal responses were recorded using a customized MR-compatible recording system (FOMRI II; Optoacoustics Ltd.). Throughout the free spoken recall phase, a white fixation dot on black background was shown on the rear-projection screen; however, participants were not required to fixate on the dot. As there was no constraint on how long the recall should be, the duration of speech varied across participants (ranged 8.8 – 55.4 min, M = 30.2 min, SD = 12.7 min). In four participants (sub-08, sub-10, sub-15, and sub-16), the free spoken recall phase was divided into two scanning runs as their speech exceeded the 35 min scanner limit. One other participant (sub-18) also had two scanning runs as she needed to take a break during recall. All other participants had a single free spoken recall scanning run.

2.4. MRI Data Acquisition

MR images were acquired at Princeton Neuroscience Institute using a 3T Siemens Prisma scanner with a 64-channel head/neck coil. To collect functional images, we used a T2*-weighted multiband accelerated echo-planar imaging sequence (TR = 1.5 s; TE = 39 ms; flip angle = 50°; acceleration factor = 4; 60 oblique axial slices; grid size 96 × 96; voxel size 2 × 2 × 2 mm³). We also collected two fieldmap magnitude images (TR = 0.52 s; TE = 4.92 or 7.38 ms; flip angle = 55°; 60 oblique axial slices; grid size 64 × 64; voxel size 3 × 3 × 2 mm³) and a phase difference image (TR = 0.52 s; TE = 7.38 ms; flip angle = 55°; 60 oblique axial slices; grid size 64 × 64; voxel size 3 × 3 × 2 mm³) to correct for magnetic field inhomogeneity. We also collected whole-brain high-resolution anatomical images using a T1-weighted MPRAGE pulse sequence. We reused existing anatomical images for participants who had already been scanned for other projects at Princeton Neuroscience Institute; thus, scanning parameters for the anatomical images varied across subjects (see Table 3).

Table 3.

Parameters used for high-resolution anatomical image acquisition.

Participant ID	Number of sagittal slices	Grid size	Voxel size (mm3)
sub-01	176	255 × 200	1 × 1 × 1
sub-02	192	255 × 186	.9 × 0.86 × 0.86
sub-03	176	255 × 189	1 × 1 × 1
sub-04	176	255 × 198	1 × 1 × 1
sub-05	176	255 × 200	1 × 1 × 1
sub-06	176	255 × 200	1 × 1 × 1
sub-07	192	241 × 171	.9 × 0.86 × 0.86
sub-08	192	256 × 256	.9 × 0.86 × 0.86
sub-09	176	255 × 200	1 × 1 × 1
sub-10	176	256 × 256	1 × 1 × 1
sub-11	176	255 × 200	1 × 1 × 1
sub-12	192	255 × 181	.9 × 0.86 × 0.86
sub-13	176	255 × 200	1 × 1 × 1
sub-14	176	255 × 200	1 × 1 × 1
sub-15	176	255 × 200	1 × 1 × 1
sub-16	176	256 × 256	1 × 1 × 1
sub-17	176	255 × 200	1 × 1 × 1
sub-18	192	256 × 256	.9 × 0.86 × 0.86
sub-19	176	256 × 256	1 × 1 × 1
sub-20	192	256 × 256	.9 × 0.86 × 0.86

2.5. Recall Transcripts

Independent coders manually transcribed the audio recording of each subject's free spoken recall and segmented each transcript into discrete utterances based on pauses in the speech and changes in the topic. Independent coders also identified the beginning and end timestamps of each utterance to record the onsets and durations of utterances.

Ethics Statements

The current research was conducted in accordance with The Code of Ethics of the World Medical Association (Declaration of Helsinki). We obtained informed consent from all participants and performed the research following procedures approved by the Princeton University Institutional Review Board (protocol #5516).

CRediT authorship contribution statement

Hongmi Lee: Data curation, Software, Writing – original draft, Writing – review & editing. Janice Chen: Conceptualization, Methodology, Investigation, Supervision, Funding acquisition, Writing – review & editing. Uri Hasson: Conceptualization, Methodology, Supervision, Funding acquisition, Writing – review & editing.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Data Availability

• Film Festival (Original data) (OpenNeuro).