Predicting whole-brain neural dynamics from prefrontal cortex functional near-infrared spectroscopy signal during movie-watching

Abstract

Functional near-infrared spectroscopy (fNIRS) offers a portable, cost-effective alternative to functional magnetic resonance imaging (fMRI) for noninvasively measuring neural activity. However, fNIRS measurements are limited to cortical regions near the scalp, missing important medial and deeper brain areas. We introduce a predictive model that maps prefrontal fNIRS signals to whole-brain fMRI activity during movie-watching. By aligning neural responses to a common audiovisual stimulus, our approach leverages shared dynamics across imaging modalities to map fNIRS signals to broader neural activity patterns. We scanned participants with fNIRS and utilized a publicly available fMRI dataset of participants watching the same TV episode. The model was trained on the first half of the episode and tested on a held-out participant watching the second half to assess cross-individual and cross-stimulus generalizability. The model significantly predicted fMRI time courses in 66 out of 122 brain regions, including areas otherwise inaccessible to fNIRS. It also replicated intersubject functional connectivity patterns and retained semantic information about the movie content. The model generalized to an independent dataset from a different TV series, suggesting it captures robust cross-modal mappings across stimuli. Our publicly available models enable researchers to infer broader neural dynamics from localized fNIRS data during naturalistic tasks.

Introduction

Functional magnetic resonance imaging (fMRI) offers the spatial resolution to study activity of specific brain regions while measuring whole-brain activity (Bandettini 2020). However, fMRI studies are constrained to tasks that can be conducted in a confined environment while participants lie still in the magnetic resonance imaging (MRI) scanner. fMRI is thus not conducive for studying many naturalistic behaviours, such as learning in a classroom environment, freely moving in a physical space, or brainstorming in groups (Shamay-Tsoory and Mendelsohn 2019). The noise generated by the scanner and the restriction on movement is also challenging for children, the elderly, and individuals with sensory sensitivities (Lueken et al. 2011, Greene et al. 2016, Hausman et al. 2022). Furthermore, the high costs associated with fMRI scanning limit the feasibility of large-scale studies that provide sufficient statistical power to study individual differences (Grady et al. 2021).

Functional Near-Infrared Spectroscopy (fNIRS) is emerging as a promising alternative (Burns et al. 2019, Pinti et al. 2020). Similar to fMRI, fNIRS measures neural activity indirectly through the hemodynamic response (Huppert et al. 2006, Yücel et al. 2017), and prior studies have found high temporal and spatial correlation in the activation profile measured by fNIRS and fMRI during the same task (Cui et al. 2011, Sato et al. 2013, Noah et al. 2015, Liu et al. 2017). Relative to fMRI, fNIRS is portable, less expensive, and more tolerant of movement. These advantages have made it suitable for studying behaviour in naturalistic settings, including face-to-face social interactions (Suda et al. 2010, Hirsch et al. 2021), affective touch (Bennett et al. 2014), actors in a theatre performance (Hamilton et al. 2018), and physical activity (Byun et al. 2014, Ono et al. 2015). fNIRS, however, is limited to measuring activity in cortical regions near the scalp (Yücel et al. 2017).

How can researchers use fNIRS to study regions beyond the cortical surface? One potential strategy is to ‘infer’ brain activity using predictive models that map fNIRS activity onto fMRI activity by leveraging functional correlations between brain regions (Liu et al. 2015, Balters et al. 2023). For example, using simultaneous fNIRS-fMRI, Liu et al. (2015) demonstrated that fMRI activity in deep brain regions could be predicted from fNIRS activity measured at the scalp. Despite the success of these models, it remains unclear how well they would perform when the fNIRS and fMRI data are derived from different groups of subjects. This is a crucial consideration, as it is not always feasible to collect simultaneous fNIRS and fMRI data from the same participant. Furthermore, previous models have been trained on data collected while participants performed cognitive tasks consisting of repeated trials from a small number of experimental conditions (Liu et al. 2015, Balters et al. 2023), and may not generalize to the naturalistic tasks for which fNIRS holds the greatest promise. Indeed, naturalistic behaviours, such as having a spontaneous conversation, tap into multiple cognitive processes simultaneously and elicit a wider range of neural responses (Matusz et al. 2019). This simultaneous activation of diverse and overlapping brain networks may result in more complex and distributed neural patterns, making it unclear whether predictive models trained on traditional tasks would generalize across different naturalistic behaviours.

The goal of this study is to develop a predictive model that maps fNIRS signals to continuous whole-brain fMRI neural dynamics during movie-watching. Our study advances past work in two significant ways: first, we train a model that generalizes across individuals, addressing the challenge of applying predictive models to data from different participants; second, we train and test our predictive model on neural activity collected during a naturalistic task, movie-watching, which more closely mirrors the complexity and cognitive demands of everyday experiences. We first used fNIRS to measure participants’ neural activity as they watched a TV episode. Due to a limited number of available optodes, optodes were placed to optimize coverage of the prefrontal cortex (PFC), a decision that we had also made in our earlier work (Burns et al. 2019, Lyu et al. 2024).

We targeted the PFC due to extensive prior work implicating the region in processing complex dynamic audiovisual narratives (Baldassano et al. 2018, Rowland et al. 2018) and higher-order cognition more broadly (Friedman and Robbins 2022). Furthermore, the PFC is a heterogeneous structure composed of multiple areas that are part of distinct large-scale functional brain networks (Menon and D’Esposito 2022). For example, the dorsomedial PFC and dorsolateral PFC are functionally coupled with the default mode network and frontoparietal network, respectively. Our model can thus leverage the diverse connections of the PFC to infer neural dynamics across the brain. From a practical perspective, the fNIRS signal in the PFC is often cleaner due to easier access and thinner hair coverage relative to other parts of the scalp.

Using a publicly available fMRI dataset where participants watched the same TV episode, we adapted a principal component regression approach to train a model that predicts whole-brain fMRI data from fNIRS data. The model was trained on the first half of the episode and tested on the second half of the episode in a leave-one-participant-out (LOO) approach. In other words, the model was tested on data from a participant and stimuli that it was not trained on, which allowed us to assess cross-individual and cross-stimulus generalizability. To evaluate the information preserved in the fNIRS-to-fMRI mapping, we used textual embeddings of the TV episode to build neural encoding models (Huth et al. 2016, Goldstein et al. 2022, Caucheteux et al. 2023) that mapped semantic information onto fMRI activity. The successful generalization of the encoding models across real and predicted fMRI activity would suggest that fNIRS-fMRI mapping retained information about the episode’s semantic content. Altogether, our study introduces a novel approach that combines the flexibility of fNIRS with the spatial coverage of fMRI, offering new possibilities for studying brain dynamics in naturalistic contexts.

Materials and methods

Participants

Thirty individuals were recruited for the fNIRS study from the University of Chicago community and provided informed consent prior to the study. Experimental procedures were approved by the University of Chicago Institutional Review Board. One participant was excluded because of poor data quality (see fNIRS data acquisition), yielding an effective sample size of 29 participants (15 females, 13 males, 1 nonbinary; age: M = 19.69, SD = 0.93).

Stimuli

Participants were scanned using fNIRS as they watched a 48 min 6 s segment from an episode of the BBC television series ‘Sherlock’. The stimulus was chosen due to the availability of a publicly available fMRI dataset of participants watching the same segment (see fMRI dataset). The segment was divided into two runs (Run 1: 23 min; Run 2: 25 min 6 s), and a short cartoon was padded to the beginning of each run to mitigate the confounding effects of stimulus-onset.

fNIRS data acquisition

fNIRS data were collected using a NIRSport2 fNIRS device (NIRx Medical Technologies) with a sampling rate of 10.1725 Hz at wavelengths of 760 and 850 nm. The fNIRS device layout consisted of 20 channels composed of 8 source optodes and 7 detector optodes using the unambiguously illustrated (UI) 10/10 external position system (Jurcak et al. 2007). Raw fNIRS data were preprocessed in MATLAB using custom scripts that utilized the Homer2 package (Huppert et al. 2009). Preprocessing steps are detailed in the Supplementary Methods and included adjusting for hemodynamic lag, removal of noisy channels, bandpass filtering (0.005–0.5 Hz) to remove physiological noise, and removal of motion artefacts using targeted principal component analysis (PCA) (Yücel et al. 2014). Optical density signals were converted to changes in oxygenated (HbO), deoxygenated (HbR), and total (HbT) haemoglobin concentrations following the modified Beer–Lambert law (ppf value = 6). Data were z-scored separately for each run and each participant. We used the HbO time courses for analyses because of the stronger signal amplitude and higher correlation to fMRI blood-oxygen-level-dependent (BOLD) signals (Strangman et al. 2002, Tong and Frederick 2010, Duan et al. 2012). HbO time courses were resampled to match the sampling frequency of the fMRI dataset (TR = 1.5 s). Participants with ≥ 3 channels with missing values (i.e. 15% of channels) in either run were excluded from subsequent analysis (n = 1).

fMRI dataset

We utilized the publicly available ‘Sherlock’ dataset, in which 17 participants viewed the same stimuli in two runs while undergoing fMRI (Chen et al. 2017). The preprocessed functional images were downloaded from the Princeton University DataSpace repository (see the Supplementary material).

HbO-BOLD correlation analyses

For each fNIRS channel, we computed the average HbO activity across participants to obtain the group-mean HbO time course. To identify corresponding locations in the fMRI data, we estimated the Montreal Neurological Institute (MNI) coordinates of each channel using an anchor-based probabilistic conversion atlas (Tsuzuki et al. 2012). We then created a spherical region of interest (ROI) with a 5-mm radius centred on the MNI coordinates of each channel. The BOLD time courses were averaged within each ROI and across all participants in the fMRI sample to obtain the group-mean BOLD time course. We computed the Pearson correlation between the group-mean HbO and BOLD time courses at matching locations. Statistical significance was assessed using a nonparametric permutation test where the empirical correlation coefficient was compared against a null distribution generated by repeating the analysis with phase-randomized group-mean BOLD time courses, as implemented in the ‘nltools’ package. A right-tailed P-value was calculated for each channel using the formula:

(1)

Across the 20 channels, P-values were corrected for multiple comparisons by controlling for the false discovery rate (FDR) at q < 0.05 (Benjamini and Hochberg 1995).

As control analyses, we correlated each fNIRS channel with the group-mean BOLD time course from the primary auditory cortex, the primary visual cortex, and the average BOLD signal across all ROIs (global grey matter signal) (see the Supplementary material).

fNIRS-fMRI predictive modelling

We parcellated the fMRI data into 114 cortical ROIs following the Yeo atlas (Yeo et al. 2011). Each cortical ROI was labelled with one of the seven functional networks (visual—VIS; somatomotor—SM; dorsal attention—DAN; salience/ventral attention—VAN; limbic—LIMB; control—CONT; default mode—DMN) according to Yeo et al. (2011). Subcortical regions (SUBC) were parcellated into eight ROIs following the subcortical nuclei masks of the Brainnetome atlas (Fan et al. 2016), and included the bilateral amygdala, basal ganglia, hippocampus, and thalamus. For each participant, we averaged the BOLD time courses of all voxels within each ROI, yielding 122 time courses.

We adapted a principal component regression (henceforth aPCR; Fig. 1) approach to predict the whole-brain fMRI BOLD time courses from fNIRS HbO time courses. During model training, a PCA was performed separately on the fNIRS and fMRI time courses in the training set to extract fNIRS and fMRI PCs capturing 90% of variance in the original data (average number of PCs across LOO iterations: fNIRS—12.8, fMRI—49). Because the PCA does not allow missing values, we imputed missing values with the group mean (see Supplementary Methods). We then fit a linear regression model predicting fMRI PCs from fNIRS PCs:

Figure 1.

Model training and evaluation. The aPCR model was trained and tested following an across-run, LOO approach. Training involved applying separate PCAs to extract PCs capturing 90% of the variance in the fNIRS and fMRI data. Linear regression was then used to predict fMRI PCs from fNIRS PCs. Training data consisted of Run 1 data from all but one fNIRS participant and all fMRI participants. Testing data consisted of Run 2 data from the held-out fNIRS participant. Model performance was evaluated by comparing the average predicted fMRI time courses with the group mean fMRI time courses using Pearson’s correlation.

Train-Test split

Model training and evaluation were performed using an LOO cross-validation approach where the model was trained on data from Run 1 and tested on data from Run 2 of a held-out fNIRS participant. Leaving out an fNIRS participant allowed us to evaluate the model’s ability to generalize to unseen fNIRS data. For each LOO iteration, each of the 28 remaining fNIRS participants was paired with each of the 17 fMRI participants, resulting in 28 × 17 = 476 unique pairings. Data from each pairing were then concatenated along the time axis, such that each fNIRS participant’s data was used to predict each fMRI participant’s data in the aPCR model. This approach exposed the model to all possible data pairings, improving its ability to generalize to new participants.

Model evaluation

We tested the trained aPCR models on the Run 2 fNIRS time course of the held-out participant. In other words, the model was tested on data from a participant and a part of the movie that was not in the training data, allowing us to test whether the model generalized to new participants and unseen stimuli. For each LOO iteration, the model’s output consisted of the predicted time courses for the fMRI PCs, which were then back-projected to the original ROI space by inverting the PCA transformation, resulting in predicted BOLD time courses for 122 ROIs. For each ROI, we computed the average predicted fMRI time course across all LOO iterations and then correlated this average predicted time course with the average fMRI time course from the real data as our measure of predictive accuracy. Statistical significance was assessed by comparing the true correlation values against a null distribution generated by repeating the correlation 1000 times with phase-randomized average fMRI time courses. P-values were computed following Equation (1) and corrected for multiple comparisons across the 122 ROIs by controlling for FDR at q < 0.05.

To contextualize model performance, we also computed the proportion of the noise ceiling (PNC) captured by the model for each ROI. Cronbach’s α was calculated from the Run 2 fMRI time courses across participants as an estimate of the maximum expected model performance given intersubject variability and measurement noise (Jiahui et al. 2020). As Cronbach’s α reflects shared variance (r²), we took the square root to obtain a ceiling in correlation (r) space, and divided the model’s Pearson r by to compute the PNC.

To assess model accuracy by functional network, we grouped the 122 ROIs into the seven cortical networks defined by the Yeo atlas and one subcortical network that included all subcortical ROIs. For each network, we computed the proportion of significant ROIs and the median model performance. We computed the median correlation value as it does not require Fisher transformation to normalize the distribution of correlation coefficients (Chen et al. 2016), and is a measure of model performance that relies on fewer assumptions.

Intersubject functional connectivity analyses

To examine whether model predictions recapitulated whole-brain patterns of functional correlations, we compared intersubject functional connectivity (ISFC) patterns between the observed and predicted fMRI time courses for Run 2. We first calculated the Pearson correlation between the time course of one ROI for that participant and the time course of another ROI across all other participants. This process was repeated pairwise for every pair of ROIs, and then repeated across all participants. We then computed the median correlation value for each pair of ROIs. This procedure was performed separately for the observed and predicted fMRI time courses, resulting in two average ISFC matrices. The diagonal of an ISFC matrix reflects the ISC of each ROI and was excluded from all ISFC analyses. As an ISFC matrix is not symmetrical along the diagonal, we followed Simony et al. (2016) and averaged the corresponding cells in the upper and lower triangles to obtain a single value representing the ISFC between two regions. We computed the Pearson correlation between predicted and observed ISFC, with statistical significance assessed using a nonparametric Mantel test (Mantel 1967, Kriegeskorte et al. 2008).

Semantic encoding models

We obtained detailed annotations of the movie content provided by Chen et al. (2017). These annotations divided the movie into 50 scenes spanning 1000 segments, with annotations describing what happened during each segment (e.g. Sherlock picks up a small pink suitcase from a chair and brings it back into the living room). The annotation of each segment was converted into a 512-dimensional vector embedding using the Universal Sentence Encoder (Cer et al. 2018). To align these segment embeddings with the fMRI data, we resampled the embeddings to match the TR of the fMRI scans. The segment embedding of TR t is denoted as SEG_t.

To capture the preceding context of a sentence, we constructed a context embedding that models the accumulation of semantic information over time. This context embedding, CTX, was initialized at zero at the beginning of each scene, reflecting the reset of context at event boundaries (Zacks et al. 2007, Pu et al. 2022). At each TR t, CTX_t was updated to be the average of CTX_t − 1 and SEG_t − 1. CTX_t was then concatenated with SEG_t, resulting in a 1024-dimensional vector that reflects the semantic content of both the previous context and the current input. Our goal was to fit encoding models that map semantic information to observed and predicted fMRI data (Naselaris et al. 2011, Deniz et al. 2019, Caucheteux et al. 2023). To prevent overfitting, we reduced the 1024-dimensional vector to 32 dimensions using PCA (Tikochinski et al. 2023, 2025). We refer to the resulting 32-dimensional vector as the TR’s semantic embedding, or SEM_t.

Consistent with our leave-one-run-out testing approach, PCA was fit to semantic embeddings from Run 1. Analyses were limited to the 66 out of 122 ROIs where the fNIRS-fMRI model with above-chance prediction performance. Following previous work by Caucheteux and colleagues (2023), we fit a ridge regression model (α = 100) to predict the observed average ROI’s time course from the 32-dimensional SEM vectors from the observed Run 1 BOLD time courses. We then tested the model on the observed and predicted BOLD time course from Run 2. Statistical significance was assessed by retraining the encoding model on phase-randomized observed Run 1 BOLD time courses and applying the retrained model to the test data. This procedure was repeated 1000 times to generate a null distribution, with P-values computed following Equation (1) and controlling for FDR at q < 0.05.

Results

Correlation between fNIRS and fMRI time courses during movie-watching

We used fNIRS to scan 29 participants as they watched a 48-min segment of a television episode. We also utilized a publicly available fMRI dataset where participants viewed the same segment. In both datasets, the segment was divided into two runs. To examine the extent to which fNIRS activity matched fMRI activity, we correlated the group-mean HbO time courses of the 20 fNIRS channels with group-mean fMRI BOLD time courses at corresponding locations. Of the 20 ROIs, 18 showed significant correlation between the group-mean BOLD time course and the matching group-mean HbO time course (median r = 0.204, FDR q < 0.05; Fig. 2; Supplementary Table S1). These results indicate that the hemodynamic responses captured by fNIRS mirrored the neural dynamics measured using fMRI as participants watched the same movie.

Figure 2.

fNIRS and fMRI time courses at matching locations were correlated during movie-watching. (a) Location of 20 fNIRS channels. (b) Brain maps show the Pearson r between the 20 fNIRS channels and corresponding fMRI ROIs in the PFC, thresholded at FDR q < 0.05. Statistical significance was computed using a phase-randomized permutation test. Of the 20 fNIRS channels, 18 exhibited significant correlation with the corresponding fMRI ROI. **q < 0.01; *q < 0.05.

As control analyses, we also computed the correlation between each group-mean HbO time course with group-mean BOLD time course at the primary auditory cortex, primary visual cortex, and the global grey matter signal derived by averaging BOLD activity of all voxels across all ROIs. None of the 20 channels were significantly correlated with the three control time courses (FDR q > 0.05; median r: primary auditory cortex = 0.012, primary visual cortex = 0.024, global grey matter signal = 0.028; see Supplementary Table S2 for correlation values of each ROI), indicating spatial specificity in the correlation between fNIRS and fMRI time courses.

Predicting whole-brain fMRI signal from prefrontal fNIRS

We trained our aPCR model on the 20 prefrontal fNIRS channels to predict whole-brain fMRI time courses (see ‘Methods’). The model significantly predicted Run 2 fMRI time courses in 66 out of the 122 ROIs (q < 0.05; Fig. 3a; see Supplementary Fig. S1 for PNC map). These regions included prefrontal areas covered by the fNIRS channels, including the dorsolateral PFC (left: r = 0.30, PNC = 0.33; right: r = 0.29, PNC = 0.33), dorsomedial PFC (dmPFC; left: r = 0.45, PNC = 0.49; right: r = 0.44, PNC = 0.49), ventrolateral PFC (left: r = 0.38, PNC = 0.45; right: r = 0.36, PNC = 0.43), and ventromedial PFC (left: r = 0.22, PNC = 0.29; right: r = 0.20, PNC = 0.27), as well as in temporal and parietal regions not covered by the fNIRS channels, including the precuneus/posterior cingulate (left: r = 0.29, PNC = 0.32; right: r = 0.26, PNC = 0.28), temporal parietal junction (TPJ; left: r = 0.38, PNC = 0.40; right: r = 0.38, PNC = 0.40), intraparietal lobules (left: r = 0.39, PNC = 0.44; right: r = 0.36, PNC = 0.41), intraparietal sulcus (left: r = 0.30, PNC = 0.32; right: r = 0.30, PNC = 0.33), lateral temporal cortex (left: r = 0.50, PNC = 0.52; right: r = 0.44, PNC = 0.46), and the temporal poles (left: r = 0.21, PNC = 0.29; right: r = 0.26, PNC = 0.34). Among SUBC, model performance was significantly above chance in the basal ganglia (left: r = 0.15, PNC = 0.19, right: r = 0.18, PNC = 0.24). Model performance was highest in the default mode network (DMN; median r = 0.30, PNC = 0.35; percentage significant ROIs = 84.6%) and control network (CONT; median r = 0.296, PNC = 0.327; percentage significant ROIs = 80.8%; Fig. 3b). In contrast, model performance was lowest in the somatomotor network (SM; median r = 0.027, PNC = 0.036; percentage significant ROIs = 0%).

Figure 3.

Model performance of the fNIRS-fMRI predictive model. (a) Brain maps showing correlation between average observed BOLD fMRI time courses and average predicted BOLD fMRI time courses, thresholded at FDR q < 0.05. LPFC: lateral PFC, MPFC: medial PFC, PCC: posterior cingulate cortex, IPS: intraparietal sulcus, IPL: intraparietal lobules; TPJ: temporal parietal junction. (b) Predictive accuracy by functional network. Height of bar graphs indicates median r with circles indicating individual ROIs. Fractions denote the proportion of ROIs significant within a network. See the methods for labels of individual networks.

We further compared ISC in the predicted and observed fMRI data (Supplementary Fig. S2). Both datasets showed high ISC in default mode and control network regions, consistent with the model capturing shared, high-level responses to the movie. In contrast, ISC was substantially lower in primary visual and auditory cortices in the predicted data, likely reflecting the limited ability of prefrontal fNIRS to track low-level sensory features.

Predicted ISFC patterns from prefrontal fNIRS

To further explore the utility of prefrontal fNIRS data in capturing whole-brain dynamics, we examined whether the predicted fMRI time courses could recapitulate ISFC patterns. ISFC computes functional connectivity across participants and reflects the pattern of whole-brain functional correlations elicited by a shared stimulus (Simony et al. 2016). ISFC computed from the observed BOLD was significantly correlated with ISFC computed from the predicted BOLD (r = 0.42, P < .001; Fig. 4), indicating that the predicted BOLD successfully reproduced patterns of stimulus-driven functional correlations observed in the actual fMRI data.

Figure 4.

ISFC matrices computed from A. observed and B. predicted BOLD. Each cell denotes the median ISFC of a pair of ROIs. The correspondence between the two ISFC matrices is computed using a Mantel test. For our analysis, we averaged the top and bottom triangles of the ISFC matrices and excluded the diagonal such that each pair of ROIs is only considered once.

Assessing semantic information encoded in predicted BOLD

We fit a neural encoding model to predict the observed Run 1 BOLD time courses from semantic embeddings of the movie content (see ‘Methods’). When evaluated on observed BOLD data in Run 2, the encoding model had above-chance accuracy in 51 out of the 66 ROIs where the fNIRS-fMRI model exhibited significant predictive accuracy (FDR q < 0.05; Fig. 5a), indicating that the observed BOLD time courses encode information about the semantic content of the movie. Next, to assess whether the predicted BOLD time courses retained this semantic information, we tested the same encoding model on the Run 2 predicted BOLD time courses. Model accuracy was above chance in 12 out of the 66 ROIs (FDR q < 0.05; Fig. 5b), including the dorsomedial PFC and the network of brain regions that constitute the language network (i.e. dorsolateral PFC, lateral temporal cortex) (Fedorenko et al. 2024). These results suggest that the predicted BOLD time courses retained semantic information of the movie content encoded in the original BOLD signal during movie viewing.

Figure 5.

Prediction accuracy of the semantic encoding model. An encoding model was trained to predict Run 1 BOLD responses from semantic embeddings of the movie content. (a) Prediction accuracy of the model on Run 2 observed BOLD responses. (b) Prediction accuracy of the model on Run 2 predicted BOLD responses. Statistical significance was assessed by retraining the encoding model on phase-randomized training data. Brain maps are thresholded at FDR q < 0.05.

Cross-stimulus generalization to a different movie

To test whether the model generalizes to a different stimulus, we applied it to an independent fNIRS dataset in which participants watched an episode of a different TV series (‘Friday Night Lights’; see the Supplementary material). The model significantly predicted fMRI activity from a separate group of participants who watched the same episode. A model trained on Run 1 of ‘Sherlock’ significantly predicted fMRI activity in 66 ROIs (Fig. 6a), while a model trained on both runs significantly predicted fMRI activity in 84 ROIs (FDR q < 0.05; Fig. 6b).

Figure 6.

Model performance of the fNIRS-fMRI predictive model on an independent dataset (‘Friday Night Lights’). (a) Predictive accuracy when the model is trained on fNIRS and fMRI data from Run 1 of ‘Sherlock’. (b) Predictive accuracy when the model is trained on fNIRS and fMRI data from both runs of ‘Sherlock’. Predictive accuracy is measured as the correlation between predicted and observed fMRI time courses, thresholded at FDR q < 0.05. Brain plots show significant ROIs (r-values) projected onto cortical and subcortical surfaces. Bar plots show median r for each network with individual ROI r-values overlaid as grey dots. Fractions above each bar denote the number of significant ROIs over total ROIs within each network.

Discussion

We developed a predictive model that maps prefrontal fNIRS activity to whole-brain fMRI activity while participants viewed audio-visual movies. Our model significantly predicted fMRI time courses above chance in a substantial number of brain regions, including areas not typically accessible by fNIRS, such as the precuneus, temporal poles, and basal ganglia. Furthermore, an encoding model trained to predict observed fMRI data from semantic embeddings showed above-chance accuracy when applied to the predicted fMRI time courses generated by our model, suggesting that the model-generated fMRI time courses retained semantic information encoded in fMRI activity. Overall, our findings demonstrate the feasibility of using fNIRS to infer broader brain activity patterns, opening up new possibilities for investigating brain function during complex, real-world contexts.

Model accuracy was highest in the default mode network and control network, both of which are critically involved in higher-order cognitive functions (Raichle 2015, Gratton et al. 2018, Menon and D’Esposito 2022). This suggests that the current model may be well suited for investigating cognitive processes that rely on these networks, including social cognition, cognitive control, introspection, and memory retrieval. Conversely, model prediction accuracy was lowest in the somatomotor network. This discrepancy is likely due to the nature of the task we employed rather than a limitation of the approach. Watching a movie does not involve tactile stimulation or movement and is thus unlikely to engage somatomotor areas. Future work can potentially expand the coverage of our model by incorporating paradigms that engage the somatomotor network (e.g. playing an instrument).

It is important to note that our approach approximates neural activity based on functional correlations, and highly connected areas may serve related but distinct functions. For example, the dmPFC and TPJ are both part of the default mode network and are often coactivated during social interaction, but may support distinct processes (Saxe 2006, Van Overwalle 2009, Konovalov et al. 2021). It is unclear whether our model captures the level of granularity needed to differentiate between the distinct roles of such interconnected regions. Instead, we see the utility of our model as a tool for generating hypotheses about broader patterns of brain activity that can guide future investigations with fMRI. For instance, if fNIRS data collected during classroom instruction predicts joint activation in dmPFC and TPJ, it could suggest their involvement in social learning. This could then motivate a targeted fMRI study using curated clips of classroom interactions that vary in types of social ambiguity, to test whether these regions support distinct processes such as belief attribution or uncertainty monitoring.

Our approach shares similarities with functional alignment techniques, such as hyperalignment (Haxby et al. 2011, 2020) and the Shared Response Model (SRM; Chen et al. 2015). These methods aim to align neural responses across individuals to a common representational space via a shared stimulus. Our approach is similar in that we leverage the shared neural dynamics elicited by naturalistic stimuli to align neural responses. However, it is unique in that it aligns responses collected using different neuroimaging modalities, each with different spatial resolutions and coverage. Using this approach, we were able to predict the average fMRI time course of participants watching an unseen stimulus based on fNIRS data from a different participant. We further show that the model generalizes to an independent dataset of participants watching an episode from a different TV series, suggesting it captures robust cross-modal mappings that can be applied broadly without retraining. We note that all datasets consisted of college-aged adults. As the model relies on functional connectivity profiles, which are known to vary across the lifespan (Geerligs et al. 2015), future work will be needed to assess its generalizability to different age groups.

As our model learns a group-level mapping between fNIRS and fMRI, it only captures responses that are consistent across individuals. However, some brain regions tend to exhibit more idiosyncratic responses. For example, ventromedial prefrontal cortex (VMPFC) responses tend to be idiosyncratic when participants watch a TV episode (Chang et al. 2021). Similarly, we previously showed that VMPFC responses while processing social situations are modulated by individual differences in social beliefs (Lyu et al. 2024). In line with these findings, the accuracy of our fNIRS-fMRI model was lower in the VMPFC relative to other prefrontal regions. Measuring the same individual watching the same video using both fNIRS and fMRI may enable the development and validation of individual-specific models. These personalized models could then be used to predict fMRI activity from fNIRS data of the participant performing tasks that are not amenable to fMRI. If this approach proves effective, incorporating a movie stimulus into fMRI scans could be a worthwhile strategy to allow for the alignment of an individual’s brain across modalities. This direction aligns with the individualized neural tuning framework (Feilong et al. 2023), which learns subject-specific mappings from shared stimuli and could inspire future extensions of our model.

To examine the information content encoded in the predicted fMRI signal, we utilized text embedding models, which convert text into high-dimensional representations of semantic meaning. Consistent with prior work, we demonstrated that an encoding model trained to predict BOLD time courses from semantic embeddings of the movie content exhibited above-chance accuracy, confirming that the BOLD time courses captured semantic information (Huth et al. 2016, Pereira et al. 2018, Caucheteux et al. 2023). Importantly, this encoding model showed above-chance accuracy when applied to the BOLD time courses generated by our fNIRS-fMRI model, with the highest prediction accuracy observed in the dorsolateral PFC, dorsomedial PFC, and lateral temporal cortex, which are known to support language processing and narrative interpretation (Leong et al. 2020, Yeshurun et al. 2021, Fedorenko et al. 2024). These results suggest that the predicted BOLD retains content-specific information about participants’ cognitive experiences, highlighting the potential of using our approach to study how the brain encodes complex, naturalistic experiences in real-world contexts where fMRI is not feasible.

The overarching objective of our project is to develop a tool that enhances the versatility of fNIRS. To that end, we have made our models publicly available at https://github.com/ycleong/fNIRS-fMRI_models. We are committed to the continued development of this tool, including training and validating our models on additional stimuli and tasks. We also invite the fNIRS community to utilize and contribute to this tool, with the goal of enhancing the utility of fNIRS and facilitating its application in studying complex, real-world behaviours.

Author contributions

Shan Gao and Yuan Chang Leong (Conceptualization, Methodology), Shan Gao (Investigation [lead], Data curation [lead], Formal analysis [lead], Software [lead], Validation [lead], Visualization [lead], Writing—original draft [lead]), Ryleigh Nash (Validation [supporting], Visualization [supporting], Formal analysis [supporting], Software [supporting]), Shannon Burns (Software), Shan Gao, Ryleigh Nash, Yuan Chang Leong (Writing—review & editing [equal]), Shannon Burns (Writing—review & editing [supporting]), and Yuan Chang Leong (Supervision, Resources, Project administration)

Supplementary data

Supplementary data is available at SCAN online.

Conflict of interest:

None declared.

Funding

None declared.

Data availability

Preprocessed MRI data from the ‘Sherlock’ dataset can be accessed at Princeton’s DataSpace, https://dataspace.princeton.edu/jspui/handle/88435/dsp01nz8062179. Preprocessed fNIRS data can be accessed at https://github.com/ycleong/fNIRS-fMRI_models.