Skip to main content
PMC home page
eLife logoLink to eLife
. 2021 May 7;10:e62376. doi: 10.7554/eLife.62376

fMRI-based detection of alertness predicts behavioral response variability

Sarah E Goodale 1,2,✉, Nafis Ahmed 3, Chong Zhao 3, Jacco A de Zwart 4, Pinar S Özbay 4, Dante Picchioni 4, Jeff Duyn 4, Dario J Englot 1,2,3,5,6, Victoria L Morgan 1,2,5,6, Catie Chang 1,2,3,✉
Editors: Sanmi Koyejo7, Christian Büchel8
PMCID: PMC8104962  PMID: 33960930

Abstract

Levels of alertness are closely linked with human behavior and cognition. However, while functional magnetic resonance imaging (fMRI) allows for investigating whole-brain dynamics during behavior and task engagement, concurrent measures of alertness (such as EEG or pupillometry) are often unavailable. Here, we extract a continuous, time-resolved marker of alertness from fMRI data alone. We demonstrate that this fMRI alertness marker, calculated in a short pre-stimulus interval, captures trial-to-trial behavioral responses to incoming sensory stimuli. In addition, we find that the prediction of both EEG and behavioral responses during the task may be accomplished using only a small fraction of fMRI voxels. Furthermore, we observe that accounting for alertness appears to increase the statistical detection of task-activated brain areas. These findings have broad implications for augmenting a large body of existing datasets with information about ongoing arousal states, enriching fMRI studies of neural variability in health and disease.

Research organism: Human

Introduction

Brain function and behavior vary naturally over time. Behavioral responses can change even across repeated presentations of the same stimulus, and altered levels of neural variability may index individual differences in cognitive function (Dinstein et al., 2015; van Kempen et al., 2019; Cohen, 2018). One major determinant of behavioral and neural variability is the brain’s continuously fluctuating level of arousal, referring to the dimension of functional states linked with phenomena including alert wakefulness, drowsiness, and sleep. Growing evidence indicates that levels of arousal closely interact with processes underlying decision-making (van Kempen et al., 2019), visuomotor performance (Makeig et al., 2000; Poudel et al., 2018), perception (Mather and Sutherland, 2011), and attention (Foucher et al., 2004), motivating incorporation of arousal levels into models of healthy and pathological brain function (Lee et al., 2014; Nassar et al., 2012; Englot et al., 2008; Motelow et al., 2015; Salomon et al., 2011; Nashiro and Mather, 2011; Jawinski et al., 2019).

Functional MRI allows for investigating how behavioral variability may be underpinned by the activity in distinct spatial locations, or across large-scale functional networks, in the human brain. For example, variations in attentional (Rosenberg et al., 2016), motor, (Fox et al., 2006), and perceptual (Sadaghiani et al., 2009) responses, have been linked to fluctuations in spontaneous functional magnetic resonance imaging (fMRI) signals, observed both in focal regions of the brain (Fox et al., 2006; Sadaghiani et al., 2009; Hesselmann et al., 2008a; Hesselmann et al., 2008b) and in large-scale network structure (Rosenberg et al., 2016; Sadaghiani et al., 2009; Kelly et al., 2008; Thompson et al., 2013; Sadaghiani et al., 2015; Boly et al., 2007; Coste et al., 2011). Temporal variation in spontaneous fMRI signals and connectivity have also been found to be altered in neuropsychiatric disorders as well as in aging (Yang et al., 2014; Nomi et al., 2017; Calhoun et al., 2014). However, relatively few fMRI studies have examined the specific role of dynamically fluctuating arousal states in shaping ongoing neural or behavioral variability (Dinstein et al., 2015; Liu and Falahpour, 2020). One barrier stems from practical challenges linked with gathering established measures of arousal (such as pupillometry and EEG) during routine fMRI scans – including the need for additional setup time, MRI-compatible hardware, and post-processing for removal of MR-related artifacts. Therefore, the ability to estimate fluctuations in brain arousal from fMRI data itself would broaden the ability to study dynamic interactions between arousal, behavior, and large-scale brain activity. It could also improve the mechanistic interpretation of fMRI analyses, particularly in populations known to have altered arousal and autonomic regulation (Cohen-Zion and Ancoli-Israel, 2004; Hegerl et al., 2016).

Prior work on the detection of arousal states from fMRI alone has demonstrated that distinct EEG-defined sleep stages may be classified based on fMRI correlation patterns calculated across 1–2 min sliding windows (Tagliazucchi and Laufs, 2014; Tagliazucchi et al., 2012; Altmann et al., 2016). Moreover, it was found that many subjects can begin to lose wakefulness even within the first few minutes of an fMRI scan (Tagliazucchi and Laufs, 2014). Focusing on levels of arousal across the waking state, Wang et al., 2016 identified dynamic connectivity patterns, calculated in 40 s sliding windows, whose expression at each window tracked slow variations in subjects’ performance in an auditory vigilance task. In a complementary line of work, we recently developed an approach for extracting continuous fluctuations in arousal from fMRI data alone, this time at the frame-by-frame (TR) temporal resolution of fMRI data (Chang et al., 2016). The extracted ‘fMRI arousal index’ was found to correlate with intracranial electrophysiological arousal and spontaneous eye opening/closing in macaque monkeys (Chang et al., 2016), as well as with scalp EEG in human subjects (Falahpour et al., 2018). It is possible that the higher temporal resolution of this approach may also provide the opportunity of capturing arousal-dependent behavioral variability during task performance, in a temporally localized manner. If such a measure can indeed track trial-wise neural state changes, it may also open the possibility of incorporating arousal states into analysis of the many task fMRI datasets that lack EEG or pupillometry measures.

To examine this possibility, here we simultaneously record EEG, fMRI, and behavioral responses and examine whether this time-resolved marker of alertness (Chang et al., 2016; Falahpour et al., 2018) – derived from fMRI data alone – can be harnessed to track trial-to-trial variations in responses to sensory stimuli. We investigate whether an fMRI index of the ongoing arousal state, estimated within a short (5 s) interval before each trial, is predictive of a subject’s response to the upcoming stimulus. We also probe the impact of including fMRI-derived arousal covariates in standard statistical analyses of fMRI task activation.

Results

fMRI-derived alertness index tracks continuous variations in EEG alertness

In this study, we use an approach for inferring fluctuations in alertness from fMRI data alone, at the temporal resolution of the fMRI scan (Chang et al., 2016; Falahpour et al., 2018). Briefly, this is carried out by taking the running spatial projection of a spatial map linked with alertness (‘template map’) onto each successive fMRI volume in the new scan, resulting in a continuous time course of estimated alertness sampled at each fMRI volume (TR). A schematic of this approach is provided in Figure 1, with further details in Methods.

Figure 1. fMRI-based inference of alertness fluctuation.

Figure 1.

Open in a new tab

(A) A spatial template, created a priori (here, using resting-state scans with simultaneous EEG–fMRI), is projected onto each successive volume of a different, auditory task fMRI scan (not used in the template creation) via spatial correlation. These correlation values trace out a time course of estimated arousal fluctuation, which we refer to as the ‘fMRI alertness index’. (B) To assess the ability to predict electrophysiological alertness in the task scan, the estimated fMRI alertness index (red) is compared with an established EEG vigilance index that was collected simultaneously (the alpha/theta power ratio, convolved with a hemodynamic response function; blue). (C) The reaction time (RT) to each auditory stimulus is represented by an ‘X’ when the subject responded to a given trial and ‘O’ when they missed.

Importantly, for unbiased evaluation of the fMRI alertness index, here we derive the template map from a set of resting-state EEG–fMRI scans (where subjects rested passively with eyes closed) and apply it to estimate continuous alertness fluctuations in a separate set of auditory task scans (in which subjects were presented with binaural tones at long and unpredictable inter-stimulus intervals, with eyes closed; see Materials and methods). Subjects were requested to press a button as soon as possible upon hearing a tone. For all results in this study, the EEG data from the auditory task scans is only used for validation purposes and does not enter into the creation of the template or the fMRI alertness index.

We first determined how well this ‘fMRI alertness index’ could track an established index of continuous electrophysiological arousal during the auditory task scans. To quantify this agreement, we calculated the temporal correlation between the estimated fMRI alertness index and a simultaneously measured ‘EEG alertness index’ across each of the auditory task scans (Materials and methods). Figure 2A,B depicts example time courses of the fMRI alertness index (derived independently of these scans’ EEG data), superimposed on the measured EEG alertness index for comparison. Figure 2C shows the temporal cross-correlation between these two waveforms across the set of task scans, relative to a null distribution that was created by randomly permuting the fMRI alertness index and EEG alertness index across the task scans.

Figure 2. Cross-correlation between estimated fMRI alertness index and measured EEG alertness index.

(A, B) fMRI alertness index (red), superimposed on the EEG alertness index (blue), for two example scans. Note that the EEG data from these scans was not used in the creation of their fMRI alertness time courses, and are only used to evaluate the ability of the fMRI alertness index to track electrophysiological arousal. (C) Temporal cross-correlation between the fMRI alertness index and EEG alertness index (mean ± SE, n = 12 scans; red) together with a null distribution (gray) constructed for statistical comparison (see Materials and methods).

Figure 2.

Figure 2—figure supplement 1. Further analysis of the reliability and across-subject generalizability of the alertness estimation approach.

Figure 2—figure supplement 1.

Subjects were broken into sub-groups (Supplementary file 2) such that the two train/test comparisons shown in (A, B) involved disjoint sets of subjects in training and testing.(A, B) Across-subject testing of temporal cross-correlation between the fMRI alertness index and EEG alertness index, across subjects in different groups (mean ± SE, n = 6 scans). (C, D) Between-subject testing of temporal cross-correlation between the fMRI alertness index and EEG alertness index from subjects in the same groups (mean ± SE, n = 6 scans).
Figure 2—figure supplement 2. Lasso regression model for alertness prediction.

Figure 2—figure supplement 2.

Maximum temporal cross-correlation of the EEG alertness index with the fMRI alertness index, where the latter is predicted from lasso regression models (x-axis of each plot) or the template approach (y-axis). For the lasso, results are shown for models whose input consisted of (A) voxelwise fMRI time courses and (B) ROI time courses extracted from the Shen 268 atlas.
Figure 2—figure supplement 3. Model performance when trained on task data and tested on resting-state data.

Figure 2—figure supplement 3.

A template map derived from the task fMRI scans was applied to the resting-state data, where the cross-correlation with the EEG alertness index was calculated (red). For reference, the original analysis (in which the template is trained on resting-state data and applied to the task data, also shown in Figure 2C, is superimposed in blue).
Open in a new tab

To probe the across-subject reproducibility of fMRI alertness prediction within our dataset, we performed an additional cross-validation analysis in which subjects were divided into two distinct groups. Model training was performed on the resting-state data from one group and evaluated on the task data of the other group. Cross-correlations between the estimated resting-state fMRI alertness index and the measured EEG alertness index were comparable to the original results (Figure 2—figure supplement 1).

Variations in trial-by-trial response times relate to arousal state and are tracked by an fMRI alertness index

We next aimed to test whether the fMRI alertness index could provide a marker of arousal-dependent behavioral variability in an auditory task. For each scan, all trials were first binned into ‘hits’ and ‘misses’, corresponding to whether or not the subject pressed the button in response to the auditory stimulus. The event-locked EEG spectrogram showed clear evidence that hit and miss trials were accompanied by distinct arousal states, with the hit spectrogram showing greater power in the alpha band (8–12 Hz, consistent with increased arousal, particularly at stimulus onset) and the miss spectrogram showing greater power in the lower frequency bands commonly associated with increased drowsiness or sleep (Figure 3A).

Figure 3. fMRI alertness index differentiates between auditory task response (hit) and misses.

Figure 3.

Open in a new tab

(A) Event-locked EEG spectrogram for hits and misses. During hits, the elevated power in the alpha band may signify higher alertness; during misses, lower levels of alpha and increased power in lower frequencies suggest lower alertness. Time = 0 corresponds to stimulus onset. (B) Event-locked fMRI alertness index. Within each subject, the fMRI alertness index surrounding each stimulus was averaged across trials with hits (red) and misses (blue), and the across-subject means are represented by solid lines (shaded area is standard error; n = 12 scans for hits, n = 9 scans for misses). Across subjects, the fMRI alertness index predicted significantly greater levels of alertness in hits compared to misses, both within the pre-stimulus interval (5 s prior to stimulus onset; light gray region) and post-stimulus interval (10 s after stimulus onset; striped gray region). A wider time interval is shown for more complete visualization. (C) Visualization of individual subjects’ effects. Within individual scans, the fMRI alertness index was averaged within the 5 s pre-stimulus interval of each trial. Scans with only ‘hits’ are marked as gray circles; no scans had only misses. Within subjects who had both hits and misses, a significant difference in the pre-stimulus fMRI alertness index was found (p=0.01, paired t-test; d = 1.11).

The time course of the fMRI alertness index closely mirrored the EEG response. Significant differences between hits and misses were observed in both the pre-stimulus (p=3.34e-03, effect size [Cohen’s d] = 1.24) and post-stimulus (p=6.40e-06, d = 2.39) intervals, defined, respectively, as the intervals 5 s before, and 10 s after, the stimulus onset (Figure 3B; two-sample t-tests). The pronounced post-stimulus peak in the hits trials may again be indicative of stimulus-induced changes in alertness, which manifests with a hemodynamic response delay. Within individual subjects, trials with hits were associated with higher levels of the fMRI alertness index in the pre-stimulus interval compared to trials with misses (Figure 3C). Of those subjects who had both hits and misses, all but two followed the expected trend, and the differences between hits and misses were significant (t(8)=3.35, p=0.01, paired t-test; d = 1.11). Our focus was primarily on the pre-stimulus interval, as it can more closely represent the internal state of the brain without contribution from changes in arousal induced by the stimulus itself.

We then further aimed to determine whether the fMRI alertness index could account for variability in reaction times to the upcoming stimulus, in trials with hits. We again focus primarily on the pre-stimulus interval. Here, hits were further binned into an equal number of ‘slow’ and ‘fast’ trials, using a reaction-time threshold of 565 ms (median of responses pooled across all subjects). Within the 10 subjects that had both fast hits and slow hits, fast hits were associated with higher levels of the pre-stimulus fMRI alertness index compared to slow hits (t(9)=4.52, p=0.001, paired t-test; Cohen’s d = 1.43; Figure 4A), and all showed higher mean values during fast compared to slow hit trials. Similar findings were obtained when using subject-specific divisions into fast and slow trials (top and bottom 10%, respectively; Figure 4—figure supplement 1). This finding supports the notion that pre-stimulus patterns in spontaneous brain activity encode behaviorally relevant internal states within individual subjects, which may be quantified using an fMRI pattern-recognition approach.

Figure 4. fMRI alertness index differentiates between trials with fast and slow reaction times.

(A) Within individual subjects, the pre-stimulus fMRI alertness index was averaged across fast and slow reaction-time trials. Subjects with only ‘slow hits’ are marked as gray circles (none had only fast hits). There was a significant difference between fast and slow hits within subjects who exhibited both fast and slow responses (p=0.001, paired t-test; d = 1.43). (B) Three types of behavioral responses (fast hit, slow hit, and miss) were examined for the fMRI alertness index, as well as the EEG alpha/theta ratio for comparison. In the pre-stimulus interval, both the EEG alpha/theta ratio and the fMRI alertness index showed a main effect of behavioral response across subjects; p<0.05, (one-way ANOVA). Tukey–Kramer post hoc tests, controlled for multiple comparisons, indicated pairwise differences for fast hits (orange) versus slow hits (yellow) for EEG and between fast hits and misses (blue) for both EEG and fMRI (*p<0.05). (C) In the post-stimulus interval, a main effect of behavioral response was also found for both EEG and fMRI, and post-hoc tests indicated significant differences in both fast hits and slow hits versus misses (*p<0.05). Error bars represent standard error.

Figure 4.

Figure 4—figure supplement 1. fMRI alertness index differentiates between the 10% fastest and slowest reaction times in each individual subject.

Figure 4—figure supplement 1.

(A) Within individual scans, the fMRI alertness index was averaged within the pre-stimulus interval, and separately within the 10% of each subject’s fastest and slowest response-time trials. Within scans, significant differences were found in the pre-stimulus fMRI alertness index between these fast vs. slow responses (*p=0.007; paired t-test; d = 0.95). (B, C) The analysis of Figure 4B,C was repeated using the 10% of subjects’ slowest and fastest responses to define the ‘fast hit’ (orange) and ‘slow hit’ (yellow) trials.
Figure 4—figure supplement 2. Trial-by-trial correspondence between EEG alpha/theta power and fMRI alertness index during pre-stimulus interval.

Figure 4—figure supplement 2.

To account for hemodynamic delays, the EEG alpha/theta power and fMRI alertness index were calculated in time-interval offset from one another: (−5.16 s to 2.64 s for EEG and −1.8 s to +0.72 s for fMRI, where t = 0 represents stimulus onset). Each plot corresponds to a subject, and each datapoint corresponds to a trial. Only trials with behavioral responses (hits) are shown. Here, the EEG alpha/theta ratio was not convolved with a hemodynamic response function and is presented in its original units.
Figure 4—figure supplement 3. Effects of regressing out low-frequency respiratory and cardiac fluctuations from the fMRI signal.

Figure 4—figure supplement 3.

(A) For each scan, the agreement between the fMRI alertness index and the EEG alertness index was quantified before and after regressing out variations linked with low-frequency RV and HR. (B) After regressing out physiology (RV and HR), the fMRI alertness index still differed significantly between trial types in both the pre-stimulus (F(2) = 4.58, p=0.02) and post- stimulus (F(2) = 8.92, p=0.1e-2) intervals. Post hoc analyses were performed using Tukey–Kramer method for multiple comparisons (*p<0.05). Error bars shown represent standard error. (Compare to Figure 4B for results from data from which physiological waveforms were not removed.)
Figure 4—figure supplement 4. Evaluation of fMRI alertness index with high-motion subjects removed.

Figure 4—figure supplement 4.

(A) Temporal cross-correlation between the fMRI alertness index and EEG alertness index in all subjects (mean ± SE, n = 12 scans; blue) and only those subjects without high motion (mean ± SE, n = 11 scans; red). (B) Within individual scans, the fMRI alertness index was averaged within the 5 s pre-stimulus interval of each trial. At the group level, significant differences were found between hits and misses (p=0.015; paired t-test, d = 1.29). (C) Within individual subjects, the pre-stimulus fMRI alertness index was averaged across fast and slow reaction-time trials. There was a significant difference between fast and slow hits across subjects (p=0.002; paired t-test, d = 0.94).
Figure 4—figure supplement 5. Reaction time histogram.

Figure 4—figure supplement 5.

Histogram across all subjects’ reaction times, pooled across all trials. We then additionally examined whether the fMRI alertness index is modulated across all three trial types (fast, slow, and missed trials), performing an identical analysis on the EEG alpha/theta ratio for comparison. This was calculated using one-way ANOVA, with post hoc pairwise tests controlled for multiple comparisons (Tukey–Kramer method). In the pre-stimulus interval, both fMRI and EEG had a main effect of behavioral response (EEG: F(2) = 13.60, p=7.45e-05; fMRI: F(2) = 8.31, p=0.15e-02; Figure 4B). Post hoc analyses indicated a significantly elevated alpha/theta ratio (corresponding to higher electrophysiological alertness) in ‘fast hits’ versus ‘misses’, and ‘fast hits’ versus ‘slow hits’ with EEG, and for ‘fast hits’ versus ‘misses’ with fMRI.
Open in a new tab

Although our initial focus was on the pre-stimulus interval, we also investigated the modulation of alertness measures following stimulus onset. In the 10 s post-stimulus interval, there was also a main effect of behavioral response for both EEG alpha/theta and the fMRI alertness index (EEG: F(2) = 8.82, p=0.1e-02, fMRI: F(2) = 14.12, p=5.75e-05; Figure 4C). Both EEG and fMRI showed post hoc pairwise differences between both fast and slow responses compared to misses, and neither showed a difference between fast and slow hits. A potential explanation is that, while a lower level of alertness before the stimulus led to slower reaction times in ‘slow’ trials, the incoming auditory stimulus led to a subsequent increase in alertness, approaching that of fast trials. We also examined the trial-by-trial correspondence between the fMRI alertness and EEG alpha/theta power using an offset that accounts for approximate hemodynamic delays (Figure 4—figure supplement 2).

Finally, while multi-echo independent component analysis (ICA) has proven effective in reducing head-motion effects (Kundu et al., 2013), we also ensured that scans with frame-to-frame head motion exceeding our voxel size (3 mm; corresponding to one resting-state and one task scan; Supplementary file 3) did not impact the major findings of this study. Removing these high-motion scans from the analysis had minimal impact on the correspondence between the fMRI alertness index with EEG and behavior (Figure 4—figure supplement 4).

A small number of voxels may suffice for detecting alertness with fMRI

In a previous study of macaque monkeys, we demonstrated that the whole-brain vigilance template could be reduced to its top ~1% of positive and negative voxels without considerable loss in predictive power (Chang et al., 2016). In addition, despite the much larger number of negative compared to positive voxels, retaining voxels of both signs was found to be critical (Chang et al., 2016). To determine whether a small number of template voxels also suffices for inferring EEG and task response variability in human subjects, we performed a similar experiment, successively thresholding the template to include different percentages of its highest-magnitude positive and negative voxels. We also examined the impact of including only positive- or only negative-valued voxels from the template. When using only positive or only negative voxels, performance was found to decline as smaller fractions of voxels were used (Figure 5). By contrast, when jointly including positive and negative voxels, we found that there was little change in the predictive power, even at thresholds down to 1%.

Figure 5. Effects of spatially reducing the vigilance template.

Figure 5.

Open in a new tab

(A) The template was successively reduced to retain the indicated fraction of highest-magnitude voxels drawn from the set of negative, positive, or both negative- and positive-signed voxels. These reduced voxel sets were used as templates for calculating the fMRI alertness index in each scan. The x-axis indicates the fraction of voxels relative to the whole-brain template; that is, at x = 0.2, we used the highest 20% of positive values (red) and the highest 20% of negative voxels (blue), then combined them to make a joint positive and negative template (magenta). To focus on relative effects, all three lines were normalized to their value at the fraction of 1.0. (B) Template map produced by retaining only the top 1% of positive and negative voxels of the original whole-brain template, respectively. (C) The pre-stimulus fMRI alertness index, generated from this reduced (‘1%”) template, for fast, slow, and missed trials (see Figure 4B for analogous results using the whole-brain template).

Here, the highest-magnitude (top 1%) voxels with negative sign were localized mainly to primary visual, auditory, and sensory/motor cortices, and those with the highest-magnitude positive sign were found in the ventricles (Figure 5b). This highly reduced template map also generated an fMRI alertness index that replicated behavioral results obtained from the whole-brain template map. The fMRI alertness index generated from the reduced template (top 1% positive, top 1% negative voxels), calculated in the pre-stimulus interval, showed a significant main effect of behavioral response (F(2)=10.09, p=0.0005; Figure 5c). This finding suggests that despite the broad nature of arousal-related signal changes across the brain, the information is inherently low dimensional, and a small fraction of voxels may suffice for detecting fluctuations in alertness.

Modeling fluctuations in alertness impacts inference of task activation

Beyond providing information about the brain’s ongoing arousal levels, an fMRI-derived alertness index may potentially allow for more reliable detection of task-evoked activation, by offering a means for explicitly modeling state-related fMRI signal changes across an experiment. To examine this possibility, we performed a standard general linear model (GLM) analysis of the auditory task responses (convolved with a canonical hemodynamic response function [HRF]) and quantified the impact of including the fMRI alertness index as a covariate of no interest. For all analyses, a regressor capturing the onset of missed stimuli (convolved with a canonical HRF) was also included as a covariate of no interest. As shown in Figure 6A, a model that included only the event-related task responses resulted in weak activation in the expected bilateral auditory and left motor regions, along with negative BOLD signal change in a number of cortical regions, with some resemblance to the pattern of correlations with EEG arousal (Figure 1, Figure 6—figure supplement 1). Including the fMRI alertness index as a covariate reduced the negative responses and increased t-scores in bilateral auditory cortex, left motor cortex (the region expected to be associated with right-handed button presses), and the contralateral cerebellum (Figure 6B). Interestingly, this covariate also markedly increased the activation in areas associated with the ‘salience network’ (including the anterior cingulate cortex and insula [Menon and Uddin, 2010], which are known to co-activate with externally salient or oddball stimuli Crottaz-Herbette and Menon, 2006); this may perhaps to be expected given the long and unpredictable inter-stimulus intervals in this task. Using the EEG alertness index as a regressor also reduced the negative activation and increased the t-statistics in auditory, motor, and salience areas (Figure 6C), though surprisingly, to a somewhat lesser extent than with the fMRI index.

Figure 6. Modeling alertness fluctuations in fMRI analysis impacts statistical inference of task-evoked activation.

General linear model analyses of responses to event-related auditory task. For each subject, beta maps were calculated for event-related responses to the auditory stimuli, resulting in group-level t-statistics (voxelwise one-sample t-test) for (A) a model that includes regressors only for the task stimuli; (B) a model that includes task regressors along with the fMRI alertness index as a covariate; and (C) a model that includes task regressors along with the EEG alertness index as a covariate.

Figure 6.

Figure 6—figure supplement 1. Comparison between event-related task response and vigilance template.

Figure 6—figure supplement 1.

General linear model analysis of the event-related task responses (A) was accompanied by negative BOLD signal changes in a number of cortical regions, resembling the pattern of correlations with an EEG index of alertness (B).
Open in a new tab

Comparison to lasso regression model

Finally, to investigate how this template-based method compares against a more conventional temporal regression approach, we also constructed lasso regression models (see Materials and methods for further details). The lasso model yielded correlations with the EEG alertness index that were similar to, though slightly lower than, those generated by the template-based approach (Figure 2—figure supplement 2). Furthermore, the lasso model that was trained on fMRI time series from a 268-region atlas (Shen et al., 2013) was superior to that which used the time series from all voxels as input.

Discussion

The present results indicate that ongoing variations in subjects’ alertness, as directly estimated from fMRI data alone, may track both concurrently measured EEG and inter-trial behavioral responses. These findings are in line with a growing literature indicating that time-varying, internal state changes may be decoded based on patterns in fMRI data, which may be harnessed to study specific dimensions of cognitive and physiological variability such as arousal, attention, and emotion (Rosenberg et al., 2016; Thompson et al., 2013; Chang et al., 2015). In addition, the ability to estimate vigilance during fMRI tasks allows for modeling state-dependent variability in task performance (Podvalny et al., 2019) and BOLD responses, which in turn may improve the reproducibility and interpretation of fMRI datasets that lack EEG or other external vigilance measures.

Changes in arousal are coordinated by ascending projections from the brainstem, via regions including thalamus, basal forebrain regions, and posterior hypothalamus (Jones, 2005; Saper et al., 2005), and manifest in altered responsiveness to sensory stimuli (Ogilvie, 2001). During periods of drowsiness and light sleep, large-amplitude and extensive fMRI signal changes have been observed in which much of the cortex is modulated together, and certain subcortical structures (including the thalamus) display opposing fluctuations (Chang et al., 2016; Falahpour et al., 2018; Olbrich et al., 2009; Goldman et al., 2002; Moosmann et al., 2003; Ong et al., 2015; Poudel et al., 2014; Liu et al., 2012; Feige et al., 2005). These inversely related signals may be the hemodynamic correlates of subcortical arousal modulation, and perhaps reflect inhibition of incoming sensory signals to the cortex by the thalamus in a feedback role (Sherman and Guillery, 2002). Here, we use the temporal expression of an EEG arousal-linked fMRI signal pattern (‘template’) to estimate a subject’s alertness fluctuation across the course of an fMRI scan. In addition to finding that such an fMRI-based inference of alertness correlates with electrophysiological arousal, consistent with prior work (Chang et al., 2016; Falahpour et al., 2018), the use of binaural tones here enabled us to investigate its bidirectional interaction with sensory stimuli.

In the pre-stimulus interval, the fMRI alertness index was indicative of upcoming behavioral responses. This finding extends a body of prior work linking fMRI spontaneous activity with behavior and perception (e.g., Sadaghiani et al., 2009; Kelly et al., 2008; Thompson et al., 2013; Sadaghiani et al., 2015), and it is also interesting to consider the potential links between the present findings and large-scale network patterns that have been previously shown to predict vigilance or tonic alertness. In particular, higher levels of pre-stimulus activity in the default-mode network and/or salience (cingulo-opercular) network, and lower levels in task-positive/dorsal attention networks, have been associated with auditory stimulus detection (Sadaghiani et al., 2009) and faster reaction times in visual psychomotor vigilance task (Thompson et al., 2013). In line with these observations, our template map – derived from EEG spectral power fluctuations – is negative in areas spanning dorsal attention network and had weak positive correlations in the anterior cingulate cortex, a key node of the salience network; however, we did not see group-level positive effects in the default-mode network. Future work may investigate the degree to which shared and distinct physiological mechanisms account for these observations, as well as the extent to which arousal interacts with other neural sources of behavioral variability, such as mind wandering (Kucyi et al., 2016).

Given that broad regions of cortex together exhibited correlated negative responses to arousal, we also investigated whether a much smaller set of voxels may suffice for extracting an alertness measure from fMRI. Indeed, we found that retaining even just 1% of positive voxels and 1% of the negative voxels produced only a 5.7% decrease in performance on average (in terms of correlation with EEG alertness). In addition, the pre-stimulus alertness index – estimated from this highly reduced template – replicated the main effect of behavioral response seen with that estimated from the whole-brain template. This effect suggests that a joint pattern of positive and negative voxels may be most critical for decoding arousal with the present approach, consistent with our prior results in macaque monkeys (Chang et al., 2016). However, unlike in the macaque study, where the strongest positive voxels in the template (i.e., those which had the greatest positive correlation with alertness) were localized to thalamus, here we found the strongest positive correlations in cerebrospinal fluid (CSF), though weaker positive clusters were also found in mediodorsal thalamus and anterior cingulate cortex. One potential difference may come from the manner in which the template map was derived – here from the EEG alpha/theta ratio during eyes closed, and in the macaque study from spontaneous eye opening/closure. Positive correlations between an arousal index and CSF have been seen in prior human EEG–fMRI studies that use a continuous EEG power ratio (alpha to theta and delta bands Falahpour et al., 2018) and discrete EEG-defined vigilance staging (Olbrich et al., 2009), where ventricular signals in the latter were suggested to potentially relate to effects of heart rate (HR) on fMRI data (de Munck et al., 2008). A pattern very similar to the reduced template, with opposing gray-matter and ventricular signals, was also reported to comprise the first principal component of the global fMRI signal that is modulated by sleep depth (McAvoy et al., 2019). An inverse relationship between CSF and gray-matter signals has also been found to correlate with EEG (Fultz et al., 2019), and both EEG and autonomic responses (Özbay et al., 2019), during sleep. One mechanistic explanation offered recently is that widespread fMRI signal changes, generated by either neuronal or autonomic activity, lead to antagonistic fMRI signal changes in the downstream venous vasculature, in part due to a disjoint timing and amplitude relationship of blood volume and blood oxygenation effects (Özbay et al., 2018). Further work will be necessary to establish potential links between these observations and those of studies which have observed CSF fMRI signals in other EEG frequencies (Kiviniemi et al., 2016).

A general challenge in understanding the effects of arousal in fMRI stems from the complex assortment of neural, metabolic, and physiological factors contributing to BOLD fMRI signals (Liu, 2016; Duyn et al., 2020). Fluctuations in arousal are closely coupled with physiological changes, such as changes in HR and respiratory patterns (Özbay et al., 2019; Yuan et al., 2013). Therefore, it is hard to disentangle these various effects with certainty. However, when the goal is to detect vigilance changes from fMRI, systemic physiological effects in fMRI signals may, in fact, contribute useful state-related information. In additional experiments, we found that projecting out low-frequency fluctuations in HR and RV generally reduced the correlation between the fMRI and EEG alertness measures (Figure 4—figure supplement 3 and Materials and methods), suggesting that retaining low-frequency systemic physiological components may present useful information when the goal is to extract arousal fluctuations from the fMRI data. The whole-brain template derived here also overlaps very closely with a spatial pattern linked with sympathetic arousal in stage-2 sleep (Özbay et al., 2019; Özbay et al., 2018) as well as in drowsy wakefulness and stage-1 sleep (Chang et al., 2018), whereby gray-matter regions (predominantly primary sensory cortex) are anti-correlated with BOLD signal in deep white matter and CSF. Our template map may, therefore, reflect in part the concurrent modulation of central and peripheral arousal, and their systemic physiological correlates.

Beyond providing information about ongoing arousal levels, an fMRI-derived alertness index may also allow for more precisely identifying task-evoked activation. While it is widely recognized that natural drifts in wakefulness are common during fMRI, few studies explicitly model dynamic arousal fluctuations in standard fMRI task analyses, as concurrent EEG and pupillometry is relatively uncommon during routine fMRI scans. Here, we observed that including the fMRI alertness index as a covariate increased the statistical significance of task-active sensory and motor regions as well as canonical salience network regions (Menon and Uddin, 2010; Crottaz-Herbette and Menon, 2006), and reduced widespread cortical deactivations. Regarding the latter, it is also worth noting that when only the task stimuli are modeled, the resulting activation map bears some resemblance to the vigilance template itself, showing similar patterns of negative BOLD signal changes and positive signals in white matter (Figure 6—figure supplement 1). This phenomenon might be explained by task-induced arousal and autonomic fluctuations, which are mitigated when including an arousal covariate. However, an important caveat is that the ‘true’ activation state of the brain is not known; for instance, here one cannot be certain whether diminishing the negative responses with an alertness covariate is also removing local neural effects. We also note that this analysis was not intended to suggest an optimal set of nuisance covariates, but to provide an indication of how covarying for measures of alertness may impact fMRI analyses. Future work may investigate this phenomenon in a wider range of task contexts.

A major goal of template-based alertness estimation is to provide an avenue for researchers to infer fluctuation in alertness from fMRI data when external measures such as EEG and pupillometry are not available in the scanner, which is the case for most existing fMRI data and publicly shared databases. Given a template map trained on a set of EEG–fMRI data (such as this one), researchers could apply it to other fMRI datasets to extract time courses of alertness fluctuation. The present results suggest that, in addition to correlating with an electrophysiological alertness index, this fMRI alertness index may be sensitive to brain states that modulate behavioral responses. Nonetheless, the approach currently has several limitations. One is that it currently detects primarily relative, rather than absolute, levels of alertness across a scan. Since spatial correlation is used to determine a frame-by-frame alertness measure, the fMRI alertness index is restricted between −1 and 1. In addition, as fMRI signals are often detrended to remove scanner artifacts, such an index may not detect slower baseline shifts in alertness. In addition, we and others observe inter-subject and inter-scan differences in the performance of fMRI alertness estimation (Falahpour et al., 2018). Such differences may partially reflect the degree of alertness fluctuation present in the fMRI data itself (with greater modulation of alertness lending itself to better prediction), but these questions are currently under investigation.

One key limitation of this study is the small sample size. As a result, the effect sizes of the current results may be overestimated, and findings may not generalize well to the population from which they are sampled. Future work should be carried out with this approach using larger sample sizes as well as other experimental conditions. Nonetheless, there are several factors that increase our confidence in the present results: our scans are long (24.5 min per scan), and the replication analyses carried out in Figure 2—figure supplement 1 and Figure 4—figure supplements 1 and 4 further support the generalizability of this approach. Furthermore, although the present study centered on predicting task-related behavioral responses, we also performed an analysis in which the template was constructed from the task scans, and the fMRI alertness index was evaluated on the held-out resting-state scans, obtaining comparable results (Figure 2—figure supplement 3). Another limitation is that here, we only considered a template-based approach and a lasso regression model to predict alertness. It is possible that other approaches could be used to extract dynamic measures from alertness from fMRI. However, our current results with these two models support the notion that vigilance can be robustly estimated from fMRI data, and indicate that there may be multiple approaches for extracting this information. Further investigation of alternate linear models for vigilance prediction would be an interesting area to develop in future work, including the use of other regularization methods as well as parcellation atlases for deriving input time series.

In conclusion, our results indicate that time-resolved predictions of alertness, directly estimated in fMRI data without the need for simultaneous EEG, may capture inter-trial behavioral responses and continuous internal state variations. These findings have implications for broadening the study of brain arousal, and its interactions with cognitive and perceptual variability, in the large number of fMRI studies that lack external vigilance measures.

Materials and methods

Subjects and data acquisition

Simultaneous EEG–fMRI data were acquired from 14 healthy, right-handed adult subjects (eight females, aged 26 ± 4 years). All subjects provided written informed consent, and human subjects protocols were approved by the Institutional Review Boards of the National Institutes of Health and Vanderbilt University. From these subjects, scans acquired in the eyes-closed resting state and/or with a sparse auditory task (described below) were considered for analysis. One scan was excluded due to a buffer overflow error in the EEG data acquisition. This yielded a total of 23 scans (11 resting-state scans and 12 auditory task scans), each lasting 24.5 min. Further details about the scans corresponding to each subject are provided in Supplementary file 1.

MRI data were acquired on a 3T Siemens Prisma scanner (Siemens, Erlangen, Germany) with a Siemens 64-channel head/neck coil. A high-resolution, T1-weighted structural image (TR = 2200 ms, TE = 4.25 ms, flip angle = 9 deg, inversion time = 1000 ms, matrix = 256×256, 160 sagittal slices, 1 mm isotropic) was acquired for anatomic reference. BOLD fMRI data were acquired with a multi-echo, gradient-echo EPI sequence, with the following parameters: flip angle = 75 deg, TR = 2100 ms, echo times = 13.0, 29.4, and 45.7 ms, voxel size = 3 × 3 × 3 mm3, slice gap = 1 mm, matrix size = 82×50, 30 axial slices, acceleration factor = 2. The duration of each scan was 24.5 min (700 volumes). Scalp EEG was acquired simultaneously with fMRI using a 32-channel MR-compatible system (BrainAmps MR, Brain Products GmbH) at a sampling rate of 5 kHz and was synchronized to the scanner’s 10 MHz clock to facilitate reduction of MR gradient artifacts. EEG channels were referenced to FCz. Photoplethysmography (PPG) and respiration belt signals were also acquired during the scans (Biopac, Goleta, CA). The PPG transducer was placed on the left index finger, and MRI scanner (slice) triggers were recorded together with the physiological and EEG signals for data synchronization.

For the resting-state scans, subjects were instructed to keep their eyes closed and to stay awake as best as possible. For the auditory task, binaural tones were delivered (VisuaStim Digital; Resonance Technology, Northridge, CA) with randomized inter-stimulus intervals. For the auditory task, data were drawn from scans having either of two versions of this task, where the two versions differed only in the timing of tone delivery. In one version (corresponding to half of the scans), the inter-stimulus interval (ISI) ranged between 29 and 41 s (41 trials per scan); in the other version, tones were presented with ISI ranging from 8 to 88 s, from which we retained only those which occurred at least 24 s after the previous stimulus (and at least 10 s prior to the subsequent stimulus), resulting in 29 trials per scan. Subjects were asked to make a right-handed button press as soon as possible upon hearing a tone (Lim and Dinges, 2008). Auditory tones were delivered via earbuds placed over subjects’ ear plugs. During the task scans, subjects were also instructed to keep eyes closed. One rationale for using an eyes-closed auditory task (e.g., as opposed to an eyes-open or visual task) is that it avoids the possibility of subjects missing a stimulus due to variation in their gaze.

Although the instruction was for subjects to stay awake, the long scans were designed to elicit variability in vigilance state – particularly, reduced wakefulness and drifts between wakefulness and light sleep. A random half of the resting-state and task data were independently sleep staged by two certified technologists at the Vanderbilt University Medical Center. Standard sleep scoring was carried out in 30 s epochs, according to standard AASM criteria (American Academy of Sleep Medicine, 2007). These ratings indicated that, across all scans combined, 85.6% of epochs were rated as ‘wake’, suggesting that subjects were mostly alert. It is therefore likely state sleep was not a major factor in our experiments.

fMRI and EEG pre-processing

Within each scan, the time series of 700 volumes corresponding to each of the three echo times were processed in the following way. For all three of these echo time series, the first seven volumes were dropped to allow magnetization to reach steady state. Then, motion coregistration (six-parameter rigid body alignment) and slice-timing correction were carried out using the functions 3dvolreg and 3dTshift in AFNI (https://afni.nimh.nih.gov/afni). For motion coregistration, the alignment parameters were estimated only for the time series of the second (middle) echo, and the resulting parameters were applied to the time series of all three echoes. Maximum frame-to-frame displacement was also calculated, as shown in Supplementary file 3. Following this initial processing, multi-echo ICA denoising was carried out using tedana 0.0.9a (Elizabeth and Salo, 2020; Kundu et al., 2012; Kundu et al., 2013). Briefly, this procedure aims to separate fMRI signal components generated from BOLD mechanisms from those arising from non-BOLD signal sources (such as scanner drifts and head motion). Spatial ICA is performed on the multi-echo fMRI data, and the resulting components are sorted using automated criteria to determine those which exhibit linear scaling in their percent-signal-change as a function of echo time (‘BOLD’ components) and those which do not (‘non-BOLD’ components). The data are then re-constructed using only the former. A more complete description of this approach can be found in Kundu et al., 2012, and several studies have demonstrated its efficacy in reducing artifacts including head motion, and aliased cardiac pulsatility and breath-to-breath respiratory artifacts (Kundu et al., 2013), the latter of which are otherwise handled with methods such as RETROICOR (Glover et al., 2000). However, fMRI variance due to low-frequency respiratory volume (RV; Birn et al., 2006) and cardiac rate (Shmueli et al., 2007) fluctuations are unlikely to be removed by this procedure, as they arise from a BOLD mechanism (Kundu et al., 2012). Here, we calculated RV and cardiac rate signals but did not initially project them out of the data, as their impact was investigated in the analysis (see Methods). Following multi-echo denoising, fMRI data were nonlinearly registered to a standard-space MNI152 template using SPM (https://www.fil.ion.ucl.ac.uk/spm/), followed by fourth-order polynomial detrending and spatial smoothing (FWHM 3 mm) in AFNI.

For the EEG data, reduction of gradient and ballistocardiogram (BCG) artifacts was carried out using BrainVision Analyzer 2 (Brain Products, Munich, Germany), with parameters and procedures described in Moehlman et al., 2019. Briefly, gradient artifact reduction followed the average artifact subtraction technique (Allen et al., 2000) using slice triggers. Following gradient artifact correction, EEG data were downsampled to 250 Hz. BCG artifact correction proceeded in two steps. First, an artifact template locked to cardiac R-peaks was subtracted from the data, after accounting for an estimated temporal offset between the R-peak and the BCG artifact. Second, independent component analysis (ICA) was performed on the template-subtracted data, and components likely related to residual BCG were removed by re-constructing the data (via inverse ICA) without the artifactual components. Putative BCG artifact components were manually determined based on the presence of temporal deflections locked with the cardiac cycle, a spatial topography consistent with BCG artifact, and a relatively large contribution to the global field power. For this study, we removed at most two components per scan, to avoid removing potential signals of interest.

EEG-based alertness index

A measure of alertness was computed from the EEG signal (averaged over channels P3, P4, Pz, O1, O2, Oz) within the 2.1 s interval of each fMRI time point (TR) by taking the ratio of the root mean square (rms) amplitude in the 8–12 Hz range over the rms amplitude in the 3–7 Hz range (alpha/theta ratio). Various EEG-based metrics have been associated with wakefulness; however, many are related to the ratio of power in middle frequency bands (i.e., alpha, beta) to the power in lower frequency bands (i.e., delta, theta) (Olbrich et al., 2009; Klimesch, 1999; Oken et al., 2006; Jobert et al., 1994; Wong et al., 2013; Horovitz et al., 2008). The alpha/theta ratio has previously been used in several human EEG–fMRI studies (Horovitz et al., 2008; Laufs et al., 2006).

The EEG alpha/theta time course was temporally aligned to the fMRI time course by removing the first seven timepoints. For analyses that directly correlate EEG with the fMRI alertness index (Figures 1, 2, 5, and 6), the aligned EEG alpha/theta time course was mean-centered and convolved with the default gamma-variate HRF provided in SPM (https://www.fil.ion.ucl.ac.uk/spm/) to account for approximate hemodynamic delays in relating EEG to fMRI, and band-pass filtered with nominal cutoffs from 0.01 to 0.2 Hz (using the bandpass function in MATLAB) to approximate the bandwidth of the fMRI data. The resulting signal is referred to as the ‘EEG alertness index’. For the analyses in Figure 4, Figure 4—figure supplement 2, we use the raw (not hemodynamically filtered) version of the EEG alpha/theta ratio.

fMRI-based alertness index

A template-based fMRI-based estimate of alertness was calculated according to the approach in Chang et al., 2016; Falahpour et al., 2018; Figure 1. Briefly, this approach centers on the use of a voxelwise spatial pattern (‘template’ map), indicating the degree to which a given voxel increases or decreases its activity in concert with fluctuations in alertness, as measured using EEG. Given such a template map (derived from a set of scans where concurrent EEG is available), one may then apply it to derive a time course of alertness in a new scan using the fMRI data alone (i.e., even if that new scan does not contain simultaneous EEG). To allow for unbiased estimates of performance, the template map was constructed using the set of resting-state scans, and its performance was analyzed in the set of auditory task scans. To create the template, the preprocessed fMRI data from each subject’s resting-state scan was temporally correlated, on a voxelwise basis, with the subject’s corresponding EEG alertness index, resulting in a voxelwise spatial map of Pearson's correlation coefficients. The resulting correlation maps underwent a Fisher-Z transformation, and were averaged across subjects to create a single, average template map (‘vigilance template’). Given this group-average vigilance template (3D spatial map), a time course of alertness (‘fMRI alertness index’) was estimated for each auditory task scan’s fMRI data without using the EEG, as auditory task scans were not used in forming the template map. After temporal z-scoring, a time course of alertness (‘fMRI alertness index’) was estimated for each task scan by projecting the vigilance template, on a volume-by-volume (TR-by-TR) basis, onto each volume of the task scan. To validate the correspondence between the fMRI alertness index and the EEG alertness index on the task data, we calculated the temporal cross-correlation between these two waveforms (Figure 2A,B). A null distribution was generated by taking 1000 random permutations (by shuffling the assignment of EEG and fMRI alertness time courses between subjects) across the task scans. Their collective means and standard errors were compared in Figure 2C.

To further investigate the reproducibility of fMRI alertness prediction, subjects were divided into two non-overlapping subgroups for both resting and task conditions (Supplementary file 2). We then created a template from the resting-state data of one group and applied it to the task data of the other (non-overlapping) group (Figure 2—figure supplement 1). Furthermore, to test whether a model trained on task data can be used to predict vigilance in resting state, we also performed the reverse, constructing our template this time with the task scans from each subject group, and evaluating the fMRI alertness index on the held-out resting-state scans (Figure 2—figure supplement 3).

Relationship between EEG/fMRI alertness and behavioral responses

To investigate the fMRI alertness index (and its comparison to EEG) in relation to trial-by-trial behavioral responses, we extracted segments of these waveforms in the [−5 s to 10 s] interval around each stimulus onset. To achieve more precise alignment with the stimulus onsets, these time courses were first interpolated 2.5-fold to a 0.84 s temporal grid to obtain a finer sampling axis. Each segment was then sorted according to whether it was a ‘hit’ (button press response) or a ‘miss’ (no button press response or a response of >5 s). This resulted in a total of 310 hits and 90 misses, pooled across scans. The ‘hits’ were further separated based on the subject’s reaction time into a slow hit (>565 ms, n = 155) and a fast hit (<565 ms, n = 155), where the threshold was defined by the median across all subjects’ pooled trials. This threshold is close to the 500 ms value commonly used to define performance lapses (Lim and Dinges, 2008).

For the time courses shown in Figure 3B, temporal segments of the fMRI alertness index corresponding to ‘hits’ were first averaged across trials within each subject, and the mean and standard error across subjects at each time point are depicted by the shaded error bars. The same was repeated for ‘misses’. For further analyses across behavioral responses (Figure 3C, Figure 4), time points of the fMRI (or EEG) data segments for each trial were averaged within the indicated interval of interest (e.g., ‘pre-stimulus’); then, within each scan, these values were averaged across trials of a given type (e.g., ‘miss’), resulting in one value per scan (per trial type). Comparison between trial types within the pre-stimulus interval were carried out using paired t-tests. Modulation of the fMRI alertness index (or EEG alpha/theta power) across fast, slow, and missed responses was assessed using a one-way ANOVA, with post hoc multiple comparisons analyses using the Tukey–Kramer method. Cohen’s d was used to assess effect sizes. An additional analysis was performed investigating the use of intra-individual threshold divisions as a comparison, considering the 10% fastest and slowest trials for each subject (Figure 4—figure supplement 2). A histogram of all subjects’ reaction times is provided in Figure 4—figure supplement 5.

Template thresholding experiment

Our prior work indicates that only a small fraction of the spatial template may be necessary for accurate vigilance prediction in macaques (Figure S9 in Chang et al., 2016). Based on these findings, we wanted to observe the impact of thresholding the template to smaller and smaller numbers of voxels in human subjects. This was performed sequentially from the full template down to 1% of the voxels with the strongest positive and negative correlation with the fMRI alertness index. Starting from the full template map (Figure 1), at each iteration a new template was created by reducing the voxel sets to the fraction of highest positive and negative correlations from the following percentages: 100%, 80%, 60%, 40%, 20%, 10%, 5%, and 1%. The fMRI alertness index derived from the reduced voxel template was compared via temporal cross-correlation over all time points to the EEG alertness index for all subjects, over a set of lags from −10 to 10 TRs (Figure 5). This process was repeated using only the top percentage of negative-valued voxels in the template following the same sequence of thresholds, as well as again with the top percentages of positive-valued voxels.

General linear model analysis of auditory-stimulus-evoked responses

To test whether explicitly modeling alertness fluctuations could improve the statistical inference of task-evoked activation, we performed a standard GLM analysis on the auditory stimulus-evoked responses. We tested three models: the first model contained only the task stimuli (where trials with and without responses were separated into two regressors), the second model included task regressors along with the EEG alertness index, and the third model contained the task regressors and the fMRI alertness index as regressors. For each model, we calculated the beta map for the regressor corresponding to trials that had responses and applied a voxelwise one-sample t-test across scans (Figure 6).

Impact of low-frequency physiological signals on fMRI alertness prediction

Given that changes in alertness are accompanied by changes in systemic physiological processes such as breathing and HR (Özbay et al., 2019; Yuan et al., 2013), we also asked how the prediction of EEG alertness from fMRI would be impacted if low-frequency respiration and cardiac effects were first removed from the fMRI data. We hypothesized that if changes in breathing and HR strongly co-vary with alertness, removing these components from fMRI data might adversely impact the inference of alertness from fMRI.

To examine this question, we re-constructed the template map and the calculated fMRI alertness index after regressing out RV and HR from all fMRI data. Specifically, following procedures in Chen et al., 2020; Chang and Glover, 2009, an RV time course was calculated as the standard deviation in a 6 s sliding window of the respiration belt waveform, and an HR time course was calculated as the (inverse of the) mean inter-beat-interval from the PPG waveform, again in a 6 s window around each TR. The resulting signals were zero-meaned and convolved with respiration (Birn et al., 2008) and cardiac (Chang et al., 2009) response functions, respectively, along with their temporal and dispersion derivatives, and band-pass filtered from 0.01 to 0.2 Hz. The basis set was then regressed out of the fMRI signal at each voxel.

To quantify the impact on fMRI alertness estimation, the maximum temporal cross-correlation was calculated between the fMRI alertness index and the EEG alertness index before and after physiological removal (Figure 4—figure supplement 3A). To assess the degree to which the fMRI alertness index is modulated with behavioral response after this physiological regression step, we performed the same statistical analysis on the ‘fast hit’, ‘slow hit’, and ‘miss’ response types (Figure 4—figure supplement 3B). The correspondence between the fMRI alertness index and the EEG alertness index was reduced in most subjects following physiological signal removal. Nonetheless, the fMRI alertness index exhibited similar trial-wise modulation as the original data.

Lasso-based fMRI alertness index

While our primary analyses used a spatial template-based prediction approach, we also examine an alternate, temporal (lasso) regression model. Just as with the template method, here we train the lasso model on the resting-state fMRI scans and evaluate the performance on the task scans. Two models were considered: one that uses the time courses of all fMRI voxels as input to the model and another that uses ROI time courses extracted from 268 regions of interest (Shen et al., 2013) as an initial dimensionality reduction step. For each approach, the resting-state scans were first temporally concatenated across subjects. We then fit a lasso model to this concatenated resting-state time series, using 10-fold cross-validation to choose the regularization parameter (lambda). We then applied the resulting lasso coefficients to the task data, to estimate an alertness time course in each task scan (Figure 2—figure supplement 2).

Statistics and software

All statistical tests are two tailed. Details of the statistical methods are provided in the sections ‘Relationship between EEG/fMRI alertness and behavioral responses’ and ‘General linear model analysis of auditory-stimulus-evoked responses’ above. All analyses were performed in MATLAB (R2018b).

Acknowledgements

This work was supported by NIH grants K22 ES028048 (CC), R01 NS112252 (DJE, CC), R01 NS108445 and R01 NS110130 (VLM), and an NSF Graduate Research Fellowship (SEG). This work was also supported in part by the Intramural Research Program of NINDS, NIH.

Funding Statement

The funders had no role in study design, data collection and interpretation, or the decision to submit the work for publication.

Contributor Information

Sarah E Goodale, Email: sarah.e.goodale@vanderbilt.edu.

Catie Chang, Email: catie.chang@vanderbilt.edu.

Sanmi Koyejo, University of Illinois at Urbana-Champaign, United States.

Christian Büchel, University Medical Center Hamburg-Eppendorf, Germany.

Funding Information

This paper was supported by the following grants:

  • National Institutes of Health K22 ES028048 to Catie Chang.

  • National Institutes of Health R01 NS112252 to Dario J Englot, Catie Chang.

  • National Institutes of Health R01 NS108445 to Victoria L Morgan.

  • National Institutes of Health R01 NS110130 to Victoria L Morgan.

  • National Science Foundation Graduate Research Fellowship to Sarah E Goodale.

Additional information

Competing interests

No competing interests declared.

Author contributions

Conceptualization, Formal analysis, Investigation, Methodology, Writing - original draft, Writing - review and editing.

Software, Investigation, Writing - review and editing.

Software, Investigation, Writing - review and editing.

Software, Investigation, Methodology, Writing - review and editing.

Investigation, Writing - review and editing.

Methodology, Writing - review and editing.

Funding acquisition, Methodology, Writing - review and editing.

Funding acquisition, Methodology, Writing - review and editing.

Funding acquisition, Methodology, Writing - review and editing.

Conceptualization, Formal analysis, Supervision, Funding acquisition, Investigation, Methodology, Writing - original draft, Project administration, Writing - review and editing.

Ethics

Human subjects: All subjects provided written informed consent, and procedures were approved by the Institutional Review Boards of the National Institutes of Health (Protocol 00-N-0082) and Vanderbilt University (IRB #181540).

Additional files

Supplementary file 1. Scan details.

‘x’ indicates whether a subject contributed an eyes-closed resting state (ECR) scan and/or an eyes-closed auditory task (ECT) scan. Some subjects did not complete both ECR and ECT scans.

elife-62376-supp1.xlsx (8.9KB, xlsx)
Supplementary file 2. Data subsets for cross-validation of template reproducibility.

Resting- state scans and task scans were broken down into two non-overlapping subgroups so that one subject’s rest and task scans did not cross into a different group.

elife-62376-supp2.xlsx (9.3KB, xlsx)
Supplementary file 3. Maximum frame-to-frame displacement for each subject’s rest and task scans, as calculated during the motion coregistration pre-processing step.
elife-62376-supp3.xlsx (9.4KB, xlsx)
Transparent reporting form

Data availability

Data files supporting the main findings are available on Open Science Framework, and code is available at https://github.com/neurdylab/fMRIAlertnessDetection (copy archived at https://archive.softwareheritage.org/swh:1:rev:db5f3ef49e53585412627984c44b9db9b188a868).

The following dataset was generated:

Goodale SE, Ahmed N, Zhao C, de Zwart JA, Özbay PS, Picchioni D, Duyn JH, Englot DJ, Morgan VL, Chang C. 2020. fMRI prediction of alertness and behavioral variability: data files. Open Science Framework.

References

  1. Allen PJ, Josephs O, Turner R. A method for removing imaging artifact from continuous EEG recorded during functional MRI. NeuroImage. 2000;12:230–239. doi: 10.1006/nimg.2000.0599. [DOI] [PubMed] [Google Scholar]
  2. Altmann A, Schröter MS, Spoormaker VI, Kiem SA, Jordan D, Ilg R, Bullmore ET, Greicius MD, Czisch M, Sämann PG. Validation of non-REM sleep stage decoding from resting state fMRI using linear support vector machines. NeuroImage. 2016;125:544–555. doi: 10.1016/j.neuroimage.2015.09.072. [DOI] [PubMed] [Google Scholar]
  3. American Academy of Sleep Medicine The AASM Manual for the Scoring of Sleep and Associated Event. American academy of sleep medicine, Westchester, IL, 2007.2007. [Google Scholar]
  4. Birn RM, Diamond JB, Smith MA, Bandettini PA. Separating respiratory-variation-related fluctuations from neuronal-activity-related fluctuations in fMRI. NeuroImage. 2006;31:1536–1548. doi: 10.1016/j.neuroimage.2006.02.048. [DOI] [PubMed] [Google Scholar]
  5. Birn RM, Smith MA, Jones TB, Bandettini PA. The respiration response function: the temporal dynamics of fMRI signal fluctuations related to changes in respiration. NeuroImage. 2008;40:644–654. doi: 10.1016/j.neuroimage.2007.11.059. [DOI] [PMC free article] [PubMed] [Google Scholar]
  6. Boly M, Balteau E, Schnakers C, Degueldre C, Moonen G, Luxen A, Phillips C, Peigneux P, Maquet P, Laureys S. Baseline brain activity fluctuations predict somatosensory perception in humans. PNAS. 2007;104:12187–12192. doi: 10.1073/pnas.0611404104. [DOI] [PMC free article] [PubMed] [Google Scholar]
  7. Calhoun VD, Miller R, Pearlson G, Adalı T. The chronnectome: time-varying connectivity networks as the next frontier in fMRI data discovery. Neuron. 2014;84:262–274. doi: 10.1016/j.neuron.2014.10.015. [DOI] [PMC free article] [PubMed] [Google Scholar]
  8. Chang C, Cunningham JP, Glover GH. Influence of heart rate on the BOLD signal: the cardiac response function. NeuroImage. 2009;44:857–869. doi: 10.1016/j.neuroimage.2008.09.029. [DOI] [PMC free article] [PubMed] [Google Scholar]
  9. Chang LJ, Gianaros PJ, Manuck SB, Krishnan A, Wager TD. A sensitive and specific neural signature for Picture-Induced negative affect. PLOS Biology. 2015;13:e1002180. doi: 10.1371/journal.pbio.1002180. [DOI] [PMC free article] [PubMed] [Google Scholar]
  10. Chang C, Leopold DA, Schölvinck ML, Mandelkow H, Picchioni D, Liu X, Ye FQ, Turchi JN, Duyn JH. Tracking brain arousal fluctuations with fMRI. PNAS. 2016;113:4518–4523. doi: 10.1073/pnas.1520613113. [DOI] [PMC free article] [PubMed] [Google Scholar]
  11. Chang C, Özbay C, de Zwart C, Picchioni C, Chappel-Farley C, Mandelkow C, Duyn C. Covariation of pulse oximetry amplitude and BOLD fMRI across vigilance states. Proc. Intl. Soc. Mag. Reson. Med.2018. [Google Scholar]
  12. Chang C, Glover GH. Effects of model-based physiological noise correction on default mode network anti-correlations and correlations. NeuroImage. 2009;47:1448–1459. doi: 10.1016/j.neuroimage.2009.05.012. [DOI] [PMC free article] [PubMed] [Google Scholar]
  13. Chen JE, Lewis LD, Chang C, Tian Q, Fultz NE, Ohringer NA, Rosen BR, Polimeni JR. Resting-state "physiological networks". NeuroImage. 2020;213:116707. doi: 10.1016/j.neuroimage.2020.116707. [DOI] [PMC free article] [PubMed] [Google Scholar]
  14. Cohen JR. The behavioral and cognitive relevance of time-varying, dynamic changes in functional connectivity. NeuroImage. 2018;180:515–525. doi: 10.1016/j.neuroimage.2017.09.036. [DOI] [PMC free article] [PubMed] [Google Scholar]
  15. Cohen-Zion M, Ancoli-Israel S. Sleep in children with attention-deficit hyperactivity disorder (ADHD): a review of naturalistic and stimulant intervention studies. Sleep Medicine Reviews. 2004;8:379–402. doi: 10.1016/j.smrv.2004.06.002. [DOI] [PubMed] [Google Scholar]
  16. Coste CP, Sadaghiani S, Friston KJ, Kleinschmidt A. Ongoing brain activity fluctuations directly account for intertrial and indirectly for intersubject variability in stroop task performance. Cerebral Cortex. 2011;21:2612–2619. doi: 10.1093/cercor/bhr050. [DOI] [PubMed] [Google Scholar]
  17. Crottaz-Herbette S, Menon V. Where and when the anterior cingulate cortex modulates attentional response: combined fMRI and ERP evidence. Journal of Cognitive Neuroscience. 2006;18:766–780. doi: 10.1162/jocn.2006.18.5.766. [DOI] [PubMed] [Google Scholar]
  18. de Munck JC, Gonçalves SI, Faes TJ, Kuijer JP, Pouwels PJ, Heethaar RM, Lopes da Silva FH. A study of the brain's resting state based on alpha band power, heart rate and fMRI. NeuroImage. 2008;42:112–121. doi: 10.1016/j.neuroimage.2008.04.244. [DOI] [PubMed] [Google Scholar]
  19. Dinstein I, Heeger DJ, Behrmann M. Neural variability: friend or foe? Trends in Cognitive Sciences. 2015;19:322–328. doi: 10.1016/j.tics.2015.04.005. [DOI] [PubMed] [Google Scholar]
  20. Duyn JH, Ozbay PS, Chang C, Picchioni D. Physiological changes in sleep that affect fMRI inference. Current Opinion in Behavioral Sciences. 2020;33:42–50. doi: 10.1016/j.cobeha.2019.12.007. [DOI] [PMC free article] [PubMed] [Google Scholar]
  21. Elizabeth D, Salo T. ME-ICA/tedana: 0.0.9a. 0.0.9aZenodo. 2020 doi: 10.5281/zenodo.3786890. [DOI]
  22. Englot DJ, Mishra AM, Mansuripur PK, Herman P, Hyder F, Blumenfeld H. Remote effects of focal hippocampal seizures on the rat neocortex. Journal of Neuroscience. 2008;28:9066–9081. doi: 10.1523/JNEUROSCI.2014-08.2008. [DOI] [PMC free article] [PubMed] [Google Scholar]
  23. Falahpour M, Chang C, Wong CW, Liu TT. Template-based prediction of vigilance fluctuations in resting-state fMRI. NeuroImage. 2018;174:317–327. doi: 10.1016/j.neuroimage.2018.03.012. [DOI] [PMC free article] [PubMed] [Google Scholar]
  24. Feige B, Scheffler K, Esposito F, Di Salle F, Hennig J, Seifritz E. Cortical and subcortical correlates of electroencephalographic alpha rhythm modulation. Journal of Neurophysiology. 2005;93:2864–2872. doi: 10.1152/jn.00721.2004. [DOI] [PubMed] [Google Scholar]
  25. Foucher JR, Otzenberger H, Gounot D. Where arousal meets attention: a simultaneous fMRI and EEG recording study. NeuroImage. 2004;22:688–697. doi: 10.1016/j.neuroimage.2004.01.048. [DOI] [PubMed] [Google Scholar]
  26. Fox MD, Snyder AZ, Zacks JM, Raichle ME. Coherent spontaneous activity accounts for trial-to-trial variability in human evoked brain responses. Nature Neuroscience. 2006;9:23–25. doi: 10.1038/nn1616. [DOI] [PubMed] [Google Scholar]
  27. Fultz NE, Bonmassar G, Setsompop K, Stickgold RA, Rosen BR, Polimeni JR, Lewis LD. Coupled electrophysiological, Hemodynamic, and cerebrospinal fluid oscillations in human sleep. Science. 2019;366:628–631. doi: 10.1126/science.aax5440. [DOI] [PMC free article] [PubMed] [Google Scholar]
  28. Glover GH, Li TQ, Ress D. Image-based method for retrospective correction of physiological motion effects in fMRI: retroicor. Magnetic Resonance in Medicine. 2000;44:162–167. doi: 10.1002/1522-2594(200007)44:1&#x0003c;162::AID-MRM23&#x0003e;3.0.CO;2-E. [DOI] [PubMed] [Google Scholar]
  29. Goldman RI, Stern JM, Engel J, Cohen MS. Simultaneous EEG and fMRI of the alpha rhythm. NeuroReport. 2002;13:2487–2492. doi: 10.1097/00001756-200212200-00022. [DOI] [PMC free article] [PubMed] [Google Scholar]
  30. Hegerl U, Sander C, Hensch T. Systems Neuroscience in Depression. Academic Press; 2016. [DOI] [Google Scholar]
  31. Hesselmann G, Kell CA, Eger E, Kleinschmidt A. Spontaneous local variations in ongoing neural activity Bias perceptual decisions. PNAS. 2008a;105:10984–10989. doi: 10.1073/pnas.0712043105. [DOI] [PMC free article] [PubMed] [Google Scholar]
  32. Hesselmann G, Kell CA, Kleinschmidt A. Ongoing activity fluctuations in hMT+ Bias the perception of coherent visual motion. Journal of Neuroscience. 2008b;28:14481–14485. doi: 10.1523/JNEUROSCI.4398-08.2008. [DOI] [PMC free article] [PubMed] [Google Scholar]
  33. Horovitz SG, Fukunaga M, de Zwart JA, van Gelderen P, Fulton SC, Balkin TJ, Duyn JH. Low frequency BOLD fluctuations during resting wakefulness and light sleep: a simultaneous EEG-fMRI study. Human Brain Mapping. 2008;29:671–682. doi: 10.1002/hbm.20428. [DOI] [PMC free article] [PubMed] [Google Scholar]
  34. Jawinski P, Kirsten H, Sander C, Spada J, Ulke C, Huang J, Burkhardt R, Scholz M, Hensch T, Hegerl U. Human brain arousal in the resting state: a genome-wide association study. Molecular Psychiatry. 2019;24:1599–1609. doi: 10.1038/s41380-018-0052-2. [DOI] [PubMed] [Google Scholar]
  35. Jobert M, Schulz H, Jähnig P, Tismer C, Bes F, Escola H. A computerized method for detecting episodes of wakefulness during sleep based on the alpha slow-wave index (ASI) Sleep. 1994;17:37–46. doi: 10.1093/sleep/17.1.37. [DOI] [PubMed] [Google Scholar]
  36. Jones BE. From waking to sleeping: neuronal and chemical substrates. Trends in Pharmacological Sciences. 2005;26:578–586. doi: 10.1016/j.tips.2005.09.009. [DOI] [PubMed] [Google Scholar]
  37. Kelly AM, Uddin LQ, Biswal BB, Castellanos FX, Milham MP. Competition between functional brain networks mediates behavioral variability. NeuroImage. 2008;39:527–537. doi: 10.1016/j.neuroimage.2007.08.008. [DOI] [PubMed] [Google Scholar]
  38. Kiviniemi V, Wang X, Korhonen V, Keinänen T, Tuovinen T, Autio J, LeVan P, Keilholz S, Zang YF, Hennig J, Nedergaard M. Ultra-fast magnetic resonance encephalography of physiological brain activity - Glymphatic pulsation mechanisms? Journal of Cerebral Blood Flow & Metabolism. 2016;36:1033–1045. doi: 10.1177/0271678X15622047. [DOI] [PMC free article] [PubMed] [Google Scholar]
  39. Klimesch W. EEG alpha and theta oscillations reflect cognitive and memory performance: a review and analysis. Brain Research Reviews. 1999;29:169–195. doi: 10.1016/S0165-0173(98)00056-3. [DOI] [PubMed] [Google Scholar]
  40. Kucyi A, Esterman M, Riley CS, Valera EM. Spontaneous default network activity reflects behavioral variability independent of mind-wandering. PNAS. 2016;113:13899–13904. doi: 10.1073/pnas.1611743113. [DOI] [PMC free article] [PubMed] [Google Scholar]
  41. Kundu P, Inati SJ, Evans JW, Luh WM, Bandettini PA. Differentiating BOLD and non-BOLD signals in fMRI time series using multi-echo EPI. NeuroImage. 2012;60:1759–1770. doi: 10.1016/j.neuroimage.2011.12.028. [DOI] [PMC free article] [PubMed] [Google Scholar]
  42. Kundu P, Brenowitz ND, Voon V, Worbe Y, Vértes PE, Inati SJ, Saad ZS, Bandettini PA, Bullmore ET. Integrated strategy for improving functional connectivity mapping using multiecho fMRI. PNAS. 2013;110:16187–16192. doi: 10.1073/pnas.1301725110. [DOI] [PMC free article] [PubMed] [Google Scholar]
  43. Laufs H, Holt JL, Elfont R, Krams M, Paul JS, Krakow K, Kleinschmidt A. Where the BOLD signal Goes when alpha EEG leaves. NeuroImage. 2006;31:1408–1418. doi: 10.1016/j.neuroimage.2006.02.002. [DOI] [PubMed] [Google Scholar]
  44. Lee TH, Baek J, Lu ZL, Mather M. How arousal modulates the visual contrast sensitivity function. Emotion. 2014;14:978–984. doi: 10.1037/a0037047. [DOI] [PMC free article] [PubMed] [Google Scholar]
  45. Lim J, Dinges DF. Sleep deprivation and vigilant attention. Annals of the New York Academy of Sciences. 2008;1129:305–322. doi: 10.1196/annals.1417.002. [DOI] [PubMed] [Google Scholar]
  46. Liu Z, de Zwart JA, Yao B, van Gelderen P, Kuo LW, Duyn JH. Finding thalamic BOLD correlates to posterior alpha EEG. NeuroImage. 2012;63:1060–1069. doi: 10.1016/j.neuroimage.2012.08.025. [DOI] [PMC free article] [PubMed] [Google Scholar]
  47. Liu TT. Noise contributions to the fMRI signal: an overview. NeuroImage. 2016;143:141–151. doi: 10.1016/j.neuroimage.2016.09.008. [DOI] [PubMed] [Google Scholar]
  48. Liu TT, Falahpour M. Vigilance effects in Resting-State fMRI. Frontiers in Neuroscience. 2020;14:321. doi: 10.3389/fnins.2020.00321. [DOI] [PMC free article] [PubMed] [Google Scholar]
  49. Makeig S, Jung TP, Sejnowski TJ. Awareness during drowsiness: dynamics and electrophysiological correlates. Canadian Journal of Experimental Psychology/Revue Canadienne De Psychologie Expérimentale. 2000;54:266–273. doi: 10.1037/h0087346. [DOI] [PubMed] [Google Scholar]
  50. Mather M, Sutherland MR. Arousal-Biased competition in perception and memory. Perspectives on Psychological Science. 2011;6:114–133. doi: 10.1177/1745691611400234. [DOI] [PMC free article] [PubMed] [Google Scholar]
  51. McAvoy MP, Tagliazucchi E, Laufs H, Raichle ME. Human non-REM sleep and the mean global BOLD signal. Journal of Cerebral Blood Flow & Metabolism. 2019;39:2210–2222. doi: 10.1177/0271678X18791070. [DOI] [PMC free article] [PubMed] [Google Scholar]
  52. Menon V, Uddin LQ. Saliency, switching, attention and control: a network model of insula function. Brain Structure and Function. 2010;214:655–667. doi: 10.1007/s00429-010-0262-0. [DOI] [PMC free article] [PubMed] [Google Scholar]
  53. Moehlman TM, de Zwart JA, Chappel-Farley MG, Liu X, McClain IB, Chang C, Mandelkow H, Özbay PS, Johnson NL, Bieber RE, Fernandez KA, King KA, Zalewski CK, Brewer CC, van Gelderen P, Duyn JH, Picchioni D. All-night functional magnetic resonance imaging sleep studies. Journal of Neuroscience Methods. 2019;316:83–98. doi: 10.1016/j.jneumeth.2018.09.019. [DOI] [PMC free article] [PubMed] [Google Scholar]
  54. Moosmann M, Ritter P, Krastel I, Brink A, Thees S, Blankenburg F, Taskin B, Obrig H, Villringer A. Correlates of alpha rhythm in functional magnetic resonance imaging and near infrared spectroscopy. NeuroImage. 2003;20:145–158. doi: 10.1016/S1053-8119(03)00344-6. [DOI] [PubMed] [Google Scholar]
  55. Motelow JE, Li W, Zhan Q, Mishra AM, Sachdev RN, Liu G, Gummadavelli A, Zayyad Z, Lee HS, Chu V, Andrews JP, Englot DJ, Herman P, Sanganahalli BG, Hyder F, Blumenfeld H. Decreased subcortical cholinergic arousal in focal seizures. Neuron. 2015;85:561–572. doi: 10.1016/j.neuron.2014.12.058. [DOI] [PMC free article] [PubMed] [Google Scholar]
  56. Nashiro K, Mather M. Effects of emotional arousal on memory binding in normal aging and Alzheimer's disease. The American Journal of Psychology. 2011;124:301–312. doi: 10.5406/amerjpsyc.124.3.0301. [DOI] [PMC free article] [PubMed] [Google Scholar]
  57. Nassar MR, Rumsey KM, Wilson RC, Parikh K, Heasly B, Gold JI. Rational regulation of learning dynamics by pupil-linked arousal systems. Nature Neuroscience. 2012;15:1040–1046. doi: 10.1038/nn.3130. [DOI] [PMC free article] [PubMed] [Google Scholar]
  58. Nomi JS, Bolt TS, Ezie CEC, Uddin LQ, Heller AS. Moment-to-Moment BOLD signal variability reflects regional changes in neural flexibility across the lifespan. The Journal of Neuroscience. 2017;37:5539–5548. doi: 10.1523/JNEUROSCI.3408-16.2017. [DOI] [PMC free article] [PubMed] [Google Scholar]
  59. Ogilvie RD. The process of falling asleep. Sleep Medicine Reviews. 2001;5:247–270. doi: 10.1053/smrv.2001.0145. [DOI] [PubMed] [Google Scholar]
  60. Oken BS, Salinsky MC, Elsas SM. Vigilance, alertness, or sustained attention: physiological basis and measurement. Clinical Neurophysiology. 2006;117:1885–1901. doi: 10.1016/j.clinph.2006.01.017. [DOI] [PMC free article] [PubMed] [Google Scholar]
  61. Olbrich S, Mulert C, Karch S, Trenner M, Leicht G, Pogarell O, Hegerl U. EEG-vigilance and BOLD effect during simultaneous EEG/fMRI measurement. NeuroImage. 2009;45:319–332. doi: 10.1016/j.neuroimage.2008.11.014. [DOI] [PubMed] [Google Scholar]
  62. Ong JL, Kong D, Chia TT, Tandi J, Thomas Yeo BT, Chee MW. Co-activated yet disconnected-Neural correlates of eye closures when trying to stay awake. NeuroImage. 2015;118:553–562. doi: 10.1016/j.neuroimage.2015.03.085. [DOI] [PubMed] [Google Scholar]
  63. Özbay PS, Chang C, Picchioni D, Mandelkow H, Moehlman TM, Chappel-Farley MG, van Gelderen P, de Zwart JA, Duyn JH. Contribution of systemic vascular effects to fMRI activity in white matter. NeuroImage. 2018;176:541–549. doi: 10.1016/j.neuroimage.2018.04.045. [DOI] [PMC free article] [PubMed] [Google Scholar]
  64. Özbay PS, Chang C, Picchioni D, Mandelkow H, Chappel-Farley MG, van Gelderen P, de Zwart JA, Duyn J. Sympathetic activity contributes to the fMRI signal. Communications Biology. 2019;2:421. doi: 10.1038/s42003-019-0659-0. [DOI] [PMC free article] [PubMed] [Google Scholar]
  65. Podvalny E, Flounders MW, King LE, Holroyd T, He BJ. A dual role of prestimulus spontaneous neural activity in visual object recognition. Nature Communications. 2019;10:3910. doi: 10.1038/s41467-019-11877-4. [DOI] [PMC free article] [PubMed] [Google Scholar]
  66. Poudel GR, Innes CR, Bones PJ, Watts R, Jones RD. Losing the struggle to stay awake: divergent thalamic and cortical activity during microsleeps. Human Brain Mapping. 2014;35:257–269. doi: 10.1002/hbm.22178. [DOI] [PMC free article] [PubMed] [Google Scholar]
  67. Poudel GR, Innes CRH, Jones RD. Temporal evolution of neural activity and connectivity during microsleeps when rested and following sleep restriction. NeuroImage. 2018;174:263–273. doi: 10.1016/j.neuroimage.2018.03.031. [DOI] [PubMed] [Google Scholar]
  68. Rosenberg MD, Finn ES, Scheinost D, Papademetris X, Shen X, Constable RT, Chun MM. A neuromarker of sustained attention from whole-brain functional connectivity. Nature Neuroscience. 2016;19:165–171. doi: 10.1038/nn.4179. [DOI] [PMC free article] [PubMed] [Google Scholar]
  69. Sadaghiani S, Hesselmann G, Kleinschmidt A. Distributed and antagonistic contributions of ongoing activity fluctuations to auditory stimulus detection. Journal of Neuroscience. 2009;29:13410–13417. doi: 10.1523/JNEUROSCI.2592-09.2009. [DOI] [PMC free article] [PubMed] [Google Scholar]
  70. Sadaghiani S, Poline J-B, Kleinschmidt A, D’Esposito M. Ongoing dynamics in large-scale functional connectivity predict perception. PNAS. 2015;112:8463–8468. doi: 10.1073/pnas.1420687112. [DOI] [PMC free article] [PubMed] [Google Scholar]
  71. Salomon R, Bleich-Cohen M, Hahamy-Dubossarsky A, Dinstien I, Weizman R, Poyurovsky M, Kupchik M, Kotler M, Hendler T, Malach R. Global functional connectivity deficits in schizophrenia depend on behavioral state. Journal of Neuroscience. 2011;31:12972–12981. doi: 10.1523/JNEUROSCI.2987-11.2011. [DOI] [PMC free article] [PubMed] [Google Scholar]
  72. Saper CB, Scammell TE, Lu J. Hypothalamic regulation of sleep and circadian rhythms. Nature. 2005;437:1257–1263. doi: 10.1038/nature04284. [DOI] [PubMed] [Google Scholar]
  73. Shen X, Tokoglu F, Papademetris X, Constable RT. Groupwise whole-brain parcellation from resting-state fMRI data for network node identification. NeuroImage. 2013;82:403–415. doi: 10.1016/j.neuroimage.2013.05.081. [DOI] [PMC free article] [PubMed] [Google Scholar]
  74. Sherman SM, Guillery RW. The role of the thalamus in the flow of information to the cortex. Philosophical Transactions of the Royal Society of London. Series B: Biological Sciences. 2002;357:1695–1708. doi: 10.1098/rstb.2002.1161. [DOI] [PMC free article] [PubMed] [Google Scholar]
  75. Shmueli K, van Gelderen P, de Zwart JA, Horovitz SG, Fukunaga M, Jansma JM, Duyn JH. Low-frequency fluctuations in the cardiac rate as a source of variance in the resting-state fMRI BOLD signal. NeuroImage. 2007;38:306–320. doi: 10.1016/j.neuroimage.2007.07.037. [DOI] [PMC free article] [PubMed] [Google Scholar]
  76. Tagliazucchi E, von Wegner F, Morzelewski A, Brodbeck V, Laufs H. Dynamic BOLD functional connectivity in humans and its electrophysiological correlates. Frontiers in Human Neuroscience. 2012;6:339. doi: 10.3389/fnhum.2012.00339. [DOI] [PMC free article] [PubMed] [Google Scholar]
  77. Tagliazucchi E, Laufs H. Decoding wakefulness levels from typical fMRI resting-state data reveals reliable drifts between wakefulness and sleep. Neuron. 2014;82:695–708. doi: 10.1016/j.neuron.2014.03.020. [DOI] [PubMed] [Google Scholar]
  78. Thompson GJ, Magnuson ME, Merritt MD, Schwarb H, Pan WJ, McKinley A, Tripp LD, Schumacher EH, Keilholz SD. Short-time windows of correlation between large-scale functional brain networks predict vigilance intraindividually and interindividually. Human Brain Mapping. 2013;34:3280–3298. doi: 10.1002/hbm.22140. [DOI] [PMC free article] [PubMed] [Google Scholar]
  79. van Kempen J, Loughnane GM, Newman DP, Kelly SP, Thiele A, O'Connell RG, Bellgrove MA. Behavioural and neural signatures of perceptual decision-making are modulated by pupil-linked arousal. eLife. 2019;8:e42541. doi: 10.7554/eLife.42541. [DOI] [PMC free article] [PubMed] [Google Scholar]
  80. Wang C, Ong JL, Patanaik A, Zhou J, Chee MW. Spontaneous eyelid closures link vigilance fluctuation with fMRI dynamic connectivity states. PNAS. 2016;113:9653–9658. doi: 10.1073/pnas.1523980113. [DOI] [PMC free article] [PubMed] [Google Scholar]
  81. Wong CW, Olafsson V, Tal O, Liu TT. The amplitude of the resting-state fMRI global signal is related to EEG vigilance measures. NeuroImage. 2013;83:983–990. doi: 10.1016/j.neuroimage.2013.07.057. [DOI] [PMC free article] [PubMed] [Google Scholar]
  82. Yang GJ, Murray JD, Repovs G, Cole MW, Savic A, Glasser MF, Pittenger C, Krystal JH, Wang X-J, Pearlson GD, Glahn DC, Anticevic A. Altered global brain signal in schizophrenia. PNAS. 2014;111:7438–7443. doi: 10.1073/pnas.1405289111. [DOI] [PMC free article] [PubMed] [Google Scholar]
  83. Yuan H, Zotev V, Phillips R, Bodurka J. Correlated slow fluctuations in respiration, EEG, and BOLD fMRI. NeuroImage. 2013;79:81–93. doi: 10.1016/j.neuroimage.2013.04.068. [DOI] [PubMed] [Google Scholar]

Decision letter

Editor: Sanmi Koyejo1

In the interests of transparency, eLife publishes the most substantive revision requests and the accompanying author responses.

Acceptance summary:

This manuscript evaluates and extends a method for estimating levels of alertness in fMRI. The primary contributions include alertness prediction using a subset of voxels and the alertness measure's usage as a regressor in task fMRI. Reducing the required spatial map from the whole brain to a thresholded set of voxels is an interesting methodological contribution. Further, results suggest that this approach can help address alertness as a possible confound in task fMRI analyses.

Decision letter after peer review:

Thank you for submitting your article "fMRI-based detection of alertness predicts behavioral response variability" for consideration by eLife. Your article has been reviewed by 3 peer reviewers, one of whom is a member of our Board of Reviewing Editors, and the evaluation has been overseen by Christian Büchel as the Senior Editor. The reviewers have opted to remain anonymous.

The reviewers have discussed the reviews with one another and the Reviewing Editor has drafted this decision to help you prepare a revised submission.

As the editors have judged that your manuscript is of interest, but as described below that additional experiments are required before it is published, we would like to draw your attention to changes in our revision policy that we have made in response to COVID-19 (https://elifesciences.org/articles/57162). First, because many researchers have temporarily lost access to the labs, we will give authors as much time as they need to submit revised manuscripts. We are also offering, if you choose, to post the manuscript to bioRxiv (if it is not already there) along with this decision letter and a formal designation that the manuscript is "in revision at eLife". Please let us know if you would like to pursue this option. (If your work is more suitable for medRxiv, you will need to post the preprint yourself, as the mechanisms for us to do so are still in development.)

Summary:

This manuscript evaluates multimodal fMRI-EEG data to estimate and assess attention in the fMRI signal based on its correlation with the EEG signal. The study of attention using fMRI is a well-studied topic, including somatosensory, visual, and auditory paradigms. Still, this study adds to the literature by using multimodal data to create the fMRI model. The discussed method follows prior work on the same approach and explores using a subset of voxels and applications as an fMRI signal regressor.

Essential revisions:

1. Conceptually, improved methods for addressing alertness confounds are a valuable contribution. The evaluation of the alertness index via the auditory task is a useful contribution. In terms of specific contributions, the reduction in the required spatial map, from the whole brain to a thresholded set of voxels, is an interesting methodological contribution. However, perhaps the main methodological contribution can be achieved more effectively using a direct approach, e.g., using a lasso model to predict the EEG alertness score using the fMRI time series. The lasso is considered a relatively standard statistical model, so it is not clear that the slight reduction in complexity is worth the loss in performance. At the very least, the authors should compare this approach to the presented thresholded correlation.

2. Reproducibility should be described more clearly. Reviewers recommend using cross-validation, dividing the data into 2 sets. The models can be built separately on each set, then compared. Also, one set's model can be used to evaluate prediction in the other set, and thus calculate sensitivity and specificity for this model. This would be complementary to the permutation testing already performed, as permutation testing is somewhat limited because these signals appear highly periodic (e.g., Figure 2) and thus can likely mismatch easily.

3. For the rsfMRI, participants were instructed to stay awake. Was this evaluated? There is a concern that participants might fall asleep during the 24.5m duration of the scan.

4. The auditory fMRI task made use of different tones. How were these tones delivered? Where earbuds used?

5. In the processing, it is stated that motion coregistration has been performed. Was there a quality check for excessive movement? When a participant moves more than the voxel size, the data are often discarded.

6. While testing hit/miss is clear, testing reaction times are limited to faster or slower than a single time (565 ms). As the study describes this method's use on an individual basis, consider intraindividual methods previously published, e.g., those from Thompson et al. 2013 (Citation 19), which includes separation by individual means, and also a separation of individuals into fast and slow groups.

7. A reaction time threshold of 565ms was used, but it remains unclear what the reason is for splitting the trails in an equal number of fast and slow trails. A reaction time of 565ms is still fast, and there can be a lot of variation between the fast and slow trails. A non-response was recorded if there was no button press within 5s. Therefore, perhaps fast can also be stated as below 2.5s and slow above 2.5s. The reference to Lim and Dinges (2008) indeed used a threshold at 0.5s but in a different setting.

8. A sample of 14 participants was investigated in this study. Would this sample be sufficient to prove the point of predicting fMRI results? There is no apriori sample size calculation, and therefore, it remains unknown if the data are sufficient to prove the point. In fMRI, usually larger samples up to 80 and more used to reduce variability/noise in the measurement.

9. The authors seem to indicate that the alertness metric could be used in resting-state fMRI data. Can't this be tested on either the authors' data or a publically available database? All that is needed is data with an index of alertness (such as eye-tracking) or even self-report of alertness/sleepiness.

10. Figure 5B shows positive (red) correspondence in a ventricle. This makes one wonder if this method is picking up on the EEG-linked CSF fluctuations reported elsewhere, e.g., Kiviniemi, V. et al. (2016). J Cereb Blood Flow Metab 36(6): 1033-1045. Can the authors discuss this?

11. What software was used for the processing of the data? Is the code available online? Please share via GitHub. Also, it remains unclear which statistical software was used for the statistical analyses.

[Editors' note: further revisions were suggested prior to acceptance, as described below.]

Thank you for submitting your article "fMRI-based detection of alertness predicts behavioral response variability" for consideration by eLife. Your article has been overseen by a Reviewing Editor and Christian Büchel as the Senior Editor.

The Reviewing Editor has drafted this to help you prepare a revised submission.

Essential revisions:

The reviewing editor believes that this manuscript is much improved and has addressed the conceptual and scientific concerns raised in the first round of reviews. While concerns about the sample size remain, the reviewing editor believes that this manuscript can be accepted as the contributions are primarily methodological.

The reviewing editor requests that the authors address the sample size limitations more directly in the main manuscript. In addition to the justification given in the response, consider adding language to the manuscript that clarifies that the findings may not generalize to the population from which they have sampled. Authors should also consider mentioning that their small sample likely overestimates the effect size of their results.

eLife. 2021 May 7;10:e62376. doi: 10.7554/eLife.62376.sa2

Author response

Essential revisions:

1. Conceptually, improved methods for addressing alertness confounds are a valuable contribution. The evaluation of the alertness index via the auditory task is a useful contribution. In terms of specific contributions, the reduction in the required spatial map, from the whole brain to a thresholded set of voxels, is an interesting methodological contribution. However, perhaps the main methodological contribution can be achieved more effectively using a direct approach, e.g., using a lasso model to predict the EEG alertness score using the fMRI time series. The lasso is considered a relatively standard statistical model, so it is not clear that the slight reduction in complexity is worth the loss in performance. At the very least, the authors should compare this approach to the presented thresholded correlation.

The reviewers bring up a very interesting suggestion. To examine this point, we now compare the template-based approach to a lasso model. Just as we had done with the template method, here we train the lasso model on the resting-state fMRI scans and evaluate the performance on the task scans. For the lasso, we tested two models: one using a 268-region atlas (Shen et al., 2013) to parcellate the fMRI data into ROI time series as input to the model, and one using the fMRI time series from all voxels. For each approach, we trained a lasso model on the resting state data (temporally concatenated across subjects), using 10-fold cross-validation to choose the regularization parameter λ. We then applied the resulting coefficients (β weights) to the task fMRI data, to estimate an alertness time course for each task scan.

Figure 2—figure supplement 2 shows the correlation between the lasso-estimated fMRI arousal index and the EEG alertness index in each of the task scans. For comparison, correlations resulting from the fMRI alertness index estimated from the template-based approach are also shown. This figure shows that although our template method performed slightly better, the lasso model gave similar predictions, especially when using the 268-region atlas ROIs as input. This finding supports the notion that vigilance can be robustly estimated from fMRI data, and indicates that there may be multiple approaches for successfully extracting this information. Moreover, the initial feature reduction using a 268-region atlas seemed to improve the prediction beyond using a voxelwise lasso model. We believe that further investigating alternate linear models for vigilance prediction is an interesting area that we plan to develop in future work, including the use of other regularization methods such as ridge regression and elastic net, as well as alternate atlases for ROI definition. We now include these results as supplementary material, and have added the following to the Results (p.16), Discussion (p.22), and Methods (p. 33):

Results pg 16: “Finally, to investigate how this template-based method compares against a more conventional temporal regression approach, we also constructed lasso regression models (see Methods for further details). […] Further, the lasso model that was trained on fMRI time series from a 268-region atlas (Shen et al., 2013) was superior to that which used the time series from all voxels as input.”

Discussion p. 22: “Further, in this study we only considered a template-based approach and a lasso regression model to predict alertness. […] Further investigation of alternate linear models for vigilance prediction would be an interesting area to develop in future work, including the use of other regularization methods as well as parcellation atlases for deriving input time series.”

Methods p. 33: “While our primary analyses used a spatial template-based prediction approach, we also examine an alternate, temporal (lasso) regression model. Just as with the template method, here we train the lasso model on the resting-state fMRI scans, and evaluate the performance on the task scans. […] We then applied the resulting lasso coefficients to the task data, to estimate an alertness time course in each task scan.”

2. Reproducibility should be described more clearly. Reviewers recommend using cross-validation, dividing the data into 2 sets. The models can be built separately on each set, then compared. Also, one set's model can be used to evaluate prediction in the other set, and thus calculate sensitivity and specificity for this model. This would be complementary to the permutation testing already performed, as permutation testing is somewhat limited because these signals appear highly periodic (e.g., Figure 2) and thus can likely mismatch easily.

We agree that we could more clearly show the reproducibility of our results. First, we’d like to clarify that our permutation testing was carried out by shuffling across-scans, rather than shifting signals within scans, to avoid issues of periodicity. I.e., the EEG signals (entire time series) were randomly mismatched with the fMRI data of the other subjects. We have now clarified this in the manuscript on p.28. We would also like to highlight that our original analysis used a hold-out validation scheme, in which we trained the model (derived the template) from the set of resting-state scans and applied it to the task scans. However, although disjoint sets of scans were used for training and testing, some of the same subjects contributed both a resting-state and a task scan, as detailed in our original Supplementary Table (Supplementary File 1). To further probe the reliability and across-subject generalizability of our approach, we have now divided the subjects into two separate groups, and examine the performance when training/testing on all 4 possible combinations. As shown below, we find that these various train-test splits have similar performance. We have added this to our Results and Methods sections on p. 7 and p. 29-30, respectively:

Results: “To probe the across-subject reproducibility of fMRI alertness prediction within our dataset, we performed an additional cross-validation analysis in which subjects were divided into two distinct groups. Model training was performed on the resting-state data from one group, and evaluated on the task data of the other group. Cross-correlations between the estimated resting-state fMRI alertness index and the measured EEG alertness index were comparable to the original results (Figure 2—figure supplement 1).”

Methods: “To further investigate the reproducibility of fMRI alertness prediction, subjects were divided into two non-overlapping subgroups for both resting and task conditions (Supplementary File 2). […] Further, to test whether a model trained on task data can be used to predict vigilance in resting state, we also performed the reverse, constructing our template this time with the task scans from each subject group, and evaluating the fMRI alertness index on the held-out resting-state scans (Figure 2—figure supplement 3).”

3. For the rsfMRI, participants were instructed to stay awake. Was this evaluated? There is a concern that participants might fall asleep during the 24.5m duration of the scan.

Although the instruction was for subjects to stay awake, the long scans were designed to elicit variability in vigilance state – particularly, reduced wakefulness and drifts between wakefulness and light sleep – in order to estimate the fMRI pattern that tracks alertness fluctuations as well as evaluate its use for detecting alertness in situations during which the subject may be drowsy or lightly sleeping during a task. However, the point raised by the reviewers is important for several reasons: (i) characterizing sleep stage would give a clearer picture of the data from which our present results are generated, and (ii) it is an open question whether a template map derived from data that includes N1 or N2 sleep is just as effective as one derived from the wakeful but drowsy state.

Half of our resting state and task data have been independently sleep scored by two certified sleep techs at the Vanderbilt University Medical Center. Standard sleep scoring was carried out in 30-second epochs, according to standard AASM criteria. These ratings indicated that, across all of the staged scans combined, 85.6% of epochs were rated as ‘awake’, suggesting that subjects were mostly alert. On pg. 24 in Methods, we now include:

“Although the instruction was for subjects to stay awake, the long scans were designed to elicit variability in vigilance state – particularly, reduced wakefulness and drifts between wakefulness and light sleep. […] It is therefore likely state sleep was not a major factor in our experiments”.

4. The auditory fMRI task made use of different tones. How were these tones delivered? Where earbuds used?

We have added this information into our Methods (p. 24): “Auditory tones were delivered via earbuds placed over subjects’ ear plugs.”

5. In the processing, it is stated that motion coregistration has been performed. Was there a quality check for excessive movement? When a participant moves more than the voxel size, the data are often discarded.

We agree it is important to consider head motion. We first note that our data is collected via multi-echo acquisition, and the multi-echo ICA denoising approach described in Kundu et al. (2012, 2013) was applied. This approach has been demonstrated to be very effective in reducing head motion effects (Kundu et al., PNAS 2013) by isolating BOLD from non-BOLD components based on the echo-time dependence of fMRI percent signal change. However, we have now also performed quality checks for excessive movement. For each scan, we have calculated the maximum frame-to-frame displacement, finding that only two subjects had a maximum displacement that exceeded our voxel size (3 mm), see Supplementary File 3. We additionally find that removing these subjects had no effect on the main results, (Figure 4—figure supplement 4). This has now been added into the Methods section (p.25) and Results (p. 12):

p. 25: “Maximum frame-to-frame displacement was also calculated, as shown in Supplementary File 3.”

p. 12: “Finally, while multi-echo ICA has proven effective in reducing head-motion effects (Kundu et al., PNAS 2013), we also ensured that scans with frame-to-frame head motion exceeding our voxel size (3 mm; corresponding to 1 resting-state and 1 task scan; Supplementary File 3) did not impact the major findings of this study. Removing these high-motion scans from the analysis had minimal impact on the correspondence between the fMRI alertness index with EEG and behavior (Figure 4—figure supplement 4).”

6. While testing hit/miss is clear, testing reaction times are limited to faster or slower than a single time (565 ms). As the study describes this method's use on an individual basis, consider intraindividual methods previously published, e.g., those from Thompson et al. 2013 (Citation 19), which includes separation by individual means, and also a separation of individuals into fast and slow groups.

We thank the reviewer for this suggestion. We have updated our explanation of our threshold selection in our methods which was found by taking the median of all trials. Our use of 565ms as the threshold came from the median across all pooled reaction times. However, we agree that it is also interesting to look at intra-individual thresholds. So now, for each scan, we also took the fastest and slowest 10% of trials (as has been used in psychomotor vigilance tasks; Loh et al., 2004), and report those results in Figure 4—figure supplement 1.

The paired t-test was still significant. One difference in the bar plots (compared to Figure 4B-C) is that the ‘slow’ (yellow) alertness index is closer to that of the ‘miss’ (blue). This would make sense, given that this analysis considers the slowest 10% of trials; this pattern is also evident in the EEG, lending further confidence to our result. We also considered a between-subjects split (i.e., assigning entire subjects into a ‘fast’ or ‘slow’ group based on their median RT), but failed to find a significant separation (p>0.05, two-sample t-test). One potential reason is because the variance of reaction times within some subjects were quite large.

We have now described the new findings in Results p.9-10 and Methods p.31 of the revised manuscript.

7. A reaction time threshold of 565ms was used, but it remains unclear what the reason is for splitting the trails in an equal number of fast and slow trails. A reaction time of 565ms is still fast, and there can be a lot of variation between the fast and slow trails. A non-response was recorded if there was no button press within 5s. Therefore, perhaps fast can also be stated as below 2.5s and slow above 2.5s. The reference to Lim and Dinges (2008) indeed used a threshold at 0.5s but in a different setting.

We’d like to reference essential revision 6 for the first part of this comment. We have considered that there are alternate ways to find the threshold and have now also performed an intra-individual separation into fast and slow trials. With regard to the 2.5s threshold, we have now also considered this, but find that across all subjects and trials, there was only one reaction time slower than 2.5s (histogram of reaction times Figure 4—figure supplement 5; now described in the Methods on p. 31). Thus, it appears that our alertness index can discriminate between slow/fast reaction times even amongst a set of relatively fast reaction times, supporting its sensitivity as a marker of underlying brain state.

8. A sample of 14 participants was investigated in this study. Would this sample be sufficient to prove the point of predicting fMRI results? There is no apriori sample size calculation, and therefore, it remains unknown if the data are sufficient to prove the point. In fMRI, usually larger samples up to 80 and more used to reduce variability/noise in the measurement.

We acknowledge the reviewers’ concern about sample size, and have now pointed this out as a limitation in the Discussion. Nonetheless, there are several factors that increase our confidence in the present results. First, our scans are long (24.5 min per scan) and 310 total trials were available for mining behavioral responses. Second, the additional analyses suggested by the reviewers (see Comment 2 above) also support the generalization ability of the approach. Third, we also note that our subject sample size is on par with other multi-modal (EEG-fMRI) studies that have examined vigilance effects in fMRI and, importantly, whose results have since been replicated in independent studies. These include Falahpour et al., 2018 (10 subjects, used to investigate the same template-based vigilance prediction on resting-state fMRI; similar effects are replicated in our current study), and Olbrich et al., 2009 (15 subjects, used to evaluate fMRI activity associated with vigilance stage changes [with similar effects replicated in Ong et al., 2015]). Similarly, Hesselmann et al. 2008 used 12 subjects to study the ability to predict behavior from pre-trial fMRI activity, and Poudel et al. 2014 used a subset of 14 subjects to map activity associated with microsleeps. Regardless, we do agree that our subject size is a limitation and we are looking to expand on this for future studies. We have added this into the Discussion on pg. 22:

“Another limitation of this study is the relatively small sample size, and future work will be carried out with this approach using larger sample sizes as well as other experimental conditions. Nonetheless, there are several factors that increase our confidence in the present results: our scans are long (24.5 min per scan), and the replication analyses carried out in Figure 2—figure supplement 1 and Figure 4—figure supplements 1 and 4 further support the generalizability of this approach.”

9. The authors seem to indicate that the alertness metric could be used in resting-state fMRI data. Can't this be tested on either the authors' data or a publically available database? All that is needed is data with an index of alertness (such as eye-tracking) or even self-report of alertness/sleepiness.

We have now tried to clarify in this paper that this alertness metric has been successfully applied to resting-state data in previous work (Falahpour et al., 2018 in humans; Chang et al., 2016 in macaques). The main goal of the present study was to investigate the ability of this alertness metric to predict behavioral response variability throughout a task, and at a fine temporal resolution (i.e., looking in a 5s pre-stimulus window before the trial). In doing so, our resting-state scans served as a “training set” from which we created the template, and all evaluation was applied to the held-out (task) scans. However, as the reviewers point out, we can also examine the performance on resting-state fMRI using the current dataset. We have now provided an additional analysis in which we construct the template with the task scans and evaluate the alertness prediction on the held-out resting-state scans (see reply to Comment 2 above, and Figure 2—figure supplement 3). In addition, another recent study (Gu et al., 2020) used the same approach independently to calculate vigilance in resting-state Human Connectome Project data, and found it agreed well with self-reported sleepiness levels. We have added this to pg. 22:

“Further, although the present study centered on predicting task-related behavioral responses, we also performed an analysis in which the template was constructed from the task scans, and the fMRI alertness index was evaluated on the held-out resting-state scans, obtaining comparable results (Figure 2—figure supplement 3).”

10. Figure 5B shows positive (red) correspondence in a ventricle. This makes one wonder if this method is picking up on the EEG-linked CSF fluctuations reported elsewhere, e.g., Kiviniemi, V. et al. (2016). J Cereb Blood Flow Metab 36(6): 1033-1045. Can the authors discuss this?

Thank you for this comment, we agree it would be great to discuss this point. In experiments with widespread BOLD changes, we have often seen an antipolar change in or near CSF, and in deep white matter (especially periventricularly). One potential mechanism is that these changes in CSF signals are at least in part due to changes in CBV, where CBV increases result in CSF volume decreases or vice versa (e.g., Ozbay, P.S. Neuroimage 2018, 176: 541). This is expected in peri-venous CSF spaces that are abundant in peri-ventricular white matter, within the ventricles near choroid plexus. In these regions, signal changes associated with CSF volume changes may substantially precede (and outweigh) BOLD effects. It has been reported that conditions that lead to vasoaction also change the EEG baseline shifts (Nita D.A. J Neurophysiol 2004, 92:1011), and this has also been suggested in the work of Kiviniemi et al. One difference between our study and the work of Kiviniemi et al. is that the latter used DC EEG data, whereas our fMRI arousal template was based on EEG band limited power, which effectively discarded the DC signal. Therefore, we believe our template may be picking up other mechanisms, but future work is necessary to examine this in more detail.

We have added a brief discussion of this possibility on p. 19.

“A plausable mechanistic explanation offered recently is that widespread fMRI signal changes generated by either neuronal or autonomic activity lead to antagonistic fMRI signal changes in the downstream venous vasculature, in part due to a disjoint timing and amplitude relationship of blood volume and blood oxygenation effects56. Further work will be necessary to investigate potential links between these observations and those of studies which have observed CSF fMRI signals in other EEG frequencies (Kiviniemi et al., 2016). ”

11. What software was used for the processing of the data? Is the code available online? Please share via GitHub. Also, it remains unclear which statistical software was used for the statistical analyses.

We have updated our methods to specify the software for processing and statistical analyses. All analyses were performed in MATLAB (R2018b). This has been updated on p. 34. The code is now available via GitHub, and a link is now provided in the Data Availability field in the submission site.

References:

1. Shen X, Tokoglu F, Papademetris X, Constable RT. Groupwise whole-brain parcellation from resting-state fMRI data for network node identification. Neuroimage. 2013 Nov 15;82:403-15. doi: 10.1016/j.neuroimage.2013.05.081.

2. American Academy of Sleep Medicine. (American Academy of Sleep Medicine, Westchester, IL, 2007).

3. Kundu, P., Inati, S. J., Evans, J. W., Luh, W.-M. and Bandettini, P. A. Differentiating BOLD and non-BOLD signals in fMRI time series using multi-echo EPI. Neuroimage 60, 1759-1770, doi:10.1016/j.neuroimage.2011.12.028 (2012).

4. Kundu, P. et al. Integrated strategy for improving functional connectivity mapping using multiecho fMRI. Proc. Natl. Acad. Sci. U. S. A. 110, 16187-16192, doi:10.1073/pnas.1301725110 (2013).

5. Thompson, G. J. et al. Short-time windows of correlation between large-scale functional brain networks predict vigilance intraindividually and interindividually. Human Brain Mapping 34, 3280-3298, doi:10.1002/hbm.22140 (2013).

6. Loh, Sylvia, et al. "The validity of psychomotor vigilance tasks of less than 10-minute duration." Behavior Research Methods, Instruments, and Computers 36.2 (2004): 339-346.

7. Falahpour, M., Chang, C., Wong, C. W. and Liu, T. T. Template-based prediction of vigilance fluctuations in resting-state fMRI. Neuroimage 174, 317-327, doi:10.1016/j.neuroimage.2018.03.012 (2018).

8. Olbrich, S. et al. EEG-vigilance and BOLD effect during simultaneous EEG/fMRI measurement. Neuroimage 45, 319-332, doi:10.1016/j.neuroimage.2008.11.014 (2009).

9. Ong, Ju Lynn, et al. "Co-activated yet disconnected—Neural correlates of eye closures when trying to stay awake." Neuroimage 118 (2015): 553-562.

10. Hesselmann, G., Kell, C. A., Eger, E. and Kleinschmidt, A. Spontaneous local variations in ongoing neural activity bias perceptual decisions. Proc. Natl. Acad. Sci. U. S. A. 105, 10984-10989, doi:10.1073/pnas.0712043105 (2008).

11. Poudel, G. R., Innes, C. R. H., Bones, P. J., Watts, R. and Jones, R. D. Losing the struggle to stay awake: divergent thalamic and cortical activity during microsleeps. Hum. Brain Mapp. 35, 257-269, doi:10.1002/hbm.22178 (2014).

12. Chang, C. et al. Tracking brain arousal fluctuations with fMRI. Proc. Natl. Acad. Sci. U. S. A. 113, 4518-4523, doi:10.1073/pnas.1520613113 (2016).

13. Gu, Y., Han, F., Sainburg, L. E., and Liu, X. (2019). Transient arousal modulations are responsible for resting-state functional connectivity changes associated with head motion. bioRxiv, 444463.

14. Kiviniemi, V., Wang, X., Korhonen, V., Keinänen, T., Tuovinen, T., Autio, J.,.… and Nedergaard, M. (2016). Ultra-fast magnetic resonance encephalography of physiological brain activity–glymphatic pulsation mechanisms?. Journal of Cerebral Blood Flow and Metabolism, 36(6), 1033-1045.

[Editors' note: further revisions were suggested prior to acceptance, as described below.]

Essential revisions:

The reviewing editor believes that this manuscript is much improved and has addressed the conceptual and scientific concerns raised in the first round of reviews. While concerns about the sample size remain, the reviewing editor believes that this manuscript can be accepted as the contributions are primarily methodological.

The reviewing editor requests that the authors address the sample size limitations more directly in the main manuscript. In addition to the justification given in the response, consider adding language to the manuscript that clarifies that the findings may not generalize to the population from which they have sampled. Authors should also consider mentioning that their small sample likely overestimates the effect size of their results.

We agree that it would be helpful to address this point more directly in the manuscript. We now mention both points suggested by the Reviewing Editor: (1) that findings may not generalize to the population; (2) the small sample size may have led to overestimation of effect size. This material is now included on p. 22 of the revised submission:

“One key limitation of this study is the small sample size. As a result, the effect sizes of the current results may be overestimated, and findings may not generalize well to the population from which they are sampled. Future work should be carried out with this approach using larger sample sizes as well as other experimental conditions.”

Additional note: During the last review period, we had also come across 3 relevant references that we also wish to cite. The added references are listed below, and have been cited on p.3 and 19:

4. Makeig, S., Jung, T. P. and Sejnowski, T. J. Awareness during drowsiness: dynamics and electrophysiological correlates. Can J Exp Psychol 54, 266-273 (2000).

5. Poudel, G. R., Innes, C. R. H. and Jones, R. D. Temporal evolution of neural activity and connectivity during microsleeps when rested and following sleep restriction. Neuroimage 174, 263-273, doi:10.1016/j.neuroimage.2018.03.031 (2018).

57. McAvoy, M. P., Tagliazucchi, E., Laufs, H. and Raichle, M. E. Human non-REM sleep and the mean global BOLD signal. J Cereb Blood Flow Metab 39, 2210-2222, doi:10.1177/0271678X18791070 (2019).

Associated Data

    This section collects any data citations, data availability statements, or supplementary materials included in this article.

    Data Citations

    1. Goodale SE, Ahmed N, Zhao C, de Zwart JA, Özbay PS, Picchioni D, Duyn JH, Englot DJ, Morgan VL, Chang C. 2020. fMRI prediction of alertness and behavioral variability: data files. Open Science Framework. [DOI]

    Supplementary Materials

    Supplementary file 1. Scan details.

    ‘x’ indicates whether a subject contributed an eyes-closed resting state (ECR) scan and/or an eyes-closed auditory task (ECT) scan. Some subjects did not complete both ECR and ECT scans.

    elife-62376-supp1.xlsx (8.9KB, xlsx)
    Supplementary file 2. Data subsets for cross-validation of template reproducibility.

    Resting- state scans and task scans were broken down into two non-overlapping subgroups so that one subject’s rest and task scans did not cross into a different group.

    elife-62376-supp2.xlsx (9.3KB, xlsx)
    Supplementary file 3. Maximum frame-to-frame displacement for each subject’s rest and task scans, as calculated during the motion coregistration pre-processing step.
    elife-62376-supp3.xlsx (9.4KB, xlsx)
    Transparent reporting form
    elife-62376-transrepform.pdf (302.7KB, pdf)

    Data Availability Statement

    Data files supporting the main findings are available on Open Science Framework, and code is available at https://github.com/neurdylab/fMRIAlertnessDetection (copy archived at https://archive.softwareheritage.org/swh:1:rev:db5f3ef49e53585412627984c44b9db9b188a868).

    The following dataset was generated:

    Goodale SE, Ahmed N, Zhao C, de Zwart JA, Özbay PS, Picchioni D, Duyn JH, Englot DJ, Morgan VL, Chang C. 2020. fMRI prediction of alertness and behavioral variability: data files. Open Science Framework.

    Articles from eLife are provided here courtesy of eLife Sciences Publications, Ltd

    RESOURCES