A Functional Connectivity-Based Neuromarker of Sustained Attention Generalizes to Predict Recall in a Reading Task

Abstract

Sustaining attention to the task at hand is a crucial part of everyday life, from following a lecture at school to maintaining focus while driving. Lapses in sustained attention are frequent and often problematic, with conditions such as attention deficit hyperactivity disorder affecting millions of people worldwide. Recent work has had some success in finding signatures of sustained attention in whole-brain functional connectivity (FC) measures during basic tasks, but since FC can be dynamic and task-dependent, it remains unclear how fully these signatures would generalize to a more complex and naturalistic scenario. To this end, we used a previously defined whole-brain FC network – a marker of attention that was derived from a sustained attention task – to predict the ability of participants to recall material during a free-viewing reading task. Though the predictive network was trained on a different task and set of participants, the strength of FC in the sustained attention network predicted reading recall significantly better than permutation tests where behavior was scrambled to simulate chance performance. To test the generalization of the method used to derive the sustained attention network, we applied the same method to our reading task data to find a new FC network whose strength specifically predicts reading recall. Even though the sustained attention network provided significant prediction of recall, the reading network was more predictive of recall accuracy. The new reading network’s spatial distribution indicates that reading recall is highest when temporal pole regions have higher FC with left occipital regions and lower FC with bilateral supramarginal gyrus. Right cerebellar to right frontal connectivity is also indicative of poor reading recall. We examine these and other differences between the two predictive FC networks, providing new insight into the task-dependent nature of FC-based performance metrics.

1. Introduction

Sustained attention, or the ability to maintain focus on something for an extended period, is an important and pervasive piece of daily life. Friends must sustain attention to a conversation, students must sustain attention to an exam, and drivers must sustain attention to the road. If a person in these situations becomes distracted by either internal or external sources, this attentional lapse could result in consequences ranging from a poor grade to a car accident. The problematic nature of sustained attentional lapses is central to disorders of attention such as attention deficit hyperactivity disorder (ADHD), which by some measures has affected 11.8% of American children ages 12-17 (Pastor et al. 2015). In addition, fluctuations in sustained attention will modulate performance on any psychological task, accounting for some of the inter-trial and inter-participant variability observed in both behavioral and neurological measures (Weissman et al. 2006).

In recent years, neuroimaging research has increasingly sought to understand how sustained attention is maintained and what happens when it lapses. To gain further insight into this process, researchers have looked for a neural signature of sustained attention that could indicate how individuals are likely to perform on assessments of vigilance and attention. Recent work has found signatures of sustained attention, mind-wandering, and daydreaming in the multi-voxel patterns in task-related visual regions (deBettencourt et al. 2015), the functional connectivity (FC) in the default-mode network (Kucyi and Davis 2014), and participants’ moment-to-moment pupil dilation (Grandchamp, Braboszcz, and Delorme 2014). Cross-modal signatures have even been combined to detect lapses in sustained attention-based behavior on a single-trial basis (Mittner et al. 2014). But these studies, like many investigations of sustained attention, have focused on time-resolved, within-participant markers of attentional lapses.

Other studies have found neural signatures that predict how well a given individual will perform on a sustained attention task. One promising result comes from Rosenberg, Finn and colleagues (Rosenberg, Finn, et al. 2016), who identified whole-brain, widely distributed networks whose FC during the gradual-onset continuous performance task (GradCPT) (Esterman et al. 2013) predicted a participant’s relative task performance. Increased FC in their “positive” network indicated that a participant would perform better, and increased FC in their “negative” network indicated that a participant would perform worse. These networks generalized well to predict ADHD symptom ratings in an independent dataset from a participant’s resting-state FC (Rosenberg, Finn, et al. 2016). Healthy adults given a single dose of methylphenidate (Ritalin) showed greater FC in the positive network and lower FC in the negative network than unmedicated controls (Rosenberg, Zhang, et al. 2016). One might expect that these networks would generalize across naturalistic tasks as they have across conditions, but this idea has yet to be tested empirically.

The sustained attention studies above have relied on pared-down vigilance or go/no-go tasks such as the GradCPT or sustained attention to response task (SART). The extent to which FC signatures predicting performance on one of these tasks will generalize to more naturalistic scenarios remains unclear, largely because FC is dynamic and task-dependent (Gonzalez-Castillo et al. 2015; Shirer et al. 2012) and the task-dependent components of FC may interact with the components that are used to predict inter-individual differences. The level of generalization may determine how closely linked the sustained attention signatures are to real-life symptoms, and how well they can inform future diagnostics and interventions.

The (Rosenberg, Finn, et al. 2016) sustained attention study may also generalize at the methods level. If the same method used to derive the sustained attention network can be applied to naturalistic task data to improve prediction over the sustained attention network itself, it would suggest that attention is only one aspect of task prediction, and future FC-based diagnostics and interventions would be more accurate if they include factors more directly measured from the naturalistic scenario of interest. If, on the other hand, the naturalistic task data produced a similar network or provided no improvement in prediction, it would suggest that other components of performance are not encoded in the functional connectome the way that sustained attention is.

To this end, we measured the functional connectivity in the sustained attention networks from Rosenberg, Finn et al. (2016) in data from a naturalistic reading task. We evaluated the ability of the attention network connectivity to predict participants’ reading recall performance. To explore generalization at the methods level, we also applied the method from (Rosenberg, Finn, et al. 2016) to the naturalistic reading dataset to derive a network specifically chosen to predict reading recall. Finally, to explore the generalizability of new networks trained like this one, we examined the spatial differences between these informative FC signatures for clues into the cognitive functions and task-specific factors driving their performance. The results show that a participant’s performance on each task is strongly related to FC in widespread networks, and that the spatial distributions of these networks are informed by the specific cognitive demands of the tasks.

2. Materials and Methods

2.1. Participants

Twenty-eight healthy participants (11 male, 1 left-handed, 1 ambidextrous, ages 20-42, mean ± std age 25.4 ± 4.6) took part in the study. Informed consent was obtained from each participant under protocol 99-M-0170 approved by the Combined Neuroscience Institutional Review Board at the National Institutes of Health.

2.2. Experimental Paradigm

Visual and auditory stimuli were presented to participants using PsychoPy software (Peirce 2008). Participants were presented with the transcript of lectures from Yale’s CLCV 205: Introduction to Ancient Greek History (http://oyc.yale.edu/classics/clcv-205). This subject matter was chosen because understanding it would not require background knowledge, but participants were unlikely to have learned it before. Excerpts from Lectures 2, 5, 7, and 10 were chosen because they presented information that did not draw heavily on other lectures or common knowledge.

Participants were presented with 10 “pages” of practice reading from the class’ introductory lecture during the structural MRI scan before the main task began. Each “page” consisted of 9 lines of black text (3-40, median 38, characters per line, Courier New font, character size 0.33×0.33 degrees) displayed on a gray screen that subtended a visual angle of approximately 14×11 visual degrees. This practice session was designed to help the participants become comfortable with the display and response box. There was no time limit on each page, and the material from the practice reading was not tested.

Participants completed 2-6 runs (median 5) of tested reading, each consisting of two 15-page blocks. Two participants were removed from further analysis because they completed fewer than 4 runs; the final cohort of participants completed 4-6 runs (median 5). Before each page, the participant was asked to fixate on a cross in the upper-left-hand corner of the screen, which was displayed for 2 seconds (this was designed to standardize eye movements and correct for eye tracker drift). A page was then displayed for 14 seconds (or until the participant advanced the page by pressing a button). When only 2 seconds remained, the text started to fade to signal that time was almost up. Participants were allowed to move their eyes freely across the page, but they were asked to advance the page when they finished rather than read the page a second time.

As they read, participants were presented with auditory distractions in the form of irrelevant speech: the same professor giving an unrelated lecture. On each page, the sound was randomly selected (with replacement and with equal probability) to be either speech or noise (phase-scrambled audio from one of the lectures). A cue preceded each 15-page block, indicating whether the participant should ignore the speech and attend to the reading alone, or attend to both the reading and speech at the same time. The first cue was presented before the scan, and the participant verbally confirmed that they understood. When the scan was started, a fixation cross was displayed for 8 s to allow the scanner to reach steady-state. The cue between blocks was 6 s long. A run contained one block of each type, and the order was counter-balanced across runs. Each run lasted 4.8-8.1 minutes (median 7.4). Auditory distraction information will be used in future distraction-focused analyses, but for the purposes of the current analysis, the distractions and conditions were ignored and all pages were treated equally. Supplementary Figure S1 investigates the potential impacts of these conditions on our findings (see Results section for more details).

Lecture 2 was always presented as reading to give a common behavioral reference point, and the material for the remaining text, attended speech, and ignored speech were selected randomly (without replacement) from the other lectures. Each page of reading or clip of speech resumed where the previous one left off until the lecture excerpt ended; then a new lecture excerpt started.

After each run, participants were presented with multiple-choice recall questions about what they read and heard. These were similar to what teachers would call “reading comprehension” questions, but we will use the term “recall” to denote that performance requires both an understanding of the information presented and the ability to retain that information and recall it at the end of a run. The questions tested important but briefly covered material, not overall themes. There was approximately one question for every three pages of reading. The question order was randomized, and participants could answer at their own pace. At the end of the scanning session, participants were asked whether they had any prior experience with the topic that helped them answer the questions. One participant said that they had, and this participant was removed from the analysis. All participants performed significantly above the chance level of 25% correct (binomial test, p<0.005).

A visual depiction of the task can be seen in Figure 1. Transcripts of the lecture excerpts and questions are available in the supplementary materials.

FIGURE 1.

A single run of the reading task. Two visual or audio stimuli arranged vertically indicates that one of these options was chosen at random. At the start of each half-block, the participant was cued about whether to attend the speech in addition to the reading. One cue (attend reading only or attend both speech and reading) was presented before the scan. After reading 15 pages, the other cue was presented and the participant read the next 15 pages. Each page lasted 14 s unless the participant pushed a button to advance. On each page, speech or phase-scrambled noise was selected randomly and played. After each run, the participant was asked 10-15 recall questions on the reading and speech from that run. Questions were self-paced.

2.3. fMRI Acquisition

fMRI data were acquired using a General Electric (GE) 750x scanner (Waukesha, WI). The scanner’s body coil was used for RF transmission. A 32-channel GE receive-only head coil was used for signal reception. A T₁-weigted anatomical image and a proton density-weighted anatomical image were acquired for each participant using an MP-RAGE sequence (192 Axial slices per volume, slice thickness = 1mm, FOV = 256×256 mm, image matrix = 256×256). This was followed by a high-order shim and an ASSET calibration. Task data were acquired using a multi-echo echo-planar imaging (EPI) sequence with echo times at 14.6, 26.8, and 39.0 ms (TR = 2 s, voxel size = 3×3×3.5 mm, image matrix = 72×72, FOV = 21.6×21.6 cm, slice thickness = 3.5mm, slice spacing = 0.0mm, flip angle = 77 deg, 37 oblique slices, 252 acquisitions per run, ASSET acceleration = 3).

2.4. Ocular Data Acquisition

Infrared images of the eye were recorded using an MRI-compatible camera (Avotec Real Eye Model RE-5701), and eye tracking and pupillometry data were sampled at 60 Hz using a SensoMotoric Instruments iViewX tracker (Teltow, Germany).

2.5. fMRI Pre-Processing

Structural and functional MRI data were pre-processed using the AFNI software package (Cox 1996). The T1-weighted anatomical image scan was bias-corrected by dividing by the PD image, then the skull was removed. The following pre-processing steps were applied to the EPI data: (1) discarding of the first 3 TRs to wait for steady-state to be achieved, (2) censoring TRs in which >10% of voxels are considered “outliers” because they deviate greatly from their detrended timecourse (using AFNI command 3dToutcount), (3) de-spiking to replace values deviating extremely from their voxel’s detrended timecourses with less extreme values (AFNI program 3dDespike), (4) slice timing correction (AFNI program 3dTshift), (5) head motion estimation and linear transformation to MNI space using the bias-corrected (T1-weighted / PD) anatomical scan (using AFNI programs align_epi_anat, 3dvolreg, and @auto_tlrc). In this step, the second echo of each TR was registered to the second echo of the first TR in the run. This first TR was registered to the bias-corrected anatomical scan, and the anatomical scan was registered to an MNI atlas. The registrations were combined into a single transform for each TR and applied to all 3 echoes in that TR.

Next, multi-echo independent component analysis (MEICA) was applied using the ME-ICA software package (version 2.5, beta 11, https://bitbucket.org/prantikk/me-ica) (Kundu et al. 2012). This de-noising technique decomposes the signal into independent components, accepting or rejecting components based on their pattern of signal amplitude decay across echoes. While a BOLD source’s percent signal change increases linearly with echo time, many noise sources have no dependence on TE. This distinction can be used to separate BOLD-like sources from noise-like sources. MEICA has been shown to improve resting-state network detection (Kundu et al. 2012), remove slow drifts in resting-state fMRI (Evans et al. 2015), and increase effect size estimates in task-based fMRI (Gonzalez-Castillo et al. 2016; Lombardo et al. 2016).

The MEICA algorithm acted on our data as follows. It first computed a voxel-wise linear weighted combination of time series for all available echoes (with weights based on voxel-wise estimates of T₂*). This step resulted in a new single time series per voxel with maximized contrast-to-noise ratio (Poser et al. 2006), which was then used as input to an Independent Component Analysis (ICA) module that decomposed the data into a series of spatial components and their associated time series. ICA is commonly used in single-echo fMRI datasets to find both networks of interest (e.g., DMN, attention network) and artifacts. On single-echo data, the selection of artifactual components for later removal must be done manually. When multi-echo data is available, well-established differences in echo time dependence profiles of artifactual (e.g., no dependence) and non-artifactual (e.g., linear dependence) ICA components can be used to automatically detect and remove those artifactual components. It is this ICA component characterization, labeling and removal of artifactual components that constitute the last step of the MEICA algorithm. Additional details on the MEICA algorithm can be found here (Gonzalez-Castillo et al. 2016; Kundu et al. 2012; Olafsson et al. 2015). MEICA removed a median 23.5% (interquartile range 17%) of the components from each run, accounting for a median 78.0% (interquartile range 13%) of the variance. Much of this variance is attributable to the contributions of scanner drift and motion-related artifacts that would be removed in a standard pre-processing pipeline: a control analysis that regressed out motion and 4^th-order polynomials from single-echo, pre-MEICA data removed a median 56.2% (interquartile range 14.1%) of the variance.

We then performed voxel-wise intensity normalization and scaled to percentage units. Finally, we ran a regression to remove covariates of no interest including 6-parameter motion time courses and their derivatives from each run, band-pass filtering from 0.01-0.1 Hz, and the mean signal from white matter and CSF. TRs with motion over 0.2 mm were censored. 6 participants who had more than 15% of their TRs censored were removed from further analysis.

2.6. Whole-Brain Functional Connectivity Calculations

Functional connectivity was analyzed using custom scripts in AFNI and MATLAB and visualized using the CONN toolbox (v17, https://www.nitrc.org/projects/conn) (Whitfield-Gabrieli and Nieto-Castanon 2012) and SUMA (Saad et al. 2004). FC was calculated between ROIs or “nodes” derived using the 268-ROI Shen atlas (Shen et al. 2013), a parcellation that includes cortical, subcortical, and cerebellar structures. The FC between a pair of nodes will be called an “edge.” The first principal component of the voxel-wise activity in each node was calculated (using AFNI program 3dmaskSVD), and its time course was used as a measure of activity in that node. Using the first principal component is akin to taking an average across the ROI, but it reduces the sensitivity of the node’s timecourse to outlier or noisy voxels (Gonzalez-Castillo et al. 2015; James et al. 2013). Pearson correlation coefficients (r) were calculated between the time courses of each node pair (across the reading period of each run) and normalized using Fisher’s r-to-z′ transformation. The mean z′ value across all runs for a given participant was used as the edge strength, or FC between node pairs. The resulting FC matrices were used to evaluate the strength of FC in the GradCPT and Visual/Auditory Language networks examined below, and to identify the “Reading networks” whose FC strength would predict reading recall performance.

2.7. GradCPT and Vis/Aud Language Networks

The GradCPT positive and negative networks are the network masks derived from the GradCPT dataset in (Rosenberg, Finn, et al. 2016). In the GradCPT, participants viewed a slowly changing series of city or mountain scenes. They were instructed to press a button when they saw the (more common) city scenes and withhold a press during the (less common) mountain scenes. The GradCPT networks were trained to predict the performance metric d’, which is a function of a participant’s hit rate and false alarm rate during the task (d’ = Z(hit rate) – Z(false alarm rate), where function Z(p) is the inverse Gaussian cumulative distribution function). Each reading participant’s GradCPT network strength was found by calculating their mean FC in the edges of the GradCPT network masks (edges in the negative mask were first multiplied by -1). The process used to derive the GradCPT network is the same as that used to derive the Reading network, described in the next section and in Supplementary Figure S2.

To compare results with another potential metric of task performance, we found another pair of networks associated with visual and auditory language from the NeuroSynth database (Poldrack et al. 2012; Yarkoni et al. 2011). These will be referred to as the Vis/Aud Language networks. When visual and auditory stimuli are presented simultaneously, the attended modality has higher activity in its sensory cortex (Johnson and Zatorre 2005; Johnson and Zatorre 2006). So we might expect higher activation in (and FC between) visual language regions and lower activation/FC in auditory language regions to indicate a stronger shift in attention towards the reading and away from the auditory distractions. To find regions associated with these topics, we used NeuroSynth’s “reverse inference maps,” which find locations preferentially activated by one topic over others. The maps contain voxels where the likelihood that a topic’s terms are used given the presence of activation (P(Topic|Activation)) is high (exceeding chance at an FDR-corrected q=0.01). (Yarkoni et al. 2011). We used the reverse inference map for Topic 15/100 (“Reading/Words/Language”) as the visual language mask and the reverse inference map for Topic 86/100 (“Speech/Auditory/Sounds”) as the auditory language mask. ROIs that were at least 15% inside the visual mask or auditory mask (15% chosen because it produced roughly the same number of edges as the other networks) were used to generate the networks: the positive network consisted of all edges connecting pairs of nodes in the visual language mask, and the negative network consisted of all edges connecting pairs of nodes in the auditory language mask. The nodes included in the visual and auditory language masks are visualized in Supplementary Figure S3. As before, each reading participant’s network strength was found by calculating their mean FC in the edges of the Vis/Aud Language network mask.

2.8. Reading Networks

If the GradCPT networks successfully generalize to predict reading recall, it raises new questions about the encoding of behavioral abilities in the functional connectome. If sustained attention holds a unique position in the FC matrix, then the GradCPT might perform just as well as one derived for our task. If not, the ways in which a network derived for our task differs spatially from the GradCPT network might provide clues into how each is specific to the task behavior on which it was trained. We therefore derived a pair of “Reading networks” specifically trained to predict performance on our reading recall questions.

The Reading networks were calculated from the reading data using the same leave-one-out (LOO) procedure as the GradCPT network, outlined in (Rosenberg, Finn, et al. 2016) and (Xilin Shen et al. 2017). One participant was left out as the “testing” participant, and then each edge was evaluated to see how well its strength correlated with behavioral performance across the remaining, “training” participants. MATLAB’s robust regression function robustfit was used to reduce the influence of outliers with a bisquare weighting function of residuals (Wager et al. 2005). For each LOO iteration, edges that were positively or negatively correlated with p values below a threshold (initially p<0.01 to match (Rosenberg, Finn, et al. 2016)) were included in a “positive” or “negative” network mask. The “network strength” of the left-out participant was found by calculating the participant’s mean FC in the edges of the LOO iteration’s network masks (edges in the negative mask were first multiplied by -1). The original method in (Rosenberg, Finn, et al. 2016) performed a sum and GLM fit to map network strengths onto behavioral measures. In our analysis, a participant’s absolute reading recall score does not have a meaning, so we focus our analysis only on predicting relative values (greater or lesser scores than other readers). We therefore removed the GLM to fit fewer parameters, and we changed the sum to a mean to be less sensitive to the number of edges selected in each LOO iteration. Supplementary Figure S2 illustrates the full LOO process.

We found “overall” network masks by calculating the overlap between all LOO iterations’ network masks (i.e., only edges included in all LOO iterations were included in the overall network masks). These overall network masks were used only to visualize the edges of importance or apply to an independent dataset, not to evaluate the predictive ability of the network.

The LOO iterations are trained on overlapping datasets, so we evaluated their output using permutation tests in addition to parametric statistics. We randomized the behavior vector 10,000 times and identified predictive networks using the LOO procedure. The distribution across permutation tests was used to derive p values.

2.9. Correlations with Behavior

To evaluate the ability of the networks to predict behavioral performance, we used the percent of reading recall questions answered correctly as our behavioral measure. When evaluating the Reading networks, the network strength for each participant was calculated using the mask from the LOO iteration where that participant was left out. After all iterations were completed, we found the Pearson correlation coefficient between the network strength and behavior across participants. When evaluating the GradCPT and Vis/Aud Language networks, the Pearson correlation coefficient between the network strength and behavior across participants was used to evaluate predictive ability. Based on previous findings reported by (Rosenberg, Finn, et al. 2016), we had a strong prior hypothesis that network strength (i.e., mean FC in positive network minus negative network edges) would be positively correlated with behavior, so one-tailed statistical tests were used.

3. Results

3.1. Identification of Network Edges

Using the methods described above, we obtained three pairs of networks. In each of these three, the pair consists of a “positive network” in which we hypothesize that FC strength should correlate positively with behavior and a “negative network” in which it should correlate negatively. (1) The GradCPT Networks, which were derived from the GradCPT data in (Rosenberg, Finn, et al. 2016) and whose strength correlated with behavioral performance on that task. (2) The Vis/Aud Language Networks, which were derived from NeuroSynth topic maps of voxels associated with visual and auditory language terms in the literature. (3) The Reading Networks, which were derived from the Reading data in this study and whose strength correlated with performance on the recall questions in the LOO procedure.

The newly identified Reading networks included hundreds of edges, indicating that there was predictive information in the FC matrices. Individual LOO iterations produced Reading networks ranging from 354-770 edges, and the number of edges in the overall Reading networks (115 positive, 144 negative) is more than would be expected by chance based on permutation tests (positive: p_perm=0.0029; negative: p_perm=0.0013). This network size is markedly lower than the GradCPT networks (757 positive, 630 negative) despite using the same statistical threshold. The Vis/Aud Language networks had a comparable number of edges (430 positive, 259 negative). We begin by exploring the predictive abilities of the networks before moving on to discuss their spatial distributions.

3.2. Predictive Ability of Networks

The strength of FC in the GradCPT positive and negative networks, derived from the data in (Rosenberg, Finn, et al. 2016), correlated significantly with reading recall accuracy (Figure 2, left). As expected, higher connectivity in the GradCPT positive network predicted increased performance, while higher connectivity in the GradCPT negative network predicted decreased performance. These correlations all reached statistical significance (positive r=0.449, p=0.0269; negative r=-0.544, p=0.00798; combined r=0.592, p=0.00380). This combined network outperformed two sets of 10,000 permutation tests where we randomly scrambled the network matrix edges (p_edgeperm=0.0024) and randomly scrambled the behavioral performance across participants (p_perm=0.0042). This shows that the GradCPT network generalizes from the sustained attention task on which it was trained to predict reading recall in an entirely new set of participants.

FIGURE 2.

(A) Graphical representation of tasks/methods used to derive the three FC networks used in this study. Left: The GradCPT networks were trained to predict performance on the GradCPT (Rosenberg, Finn, et al. 2016). Image adapted from (Esterman et al. 2013) with permission. Middle: Maps of visual and auditory language comprehension were derived from related topic maps from NeuroSynth and used to generate the FC-based Vis/Aud Language network (see Figure S3 for detailed maps). Right: Performance on the recall questions after each block of reading were used to derive the Reading networks. (B) Correlation across participants (N=19) of network strengths from each of these three networks and accuracy on reading recall questions. LOO network strengths were used for the Reading networks. Each x represents one participant. Dotted lines represent 95% confidence intervals. The r and p values listed above each plot show the Pearson correlation coefficient and the (one-tailed) p value based on the associated permutation test where behavioral performance was scrambled across participants.

To show that this generalization of the GradCPT network to our reading dataset is not the case for all plausible FC metrics, we also evaluated the ability of the Vis/Aud Language networks to predict reading recall accuracy. FC in these networks (Figure 2, middle column) was not informative (positive: r=-0.0468, p=0.578; negative: r=-0.0271, p=0.131; combined r=0.111, p=0.325). We again ran permutation tests, and the Vis/Aud Language network failed to significantly outperform these permutations (scrambled edges, p_edgeperm=0.0932; scrambled behavior, p_perm=0.3401). Using a Steiger’s Z test (Steiger 1980), we confirmed that the GradCPT network scores were correlated significantly more strongly with reading recall accuracy than the Vis/Aud Language network scores were (p_perm=0.0433).

To see whether there are other task-related abilities encoded in the FC matrix beyond sustained attention, and to test the generalizability of the method used to derive the GradCPT network, we next used the LOO procedure to evaluate the predictive ability of the Reading networks trained on our reading task. These networks predicted recall accuracy more accurately (Figure 2, right) than the other two networks, with higher correlations in the expected directions (positive: r=0.776, negative: r=-0.781, combined r=0.826). To rule out the possibility that the structure of the data made informative networks inevitable, we performed permutation tests where we randomized the behavior across participants 10,000 times and re-ran the LOO analyses. The combined network derived from the true data significantly outperformed these permutations (p_perm=0.0008). Using Steiger’s Z tests, we determined that the Reading network scores’ correlation with reading recall performance was significantly higher than those of the GradCPT (p_perm=0.0388) or Vis/Aud Language networks (p_perm=0.0012).

The predictive ability of the Reading networks is unlikely to be due to motion artifacts, button-press behavior, or eye movements because behavioral accuracy did not correlate strongly with participant motion, reading speed, saccade rate, or blink rate (Supplementary Figure S1A-D).

Two other summary metrics did correlate significantly with behavioral accuracy: pupil dilation and global FC. Pupil dilation was defined as the mean pupil diameter during pages minus the mean diameter between pages (when the fixation cross was displayed). Participants’ pupil dilations were positively correlated with their reading recall scores (r=0.574, p_perm=0.0055 based on scrambled-behavior permutation tests). Global FC was defined as the average FC across all edges. Participants’ global FC were negatively correlated with their recall scores (r=- 0.451, p_perm=0.0342). This may help explain why the Reading negative network contained slightly more edges than the positive one. Additional information can be found in Supplementary Figure S1E-F.

3.3. Size of Networks

Comparing the spatial distribution of our new Reading networks with the GradCPT networks may provide clues into how each is specific to its task or generalizable to new ones. In order to focus our comparison on informative edges, we next investigated whether the Reading networks’ predictive ability was driven by a small number of edges or by a larger set. To do this, we swept the threshold determining how many edges were included and re-ran the LOO analyses at each threshold value. The results, seen in Figure 3, show that predictive ability increases until 73 edges are included in the combined network. After this point, the correlation between the network strength and recall accuracy slowly declines and does not benefit from including additional edges. This indicates that additional edges provide redundant or unreliable information. This could be partially explained by blurring and differences in registration across participants, which could cause informative activity in one node to spread to nearby nodes.

FIGURE 3.

The correlation between reading recall and the predictions of the LOO-derived Reading networks of varying sizes. The y axis indicates the Pearson correlation value r as in Figure 2B (right). The x axis indicates the number of edges that are consistently included in the network across all LOO iterations. Performance peaks at 73 edges (red circle and dotted line), then slowly declines as more edges are included. Labeled gray circles and dotted lines indicate the statistical threshold (uncorrected) used to select edges in each LOO iteration that produced a network of this size.

3.4. Gross Spatial Distribution of Networks

Using these results, we selected the network size (73 edges) that best predicted reading recall in order to investigate the spatial distributions of the networks. In these “optimally thresholded” networks, we selected statistical thresholds that resulted in 73 edges (overlap across LOO iterations) for both the Reading and GradCPT networks. The spatial extents of these two 73-edge networks are visualized in Figure 4. Primary visual regions contribute to both the GradCPT and Reading networks, but the two networks diverge in other ways. For example, the GradCPT networks use more cerebellar-cerebellar and temporal-temporal connections, while the Reading networks rely more on left-occipital-to-left-temporal and left-temporal-to-right-parietal connections (Figure 4B). For reference, the full 268x268 matrix of FC induced by the task (mean across participants) is included as Supplementary Figure S4.

FIGURE 4.

(A) All edges in the “optimally thresholded” GradCPT networks (left) and Reading networks (right), as seen from the top (top row) and left side (bottom row). Red lines indicate positive network edges and blue lines indicate negative network edges. ROIs are represented as spheres and colored according to their macroscale region, as seen in (B). (B) All edges grouped by macroscale region and hemisphere (L=left, R=right). Colorbar values indicate the number of edges in the positive network minus the number of edges in the negative network: a positive value means higher FC between these regions is associated with better performance on the task. Colored outlines highlight edges within a macroscale region.

In general, ipsilateral connections were more common than contralateral connections in the Reading positive network (χ²(1)=5.19, p=0.0227) but not in the negative network (χ²(1)=0.537, p=0.464). Left-left connections were more dominant than right-right connections in the positive network (χ²(1)=7.2, p=0.00729). In the GradCPT networks, ipsilateral and contralateral connections were about equally represented in the positive network (χ²(1)=0.0261, p=0.872) and contralateral connections were slightly but not significantly more common in the negative network (χ²(1)=2.26, p=0.133). The number of edges of each type are listed in Table 1.

Table 1.

Number of edges in the 73-node “optimally thresholded” GradCPT and Reading positive and negative networks, separated by hemisphere. L-L indicates edges between two left-hemisphere nodes, R-R between two right-hemisphere nodes, and L-R between one left and one right node.

	GradCPT		Reading
	Positive	Negative	Positive	Negative
L-L	13	4	16	11
R-R	10	6	4	9
L-R	22	18	8	25

If the same signal of sustained attention is influencing performance in both the GradCPT and reading tasks, we might expect high overlap between same-direction networks – i.e., between the two positive networks and the two negative networks – and low overlap between the positive network from one task and the negative network from the other. We do see this trend in the networks thresholded at p<0.01, but in a small subset of the edges. The level of overlap between the same-direction networks is above what would be expected by chance as assessed by the hypergeometric cumulative distribution function, which assesses the probability of selecting the same edges by random chance given the total number of edges and the number included in each network (positive-positive: 8 overlapping edges, p=0.000777; negative-negative: 8 overlapping edges, p=0.00107), while the overlap between opposite-direction networks is zero. (GradCPT positive – Reading negative: p=0.0457; GradCPT negative – Reading positive: p=0.129). While they do exceed chance levels, the overlapping edges represent only 6.18% of the total number of edges in the Reading networks. In the “optimally thresholded” networks, 2 edges overlapped between the GradCPT positive and Reading positive networks, exceeding chance levels (p=5.96e-6), but no other edges overlapped.

3.5. Fine Spatial Distribution of Networks

Next, we examined the fine spatial distribution of the “optimally thresholded” network nodes and edges. The nodes contributing to the GradCPT and Reading networks are visualized in Figure 5(A-B). This allows us to determine whether macroscale regions involved in both networks are using the same nodes or different ones. We see in this figure that a smaller number of occipital regions participate in the Reading networks than did in the GradCPT networks. We also note that the right temporal ROIs in the Reading and GradCPT networks are almost entirely non-overlapping, with the GradCPT networks using superior temporal regions and the Reading networks using medial temporal ones.

FIGURE 5.

(A-B) Cortical ROIs participating in the “optimally thresholded” (A) GradCPT or (B) Reading networks. “Participating” ROIs are those for which a connection to any other ROI is part of the network. (C-F) All connections in the GradCPT and Reading networks that include any ROI in each of four macroscale regions: (C) occipital, (D) temporal, (E) motor, and (F) prefrontal. Red lines indicate positive network edges and blue lines indicate negative network edges. Dot color indicates macroscale region as in Figure 4.

The macroscale region that contributed most highly to both the positive Reading network and the positive GradCPT network was the left occipital region. The ROIs to which this region is connected in both networks are visualized in Figure 5(C). In both networks, occipital regions overwhelmingly participate in the positive network. In the GradCPT networks, widely distributed occipital ROIs show higher FC with bilateral superior temporal cortex (STC) and left-lateralized motor cortex in good performers. The Reading positive network contains connections between a smaller number of occipital regions and several semantic regions including left temporal pole, an inferior frontal ROI resembling Broca’s area, a temporal-parietal region resembling Wernicke’s area, and ROIs in fusiform cortex.

Temporal regions, like occipital ones, played an outsize role in both networks (Figure 5D). But unlike occipital regions, they participate in both the positive and negative networks. In the GradCPT networks, STC ROIs showed decreased FC with contralateral STC in good performers. In the Reading networks, left temporal pole showed decreased FC with bilateral supramarginal gyrus in good performers. These are in addition to the increased FC with occipital regions mentioned previously.

In motor and prefrontal regions, we see a great deal of divergence between the GradCPT and Reading networks. Motor regions are primarily in the GradCPT positive network, where they connect with occipital regions. However, in the reading task analysis they are exclusively in the negative network. (Figure 5E). Prefrontal regions participate much more strongly in the Reading networks than they do in the GradCPT networks (Figure 5F). In the Reading networks, there appears to be a hemispheric split: left prefrontal regions participate mostly in the positive network, while right prefrontal regions participate exclusively in the negative network.

3.6. Influence of Memory-Related Structures

In order to perform well on the reading recall questions, participants must not only understand the reading: they must also maintain the relevant information in memory until the questions at the end of each run. Past studies have found FC patterns predictive of within-individual or inter-individual differences in memory performance, so we might expect those edges to be influential in our Reading networks. For example, the amplitude and FC of activity in left angular gyrus, posterior cingulate, medial prefrontal cortex, and left hippocampus are different for successful and unsuccessful recollection in several visual tasks during recall (King et al. 2015). In another study, better free recall of pictured objects was predicted by higher FC between left hippocampus and the right hippocampus or left inferior prefrontal cortex during encoding (Dickerson et al. 2007). Because the latter study identified patterns during encoding, we considered it a better fit for our task and investigated the regions it identified further. We identified the Shen ROIs located at the coordinates reported by (Dickerson et al. 2007) for the left hippocampus, right hippocampus, and left inferior prefrontal cortex (ROIs 231, 98, and 157, respectively). Across all participants in our dataset, the FC between these three ROIs did not correlate significantly with reading recall (|r|<0.2, p>0.05). Given this, it is unsurprising that these edges did not play a role in our Reading networks.

For a more expansive investigation of memory-related structures, we used the NeuroSynth topic map for topic 24/100 (“Memory/Encoding/Retrieval”) as a mask containing major memory-related structures. We used the same overlap threshold used to derive our Vis/Aud Language networks to find a set of “memory nodes” that overlapped with this mask. This identified 41 Memory nodes in regions including the left and right hippocampus, the left inferior frontal gyrus, the left angular gyrus, the left and right precuneus, and the posterior cingulate. The 820 edges connecting these nodes had no overlap with the “optimally thresholded” Reading networks, representing no significant difference from the amount expected by chance (p>0.05). The mean strength of FC in edges connecting the Memory nodes did not correlate significantly with reading recall performance (r=-0.132, p>0.05). The 41 memory nodes overlapped with 2 of the 18 nodes in the Reading positive network and 4 of the 31 nodes in the Reading negative network, again within the range expected by chance (p>0.05).

4. Discussion

Sustained attention contributes to performance in any psychological task: to do something well, you must pay some attention to it. Reading is no exception, as studies of “mindless reading” attest. Mindless reading occurs about 30% of the time by some measures, and those reading mindlessly may miss critical words or inferences in the text (Smallwood, Fishman, and Schooler 2007). In this analysis, we tested the limits of one metric of sustained attention (FC in the GradCPT networks) to predict performance on the more naturalistic and educationally relevant task of reading recall and compared it with a more targeted predictor of reading recall.

In our dataset, functional connectivity within the GradCPT networks provided information on reading recall performance, with accuracy greater than chance and information beyond the outwardly observable metrics of head motion or reading speed (Figure 2; Figure S1A-D; Figure S5). The GradCPT networks had previously proven some cross-task predictive ability, including predicting ADHD symptom severity (Rosenberg, Finn, et al. 2016) and stop-signal task performance (Rosenberg, Zhang, et al. 2016) in independent datasets. Our findings further extend its generalizability to the prediction of performance on a naturalistic, educationally relevant task.

Other metrics, like pupil dilation and global FC, provided similarly reliable metrics (Figure S1, E-F). Pupil dilation has been shown to correlate with sustained attention task performance (van den Brink, Murphy, and Nieuwenhuis 2016), and the similar FC measure of “global efficiency” has been shown to correlate with IQ (van den Heuvel et al. 2009). The informative nature of global FC is in contrast with findings from data in the original study (Rosenberg, Finn, et al. 2016), where global FC was not predictive of the GradCPT performance metric d′ (r=0.21, p=0.31, data not shown), but the GradCPT networks were highly predictive (r=0.84, p_perm<0.001). This result indicates that the GradCPT networks are augmenting signals of domain-general arousal or engagement with an informative but domain-specific signal related to details of the GradCPT task. Although the GradCPT networks generalize to predict ADHD symptoms (Rosenberg, Finn, et al. 2016) and stop-signal task performance (Rosenberg, Zhang, et al. 2016) from data acquired at rest (i.e., in the absence of visual stimuli or motor responses), researchers seeking to apply them should note that the upper bound of their predictive power is determined by the similarity of the predicted behavior to GradCPT performance. Future work defining networks on a variety of attention-related behaviors could help optimize these networks and develop domain-general measures of sustained attention (Rosenberg et al. 2017).

The Reading network is similarly domain-specific, but it has the advantage of being specific to a domain that is naturalistic and educationally relevant. As a result, the Reading network strength could potentially be used to study how well a student is retaining reading under different conditions, to test the effectiveness of a written message, or as a biomarker to track the progression of a learning disability that affects reading comprehension. The Reading and GradCPT networks could eventually become two in a collection of models designed to predict a suite of cognitive and behavioral abilities from the same dataset (Rosenberg et al. 2017).

The success of the Reading networks shows that there are FC patterns predictive of reading recall performance, and that they provide information beyond that of behavioral metrics, pupil dilation, global FC, and GradCPT network strengths (Figure S5). Many of the network’s edges are consistent with findings from the reading literature. For example, typically developing children process reading with a network that is more left-lateralized than reading disabled children (Molfese 2012; Molfese, Fletcher, and Denton 2013). This could explain why the Reading positive network edges were skewed towards left intra-hemispheric connections and the negative edges were not.

The different spatial distributions of the GradCPT and Reading networks often appear indicative of the different mental and behavioral demands of the two tasks. Perhaps the clearest illustration of this is the set of occipital connections seen in Figure 5C. The Reading networks involved a narrow set of occipital nodes, while the GradCPT networks used a more distributed set. This makes sense given that the free-viewing reading task requires processing a small visual field, while the GradCPT requires covert attention to a larger image and involves scene recognition, which could be expected to engage additional regions in the visual processing hierarchy. In the Reading positive network, occipital regions are connected with the left temporal pole, which could be conceptualized as increased communication between visual input and semantic processing (Price et al. 1997). In the GradCPT positive network, occipital regions are connected to left motor regions, which could reflect how good GradCPT performers must use visual input to inform button presses. These button presses are much more frequent and task-relevant in the GradCPT than they are in the reading task, which may explain why motor regions are absent from the Reading positive network.

Also absent from the Reading networks is FC between memory-related structures like the hippocampus and posterior cingulate cortex. This is somewhat surprising given the well-documented involvement of these structures in memory encoding and retrieval (Rugg and Vilberg 2013), and the importance of memory to our task. Past findings have found that hippocampal FC during encoding correlates with intra-subject recall performance (Dickerson et al. 2007). However, the FC between these regions during reading did not correlate with inter-individual performance differences in our recall task and were therefore absent from our Reading networks. This could be due to several factors making our paradigm different from past studies, including the continuous stream of information in our free-viewing task and the testing of high-level content rather than single words or pictures.

The Reading negative network did include many connections between left temporal regions and the right inferior temporal cortex and right supramarginal gyrus (Figure 5D), both of which have been identified as compensatory mechanisms in poor readers (Molfese, Fletcher, and Denton 2013; Simos et al. 2005). In the GradCPT, left superior temporal cortex shows greater FC with occipital cortex and lower FC with right STC in good performers. This may reflect an attentional shift by which good performers ignore auditory input (the noise of the scanner) and instead allow auditory regions to be dominated by visual information from anterior visual regions (Eckert et al. 2008).

In addition, the Reading negative network shows that increased FC between right cerebellar and right prefrontal regions was indicative of poorer reading recall (Figure 4B, Figure 5F). Both regions have been implicated in language and reading before. The cerebellum may play several roles in reading recall, ranging from imagined speech to working memory (Mariën et al. 2013; Stoodley and Schmahmann 2009). Prefrontal involvement has been linked to semantic and episodic memory (Gabrieli, Poldrack, and Desmond 1998; Tulving et al. 1994) and the presence of an extended narrative: in a previous study, prefrontal regions were engaged when listening to a continuous story, but not when the same words or sentences were scrambled to disrupt the narrative (Lerner et al. 2011). Prefrontal connectivity has also been implicated in reading performance. Impaired readers have shown decreased connectivity between the left inferior frontal lobe and several right frontal regions (Farris et al. 2011), and readers reporting greater focus during a reading task had higher resting-state FC between the anterior medial prefrontal cortex and the cerebellum (Smallwood et al. 2013). But to our knowledge, the current study is the first to associate the task-based FC between right cerebellar and right prefrontal regions with reduced recall of reading.

Resting-state functional connectivity (rs-FC) has been previously linked to performance on a visual discrimination task (Baldassarre et al. 2012) and to reading competence (Koyama et al. 2011; Smallwood et al. 2013). We extend these results by examining whole-brain FC, focusing on a combined metric for prediction, and using task data rather than resting-state data, which may have advantages such as increased signal in task-relevant areas. Koyama et al. (2011) examined the rs-FC of several seed regions including Broca’s area, Wernicke’s area, and the fusiform gyrus. Koyama et al. showed that reading ability was associated with increased rs-FC between the three reading-related regions; our results show that this is true for reading task-evoked FC as well (Figure 5). However, Koyama also found that rs-FC between the left precentral gyrus and other motor regions indicated good reading performance, but our results show that the opposite is true for task-related FC (Figure 5E).

The observed results do not appear to be driven by head or eye motion. Head motion can be associated with variation in functional connectivity patterns and behavioral variability (Van Dijk, Sabuncu, and Buckner 2012), so we were careful to rule it out as a deciding factor in our analyses. We excluded high-motion participants and samples; and we confirmed that our performance metric did not correlate with motion, button press frequency, or the average rate of blinks or saccades (Figure S5).

Further research could include reading disabled participants or children, which could expand the range of observed behavioral performance and add statistical power to the analyses. In these participants, reading might require more effort and therefore elicit greater signal changes. A limitation of the current analysis is the inability to ascribe these FC patterns to a specific term like “lexical ability,” “sustained attention,” or “distractibility.” Further research could devise control tasks to isolate each of these component concepts. But the context dependence of performance markers seen in this study reminds us that in doing so, keeping the task context naturalistic and educationally relevant will enable the findings to be translated to real-life scenarios and symptoms. It is also likely that the component concepts’ influences on FC do not combine in a simple, additive way, and further research should take care to include interactions in their analyses.

Abbreviations

FC

Functional Connectivity

GradCPT

Gradual-onset Continuous Performance Task

LOO

Leave One Out

MEICA

Multi-Echo Independent Component Analysis

STC

Superior Temporal Cortex

rs-FC

Resting-State Functional Connectivity