Skip to main content
PMC home page
The Journal of Neuroscience logoLink to The Journal of Neuroscience
. 2014 Jan 1;34(1):149–157. doi: 10.1523/JNEUROSCI.3564-13.2014

Training-Induced Changes in Inhibitory Control Network Activity

Elliot T Berkman 1,✉, Lauren E Kahn 1, Junaid S Merchant 1
PMCID: PMC3866481  PMID: 24381276

Abstract

Despite extensive research on inhibitory control (IC) and its neural systems, the questions of whether IC can be improved with training and how the associated neural systems change are understudied. Behavioral evidence suggests that performance on IC tasks improves with training but that these gains do not transfer to other tasks, and almost nothing is known about how activation in IC-related brain regions changes with training. Human participants were randomly assigned to receive IC training (N = 30) on an adaptive version of the stop-signal task (SST) or an active sham-training (N = 30) during 10 sessions across 3 weeks. Neural activation during the SST before and after training was assessed in both groups using functional magnetic resonance imaging. Performance on the SST improved significantly more in the training group than in the control group. The pattern of neuroimaging results was consistent with a proactive control model such that activity in key parts of the IC network shifted earlier in time within the trial, becoming associated with cues that anticipated the upcoming need for IC. Specifically, activity in the inferior frontal gyrus decreased during the implementation of control (i.e., stopping) and increased during cues that preceded the implementation of IC from pretraining to post-training. Also, steeper behavioral improvement in the training group correlated with activation increases during the cue phase and decreases during implementation in the dorsolateral prefrontal cortex. These results are the first to uncover the neural pathways for training-related improvements in IC and can explain previous null findings of IC training transfer.

Introduction

The neurocognitive systems involved in inhibitory control (IC) are well characterized (Boucher et al., 2007; Wiecki and Frank, 2013), but relatively little work investigates whether and how IC performance can be improved with training. The present study used an IC training experiment with pretraining and post-training neuroimaging to characterize how the neural systems involved in IC change with training.

IC is commonly studied using paradigms, such as the stop-signal task (SST), wherein an ongoing motor response is inhibited or overcome. Performance on the SST is correlated with outcomes, such as drug abuse (Monterosso et al., 2005) and regulation of craving for cigarettes among smokers attempting to quit (Berkman et al., 2011), and researchers have begun to identify the differential contributions of regions typically active during the task, including rule representation and response selection in the dorsolateral prefrontal cortex (DLPFC; Miller and Cohen, 2001), monitoring and error detection in the anterior cingulate cortex (ACC; Munakata et al., 2011), behavioral gating in the basal ganglia (BG; Frank et al., 2001), and response inhibition in an interaction between the subthalamic nucleus and inferior frontal gyrus (IFG) modulated by the presupplementary motor area (preSMA; Frank, 2006; Aron et al., 2007; Sharp et al., 2010).

Compared with this wealth of knowledge, the literature on IC training is relatively impoverished. Two studies have found null effects of IC training (Logan and Burkell, 1986; Cohen and Poldrack, 2008), and meta-analyses of executive function training (including working memory and attention as well as IC) find mixed evidence for training effects, and little or no evidence for training transfer (Owen et al., 2010; Shipstead et al., 2012; Melby-Lervåg and Hulme, 2013). Thus, part of the lack of research in this area might be due to the notion that IC is “untrainable” even though related executive skills (Gray et al., 2003), such as attention (Choi et al., 2012) and working memory (Olesen et al., 2004; Jaeggi et al., 2011), can improve with training. Executive skills training studies have noted that effective training protocols tend to have high dosage and employ an adaptive design to maintain an engaging level of challenge (Holmes et al., 2009; Thorell et al., 2009; Diamond and Lee, 2011), so it may be that IC is trainable but the training tasks used previously were inadequate in form, dosage, or both. An alternative approach would be to use a variable-criterion task (e.g., the SST), where difficulty adapts to performance and related factors (e.g., task motivation) as they wax and wane.

The goal of the present study is to directly address these issues using an adaptive IC training task in combination with functional neuroimaging pretraining and post-training. We hypothesize that (1) IC performance will improve with sufficient training dosage of an adaptive task, (2) there will be alterations in the activity in the IC network (or subparts of the network) as a function of training, and (3) the nature of these changes will help explain the lack of training and transfer effects in previous studies of IC training.

Materials and Methods

Participants

A total of 60 participants (27 males and 33 females) aged 18–30 years (mean, 21.63 years; SD, 2.99) were recruited from the University of Oregon campus. The ethnicity of the participants was representative of the local region: 84% Caucasian, 4% Asian or Pacific Islander, 7% Hispanic, and 5% other. Before the first session, participants were screened on the phone for handedness (right-handed only), neurological disorders, mood disorders, and any MRI contraindications (e.g., metal implants). Those participants passing screening were scheduled for a baseline functional magnetic resonance imaging (fMRI) session that took place at the Lewis Center for Neuroimaging (LCNI) at the University of Oregon. There, all participants gave informed consent according to a protocol approved by our institutional review board.

Procedure overview

The study took place in three phases (baseline, training, endpoint) across ∼23 d. In the baseline phase, whole-brain fMRI data were acquired at the LCNI while participants completed two runs of the classic SST (Verbruggen and Logan, 2008a) and then two other tasks not related to inhibitory motor control (which will therefore not be discussed further here; performance and change over time on the SST was not related to performance or change on the other tasks). Participants also completed a series of unrelated questionnaire measures after the scan. At the end of the baseline session, participants were randomly assigned to either a training group, which received training in the SST during the training phase, or a sham-training group, which received training in a two-alternative forced-choice reaction time task (i.e., the SST without stop trials). Training began 1–2 d following the baseline session (mean, 1.58 d; SD, 0.72), and took place across 10 sessions that occurred approximately every other day for 3 weeks (mean, 18.98 d; SD = 1.94). The training sessions were held in behavioral testing rooms in the Department of Psychology. All participants completed all 10 training sessions. Finally, 1–2 d following the last training session (mean, 1.70 d; SD, 1.12), participants returned to the LCNI for an endpoint fMRI session that was identical to the baseline session.

Task

Baseline and endpoint sessions.

Participants completed two 6 min runs of the SST at each of the baseline and endpoint sessions. Each trial consisted of a cue indicating the start of a trial (500 ms), followed by an arrow pointing either left or right (with 1:1 relative frequency) that served as the go signal (1000 ms), and then an intertrial interval of variable duration (mean, 1400 ms; jittered following a gamma distribution). Participants were instructed to press the left or right arrow key as quickly as possible in response to the go signal. On 25% of the trials, an auditory stop signal was played after the go signal at a variable latency known as the stop-signal delay (SSD). Participants were instructed to withhold their button press on trials in which a stop signal sounded. The SSD was adjusted by 50 ms after each stop trial using a staircase function that increased for successful stops and decreased for failed stops. Two independent staircases alternated control over the SSD in blocks of eight trials until 50% response accuracy was reached on stop trials. The critical measure, the stop-signal response time (SSRT), is an index of the efficiency of the inhibitory control process. The SSRT is calculated as the difference between the speed of the stop process and the SSD. Here, we used the integration method to estimate the speed of the stop process because it was recently shown to be less biased than the alternative mean method (Verbruggen et al., 2013a). Each run consisted of 128 trials (32 stop trials) and lasted 6:06 min. The SSRT was computed separately for each run and averaged across the two runs at each time point to create a composite measure of IC ability at baseline and endpoint.

Training sessions.

Each training session consisted of one run of the SST runs described above, modified in two ways. First, the starting value of the SSD at each session was set as the estimated SSD from the previous session, allowing for the task to continuously adapt its difficulty level to maintain 50% accuracy despite potential changes in performance across training sessions. Second, the cue indicating the start of each trial changed color (from white) if the go response time based on a 16-trial moving average increased by 1 SD (to yellow) and 2 SDs (to red). This change was implemented to discourage participants from slowing their responses to accommodate stop trials and instead encourage participants to respond as quickly as possible to the go signals, which has been shown to decrease the bias and increase the efficiency of SSRT estimation (Verbruggen et al., 2013a).

Participants in the sham-training group completed the same SST as participants in the training group but with no stop signals. Thus, estimation of the SSRT for these participants during the sham-training sessions was not possible.

Behavioral data analysis

We estimated the SSRT for both groups at baseline and endpoint, and at each training session for the training group. At the group level, these data allowed for test of the interaction between group (training, sham-training) and time (baseline, endpoint) on SSRT, and simple main effects of time within the training group and sham training groups separately, and of group at the baseline and endpoint sessions separately. Also, we estimated the best fitting linear slope of the SSRTs across the 10 training sessions for each participant in the training group using linear regression. Steeper, more negative slopes indicate more effective training in stopping efficiency throughout the course of training.

fMRI acquisition and data analysis

Neuroimaging data were acquired using a 3.0 tesla Siemens Allegra head-only scanner at the University of Oregon's Robert and Beverly Lewis Center for Neuroimaging. Blood oxygen level-dependent (BOLD) echo-planar images were acquired with a T2*-weighted gradient echo sequence (TR, 2000 ms; TE, 30 ms; flip angle, 80°; matrix size, 64 × 64; 32 contiguous axial slices with interleaved acquisition; field of view, 200 mm; slice thickness; 4 mm). This sequence also prospectively corrected for motion during acquisition with PACE (Prospective Acquisition CorrEction; Thesen et al., 2000). For each participant, a high-resolution structural T1-weighted 3D MP-RAGE pulse sequence (TR, 2500 ms; TE, 4.38 ms; TI, 1100 ms; flip angle, 8°; matrix size, 256 × 192; 160 contiguous axial slices; voxel size, 1 mm3; slice thickness, 1 mm) was acquired coplanar with the functional images. Between the functional runs, field map scans were acquired to obtain magnetization values used to correct for field inhomogeneity (TR, 500 ms; TE, 4.99 ms; flip angle, 55°; matrix size, 64 × 64; field of view, 200 mm; 32 contiguous axial slices with interleaved acquisition; slice thickness, 4 mm).

Before preprocessing, nonbrain tissue was removed from the brain images using robust skull stripping with the Brain Extraction Tool in the FMRIB (Functional Magnetic Imaging of the Brain) Software Library (http://www.fmrib.ox.ac.uk/fsl/). Image preprocessing was conducted using NeuroElf (http://www.neuroelf.net), which implements SPM8 preprocessing functions (Wellcome Department of Cognitive Neurology, London, UK; http://www.fil.ion.ucl.ac.uk/spm/). The preprocessing stream was as follows: reorientation to the anterior–posterior commissure line, realignment and coregistration of functional images to each subject's own high-resolution structural image using a six-parameter rigid body transformation model, spatial normalization using segmentation into space compatible with an MNI template, and smoothing using a 6 mm3 full-width half-maximum Gaussian kernel.

Statistical analyses were implemented in SPM8. For each subject, event-related condition effects were estimated, according to the general linear model, using a canonical hemodynamic response function, high-pass filtering (128 s), and a first-order autoregressive error structure. At the subject level, BOLD signal was modeled in a fixed effects analysis with regressors for the cue period, correct go trials, correct stop trials, incorrect stop trials, and incorrect go trials. Linear contrasts were created for each comparison of interest (i.e., correct stop vs go trials at baseline, cue period at baseline vs endpoint, and correct stop at baseline vs endpoint). These contrasts were then imported to group-level random effects analyses for inference to the population.

We used two types of thresholds at the group level. First, because we had specific interest in the IC network, we created masks to restrict our search space to regions that showed increased activity in correct stop trials relative to correct go trials at baseline, separately for each group, using a familywise error (FWE) threshold of 0.05 as implemented in SPM8. For analyses involving only one group, we used the smoothed binary mask from the other group to maximize the independence between the mask and the contrast; for analyses involving both groups, we used the mask derived from the baseline contrast including both groups. Because this latter mask collapses across groups, it is orthogonal to subsequent analyses examining differences between groups. Second, for all subsequent analyses, we applied a combined voxel-height and cluster-extent correction for multiple comparisons within these masks using the Analysis of Functional Neuroimages (AFNI) AlphaSim software (Cox, 1996). AlphaSim takes into account the size of the search space (defined by the IC or whole-brain mask) and the estimated smoothness (based on AFNI's 3dFWHMx) to generate probability estimates (using Monte Carlo simulations) of a random field of noise producing a cluster of voxels of a given size for a set of voxels passing a given voxelwise p-value threshold. Within our IC network masks, these simulations determined that a FEW-corrected false-positive probability of p < 0.05 was achieved using a voxelwise threshold of p < 0.005 combined with a spatial extent threshold of 29 voxels for analyses using the sham-training mask, 35 voxels for analyses using the training mask, and 35 voxels for analyses using the combined mask.

Our central research question is the extent and nature of the change in functional activation in the IC network due to IC training. Thus, our first analysis was to functionally define the IC network in the contrast correct stop > correct go. We then used the suprathreshold voxels from that contrast as a mask to identify regions within the IC network that changed as a function of training. We reasoned that training-related changes in IC network activity could be characterized in two ways. First, following the 2 (group: training, control) × 2 (time: baseline, endpoint) factorial ANOVA design, the effect of the training on functional activation could be characterized in terms of the interaction between group and time, and then decomposed into the simple effects of time in the training group (i.e., endpoint > baseline for training subjects) and of group following training (i.e., training > control at endpoint). Second, within the training group, the linear slope of SSRT across time within each individual participant could be used as a measure of training quality, which could be entered as a covariate to the group-level models to identify regions that were differentially affected by training quality.

In light of recent results suggesting that cognitive control training can engender a shift from “reactive” to “proactive” control (Braver et al., 2009), we conducted each of the analyses described above separately for two parts of the trial: the IC preparation phase (in the moments immediately preceding the go signal) and the IC implementation phase (in the moments following the stop trial). Separating the components of the trial in this way is consistent with the dual mechanisms of control (DMC) model, which predicts that fluctuations in cognitive control performance are related to variations in the temporal dynamics of processing mode (Braver, 2012). Consistent with DMC predictions, the IC network shows preparatory activation in response to cues indicating an increased likelihood of an IC demand in the upcoming trial (Jahfari et al., 2012). Thus, we conducted analyses to separate the effects of training on the cue (preparatory) and stop signal (implementation) phases. Those components of the design are not multicollinear because all trials had a cue phase but only 25% of trials had a stop signal.

Results

Behavioral results: changes in IC performance as a function of training

The effectiveness of the training to improve IC was tested in the group–time interaction on SSRTs assessed at the baseline and endpoint sessions in the scanner. As shown in Figure 1a, there was a significant interaction such that the difference from baseline to endpoint was significantly greater in the training group compared with the sham-training group (F(1,58) = 4.76, p < 0.05, η2 = 0.08). Decomposing this interaction into its simple effects revealed a significant effect of time in the training group (F(1,29) = 21.81, p < 0.05) of greater magnitude (η2 = 0.43) than the effect in the sham-training group (η2 = 0.01, F(1,29) = 0.28, p > 0.6). The behavioral data from the baseline and endpoint sessions and all training sessions is shown in Table 1. The go response times are stable across the 10 training sessions, indicating that the procedure to prevent progressive slowing on go trials as a strategy to improve stopping rate was successful.

Figure 1.

Figure 1.

Open in a new tab

Behavioral improvement as a function of training. a, The significant group–time interaction on SSRT at the baseline and endpoint session for the training and sham-training groups (F(1,58) = 4.76, p < 0.05). Error bars indicate 2 SEs around the mean. b, The significant negative linear slope of SSRT across training sessions within the training group (with 95% confidence interval, dashed lines; F(1,29) = 7.17, p < 0.05). Dashed curves indicate 95% confidence interval around the linear regression line.

Table 1.

Behavioral data for baseline, endpoint, and all training sessions presented as mean (SD)

Training group
 Sham-training group
SSRT GoRT SSD % Inhib SSRT GoRT SSD % Inhib
Baseline 238.17 (47.03) 562.81 (131.58) 308.91 (154.84) 51.25 (10.36) 251.62 (115.70) 565.79 (120.98) 309.92 (141.96) 50.00 (11.85)
1 198.69 (67.81) 487.11 (124.18) 275.78 (109.88) 48.75 (13.94) — 372.71 (38.57) — —
2 189.93 (54.25) 476.78 (128.96) 281.51 (157.48) 49.17 (9.21) — 376.52 (44.71) — —
3 194.71 (81.14) 485.00 (138.81) 290.21 (178.43) 46.88 (9.92) — 377.89 (55.40) — —
4 185.71 (54.05) 479.54 (123.70) 286.20 (167.08) 50.10 (5.53) — 377.71 (51.55) — —
5 181.35 (66.08) 481.33 (136.34) 298.96 (174.08) 47.50 (8.50) — 381.44 (67.04) — —
6 183.88 (42.18) 477.28 (129.33) 285.36 (154.30) 50.73 (4.76) — 378.32 (76.84) — —
7 172.36 (42.56) 475.40 (130.35) 294.95 (154.68) 50.10 (4.46) — 373.93 (69.99) — —
8 176.56 (38.07) 472.37 (132.48) 290.00 (159.72) 49.38 (4.37) — 375.86 (76.57) — —
9 162.7 (61.32) 471.24 (128.39) 299.58 (164.37) 49.06 (5.51) — 374.63 (77.32) — —
10 161.31 (57.53) 476.92 (132.56) 303.91 (169.22) 50.63 (4.67) — 373.75 (81.26) — —
Endpoint 202.85 (28.45) 553.71 (130.26) 322.40 (140.03) 51.88 (7.42) 245.72 (98.38) 535.69 (115.14) 294.19 (134.19) 47.24 (13.10)
Open in a new tab

GoRT, Go response time; % Inhib, percentage of successful stops on stop trials.

A second analysis examined the slope of the SSRTs across the 10 training sessions for each participant in the training group. We first established the function that best characterized the slope using multilevel modeling with session at the first level and persons at the second level, and SSRT as the dependent measure. To do this, we modeled the trend across sessions using a linear, quadratic, and decelerating log and a series of step functions at the first level. When entered individually, the linear function produced the best fit (−2LL = 3048.58, where −2LL is −2 * log-likelihood), followed by a step function (with the step after the second training session, −2LL = 3054.75; other step functions produced worse fit), a quadratic function (−2LL = 3057.38), and a logarithmic (−2LL = 3163.59) function, all with seven model parameters. Also, the linear-only model had the smallest within-person (Level 1) variability (s2) of all these models. None of the other functions significantly improved upon a base linear model when added to it (compared using χ2 change tests between nested models). Based on these models, we concluded that a linear model is an appropriate and parsimonious model for the slope of SSRT across sessions. We then estimated the linear slope across the 10 sessions for each participant in the training group. This analysis indicated that the mean of the individual participant slopes was greater than zero (mean, −3.97; SD, 6.93; t(29) = 3.14; p < 0.05). Overall, as shown in Figure 1b, there was a negative slope across sessions such that the SSRT became faster with each training session (F(1,29) = 7.17, p < 0.05). There was no significant correlation between SSRT slope and change in SSRT from baseline to endpoint in the training group (r = 0.19; p, not significant). Because of the increased reliability of the slope metric (based on N = 10 sessions) compared with the pretraining-to-post-training change metric (based on N = 2 sessions), we used the former to quantify training quality.

Defining the IC network: main effect of task at baseline

The main effect of stopping in the correct stop > correct go contrast at baseline (collapsed across groups) revealed a network of regions including but not limited to bilateral IFG, right middle frontal gyrus (rMFG), bilateral insula, bilateral dorsal ACC (dACC), right presupplementary motor area, and bilateral caudate (Fig. 2; Table 2) that are commonly found in studies of response inhibition and executive control (Cohen et al., 2013). The regions identified in this collapsed analysis are hereafter referred to as the IC network, and the image from this contrast is used as a mask for subsequent analyses involving both groups. Separate masks were created for the training and control groups at baseline to be used as independent masks for analyses involving only one group (e.g., the simple effect of time in the training group is masked by the baseline image from the control group).

Figure 2.

Figure 2.

Open in a new tab

The IC network as identified by the correct stop > correct go contrast at baseline. The image shows the activation for the training group in yellow, the sham-training group in red, and the overlap in orange. Contrast thresholded at FWE rate of 0.05 as implemented in SPM8.

Table 2.

Stop > go; both groups at T1

Anatomical region x y z t z k
Right inferior frontal gyrus 48 18 −9 14.75 4318*
42 9 30 11.30
Right middle frontal gyrus 45 24 27 9.47
39 36 24 8.66
36 3 54 8.19
Right insula 36 15 −6 13.21
Bilateral dorsal anterior cingulate 3 36 12 8.59
6 24 33 12.12
−6 21 39 8.61
Right supplementary motor area 12 12 63 10.26
Right caudate 12 9 3 10.93
Right thalamus 9 −9 0 10.02
Right midbrain 3 −30 −6 7.85
Right middle temporal gyrus 51 −27 −9 12.33
Right superior temporal gyrus 63 −42 15 13.25
60 −33 6 12.55
51 3 −18 9.35
Right inferior parietal lobule 48 −45 39 12.03
Left inferior frontal gyrus −42 18 0 9.75 1386*
Left insula −33 24 −12 13.14
−27 15 6 9.57
Left caudate −12 6 3 9.06
Left thalamus −6 −12 −3 7.27
Left superior temporal gyrus −54 −3 −12 7.82
−54 −24 0 9.90
−54 −45 12 11.19
Left inferior parietal lobule −60 −45 30 10.42
−51 −45 48 7.46
Right cingulate 6 −18 27 8.00 6.56 68
Right precuneus 9 −66 42 5.97 5.26 7
Right middle frontal gyrus −42 33 24 5.76 5.11 2
Open in a new tab

Corrected using FWE correction of p = 0.05.

*All regions listed below k = 4318 and k = 1386 (until the next listed k value) are part of the same cluster.

Neuroimaging results: change in task-related activation as a function of training

Group–time interaction

This analysis identified regions within the IC network (Table 2) that increased or decreased from baseline to endpoint differentially between the two groups. There were no regions that showed a significant group–time interaction in their activation during stop trials (Table 3). However, during the cue phase, a cluster in the right inferior gyrus (51, 18, −3) showed greater increases in activation from baseline to endpoint in the training group compared with the control group (Fig. 3; Table 4).

Table 3.

Stop > rest; interaction between group and time and simple effects

Anatomical region x y z k t z
Increases for training > sham — — —
Decreases for training > sham — — —
Increases for training Right putamen 21 9 −3 47 4.64 3.98
Decreases for training Right angular gyrus 60 −51 30 191 4.85 4.11
Right supramarginal gyrus 48 −51 42 3.50 3.17
Right IFG/temporal pole 54 12 −6 37 4.23 3.70
Right ACC/preSMA 12 21 51 32 4.36 3.79
Posterior DMPFC 6 24 60 3.47 3.15
rMFG 48 3 45 126 4.06 3.58
Increases for sham — — —
Decreases for sham — — —
Training > sham at endpoint — — —
Sham > training at endpoint — — —
Training > sham at baseline — — —
Sham > training at baseline — — —
Open in a new tab

Corrected using AlphaSim; voxelwise threshold of p = 0.005, cluster size k > 29 for training, 35 for sham, and 35 for analyses with both groups; search conducted within the masks shown in Figure 2.

Figure 3.

Figure 3.

Open in a new tab

The group–time interaction in the right IFG during the cue phase. The right IFG showed greater increases immediately preceding each trial from pretraining to post-training in the training group compared with the sham-training group.

Table 4.

Cue > rest; interaction between group and time and simple effects

Anatomical region x y z k t z
Increases for training > sham Right inferior frontal gyrus 51 18 −3 50 3.43 3.34
Decreases for training > sham — — —
Increases for training Right supramarginal gyrus 54 −45 39 18* 3.13 2.88
Decreases for training Right putamen 21 9 −3 98 5.57 4.56
Left putamen −24 9 −3 31 3.86 3.44
Increases for sham — — —
Decreases for sham Left thalamus −15 −3 9 89 4.87 4.11
Left caudate −9 9 −3 3.63 3.25
Right putamen 15 6 −9 55 4.79 4.06
Training > sham at endpoint Right inferior frontal gyrus 45 18 6 35 3.86 3.62
Sham > training at endpoint — — —
Training > sham at baseline — — —
Sham > training at baseline — — —
Open in a new tab

Corrected using AlphaSim; voxelwise threshold of p = 0.005, cluster size k > 29 for training, 35 for sham, and 35 for analyses with both groups unless otherwise noted; search conducted within the masks shown in Figure 2.

*This cluster emerged at a slightly lower threshold but was significant at the default threshold in the whole-sample (N = 60) mask.

Changes from pretraining to post-training in the training group

As shown in Table 3 and Figure 4 (top right), only one region increased significantly during stop trials as a result of training: the right putamen (21, 9, −3). However, during stopping from pretraining to post-training, several regions decreased, including the right IFG (54, 12, −6) and ACC (12, 21, 51; Fig. 4, bottom right). In the cue phase of the trial, activation increased in right supramarginal gyrus (54, −45, 39; Fig. 4, top left; Table 4) and decreased in the putamen bilaterally (21, 9, −3; −21, 9, 0; Fig. 4, bottom left; Table 4). Thus, consistent with the DMC model of adjustments in time, activation in the putamen shifted from preparation to implementation as a function of training.

Figure 4.

Figure 4.

Open in a new tab

Simple effects of time (from pretraining to post-training) in the training group during the preparation (cue) phase (left side) and implementation (stopping) phase (right side). Increases over time are shown on the top and decreases are on the bottom. Activation in supramarginal gyrus increased during preparation and decreased during implementation, along with ACC/preSMA, right IFG, and rMFG. Conversely, activation in putamen decreased during preparation and increased during implementation. Asterisk indicates cluster that emerged at a slightly lower threshold but was significant at the default threshold in the whole-sample (N = 60) mask.

Group differences after training

At the endpoint session, both groups had completed 10 laboratory behavioral sessions but only the training group had practiced IC. As in the interaction analysis, there were no significant changes during stop trials (Table 3). However, during the cue phase, the training group showed greater activation than the control group in the right IFG (45, 18, 6) and left angular gyrus (−60, −48, 18; Fig. 5; Table 4), suggesting that activation in these regions shifted proactively from IC implementation to IC preparation.

Figure 5.

Figure 5.

Open in a new tab

Simple effect of group (training group > sham-training) at the endpoint session following training during the preparation (cue) phase. The training group showed greater activation in right IFG at endpoint than the sham-training group, and increased to a greater extent over time.

Neuroimaging results: regions correlated with training quality

Finally, as an additional way of examining training-related changes in functional activation, we identified regions within the IC network whose changes over time were moderated by training slope. These regions showed more or less change in activation (from baseline to endpoint) depending on the effectiveness of the training as measured by the degree of linear change throughout the training sessions. We found that two regions of the DLPFC showed greater amounts of change from pretraining to post-training to the extent that the slope of the SSRT across the training sessions was more negative (i.e., steeper improvements in stopping efficiency). Increases in activity in a region in the right lateral PFC (inferior/middle frontal gyrus; 33, 33, 27) during stop trials was positively correlated with SSRT slope, and increases in activity in a region in the left DLPFC (superior/middle frontal gyrus; 33, 42, 24) during the cue phase was negatively correlated with SSRT slope (Table 5). Because negative slopes corresponded with higher training effectiveness, these results suggest that increases in lateral PFC during IC preparation, and decreases in lateral PFC during IC implementation, correlated with greater training effectiveness. The fact that more effective training (as indexed by steeper SSRT slopes across training sessions) was associated with lateral PFC activity during both IC preparation and implementation implicates this region as a possible locus of training-related improvements in IC performance (Fig. 6).

Table 5.

Correlations in the training group of change in activation over time (endpoint > baseline) with training slope (more negative slope = more effective training)

Anatomical region x y z k t z
Cue > rest
    Positive correlation — — —
    Negative correlation Right middle frontal gyrus 24 42 30 32 3.99 3.52
Stop > rest
    Positive correlation Right middle frontal gyrus 36 30 27 52 4.74 4.03
    Negative correlation — — —
Open in a new tab

Corrected using AlphaSim; voxelwise threshold of p = 0.005, cluster size k > 29; search conducted within the mask shown in Figure 2.

Figure 6.

Figure 6.

Open in a new tab

Regions where activity was moderated by the effectiveness of training as measured by the linear slope of SSRT across training sessions. Greater training effectiveness (i.e., steeper negative slopes) were associated with increased activation in DLPFC during preparation (left) and decreased activation during implementation (right).

Discussion

The present research tested whether IC performance could be improved with training and characterized the associated changes in neural activity. We verified that performance on one IC task (the stop-signal) improves with training, and found that improvement followed a linear pattern at least through the first 10 sessions. We also found a complex pattern of changes in neural activation in both the IC preparation (cue) phase and the IC implementation (stopping) phase. The pattern of changes follows predictions of the DMC model regarding proactive temporal shifts in some regions such that activation initially recruited during IC implementation shifts earlier in time to engage in response to cues that predict the upcoming need for control. This interpretation at least partially explains why IC training has generally failed to transfer to new tasks; to the extent that training creates an association between activation in the IC network and specific cues that predict the upcoming need for IC, then training will not generalize to novel tasks that do not include the same anticipatory cues as the training task.

The region that most clearly shows the proactive shift pattern is the right IFG. In the training group, its activation decreased from before to after training during implementation but increased relative to the sham-training group during preparation. A shift of activation to the preparatory phase of the trial is consistent with behavioral data showing that participants make proactive adjustments to their response strategy on the SST when given cues indicating the likelihood of an impending stop trial (Verbruggen and Logan, 2009). The fact that the right IFG (and also the supramarginal gyrus) selectively showed this pattern is evidence that these regions are involved in preparing for IC by adjusting performance in a rule-based way rather than implementing IC per se (Banich, 2009; Hampshire et al., 2010; Munakata et al., 2011). Furthermore, the right IFG cluster reported here overlaps with the one reported by Lenartowicz and colleagues (2011), who found right IFG activation in response to cues that had previously been paired with a stop signal but no longer required inhibition. Similarly, right IFG activates to masked stop cues in the absence of stopping (van Gaal et al., 2010). However, unlike in the current study, the association between the cues and stopping was perfectly predictive (rather than probabilistic), which affects the strength of the association and subsequent performance (Verbruggen and Logan, 2008b). Our results indicate that training may be an effective way of increasing the association between cues and activation in IC-preparatory regions, even when the association is probabilistic.

The relationship between pretraining-to-post-training changes in lateral PFC and training quality further supports the DMC model of temporal control adjustments. Training quality correlated with increases in right lateral PFC activity during the preparation phase, and decreases in a nearby region of right lateral PFC during the implementation phase. The correlated change between training improvement and proactive shifts in right lateral PFC suggest a role for this region in forming predictive associations between particular contextual cues and IC, though not necessarily implementing the IC directly. One study found proactive shifts in this lateral PFC region following cognitive control training that did not include IC training (Braver et al., 2009), which, together with the results here, presents the possibility that the lateral PFC might serve to anticipate upcoming control demands across a range of executive functioning domains.

Unlike the proactive shift shown in several cortical regions, the right putamen evidenced an almost opposite, “reactive” shift. Specifically, activation in this region (part of the dorsal striatum) increased from baseline to endpoint in the training group during implementation and decreased during preparation, suggesting that this region may have strengthened with practice and narrowed in its specificity to implementation. With repeated practice at the SST wherein participants presumably learned well the timing of the task, activation in this region became more exclusively associated with the implementation of stopping, and unlike other regions, did not come to anticipate stopping during the cue phase of the trial. This is consistent with other findings suggesting that connectivity between PFC and BG during implementation is weakest when IC is expected (Jahfari et al., 2012), presumably because the action control network (Frank et al., 2001; Redgrave et al., 2010; Majid et al., 2013; including the putamen and the BG more broadly) has been proactively prepared. The contrasting pattern of changes with training between the BG and the cortical regions above highlight their roles in reactive and proactive IC, respectively.

A third set of regions showed yet another pattern of change characterized by decreases in activation during stopping from pretraining to post-training in the training group but without corresponding increases during the cue phase. Brain areas showing this pattern included the dACC, preSMA, and the posterior rMFG. In contrast with the putamen, which increased its activation during stopping with practice, these regions apparently reduced activation during IC implementation. One potential explanation for this is an increase in efficiency (Hockey, 1997; Gray et al., 2005) in these regions in the sense that they show less activation during equal or superior task performance. This explanation might account for the differences between the effect of training on these regions, which are cortical, and the putamen, which is subcortical. Namely, it may be the case that practice generates a more efficient (i.e., smaller BOLD) response in the cortex but a more robust (i.e., larger BOLD) response in the striatum. Another, more provocative explanation for this pattern, at least in the dACC, is that participants learned through training to be less distressed or frustrated during stop trials, and that these affective responses account for the dACC activity during the SST (Spunt et al., 2012) and related tasks (Inzlicht and Al-Khindi, 2012). If indeed dACC indexes negative affect in response to stop signals, one prediction that follows from this result is that training reduces the experience of negative affect during the SST through habituation or some other form of emotion regulation. An intriguing further possibility is that performance increases gained through training on this task may be mediated through reductions in negative affect or affective arousal, which has been shown to impair performance on this task (Verbruggen and Houwer, 2007). Further research probing emotional experience across training sessions will help disentangle these possibilities.

One limitation of the results is that many of the regions appearing in the simple effects (i.e., changes from pretraining to post-training in the training group, group differences at endpoint) do not also appear in the full group–time interaction. We believe this is because the size of the training effect is small and could not be detected in all regions even with our sample, which is relatively large (N = 60) for a neuroimaging study with repeated sessions within subjects. This argument is supported by the fact that each of the regions appearing in the simple effects in Tables 3 and 4 do emerge in the group–time interaction at a relaxed threshold (p < 0.05 voxelwise with a 25-voxel extent). Nonetheless, three previous IC training studies with smaller dosages of training and less spacing between sessions compared with our design have reported null effects of training on SSRT (Logan and Burkell, 1986; Cohen and Poldrack, 2008; Ditye et al., 2012), let alone transfer, and evidence from other lines of research on brief IC training (e.g., testing for carryover effects within a single session) converges on the notion that training effects in this area, if any, are small (Guerrieri et al., 2012; Verbruggen et al., 2013b). Thus, a challenge for future research on IC training will be to identify protocols with greater effects. One promising direction is to use transcranial direct current stimulation over right IFG, which has been found to boost the effect of training on performance relative to sham stimulation (Ditye et al., 2012). This result also highlights the idea that change in right IFG may moderate or even mediate the effects of behavior change training and interventions (Berkman et al., 2012).

The present research provides important clues about how to maximize the efficiency of IC training and develop effective IC training interventions. The DMC model predicts proactive shifts in activation in key parts of the IC network (e.g., the right IFG) when those cues reliably predict the need for IC in the immediate future. Thus, one marker of successful training is increased preparatory activation in parts of the IC network that anticipate the need to engage IC. If this proactive shift could generalize into other domains (e.g., risky decisions during gambling), it might increase subsequent IC efficiency and reduce impulsive responding in those domains (Verbruggen et al., 2012). Research in our laboratory is testing this possibility by importing cues from real-life IC challenges (e.g., cigarette cues for smokers attempting to quit) into an adapted training task, and by exporting cues from a basic SST (e.g., the stop signal beep) into untrained tasks. The results of these studies and those from other laboratories will demonstrate the utility of knowledge about the basic neurocognitive changes engendered by IC training for directly informing clinically relevant and effective interventions to reduce health-risking behaviors.

Footnotes

This work was supported by University of Oregon faculty research funds. Magnetic resonance imaging was performed at the Robert and Beverly Lewis Center for Neuroimaging at the University of Oregon. We thank Jolinda Smith for her guidance in sequence selection, Scott Watrous and Chuck Theobold for their assistance in image acquisition, and two anonymous reviewers and the Social and Affective Neuroscience Laboratory at the University of Oregon for helpful comments.

The authors declare no competing financial interests.

References

  1. Aron AR, Behrens TE, Smith S, Frank MJ, Poldrack RA. Triangulating a cognitive control network using diffusion-weighted magnetic resonance imaging (MRI) and functional MRI. J Neurosci. 2007;27:3743–3752. doi: 10.1523/JNEUROSCI.0519-07.2007. [DOI] [PMC free article] [PubMed] [Google Scholar]
  2. Banich MT. Executive function: the search for an integrated account. Curr Dir Psychol Sci. 2009;18:89–94. doi: 10.1111/j.1467-8721.2009.01615.x. [DOI] [Google Scholar]
  3. Berkman ET, Falk EB, Lieberman MD. In the trenches of real-world self-control: neural correlates of breaking the link between craving and smoking. Psychol Sci. 2011;22:498–506. doi: 10.1177/0956797611400918. [DOI] [PMC free article] [PubMed] [Google Scholar]
  4. Berkman ET, Graham AM, Fisher PA. Training self-control: a domain-general translational neuroscience approach. Child Dev Perspect. 2012;6:374–384. doi: 10.1111/j.1750-8606.2012.00248.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  5. Boucher L, Palmeri TJ, Logan GD, Schall JD. Inhibitory control in mind and brain: an interactive race model of countermanding saccades. Psychol Rev. 2007;114:376–397. doi: 10.1037/0033-295X.114.2.376. [DOI] [PubMed] [Google Scholar]
  6. Braver TS. The variable nature of cognitive control: a dual mechanisms framework. Trends Cogn Sci. 2012;16:106–113. doi: 10.1016/j.tics.2011.12.010. [DOI] [PMC free article] [PubMed] [Google Scholar]
  7. Braver TS, Paxton JL, Locke HS, Barch DM. Flexible neural mechanisms of cognitive control within human prefrontal cortex. Proc Natl Acad Sci U S A. 2009;106:7351–7356. doi: 10.1073/pnas.0808187106. [DOI] [PMC free article] [PubMed] [Google Scholar]
  8. Choi H, Chang LH, Shibata K, Sasaki Y, Watanabe T. Resetting capacity limitations revealed by long-lasting elimination of attentional blink through training. Proc Natl Acad Sci U S A. 2012;109:12242–12247. doi: 10.1073/pnas.1203972109. [DOI] [PMC free article] [PubMed] [Google Scholar]
  9. Cohen JR, Poldrack RA. Automaticity in motor sequence learning does not impair response inhibition. Psychon Bull Rev. 2008;15:108–115. doi: 10.3758/PBR.15.1.108. [DOI] [PubMed] [Google Scholar]
  10. Cohen JR, Berkman ET, Lieberman MD. Intentional and incidental self-control in ventrolateral PFC. In: Stuss DT, Knight RT, editors. Principles of frontal lobe function. 2nd ed. New York: Oxford UP; 2013. pp. 417–440. [Google Scholar]
  11. Cox RW. AFNI: software for analysis and visualization of functional magnetic resonance neuroimages. Comput Biomed Res. 1996;29:162–173. doi: 10.1006/cbmr.1996.0014. [DOI] [PubMed] [Google Scholar]
  12. Diamond A, Lee K. Interventions shown to aid executive function development in children 4 to 12 years old. Science. 2011;333:959–964. doi: 10.1126/science.1204529. [DOI] [PMC free article] [PubMed] [Google Scholar]
  13. Ditye T, Jacobson L, Walsh V, Lavidor M. Modulating behavioral inhibition by tDCS combined with cognitive training. Exp Brain Res. 2012;219:363–368. doi: 10.1007/s00221-012-3098-4. [DOI] [PubMed] [Google Scholar]
  14. Frank MJ. Hold your horses: a dynamic computational role for the subthalamic nucleus in decision making. Neural Netw. 2006;19:1120–1136. doi: 10.1016/j.neunet.2006.03.006. [DOI] [PubMed] [Google Scholar]
  15. Frank MJ, Loughry B, O'Reilly RC. Interactions between frontal cortex and basal ganglia in working memory: a computational model. Cogn Affect Behav Neurosci. 2001;1:137–160. doi: 10.3758/CABN.1.2.137. [DOI] [PubMed] [Google Scholar]
  16. Gray JR, Chabris CF, Braver TS. Neural mechanisms of general fluid intelligence. Nat Neurosci. 2003;6:316–322. doi: 10.1038/nn1014. [DOI] [PubMed] [Google Scholar]
  17. Gray JR, Burgess GC, Schaefer A, Yarkoni T, Larsen RJ, Braver TS. Affective personality differences in neural processing efficiency confirmed using fMRI. Cogn Affect Behav Neurosci. 2005;5:182–190. doi: 10.3758/CABN.5.2.182. [DOI] [PubMed] [Google Scholar]
  18. Guerrieri R, Nederkoorn C, Jansen A. Disinhibition is easier learned than inhibition. The effects of (dis)inhibition training on food intake. Appetite. 2012;59:96–99. doi: 10.1016/j.appet.2012.04.006. [DOI] [PubMed] [Google Scholar]
  19. Hampshire A, Chamberlain SR, Monti MM, Duncan J, Owen AM. The role of the right inferior frontal gyrus: inhibition and attentional control. Neuroimage. 2010;50:1313–1319. doi: 10.1016/j.neuroimage.2009.12.109. [DOI] [PMC free article] [PubMed] [Google Scholar]
  20. Hockey GR. Compensatory control in the regulation of human performance under stress and high workload: a cognitive-energetical framework. Biol Psychol. 1997;45:73–93. doi: 10.1016/S0301-0511(96)05223-4. [DOI] [PubMed] [Google Scholar]
  21. Holmes J, Gathercole SE, Dunning DL. Adaptive training leads to sustained enhancement of poor working memory in children. Dev Sci. 2009;12:F9–F15. doi: 10.1111/j.1467-7687.2009.00848.x. [DOI] [PubMed] [Google Scholar]
  22. Inzlicht M, Al-Khindi T. ERN and the placebo: a misattribution approach to studying the arousal properties of the error-related negativity. J Exp Psychol Gen. 2012;141:799–807. doi: 10.1037/a0027586. [DOI] [PubMed] [Google Scholar]
  23. Jaeggi SM, Buschkuehl M, Jonides J, Shah P. Short- and long-term benefits of cognitive training. Proc Natl Acad Sci U S A. 2011;108:10081–10086. doi: 10.1073/pnas.1103228108. [DOI] [PMC free article] [PubMed] [Google Scholar]
  24. Jahfari S, Verbruggen F, Frank MJ, Waldorp LJ, Colzato L, Ridderinkhof KR, Forstmann BU. How preparation changes the need for top-down control of the basal ganglia when inhibiting premature actions. J Neurosci. 2012;32:10870–10878. doi: 10.1523/JNEUROSCI.0902-12.2012. [DOI] [PMC free article] [PubMed] [Google Scholar]
  25. Lenartowicz A, Verbruggen F, Logan GD, Poldrack RA. Inhibition-related activation in the right inferior frontal gyrus in the absence of inhibitory cues. J Cogn Neurosci. 2011;23:3388–3399. doi: 10.1162/jocn_a_00031. [DOI] [PubMed] [Google Scholar]
  26. Logan GD, Burkell J. Dependence and independence in responding to double stimulation: a comparison of stop, change, and dual-task paradigms. J Exp Psychol Hum Percept Perform. 1986;12:549–563. doi: 10.1037/0096-1523.12.4.549. [DOI] [Google Scholar]
  27. Majid DS, Cai W, Corey-Bloom J, Aron AR. Proactive selective response suppression is implemented via the basal ganglia. J Neurosci. 2013;33:13259–13269. doi: 10.1523/JNEUROSCI.5651-12.2013. [DOI] [PMC free article] [PubMed] [Google Scholar]
  28. Melby-Lervåg M, Hulme C. Is working memory training effective? A meta-analytic review. Dev Psychol. 2013;49:270–291. doi: 10.1037/a0028228. [DOI] [PubMed] [Google Scholar]
  29. Miller EK, Cohen JD. An integrative theory of prefrontal cortex function. Ann Rev Neurosci. 2001;24:167–202. doi: 10.1146/annurev.neuro.24.1.167. [DOI] [PubMed] [Google Scholar]
  30. Monterosso JR, Aron AR, Cordova X, Xu J, London ED. Deficits in response inhibition associated with chronic methamphetamine abuse. Drug Alcohol Depend. 2005;79:273–277. doi: 10.1016/j.drugalcdep.2005.02.002. [DOI] [PubMed] [Google Scholar]
  31. Munakata Y, Herd SA, Chatham CH, Depue BE, Banich MT, O'Reilly RC. A unified framework for inhibitory control. Trends Cogn Sci. 2011;15:453–459. doi: 10.1016/j.tics.2011.07.011. [DOI] [PMC free article] [PubMed] [Google Scholar]
  32. Olesen PJ, Westerberg H, Klingberg T. Increased prefrontal and parietal activity after training of working memory. Nat Neurosci. 2004;7:75–79. doi: 10.1038/nn1165. [DOI] [PubMed] [Google Scholar]
  33. Owen AM, Hampshire A, Grahn JA, Stenton R, Dajani S, Burns AS, Howard RJ, Ballard CG. Putting brain training to the test. Nature. 2010;465:775–778. doi: 10.1038/nature09042. [DOI] [PMC free article] [PubMed] [Google Scholar]
  34. Redgrave P, Rodriguez M, Smith Y, Rodriguez-Oroz MC, Lehericy S, Bergman H, Agid Y, DeLong MR, Obeso JA. Goal-directed and habitual control in the basal ganglia: implications for Parkinson's disease. Nat Rev Neurosci. 2010;11:760–772. doi: 10.1038/nrn2915. [DOI] [PMC free article] [PubMed] [Google Scholar]
  35. Sharp DJ, Bonnelle V, De Boissezon X, Beckmann CF, James SG, Patel MC, Mehta MA. Distinct frontal systems for response inhibition, attentional capture, and error processing. Proc Natl Acad Sci U S A. 2010;107:6106–6111. doi: 10.1073/pnas.1000175107. [DOI] [PMC free article] [PubMed] [Google Scholar]
  36. Shipstead Z, Hicks KL, Engle RW. Cogmed working memory training: does the evidence support the claims? J Appl Res Mem Cogn. 2012;1:185–193. doi: 10.1016/j.jarmac.2012.06.003. [DOI] [Google Scholar]
  37. Spunt RP, Lieberman MD, Cohen JR, Eisenberger NI. The phenomenology of error processing: the dorsal ACC response to stop-signal errors tracks reports of negative affect. J Cogn Neurosci. 2012;24:1753–1765. doi: 10.1162/jocn_a_00242. [DOI] [PubMed] [Google Scholar]
  38. Thesen S, Heid O, Mueller E, Schad LR. Prospective acquisition correction for head motion with image-based tracking for real-time fMRI. Magn Reson Med. 2000;44:457–465. doi: 10.1002/1522-2594(200009)44:3<457::aid-mrm17>3.0.co;2-r. [DOI] [PubMed] [Google Scholar]
  39. Thorell LB, Lindqvist S, Bergman Nutley S, Bohlin G, Klingberg T. Training and transfer effects of executive functions in preschool children. Dev Sci. 2009;12:106–113. doi: 10.1111/j.1467-7687.2008.00745.x. [DOI] [PubMed] [Google Scholar]
  40. van Gaal S, Ridderinkhof KR, Scholte HS, Lamme VA. Unconscious Activation of the Prefrontal No-Go Network. J Neurosci. 2010;30:4143–4150. doi: 10.1523/JNEUROSCI.2992-09.2010. [DOI] [PMC free article] [PubMed] [Google Scholar]
  41. Verbruggen F, Houwer JD. Do emotional stimuli interfere with response inhibition? Evidence from the stop signal paradigm. Cogn Emot. 2007;21:391–403. doi: 10.1080/02699930600625081. [DOI] [Google Scholar]
  42. Verbruggen F, Logan GD. Response inhibition in the stop-signal paradigm. Trends Cogn Sci. 2008a;12:418–424. doi: 10.1016/j.tics.2008.07.005. [DOI] [PMC free article] [PubMed] [Google Scholar]
  43. Verbruggen F, Logan GD. Automatic and controlled response inhibition: associative learning in the go/no-go and stop-signal paradigms. J Exp Psychol Gen. 2008b;137:649–672. doi: 10.1037/a0013170. [DOI] [PMC free article] [PubMed] [Google Scholar]
  44. Verbruggen F, Logan GD. Proactive adjustments of response strategies in the stop-signal paradigm. J Exp Psychol Hum Percept Perform. 2009;35:835–854. doi: 10.1037/a0012726. [DOI] [PMC free article] [PubMed] [Google Scholar]
  45. Verbruggen F, Adams R, Chambers CD. Proactive motor control reduces monetary risk taking in gambling. Psychol Sci. 2012;23:805–815. doi: 10.1177/0956797611434538. [DOI] [PMC free article] [PubMed] [Google Scholar]
  46. Verbruggen F, Chambers CD, Logan GD. Fictitious inhibitory differences: How skewness and slowing distort the estimation of stopping latencies. Psychol Sci. 2013a;24:352–362. doi: 10.1177/0956797612457390. [DOI] [PMC free article] [PubMed] [Google Scholar]
  47. Verbruggen F, Adams RC, van't Wout F, Stevens T, McLaren IP, Chambers CD. Are the effects of response inhibition on gambling long-lasting? PLoS One. 2013b;8:e70155. doi: 10.1371/journal.pone.0070155. [DOI] [PMC free article] [PubMed] [Google Scholar]
  48. Wiecki TV, Frank MJ. A computational model of inhibitory control in frontal cortex and basal ganglia. Psychol Rev. 2013;120:329–355. doi: 10.1037/a0031542. [DOI] [PubMed] [Google Scholar]

Articles from The Journal of Neuroscience are provided here courtesy of Society for Neuroscience

RESOURCES