Sleep Deprivation Promotes Habitual Control over Goal-Directed Control: Behavioral and Neuroimaging Evidence

Abstract

Sleep is one of the most fundamental processes of life, playing an important role in the regulation of brain function. The long-term lack of sleep can cause memory impairments, declines in learning ability, and executive dysfunction. In the present study, we evaluated the effects of sleep deprivation on instrumental learning behavior, particularly goal-directed and habitual actions in humans, and investigated the underlying neural mechanisms. Healthy college students of either gender were enrolled and randomly divided into sleep deprivation group and sleep control group. fMRI data were collected. We found that one night of sleep deprivation led to greater responsiveness to stimuli that were associated with devalued outcomes in the slips-of-action test, indicating a deficit in the formation of goal-directed control and an overreliance on habits. Furthermore, sleep deprivation had no effect on the expression of acquired goal-directed action. The level of goal-directed action after sleep deprivation was positively correlated with baseline working memory capacity. The neuroimaging data indicated that goal-directed learning mainly recruited the ventromedial PFC (vmPFC), the activation of which was less pronounced during goal-directed learning after sleep deprivation. Activation of the vmPFC during goal-directed learning during training was positively correlated with the level of goal-directed action performance. The present study suggests that people rely predominantly on habits at the expense of goal-directed control after sleep deprivation, and this process involves the vmPFC. These results contribute to a better understanding of the effects of sleep loss on decision-making.

SIGNIFICANCE STATEMENT Understanding the cognitive consequences of sleep deprivation has become extremely important over the past half century, given the continued decline in sleep duration in industrialized societies. Our results provide novel evidence that goal-directed action may be particularly vulnerable to sleep loss, and the brain mechanism underlying this effect was explored. Elucidation of the effects of sleep deprivation on decision-making will deepen our understanding of the function of sleep, emphasizing the role of sleep in cognitive impairments and mental health.

Introduction

Instrumental behavior is an elementary type of learning whereby subjects obtain knowledge about the consequences of actions to obtain rewards and avoid punishments, which stimulates motivated behavior in daily life (Shanks, 1993). Substantial animal and human studies have suggested that two systems are involved in instrumental behavior: a stimulus-response habit learning process (i.e., model-free system) and a goal-directed process (i.e., model-based system) (Balleine and Dickinson, 1998; Balleine and O'Doherty, 2010). The former system encodes the relationship between a stimulus and a corresponding response and is insensitive to changes in outcome value. This type of behavior provides significant benefits for cognitive efficiency, but flexibly adjusting behavior is difficult when outcome valuation suddenly changes. While in a goal-directed process, the formation of action-consequence associations makes it sensitive to changes in outcome values and the contingency between actions and outcomes. Based on these characteristics, the sensitivity to outcome devaluation is usually used to indicate on which system individuals mainly rely in instrumental learning. Flexible and changeable behaviors are important for adapting to an ever-changing environment, and an imbalance between these two systems may result in impulsivity and underlie dysfunctional clinical conditions, such as addiction (Barker and Taylor, 2014; Everitt and Robbins, 2016), obsessive-compulsive disorder (Gillan et al., 2011, 2014; Gillan and Robbins, 2014), and obesity (Horstmann et al., 2015).

The importance of sleep for cognition has been explored for many years. However, insufficient sleep is commonplace in modern society, including sleep disorders and various sleep problems, such as staying late at work, jet lag, and shift work. A lack of sleep affects cognition in many ways and can negatively impact alertness, learning, memory, and executive function (Chee and Chuah, 2008; Killgore, 2010; Jackson et al., 2013; Krause et al., 2017). Previous studies revealed that sleep-deprived individuals had greater impulsiveness and emitted faster incorrect responses (Anderson and Platten, 2011; Killgore et al., 2011). Sleep problems are also broadly associated with impulse-related disorders and behaviors. A longitudinal epidemiological survey of young adults found that prior insomnia significantly increased the risk of illicit drug use (Breslau et al., 1996). More recent researches suggested that sleep disturbances are also a risk factor for relapse in addiction (Brower and Perron, 2010). In addition to the impulsivity, executive control function is also sensitive to sleep deprivation (SD) (Jones and Harrison, 2001). The evidence above suggests that the association between sleep loss and impulsivity-related behaviors may be mediated by a universal mechanism, in which sleep loss impacts the balance of goal-directed control and habitual control.

Several animal and human studies have demonstrated the involvement of the corticobasal ganglia network in instrumental learning (for review, see Yin and Knowlton, 2006). The prefrontal and parietal association cortices (mPFC and medial orbitofrontal cortex in humans and prelimbic cortex in rats) and associative striatum (caudate in humans and dorsomedial striatum in rats) participate in goal-directed actions. The posterior putamen in humans and the dorsolateral striatum in rats play a role in controlling habitual actions (Balleine and O'Doherty, 2010). Previous studies found that sleep loss affects the function of brain areas that are involved in goal-directed behavior; for example, SD decreases brain metabolism, especially in the PFC (Thomas et al., 2000; Muzur et al., 2002).

Building on prior findings, the present study had two objectives. First, we used a total SD paradigm to investigate the effects of sleep loss on instrumental learning, especially on the balance between goal-directed and habitual control. Second, using fMRI, we explored the neural mechanisms that underlie the effects of SD on the balance between goal-directed and habitual action.

Materials and Methods

Participants

Ninety-four healthy college students were enrolled in the study through advertisements. All of the participants met the following inclusion criteria: (1) right-handed, (2) nonsmokers, (3) regular sleeping habits (sleep no less than 6.5 h per night), (4) not on any long-term medications, (5) no history of sleep disorders or psychiatric/neurological disorders, (6) no shift work in the past 3 months, (7) no traveling to a time zone with more than a 4 h time difference in the past 3 months, (8) <3 caffeinated drinks per day, and (9) no extreme score on the Horne-Östberg Morningness-Eveningness Questionnaire (MEQ, i.e., scores between 35 and 65) (Horne and Ostberg, 1976). The participants who underwent fMRI scanning were free from MRI contraindications.

In both experiments, the participants were randomly assigned to the sleep control (SC) group or SD group.

In Experiment 1, 24 participants were assigned to the SC group, and 29 were assigned to the SD group. Two were excluded in the SC group because of a failure to keep the sleep schedule (>2 h deviation; n = 1) and a failure to meet the learning criterion (n = 1), and 4 were excluded in the SD group because of an inability to stay awake during SD (n = 2) and a failure to meet the learning criterion (n = 2), resulting in 47 participants being included in the final analysis in Experiment 1 (SC group: n = 22; SD group: n = 25). In addition, due to technical issues, scanning data could not be analyzed for 2 other participants in the SD group, leaving n = 23 in the SD group for the imaging data analysis.

In Experiment 2, 19 participants were assigned to the SC group, and 22 were assigned to the SD group. Two were excluded in the SC group because of a failure to meet the learning criterion, and 3 were excluded in the SD group because of an inability to stay awake during SD (n = 2) and a failure to meet the learning criterion (n = 1), resulting in 36 participants being included in the final analysis in Experiment 2 (SC group: n = 17; SD group: n = 19). There was no overlap in the participants involved in each of the experiments. Table 1 provides demographic data, psychological traits, and sleep evaluations for each experimental group.

Table 1.

Demographic data, psychological traits, and sleep evaluation^a

	Experiment 1			Experiment 2
	SC (n = 22)	SD (n = 25)	p	SC (n = 17)	SD (n = 19)	p
Age (yr)	23.3 ± 1.2	23.2 ± 1.7	0.868	24.0 ± 1.2	24.1 ± 1.5	0.819
Male, n (%)	13 (59.1)	14 (56.0)	0.831	9 (52.9)	9 (47.4)	0.738
BMI (kg/m2)	21.6 ± 2.5	21.0 ± 2.6	0.433	20.5 ± 2.1	21.4 ± 2.7	0.265
Education (yr)	17.0 ± 1.3	16.7 ± 1.6	0.582	18.1 ± 1.2	18.0 ± 1.3	0.785
BDI	1.4 ± 2.0	2.3 ± 2.5	0.157	2.6 ± 1.9	1.7 ± 1.9	0.162
HAMA	2.2 ± 2.1	3.6 ± 3.4	0.105	3.7 ± 3.6	2.1 ± 2.2	0.103
BIS	62.5 ± 4.2	62.7 ± 9.5	0.951	62.6 ± 7.9	61.5 ± 6.3	0.656
Attentional impulsiveness	21.9 ± 2.7	21.4 ± 3.5	0.585	21.1 ± 3.0	21.5 ± 3.3	0.699
Motor impulsiveness	15.5 ± 3.0	15.6 ± 2.5	0.895	15.8 ± 3.8	15.9 ± 3.0	0.873
Nonplanning impulsiveness	25.2 ± 3.2	25.8 ± 4.9	0.664	25.8 ± 3.2	24.1 ± 3.5	0.144
MoCA	29.0 ± 0.8	28.8 ± 1.2	0.502	29.0 ± 1.0	28.9 ± 0.8	0.860
WM capacity	51.8 ± 12.2	47.2 ± 18.3	0.326	46.5 ± 12.8	51.6 ± 11.6	0.218
ESS	5.8 ± 2.6	6.8 ± 2.8	0.220	8.1 ± 3.6	7.1 ± 2.6	0.368
PSQI	3.0 ± 1.4	3.1 ± 1.7	0.253	4.4 ± 2.1	3.7 ± 1.8	0.309
Habitual sleep duration (h)b	7.3 ± 0.5	7.5 ± 0.6	0.718	7.5 ± 0.8	7.2 ± 0.6	0.214
MEQ	54.5 ± 7.1	52.3 ± 7.2	0.313	52.0 ± 7.9	55.4 ± 5.8	0.160

^aData are mean ± standard deviation. BMI, Body mass index.

^bHabitual sleep duration came from participants' answers to the question in PSQI; that is, “How many hours of actual sleep did you get at night during the past month?”

All of the participants were instructed to refrain from drug, alcohol, and caffeine consumption for 24 h before the study to maintain a normal sleep–wake rhythm, verified by daily sleep logs and wrist actigraphy (Actiwatch, Philips Respironics). Each subject provided written informed consent and was paid for participation in the study. The study protocol was approved by the Institutional Review Board of Peking University.

Experimental design

In Experiment 1, we used a “between-subjects” study design to investigate the effect of acute sleep loss on instrumental learning. The participants underwent both instrumental training and test phases following either a night of sleep (SC group) or SD (SD group). During the training phase, we collected both behavioral and imaging data to elucidate the neural mechanisms.

We could not get a conclusion whether the results in Experiment 1 were due to the adverse effects of SD on the formation of the instrumental learning (e.g., initial encoding or acquisition) or on both processes, including formation and expression (e.g., retrieval process). Based on this, in Experiment 2, instrumental training was performed before the sleep intervention to determine whether SD affected the expression of instrumental behavior. Only behavioral data were collected for Experiment 2. The overall experimental design described above is illustrated in Figure 1.

Figure 1.

Experimental design and protocol. Top, In Experiment 1, participants performed both instrumental training and test phases following either a night of sleep (SC group) or SD. Both behavioral and imaging data were collected during the training phase. Bottom, In Experiment 2, both groups performed instrumental training first. Then after a night of sleep (SC group) or SD, the test was conducted to observe the effects of SD solely on the expression of instrumental behavior.

Procedure

Three days before the experiment, the participants made their first visit to the laboratory. They were briefed on the study protocol and requirements, and all of the participants completed a demographic questionnaire in addition to completing the Beck Depression Inventory (BDI), Hamilton Anxiety Scale (HAMA), Epworth Sleepiness Scale (ESS), Pittsburgh Sleep Quality Index (PSQI), MEQ, Barratt Impulsiveness Scale (BIS-11), and Montreal Cognitive Assessment (MoCA). Afterward, working memory was assessed using an automated version of the operation span (Ospan) procedure (Unsworth et al., 2005). To monitor sleep patterns, each participant was asked to complete a sleep diary and was given a wrist Actiwatch that had to be worn until they completed the study. Only subjects with good sleep habits (slept >6.5 h per night, went to sleep no later than 1:00 A.M., and woke up no later than 9:00 A.M.) were invited to participate in subsequent sessions. The average sleep duration for three nights before the study initiation did not differ between the two groups (Experiment 1: 7.59 ± 0.45 [SC] vs 7.68 ± 0.64 [SD], p = 0.163; Experiment 2: 7.22 ± 0.57 [SC] vs 7.53 ± 0.62 [SD], p = 0.639).

In each experiment, participants who met the selection criteria were randomly assigned to one of two sleep intervention groups: the SC group and the SD group. This group assignment remained unknown to the participants until the beginning of the corresponding sleep intervention.

In Experiment 1, we explored the effects of SD on subsequent instrumental learning. Both the SC group and SD group arrived at the laboratory at 9:00 P.M. on the evening of the experiment. The SC group were given a 9 h sleep opportunity. They were monitored by wrist actigraphy and awakened at 7:00 A.M. the next morning. All SC participants slept at least 6.5 h. To minimize any effects of sleep inertia on performance, participants performed a 10 min psychomotor vigilance task (PVT) (Basner and Dinges, 2011), and sleepiness was assessed with a visual analog scale at 7:30 A.M. Subsequently, the participants performed the instrumental learning task in the fMRI scanner at 8:00 A.M. In the SD group of Experiment 1, the participants stayed awake the entire night. They were assessed for subjective sleepiness with visual analog scale and completed the PVT to measure their alertness during the first 10 min of every hour from 10:00 P.M. until 6:00 A.M. The SD group was monitored continually by research assistants. SD participants were allowed to engage in nonstrenuous activities, such as reading, watching videos, and conversing with fellow participants. fMRI scanning was performed at ∼6:00 A.M., which was considered the circadian nadir (Kong et al., 2014).

The procedure for Experiment 2 was similar to Experiment 1, with the exception of the timing of instrumental learning. Both the SC and SD groups underwent instrumental learning to acquire the same information before the SC or SD condition. The next morning, the behavioral test was conducted on both groups (Fig. 1).

The timing of the behavioral tests after one night of SD (6:00 A.M.) was determined differently from that in the sleep control group (8:00 A.M.). It represents the time when vigilance hits a nadir after a night of SD (Doran et al., 2001; Graw et al., 2004). Most vehicular accidents following SD occur between 2:00 A.M. and 6:00 A.M. (Horne and Reyner, 1995). Cognitive function around this time could represent the social influence of SD. Despite the above considerations and information, the current data cannot rule out the possibility of clock-dependent changes in behaviors. The results may represent a combination of homeostatic and circadian influences on neurobehavioral performance.

Instrumental task

The instrumental task was based on the work of the following: de Wit et al. (2009), Gillan et al. (2011), and Sjoerds et al. (2013), which includes training phase, slips-of-action test, baseline test, and explicit knowledge questionnaires (Fig. 2).

Figure 2.

Instrumental learning task description. A, The training phase. Participants were shown various closed boxes with one kind of fruit (i.e., the stimulus) on the outside of each box. They were required to press the correct key (left key or right key, i.e., the response) to open the box, and obtain another fruit (i.e., the outcome) inside the box and subsequently earn points. If the response was incorrect, an empty box was shown with no points given. Participants learned the stimulus-outcome-response associations by trial and error. B, Three discrimination types (standard, congruent, and incongruent discriminations) were used for this task. C, The slips-of-action test. This test directly assessed the balance between goal-directed and habitual learning systems. First, six open boxes with outcomes inside were shown, two of which were marked with crosses. These two outcomes would lead to the subtraction of points, and participants should withhold any response to the corresponding stimuli (No-Go trials), whereas the other four outcomes were valuable and participants should make the correct response (Go trials). D, Baseline test of inhibition control. This was a control test of slips-of-action test for general inhibitory impairments. First, six closed boxes with stimuli onside were shown, two of which were marked with crosses. These two stimuli would lead to the subtraction of points and participants should withhold any response to these two boxes (No-Go trials), whereas the other four stimuli were valuable and participants should make the correct response (Go trials). E, Explicit knowledge questionnaire. Participants were examined on the corresponding response and outcome for each stimulus with paper-and-pencil questionnaires.

Instrumental training phase.

During the training phase, participants learned by trial and error to respond (R) to stimuli (S) to gain outcomes (O), that is, the stimulus-outcome-response (S-O-R) associations. As shown in Figure 2A, in each trial, a box with one kind of fruit on it appeared on the screen for 2 s, which functioned as a stimulus. The participants were required to press the left key or right key (response) to open the box. If their response was correct, the box would open, and the participants obtained another fruit (outcome) inside the box and earned points. If the response was incorrect, an empty box was shown, and no points were awarded. The feedback was always shown for 1 s. Based on this trial-and-error procedure, the participants learned the correct response for each stimulus and corresponding outcome. For each trial, the reaction time was limited to 2 s. If the reaction time exceeded 2 s, then the screen showed “Timed out” for 1 s. Timeout trials were not included in the statistical analysis.

The stimuli that were presented in the experiments were divided into three categories: standard, congruent, and incongruent discriminations (Fig. 2B). For standard discrimination, four different kinds of fruits were presented, which functioned as stimuli and outcomes. For congruent discrimination, the stimuli and outcomes were the same fruit pictures. For incongruent discrimination, each fruit functioned as the stimulus or outcome in different reactions. The training phase consisted of six blocks, and each stimulus appeared two times in each block in a random order, resulting in a total of 72 trials. The task used different types of discrimination to distinguish goal-directed and habitual systems that are involved in instrumental learning. For standard discrimination and congruent discrimination, both goal-directed (S-O-R) and habitual (Stimulus-Response [S-R]) systems were involved. For incongruent discrimination, the goal-directed system led to interference, and goal-directed and habitual strategies would produce conflict. Therefore, the participants would use the S-R strategy (i.e., habitual system) to guide learning behavior. We further recorded imaging data during instrumental learning to investigate the neural basis of behavioral changes that were caused by SD.

Slips-of-action test.

The purpose of this test was to assess whether participants were flexible to adjust their behavior to changes in the desirability of outcomes, that is, the balance between goal-directed and habitual actions. As shown in Figure 2C, at the start of each block, six open boxes that showed the outcomes appeared on the screen for 5 s. Two boxes (different types and corresponding to different responses) were marked with “×,” indicating that the outcomes in these two boxes were devalued and the corresponding stimuli no longer produced a score. Afterward, the participants were shown different closed boxes with stimulus pictures on them. If the outcome that corresponded to the stimulus was valuable (Go trial), then the correct response should be made; otherwise (No-Go trial), any response should be forbidden. Each closed box was shown for 2 s, and no feedback was given after the response to exclude the possibility of new learning. This test consisted of six blocks, and each block began with six open boxes, consisting of 24 trials each for a total of 144 trials. We compared the rates of response to stimuli that were associated with valuable and devalued outcomes. Selectively responding toward valuable outcomes was indicative of dominant goal-directed control, which was mediated by evaluation of the consequent outcome. Conversely, a failure to withhold responses to stimuli that were associated with devalued outcomes indicated predominance of the habitual system via direct S-R associations (Gillan et al., 2011; de Wit et al., 2012; Delorme et al., 2016). This was the most critical behavioral test in the present study.

Baseline test of response inhibition.

This test was used to determine whether impaired performance on the slips-of-action test could be due to a general inhibitory deficit. The procedure was similar to the slips-of-action test, with the exception that partial discriminative stimuli were devalued, rather than the outcomes (Fig. 2D). At the start of each block, six closed boxes that showed the stimuli appeared on the screen for 5 s. Two boxes (different types and corresponding to different responses) were marked with “×,” indicating that these two boxes were devalued and produced no score. This test consisted of six blocks, and each block began with six open boxes, consisting of 24 trials each for a total of 144 trials. Because performance did not depend on the ability to exert goal-directed control, a high rate of response to devalued stimuli indicated an impairment in response inhibition. This task was used as a control for the possibility that excessive response toward devalued outcomes in the slips-of-action test was purely related to outcome devaluation insensitivity (Geurts and de Wit, 2014; Delorme et al., 2016; Ersche et al., 2016). The order of the slips-of-action test and baseline test was random.

Explicit knowledge questionnaires.

A questionnaire (Fig. 2E) was applied to detect explicit knowledge about the S-R-O. It consisted of six questions, and each question consisted of an S-R part and an S-O part.

Working memory assessment

The present study adopted an Ospan task that was designed by Unsworth et al. (2005) to evaluate working memory capacity. In this task, the participants were required to perform simple arithmetic within the prescribed time and simultaneously remember the letters that appeared after each calculation. The result was the Ospan score: sum of letters selected for all correctly selected sets. The Chinese version of the Ospan task was used (http://englelab.gatech.edu/tasks.html).

fMRI data acquisition

Images were acquired using a GE-MR750 3.0 tesla scanner with a standard head coil in the MRI Research Center, Peking University. The scanning included functional and anatomical imaging. T2*-weighted functional images were acquired in 40 axial slices parallel to the AC-PC line with no interslice gap, affording full-brain coverage. Images were acquired using an EPI pulse sequence, with a repetition time of 2000 ms, echo time of 30 ms, flip angle of 90°, FOV of 192 mm × 192 mm, and 3 mm × 3 mm × 3 mm voxels. A high-resolution, whole-brain structural scan (1 mm³ isotropic voxel MPRAGE) was acquired after functional imaging.

fMRI data preprocessing and analysis

Image preprocessing and analysis were conducted using Statistical Parametric Mapping SPM8 software (Wellcome Trust Department of Cognitive Neurology, London). Images were slice time-corrected, motion-corrected, resampled to 3 × 3 × 3 isotropic voxels, normalized to MNI space, spatially smoothed using a 6 mm FWHM Gaussian filter, and temporally filtered using a high-past filter with 1/120 Hz cutoff frequency. Correction for temporal autocorrelations using AR(1) was also performed.

In the first-level (within-participant) analysis, events were modeled with boxcar regressors (duration = 2 s) convolved with a standard HRF. The six rigid body parameters were included to account for head motion artifacts. The responses for each trial type were modeled separately compared with fixation. Parameter estimates of the three trial types were then entered for second-level analysis (Sjoerds et al., 2013).

During the second-level (group) analysis, we first verified the previous results and assessed goal-directed learning in the SC group by contrasting the standard and congruent trials (both involving goal-directed and habit learning) against the incongruent trials (only involving habit learning), that is, [standard > incongruent] and [congruent > incongruent] contrasts. Second, with a full-factorial mixed ANOVA model, we compared the two groups with regard to goal-directed learning contrast [standard > incongruent]. Third, we additionally compared the habit learning with the incongruent trials (only involving habit learning) across the two experimental groups. First-level parameter estimates for the above contrasts were entered into the second-level analysis.

Whole-brain–based analyses were conducted for the imaging data. Only activation in areas of a priori interest were performed. Our a priori ROIs were ventromedial PFC (vmPFC), caudate, and putamen, because these areas are known to be involved in goal-directed and habit learning (de Wit et al., 2009; Brovelli et al., 2011; Sjoerds et al., 2013). For separate SC group analysis, the effects are reported at a level of significance of p < 0.05 at the cluster level using false discovery rate (FDR) correction. For analysis of the two groups, the main effects were reported at significance p < 0.05 whole-brain FWE corrected for multiple comparisons. Group comparisons and regression analysis were examined at p < 0.005 with a cluster extent of 5 voxels. Furthermore, group comparisons were corrected for multiple comparisons using small-volume correction with an 8 mm-radius sphere that was centered on coordinates derived from previous studies that reported brain areas of interest during instrumental conditioning: vmPFC (x = 0, y = 33, z = −12) (Gläscher et al., 2009) and (x = −8, y = 32, z = −8) (Sebold et al., 2017); and caudate (x = −18, y = 24, z = 6) (Brovelli et al., 2011).

Statistical analysis

Demographic data and behavioral performance were analyzed using SPSS 20.0 software (IBM). We used t tests to examine group differences in demographic data and psychological traits. The distribution of genders in the SC and SD groups was analyzed using the χ² test. Repeated-measures ANOVA was used for instrumental learning, slips-of-action test, baseline test, and explicit knowledge test data. Post hoc tests were corrected for multiple comparisons using Bonferroni correction. We assessed the association between task performance and working memory capacity and the level of attention in regression analysis. All of the tests were two-tailed. Values of p < 0.05 were considered statistically significant.

Results

Group characteristics

Demographic data, psychological traits, and sleep characteristics are shown in Table 1. In both experiments, independent-samples t tests revealed no significant differences in age, body mass index, years of education, BDI scores, HAMA scores, BIS scores (including all three dimensions), MoCA scores, working memory capacity, ESS scores, PSQI scores, or MEQ scores (all p > 0.1). No significant difference in gender distribution was found (p = 0.831 for Experiment 1, p = 0.738 for Experiment 2).

Sleepiness and psychomotor vigilance

Compared with the SC group, subjective sleepiness significantly increased in the SD group (p < 0.001 for both experiments). The SD group also presented a reduction of psychomotor vigilance, reflecting sustained attention, as indexed by a decrease in average response speed (p < 0.001 for both experiments) and an increase in lapses (reaction time ≥ 500 ms) (p < 0.001 for both experiments) in the PVT. These results indicate that the SD manipulation was successful (Table 2).

Table 2.

Sleepiness and psychomotor vigilance performance^a

	Experiment 1			Experiment 2
	SC (n = 22)	SD (n = 25)	p	SC (n = 17)	SD (n = 19)	p
Sleepiness	2.04 ± 1.67	7.90 ± 1.96	<0.001	1.87 ± 1.26	7.40 ± 2.43	<0.001
PVT
Average response speed	3.24 ± 0.24	2.71 ± 0.42	<0.001	3.30 ± 0.29	2.80 ± 0.32	<0.001
Lapses	3.39 ± 1.67	7.12 ± 3.39	<0.001	2.34 ± 1.17	5.35 ± 2.50	<0.001
Instrumental training phase
Nonresponses (%)	1.45 ± 1.32	2.06 ± 2.34	0.291	3.51 ± 1.81	4.82 ± 3.93	0.200

^aData are mean ± standard deviation.

Effects of SD on instrumental learning in Experiment 1

Discrimination training

We first compared the rates of nonresponses and found no significant difference between the SC and SD groups (t₍₄₅₎ = −1.068, p = 0.291; Table 2). During the training phase, the participants learned the S-O-R associations by trial and error. The three-way repeated-measures ANOVA (Group [SC, SD] × Congruence Type [congruent, standard, incongruent] × Block) revealed that accuracy between the SC and SD groups gradually increased as training progressed (main effect of Block: F_(5,225) = 69.750, p < 0.001), indicating a learning effect. No main effect of Group was found (F_(1,45) = 0.119, p = 0.732), with no Group × Block interaction (F_(5,225) = 1.410, p = 0.221; Fig. 3A). No difference in S-O-R learning was found between the SC group and SD group. In the last block, accuracy was above the level of chance (50%) for both groups. We also found a main effect of Congruence Type (F_(2,90) = 26.280, p < 0.001). The post hoc analysis revealed better accuracy for both congruent and standard types than incongruent type (all p < 0.001), with no difference between the congruent and standard types (p = 0.216). No Group × Congruence Type (F_(2,90) = 1.458, p = 0.238) or Group × Congruence Type × Block (F_(10,450) = 0.389, p = 0.951) interaction was found.

Figure 3.

Effects of SD on instrumental learning (Experiment 1). Response accuracy gradually increased (A) and reaction time decreased (B) over the six blocks of learning in the SC group (n = 22) and SD group (n = 25). Dotted line indicates the level of chance (50%). C, In the slips-of-action test, the SD group responded more often to stimuli that were associated with devalued outcomes compared with the SC group. D, The average differential score between responses to valuable outcomes minus responses to devalued outcomes was lower in the SD group. E, No group differences were found in the level of responding for valuable stimuli or devalued stimuli. F, No significant difference was found between the two groups in average differential scores between responses to valuable stimuli minus responses to devalued stimuli. G, Knowledge of associated outcomes was significantly worse in the SD group. Error bars indicate SEM. *p < 0.05, compared with SC group.

The repeated-measures ANOVA showed that reaction time gradually decreased across blocks (main effect of Block: F_(5,225) = 46.292, p < 0.001), with no main effect of Group (F_(1,45) = 0.090, p = 0.766) and no Group × Block interaction (F_(5,225) = 0.619, p = 0.685; Fig. 3B). The post hoc analysis showed that participants responded faster for the congruent and standard types than for the incongruent type (all p < 0.001), with no difference between the congruent type and standard type (p = 0.720). Therefore, no trade-off was observed between accuracy and reaction time during training.

Slips-of-action test and baseline test

During the slips-of-action test, the goal-directed and habitual systems competed for behavioral control, thus serving as an index of the balance between these two systems. Three-way repeated-measures ANOVA (Group [SC, SD] × Value [valuable, devalued] × Congruence Type [congruent, standard, incongruent]) revealed a significant main effect of Value (F_(1,45) = 680.712, p < 0.001), suggesting that the participants responded significantly less to stimuli that were associated with devalued outcomes, indicating a pronounced devaluation effect. A trend toward a Group × Value interaction was found (F_(1,45) = 3.612, p = 0.064). Bonferroni-corrected post hoc analysis indicated no group difference in the level of responding to stimuli that were associated with valuable outcomes (F_(1,45) = 1.294, p = 0.261), but the level of responding to stimuli that were associated with devalued outcomes was higher in the SD group than in the SC group (F_(1,45) = 4.492, p = 0.040; Fig. 3C). Responses to devalued outcomes implied a lack of sensitivity to changes in outcome value and were thus indicative of the dominance of habitual response control, that is, more slips of action in the SD group we observed indicating an overreliance on the habit system.

We then calculated the differential score (i.e., difference between levels of responding for stimuli associated with valuable outcomes and devalued outcomes). A differential score of 100 indicated complete goal-directed behavior, and 0 indicated complete habitual behavior. The independent-samples t test showed that the level of goal-directed behavior in the SD group was significantly lower than that in the SC group (p = 0.046; Fig. 3D).

In the baseline test, the three-way ANOVA revealed a significant main effect of Value (F_(1,45) = 12121.555, p < 0.001) and a trend toward a Group × Value interaction (F_(1,45) = 3.054, p = 0.087). Post hoc analysis showed that the difference in responding between the SC and SD groups was not statistically significant for valuable stimuli (F_(1,45) = 3.564, p = 0.065) or devalued stimuli (F_(1,45) = 1.620, p = 0.210; Fig. 3E). No significant difference in differential score was found between the SC group and SD group (p = 0.117; Fig. 3F), indicating that the effects of SD on goal-directed/habitual control were not attributable to impairments in inhibitory control.

Explicit knowledge questionnaires

The three-way repeated-measures ANOVA (Group [SC, SD] × Question [response, outcome] × Congruence Type [congruent, standard, incongruent]) revealed a significant main effect of Congruence Type (F_(2,90) = 8.585, p < 0.001). The post hoc analysis showed that the participants' knowledge of the incongruent type was less than the congruent (p = 0.001) and standard (p = 0.020) types. A significant Group × Question interaction was found (F_(1,45) = 9.002, p = 0.004). Post hoc analysis showed that knowledge of the appropriate responses to the stimuli did not differ between the SC and SD groups (F_(1,45) = 0.800, p = 0.376), whereas knowledge of the associated outcomes was significantly worse in the SD group (F_(1,45) = 6.582, p = 0.014; Fig. 3G).

Correlation analysis of behavioral data in Experiment 1

The correlation analysis showed that the level of goal-directed action (differential score) in the SD group was positively correlated with working memory capacity (r = 0.463, p = 0.020; Fig. 4A), with no such correlation in the SC group (r = 0.105, p = 0.642). These results indicated that, in the SD group, participants with high working memory capacity presented more goal-directed control and avoided overreliance on the habitual system, whereas participants with low working memory capacity overly relied on habitual control and suffered more adverse effects from SD.

Figure 4.

Correlation analysis of behavioral data (Experiment 1). A, A positive correlation was found between working memory capacity and the average differential score between responses to valuable outcomes minus responses to devalued outcomes in the SD group (n = 25). No significant correlation was found between the differential score in the SD group and the degree of subjective sleepiness (B), number of attention lapses (C), or average reaction time (D) in the PVT. WM, Working memory.

To elucidate the relationship between behavioral changes and attention, we also performed correlation analysis between the differential score and subjective sleepiness and objective attention before the task. In the SD group, no correlations were found between the differential score and the degree of subjective sleepiness (r = −0.070, p = 0.739; Fig. 4B), number of attention lapses (r = 0.135, p = 0.520; Fig. 4C), or average reaction time (r = −0.117, p = 0.577) in the PVT (Fig. 4D). In addition, there was no significant correlation between the differential score and the rate of nonresponses during the training phase (r = −0.232, p = 0.264), which represented the attention level in the main task. These results suggest that the decline in sustained attention did not completely explain the effects of SD on goal-directed/habitual action.

Brain activation related to goal-directed learning in the SC group in Experiment 1

We first analyzed brain activation associated with goal-directed learning in the SC group. Contrast that corresponded to the main effect of goal-directed learning produced activation in the vmPFC ([standard > incongruent]: Z = 4.17, p_FDR < 0.05; [congruent > incongruent]: Z = 4.74, p_FDR < 0.05; Fig. 5A,C). Brain signals (parameter estimates) were extracted and plotted from the peak points, with significant differences between groups. As expected, because of the location of this region within the default mode network, it exhibited below-baseline activation (Fig. 5B,D, negative β coefficients). In both contrasts, the vmPFC exhibited higher activation in response to standard and congruent types compared with the incongruent type. These results indicate that the vmPFC is involved in the process of goal-directed learning under well-rested conditions.

Figure 5.

Goal-directed learning-related brain activation in the SC group (Experiment 1). A, Contrast between cue-outcome standard and incongruent trials produced activation in the vmPFC (MNI coordinate: 3, 54, −15). B, Parameter estimates of the peak voxel in the vmPFC for contrast [standard > incongruent]. C, Contrast between cue-outcome congruent and incongruent trials produced activation in the vmPFC (MNI coordinate: 3, 45, −9). D, Parameter estimates of the peak voxel in the vmPFC for contrast [incongruent > incongruent]. *p < 0.05, compared with incongruent trials. n = 22.

Comparisons of brain activation related to instrumental learning in the SD and SC groups in Experiment 1

Goal-directed learning [standard > incongruent]

The main effect of goal-directed learning showed vmPFC involvement (Z = 5.67, p_FWE < 0.05). Group comparisons showed lower vmPFC activation in the SD group than in the SC group (x = −3, y = 30, z = −9; Z = 3.33, p_FWE-SVC = 0.016; Fig. 6A). The peak β weights were significantly higher in the SC group (t₍₄₃₎ = 2.758, p = 0.009; Fig. 6B). The SD group exhibited no regions with greater activation than the SC group.

Figure 6.

Comparisons of vmPFC activation during goal-directed behavior in the SD group and SC group (Experiment 1). A, Activation of the vmPFC (MNI coordinate: −3, 30, −9) was significantly higher in the SC group (n = 22) than in the SD group (n = 23) in the contrast of [standard > incongruent], which presented goal-directed learning. Displayed at p < 0.005 whole-brain uncorrected, extent threshold > 5 voxels. B, Parameter estimates were significantly lower in the SD group. C, Activation of the vmPFC during goal-directed learning was positively correlated with the level of goal-directed performance in the test in the SD group. **p < 0.01, compared with SC group.

Habitual learning (incongruent trials)

We additionally compared the groups with regard to habit learning and found that the caudate was more active in the SC group than in the SD group (x = −15, y = 21, z = 12; Z = 3.45, p_FWE-SVC = 0.011). The SD group did not present greater activation in the areas of interest compared to the SC group. Therefore, our data indicated that the imbalance between goal-directed action and habitual action after SD resulted from a deficit in goal-directed action but not an exaggeration of habitual action.

Correlation analysis between imaging and behavioral results in Experiment 1

Because the level of goal-directed action in the slips-of-action test (responding for valuable outcomes minus responding for devalued outcomes) was the most critical behavioral testing indicator in the present study, we performed a correlation analysis between vmPFC activation during goal-directed learning contrast [standard > incongruent] and the level of goal-directed action. The results showed that vmPFC activation was positively correlated with the subsequent behavioral performance in the SD group (r = 0.519, p = 0.011; Fig. 6C), whereas this correlation was not significant in the SC group (r = 0.279, p = 0.208).

Effects of SD on the expression of goal-directed and habitual action

Based on the above results in Experiment 1, goal-directed action was impaired after SD. However, the results did not reveal whether SD affected instrumental processes that are involved in formation or expression. Thus, we varied the timing of instrumental learning in Experiment 2.

We first compared the rates of nonresponses and found no significant difference between the two groups (t₍₄₅₎ = −1.308, p = 0.200; Table 2). During the training phase, the three-way ANOVA (Group × Congruence Type × Block) showed a learning effect (main effect of Block: F_(5,170) = 52.032, p < 0.001), with no main effect of Group (F_(1,34) = 0.001, p = 0.981) and no Group × Block interaction (F_(5,170) = 0.439, p = 0.821; Fig. 7A), which was similar to Experiment 1.

Figure 7.

Effects of SD on the expression of goal-directed and habitual actions (Experiment 2). A, Response accuracy gradually increased over the six blocks of learning in both SC group (n = 17) and SD group (n = 19). B, In the slips-of-action test, the levels of responding for stimuli that were associated with valuable outcomes and devalued outcomes were similar in the two groups. C, No group differences were found in the level of responding for valuable stimuli or devalued stimuli. D, Knowledge of the associated responses and outcomes did not differ between the two groups. Error bars indicate SEM.

In the slips-of-action test, the three-way ANOVA (Group × Value × Congruence Type) showed no main effect of Group (F_(1,34) = 3.098, p = 0.085) and no Group × Value interaction (F_(1,34) = 0.331, p = 0.569). Post hoc analysis showed no group differences in the level of responding for stimuli that were associated with valuable (F_(1,34) = 0.089, p = 0.767) or devalued (F_(1,34) = 0.625, p = 0.435) outcomes (Fig. 7B). Similarly, no group differences were found in the baseline test (F_(1,34) = 0.247, p = 0.622 for valuable stimuli; F_(1,34) = 0.084, p = 0.774 for devalued stimuli; Fig. 7C). With regard to explicit knowledge, no significant differences in appropriate responses to the stimuli (F_(1,34) = 0.000, p = 1.00) or knowledge of the associated outcomes (F_(1,34) = 0.062, p = 0.804) were found (Fig. 7D).

The results of Experiment 2 suggested that, after successfully learning goal-directed actions, SD did not affect its expression, thus indicating the adverse effects of SD on the formation of goal-directed action.

Discussion

The present study provides evidence of the effects of SD on instrumental learning. We found that SD did not affect the ability of individuals to adjust their decision-making behavior based on feedback. However, SD affected the perception of the content of outcomes, consequently making the participants less sensitive to changes in outcome value and eventually leading to overreliance on habitual control. The neuroimaging results indicated that the vmPFC mediated the behavioral changes in sleep-deprived people.

Adverse effects on cognition after SD

Unlike the broad consensus in the alertness and attention cognitive domain, the idiographic detrimental effects of sleep loss on many higher-level cognitive capacities, including decision-making and inhibitory control, are far more complex (Jackson et al., 2013). This complexity reflects the fact that cognitive performance involves numerous component processes (e.g., rational influences and emotional factors) and the heterogeneity of research tasks. Taking decision-making as an example, it was found that sleep-deprived subjects inclined to prefer riskier selections (Killgore et al., 2006, 2007b), whereas other evidence showed that SD altered decision-making by altering information strategies without altering risk preferences (Mullette-Gillman et al., 2015). A recent event-related potential study showed that SD influenced decision-making by affecting individuals' responses to feedback stimuli (Liu and Zhou, 2016). This raises an assumption that the detrimental effects of sleep loss on decision-making are, at least in part, due to impairments in perception and the application of feedback information.

In the present study, an instrumental learning and outcome devaluation paradigm was used. This required the coding of outcome value, to investigate the effects of SD on cognition. In Experiment 1, both groups learned S-R associations equally, given that response accuracy did not differ between the two groups in the training phase. Furthermore, the explicit knowledge of S-R associations did not differ between the groups in the explicit knowledge test. With regard to S-O associations, the slips-of-action test indicated that sleep-deprived participants were insensitive to changes in outcome value and presented inferior explicit knowledge of S-O associations, in which SD made individuals more prone to habitual control. In other words, SD altered learning strategies in the instrumental behavior task, from a learning strategy mainly based on goal-directed S-O-R associations to one mainly based on habitual S-R associations.

A recent study employed a reversal learning decision task (an adapted version of Go/No Go task) and found that sleep-deprived subjects exhibited profound impairments in adapting to abrupt changes in feedback contingencies (Whitney et al., 2015). Skin conductance responses to outcome feedback also decreased, suggesting that a reduction of affective valence of the feedback was related to impairments in reversal learning decision-making. This is consistent with the main behavioral findings of our study, in which SD affected the processing of cognitive and/or affective dimensions of feedback information (Venkatraman et al., 2007), resulting in a lack of sensitivity to response outcomes (i.e., what was contained in the box).

We also found that SD affected the formation but not expression of goal-directed action. This is consistent with previous findings that the maintenance of successful coded stimuli in sleep-deprived individuals was the same as in control subjects (Whitney et al., 2015). Recently, Liljeholm et al. (2015) found that the acquisition and expression of goal-directed/habitual control had different neural correlates. Animal studies suggested that damage to the mPFC impaired the acquisition of goal-directed learning rather than its expression (Ostlund and Balleine, 2005). Therefore, we speculate that the acquisition and expression of goal-directed learning are mediated by different brain regions that are differentially affected by SD.

Working memory capacity predicts the effects of sleep deprivation on goal-directed action

Working memory is part of the central executive and involves the temporary storage and manipulation of information. Previous studies have shown that goal-directed control mainly depends on the PFC, orbitofrontal cortex, and other brain regions (Otto et al., 2013). In the present study, we explored the relationship between baseline working memory capacity and the balance between the goal-directed and habitual systems after SD. The results suggested that, although the overall level of goal-directed control decreased after SD, individuals with higher working memory capacity presented more goal-directed control and avoided excessive reliance on the habitual system. Numerous studies have reported that the susceptibility of an individual's cognitive performance to sleep loss is trait-dependent (Van Dongen et al., 2004; Killgore et al., 2007a). The results indicated that working memory capacity at baseline predicted the extent to which one's goal-directed behavior was affected by SD.

This raises the question of whether the effects of SD on goal-directed behavior are mediated by working memory. The present experimental design cannot reliably resolve this issue because we did not measure working memory after the SC and SD conditions. However, a wealth of evidence suggests that working memory is indeed impaired by SD. Behaviorally, a meta-analysis found that SD adversely affected both accuracy and response time during working memory tasks (Lim and Dinges, 2010). From a neuroimaging mechanism perspective, impairments in working memory during SD were reported to be associated with reductions of activity in the dorsolateral prefrontal cortex and posterior parietal cortex (Chee and Choo, 2004; Choo et al., 2005; Chee and Chuah, 2007; Lythe et al., 2012).

Notably, no significant correlation was observed between baseline working memory capacity and the level of goal-directed control in the SC group. A reasonable explanation is that processing capacity can cope with general cognitive tasks under normal circumstances (e.g., after adequate sleep). In the case of sleep loss, individuals need to expend a portion of their capacity to combat the negative effects of sleep loss during long periods of wakefulness, thus reducing the ability to process additional information (Kong et al., 2011). Consequently, cognition in individuals with low working memory capacity is more likely to be affected by sleep loss.

Effects of SD on vmPFC function

In the present study, SD affected vmPFC activation during goal-directed learning, and the level of activation was positively correlated with the level of goal-directed behavior. The vmPFC is involved in various cognitive processes, such as decision-making (Hare et al., 2009), emotion regulation (Mobbs et al., 2007), and learning and memory (Gerraty et al., 2014). Animal studies showed that the vmPFC also plays an important role in updating the value of rewards (Laskowski et al., 2016). Using a multivariate classification analysis approach, McNamee et al. (2015) found that the vmPFC and other brain areas were involved in the information processing of associations among stimuli, responses, and outcomes in goal-directed learning, whereas brain areas that are involved in habitual learning (e.g., posterior putamen) did not participate in processing outcomes.

Insufficient sleep can affect the function of the vmPFC, which has been demonstrated in multiple models. The value signal represented in the vmPFC was altered in sleep-deprived individuals during decision-making tasks (Libedinsky et al., 2011). In patients with insomnia, mPFC activation was lower than in healthy controls during a category fluency task, which recovered after sleep therapy (Altena et al., 2008). In the present study, the degree of vmPFC involvement in goal-directed learning was reduced after SD, resulting in impairments in goal-directed control.

Limitations of our study

Our findings should be acceptable with the following limitations. First, we applied a between-subjects design. A within-subjects design may allow a better assessment of interindividual differences. However, the choice of a within-subjects or between-subjects design depends considerably on the specific research question and specific tests. Some domains do not lend themselves to be repeatedly measured due to the learning effects. Such potential learning effects would be more pronounced in the SD condition because there might be a ceiling effect in the SC condition, which would obscure the effects of SD. Future repeated-measures experiments with a within-subjects design will be necessary to investigate whether there are learning effects with multiple exposures to the tasks we currently use and how much impact they will have on the findings. Second, we used different test time schedules in SC (at 8:00 A.M.) and SD (at 6:00 A.M.) groups. This was designed to simulate the difference in performance obtained at the start of the workday and the effect of having to work through a night shift without sleep. To be precise, both homeostatic and circadian effects may exist in the present study. However, previous findings indicate that the homeostatic pressure caused by extended waking is sufficient to mitigate the impact of circadian clock-dependent alerting upon cognitive performance (Wright et al., 2012). Our findings are unlikely to be justified solely by circadian effects.

In conclusion, sleep-deprived individuals overrelied on habitual learning at the expense of goal-directed learning, which was mediated by the vmPFC. The extent to which SD affected goal-directed action could be predicted by working memory capacity. These findings broaden our understanding of the adverse effects of SD on cognition, especially learning strategies, and the neural basis of sleep-related disorders.