New insights into the cognitive effects of sleep deprivation by decomposition of a cognitive throughput task

Abstract

Study Objectives

A cognitive throughput task known as the Digit Symbol Substitution Test (DSST) (or Symbol Digit Modalities Test) has been used as an assay of general cognitive slowing during sleep deprivation. Here, the effects of total sleep deprivation (TSD) on specific cognitive processes involved in DSST performance, including visual search, spatial memory, paired-associate learning, and motor response, were investigated through targeted task manipulations.

Methods

A total of 12 DSST variants, designed to manipulate the use of specific cognitive processes, were implemented in two laboratory-based TSD studies with N = 59 and N = 26 subjects, respectively. In each study, the Psychomotor Vigilance Test (PVT) was administered alongside the DSST variants.

Results

TSD reduced cognitive throughput on all DSST variants, with response time distributions exhibiting rightward skewing. All DSST variants showed practice effects, which were however minimized by inclusion of a pause between trials. Importantly, TSD-induced impairment on the DSST variants was not uniform, with a principal component analysis revealing three factors. Diffusion model decomposition of cognitive processes revealed that inter-individual differences during TSD on a two-alternative forced choice DSST variant were different from those on the PVT.

Conclusions

While reduced cognitive throughput has been interpreted to reflect general cognitive slowing, such TSD-induced impairment appears to reflect cognitive instability, like on the PVT, rather than general slowing. Further, comparisons between task variants revealed not one, but three distinct underlying processes impacted by sleep deprivation. Moreover, the practice effect on the task was found to be independent of the TSD effect and minimized by a task pacing manipulation.

Statement of Significance.

The effects of sleep deprivation on task performance are frequently described in terms of overall changes in cognitive functioning (e.g. cognitive slowing or cognitive instability), independent of the cognitive processes involved. After dissociating the practice effect from the sleep deprivation effect on a cognitive throughput task, we found that sleep deprivation induces cognitive instability in this task, comparable to the instability seen in Psychomotor Vigilance Test performance. This suggests shared underlying mechanisms between the two tasks and casts doubt on the common interpretation of reduced cognitive throughput as reflecting general cognitive slowing. However, we also found that sleep deprivation causes cognitive process-dependent deficits, suggesting the involvement of distinct neural pathways that show differential vulnerability to sleep deprivation.

An extensive body of empirical literature has documented systematic declines in human performance during sleep deprivation across a wide range of performance tasks and cognitive domains [1]. These pervasive performance declines have been seen as evidence to support theories of sleep loss operating at the global system level. Theoretical perspectives commonly discussed in the literature posit that the effects of sleep loss on performance are due to failure to maintain effort to perform [2]; slowed cognitive processing [3]; lapses of attention [4]; and/or increased attentional variability due to wake state instability [5]. These theories have provided useful characterizations of the kinds of degradations that are observed when people are asked to perform cognitive tasks under conditions of sleep deprivation.

These characterizations may be insufficient, however, as they do not explain a number of phenomena documented over the last two decades. For example, distinct components of cognition, such as sustained attention, working memory, and attentional control, are differentially susceptible to degradation due to sleep deprivation [6–9]. Also, inter-individual differences in the degree of performance degradation during a given period of sleep deprivation can vary across different cognitive tasks, such that a person may be resilient to impairment from sleep loss on one task yet vulnerable on another [10]. These and other observations [11–16] indicate that global, system-level theories of the effects of sleep deprivation on cognition are incomplete, and that both global and local effects must be considered to understand the impact on performance [17].

Psychological principles underscore the importance of evaluating the effects of sleep deprivation on individual task components, in order to address the “task impurity problem” of intertwined effects of sleep deprivation on distinct cognitive processes [18]. A recently proposed bottom-up perspective on sleep deprivation and performance, whereby overall impairment is an emergent property from the effects of sleep deprivation on the underlying cognitive processes [19], is compatible with theories developed within cognitive science [20, 21]. Research using integrated computational theories has yielded detailed accounts of the deleterious effects of sleep loss on different cognitive processes and overall performance [22–25]. A key implication of this research is that understanding the effects of sleep deprivation on performance of a given task requires careful analysis of task requirements to assess the role of different components of cognition in performance [17]. In the present article, we pursue this for a cognitive throughput task. The specific task is herein referred to as the Digit Symbol Substitution Test (DSST) for consistency with previous sleep deprivation literature [26, 27], although this task has also been referred to as the Symbol Digit Modalities Test [28, 29] or the Symbol Digit Substitution Task [30, 31].

The DSST requires matching of stimuli consisting of symbols and digits to each other using a given digit–symbol pairing key, as fast as possible while remaining accurate. The task was originally developed in the context of intelligence testing to measure paired-associate learning through repetition [32]. Since then, the task has been used primarily as a measure of cognitive throughput—defined as the number of correct responses within an allotted time interval—which is presumed to be a measure of cognitive processing speed [33]. Cognitive throughput on the DSST decreases with age, which has been interpreted as evidence of cognitive slowing with aging [34]. To what extent the effects of sleep deprivation on cognition are equivalent to those of aging is an outstanding issue of debate [35, 36].

A computerized version of the DSST, here referred to as the Standard DSST (Figure 1), has been used to investigate changes in performance due to sleep loss [37]. In this version, subjects type digits (1–9) corresponding to symbols shown in a digit–symbol pairing key presented at the top of the screen, using the index finger of their dominant hand to respond. This is an inverse form of the original paper-and-pencil version [32], which required subjects to draw the symbol associated with a provided digit. Cognitive throughput on the Standard DSST decreases with total sleep deprivation (TSD) and sustained sleep restriction [26, 29], which has been interpreted as evidence that sleep loss results in slowed cognitive processing [27]. In the absence of sleep loss, performance on the Standard DSST improves with repeated exposure to the task, revealing a practice effect that continues across dozens of task administrations [26]. Whether the brain mechanisms underlying the practice effect are related to those underlying the sleep deprivation effect on DSST performance has remained unclear.

Figure 1.

Sample screenshot of the computerized Standard DSST. A probe symbol is presented in the box at the bottom of the screen (here: plus symbol). The subject’s task is to respond by typing the corresponding digit using the digit–symbol pairing key shown at the top of the screen (here: correct response is “3”). After responding, a new probe symbol is presented.

While the task itself is relatively simple and straightforward, from a cognitive science standpoint, the Standard DSST is a more complex task, in which good performance relies on multiple, distinct components of cognition. These include, among others, visual search, spatial memory, paired-associate learning, and motor action (as described in Table 1). In two laboratory studies conducted for the present research, we sought to determine whether any of these four components of cognition are specifically the source of sleep deprivation-induced performance impairment on the DSST, or whether there may be a global effect on cognition that non-specifically impairs all cognitive functions. We therefore developed six variants of the DSST with the purpose of dissociating the effects of sleep deprivation on the four component processes, as shown in Table 2.

Table 1.

The four component processes isolated and manipulated in the current research

Component	Function	Example use
Visual search	Used when the subject scans the digit–symbol pairing key to find the current probe symbol	“Where is the plus symbol?”
Spatial memory	May gradually be incorporated across the task duration as the subject learns the approximate location of a symbol in the digit–symbol pairing key, allowing faster location of the probe symbol and associated digit	“The plus symbol is toward the left side of the digit–symbol pairing key.”
Paired- associate learning	May also be gradually developed across the task duration as the subject learns the digit–symbol pairing, allowing faster responding through reduced need to consult the digit–symbol pairing key	“The plus is paired with the number 3.”
Motor action	Used when the subject moves the index finger of the dominant hand to press the desired number key on the top of the keyboard	Using the index finger of the dominant hand to press the number 3.

Table 2.

The six DSST variants used in the present research, including name, description, and components manipulated in each, as well as two example screenshots

We compared performance on the six variants of the DSST to each other, and to a well-established assay of vigilant attention, the Psychomotor Vigilance Test (PVT) [3]. Performance on the PVT during sleep deprivation is characterized by a pronounced time-on-task effect involving increasing response variability across the 10-min task duration [5]. Previous research has found that inter-individual differences in performance impairment due to sleep deprivation do not align between the Standard DSST and the PVT, when compared among a panel of 13 neurobehavioral tests [10]. This discrepancy could be seen as evidence of the involvement of distinct cognitive processes in the two tasks and/or that these tasks behaved more similarly to other tasks in the panel than to each other [19]. While the DSST is traditionally viewed as a measure of cognitive throughput or processing speed [33], the PVT is considered a measure of vigilant attention [3]. Both tasks include basic components of information processing, such as visual stimulus detection, but also have different task-specific processes. These include visual search, spatial memory, paired-associate learning, and certain motor response components for the DSST (see Table 1); and vigilant attention for the PVT. The Standard DSST and the PVT also differ in the pacing of the task, where the Standard DSST is subject-paced (each response immediately begins the next trial) and the PVT is computer-paced (each response is followed by a computer-driven delay before the next trial). In the case of the PVT, this delay between trials includes a variable inter-trial interval. Therefore, to investigate the impact of task pacing on DSST performance during sleep deprivation, we developed a second set of six variants of the DSST, which differed from the first set in that the variants in the second set were not purely subject-paced. That is, in the first set the next stimulus was presented immediately after each response, whereas in the second set there was a fixed computer-pacing with a 1-s delay prior to presentation of the next stimulus.

In Study 1, we administered the six subject-paced variants of the DSST, and the PVT, during 38 h of TSD. In Study 2, we administered the six DSST variants with 1-s delays, and the PVT, during 62 h of TSD. Using the data from these two studies, we set out to investigate the following five aims:

• (1) To determine the effect of sleep deprivation on response time (RT) distributions on the DSST, and what it tells us about the DSST as an assay of cognitive processing speed during sleep deprivation;

• (2) To examine the nature of the practice effect on the DSST, and the extent to which it is related to the effect of sleep deprivation on DSST performance;

• (3) To examine the effects of sleep deprivation on distinct cognitive components involved in performance of the DSST and PVT, and what they tell us about the underlying neural processes driving these cognitive components;

• (4) To compare the role of subject- versus computer-based pacing, as well as fixed- versus variable-computer-based pacing, in the DSST and PVT performance impairment due to sleep deprivation, and determine what that reveals about the underlying brain mechanisms; and

• (5) To determine what are the unifying mechanisms, if any, that underpin performance impairment in the DSST and the PVT during sleep deprivation, and what are the theoretical implications for the effects of sleep deprivation on brain functioning.

Methods

A total of 86 subjects underwent one of two laboratory-based TSD studies. These studies have been previously described [8, 13], but the DSST data are presented here for the first time. During each of the two studies, cognitive performance on six DSST variants was measured across a period of sustained wakefulness. Both studies were conducted at the Sleep and Performance Research Center of Washington State University, Spokane and approved by the Washington State University Institutional Review Board. Subjects provided written, informed consent, and were financially compensated for their participation.

TSD studies

Study 1

N = 59 healthy young adults (mean age ± SD = 27.2 ± 4.6 y, 30 females) stayed in the laboratory for 4 consecutive days (3 nights) in groups of up to 4. They were randomly assigned, with a 2:1 ratio, to either a TSD condition (n = 39) or a well-rested control condition (n = 20). Subjects in the TSD condition had 10 h time in bed (10:00 pm–08:00 am) for baseline sleep on the first night and for recovery sleep on the third night, with 38 h TSD between. Subjects in the control condition had 10 h time in bed (10:00 pm–08:00 am) during each of the three nights in the laboratory (Figure 2, top). Three subjects were omitted from analyses; for two subjects, there was a technical issue that affected randomization, so that the DSST variants were not administered the correct number of times, and one subject had poor compliance and a high proportion of errors across all stages of the protocol. The final sample, therefore, was comprised of N = 56 subjects, with n = 37 subjects in the TSD condition and n = 19 subjects in the control condition. One subject in the TSD condition did not use the correct response keys in the Two-Alternative Forced Choice (2AFC) variant of the DSST; this subject was therefore excluded from the 2AFC DSST analyses only.

Figure 2.

Experimental design of Studies 1 and 2. White bars indicate scheduled wake periods. Black bars indicate scheduled sleep periods (10:00 pm–08:00 am); gray bars indicate scheduled sleep periods (10:00 pm–08:00 am) for subjects in the control condition only. Numbers indicate sessions with administrations of each of the six DSST variants. Every session (0–5 in Study 1 and 0–15 in Study 2) consisted of two test bouts represented by black dots connected with a dashed line. Each test bout included three DSST variants so that each session contained the full complement of six variants. Session 0 was used for practice in each study and was not included in the analyses. In Study 1 (top), sessions 1 and 2 were administered during the baseline period, sessions 3 and 4 during the TSD period or well-rested control period, and session 5 during the recovery period. In Study 2 (bottom), sessions 1 through 5 were administered during the baseline period, sessions 6 through 10 during the TSD period or well-rested control period, and sessions 11 through 15 during the recovery period. In Study 1, subject-paced versions of the DSST variants were administered; in Study 2, there was a 1-s delay after each response. In both studies, each test bout (black dot) also included a PVT.

Study 2

N = 26 healthy young adults (mean age ± SD = 25.9 ± 4.0 y, 10 females) stayed in the laboratory for 7 consecutive days (6 nights) in groups of up to 4. They were randomly assigned to either a TSD condition (n = 13) or a well-rested control condition (n = 13). Subjects in the TSD condition had 10 h time in bed (10:00 pm–08:00 am) for baseline sleep on the first two nights and for recovery sleep on the last two nights, with 62 h TSD between. Subjects in the control condition had 10 h time in bed (10:00 pm–08:00 am) during each of the six nights in the laboratory (Figure 2, bottom).

Subjects

Subjects were healthy males and females recruited from the community, aged between 22 and 40 years. Subjects were physically and psychologically healthy as assessed by physical examination and history, with no clinical disorders and/or illnesses and no current medical or drug treatment, excluding oral contraceptives. Blood chemistry, urine drug screen, and breathalyzer test showed no clinically significant abnormalities or traces of drugs or alcohol. Subjects reported no current psychiatric illness, no clinically relevant history of psychiatric illness, no history of drug or alcohol abuse in the past year, no history of methamphetamine abuse, were not current smokers, were not vision or hearing impaired, unless corrected to normal, and had no history of moderate to severe brain injury, learning disability, or previous adverse reaction to sleep deprivation. Female subjects were not pregnant. All subjects were proficient English speakers. No subjects participated in both Studies 1 and 2.

Subjects were normal sleepers, with normal baseline polysomnogram and no sleep or circadian disorders. They reported good habitual sleep between 6 and 10 h in duration, regular bedtimes, and habitual wake times between 06:00 am and 09:00 am. They did not travel across time zones within 1 month of entering the laboratory, and reported no shift work within 3 months of entering the laboratory. During the week prior to entering the laboratory, subjects were required to maintain their habitual sleep/wake schedule and abstain from caffeine, alcohol, drugs, and napping. This was documented by means of wrist actigraphy (Actiwatch 2, Philips Respironics, Bend, OR, USA), sleep/wake diary, and a time-stamped voicemail box subjects called daily upon waking and when going to bed. Prior to entering the sleep laboratory, a urine drug screen and breathalyzer test were repeated to confirm that subjects remained free of traces of drugs and alcohol.

Between subjects in Study 1 and Study 2, there was no significant group differences in age (p = 0.24) or sex (p = 0.26). In each study, subjects averaged more than 7 h sleep per night (as recorded with wrist actigraphy) during the week prior to entering the laboratory. Further, for total sleep time (as recorded with polysomnography) during in-laboratory baseline sleep on the night immediately prior to sleep deprivation, there was no difference between subjects in the TSD-condition of Study 1 versus Study 2 (p = 0.23), see Supplementary Table S1.

Experimental procedures

While in the laboratory, each subject had his/her own room for sleeping and performance testing. Outside of sleep and test times (see below), subjects were in a common suite area inside the laboratory, where they could engage in non-vigorous activities, such as reading, board games, and watching DVD movies. The laboratory environment was highly controlled to shelter subjects from external influences. Ambient temperature was maintained at 21°C (± 1°C), light levels were fixed below 100 lux during wakefulness (and lights were off during scheduled sleep), and subjects had no contact with individuals outside the laboratory. They had no access to telephones, smartphones, personal computers, internet, live television, or radio. Subjects were monitored by trained research assistants at all times to monitor compliance with the study procedures and to ensure continuous wakefulness outside of designated sleep periods.

A computerized cognitive test battery was administered repeatedly in each study (Figure 2). Every administration of the test battery included three of the six DSST variants. Thus, subjects completed one set of all six tasks across consecutive test bout pairs; these test bout pairs are hereafter referred to as sessions. The order of the DSST variants across the sessions was randomized over subjects using a Latin Square design. In Study 1, the subject-paced versions of the DSST variants were administered; in Study 2, the versions with the 1-s delay after each response were used. In both studies, every administration of the test battery also included the PVT (so there were two PVT administrations in each session).

DSST variants

Six variants of the computerized DSST were implemented in each study. All task variants were 4 min in duration. For each variant, subjects were instructed to try to enter as many correct answers as possible within the allotted time. They were also instructed to use only the index finger of their dominant hand and the number keys across the top of the keyboard, except where otherwise noted.

In the Standard version of the task (Figure 1), subjects were shown a randomized digit–symbol pairing key on the top of the screen, with the digits (1–9) ordered from left to right. The key remained the same for the duration of each task, but varied from session to session (i.e. the number 1 could have been paired with the square symbol in the first 4-min task administration, but with a different (random) symbol in the next task administration). In a box near the bottom of the screen, one of the symbols in the key appeared as a probe. Subjects responded by entering the corresponding digit. After a response was made, a randomly selected new probe (which could not be the same as the previous probe) was presented. This was repeated through to the end of the task duration.

Subjects were provided with visual feedback on screen as to the correctness of their responses during the first 10 trials of each task administration in Study 1 and during all trials in Study 2. In Study 1, following the first 10 trials, the next stimulus was presented immediately after each response. In Study 2, there was a 1-s delay prior to presentation of the next stimulus, and the pacing of the task was thus altered—but otherwise performance testing was the same.

The DSST variants were all based on the Standard DSST, but they featured strategically selected task variations as described in Table 2. Through juxtapositions of the DSST variants administered during the two studies, the effects of sleep deprivation on distinct components of cognition could be investigated: visual search, spatial memory, paired-associate learning, and motor response. In general, the effect of TSD on a particular cognitive function cannot be isolated, as removing one cognitive component from the task changes the reliance on other functions (e.g. preventing the use of paired-associate learning increases dependence on visual search and/or spatial memory). However, comparing variants with and without the potential use of a cognitive function does reveal whether that specific function is involved in the effects of TSD on DSST performance. For example, comparison of the effects of sleep deprivation on the Standard and Moving Pairs variants of the DSST (see Table 2) elucidates whether spatial memory is involved in DSST performance impairment due to TSD. Similarly, use of a particular cognitive function may be amplified in a task variant, so that any differential effects may demonstrate whether that specific function is involved in the effects of TSD on DSST performance. For example, the effect of sleep deprivation on the Memory DSST (see Table 2) elucidates whether paired-associate learning is involved in DSST performance impairment due to TSD.

Psychomotor Vigilance Test

A computerized, 10-min PVT with a random inter-trial interval of 2–10 s was administered in each test bout prior to the set of three DSST variants. Subjects were instructed to respond by pressing a button on a response box as quickly as possible after each stimulus presentation and without making any false starts (i.e. without responding before the stimulus appeared).

RT distributions

Performance changes on DSST variants and the PVT were investigated in detail using RT distributions. This is important to distinguish possible underlying mechanisms. For example, general cognitive slowing, as is found with aging [38], would be associated with a shift to the right of the entire RT distribution (Supplementary Figure S1, left). In contrast, cognitive instability, as is found with sleep deprivation [5], would involve a skewing of the distribution to the right, such that the fastest RTs are relatively unaffected, but the right tail of the distribution becomes more prominent as slow RTs become more frequent and slower (Supplementary Figure S1, right) [39, 40].

Cumulative relative RT frequency distributions were generated for the Standard DSST, Digit Only DSST, and PVT for the TSD condition in Studies 1 and 2. These task variants were selected to illustrate comparisons between DSST variants when all potential cognitive processes are brought to bear (Standard DSST) and when minimal cognitive processes are brought to bear (Digit Only DSST), as well as comparisons against the PVT, which is most similar in underlying processes to the Digit Only DSST and has known TSD effects on RT distributions, including increased RT variability shown with rightward skewing [40]. For Study 1, RTs from morning test sessions were aggregated by phase (Baseline—Session 1; TSD—Session 3; Recovery—Session 5; see Figure 2, top). For Study 2, RTs were aggregated using the second morning test sessions per phase (Baseline—Session 4; TSD—Session 9; Recovery Session 14; see Figure 2, bottom). In the Standard DSST and Digit Only DSST, RTs were aggregated from trials with correct responses and RTs > 150 ms only. In the PVT, RTs were aggregated from trials with RTs > 150 ms (no false starts).

In order to compare different possible effects within the RT distributions, the RTs were calculated for the fastest 10 per cent, median RT, and slowest 10 per cent for each of the six DSST variants and the PVT at baseline, TSD/control, and recovery in Study 1 and Study 2 in the TSD condition (see Supplementary Table S2) and control condition (see Supplementary Table S3). The same morning test sessions as described above in the cumulative relative RT distributions were used for consistency across phases. RTs > 150 ms were pooled, including only correct responses for the DSST variants. Additionally, the Memory DSST in Study 1 included only correct responses in which the correct response was not displayed on the screen.

Diffusion model

The diffusion model was used to further analyze the effects of TSD on cognitive processing in the 2AFC DSST (see Table 2) and the PVT. The diffusion model is a computational model of cognition that describes RT distributions in terms of underlying cognitive processes for one-choice tasks such as the PVT [41] and two-choice response tasks such as the 2AFC DSST [42]. The diffusion model describes task performance in terms of a diffusion process, which represents the brain’s accumulation of evidence in central cognition toward a level representing the decision threshold for making a response (Supplementary Figure S2). For both one-choice and two-choice response tasks, it has been shown that the accumulation of evidence in central cognition is degraded by sleep deprivation [43, 44].

The diffusion process may be characterized by means of a drift ratio variable, calculated as the ratio of the mean over the standard deviation of the drift rate (with drift rate signifying the speed of evidence accumulation); and a speed-accuracy trade-off variable, calculated as the ratio of the mean speed of evidence accumulation over the level of the decision threshold (Supplementary Figure S2). Together these two variables, which may be expressed as the log transformation of the signal-to-noise ratio (LSNR), describe the fidelity of information processing in central cognition, as calculated using a method described elsewhere [44]. Here, this approach is used to enable head-to-head comparison of cognitive processing between the 2AFC DSST and the PVT.

Statistical analyses

Primary analyses of temporal changes in DSST performance were based on throughput, defined as the number of correct responses during the 4-min task duration, and on the proportion of total responses that were errors. In the Memory variant of the DSST, correct responses used for analyses included only those trials in which the correct response was not shown on-screen. Thus, the first trial of each digit–symbol pairing was omitted. In Study 1, each trial of a digit–symbol pair that followed a previous incorrect response for that pair also repeated presentation of the digit–symbol pair, and was therefore omitted as well. In Study 2, the correct pair was instead presented during the 1-s pause between trials with the accuracy feedback, so the digit-symbol pairings were not shown on-screen during the trials.

Statistical analyses were performed using SAS for Windows version 9.4 (SAS Institute Inc., Cary, NC, USA). Primary statistical analyses involved mixed-effects analysis of variance (ANOVA) with fixed effects for task variant (6 levels), condition (2 levels), session number (5 levels in Study 1; 15 levels in Study 2), and their interactions. A random effect over subjects was placed on the intercept. The two studies were analyzed separately.

To explore clustering of cognitive components across the DSST variants and the PVT in the context of inter-individual differences in responses to sleep deprivation, a principal component analysis (PCA) was applied to the combined data of the six DSST variants and PVT from subjects in the sleep deprivation condition in both studies. The primary outcome variable from each task was used (throughput for each DSST variant and lapses of attention for the PVT). Data from equivalent baseline and TSD test sessions of Studies 1 and 2 were considered. For Study 1, baseline sessions 1 and 2 and TSD sessions 3 and 4 were used; for Study 2, baseline sessions 3 and 5 and TSD sessions 8 and 10 were selected (see Figure 2). For each subject and each task, the throughput (DSST) or number of lapses (PVT) per session was computed, then averaged across the selected TSD sessions and across the selected baseline sessions. To account for any inter-individual differences in aptitude for the different tasks, the baseline grand average was then subtracted from the TSD grand average to produce a relative TSD impairment measure for each task for each subject. These relative impairments were then subjected to PCA, where factors with eigenvalues > 1 were retained and subjected to varimax rotation.

Finally, a head-to-head comparison of cognitive processing was made between the PVT and the 2AFC DSST for the TSD condition in Study 2. Based on the diffusion model (described above), the drift ratio, speed-accuracy trade-off, and LSNR were computed over the aggregated data from each of the three distinct phases of the study: baseline (sessions 1–5), TSD (session 6–10), and recovery (sessions 11–15). These variables were analyzed using a mixed-effects ANOVA with fixed effects for task (PVT/2AFC DSST), study phase (baseline/TSD/recovery), and their interaction. The interaction of task by study phase was used to assess whether TSD affected cognitive processing differently in the DSST as compared with the PVT.

Results

In this study, we investigated performance on DSST task variants, which were manipulated specifically to dissociate the effects of sleep deprivation on distinct cognitive processes. The task variants differed substantially in overall cognitive throughput, as seen in Figure 3, as the cognitive demands differ between the tasks. For instance, the Digit Only DSST does not require the visual search, spatial memory, or paired associate learning functions as used in the other tasks and, with the simplest task demands (see a digit, type that digit), has the highest throughput. In contrast, the Memory DSST, which relies on probe recall through paired associate learning, has the lowest throughput. Adding a 1-s delay after each trial (in Study 2) decreases throughput on all variants, as a portion of the 4-min task period is taken up by these inter-trial pauses, but the order of highest to lowest throughput task is the same as in Study 1. While these overall performance differences corroborate the effectiveness of the manipulation, to investigate the effects of sleep deprivation on cognitive processes, we focus on other aspects of performance, including the changes over time in the TSD condition and the control condition and inter-individual differences therein, as well as changes in the cumulative relative RT frequency distributions.

Figure 3.

Average throughput (mean ± SE) on each of the six task variants for subjects from Study 1 (top) and Study 2 (bottom) in the TSD condition (left) or the control condition (right). Light gray boxes indicate scheduled sleep opportunities. The dashed vertical lines in the TSD conditions (left) indicate midnight during the night(s) of sleep deprivation. The solid vertical lines indicate midnight during 10-h sleep opportunities. Note the different ordinate scale between Study 1 (top) and Study 2 (bottom).

Effect of sleep deprivation on DSST variants

To investigate the effect of TSD on performance across the different DSST variants, throughput in Study 1 was analyzed as a function of task variant, condition, and session. There were significant main effects of DSST variant (F_5,1560 = 1,693.63, p < 0.001), condition (F_1,1560 = 8.85, p = 0.003), and session (F_4,1560 = 29.47, p < 0.001). There was a significant interaction of variant by condition (F_5,1560 = 19.29, p < 0.001), variant by session (F_20,1560 = 1.87, p = 0.011), and condition by session (F_4,1560 = 25.14, p < 0.001). The three-way interaction of variant by condition by session was not significant (p = 0.51). As shown in Figure 3 (top left), there were large differences among the six DSST variants in overall levels of throughput, as expected based on the task components manipulated in each (see Table 2). However, all six variants were impaired by TSD in a comparable manner, with a notable decrease in throughput following the night of TSD.

In the control condition, the throughput analyses revealed a practice effect (albeit with a much smaller magnitude than the TSD effect), such that throughput increased across the test sessions on all six variants (Figure 3, top right). Across the six variants, throughput increased by an average of 4.1 correct responses per session (t_1,560 = 6.07, p < 0.001). The practice effect was also present in the TSD condition (Figure 3, top left), as throughput increased from session 1 to session 5 by an average of 2.7 correct responses per session (t_1,560 = 5.36, p < 0.001). The magnitude of the practice effect within each variant, calculated as throughput at session 5 minus throughput in session 1, was significant in the control condition for the 2AFC DSST (average increase of 7.0 correct responses per session; t_1,560 = 4.12, p < 0.001), Memory DSST (average increase of 6.9 correct responses per session; t_1,560 = 4.12, p < 0.001) and the Digit Only DSST (average increase of 3.8 correct responses per session; t_1,560 = 2.22, p = 0.026), with a trend toward significance in the Standard DSST (average increase of 3.0 correct responses per session; t_1,560 = 1.79, p = 0.074). Similarly, the magnitude of the practice effect in the TSD condition was significant in the 2AFC DSST (average increase of 4.4 correct responses per session; t_1,560 = 3.61, p < 0.001), Memory DSST (average increase of 3.6 correct responses per session; t_1,560 = 2.99, p = 0.003), and the Digit Only DSST (average increase of 2.7 correct responses per session; t_1,560 = 2.21, p = 0.027), with a trend toward significance in the Changing Pairs DSST (average increase of 2.1 correct responses per session; t_1,560 = 1.71, p = 0.087). There was no significant difference in magnitude of the practice effect between the control condition and TSD condition on any of the six DSST variants (all p ≥ 0.11).

To determine whether the decrease in throughput during TSD in Study 1 was driven by an increase in error responses, rather than fewer total responses being made, the proportion of error responses on each variant was also analyzed. There was a significant main effect of DSST variant (F_5,1560 = 344.33, p < 0.001), but not condition (p = 0.99) or session (p = 0.35). There was a significant interaction of condition by session (F_4,1560 = 2.54, p = 0.038) and a trend of significance for variant by condition (F_5,1560 = 2.07, p = 0.067), but not for variant by session (p = 0.20). The three-way interaction of variant by condition by session was also not significant (p > 0.99). While there was a significant increase in errors following the night of TSD, it did not account for the TSD-induced reduction in throughput. Across sessions, the average error proportion remained low (≤ 5 per cent) on all variants—except for the Memory DSST, in which it is not always possible to determine the correct response from the information on the display during each trial. Although the Memory variant had a higher overall error proportion (mean: 21 per cent overall), it did not show a marked increase in errors during TSD (p = 0.25; see Supplementary Figure S3).

We developed the six task variants to determine whether distinct components of cognition are specifically the source of TSD-induced performance impairment on the DSST. Our results indicate that for the components of cognition on which we focused—visual search, spatial memory, paired-associate learning, or motor response—this is not the case.

Effect of sleep deprivation with altered task pacing

Whereas in the DSST variants administered in Study 1, the next stimulus was presented immediately after each response, in Study 2, the next stimulus was presented after a fixed, computer-driven 1-s delay. To investigate the effect of TSD on performance across the different DSST variants with this altered pacing, throughput in Study 2 was analyzed as a function of task variant, condition, and session. There were significant main effects of DSST variant (F_5,2136 = 1,132.49, p < 0.001), condition (F_1,2136 = 16.07, p < 0.001), and session (F_14,2136 = 19.71, p < 0.001). There was a significant interaction of variant by condition (F_5,2136 = 38.96, p < 0.001) and condition by session (F_14,2136 = 15.81, p < 0.001), but not of variant by session (p = 0.23). The three-way interaction of variant by condition by session was also not significant (p = 0.98). As in Study 1, there were large differences among the six DSST variants in overall levels of throughput, as expected based on the task components manipulated in each (see Table 2). As shown in Figure 3 (bottom left), throughput was lower on all task variants in Study 2 compared with Study 1 due to the 1-s pause after each trial, which reduced the overall time per session available to make responses. However, the pacing manipulation of Study 2 did not fundamentally alter the TSD effect on throughput—all six variants were impaired by TSD with a notable decrease in throughput during the TSD period.

However, a marked difference between Studies 1 and 2 was seen in the control condition. The addition of the 1-s pause between response and presentation of the next stimulus in Study 2 greatly diminished the practice effect across sessions in all DSST variants except the Memory variant (Figure 3, bottom right). Across the six variants, throughput increased by an average of only 0.3 correct responses per session (as compared with 4.1 in Study 1). Thus, the practice effect does not seem to be an essential feature of the DSST (as it can be minimized without consequence), and it does not appear to play a role in the effect of sleep deprivation on DSST performance.

DSST and PVT RT distributions

To investigate possible mechanisms underlying the effect of TSD and the practice effect on the DSST, we inspected the cumulative relative RT frequency distributions of the Standard DSST, Digit Only DSST, and PVT in the TSD condition of Study 1 (Figure 4, left panels) and Study 2 (Figure 4, right panels). The TSD effect, which can be observed by comparing the TSD curves (red) to the corresponding baseline curves (blue), caused a rightward “skewing” of the RT distribution, indicating an increase in performance variability during TSD (cf. Supplementary Figure S1) for both the Standard DSST and the Digit Only DSST as well as the PVT. As expected based on previous findings for the PVT [39], the degree of skewing was greater in Study 2, which had the longer duration of TSD. This effect was further corroborated by calculating RTs for the fastest 10 per cent of responses, the median, and the slowest 10 per cent of responses in all DSST variants and the PVT in both studies (Supplementary Tables S2 and S3). The predominant TSD effect on all tasks in both studies was the increased proportion of slowed RTs. Whereas the increase in RT was near negligible for the fastest 10 per cent and moderate for the median, it was profound for the slowest 10 per cent (Supplementary Table S2). This type of a response to sleep deprivation has consistently been found in the PVT [40] and is indicative of state instability, which indicates that the DSST and the PVT both capture a similar TSD-induced impairment.

Figure 4.

Cumulative relative RT frequency distributions by study phase (baseline [blue], TSD [red], and recovery [gray]) for the TSD condition in Study 1 (left) and Study 2 (right). The graphs show the proportion of responses as a function of RT (plotted in 10 ms bins). Note that the scale of the abscissa is different for the PVT (bottom panels) than the Standard DSST and Digit Only DSST (top and middle panels).

In contrast with the TSD effect, the practice effect, which can be observed by comparing the recovery curves (gray) to the corresponding baseline curves (blue), caused a leftward “shift” of the RT distribution, indicating increased performance speed (cf. Supplementary Figure S1). This was observed primarily for the Standard DSST and Digit Only DSST in Study 1, as in Study 2 the practice effect was reduced by the 1-s pause between trials (Figure 3, bottom right) and the PVT does not exhibit any significant practice effect as previously shown [26, 45]. Similarly, the fastest 10 per cent, median, and slowest 10 per cent RTs became progressively faster across the baseline, control, and recovery phases of the study for all DSST variants, but not the PVT (Supplementary Table S3). This improvement was particularly evident in Study 1, in which the practice effect occurred across 5 administrations of each DSST variant, whereas in Study 2, the practice effect is reduced and any RT improvement occurs more gradually across 15 administrations of each DSST variant.

These results show that although responses are generally slower on the DSST compared with the PVT, sleep deprivation induces increased cognitive variability similarly in both task types—regardless of whether responses are subject-paced (as in the DSST) or computer-paced (as in the PVT), and regardless of whether relevant cognitive components are comprehensively included (Standard DSST) or maximally stripped from the task (Digit Only DSST). Yet regardless of the cognitive processing demand in each of the versions of the DSST, the data show a cognitive “variability” effect of sleep deprivation rather than a cognitive “slowing” effect. This finding is not compatible with an interpretation of DSST performance as an assay of general cognitive slowing during sleep deprivation.

Furthermore, these results demonstrate that the practice effect on the DSST is fundamentally distinct from the TSD effect, as it involves a change in general cognitive speed rather than variability. However, because the practice effect was seen in the DSST variants in Study 1 only, when there was no 1-s pause between trials, and because the PVT is devoid of a practice effect [45]—even when the demand characteristics for task performance are essentially the same (i.e. to respond accurately to the probes presented as quickly as possible)—it is reasonable to suspect that subject-based pacing of task performance is responsible for the practice effect on the DSST.

Inter-individual differences across task variants

To explore cognitive components in the context of inter-individual differences in responses to TSD, we investigated clustering across the DSST variants and the PVT in Studies 1 and 2 by means of PCA of mean impairment (reduction in DSST throughput or increase in PVT lapses) during TSD relative to baseline. The PCA revealed three dominant factors with eigenvalues > 1, which together explained 71.6 per cent of the variance. As shown in Supplementary Table S4, the Standard DSST, Digit Only DSST, and PVT loaded primarily on the first factor; the Moving Pairs DSST, Changing Pairs DSST, and 2AFC DSST loaded primarily on the second factor; and the Memory DSST loaded primarily on the third factor. Each task loaded primarily on one factor only. Notably, the Memory DSST overwhelmingly loaded to the third factor (factor 3 loading = 0.952). This suggests that the inter-individual differences in the effect of TSD on paired-associate learning, which is especially important in the Memory DSST, is distinct from the effect of TSD on other components of cognition investigated here.

Diffusion model analysis of information processing

In order to further compare information processing within individuals between tasks, a diffusion model analysis was used with the 2AFC DSST and PVT data from Study 2. For each subject for both the PVT and 2AFC DSST, the drift ratio, speed-accuracy trade-off, and LSNR were analyzed as a function of task, phase, and task by phase. A significant interaction of task by phase would indicate inter-individual differences in information processing varying between the two tasks. This task by phase interaction was significant for drift ratio (F_2,60 = 3.19, p = 0.048) and for speed-accuracy trade-off (F_2,60 = 4.85, p = 0.011), but not for LSNR (p = 0.12). Although both the 2AFC variant of the DSST and the PVT showed substantial impairment during TSD, the diffusion model analyses revealed that information processing for the two tasks was differentially degraded between the tasks within individuals.

Discussion

Based on its use in the literature on aging, the DSST has previously been interpreted as a global assay of cognitive processing speed. Thus, reductions in DSST throughput observed during sleep deprivation have been interpreted as evidence of slowed processing. By contrast, in the PVT, sleep deprivation has been shown to cause cognitive instability, which involves an increase in RT variability and thereby causes an increase in slow RTs but does not necessarily involve general slowing. Recognizing that the DSST is subject-paced and the PVT is computer-paced, and the tasks are therefore not readily comparable, these contrasting findings have not previously been reconciled. Additionally, the DSST involves a range of cognitive processes, which have not been previously differentiated, and has a practice effect, the nature of which has not been previously well understood. Here we conducted two laboratory TSD studies using the PVT and variants of the DSST, which were specifically designed to manipulate the involvement of specific cognitive processes, to shed light on these unresolved issues.

Sleep deprivation induces RT variability on the DSST

Performance on all DSST variants was impaired during sleep deprivation in both Studies 1 and 2, as evidenced by a substantial decrease in throughput (number of correct responses per test bout; see Figure 3, top and bottom left). To investigate this in greater depth, the effects of TSD on the shape of cumulative relative RT frequency distributions were compared for the Standard DSST, Digit Only DSST, and PVT (Figure 4, red curves compared with blue curves). Previous research has shown that increased performance variability is a signature effect of TSD on the PVT [46, 47]. This finding is replicated here in the PVT, and both the Standard DSST and the Digit Only DSST exhibited the same rightward skewing of the distribution during TSD. Thus, a similar effect was seen across tasks regardless of the particular cognitive components, task pacing, and the duration of sleep deprivation. Note also that even in Study 2, which involved 24 h more TSD than Study 1, the fastest RTs were only slightly shifted in response to sleep deprivation. This pattern of TSD effects is not compatible with a “general” slowing of cognitive processing, and indicates that in the context of sleep deprivation the DSST should not be interpreted as an assay of cognitive slowing. Rather, the dominant source of impairment on the DSST appears to be the same instability phenomenon that also underlies impairment on the PVT.

The practice effect on the DSST is independent of the sleep deprivation effect

As has been reported previously, there is a significant practice effect on the DSST [26]. In Study 1, the practice effect was observed in all DSST variants. It can be most readily seen in the control condition (Figure 3, top right), but it is also visible in the TSD condition as improved performance after recovery sleep compared with baseline (Figure 3, top left), as has been observed previously [27]. The magnitude of the practice effect in the TSD condition was not significantly different than that in the control condition. Inspection of the RT distributions sheds further light on this (Figure 4, gray curve relative to blue curve). That is, the practice effect in Study 1 involved a leftward “shift” of the RT distribution. In contrast to the TSD effect, which primarily impacted the slowest RTs while largely preserving the fastest RTs, the practice effect produced a similar improvement in the fastest 10 per cent, median, and slowest 10 per cent of responses (Supplementary Table S3). This type of change reflects a general response speeding and is therefore not only opposite to, but also fundamentally different from the TSD effect.

In contrast with the DSST, there was no practice effect in the PVT, in agreement with previous reports [45]. Study 2 provided some insight into what causes a practice effect in one task and not another. Namely, the magnitude of the practice effect on the DSST was considerably diminished by manipulating the task pacing, from a stimulus being presented immediately after a response to a stimulus being presented following a 1-s pause after a response (Figure 3, bottom right). This changed the pacing of the task from being entirely dependent on the response speed of the subject to more dependent on a computer-driven delay between trials, similar to the PVT, in which a stimulus is presented after an inter-stimulus interval of 2–10 s. The learning that generates the practice effect in the Study 1 DSST variants may be related to an improvement in response efficiency that can be achieved by increasingly automating the motor response (pressing a key or button). That is, subjects may be improving their ability to turn their attention to the next stimulus while they are still in the (automated) process of executing the previous response. This would allow them to become increasingly efficient, and thus faster, in the self-paced DSST variants of Study 1, while not offering any meaningful benefit in the 1-s delay DSST variants of Study 2 or the computer-paced PVT. Recent research has shown that carrying out a planned motor action is robust to sleep deprivation [16], which may explain why the practice effect in Study 1 is sustained through TSD. It is unclear, however, why the 2AFC DSST had a diminished practice effect during TSD in Study 1 compared with the control condition.

Task pacing does not materially affect the TSD effect

Previous literature has suggested that “task pacing” is a critical moderator of performance during sleep deprivation, as tasks that are computer-paced are primarily subject to vigilant attention deficits during TSD, whereas tasks that are subject-paced would avoid such impairment by allowing subjects to slow down as needed (speed-accuracy trade-off) [48]. However, in the present research, different types of task pacing were employed: subject-paced in the Study 1 DSST and computer-paced in both the PVT (variable interval delay) and Study 2 DSST (fixed interval delay). Under all types of task pacing (subject-paced, computer-paced with variable interval, and computer-paced with fixed interval), task performance demonstrated response variability indicative of vigilance performance decrements. The similarity is especially evident when comparing the cumulative relative RT frequency distributions of the Digit Only DSST, which is essentially just a reaction time task, to that of the PVT, which is also a reaction time task. As shown in Figure 4, both the Digit Only DSST and the PVT exhibited a rightward skewing of the response distribution during TSD that is stereotypical of vigilance impairment (middle versus bottom panels) and proportional to the amount of sleep deprivation (left versus right panels). This suggests that task pacing alone is not the determinant of whether a task shows vigilant attention deficits during sleep deprivation.

In the current research, tasks with either type of pacing (PVT or DSST) had similar demand characteristics: subjects were instructed to respond as quickly as possible while avoiding errors. In other words, there was time pressure to respond following stimulus presentation, regardless of the pacing between trials. As such, while task pacing may be important for whether or not a practice effect is seen on the task (see above), it seems that the presence of “time pressure,” rather than the pacing of the task per se, is the critical driver of whether or not sleep deprivation produces vigilant attention deficits. This idea is corroborated by our finding that vigilant attention deficits in the subject-paced DSST persisted even when the pacing incorporated a 1-s response delay after each trial, as in the DSST variants in Study 2 (Figure 4, right).

The effects of sleep deprivation depend in part on task components

Thus far, we focused on the remarkable degree of similarity between each of the DSST variants (including two different types of pacing) and the PVT. This would seem to suggest that the effects of sleep deprivation are largely universal regardless of task components, and not a single cognitive function investigated here is solely responsible for the effects of sleep deprivation on task performance. Regardless of which functions were omitted and/or manipulated, including visual search, spatial memory, paired associate learning, and motor functions, performance on the task variants showed similar impairments during sleep deprivation and improvement after recovery sleep.

However, the emergence of three distinct factors on the PCA revealed that this perspective on the data is incomplete (Supplementary Table S4). Individuals who were most vulnerable to the effects of sleep deprivation on tasks that clustered on one factor were not the most vulnerable on tasks that clustered the other two factors. The first factor, which included the Standard DSST, Digit Only DSST, and PVT, may reflect a vigilant attention component involved in straightforward, speeded responding. The second factor, which included the Moving Pairs DSST, Changing Pairs DSST, and 2AFC DSST, may reflect stronger reliance on the visual search component. Most strikingly, the third component was dominated by the Memory DSST (loading at >95 per cent). This task variant, which relies most heavily on paired associate learning, was differentially impaired by TSD compared with all other DSST variants and the PVT. This separation is attributable to differential individual vulnerability to sleep deprivation in the Memory DSST variants compared with the other tasks. There is evidence in the literature that networks underlying paired associate learning are at least partially distinct from attentional networks used across all task versions [49]. We may therefore infer that the distinction between the three factors exposes that the effects of sleep deprivation are at least in part local, that is, neural pathway-specific.

Further support for this idea comes from the diffusion model analysis of the 2AFC DSST and PVT. The results showed task-dependent inter-individual differences in the drift ratio, which is an index of the rate of accumulation of evidence in the processing of a stimulus [41]. The task-dependence of inter-individual differences in this index is consistent with the idea that distinct neural pathways are involved, each independently affected by sleep deprivation differentially across individuals [50].

A plausible variation on this explanation is that changing the nature of a task (as in the DSST variants) moves the locus of the sleep deprivation effect and thus exposes a different bottleneck in cognitive processing. For instance, the Memory DSST forces heavy reliance on paired associate learning and, because task performance is critically dependent on this skill, a particular vulnerability is revealed that is masked in other DSST variants in which subjects may also rely on other cognitive functions to complete the task. This explanation would also be consistent with the idea that distinct neural pathways are involved, each independently affected by sleep deprivation differentially among individuals [50].

Conclusion

Sleep deprivation impaired performance on the DSST across all six task variants, with throughput being reduced regardless of which cognitive components were included or omitted. Thus, the sleep deprivation effect on DSST throughput was not specifically contained in one of the particular functions isolated by varying the nature of the task. This suggest that either the sleep deprivation effect lies in a common cognitive function that is included in all DSST variants (as well as the PVT), or there may be a more general effect causing equivalent impairment across different functions. However, the PCA results and the diffusion model analyses showed task-dependent inter-individual differences, suggesting the presence of local, cognitive or neural pathway-specific effects of sleep deprivation as well. Thus, in line with recent theoretical [17] and experimental [8] findings of both top-down and bottom-up attentional deficits in cognitive performance, we found evidence for both general (global) and specific (local) effects of sleep deprivation. Our results provide important new insights into the effects of sleep deprivation on DSST performance specifically and cognitive function in general.

Funding

This work was supported by Office of Naval Research grant N00014-13-1-0302, National Institutes of Health grant R01HL105768, Congressionally Directed Medical Research Program awards W81XWH-05-1-0099 and W81XWH-16-1-0319, Defense University Research Instrumentation Program grants FA9550-06-1-0281 and N00014-17-1-2990, and the Air Force Research Laboratory’s Warfighter Readiness Research Division.Opinions, interpretations, conclusions, and recommendations are those of the authors and are not necessarily endorsed by the Department of Defense.

Conflict of interest statement. None declared.