Skip to main content
PMC home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2018 Dec 6.
Published in final edited form as: Rehabil Psychol. 2017 Nov;62(4):464–473. doi: 10.1037/rep0000185

Motor-Free Composites From the National Institutes of Health Toolbox Cognition Battery (NIHTB-CB) for People With Disabilities

Noelle E Carlozzi 1, Siera Goodnight 1, Anya Umlauf, Robert K Heaton 2, Allen W Heinemann 3, Benjamin D Schalet, Richard C Gershon 4, David S Tulsky 5
PMCID: PMC6283101  NIHMSID: NIHMS998473  PMID: 29265867

Abstract

Purpose/Objective:

The National Institutes of Health Toolbox Cognition Battery (NIHTB-CB) includes a group of brief measures (i.e., 30 min) designed to assess language, processing speed, working memory, episodic memory, and executive functioning. These subtests can be combined to create composite scores that reflect fluid and crystallized cognition, as well as overall cognition. The battery is of limited utility with individuals who have impaired upper extremity motor functioning. This manuscript examines the accuracy of the Oral Symbol Digit Modalities Test as a substitute for the Pattern Comparison Processing Speed Test for computing motor-free composite scores.

Research Method/Design:

Individuals with spinal cord injury (SCI; n [H11005] 188), traumatic brain injury (TBI; n [H11005] 159), or stroke (n [H11005] 180) completed the NIHTB-CB. We used the Oral Symbol Digit Modalities Test to create a Motor-Free Pattern Comparison score; this was used to create revised, Motor-Free Composite scores for Fluid Cognition and Overall Cognition.

Results:

Although there were statistically significant overall differences between the two Fluid and Overall Cognition composite scores for some of the clinical groups (scores based on the motor-free approach were significantly higher than the original score), these differences were small and partly because of overclassification of impaired processing speed in participants with motor impairment. There was good to substantial agreement with regard to “impairment” classification between the two sets of Original and Motor-Free composite scores.

Conclusions/Implications:

Although the Motor-Free scores are not a perfect match for the Original Composite scores, they provide a reliable and valid way to examine overall and fluid cognition in individuals with upper extremity motor impairments.

Impact and Implications

Although the National Institutes of Health Toolbox Cognition Battery (NIHTB-CB) is designed as “common currency” measures that are useful across the life span (ages 3–85) and clinical conditions, there are limitations to administering the full battery to individuals with certain disabilities, especially those with limited upper extremity motor functioning. This study examined the clinical utility of Motor-Free Composite scores that can be used in individuals with upper extremity motor functioning impairments. While the Motor-Free Composite scores are not perfect matches with the Original Composite scores, they provide reliable and valid alternatives when using the NIHTB-CB in individuals with upper extremity motor impairments that interfere with their ability to complete subtests that include a motor component. proceeding Data from this study provide clinicians and researchers with an accessible version of the NIHTB-CB for individuals with upper extremity motor impairments.

Keywords: cognition, NIHTB, spinal cord injury, traumatic brain injury, stroke

Introduction

The National Institutes of Health (NIH) Blueprint for Neuroscience Research was initiated to “accelerate the pace of discovery” in neuroscience research and practice (www.neuroscienceblueprint.nih.gov). As a part of this initiative, the NIH Toolbox (NIHTB) for Neurological and Behavioral Function was developed to provide a series of common data elements across four domains (cognition, emotion, motor, and sensory-perceptual functioning) that could be used across clinical populations and settings. The NIHTB Cognitive Functioning Battery (NIHTB-CB), one of the four components of the NIHTB, includes a brief series (i.e., 30 min total) of measures designed to assess language, processing speed, working memory, episodic memory, and executive functioning (Weintraub et al., 2013, 2014). The NIHTB-CB is comprised of seven core subtests (Picture Vocabulary, Oral Reading Recognition, Picture Sequence Memory, Pattern Comparison Processing Speed, List Sorting Working Memory, Dimensional Change Card Sort, and Flanker Inhibitory Control and Attention). In the development process of the NIHTB-CB, two of the current authors (NEC and DST) suggested potential use of a motor-free supplemental processing speed subtest (Oral Symbol Modalities Test [OSD]; Koh et al., 2011; Rao et al., 1991) as an alternative to the Pattern Comparison subtest.

Although the seven core subtests can be combined to create composite scores that reflect fluid cognition (comprised of Picture Sequence Memory, Pattern Comparison Processing Speed, List Sorting Working Memory, Dimensional Change Card Sort, and Flanker Inhibitory Control and Attention) and crystallized cognition (comprised of Picture Vocabulary and Oral Reading Recognition), as well as overall cognition (comprised of the average of the fluid and crystallized measures; Heaton et al., 2014), the OSD test was not included in the composite scores, nor reported previously in NIHTB-CB publications. Yet at the same time, there are limitations to administering the full battery to individuals with certain disabilities, especially the individuals with limited upper extremity motor functioning, which prohibits or interferes with completion of multiple NIHTB-CB tests; in fact, previous work has indicated that nonstandard administration is needed in some individuals with disabilities because of problems with accessibility (Magasi et al., in press). This particularly limits the ability to utilize the composite scores in clinical research with individuals who have upper extremity impairments. In such individuals, poor performance on tests like the Pattern Comparison Processing Speed Test may result from impaired motor function rather than processing speed. To at least partially remedy this problem, we provide detailed information about the OSD, and present motor-free composite scores that can be utilized when testing with individuals who have relevant motor disabilities. Data from a larger study that examined the full NIHTB in individuals with spinal cord injury, traumatic brain injury, and stroke (Carlozzi et al., in press; Carlozzi et al., in press; Cohen et al., in press; Tulsky et al., in press) were used.

Thus, this article is designed to provide details for converting the NIHTB OSD to Motor-Free Pattern Comparison T scores, and information about using these scores to generate Motor-Free Composite scores; we also provide preliminary normative standards to support these scores so that they can be used as an accommodation when individuals have a disability with upper extremity limita tions. These data will enable NIHTB-CB users to obtain scores when tests that require upper extremity motor functioning cannot be validly administered (standard Pattern Comparison Processing Speed, Flanker Inhibitory Control and Attention, and Dimensional Change Card Sort).

Method

Participants

There were 527 individuals with a medically documented traumatic spinal cord injury (SCI; n = 188), traumatic brain injury (TBI; n = 159), or stroke (n = 180) who completed the NIHTBCB. Participants were recruited through the Rehabilitation Institute of Chicago, the University of Michigan, and Washington University. Participants were at least 18 years old, a minimum of 12-months postinjury, able to comprehend English, and had a medically confirmed diagnosis of SCI, TBI, or stroke (described below). Exclusion criteria consisted of cognitive impairments because of other diagnoses or conditions, and the presence of aphasia as determined by the Frenchay Aphasia Screening Test (Enderby & Crow, 1996; Enderby, Wood, Wade, & Hewer, 1987; Salter, Jutai, Foley, Hellings, & Teasell, 2006). Data were collected in accordance with approvals by local institutional review boards. Participants with SCI included individuals with a medically documented acute traumatic lesion of neural elements in the spinal canal that resulted in permanent sensory or motor deficits (American Spinal Injury Association, 2002). Individuals with TBI had a medically confirmed diagnosis of complicated mild (Williams, Levin, & Eisenberg, 1990), moderate (Traumatic Brain Injury Model Systems National Data Center, 2006), or severe TBI (emergency room Glasgow Coma Scale score and neuroimaging results were used to confirm TBI severity; Traumatic Brain Injury Model Systems National Data Center, 2006). Stroke diagnosis required medical documentation of rapidly developing clinical signs of focal or global disturbance of cerebral function, with symptoms lasting more than 24 hr, and with no apparent cause other than that of vascular origin (WHO MONICA Project Principal Investigators, 1988). Certain analyses were performed with separate subgroups of participants without and with dominant upper extremity motor impairment, defined as absence or presence of NIHTB-Motor Battery dominant hand 9-Hole Pegboard Task T score <40(i.e., more than 1 SD below the demographically corrected normative mean on this test of manipulatory speed).

Instruments

The NIHTB-CB (Weintraub et al., 2013).

The NIHTB-CB includes seven core measures: two crystallized cognition measures (Picture Vocabulary [Gershon et al., 2014, 2013] and Oral Reading Recognition [Gershon et al., 2014, 2013]) and five fluid cognition measures (Picture Sequence Memory [Dikmen et al., 2014], Pattern Comparison Processing Speed [Carlozzi et al., 2014], List Sorting Working Memory [Tulsky et al., 2014], Flanker Inhibitory Control and Attention [Zelazo et al., 2013], and Dimensional Change Card Sort [Zelazo et al., 2014, 2013]). In addition, the current study included a supplemental motor-free measure of processing speed: the Oral Symbol Digit Modalities test.

The Oral Symbol Digit Modalities Test (OSD; Koh et al., 2011; Rao et al., 1991).

This OSD is a measure of processing speed that involves visual scanning, tracking, and oral response, but no speeded motor response with the dominant upper extremity; it was selected (but unfortunately not as a regular component of the NIHTB-CB norming study) as a supplemental test to provide a motor-free measure of processing speed. The examinee has 120 s to match symbols (i.e., geometric line drawings such as “ [H11005] “) with digits (i.e., the numbers 1 to 9) as quickly as possible according to a “key” that is provided at the top of the page. The examinee responds by orally providing the corresponding digits for each of the symbols provided on the screen (out of a possible 144 symbols); the examiner records correct responses by checking a box according to key that is provided on a separate screen. Raw scores reflect the number of correct responses provided during the 90 s administration period. For the purposes of this study, scores were used to generate a Motor-Free Pattern Comparison score (described below).

The NIHTB 9-Hole Pegboard Test, Dominant Hand (Wang et al., 2011).

This provides a measure of speeded fine motor dexterity and the ability to coordinate the fingers and manipulate objects. Scores reflect time to completion (in seconds) for the dominant hand; higher scores indicate worse functioning. For some analyses of the current study, this measure was used to identify participants with and without motor impairments. Specifically, to be classified in the “motor impairment group,” participants must have a T score <40 on the dominant hand NIHTB 9-Hole Pegboard Test (i.e., score >2.0 SD below the mean; n = 101 individuals with SCI, n = 78 for individuals with TBI, and n = 89 for individuals with stroke). For individuals identified as having motor impairment, missing scores were winsorized (a process of limiting extreme values, or in this case missing values to due significant motor impairments that precluded test administration, in statistical data) for all of the NIHTB tests that required timed motor responses (i.e., participants were given a T score that was 1 point lower than the lowest obtained score for the aforementioned timed tests).

Scoring for the Motor-Free Pattern Comparison.

We examined fully corrected (age, education, sex, and race/ethnicity) NIHTB-CB scores for the Core Battery Measures (Casaletto et al., 2015; described below) and developed a Motor-Free Pattern Comparison score using data from the OSD. We used equipercentile linking (Choi, Schalet, Cook, & Cella, 2014; Kolen & Brennan, 2014; Noonan et al., 2012) to generate estimated raw Pattern Comparison scores from raw OSD scores for participants with normal T scores (≥40) on the Dominant Hand Pegboard (analyses were conducted in LEGS Version 2.0; Brennan, 2004); Table 1 provides the raw score conversion table. Then, using the published norms (Casaletto et al., 2015), fractional polynomial models were applied to these estimated raw Pattern Comparison scores to obtain fully corrected T scores (corrected for race/ethnicity (i.e., Non-Hispanic White or Asian, African American, Hispanic), age, education, and sex). The resulting Motor-Free Pattern Comparison T score represents an individual’s processing speed performance relative to normal age-, education-, sex-, and race/ethnicity-matched peers.

Table 1.

Oral Symbol Digit Raw Score to Pattern Comparison Raw Score Conversion Table

OSD raw score Pattern comparison raw score
3 23
4 23.7
5 24.7
6 25.7
7 28.7
8 28.8
9 28.8
10 28.8
11 28.8
12 28.8
13 28.8
14 28.8
15 29
16 29.3
17 31.5
18 31.8
19 32
20 32.2
21 32.5
22 32.5
23 32.8
24 33
25 34
26 35
27 35.5
28 36
29 36
30 36
31 36
32 36.2
33 36.6
34 36.6
35 36.7
36 37
37 37.3
38 37.4
39 38
40 39
41 39.6
42 40.2
43 41.2
44 42.7
45 43.6
46 44.4
47 45.5
48 47.1
49 47.7
50 48.2
51 48.9
52 49.2
53 49.2
54 50.3
55 51.7
56 52
57 52.4
58 53.6
59 55.2
60 56.5
61 57
62 57.4
63 57.8
64 58.2
65 58.7
66 59.2
67 59.7
68 60.2
69 60.7
70 61
71 61.3
72 61.7
73 62.5
74 62.9
75 63
76 63.6
77 65.2
78 65.8
79 66.2
80 66.7
81 67.1
82 67.4
83 68.2
84 69.5
85 70.6
86 71.1
87 71.4
88 72
89 72.9
90 73.1
91 73.4
92 74
93 74.7
94 75.2
95 75.8
96 76.5
97 77
98 77
99 77
100 77
101 77
102 77.4
103 78.6
104 79
105 79.2
106 79.4
107 79.5
108 80
109 80.5
110 82.5
111 85.5
112 85.5
113 85.5
114 85.5
115 85.5
116 87
117 87.5
118 87.5
119 87.5
120 88
121 88.5
122 88.5
123 88.5
124 88.5
125 88.5
126 88.5
127 88.5
128 88.5
129 88.5
130 88.5
131 88.5
132 89.3
133 90
Open in a new tab

Note. OSD [H11005] Oral Symbol Modalities Test.

Scoring of the other NIHTB-CB tests.

With regard to the remainder of the measures within the NIHTB-CB, demographically corrected normative standards were also utilized (Casaletto et al., 2015). In cases where there were substantial effects of demographic factors (race, ethnicity, age, education, and sex) on raw scores, fractional polynomial models were created from the raw scores of each test separately for each race/ethnicity (i.e., Non-Hispanic White or Asian, African American, Hispanic) and regressed on demographic characteristics (i.e., age, education, sex). The resulting T scores (M = 50, SD = 10) for each test represents an individual’s cognitive performance relative to their matched normal peers (matched on age, education, sex, and race/ethnicity). Composite scores were examined that reflected the NIHTB-CB Crystallized Cognition Score, Fluid Cognition Score (“Original Fluid Cognition”), and an Overall Cognition Score (“Original Overall Cognition”) that averaged the Crystallized and Fluid Composites (Heaton et al., 2014); see Table 2.

Table 2.

NIHTB-CB Subtests Used to Calculate the Original and Motor-Free Composite Scores

NIHBT-CB Composite Score Original Composite Score Motor-Free Composite Scores
Crystalized Cognition Picture Vocabularya Picture Vocabularya
Oral Reading Recognition Oral Reading Recognition
Fluid Cognition Dimensional Change Card Sorta Picture Sequence Memorya
Flanker Inhibitory Control and Attentiona List Sorting Working Memorya
Picture Sequence Memorya Estimated Pattern Comparison Processing Speed (from Oral Symbol Digit)
List Sorting Working Memorya
Pattern Comparison Processing Speed
Overall Cognition Picture Vocabularya Picture Vocabularya
Oral Reading Recognitiona Oral Reading Recognitiona
Dimensional Change Card Sorta Picture Sequence Memorya
Flanker Inhibitory Control and Attentiona List Sorting Working Memorya
Picture Sequence Memorya Estimated Pattern Comparison Processing Speed (from Oral Symbol Digit)
List Sorting Working Memorya
Pattern Comparison Processing Speed
Open in a new tab

Note. NIHTB-CB [H11005] National Institutes of Health Toolbox Cognition Battery.

a

Indicates subtests that are a part of the Core NIHTB-CB.

Development of “Motor-Free” Fluid and Overall Composite scores.

Heaton et al., (2014) created three composite scores for the NIHTB-CB (Fluid, Crystallized, and Overall Cognition; Heaton et al., 2014). While the Crystallized composite score is constructed of nonmotor tests, three of the five tests within the Fluid category require quick responses with one’s hand (e.g., clicking a mouse, pointing to stimuli on the screen) that, in the standardized administration procedures, limit the use of the test and/or the interpretation of the resulting scores and composite scores. Magasi and colleagues (in press) have provided alternative administration instructions as an attempt to make reasonable accommodations, but it is unclear what the impact of these alternative administration and response outcomes will have on scores. For the current manuscript, we created revised, “Motor-Free” Composite scores for Fluid Cognition and Overall Cognition that utilized the measures that did not rely on motor responses. Table 2 describes the measures used to calculate the “Motor-Free Fluid Cognition Score” and the “Motor-Free Total Cognition Score.” The Dimensional Change Card Sorting Test and the Flanker Tests were not included in this revised composite measure. The following formulas were used to generate the new T scores:

White/Asian:

  • Motor-Free Fluid Cognition T Score = 50 + 10*(((Mean of 3 fully corrected T scores for Motor-Free Pattern Comparison, Picture Sequence Memory, List Sorting) − 50.1414)/6.9041);

  • Motor-Free Overall Cognition T Score = 50 + 10*(((Mean of 2 fully corrected T scores for Motor-Free Fluid composite and Crystallized Composite) − 50.1133)/8.2739).

African American:

  • Motor-Free Fluid Cognition T Score = 50 + 10*(((Mean of 3 fully corrected T scores for Motor-Free Pattern Comparison, Picture Sequence Memory, List Sorting) − 50.0399)/6.9322);

  • Motor-Free Overall Cognition T Score = 50 + 10*(((Mean of 2 fully corrected T scores for Motor-Free Fluid composite and Crystallized Composite) − 50.0058)/8.6584).

Hispanic:

  • Motor-Free Fluid Cognition T Score = 50 + 10*(((Mean of 3 fully corrected T scores for Motor-Free Pattern Comparison, Picture Sequence Memory, List Sorting) − 50.0562)/7.1593);

  • Motor-Free Overall Cognition T Score = 50 + 10*(((Mean of 2 fully corrected T scores for Motor-Free Fluid composite and Crystallized Composite) − 49.8723)/8.2570).

Analysis Plan

Descriptive statistics for the study sample will be provided. In addition, descriptive information for the NIHTB-CB Core measures, Motor-Free Estimate Pattern Comparison (based upon OSD), Original Composite Scores, and Motor-Free Composite Scores will be presented separately for participants with and without motor impairments. Separate paired sample t tests were used to examine differences between the raw Original and raw Motor-Free Pattern Comparison scores separately for the three clinical groups that did not exhibit motor impairments.

Relationships among Motor-Free Composites and Original Composite scores.

In addition, interclass correlations between each Motor-Free and its corresponding Original Composite T score were conducted. Low correlations <0.4 were considered poor, 0.4 to 0.6 adequate, and ≥0.6 good/very good evidence for test score interpretation of convergent validity (Campbell & Fiske, 1959). For the combined group of individuals with brain injuries (i.e., TBI and stroke; n = 339), classification of cognitive impairment (i.e., the identification of participants that scored >1 SD below the mean; i.e., T scores <40) was also compared between the two sets of Fluid Composite scores and the two sets of Overall Cognition scores using McNemar’s test to establish known groups validity. The overall proportion of agreement was examined; ≥ 90% was considered excellent, ≥ 80% good, ≥ 70% as fair, and <70%, poor agreement (a priori minimum was specified as ≥70%). Cohen’s κ coefficients also were calculated; a κ score between 0.81 and 1 was considered almost perfect agreement, 0.61–0.80 was considered substantial agreement, 0.41–0.60 was moderate agreement, 0.21–0.40 was deemed fair agreement, and lower than 0.20 indicated no agreement (Cohen, 1960; McHugh, 2012).

A series of paired t tests were also conducted to determine if the Motor-Free Fluid Composite scores differed from the Original Fluid Composite scores (individuals with SCI, TBI, and Stroke [each of these three groups included both individuals with and without motor impairments] and a combined group of individuals with brain injuries). Effect size calculations were computed comparing the Original scores to the Motor-Free composites for each of the six clinical groups. Effect sizes are reported as Cohen’s d, with cutoffs of .20, .50, and .80 indicating small, medium, and large effects, respectively (Cohen, 1992) and with d < .20 considered not a meaningful difference.

Results

The study sample included 188 individuals with SCI (27% paraplegic complete, 22% paraplegia incomplete, 23% tetraplegia complete, 28% tetraplegia incomplete, and 1% unknown severity), 159 individuals with TBI (38% complicated mild, 10% moderate, 51% severe, and 1% unknown severity), and 180 individuals with stroke (31% mild, 26% moderate, 43% severe, and 1% unknown [numbers do not sum to 100% because of rounding]; 67% ischemic, 29% hemorrhagic and 4% unknown). Table 3 provides demographic information for each diagnosis group, and the total sample. Table 4 provides summary information for all NIHTB-CB core measures, as well as Motor-Free Estimate Pattern Comparison (based upon OSD) and Motor-Free Composite Scores; data are presented separately for participants with and without motor impairments. Because the only groups without any known brain injury are individuals with SCI, it is noteworthy that mean Original and Motor-Free scores for Pattern Comparison and the Fluid and Total Composite scores are all within the normal range (t ≥ 49) for individuals with SCI without motor impairment; by contrast, while the Original NIHTB scores on these measures were well below average (t = 39.31 to 46.58) for individuals with SCI with motor impairment, the Motor-Free scores were much higher and normal (t = 49.23 to 52.32). There were also significant differences between the Original and Motor-Free Pattern Comparison scores for all three clinical groups without motor impairments (SCI: t(60) = 3.13, p = .003; TBI: t(77) = 3.68, p < .0001, and stroke, t(85) = 4.30, p < .001); in all cases, the Motor-Free means were higher than the Original means (with the most profound difference in individuals with SCI). It was also noted that in individuals without motor impairment, the Motor-Free Pattern Comparison scores were significantly higher than the original scores on this test (all ps <.004); however, these differences did not translate into meaningful differences in the NIHTB Composite scores (in all cases, mean Motor-Free and Original Composite score differences were not significant.

Table 3.

Summary Demographic Data

Variable Spinal cord injury
(n = 188)		 Traumatic brain injury
(n = 159)		 Stroke
(n = 180)		 Combined clinical groups
(N = 527)
Age (years)
 M (SD) 45.31 (13.78) 39.22 (17.26) 55.71 (12.93) 47.02 (16.10)
Time since injury (years)
 M (SD) 12.40 (10.16) 5.78 (5.58) 2.80 (2.27) 7.13 (8.05)
Sex (%)
 Male 79.8 62.9 50.0 64.5
 Female 20.2 37.1 50.0 35.5
Race (%)
 White 62.2 74.8 43.3 59.6
 African American 26.6 13.2 47.8 29.8
 Asian .5 5.7 2.2 2.7
 American Indian/Alaskan Native .0 .6 .6 .4
 Native Hawaiian/Other Pacific Islander .0 .6 .0 .2
 Other 10.6 5.0 6.1 7.4
Ethnicity (%)
 Not Hispanic or Latino 90.4 92.5 95.0 92.6
 Hispanic or Latino 9.6 7.5 5.0 7.4
Education
 M (SD) 13.86 (2.50) 13.83 (2.50) 13.83 (2.58) 13.84(2.53)
Open in a new tab

Note. This is a subsample of the full spinal cord injury sample.

Table 4.

Descriptive Data for the NIHTB-CB Measures, Original Composite Scores, and Motor-Free Composite Scores

Spinal cord injury (without motor impairments)
(n =63)		 Spinal cord injury (with motor impairments)
(n = 125a)		 Traumatic brain injury (without motor impairments)
(n = 80)		 Traumatic brain injury (with motor impairments)
(n = 79a)		 Stroke (without motor impairments)
(n = 87)		 Stroke (with motor impairments)
(n = 93a)
NIHTB Cognition Scores M (SD) M (SD) M (SD) M (SD) M (SD) M (SD)
Crystallized Cognition 50.46 10.49 50.23 10.22 51.27 9.51 48.14 9.84 50.68 11.35 48.40 10.06
 Picture Vocabulary 51.43 9.96 50.36 10.26 51.50 8.8 47.68 10.26 50.82 11.99 49.29 10.11
 Oral Reading Recognition 49.23 10.64 49.93 10.03 51.21 11.23 49.14 10.13 50.18 10.41 48.09 10.09
Original Fluid Cognition 49.42 5.02 42.86 9.18 46.74 7.07 41.43 8.37 44.36 7.91 40.74 7.13
Motor-Free Fluid Cognition 50.28 8.77 49.23 10.20 46.84 11.55 39.72 13.41 45.14 14.23 39.44 10.93
 Picture Sequence Memory 47.84 8.20 46.45 10.60 44.10 10.77 41.33 10.88 43.41 13.28 41.50 10.73
 List Sorting 51.16 8.83 41.79 13.53 48.74 10.35 42.04 10.74 45.02 11.18 40.51 10.83
 Flanker 47.39 9.21 49.91 9.88 46.87 9.80 42.97 11.18 45.36 9.74 42.69 10.39
 DCCS 50.57 8.34 36.17 16.49 46.16 9.06 39.23 12.32 44.25 10.02 38.15 12.06
 Pattern Comparison 49.39 8.63 39.31 13.51 46.64 9.47 42.27 11.33 43.89 9.96 39.69 9.27
 Motor-Free Pattern Comparison 55.63 9.29 52.32 10.08 52.88 10.33 44.56 11.74 51.37 13.03 43.98 10.49
Original Overall Cognition 50.22 6.13 46.58 7.63 49.16 6.79 44.60 7.21 47.45 8.19 44.73 6.70
Motor-Free Overall Cognition 50.52 9.29 49.84 9.50 48.88 10.56 42.88 12.26 47.94 12.64 43.29 10.04
Open in a new tab

Note. NIHTB-CB = National Institutes of Health Toolbox Cognition Battery. Scores presented are all fully corrected T scores (M = 50, SD = 10); the Original Crystallized Cognition composite score is comprised of Picture Vocabulary and Oral Reading Recognition Test; the Original Fluid Cognition Composite Score is comprised of Picture Sequence Memory, Pattern Comparison Processing Speed, List Sorting Working Memory, Flanker Inhibitory Control, and Attention and Dimensional Change Card Sort; the Original Overall Cognition Composite Score is comprised of all 7 Core measures within the NIHTB-CB; the Fluid Cognition Motor-Free Composite Score is comprised of List Sorting Working Memory, Picture Sequence Memory, and Motor-Free Pattern Comparison; the Overall Motor-Free Composite Score is comprised of Oral Reading Recognition, Picture Vocabulary, Picture Sequence Memory, Motor-Free Pattern Comparison, and List Sorting Working Memory.

a

Those with “motor impairments” included those that were unable to complete the timed motor tests (scores were winsorized; i.e., participants were given a T score that was 1 point lower than the lowest score on that test, by participants who could take the test, for n = 24 individuals with spinal cord injury (SCI), n = 1 for individuals with traumatic brain injury (TBI), and n = 4 for individuals with stroke) and/or those individuals who had T scores <40 with the dominant hand on the 9-Hole Pegboard from the NIHTB Motor Battery (n = 202 individuals with SCI, n = 78 for individuals with TBI, and n = 89 for individuals with stroke).

Relationships Among Motor-Free Composites and Original Composite Scores

Table 5 presents the intraclass correlations among the different composite scores; in all cases, the Original and its matched Motor-Free Composite score were correlated ≥0.74, supporting convergent validity.

Table 5.

Intraclass Correlations (ICC) Among Motor-Free Composite Scores and Original Composite Scores for Combined Sample

Fluid Reasoning Overall Cognition
Original Motor-Free Original
NIHTB-CB measure/score n ICC n ICC n ICC
Fluid Reasoning
 Original Composite — —
 Motor-Free Composite 497 .80
Overall Cognition
 Original Composite 492 .86 492 .74
 Motor-Free Composite 492 .77 522 .93 492 .91
Open in a new tab

Note. NIHTB-CB = National Institutes of Health Toolbox Cognition Battery. The Original Fluid Cognition Composite Score is comprised of Picture Sequence Memory, Pattern Comparison Processing Speed, List Sorting Working Memory, Flanker Inhibitory Control and Attention, and Dimensional Change Card Sort; the Original Overall Cognition Composite Score is comprised of all 7 Core measures within the NIHTB-CB; the Fluid Cognition Motor-Free Composite Score is comprised of List Sorting Working Memory and Motor-Free Pattern Comparison; the Overall Cognition Motor-Free Score is comprised of the Oral Reading Recognition Test, Picture Vocabulary, Motor-Free Pattern Comparison, and List Sorting Working Memory. Sample sizes are variable due to missing data for participants that were unable to complete tasks that involved a motor response.

An exact McNemar’s test indicated significant differences in the number (%) of individuals with brain injury who were classified as cognitively impaired using the Motor-Free vs. Original Composites, with Motor-Free being somewhat higher (both p < .001; Table 6). With regard to both the Fluid and Overall Cognition Composites, the overall proportion of agreement was excellent, and the κ statistic indicated substantial agreement (see Table 6). Findings generally provide support for known-groups validity.

Table 6.

Classification Accuracy of the Different Composite Scores for Combined Brain-Injured Participants Without and Without Motor Impairment (i.e., TBI and Stroke; n = 339)

Composite Score No. classified as cognitively impaired by the Original Composite score no. (%) No. classified as cognitively impaired by Motor-Free composite score no. (%) No. of participants classified as impaired by both composites Overall proportion of agreement % κ
Fluid Cognition 115(35.2) 141 (43.1) 99 82.2 .63
Overall Cognition 73 (22.5) 109(33.6) 67 85.2 .639
Open in a new tab

Note. TBI = traumatic brain injury.

Table 7 presents the findings comparing the Motor-Free to the Original Composite scores. There were significant differences between the Motor-Free and Original Composite scores for individuals with SCI for both the Fluid Composites and the Overall Composites, with medium and small-to-medium effect sizes. In each case, the Motor-Free scores were higher. Although some statistically significant Overall Cognition differences were seen for the individuals with brain injury (TBI and stroke), effect sizes for these differences were below the cutoff for “small” (cutoff d = .20; observed d’s of −0.08 and −0.11).

Table 7.

Motor-Free Versus Original Composite Scores for Each Clinical Group (Individual Groups That Included Participants With and Without Motor Impairments), as Well as a Combined Total Brain-Injured Group (TBI/Stroke)

Original Fluid Reasoning Composite Score Motor-Free Fluid Reasoning Composite Score
Fluid Reasoning M SD M SD Cohen’s d df t P
SCI 45.04 8.61 50.04 9.70 .55 174 −7.49 <.0001
TBI 44.12 8.16 43.22 12.94 −.08 151 1.46 .15
Stroke 42.57 7.73 42.36 12.81 −.02 169 .36 .72
TBI/stroke 43.30 7.96 42.77 12.86 −.05 321 1.26 .21
Original Overall Cognition Composite Score Motor-Free Overall Cognition Composite Score
Overall Cognition M SD M SD Cohen’s d df T P
SCI 47.80 7.35 50.49 9.21 .32 172 −6.93 <.0001
TBI 46.90 7.34 45.79 11.69 −.11 150 2.48 .01
Stroke 46.11 7.59 45.68 11.37 −.04 167 1.08 .28
TBI/stroke 46.48 7.47 45.73 11.50 −.08 318 2.52 .01
Open in a new tab

Note. SCI = spinal cord injury; TBI = traumatic brain injury. The Original Fluid Cognition Composite Score is comprised of Picture Sequence Memory, Pattern Comparison Processing Speed, List Sorting Working Memory, Flanker Inhibitory Control and Attention, and Dimensional Change Card Sort; the Original Overall Cognition Composite Score is comprised of all 7 Core measures within the NIHTB-CB; the Motor-Free Fluid Cognition Composite Score is comprised of List Sorting Working Memory, Picture Sequence Memory, and Motor-Free Pattern Comparison; and the Motor-Free Overall Cognition Composite Score is comprised of Oral Reading Recognition, Picture Vocabulary, Picture Sequence Memory, Motor-Free Pattern Comparison, and List Sorting Working Memory; Cohen’s d were calculated by comparing the Original Scores to the Motor-Free scores for each respective clinical group.

Discussion

Findings provide support for utilizing Motor-Free composite scores in cases where a participant is unable to complete the standard NIHTB-CB processing speed measure (Pattern Comparison) because of upper extremity motor impairment. Correlations among each Motor-Free Composite with its matched Original Composite were moderate to strong.

Perhaps the best test of the usefulness of the Motor-Free Pattern Comparison can be seen in individuals with SCI without versus those with upper extremity motor impairments (see Table 4). Note that neither of these subgroups had any diagnosed brain injury, so they would not be expected to have cognitive processing speed deficits; thus, expected T scores for fluid and crystallized tests would be approximately 50. Consistent with that expectation, individuals with SCI without motor impairments had a mean Crystallized Composite score of 50.46, and mean Original and Motor-Free scores on Pattern Comparison, Fluid Composites, and Total Composites from 49.42 to55.63. It is the individuals with SCI with motor impairments that would be expected to show the largest difference between motor-free and original scores, because of the confounding effect of motor impairment on what is intended to be a cognitive test of processing speed. In fact, this is what is seen in Table 4: again, where a mean T score of about 50 would be expected (because of the absence of brain injury), for those individuals with SCI with motor impairment, the mean original Pattern Comparison score is 39.31, whereas the mean for Motor-Free Pattern Comparison is 52.32; similarly, related differences can be seen in the Composite scores.

Interpretation of original versus motor-free results for the combined group of individuals with brain injuries (TBI and stroke) is more complicated, because the mean T scores in Table 4 suggest that subgroups with motor impairment strongly suggest that they probably have more serious brain injuries and associated cognitive deficits. Thus, subgroups with motor impairments consistently obtained worse T scores on all NIHTB-CB tests that are untimed and not susceptible to motor confounds (i.e., Crystallized Cognition subtests and composites, as well as Picture Sequence Memory and List Sorting). On the other hand, it is possible that the higher scores of all individuals with brain injuries (i.e., those individuals with TBI and/or stroke with and without motor impairment) on the Motor-Free Pattern Comparison than the Original Pattern Comparison reflect somewhat reduced sensitivity of the Motor-Free scores to brain injury. Nevertheless, even if this may be true, data in Table 7 suggest that the Motor-Free Fluid and Total Composites (that include Motor-Free Pattern Comparison) are at least as sensitive to brain injury than are the Originals.

For the combined group of individuals with brain injury, there was also substantial agreement between the Motor-Free and Original Fluid Cognition scores and Motor-Free and Original Overall Cognition scores. While our findings indicate that the Motor-Free Composite scores are not a perfect match with the Original Composites (especially for the Fluid Reasoning scores), they are close enough to use with confidence in individuals that are precluded from completing the core NIHTB-CB measures because of physical limitations.

While the current study provides a possible, Motor-Free alternative to Pattern Comparison for individuals using the NIHTB-CB in clinical populations, it is also important to acknowledge several study limitations. First, analyses focused on three different clinical groups, but other diagnoses were not considered and we did not examine the impact that injury severity has on findings. Future work is needed to understand how injury severity impacts these Motor-Free Composite scores. In addition, since the OSD was designed as a Motor-Free substitute for the Pattern Comparison Processing Speed Test and there are no motor-free executive function tests that are available as part of the NIHTB-CB, the current Motor-Free composite scores do not include tests that evaluate executive function. While processing speed is typically a quite sensitive measure of cognitive impairment after brain injury (DeLuca, Chelune, Tulsky, Lengenfelder, & Chiaravalloti, 2004; Donders, Tulsky, & Zhu, 2001; Gontkovsky & Beatty, 2006), executive function is also commonly compromised in these individuals (Podell, Gifford, Bougakov, & Goldberg, 2010; Rabinowitz & Levin, 2014). Thus clinicians and researchers should consider including additional motor-free measures of executive function for a comprehensive assessment of cognition in individuals whose motor impairments may confound assessments of cognitive processing speed. Furthermore, suboptimal effort and litigation status were not evaluated for study participants, although our testing was being done for research (not clinical) purposes and examiners did not note any evidence of poor effort. In addition, while an oral version of the Pattern Comparison Processing Speed Test would provide a better analytical or measurement match with the original test, this type of assessment is currently not available for the NIHTB-CB. As the OSD was already available and conormed as a part of the NIHTB-CB, this presented the best motor-free option for a processing speed measure within this battery. Thus, future work could consider developing a more closely matched motor-free version of the Pattern Comparison Processing Speed Test that could be fully standardized, normed and integrated within the NIHTB-CB for use in individuals with motor impairments. Finally, only a small portion of the NIHTB normative sample completed the OSD, resulting in an inadequate and nonrepresentative subsample that precluded conorming the OSD with the NIHTB-CB tests. Future work is also needed in both clinical and healthy populations to examine test–retest reliability and norms for change over time of these Motor-Free Composites.

Regardless of these limitations, findings provide important information for the clinical utility of the NIHTB in individuals who have upper extremity motor impairments. Specifically, rehabilitation psychologists should consider administering the OSD from the NIHTB-CB when examining their patients, especially if these individuals have associated motor impairments that may compromise their performance on tests requiring speeded motor responses. This subtest can provide clinically relevant information about processing speed, and can be used to generate Motor-Free Composite scores that can ultimately maximize the clinical utility of the NIHTB-CB in individuals with upper extremity physical functioning limitations. Such data allow clinicians and researchers to gather clinically relevant information while simultaneously using the “common currency” that was intended in the original conceptualization of the NIHTB.

Acknowledgments

This study was supported by the NIDILRR RRTC on Improving Measurement of Medical Rehabilitation Outcomes (Award H133B090024, PI: Allen W. Heinemann). This project was also supported in part with Federal funds from the Blueprint for Neuroscience Research, National Institutes of Health, under Contract HHS-N-260-2006-00007-C.

Contributor Information

Robert K. Heaton, Department of Psychiatry, University of California San Diego;

Allen W. Heinemann, Shirley Ryan AbilityLab, Chicago, Illinois, and Department of Physical Medicine and Rehabilitation, Northwestern University Feinberg School of Medicine;

Richard C. Gershon, Department of Medical Social Sciences, Department of Preventive Medicine, and Institute for Health Services Research & Policy Studies, Feinberg School of Medicine, Northwestern University;

David S. Tulsky, Center for Health Assessment Research and Translation, and the Departments of Physical Therapy and Psychological and Brain Science, University of Delaware.

References

  1. American Spinal Injury Association. (2002). International standards for neurological and functional classification of spinal cord injury. Atlanta, GA: American Spinal Injury Association, International Spinal Cord Society. [Google Scholar]
  2. Brennan R (2004). Linking with Equivalent Group or Single Group Design (LEGS; Version 2.0) [Computer software]. Iowa City, IA: University of Iowa, Center for Advanced Studies in Measurement and Assessment (CASMA). [Google Scholar]
  3. Campbell DT, & Fiske DW (1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56, 81–105. http://dx.doi.org/10.1037/h0046016 [PubMed] [Google Scholar]
  4. Carlozzi NE, Goodnight S, Casaletto KB, Goldsmith A, Heaton RK, Wong AWK, … Tulsky DS (in press). Validation of the NIH Toolbox (NIHTB) in individuals with neurologic disorders. Archives of Clinical Neuropsychology. [DOI] [PMC free article] [PubMed] [Google Scholar]
  5. Carlozzi NE, Tulsky DS, Chiaravalloti ND, Beaumont JL, Weintraub S, Conway K, & Gershon RC (2014). NIH Toolbox Cognitive Battery (NIHTB-CB): The NIHTB Pattern Comparison Processing Speed Test. Journal of the International Neuropsychological Society, 20, 630–641. http://dx.doi.org/10.1017/S1355617714000319 [DOI] [PMC free article] [PubMed] [Google Scholar]
  6. Carlozzi NE, Tulsky DS, Wolf T, Goodnight S, Heaton R, Casaletto KB, … Heinemann A (in press). Construct validity of the NIH Toolbox Cognition Battery in individuals with stroke. Rehabilitation Psychology. [DOI] [PMC free article] [PubMed] [Google Scholar]
  7. Casaletto KB, Umlauf A, Beaumont J, Gershon R, Slotkin J, Akshoomoff N, & Heaton RK (2015). Demographically corrected normative standards for the English Version of the NIH Toolbox Cognition Battery. Journal of the International Neuropsychological Society, 21, 378–391. http://dx.doi.org/10.1017/S1355617715000351 [DOI] [PMC free article] [PubMed] [Google Scholar]
  8. Choi SW, Schalet B, Cook KF, & Cella D (2014). Establishing a common metric for depressive symptoms: Linking the BDI-II, CES-D, and PHQ-9 to PROMIS depression. Psychological Assessment, 26, 513–527. http://dx.doi.org/10.1037/a0035768 [DOI] [PMC free article] [PubMed] [Google Scholar]
  9. Cohen J (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20, 37 http://dx.doi.org/10.1177/001316446002000104 [Google Scholar]
  10. Cohen J (1992). A power primer. Psychological Bulletin, 112, 155–159. http://dx.doi.org/10.1037/0033-2909.112.1.155 [DOI] [PubMed] [Google Scholar]
  11. Cohen ML, Tulsky DS, Holdnack JA, Carlozzi NE, Wong A, Magasi S, … Heinemann A (in press). Cognition among community-dwelling individuals with spinal cord injury. Rehabilitation Psychology. [DOI] [PMC free article] [PubMed] [Google Scholar]
  12. DeLuca J, Chelune GJ, Tulsky DS, Lengenfelder J, & Chiaravalloti ND (2004). Is speed of processing or working memory the primary information processing deficit in multiple sclerosis? Journal of Clinical and Experimental Neuropsychology, 26, 550–562. http://dx.doi.org/10.1080/13803390490496641 [DOI] [PubMed] [Google Scholar]
  13. Dikmen SS, Bauer PJ, Weintraub S, Mungas D, Slotkin J, Beaumont JL, … Heaton RK (2014). Measuring episodic memory across the lifespan: NIH Toolbox Picture Sequence Memory Test. Journal of the International Neuropsychological Society, 20, 611–619. http://dx.doi.org/10.1017/S1355617714000460 [DOI] [PMC free article] [PubMed] [Google Scholar]
  14. Donders J, Tulsky DS, & Zhu J (2001). Criterion validity of new WAIS-II subtest scores after traumatic brain injury. Journal of the International Neuropsychological Society, 7, 892–898. [PubMed] [Google Scholar]
  15. Enderby P, & Crow E (1996). Frenchay aphasia screening test: Validity and comparability. Disability and Rehabilitation, 18, 238–240. http://dx.doi.org/10.3109/09638289609166307 [DOI] [PubMed] [Google Scholar]
  16. Enderby PM, Wood VA, Wade DT, & Hewer RL (1986). The Frenchay Aphasia Screening Test: A short, simple test for aphasia appropriate for non-specialists. International Rehabilitation Medicine, 8, 166–170. http://dx.doi.org/10.3109/03790798709166209 [DOI] [PubMed] [Google Scholar]
  17. Gershon RC, Cook KF, Mungas D, Manly JJ, Slotkin J, Beaumont JL, & Weintraub S (2014). Language measures of the NIH Toolbox Cognition Battery. Journal of the International Neuropsycho-logical Society, 20, 642–651. http://dx.doi.org/10.1017/S1355617714000411 [DOI] [PMC free article] [PubMed] [Google Scholar]
  18. Gershon RC, Slotkin J, Manly JJ, Blitz DL, Beaumont JL, Schnipke D, … Weintraub S. (2013). IV. NIH Toolbox Cognition Battery (CB): Measuring language (vocabulary comprehension and reading decoding). Monographs of the Society for Research in Child Development, 78, 49–69. http://dx.doi.org/10.1111/mono.12034 [DOI] [PMC free article] [PubMed] [Google Scholar]
  19. Gontkovsky ST, & Beatty WW (2006). Practical methods for the clinical assessment of information processing speed. International Journal of Neuroscience, 116, 1317–1325. http://dx.doi.org/10.1080/00207450500516537 [DOI] [PubMed] [Google Scholar]
  20. Heaton RK, Akshoomoff N, Tulsky D, Mungas D, Weintraub S, Dikmen S, … Gershon R. (2014). Reliability and validity of composite scores from the NIH Toolbox Cognition Battery in adults. Journal of the International Neuropsychological Society, 20, 588–598. http://dx.doi.org/10.1017/S1355617714000241 [DOI] [PMC free article] [PubMed] [Google Scholar]
  21. Koh CL, Lu WS, Chen HC, Hsueh IP, Hsieh JJ, & Hsieh CL (2011). Test-retest reliability and practice effect of the oral-format Symbol Digit Modalities Test in patients with stroke. Archives of Clinical Neuropsychology, 26, 356–363. http://dx.doi.org/10.1093/arclin/acr029 [DOI] [PubMed] [Google Scholar]
  22. Kolen MJ, & Brennan RL (Eds.). (2014). Test equating, scaling, and linking (3rd ed.). New York, NY: Spinger. [Google Scholar]
  23. Magasi S, Harniss M, Tulsky D, Cohen M, Heaton R, & Heinemann A (in press). Frequency of Test Accommodations for individuals with neurological conditions completing the NIH Toolbox - Cognition Battery. Rehabilitation Psychology. [DOI] [PMC free article] [PubMed] [Google Scholar]
  24. McHugh ML (2012). Interrater reliability: The kappa statistic. Biochemia Medica, 22, 276–282. http://dx.doi.org/10.11613/BM.2012.031 [PMC free article] [PubMed] [Google Scholar]
  25. Noonan VK, Cook KF, Bamer AM, Choi SW, Kim J, & Amtmann D (2012). Measuring fatigue in persons with multiple sclerosis: Creating a crosswalk between the Modified Fatigue Impact Scale and the PROMIS Fatigue Short Form. Quality of Life Research, 21, 1123–1133. http://dx.doi.org/10.1007/s11136-011-0040-3 [DOI] [PubMed] [Google Scholar]
  26. Podell K, Gifford K, Bougakov D, & Goldberg E (2010). Neuropsychological assessment in traumatic brain injury. Psychiatric Clinics of North America, 33, 855–876. http://dx.doi.org/10.1016/j.psc.2010.08.003 [DOI] [PubMed] [Google Scholar]
  27. Rabinowitz AR, & Levin HS (2014). Cognitive sequelae of traumatic brain injury. Psychiatric Clinics of North America, 37, 1–11. http://dx.doi.org/10.1016/j.psc.2013.11.004 [DOI] [PMC free article] [PubMed] [Google Scholar]
  28. Rao SM, Leo GJ, Ellington L, Nauertz T, Bernardin L, & Unverzagt F (1991). Cognitive dysfunction in multiple sclerosis. II. Impact on employment and social functioning. Neurology, 41, 692–696. http://dx.doi.org/10.1212/WNL.41.5.692 [DOI] [PubMed] [Google Scholar]
  29. Salter K, Jutai J, Foley N, Hellings C, & Teasell R (2006). Identification of aphasia post stroke: A review of screening assessment tools. Brain Injury, 20, 559–568. http://dx.doi.org/10.1080/02699050600744087 [DOI] [PubMed] [Google Scholar]
  30. Traumatic Brain Injury Model Systems National Data Center. (2006). Traumatic brain injury model systems national data base inclusion criteria. Retrieved from http://www.tbindsc.org/Documents/2010%20TBIMS%20Slide%20Presentation.pdf [Google Scholar]
  31. Tulsky DS, Carlozzi N, Chiaravalloti ND, Beaumont JL, Kisala PA, Mungas D, … Gershon R. (2014). NIH Toolbox Cognition Battery (NIHTB-CB): List sorting test to measure working memory. Journal of the International Neuropsychological Society, 20, 599–610. http://dx.doi.org/10.1017/S135561771400040X [DOI] [PMC free article] [PubMed] [Google Scholar]
  32. Tulsky DS, Carlozzi NE, Holdnack JA, Heaton RK, Wong A, Goldsmith A, & Heinemann A (in press). Using the NIH Toolbox Cognition Battery (NIHTB-CB) in Individuals with Traumatic Brain Injury. Rehabilitation Psychology. [DOI] [PMC free article] [PubMed] [Google Scholar]
  33. Wang YC, Magasi SR, Bohannon RW, Reuben DB, McCreath HE, Bubela DJ, … Rymer WZ. (2011). Assessing dexterity function: A comparison of two alternatives for the NIH Toolbox. Journal of Hand Therapy, 24, 313–321. http://dx.doi.org/10.1016/j.jht.2011.05.001 [DOI] [PMC free article] [PubMed] [Google Scholar]
  34. Weintraub S, Dikmen SS, Heaton RK, Tulsky DS, Zelazo PD, Bauer PJ, … Gershon RC. (2013). Cognition assessment using the NIH Toolbox. Neurology, 80(Suppl. 3), S54–S64. http://dx.doi.org/10.1212/WNL.0b013e3182872ded [DOI] [PMC free article] [PubMed] [Google Scholar]
  35. Weintraub S, Dikmen SS, Heaton RK, Tulsky DS, Zelazo PD, Slotkin J, … Gershon R. (2014). The cognition battery of the NIH toolbox for assessment of neurological and behavioral function: Validation in an adult sample. Journal of the International Neuropsychological Society, 20, 567–578. http://dx.doi.org/10.1017/S1355617714000320 [DOI] [PMC free article] [PubMed] [Google Scholar]
  36. WHO MONICA Project Principal Investigators. (1988). The World Health Organization MONICA Project (monitoring trends and determinants in cardiovascular disease): A major international collaboration. Journal of Clinical Epidemiology, 41, 105–114. http://dx.doi.org/10.1016/0895-4356(88)90084-4 [DOI] [PubMed] [Google Scholar]
  37. Williams DH, Levin HS, & Eisenberg HM (1990). Mild head injury classification. Neurosurgery, 27, 422–428. http://dx.doi.org/10.1227/00006123-199009000-00014 [DOI] [PubMed] [Google Scholar]
  38. Zelazo PD, Anderson JE, Richler J, Wallner-Allen K, Beaumont JL, Conway KP, … Weintraub S. (2014). NIH Toolbox Cognition Battery (CB): Validation of executive function measures in adults. Journal of the International Neuropsychological Society, 20, 620–629. http://dx.doi.org/10.1017/S1355617714000472 [DOI] [PMC free article] [PubMed] [Google Scholar]
  39. Zelazo PD, Anderson JE, Richler J, Wallner-Allen K, Beaumont JL, & Weintraub S (2013). II. NIH Toolbox Cognition Battery (CB): Measuring executive function and attention. Monographs of the Society for Research in Child Development, 78, 16–33. http://dx.doi.org/10.1111/mono.12032 [DOI] [PubMed] [Google Scholar]

RESOURCES