Skip to main content
PMC home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2018 Nov 1.
Published in final edited form as: Rehabil Psychol. 2017 Nov;62(4):474–484. doi: 10.1037/rep0000145

NIH Toolbox Cognition Tests Following Traumatic Brain Injury: Frequency of Low Scores

James A Holdnack 1, Grant L Iverson 2, Noah D Silverberg 3, David S Tulsky 4, Allen W Heinemann 5
PMCID: PMC5745050  NIHMSID: NIHMS905784  PMID: 29265868

Abstract

Purpose/Objective

To apply multivariate base rate analyses to the National Institutes of Health Toolbox Cognition Battery (NIHTB-CB) to facilitate the identification of cognitive impairment in individuals with traumatic brain injury (TBI).

Research Method/Design

In a multisite cross-sectional design, 158 participants who sustained a complicated mild or moderate TBI (n=74) or severe TBI (n=84) at least one year earlier were administered the NIHTB-CB. The NIHTB-CB is comprised of two crystallized cognition tests (reflecting premorbid ability) and five fluid cognition tests, measuring processing speed, memory, and executive functioning. Base rates for obtaining 0 to 5 low fluid cognition scores were calculated across a range of cutoffs for defining a low test score (≤25th to 5th percentiles). Base rates of low scores in the TBI sample were compared to the NIHTB-CB normative sample using diagnostic accuracy statistics.

Results

The proportion of the TBI sample obtaining low scores decreased as the cutoff for defining a low score decreased. Individuals with lower premorbid cognitive ability, as measured by NIHTB-CB crystallized composite score, tended to produce more low scores on the NIHTB-CB fluid cognition tests, even when using fully demographically-adjusted scores. Certain patterns of low scores were associated with TBI (defined as Likelihood Ratio>2.0), whereas others were non-specific, occurring almost as often in participants without TBI.

Conclusions/Implications

Premorbid ability stratified base rate tables provided in this article can guide researchers and clinicians in the interpretation of NIHTB-CB performance in adults with TBI.

Keywords: Multivariate Base Rates, Cognition, NIH Toolbox, Traumatic Brain Injury, Cognitive Impairment

Introduction

Cognitive impairment caused by moderate or severe traumatic brain injury (TBI) might have diverse neurobiological underpinnings, such as focal contusions, hematomas, diffuse axonal injury, edema, cellular dysfunction (e.g., excitotoxicity, calcium overload, oxidative stress, mitochondrial dysfunction, and inflammation), impaired synaptic transmission, cell death (necrosis or apoptosis), and axonal degeneration (Walker & Tesco, 2013). Cognition is often severely affected in the initial days and weeks, and tends to gradually improve over time (Christensen et al., 2008; Karr, Areshenkoff, & Garcia-Barrera, 2014; Schretlen & Shapiro, 2003). When considering impairment in groups of people, those with severe brain injuries are likely to have some degree of persisting or permanent cognitive impairment, and those who sustain mild TBIs are unlikely to have persisting impairment (Christensen et al., 2008; Dikmen, Machamer, & Temkin, 2001; Dikmen, Machamer, Winn, & Temkin, 1995; Draper & Ponsford, 2008; Karr et al., 2014; Levin, Grossman, Rose, & Teasdale, 1979; Ruttan, Martin, Liu, Colella, & Green, 2008; Schretlen & Shapiro, 2003). Attention, concentration, working memory, speed of processing, and memory are often most affected (Dikmen et al., 2001; Dikmen, Machamer, Powell, & Temkin, 2003; Dikmen et al., 1995; Dikmen, McLean, Temkin, & Wyler, 1986; Iverson, 2005; Lezak, Howieson, & Loring, 2004; Mearns & Lees-Haley, 1993; Spikman, Timmerman, Zomeren van, & Deelman, 1999; Whyte, Schuster, Polansky, Adams, & Coslett, 2000). Most neuropsychological recovery following moderate or severe TBI occurs within the first year (Christensen et al., 2008; Dikmen et al., 1995), although some additional recovery can occur thereafter (Millis et al., 2001). Recovery tends to be more rapid after mild TBI (Karr et al., 2014; Schretlen & Shapiro, 2003).

Clinicians and researchers typically administer several tests when evaluating cognitive impairment in people who have sustained TBIs. These tests yield multiple scores that are interpreted in combination to determine if a person has cognitive deficits. The psychometric criteria for cognitive impairment varies, but commonly used cut-offs include scores that are 1 or 1.5 standard deviations (SD) below the mean. The Diagnostic and Statistical Manual of Mental Disorders-Fifth Edition (DSM 5; American Psychiatric Association, 2013) recommends a cut-off score of 1 SD below the mean for defining mild neurocognitive disorder. Applying this criterion in clinical practice and research, however, is not straightforward. Researchers have reported that children (Brooks, Sherman, & Iverson, 2010), adults (Iverson, Brooks, & Holdnack, 2012; Schretlen, Testa, Winicki, Pearlson, & Gordon, 2008), and older adults (Brooks, Iverson, & White, 2009) with no known medical, psychiatric, or neurological conditions that could affect cognition often obtain one or more low scores when a battery of tests are administered (Binder, Iverson, & Brooks, 2009). Individuals with fewer years of education or below average intellectual abilities are more likely to obtain low scores than people with university degrees or above average intellectual abilities (Brooks, Holdnack, & Iverson, 2011). Therefore, a “one size fits all” criterion for identifying cognitive deficits, even if normative data are corrected for level of education (Brooks, Iverson, Holdnack, & Feldman, 2008), will result in greater false positives in those with below average intelligence and greater false negatives in those with above average intelligence (Brooks et al., 2008). A methodology for understanding low scores in the general population, when several cognitive tests are administered, is called multivariate base rate analyses (Brooks, Iverson, & Holdnack, 2013). Multivariate base rate analyses allow clinicians and researchers to determine if a combination or pattern of low scores is common or uncommon in the general population. This approach can be used to operationalize the psychometric criterion for DSM-5 mild neurocognitive disorder due to TBI in clinical and research settings.

Brief, efficient, and reliable computerized cognitive assessment can be useful in clinical practice and research involving those who have sustained TBIs. The National Institutes of Health (NIH) Toolbox for the Assessment of Neurological and Behavioral Function (Gershon et al., 2010; Gershon et al., 2013) Cognition Battery (Weintraub et al., 2013) is a computerized neuropsychological screening battery that has been recommended as a common data element for TBI research and clinical trials (https://www.commondataelements.ninds.nih.gov/tbi.aspx#tab=Data_Standards). The NIH Toolbox Cognition Battery (NIHTB-CB) measures attention, working memory, language, processing speed, and executive functioning. In a validation paper, Tulsky and colleagues illustrate that the NIH-TBCB is sensitive to persistent cognitive impairment following TBI, and the Fluid Composite and Picture Sequence Memory Test have the largest effect sizes (Tulsky et al., under review). However, there is considerable heterogeneity in pre-injury vulnerabilities, mechanisms of injury, and TBI pathophysiology, all of which may underlie the heterogeneity in long-term cognitive outcomes from TBI (Allen et al., 2010; Di Paola et al., 2015; Martin et al., 2016; Sigurdardottir et al., 2015).Group mean differences between people with vs. without TBI represent an “averaging” of individuals with diverse neuropsychological profiles, such as diffuse mild impairment vs. circumscribed severe impairment in a particular cognitive domain. This group-level information cannot be readily applied to address the clinical challenge of determining whether a specific person fulfills criteria for neuropsychological impairment (Chelune, 2010).

Therefore, the purpose of this study is to apply multivariate base rate analyses to the NIHTB-CB to facilitate the identification of persistent cognitive impairment following TBI. Iit is essential for researchers and clinicians to know how common it is for those with TBIs to obtain certain combinations of low scores, compared to adults without TBI. The current study estimates base rates of low scores for the NIHTB-CB in those with TBIs for cut-off scores based on the 25th, 16th, 9th, and 5th percentiles.

Methods

Participants

Three samples were used in this study: the normative sample for the NIHTB-CB and two samples of people with TBIs. The NIHTB-CB normative sample was used to estimate the population prevalence of low scores, and comprehensive base rate analyses on this battery are available (Holdnack et al., under review). To be included in the standardization sample base rates, examinees had to have complete data on all seven of the NIHTB-CB tests; 843 individuals were used in the age-adjusted multivariate base rates and 793 subjects were used in the demographically-adjusted base rate analyses because they also needed to have complete data on all demographic variables. The normative sample (N=843) had an average age of 47.4 years (SD=17.4) and an average education of 14.2 years (SD=2.5). The breakdown of the sample by education was as follows: less than 12 years=9.6%, 12 years=26.3%, 13-15 years=23.8%, 16 or more years=39.4%, and unknown=0.8%. Women comprised 65.6% of the sample. The breakdown of the sample by race was as follows: Caucasian=64.5%, African American=16.4%, Hispanic=10.7%, other=6.6%, and not provided=1.8%.

One hundred and eighty-two individuals with a medically documented complicated mild or moderate TBI (n=83), or severe TBI (n=99), were administered the NIHTB-CB as part of a larger study. These are the same clinical samples as used in a broader study relating to the NIHTB-CB following TBI (Tulsky et al., under review). They were recruited at three rehabilitation centers: Rehabilitation Institute of Chicago (Northwestern University), Washington University, and the University of Michigan. Institutional Review Board approval was obtained from each participating site. Inclusion criteria for this analysis were scores on all seven subtests of the NIHTB-CB, age at least 18 years, admission to a hospital within 24 hours of a TBI, ability to comprehend and speak English at a 5th grade level, willingness and ability to return for follow-up testing, and at least one year must have passed since their brain injury. Individuals with cognitive impairments due to other conditions such as a psychiatric disorder, Alzheimer's disease, or other dementing illness were excluded.

TBI severity was classified according to the lowest GCS score within the first 24 hours after injury (not due to intubation, sedation, or intoxication). A GCS score of ≤8 was classified as severe injury, a score of 9-12 was classified as moderate injury, and complicated mild cases had a GCS score of 13-15 with neuroimaging showing acute brain abnormality consistent with TBI, such as subarachnoid hemorrhaging or cortical contusions. If no GCS was available, TBI cases were classified based on the detailed description of their injury and confirmed by a neuropsychologist on staff independent of performance on the NIHTB-CB or traditional neurocognitive tests. See Figure 1 in Tulsky et al. (under review) for more information regarding the review of medical records and the classification of TBI severity.

The TBI sample (N=158) included 74 with a complicated mild/moderate TBI and 84 with a severe TBI; their average age was 38.6 years (SD=17.4). Their average education was 13.8 years (SD=2.5). The breakdown of the sample by education was as follows: less than 12 years=13.9%, 12 years=18.4%, 13-15 years=34.8%, and 16 or more years=32.9%. Women comprised 62.0% of the sample. The breakdown of the sample by race was as follows: Caucasian=72.2%, African American=12.0%, Hispanic=7.6%, other=7.6%, and not provided=0.6%. Their average time since injury was as follows: total sample M=5.8 years, SD=5.6, Range=1-29, complicated MTBI/moderate M=4.6 years, SD=4.2, Range=1-21, and severe TBI M=6.8 years, SD=6.3, Range=1-29.

Measures

The NIHTB-CB is comprised of seven tests that measure different aspects of crystallized and fluid cognitive abilities. The crystallized tests are Picture Vocabulary and Oral Reading Recognition. They yield individual scores and they are combined into a Crystallized Cognition Composite Score. For individuals with acquired brain injury, the Crystallized Cognition Composite Score can serve as an estimate of pre-injury general cognitive ability. This composite correlates strongly with other known “hold” tests (Gershon et al., 2014) and is resistant to the effects of TBI (Tulsky et al., under review). The fluid cognition tests are List Sorting Working Memory, Picture Sequence Memory, Pattern Comparison Processing Speed, Flanker Inhibitory Control and Attention Test (measuring both attention and executive functioning), and the Dimensional Change Card Sort Test. These five tests are combined into a Fluid Cognition Composite Score. The age-adjusted scale scores have a mean of 100 and a standard deviation (SD) of 15. The demographically-adjusted T scores are adjusted for age, gender, race (white, black, other), ethnicity (Hispanic vs. non-Hispanic), and education; they have a mean of 50 and a SD of 10 (Casaletto et al., 2015). Detailed description of the norming procedure is provided elsewhere (Casaletto et al., 2015).

One of the advantages of the NIHTB-CB is the relatively short administration time required to cover the cognitive domains most likely to be impacted by brain injury as well as providing an estimate of premorbid abilities. Total test administration time of the NIHTB-CB is approximately 30 minutes. This brief administration time enables the clinician and researcher to survey important cognitive domains without undue burden on the examinee. Furthermore, the 5 subtests designed to be sensitive to brain injury allow for a multivariate analysis of results which may improve sensitivity and specificity compared to a single score screening measure.

Results

The multivariate base rates of low scores for the five fluid tests of the NIHTB-CB for age and demographically adjusted scores in the combined TBI sample are presented in Table 1. Four different cutoff scores for defining a low score were used: ≤25th, 16th, 9th, and 5th percentiles. Individuals with lower premorbid cognitive ability tended to produce more low scores on the NIHTB-CB fluid subtests. Specifically, rates of low age-adjusted scores (i.e., ≤16th percentile) on the fluid tests were significantly correlated with the NIHTB-CB Crystallized composite (r=-.45, p<.001), Picture Vocabulary Test (r=-.46, p<.001), and Oral Reading Recognition Test (r=-.37, p<.001). This relationship was also observed for demographically-adjusted scores: Crystallized composite (r=-.30, p<.001), Picture Vocabulary Test (r=-.35, p<.001), and Oral Reading Recognition Test (r=-.19, p<.05). Table 1 provides base rates stratified by estimated premorbid ability level, in addition to overall base rates of low scores. The rightmost columns (Demographic Norms by Crystallized Composite) represent the greatest adjustments for variance associated with premorbid factors. Age, gender, race, ethnicity, education level, and estimated premorbid ability (i.e., performance on the NIHTB-CB Crystallized composite) jointly determined to which stratum an individual belongs.

Table 1. Prevalence of Low Scores on Five NIHTB-CB Fluid Measures Overall and by Ability Level in the Combined TBI Sample.

Low Scores Age Norms Age Norms by Education Level Age Norms by Crystallized Composite Demographic Norms Demographic Norms by Crystallized Composite
< 12 12 13-15 16+ < 90 90-99 100-109 110 + < 43 43-49 50-57 58 +
n=158 n=17 n=33 n=56 n=52 n=43 n=37 n=42 n=36 n=158 n=31 n=42 n=48 n=37
≤25th percentile
5 19.0 41.2 24.2 19.6 7.7 37.2 27.0 7.1 2.8 17.1 29.0 21.4 14.6 5.4
4+ 34.8 47.1 48.5 35.7 21.2 55.8 48.7 19.0 13.9 32.3 51.6 42.9 22.9 16.2
3+ 48.7 58.8 63.6 50.0 34.6 74.4 59.5 35.7 22.2 48.7 71.0 59.5 37.5 32.4
2+ 64.6 76.5 81.8 62.5 51.9 90.7 70.3 47.6 47.2 65.2 90.3 69.1 58.3 48.7
1+ 75.9 94.1 87.9 76.8 61.5 97.7 83.8 64.3 55.6 78.5 96.8 85.7 77.1 56.8
None 24.1 5.9 12.1 23.2 38.5 2.3 16.2 35.7 44.4 21.5 3.2 14.3 22.9 43.2
≤16th percentile
5 13.3 35.3 12.1 14.3 5.8 25.6 18.9 4.8 2.8 11.4 19.4 21.4 6.3 0.0
4+ 26.0 47.1 33.3 26.8 13.5 51.2 35.1 7.1 8.3 22.8 41.9 28.6 12.5 13.5
3+ 38.6 58.8 45.5 42.9 23.1 65.1 51.4 19.0 16.7 37.3 54.8 45.2 29.2 24.3
2+ 53.8 64.7 66.7 51.8 44.2 79.1 62.2 35.7 36.1 51.9 77.4 57.1 41.7 37.8
1+ 63.9 76.5 75.8 66.1 50.0 88.4 70.3 50.0 44.4 67.7 90.3 78.6 60.4 45.9
None 36.1 23.5 24.2 33.9 50.0 11.6 29.7 50.0 55.6 32.3 9.7 21.4 39.6 54.1
≤9th percentile
5 7.6 29.4 9.1 7.1 0.0 18.6 10.8 0.0 0.0 8.2 16.1 11.9 6.3 0.0
4+ 14.6 35.3 21.2 16.1 1.9 32.6 18.9 4.8 0.0 14.6 29.0 19.0 8.3 5.4
3+ 25.9 47.1 39.4 26.8 9.6 51.2 35.1 9.5 5.6 26.0 45.2 33.3 16.7 13.5
2+ 43.7 52.9 60.6 41.1 32.7 72.1 54.1 26.2 19.5 41.8 67.7 52.4 31.2 21.6
1+ 51.3 64.7 63.6 48.2 42.3 76.7 59.5 35.7 30.6 52.5 80.6 64.3 41.7 29.7
None 48.7 35.3 36.4 51.8 57.7 23.3 40.5 64.3 69.4 47.5 19.4 35.7 58.3 70.3
≤5th percentile
5 2.5 5.9 6.1 1.8 0.0 4.7 5.4 0.0 0.0 1.9 3.2 4.8 0.0 0.0
4+ 7.0 23.5 12.1 3.6 1.9 18.6 5.4 2.4 0.0 6.3 19.4 7.1 2.1 0.0
3+ 15.2 41.2 18.2 14.3 5.8 30.2 24.3 4.8 0.0 13.3 25.8 14.3 14.6 0.0
2+ 31.0 41.2 33.3 35.7 21.2 53.5 43.3 11.9 13.9 28.5 54.8 35.7 20.8 8.1
1+ 42.4 52.9 51.5 44.7 30.8 67.4 54.1 21.4 25.0 39.2 61.3 50.0 29.2 21.6
None 57.6 47.1 48.5 55.4 69.2 32.6 46.0 78.6 75.0 60.8 38.7 50.0 70.8 78.4
Open in a new tab

Note: Age norms are presented as standard scores with M=100 and SD=15, and demographically-adjusted norms are presented as T scores (M=50, SD=10).

A few examples, applying the results in Table 1, highlight the importance of premorbid ability stratified base rates (BRs). When considering the five fluid tests, 63.9% of the TBI sample had at least one age-adjusted score at or below the 16th percentile. For individuals with low crystallized ability (< 90), 88.4% had at least one low fluid age-adjusted test score. In contrast, it was less common (BR=44.4%) for individuals with high crystallized ability (110+) to have an age-adjusted score at or below the 16th percentile. That is, individuals with low crystallized ability were twice as likely to have at least one low score (≤16th percentile). BR differences between individuals with low vs. high premorbid ability tend to become more extreme as the low score cutoff becomes more conservative and/or the number of low scores increases. For example, individuals with low crystallized ability (<90) were almost four times more likely to obtain three or more age-adjusted scores at or below the 16th percentile (BR=65.1% vs. 16.7%) and almost 10 times more likely to obtain or more+ scores at or below the 9th percentile (BR=51.6% vs. 5.1%) as individuals with high crystallized ability (>110). Differences between participants in each strata of crystallized ability were also present when using fully demographically-adjusted instead of age-adjusted norms. For example, obtaining three or more scores at or below the 16th percentile occurred in 54.8% of participants with low crystallized ability and 24.3% of participants with high crystallized ability, compared to 65.1% and 16.7% based on age-adjusted norms.

Base Rates and Probability Statistics

Comprehensive analyses were undertaken to examine diagnostic accuracy statistics for patterns of low scores on the NIHTB-CB. For these analyses, the TBI subsamples were compared to the normative sample, adults from the general population. Demographically-adjusted normative scores were used (Casaletto et al., 2015).The results of these analyses are presented in Tables 2-5. A range of cutoff scores was examined (i.e., ≤25th, 16th, 9th, and 5th percentiles). The number of low scores is presented in the first column, and the base rate (i.e., prevalence) of the normative sample that obtained that number of low scores is presented in the second column. For example, 15% of the normative sample obtained three or more Fluid cognition test scores ≤25th percentile, compared to 38% of those with complicated mild or moderate TBIs (column 3) and 58% of those with severe TBIs (column 4). Specificity (column 5) represents 1.0 minus the normative base rate. The likelihood ratios (LR+ and LR-) are clinically helpful metrics that indicate how much more likely a given test result (e.g., 1+ scores at or below the 5th percentile) is to occur in an individual with and without the condition of interest (e.g., severe TBI) than in an individual without that condition of interest. The LR+ is calculated as sensitivity/(1-specificity). The higher the LR, the stronger evidence that the condition of interest is present. A LR of greater than two provides significant evidence for the presence of the condition of interest. The LR- is calculated as 1-sensitivity/specificity and represents the probability that the absence of a positive test result is associated with the condition of interest not being present. Lower scores represent greater certainty that the test results indicate the condition is not present. Scores below .5 indicate greater certainty that the condition of interest is not present. LRs for mild complicated/moderate and severe TBI groups are also provided in Table 3.

Table 2. Diagnostic Accuracy Statistics for the NIHTB-CB Tests Demographically-Adjusted Norms for TBI Subgroup Samples Applying the ≤25th Percentile Cut-off Score.

Number of Low Scores Normative Sample Base Rate Mild/Mod Base Rate Severe Base Rate Spec LR + Mild/Mod LR- Mild/Mod LR+ Severe LR - severe
Total Sample n=793 n=74 n=84
0 .38 0.28 0.15 n/a n/a n/a n/a n/a
1+ .62 0.72 0.85 0.38 1.16 0.74 1.37 0.39
2+ .33 0.57 0.73 0.67 1.73 0.64 2.21 0.40
3+ .15 0.38 0.58 0.85 2.53 0.73 3.87 0.49
4+ .08 0.23 0.40 0.92 2.88 0.84 5.00 0.65
5 .02 0.12 0.21 0.98 6.00 0.90 10.50 0.81
Crystallized Composite < 43 n=78 n=11 n=20
0 0.18 0.00 0.05 n/a n/a n/a n/a n/a
1+ 0.83 1.00 0.95 0.17 1.20 0.00 1.14 0.29
2+ 0.61 0.91 0.90 0.39 1.49 0.23 1.48 0.26
3+ 0.33 0.73 0.70 0.67 2.21 0.40 2.12 0.45
4+ 0.16 0.45 0.55 0.84 2.81 0.65 3.44 0.54
5 0.07 0.18 0.35 0.93 2.57 0.88 5.00 0.70
Crystallized Composite 43-49 n=204 n=20 n=22
0 0.38 0.20 0.09 n/a n/a n/a n/a n/a
1 + 0.62 0.80 0.91 0.38 1.29 0.53 1.47 0.24
2+ 0.37 0.55 0.82 0.63 1.49 0.71 2.22 0.29
3+ 0.19 0.40 0.77 0.81 2.11 0.74 4.05 0.28
4+ 0.12 0.25 0.59 0.88 2.08 0.85 4.92 0.47
5 0.03 0.20 0.23 0.97 6.67 0.82 7.67 0.79
Crystallized Composite 50-57 n=229 n=21 n=27
0 0.40 0.38 0.11 n/a n/a n/a n/a n/a
1+ 0.60 0.62 0.89 0.40 1.03 0.95 1.48 0.28
2+ 0.25 0.52 0.63 0.75 2.08 0.64 2.52 0.49
3+ 0.09 0.29 0.44 0.91 3.22 0.78 4.89 0.62
4+ 0.03 0.19 0.26 0.97 6.33 0.84 8.67 0.76
5 0.01 0.10 0.19 0.99 10.00 0.91 19.00 0.82
Crystallized Composite 58+ n=194 n=22 n=15
0 0.57 0.41 0.47 n/a n/a n/a n/a n/a
1+ 0.43 0.59 0.53 0.57 1.37 0.72 1.23 0.82
2+ 0.13 0.45 0.53 0.87 3.46 0.63 4.08 0.54
3+ 0.03 0.27 0.40 0.97 9.00 0.75 13.33 0.62
4+ 0.01 0.14 0.20 0.99 14.00 0.87 20.00 0.81
5 0.00 0.05 0.07 1.00 -- 0.95 -- 0.93
Open in a new tab

Note: base rate= sensitivity, spec=specificity, LR + = Likelihood Ratio Positive, LR- = Likelihood Ratio Negative

Table 5. Probability Statistics for the NIHTB-CB Tests Demographically-Adjusted Norms for TBI Subgroups Applying the ≤5th Percentile Cut-off Score.

Number of Low Scores Normative Sample Base Rate Mild/Mod Base Rate Severe Base Rate Spec LR + Mild/Mod LR- Mild/Mod LR+ Severe LR - severe
Total Sample n=793 n=74 n=84
0 .83 0.69 0.54 n/a n/a n/a n/a n/a
1+ .17 0.31 0.46 0.83 1.82 0.83 2.71 0.65
2+ .05 0.18 0.38 0.95 3.60 0.86 7.60 0.65
3+ .02 0.08 0.18 0.98 4.00 0.94 9.00 0.84
4+ .00 0.05 0.07 1.00 -- 0.95 -- 0.93
5 .00 0.03 0.01 1.00 -- 0.97 -- 0.99
Crystallized Composite < 43 n=78 n=11 n=20
0 0.66 0.55 0.30 n/a n/a n/a n/a n/a
1+ 0.34 0.45 0.70 0.66 1.32 0.83 2.06 0.45
2+ 0.15 0.27 0.70 0.85 1.80 0.86 4.67 0.35
3+ 0.06 0.09 0.35 0.94 1.50 0.97 5.83 0.69
4+ 0.02 0.09 0.25 0.98 4.50 0.93 12.50 0.77
5 0.00 0.00 0.05 1.00 -- 1.00 -- 0.95
Crystallized Composite 43-49 n=204 n=20 n=22
0 0.81 0.70 0.32 n/a n/a n/a n/a n/a
1+ 0.19 0.30 0.68 0.81 1.58 0.86 3.58 0.40
2+ 0.04 0.25 0.45 0.96 6.25 0.78 11.25 0.57
3+ 0.02 0.15 0.14 0.98 7.50 0.87 7.00 0.88
4+ 0.00 0.10 0.05 1.00 -- 0.90 -- 0.95
5 0.00 0.10 0.00 1.00 -- 0.90 -- 1.00
Crystallized Composite 50-57 n=229 n=21 n=27
0 0.89 0.67 0.74 n/a n/a n/a n/a n/a
1+ 0.11 0.33 0.26 0.89 3.00 0.75 2.36 0.83
2+ 0.01 0.19 0.22 0.99 19.00 0.82 22.00 0.79
3+ 0.00 0.10 0.19 1.00 -- 0.90 -- 0.81
4+ 0.00 0.05 0.00 1.00 -- 0.95 -- 1.00
5 0.00 0.00 0.00 1.00 -- 1.00 -- 1.00
Crystallized Composite 58+ n=194 n=22 n=15
0 0.93 0.77 0.80 n/a n/a n/a n/a n/a
1+ 0.07 0.23 0.20 0.93 3.29 0.83 2.86 0.86
2+ 0.01 0.05 0.13 0.99 5.00 0.96 13.00 0.88
3+ 0.00 0.00 0.00 1.00 -- 1.00 -- 1.00
4+ 0.00 0.00 0.00 1.00 -- 1.00 -- 1.00
5 0.00 0.00 0.00 1.00 -- 1.00 -- 1.00
Open in a new tab

Note: base rate= sensitivity, spec=specificity, LR + = Likelihood Ratio Positive, LR- = Likelihood Ratio Negative

Table 3. Probability Statistics for the NIHTB-CB Tests Demographically-Adjusted Norms for TBI Subgroup Samples Applying ≤16th Percentile Cut-off Score.

Number of Low Scores Normative Sample Base Rate Mild/Mod Base Rate Severe Base Rate Spec LR + Mild/Mod LR- Mild/Mod LR+ Severe LR - severe
Total Sample n=793 n=74 n=84
0 .54 0.41 0.25 n/a n/a n/a n/a n/a
1+ .46 0.59 0.75 0.54 1.28 0.76 1.63 0.46
2+ .21 0.43 0.60 0.79 2.05 0.72 2.86 0.51
3+ .10 0.28 0.45 0.90 2.80 0.80 4.50 0.61
4+ .04 0.16 0.29 0.96 4.00 0.88 7.25 0.74
5 .01 0.08 0.14 0.99 8.00 0.93 14.00 0.87
Crystallized Composite < 43 n=78 n=11 n=20
0 0.25 0.18 0.05 n/a n/a n/a n/a n/a
1+ 0.75 0.82 0.95 0.25 1.09 0.72 1.27 0.20
2+ 0.41 0.64 0.85 0.59 1.56 0.61 2.07 0.25
3+ 0.23 0.55 0.55 0.77 2.39 0.58 2.39 0.58
4+ 0.11 0.27 0.50 0.89 2.45 0.82 4.55 0.56
5 0.04 0.09 0.25 0.96 2.25 0.95 6.25 0.78
Crystallized Composite 43-49 n=204 n=20 n=22
0 0.52 0.30 0.14 n/a n/a n/a n/a n/a
1+ 0.48 0.70 0.86 0.52 1.46 0.58 1.79 0.27
2+ 0.24 0.40 0.73 0.76 1.67 0.79 3.04 0.36
3+ 0.13 0.25 0.64 0.87 1.92 0.86 4.92 0.41
4+ 0.05 0.20 0.36 0.95 4.00 0.84 7.20 0.67
5 0.01 0.20 0.23 0.99 20.00 0.81 23.00 0.78
Crystallized Composite 50-57 n=229 n=21 n=27
0 0.60 0.57 0.26 n/a n/a n/a n/a n/a
1+ 0.40 0.43 0.74 0.60 1.08 0.95 1.85 0.43
2+ 0.14 0.38 0.44 0.86 2.71 0.72 3.14 0.65
3+ 0.05 0.19 0.37 0.95 3.80 0.85 7.40 0.66
4+ 0.01 0.14 0.11 0.99 14.00 0.87 11.00 0.90
5 0.00 0.05 0.07 1.00 -- 0.95 -- 0.93
Crystallized Composite 58+ n=194 n=22 n=15
0 0.74 0.45 0.67 n/a n/a n/a n/a n/a
1+ 0.26 0.55 0.33 0.74 2.12 0.61 1.27 0.91
2+ 0.08 0.41 0.33 0.92 5.13 0.64 4.13 0.73
3+ 0.01 0.27 0.20 0.99 27.00 0.74 20.00 0.81
4+ 0.01 0.09 0.20 0.99 9.00 0.92 20.00 0.81
5 0.00 0.00 0.00 1.00 -- 1.00 -- 1.00
Open in a new tab

Note: base rate= sensitivity, spec=specificity, LR + = Likelihood Ratio Positive, LR- = Likelihood Ratio Negative

Considering the 16th percentile (Table 3) as the cutoff for a low score, 59% of individuals with complicated mild/moderate TBI and 75% of individuals with severe TBI have one or more low scores. Although that would suggest a high degree of sensitivity, the normative base rate of 46% indicates poor specificity (i.e., 54% or .54). The LR+ indicates that individuals with a single score at or below the 16th percentile are only 1.3 times more likely to be identified as being in the mild-complicated/moderate TBI group than being in the normative sample and 1.6 times more likely to be identified as being in the severe TBI group than being in the normative sample. Therefore, this test finding (1+ score < 16th percentile on the NIHTB-CB fluid tests) represents LR+ scores below 2.0 which suggests that it is not significantly associated with TBI. The LR- score of .76 for mild complicated/moderate indicates that the condition cannot be ruled out either (i.e., more test information is required). For severe injuries, the LR- of .46 while not completely ruling out the condition provides more evidence for its absence. In contrast, having three Fluid cognition test scores at or below the 16th percentile (Table 3) is 2.8 times more likely in the mild-complicated/moderate TBI group compared to the normative sample, and 4.5 times more likely in the severe TBI group compared to the normative sample, suggesting that a diagnosis of persistent cognitive impairment due to TBI is likely.

The base rates stratified by estimated premorbid ability allow the use of different cut-offs to maximize classification accuracy. For example, for people in the normative sample with above average cognitive ability (Crystallized Composite T=58+), only 13% obtain two or more scores ≤25th percentile, compared to 45% of those with complicated mild/moderate TBIs and 53% of those with severe TBIs (Table 2). LR+s of 3.46 and 4.08 suggest that those with TBIs and above average estimated premorbid ability who show this pattern of low scores are likely to have acquired cognitive deficits. Using a cut-off score of ≤16th percentile (Table 3), having three or more low scores is uncommon in the normative sample for those with average cognitive ability (BRs 5-13%). Having four or more low scores is uncommon for those in the normative sample with below average cognitive ability (Crystallized Composite<T=43; BR=11%). For those with above average cognitive ability (Crystallized Composite T=58+), having two or more scores ≤16th percentile occurs in only 8% of the normative sample compared to 41% and 33% of the two TBI groups (LR+s=5.1 and 4.1).

Algorithms for Identifying Cognitive Impairment

Algorithms for identifying cognitive impairment are presented in Table 6. The first row presents two algorithms for cognitive impairment that can be applied to the general population. The normative base rate represents the percentage of adults in the general population who meet one or more of the criteria when all criteria are applied simultaneously. That is, there is an “or” statement after each criterion (4+ scores or 3+ scores or 2+ scores). If a researcher or clinician wanted to require the presence of at least two low scores as the definition of cognitive deficits, and also wanted to standardize the cut-off for the pattern of impairment across the five fluid cognitive tests as being approximately one SD below the mean (e.g., per DSM-5 mild neurocognitive disorder psychometric criteria), then the criteria for impairment using demographically-adjusted T scores should be different based on a person's estimated level of crystallized intellectual ability. For example, criteria for cognitive impairment, stratified by estimated pre-injury level of cognitive functioning, could be as follows: (a) High Average Crystallized Composite (T=58+), two or more scores ≤25th percentile, normative BR=13.4%, mild-moderate TBI=45.4%, severe TBI=53.3%; (b) Average Crystallized Composite (T=50-57), two or more scores ≤16th percentile, normative BR=14.0%, mild-moderate TBI=38.1%, severe TBI=44.4%; (c) Average Crystallized Composite (T=43-49), two or more scores ≤9th percentile, normative BR=14.7%, mild-moderate TBI=35.0%, severe TBI=68.2%; and Low Average Crystallized Composite (T<43), two or more scores ≤5th percentile, normative BR=14.5%, mild-moderate TBI=27.3%, and severe TBI=70.0%. Notice that these ability-stratified criteria for cognitive impairment hold the false positive rate (i.e., the normative base rate) constant. Multiple algorithms are presented in Table 6 so that the results can be available for comparison to results from future studies. These algorithms are not exhaustive but they represent a methodology for creating criteria for identifying cognitive difficulties that incorporates knowledge of multivariate base rates that have a relatively low false positive rate (i.e., < 15%).

Table 6. Algorithms for defining cognitive impairment applied to the TBI sample.

Demographically-Adjusted Scores Criteria Normative Base Rate Complicated Mild/Moderate TBI Severe TBI LR + Mild/Mod LR+ Severe
Adult General Population 4+ scores ≤25th, or 3+ scores ≤16th, or 2+ scores ≤5th 11.8% 37.8% 53.6% 3.2 4.5
4+ scores ≤25th, or 3+ scores ≤16th, or 2+ scores ≤9th 14.2% 41.9% 53.6% 3.0 3.8
Crystallized Composite <SS90/<T43 2+ scores ≤5th 14.5% 27.3% 70.0% 1.9 5.0
4+ scores ≤16th or 3+ scores ≤9th 14.5% 45.4% 55.0% 3.2 3.9
Crystallized Composite SS90-99/T43-49 2+ scores ≤9th 14.7% 35.0% 68.2% 2.3 4.5
Crystallized Composite SS100-109/T50-57 2+ scores ≤16th 14.0% 38.1% 44.4% 2.7 3.1
3+ scores ≤25th or 2+ scores ≤16th 9.3% 40.0% 16.7% 4.4 1.9
1+ score ≤5th 10.9% 33.3% 25.9% 3.0 2.4
Crystallized Composite SS110+/T58+ 2+ scores ≤25th 13.4% 45.4% 53.4% 3.5 4.1
2+ scores ≤25th or 1+ scores ≤5th 16.0% 45.4% 53.3% 2.8 3.3
1+ scores ≤9th 15.5% 36.4% 20.0% 2.3 1.3
1+ score ≤5th 7.2% 22.7% 20.0% 3.3 2.9
Open in a new tab

Discussion

Cognitive impairment can occur following a mild, moderate, or severe TBI, and the type and severity of cognitive deficits, recovery trajectory, and risk for permanency can be difficult to predict in individual cases. The NIHTB-CB can be used by clinicians and researchers to quantify cognitive functioning in people with TBIs, and to screen people for DSM-5 mild neurocognitive disorder. This study provides validity evidence for the NIHTB-CB in TBI. Another study involving the same samples reported significant group differences between people with and without TBI, and illustrated that performance on the NIHTB-CB is associated with injury severity (Tulsky et al., under review). The present study adds two important findings. First, we illustrated that performance on the NIHTB-CB in TBI is strongly related to estimated premorbid ability. As with uninjured examinees on this battery (Holdnack et al., under review) and on various other neuropsychological test batteries (e.g., Brooks et al., 2011), individuals with TBI who had lower premorbid ability obtained more low scores. Correcting for age, education, gender, race, and ethnicity with demographically-adjusted norms was not sufficient to remove the variance associated with premorbid intellectual ability, and in fact, only modestly attenuated the differences between participants with differing levels of crystallized ability as measured by the NIHTB-CB Crystallized composite. This finding is consistent with our previous studies with other test batteries and does not reflect any known specific factors (e.g., sampling or norming method) associated with the NIHTB-CB. The clinical implication is that crystallized ability stratified base rates provide a more accurate estimate of the likelihood of persistent cognitive impairment in a given person, compared to when crystallized ability is not considered. Second, the present study identified psychometric algorithms, stratified by estimated intellectual ability that can be used to identify cognitive impairment in people with TBIs (see Table 6).

The findings from this study are relevant to clinicians and researchers working in rehabilitation settings. There is a significant body of research showing that cognitive impairment after TBI has a negative impact on daily living skills (Colantonio et al., 2004), community reintegration, and return to competitive employment (Cattelani, Lombardi, & Mazzucchi, 2002). The NIHTB-CB provides a relatively brief screening battery for identifying those at risk for poor functional outcomes and can also be used to triage patients for more comprehensive neuropsychological evaluations to assist with rehabilitation planning.

Researchers and clinicians can use the data provided in Table 6 to screen patients or research participants with TBI for DSM-5 mild neurocognitive disorder. The psychometric criteria for doing so could be having two or more low scores, considering the five fluid test scores. The cutoff for defining a low score can vary based on the person's Crystallized Composite score. For example, for people with Crystallized Composite scores between T=50-57, having two or more scores ≤16th percentile occurs in 14.0% of the normative sample, 38.1% of those with complicated mild or moderate TBIs, and 44.4% of those with severe TBIs. For people who have above average estimated intellectual ability (Crystallized Composite T=58+), the criterion for screening positively for mild neurocognitive disorder could be having two or more scores ≤25th percentile. This pattern of performance occurs in 13.4% of the normative sample, 45.4% of those with complicated mild or moderate TBIs, and 53.4% of those with severe TBIs. In Table 6, we used the Crystallized composite score as a proxy for pre-injury Crystallized abilities. Of course, obtained scores might be lower than pre-injury scores, especially in people who have sustained severe TBIs. For example, a person with a severe TBI who obtained a Crystallized Composite score of 57 would have been included in the analyses based on the criterion for the group with scores between 50 and 57, when in fact this person's pre-injury Crystallized Composite might easily have been 58 or greater. Thus, for people who were on the border of an ability classification range, applying these criteria likely resulted in an under-estimation of the true rate of cognitive impairment. Research is needed to determine if a small upward adjustment in obtained Crystallized composite scores improves the diagnostic accuracy of these algorithms in people with neurological disorders, such as severe TBI.

The present study has several limitations. All participants were assessed at a single point in time. Neuropsychological functioning after TBI is dynamic, with a general trajectory for gradual improvement. Longitudinal research is needed to examine diagnostic accuracy of the NIHTB-CB during periods of neurologic recovery after TBI. The present study did not include individuals with the mildest forms of TBI, such as those who are discharged from an Emergency Department, without admission to inpatient rehabilitation. It would have been advantageous to include a control group with traumatic injuries to regions other than the head. Stratifying participants on demographic variables and estimated premorbid cognitive ability should have partly accounted for the pre-injury dissimilarities between uninjured controls and people with TBI.

In conclusion, the present study further supports the validity of the NIHTB-CB for assessing cognition in people with post-acute mild-complicated to severe TBI. By referencing the tables provided here, researchers and clinicians can apply the diagnostic accuracy properties of NIHTB-CB to identify individual people with persistent cognitive impairment after TBI. Rehabilitation psychologists emphasize the psychosocial assets and capabilities of people with disabilities (Dunn, Ehde, & Wegener, 2016), and they promote a focus on individuals' strengths and the importance of one's self-perception on the evaluation of attributes (Shontz & Wright, 1980; Wright, 1972). Findings from this study inform rehabilitation psychologists' practice by providing NIHTB-CB score interpretations that put low scores in perspective.

Table 4. Probability Statistics for the NIHTB-CB Tests Demographically-Adjusted Norms for TBI Subgroups Applying the ≤9th Percentile Cut-off Score.

Number of Low Scores Normative Sample Base Rate Mild/Mod Base Rate Severe Base Rate Spec LR + Mild/Mod LR- Mild/Mod LR+ Severe LR - severe
Total Sample n=793 n=74 n=84
0 .68 0.57 0.39 n/a n/a n/a n/a n/a
1+ .32 0.43 0.61 0.68 1.34 0.84 1.91 0.57
2+ .08 0.34 0.49 0.92 4.25 0.72 6.13 0.55
3+ .02 0.19 0.32 0.98 9.50 0.83 16.00 0.69
4+ .01 0.08 0.20 0.99 8.00 0.93 20.00 0.81
5 .00 0.07 0.10 1.00 -- 0.93 -- 0.90
Crystallized Composite < 43 n=78 n=11 n=20
0 0.43 0.36 0.10 n/a n/a n/a n/a n/a
1+ 0.57 0.64 0.90 0.43 1.12 0.84 1.58 0.23
2+ 0.28 0.55 0.75 0.72 1.96 0.63 2.68 0.35
3+ 0.11 0.36 0.50 0.89 3.27 0.72 4.55 0.56
4+ 0.06 0.09 0.40 0.94 1.50 0.97 6.67 0.64
5 0.01 0.09 0.20 0.99 9.00 0.92 20.00 0.81
Crystallized Composite 43-49 n=204 n=20 n=22
0 0.66 0.55 0.18 n/a n/a n/a n/a n/a
1+ 0.34 0.45 0.82 0.66 1.32 0.83 2.41 0.27
2+ 0.15 0.35 0.68 0.85 2.33 0.76 4.53 0.38
3+ 0.04 0.25 0.41 0.96 6.25 0.78 10.25 0.61
4+ 0.01 0.15 0.23 0.99 15.00 0.86 23.00 0.78
5 0.01 0.15 0.09 0.99 15.00 0.86 9.00 0.92
Crystallized Composite 50-57 n=229 n=21 n=27
0 0.74 0.62 0.56 n/a n/a n/a n/a n/a
1+ 0.26 0.38 0.44 0.74 1.46 0.84 1.69 0.76
2+ 0.06 0.33 0.30 0.94 5.50 0.71 5.00 0.74
3+ 0.01 0.14 0.19 0.99 14.00 0.87 19.00 0.82
4+ 0.00 0.10 0.07 1.00 -- 0.90 -- 0.93
5 0.00 0.05 0.07 1.00 -- 0.95 -- 0.93
Crystallized Composite 58+ n=194 n=22 n=15
0 0.85 0.64 0.80 n/a n/a n/a n/a n/a
1+ 0.16 0.36 0.20 0.84 2.25 0.76 1.25 0.95
2+ 0.03 0.23 0.20 0.97 7.67 0.79 6.67 0.82
3+ 0.01 0.09 0.20 0.99 9.00 0.92 20.00 0.81
4+ 0.00 0.00 0.13 1.00 -- 1.00 -- 0.87
5 0.00 0.00 0.00 1.00 -- 1.00 -- 1.00
Open in a new tab

Note: base rate= sensitivity, spec=specificity, LR + = Likelihood Ratio Positive, LR- = Likelihood Ratio Negative

Impact.

  • The National Institutes of Health (NIH) Toolbox for the Assessment of Neurological and Behavioral Function Cognition Battery is a computerized neuropsychological screening battery that can be used as a brief, efficient, and reliable cognitive assessment in clinical practice and research involving those who have sustained traumatic brain injuries.

  • The prevalence of low scores on this battery is associated with, or influenced by, age, education, race, ethnicity, level of intelligence, and brain injury.

  • The identification of cognitive impairment using this battery can be improved by applying multivariate base rate analyses and algorithms.

Acknowledgments

Standardization data from NIH Toolbox used with permission. This research was supported by grants H133B090024, H122G070138 from the National Institute on Disability, Independent Living, and Rehabilitation Research. Support for JAH was also provided by an Institutional Development Award (IDeA) from the National Institute of General Medical Sciences of the National Institutes of Health under grant number U54-GM104941(PI: Binder-Macleod). GLI acknowledges support from the Mooney-Reed Charitable Foundation, the Heinz Family Foundation, and ImPACT Applications, Inc. This work is related in part to the TBI Endpoints Development Initiative and a grant titled Development and Validation of a Cognition Endpoint for Traumatic Brain Injury Clinical Trials.

Footnotes

Financial Disclosure: The authors have no financial interest in the NIH Toolbox.

General Disclosure: GLI has been reimbursed by the government, professional scientific bodies, and commercial organizations for discussing or presenting research relating to mild TBI and sport-related concussion at meetings, scientific conferences, and symposiums. He has a clinical practice in forensic neuropsychology involving individuals who have sustained mild TBIs. He has received honorariums for serving on research panels that provide scientific peer review of programs. He is a co-investigator, collaborator, or consultant on grants relating to mild TBI funded by several organizations. He has received research support from test publishing companies in the past, including ImPACT® Applications Systems and CNS Vital Signs. He receives royalties for one neuropsychological test (Wisconsin Card Sorting Test-64 Card Version).

Contributor Information

James A. Holdnack, Center on Assessment Research and Translation, University of Delaware

Grant L. Iverson, Department of Physical Medicine and Rehabilitation, Harvard Medical School; Spaulding Rehabilitation Hospital; MassGeneral Hospital for Children™ Sports Concussion Program; & Home Base, A Red Sox Foundation and Massachusetts General Hospital Program

Noah D. Silverberg, Division of Physical Medicine & Rehabilitation, University of British Columbia; GF Strong Rehab Centre; Vancouver, BC, Canada & Department of Physical Medicine and Rehabilitation, Harvard Medical School, Boston, MA, USA

David S. Tulsky, Center on Assessment Research and Translation & and Departments of Physical Therapy, Psychological and Brain Sciences, University of Delaware

Allen W. Heinemann, Rehabilitation Institute of Chicago

References

  1. Allen DN, Leany BD, Thaler NS, Cross C, Sutton GP, Mayfield J. Memory and attention profiles in pediatric traumatic brain injury. Archives of Clinical Neuropsychology. 2010;25:618–633. doi: 10.1093/arclin/acq051. [DOI] [PubMed] [Google Scholar]
  2. American Psychiatric Association. Diagnostic and statistical manual of mental disorders. 5th. Washington, DC: American Psychiatric Association; 2013. [Google Scholar]
  3. Binder LM, Iverson GL, Brooks BL. To err is human: “Abnormal” neuropsychological scores and variability are common in healthy adults. Archives of Clinical Neuropsychology. 2009;24:31–46. doi: 10.1093/arclin/acn001. [DOI] [PubMed] [Google Scholar]
  4. Brooks BL, Holdnack JA, Iverson GL. Advanced clinical interpretation of the WAIS-IV and WMS-IV: prevalence of low scores varies by level of intelligence and years of education. Assessment. 2011;18:156–167. doi: 10.1177/1073191110385316. [DOI] [PubMed] [Google Scholar]
  5. Brooks BL, Iverson GL, Holdnack JA. Understanding and using multivariate base rates with the WAIS-IV/WMS-IV. In: Holdnack JA, Drozdick LW, Weiss LG, Iverson GL, editors. WAIS-IV/WMS-IV/ACS: Advanced clinical interpretation. San Diego, CA: Elsevier Science; 2013. pp. 75–102. [Google Scholar]
  6. Brooks BL, Iverson GL, Holdnack JA, Feldman HH. Potential for misclassification of mild cognitive impairment: a study of memory scores on the Wechsler Memory Scale-III in healthy older adults. Journal of the International Neuropsychological Society. 2008;14:463–478. doi: 10.1017/S1355617708080521. [DOI] [PubMed] [Google Scholar]
  7. Brooks BL, Iverson GL, White T. Advanced interpretation of the Neuropsychological Assessment Battery (NAB) with older adults: Base rate analyses, discrepancy scores, and interpreting change. Archives of Clinical Neuropsychology. 2009;24:647–657. doi: 10.1093/arclin/acp061. [DOI] [PubMed] [Google Scholar]
  8. Brooks BL, Sherman EM, Iverson GL. Healthy children get low scores too: prevalence of low scores on the NEPSY-II in preschoolers, children, and adolescents. Archives of Clinical Neuropsychology. 2010;25:182–190. doi: 10.1093/arclin/acq005. [DOI] [PubMed] [Google Scholar]
  9. Casaletto KB, Umlauf A, Beaumont J, Gershon R, Slotkin J, Akshoomoff N, Heaton RK. Demographically Corrected Normative Standards for the English Version of the NIH Toolbox Cognition Battery. Journal of the International Neuropsychological Society. 2015;21:378–391. doi: 10.1017/S1355617715000351. [DOI] [PMC free article] [PubMed] [Google Scholar]
  10. Cattelani R, Tanzi F, Lombardi F, Mazzuchi A. Competitive re-employment after severe traumatic brain injury: clinical, cognitive and behavioural predictive variables. Brain Injury. 2002;16:51–64. doi: 10.1080/02699050110088821. [DOI] [PubMed] [Google Scholar]
  11. Chelune GJ. Evidence-based research and practice in clinical neuropsychology. The Clinical Neuropsychologist. 2010;24:454–467. doi: 10.1080/13854040802360574. [DOI] [PubMed] [Google Scholar]
  12. Christensen BK, Colella B, Inness E, Hebert D, Monette G, Bayley M, Green RE. Recovery of cognitive function after traumatic brain injury: a multilevel modeling analysis of Canadian outcomes. Archives of Physical Medicine and Rehabilitation. 2008;89:S3–15. doi: 10.1016/j.apmr.2008.10.002. [DOI] [PubMed] [Google Scholar]
  13. Colantonio A, Ratcliff G, Chase S, Kelsey S, Escobar M, Vernich L. Long term outcomes after moderate to severe traumatic brain injury. Disability and Rehabilitation. 2004;26:253–261. doi: 10.1080/09638280310001639722. [DOI] [PubMed] [Google Scholar]
  14. Di Paola M, Phillips O, Costa A, Ciurli P, Bivona U, Catani S, et al. Carlesimo GA. Selective Cognitive Dysfunction Is Related to a Specific Pattern of Cerebral Damage in Persons With Severe Traumatic Brain Injury. Journal of Head Trauma Rehabilitation. 2015;30:402–410. doi: 10.1097/HTR.0000000000000063. [DOI] [PubMed] [Google Scholar]
  15. Dikmen S, Machamer J, Temkin N. Mild head injury: facts and artifacts. Journal of Clinical and Experimental Neuropsychology. 2001;23:729–738. doi: 10.1076/jcen.23.6.729.1019. [DOI] [PubMed] [Google Scholar]
  16. Dikmen S, Machamer JE, Powell JM, Temkin NR. Outcome 3 to 5 years after moderate to severe traumatic brain injury. Archives of Physical Medicine and Rehabilitation. 2003;84:1449–1457. doi: 10.1016/s0003-9993(03)00287-9. [DOI] [PubMed] [Google Scholar]
  17. Dikmen S, Machamer JE, Winn R, Temkin NR. Neuropsychological outcome 1-year post head injury. Neuropsychology. 1995;9:80–90. [Google Scholar]
  18. Dikmen S, McLean A, Jr, Temkin NR, Wyler AR. Neuropsychologic outcome at one-month postinjury. Archives of Physical Medicine and Rehabilitation. 1986;67:507–513. [PubMed] [Google Scholar]
  19. Draper K, Ponsford J. Cognitive functioning ten years following traumatic brain injury and rehabilitation. Neuropsychology. 2008;22:618–625. doi: 10.1037/0894-4105.22.5.618. [DOI] [PubMed] [Google Scholar]
  20. Dunn DS, Ehde DM, Wegener ST. The foundational principles as psychological lodestars: Theoretical inspiration and empirical direction in rehabilitation psychology. Rehabilitation Psychology. 2016;61:1–6. doi: 10.1037/rep0000082. [DOI] [PubMed] [Google Scholar]
  21. Gershon RC, Cella D, Fox NA, Havlik RJ, Hendrie HC, Wagster MV. Assessment of neurological and behavioural function: the NIH Toolbox. Lancet Neurology. 2010;9:138–139. doi: 10.1016/S1474-4422(09)70335-7. [DOI] [PubMed] [Google Scholar]
  22. Gershon RC, Cook KF, Mungas D, Manly JJ, Slotkin J, Beaumont JL, Weintraub S. Language measures of the NIH Toolbox Cognition Battery. Journal of the International Neuropsychological Society. 2014;20:642–651. doi: 10.1017/S1355617714000411. [DOI] [PMC free article] [PubMed] [Google Scholar]
  23. Gershon RC, Wagster MV, Hendrie HC, Fox NA, Cook KF, Nowinski CJ. NIH Toolbox for Assessment of Neurological and Behavioral Function. Neurology. 2013;80:S2–6. doi: 10.1212/WNL.0b013e3182872e5f. [DOI] [PMC free article] [PubMed] [Google Scholar]
  24. Holdnack JA, Tulsky DS, Brooks BL, Slotkin J, Gershon R, Heinemann AW, Iverson GL. Interpreting patterns of low scores on the NIH Toolbox Cognition Battery. doi: 10.1093/arclin/acx032. under review. [DOI] [PMC free article] [PubMed] [Google Scholar]
  25. Iverson GL. Outcome from mild traumatic brain injury. Current Opinion in Psychiatry. 2005;18:301–317. doi: 10.1097/01.yco.0000165601.29047.ae. [DOI] [PubMed] [Google Scholar]
  26. Iverson GL, Brooks BL, Holdnack JA. Evidence-based neuropsychological assessment following work-related injury. In: Bush SS, Iverson GL, editors. Neuropsychological assessment of work-related injuries. New York: Guilford Press; 2012. pp. 360–400. [Google Scholar]
  27. Karr JE, Areshenkoff CN, Garcia-Barrera MA. The neuropsychological outcomes of concussion: a systematic review of meta-analyses on the cognitive sequelae of mild traumatic brain injury. Neuropsychology. 2014;28:321–336. doi: 10.1037/neu0000037. [DOI] [PubMed] [Google Scholar]
  28. Levin HS, Grossman RG, Rose JE, Teasdale G. Long-term neuropsychological outcome of closed head injury. Journal of Neurosurgery. 1979;50:412–422. doi: 10.3171/jns.1979.50.4.0412. [DOI] [PubMed] [Google Scholar]
  29. Lezak MD, Howieson DB, Loring DW. Neuropsychological Assessment. 4th. New York: Oxford University Press; 2004. [Google Scholar]
  30. Martin RM, Wright MJ, Lutkenhoff ES, Ellingson BM, Van Horn JD, Tubi M, et al. Vespa PM. Traumatic hemorrhagic brain injury: impact of location and resorption on cognitive outcome. Journal of Neurosurgery. 2016:1–9. doi: 10.3171/2016.3.JNS151781. [DOI] [PubMed] [Google Scholar]
  31. Mearns J, Lees-Haley PR. Discriminating neuropsychological sequelae of head injury from alcohol-abuse-induced deficits: a review and analysis. Journal of Clinical Psychology. 1993;49:714–720. doi: 10.1002/1097-4679(199309)49:5<714::aid-jclp2270490515>3.0.co;2-h. [DOI] [PubMed] [Google Scholar]
  32. Millis SR, Rosenthal M, Novack TA, Sherer M, Nick TG, Kreutzer JS, et al. Ricker JH. Long-term neuropsychological outcome after traumatic brain injury. Journal of Head Trauma Rehabilitation. 2001;16:343–355. doi: 10.1097/00001199-200108000-00005. [DOI] [PubMed] [Google Scholar]
  33. Ruttan L, Martin K, Liu A, Colella B, Green RE. Long-term cognitive outcome in moderate to severe traumatic brain injury: a meta-analysis examining timed and untimed tests at 1 and 4.5 or more years after injury. Archives of Physical Medicine and Rehabilitation. 2008;89:S69–76. doi: 10.1016/j.apmr.2008.07.007. [DOI] [PubMed] [Google Scholar]
  34. Schretlen DJ, Shapiro AM. A quantitative review of the effects of traumatic brain injury on cognitive functioning. [**] International Review of Psychiatry. 2003;15:341–349. doi: 10.1080/09540260310001606728. [DOI] [PubMed] [Google Scholar]
  35. Schretlen DJ, Testa SM, Winicki JM, Pearlson GD, Gordon B. Frequency and bases of abnormal performance by healthy adults on neuropsychological testing. Journal of the International Neuropsychological Society. 2008;14:436–445. doi: 10.1017/S1355617708080387. [DOI] [PubMed] [Google Scholar]
  36. Shontz FC, Wright BA. The distinctiveness of rehabilitation psychology. Professional Psychology. 1980;11:919–924. [Google Scholar]
  37. Sigurdardottir S, Andelic N, Wehling E, Roe C, Anke A, Skandsen T, et al. Schanke AK. Neuropsychological functioning in a national cohort of severe traumatic brain injury: demographic and acute injury-related predictors. Journal of Head Trauma Rehabilitation. 2015;30:E1–12. doi: 10.1097/HTR.0000000000000039. [DOI] [PubMed] [Google Scholar]
  38. Spikman JM, Timmerman ME, Zomeren van AH, Deelman BG. Recovery versus retest effects in attention after closed head injury. Journal of Clinical and Experimental Neuropsychology. 1999;21:585–605. doi: 10.1076/jcen.21.5.585.874. [DOI] [PubMed] [Google Scholar]
  39. Tulsky DS, Carlozzi NE, Holdnack JA, Heaton RK, Wong A, Goldsmith A, Heinemann AW. Using the NIH Toolbox Cognition Battery (NIHTB-CB) in individuals with traumatic brain injury. Rehabilitation Psychology. doi: 10.1037/rep0000174. under review. [DOI] [PMC free article] [PubMed] [Google Scholar]
  40. Walker KR, Tesco G. Molecular mechanisms of cognitive dysfunction following traumatic brain injury. Front Aging Neurosci. 2013;5:29. doi: 10.3389/fnagi.2013.00029. [DOI] [PMC free article] [PubMed] [Google Scholar]
  41. Weintraub S, Dikmen SS, Heaton RK, Tulsky DS, Zelazo PD, Bauer PJ, et al. Gershon RC. Cognition assessment using the NIH Toolbox. Neurology. 2013;80:S54–64. doi: 10.1212/WNL.0b013e3182872ded. [DOI] [PMC free article] [PubMed] [Google Scholar]
  42. Whyte J, Schuster K, Polansky M, Adams J, Coslett HB. Frequency and duration of inattentive behavior after traumatic brain injury: effects of distraction, task, and practice. Journal of the International Neuropsychological Society. 2000;6:1–11. doi: 10.1017/s1355617700611013. [DOI] [PubMed] [Google Scholar]
  43. Wright BA. Value-laden beliefs and principles for rehabilitation psychology. Rehabilitation Psychology. 1972;19:38–45. [Google Scholar]

RESOURCES