Prospective, Head-to-Head Study of Three Computerized Neurocognitive Assessment Tools Part 2: Utility for Assessment of Mild Traumatic Brain Injury in Emergency Department Patients

Abstract

Objectives:

The aim of this study was to evaluate the reliability and validity of three computerized neurocognitive assessment tools (CNTs; i.e., ANAM, DANA, and ImPACT) for assessing mild traumatic brain injury (mTBI) in patients recruited through a level I trauma center emergency department (ED).

Methods:

mTBI (n = 94) and matched trauma control (n = 80) subjects recruited from a level I trauma center emergency department completed symptom and neurocognitive assessments within 72 hr of injury and at 15 and 45 days post-injury. Concussion symptoms were also assessed via phone at 8 days post-injury.

Results:

CNTs did not differentiate between groups at any time point (e.g., M 72-hr Cohen’s d = –.16, .02, and .00 for ANAM, DANA, and ImPACT, respectively; negative values reflect greater impairment in the mTBI group). Roughly a quarter of stability coefficients were over .70 across measures and test-retest intervals in controls. In contrast, concussion symptom score differentiated mTBI vs. control groups acutely), with this effect size diminished over time (72-hr and day 8, 15, and 45 Cohen’s d = –.78, –.60, –.49, and –.35, respectively).

Conclusions:

The CNTs evaluated, developed and widely used to assess sport-related concussion, did not yield significant differences between patients with mTBI versus other injuries. Symptom scores better differentiated groups than CNTs, with effect sizes weaker than those reported in sport-related concussion studies. Nonspecific injury factors, and other characteristics common in ED settings, likely affect CNT performance across trauma patients as a whole and thereby diminish the validity of CNTs for assessing mTBI in this patient population. (JINS, 2017, 23, 1–11)

INTRODUCTION

Community members frequently present to hospital emergency departments (EDs) with mild traumatic brain injury (mTBI; Cassidy et al., 2004; Faul, Xu, Wald, & Coronado, 2010). In the United States, for example, approximately 2.5 million ED visits, hospitalizations, or deaths were associated with TBI in 2010 (over 80% of which are categorized as mTBI based on acute injury characteristics; Centers for Disease Control and Prevention, 2015). Despite the fact that most mTBIs occur in civilian, patient-based populations (Bazarian, McClung, Shah, et al., 2005; Centers for Disease Control and Prevention, 2015), arguably more scientific progress has been made to understand the effects of the injury, and improve its assessment and management, in the subset of mTBIs precipitated by sports participation (i.e., sport-related concussion).

One such advance has been the development of standardized assessment tools that facilitate the injury’s identification and clinical management. Computerized neurocognitive test (CNT) batteries, in particular, have been developed and are now widely used for the assessment of athletes with mTBI. Of the available CNT batteries (all of which emphasize neurocognitive assessment and in some cases include a symptom checklist), ImPACT (Immediate Post-Concussion and Cognitive Testing) is the most widely used and studied in sports settings, although other tools have been developed for athlete (e.g., Axon Sports), military (Automated Neuropsychological Assessment Metrics and Defense Automated Neurobehavioral Assessment, i.e., ANAM and DANA, respectively), and other populations (e.g., CNS Vital Signs).

Given that most mTBIs occur outside the athletic arena, it is important to determine whether the neurocognitive assessment tools developed for athletes provide clinical utility for the broader mTBI population. The ED setting is particularly challenging for diagnosing and treating mTBI—the majority of mTBI patients in this setting do not realize their symptoms are indicative of mTBI (Delaney, Abuzeyad, Correa, & Foxford, 2005), and most who appear to meet CDC criteria for mTBI do not receive this diagnosis (Powell, Ferraro, Dikmen, Temkin, & Bell, 2008). Of those who are diagnosed, a large minority (37.2%) are not given recommendations for mTBI follow-up by their physicians (Bazarian, McClung, Cheng, Flesher, & Schneider, 2005).

A major factor that complicates the identification of mTBI in ED and other civilian mTBI populations is the high prevalence of comorbid personal and clinical features that mimic the effects of mTBI, thereby rendering it difficult to isolate the effects of the injury (Furger, Nelson, Lerner, & McCrea, 2016; Luoto et al., 2013; Mathias, Harman-Smith, Bowden, Rosenfeld, & Bigler, 2014). To the degree that neurocognitive tests are sensitive to mTBI in ED patients, their use could facilitate its identification and the triaging of affected patients into appropriate education and clinical management opportunities.

The automated nature of CNT administration (alongside other features; see Covassin, Elbin, & Stiller-Ostrowski, 2009; Meehan, d’Hemecourt, Collins, Taylor, & Comstock, 2012) may facilitate their adoption in medical settings over traditional paper-and-pencil neuropsychological tests. However, very few studies have examined the performance of CNTs in patients recruited from EDs. One study of ImPACT found statistically slower reaction time/processing speed (but equivalent verbal and visual memory) in mTBI versus control participants tested in the ED (Peterson, Stull, Collins, & Wang, 2009), whereas another study found a moderate effect of mTBI on immediate memory (as measured by the Rapid Screen of Concussion computerized battery) but nonsignificant effects of six other indices after adjusting for confounding variables (recent alcohol use; Sheedy, Geffen, Donnelly, & Faux, 2006).

Project Head to Head was a prospective study of the reliability, validity, and clinical utility of four CNTs for the assessment of mTBI that enrolled subjects in both sport and civilian (ED) samples from 2012 to 2014. Here, we present results from the ED arm of the study, which collected data on three CNTs (ANAM, DANA, ImPACT). (Corresponding findings from the sports arm of the study have been published previously; Nelson, LaRoche, et al., 2016.) We present the test-retest reliability and validity (group-level sensitivity/discrimination) of these neuropsychological measures in patients recruited from an ED with uncomplicated mTBI (as well as matched trauma controls) who were assessed at several time points post-injury.

Findings for the CNTs are compared to those of a brief postconcussive symptom checklist (Sport Concussion Assessment Tool – 3; SCAT3). Based on the small body of prior work on the performance of these measures to detect mTBI in similar patient populations, we expected to observe significant and moderate-to-large differences between mTBI and control groups in SCAT3 symptoms, with modest but statistically significant differences between groups on some CNT measures. Test-retest reliability evaluated in the control group was expected to be modest, given findings of limited stability of these measures in uninjured athlete populations (e.g., Nelson, LaRoche, et al., 2016), where sample characteristics (as compared to this ED sample) could be expected to maximize measurement stability.

METHOD

Participants

The sample was derived from participants in Project Head to Head, which enrolled a convenience sample of ED patients treated between September 2012 and May 2014 at an academic medical center that serves the emergent needs of the local community as well as being the region’ s only level I trauma center. Participants completed informed consent before their first evaluation and were compensated $210 for their time and effort in completing all assessments. Testing procedures were approved by the Institutional Review Board at the Medical College of Wisconsin.

We monitored each patient being treated in the ED and attempted to approach any patient who appeared to meet inclusion and exclusion criteria specified below (e.g., age 18–45 years old, exposed to a common cause of mTBI; Faul et al., 2010). Possible causes of mTBI included motor vehicle-traffic crashes, falls, assaults, and struck by/against an object events. Figure 1 is a flow chart of each screening and enrollment stage. Of the 181 participants who completed informed consent, 98 met criteria for mTBI and 83 were enrolled as trauma controls. One of these subjects did not complete any assessment procedures after consenting to participate, 2 were withdrawn after the first assessment when it was discovered they met exclusionary criteria (neurologic disorder and positive head CT), and 4 were excluded due to evidence of invalid neurocognitive test results (see the Data Analyses section), yielding a final sample for analysis of 94 mTBI participants and 80 controls.

Fig. 1.

Patient screening and recruitment flow chart. Screening of emergency department (ED) admissions occurred in real-time, and patients who met eligibility criteria were considered for approach. As study procedures were conducted outside the ED, patients who were approached about the study were allowed to either schedule a study appointment or consent to be contacted about the study at a later time.

Inclusion and Exclusion Criteria

Our definition of mTBI was based on that of the study sponsor, the U.S. Department of Defense: “mTBI is defined as an injury to the brain resulting from an external force and/or acceleration/deceleration mechanism from an event such as a blast, fall, direct impact, or motor vehicle accident which causes an alteration in mental status typically resulting in the temporally related onset of symptoms such as headache, nausea, vomiting, dizziness/balance problems, fatigue, insomnia/sleep disturbances, drowsiness, sensitivity to light/noise, blurred vision, difficulty remembering, and/or difficulty concentrating” (Helmick et al., 2006, p. 2).

Inclusion criteria for participation were age (18–45 years, the age range of interest to the study sponsor), loss of consciousness less than 30 min, posttraumatic amnesia less than 24 hr, no acute intracranial findings on brain imaging (if available), proficiency in English, and presenting for the initial assessment within 72 hr of injury. Subjects were excluded if they had an injury that precluded participation in the study protocol (e.g., hand injury that prevented use of a computer mouse), current diagnosis of a psychotic disorder,¹ history or clinical suspicion of other conditions (e.g., epilepsy, stroke, dementia) known to cause cognitive dysfunction, and history of moderate or severe TBI. Trauma controls met the inclusion criteria but sustained orthopedic injuries rather than mTBIs. Controls were also excluded if they had an mTBI within the last 6 months.

Assessment Protocol

The study protocol involved completion of post-injury examinations conducted within 72 hr of injury and at 8 (± 1), 15 (±2), and 45 (±5) days post-injury. Mean (SD) time from injury to follow-up was 39.03 (21.62) hr, 7.94 (1.16) days, 14.63 (1.54) days, and 43.94 (3.93) days for the 72-hr, 8-, 15-, and 45-day time points, respectively. Tests were individually proctored by a research assistant in a quiet setting, nearly always with only one participant being examined at a time. Participants were either first assessed immediately after being medically discharged from the ED or they made an appointment to return another time. The assessment began with a one-on-one interview of contact information, demographics, and health history information followed by a neuropsychological assessment battery. Follow-up assessments began with an interview about subjects’ recoveries followed by the same neuropsychological assessment battery. To reduce the burden on participants to come to our office for assessment, day 8 follow-up appointments were completed via phone and only obtained recovery and SCAT3 symptom ratings (see below).

The testing protocol consisted of, in order, the: Wechsler Test of Adult Reading (WTAR; Wechsler, 2001), CNT #1, Standardized Assessment of Concussion (SAC; McCrea et al., 1998), Sport Concussion Assessment Tool – 3^rd edition (SCAT3) symptom checklist (McCrory et al., 2013), CNT #2, Green’s Medical Symptom Validity Test (MSVT; Green, 2003), Satisfaction With Life Scale (SWLS; Diener, Emmons, Larsen, & Griffin, 1985), Brief Symptom Inventory-18 (BSI-18; Derogatis, 2001), Posttraumatic Stress Disorder Checklist – Civilian Version (PCL-C; Weathers, Litz, Huska, & Kean, 1994) and the Balance Error Scoring System (BESS; Guskiewicz, Ross, & Marshall, 2001). The MSVT was not performed at the day 15 assessment.

Each subject took two of three CNTs: Automatic Neuropsychological Assessment Metrics (ANAM v. 4.3; Vista Life Sciences), Defense Automated Neurobehavioral Assessment [DANA; U.S. Navy Bureau of Medicine and Surgery (BUMED)] and Immediate Post-Concussion Assessment and Cognitive Testing (ImPACT, Online version; ImPACT Applications Inc.). These were selected by the study’s principal investigator and advisors to match the most widely used CNTs in sports medicine and military settings at the time of study design. CNT pairs were filled sequentially during the course of the study with order of administration counterbalanced across participants. Once assigned to an order of administration, each subject completed the same order at all follow-up examinations.

Neuropsychological Tests

ANAM and ImPACT have been described in more detail in publications from the sports arm of this study and are summarized more briefly here (Chin, Nelson, Barr, McCrory, & McCrea, 2016; Nelson, LaRoche, et al., 2016; Nelson, Pfaller, Rein, & McCrea, 2015).

ANAM

The version of ANAM used was comprised of eight subtests: Simple Reaction Time (SRT), Code Substitution-Learning (CDS), Procedural Reaction Time (PRT), Mathematical Processing (MTH), Matching to Sample (M2S), Code Substitution-Delayed (CDD), Simple Reaction Time 2 (SR2), and Go/No-Go (GNG), producing one throughput score for each subtest (except GNG, for which the D-prime score was used) as a well as Composite Score that aggregates the subtests’ throughput scores (Vincent et al., 2012).

DANA

The DANA BRIEF examination is comprised of nine cognitive tasks: Simple Reaction Time (SRT), Code Substitution (CDS), Procedural Reaction Time (PRT), Spatial Processing (SP), Choice Reaction/Go/No-Go (GNG), Code Substitution Recall (CDR), Matching to Sample (M2S), Memory Search (MS), and a second trial of Simple Reaction Time (SR2). Each subtest produces a throughput score. Psychological screening measures included in DANA were not evaluated for the purpose of this study.

ImPACT

ImPACT is comprised of six tasks which yield four neurocognitive composite scores: Verbal Memory (VERM), Visual Memory (VISM), Visual Motor Speed (VMS), and Reaction Time (RT).

SCAT3 symptom checklist

The SCAT3 (McCrory et al., 2013) symptom checklist is a 22-item self-report checklist of post-concussive symptoms (each rated from 0 to 6, with the resulting symptom severity scores ranging from 0 to 132). Higher scores reflect greater symptom burden.

Data Analyses

Sample and measures

Analyses included the aforementioned indices derived from each CNT as well as the SCAT3 symptom severity score. Subjects with evidence of invalid test performance, as determined by failure of one or more indices of the MSVT, were excluded from analyses by time point (at day 15, where the MSVT was not administered, we used the more conservative estimate of any MSVT failure across the other time points). In all, only 4 subjects failed to provide any valid assessment data, resulting in 94 mTBI and 80 control subjects reported on below. Cell sample sizes for the valid data analyzed here are presented in Table 3. Analyses did not exclude subjects based on the performance validity output embedded in ImPACT and ANAM due to: (a) our preference to not bias results in favor of select neurocognitive measures (e.g., due to the fact that DANA does not contain performance validity measures), (b) because some performance validity measures (in particular those of ImPACT) were developed for the purpose of establishing invalid performance in healthy athletes assessed pre-injury (and, therefore, are of unclear validity in this injured trauma sample), and (c) to maximize the sample size available for each analysis.

Table 3.

Descriptive statistics M (SD)

	< 72 Hours		Day 15		Day 45
	mTBI	Control	mTBI	Control	mTBI	Control
	(n = 88)	(n = 73)	(n = 75)	(n = 69)	(n = 68)	(n = 68)
SCAT3 symptoms	32.25 (26.14)	15.63 (14.82)	20.47 (21.86)	11.49 (13.40)	13.06 (17.62)	8.21 (9.06)
ANAM	(n = 40)	(n = 27)	(n = 38)	(n = 27)	(n = 29)	(n = 27)
Composite	–.82 (1.56)	–.55 (1.57)	–.41 (1.25)	.07 (2.01)	–.32 (1.44)	.24 (1.41)
SRT	214.08 (54.35)	200.88 (67.58)	224.93 (46.18)	229.41 (39.25)	223.24 (42.34)	235.55 (43.28)
CDS	46.61 (11.82)	48.52 (13.47)	53.93 (9.74)	53.31 (14.03)	53.84 (11.72)	58.90 (16.52)
PRO	95.06 (21.92)	96.36 (22.98)	97.28 (17.79)	102.81 (23.05)	96.35 (19.41)	101.65 (21.07)
MTH	18.02 (6.73)	22.00 (6.61)	20.42 (7.51)	23.84 (21.53)	22.27 (13.15)	24.10 (8.11)
M2S	27.14 (9.37)	30.00 (12.39)	26.91 (8.74)	28.03 (11.57)	26.59 (9.16)	27.61 (9.29)
CDD	42.57 (21.83)	39.64 (18.31)	43.47 (15.07)	45.81 (16.85)	46.92 (16.90)	47.42 (18.49)
SR2	196.22 (53.24)	209.67 (50.02)	202.53 (58.15)	219.63 (43.61)	197.84 (48.58)	218.68 (48.00)
GNG	3.16 (1.40)	3.58 (1.52)	3.93 (1.40)	4.28 (1.49)	3.70 (1.35)	4.39 (1.82)
DANA	(n = 64)	(n = 53)	(n = 59)	(n = 46)	(n = 49)	(n = 49)
RT	134.62 (32.74)	134.76 (28.20)	141.31 (28.93)	147.64 (26.26)	141.63 (24.98)	142.43 (26.90)
CDS	41.76 (8.66)	40.44 (9.99)	44.98 (9.53)	43.78 (11.17)	45.86 (10.64)	44.92 (10.26)
PRT	90.12 (18.65)	90.89 (16.39)	96.55 (15.77)	93.93 (18.59)	92.92 (15.76)	93.56 (17.68)
SP	28.59 (7.13)	27.56 (7.09)	32.71 (8.82)	30.59 (8.81)	33.10 (8.24)	30.52 (7.41)
GNG	76.96 (32.70)	78.58 (31.91)	87.10 (31.66)	83.20 (36.04)	80.50 (36.68)	86.35 (31.78)
CDD	49.35 (12.84)	47.38 (14.19)	52.90 (15.58)	50.14 (16.23)	52.99 (13.45)	51.29 (16.17)
M2S	26.90 (9.41)	26.14 (9.89)	28.18 (9.39)	25.25 (8.89)	26.04 (8.93)	25.85 (7.39)
MS	55.90 (18.29)	56.64 (19.16)	59.67 (18.02)	61.86 (19.27)	62.69 (18.39)	63.87 (18.33)
SR2	133.88 (33.09)	139.76 (29.2)	140.65 (32.00)	137.05 (39.06)	135.56 (30.54)	141.19 (30.00)
ImPACT	(n = 43)	(n = 47)	(n = 38)	(n = 40)	(n = 31)	(n = 41)
VERM	78.06 (12.45)	78.78 (14.68)	81.88 (12.95)	79.85 (14.06)	80.82 (14.72)	83.37 (12.05)
VISM	64.36 (14.79)	62.40 (14.87)	60.88 (13.10)	65.22 (15.64)	62.76 (14.33)	63.24 (15.56)
VMS	32.53 (6.88)	33.06 (7.49)	34.96 (7.71)	35.90 (8.17)	34.34 (7.41)	36.11 (8.21)
RT	.67 (0.11)	.68 (0.12)	.67 (0.15)	.65 (0.17)	.68 (0.13)	.66 (0.18)

Note. SCAT3 Symptom variable reflects the symptom severity score. SCAT3 symptom severity day 8 M (SD) for mTBI and control group = 25.23 (22.57) and 13.47 (16.15), respectively.

However, because we were interested in the degree to which the CNT performance validity criteria affected our results, major analyses (e.g., test–retest reliability, effect sizes of mTBI vs. control group differences) were re-run for the three CNTs after excluding subjects whose embedded performance validity measures indicated questionably valid data. For DANA, we applied the major validity criteria used by ANAM given the high level of similarity between these tests (i.e., excluding relevant subtests where accuracy < 56%; see Nelson et al., 2015). The findings of these additional analyses (which yielded extremely modest changes in results from those of the full sample) are summarized in the Results section.

Stability

Test–retest reliability for each scale was quantified for the control sample using both Pearson correlations (r) and intraclass correlations (ICC; two-way mixed; absolute agreement). Although the injured status of this control group could be expected to diminish performance stability, this was nevertheless of interest to provide a comparison to prior published data on the stability of these measures in healthy athletes (Chin et al., 2016; Nelson, LaRoche, et al., 2016). Test–retest intervals were selected from varying combinations of 72-hr, day 15, and day 45 time points to achieve a range of intervals. This yielded mean intervals of 13 (72-hr vs. day 15), 29 (day 15 vs. day 45), and 42 (72-hr vs. day 45) days. To extract an overall stability metric across indices within each CNT, correlation coefficients were converted to z scores using the Fisher transformation, averaged together, and back-transformed into the mean correlation coefficients presented in Table 2.

Table 2.

Test-retest reliability of SCAT3 and CNT measures in trauma controls using Pearson (and intraclass) correlations

	Test–retest interval
	13 Days	29 Days	42 Days
SCAT3 symptoms	.78 (.74)	.60 (.54)	.55 (.42)
ANAM
Composite	.39 (.35)	.75 (.72)	.58 (.51)
SRT	.13 (.10)	.51 (.51)	.20 (.15)
CDS	.38 (.35)	.84 (.79)	.32 (.24)
PRT	.50 (.49)	.26 (.27)	.61 (.60)
MTH	.74 (.41)	.72 (.48)	.85 (.79)
M2S	.67 (.67)	.63 (.62)	.76 (.73)
CSD	.72 (.66)	.80 (.80)	.75 (.68)
SR2	.41 (.40)	.30 (.30)	−.05 (−.05)
GNG	.46 (.41)	.41 (.41)	.30 (.27)
Mean	.51 (.62)	.62 (.57)	.54 (.48)
DANA
SRT	.68 (.61)	.81 (.80)	.56 (.55)
CDS	.80 (.75)	.79 (.79)	.67 (.62)
PRT	.75 (.73)	.78 (.78)	.59 (.59)
SP	.65 (.60)	.69 (.69)	.57 (.53)
GNG	.77 (.76)	.74 (.73)	.53 (.52)
CDR	.55 (.54)	.74 (.75)	.43 (.42)
M2S	.61 (.61)	.64 (.62)	.51 (.49)
MS	.31 (.31)	.39 (.39)	.46 (.43)
SR2	.68 (.66)	.62 (.61)	.61 (.61)
Mean	.66 (.63)	.70 (.70)	.55 (.53)
ImPACT
VERM	.57 (.58)	.60 (.59)	.50 (.47)
VISM	.66 (.66)	.52 (.52)	.52 (.53)
VMS	.79 (.75)	.83 (.84)	.84 (.79)
RT	.58 (.55)	.53 (.53)	.45 (.41)
Mean	.62 (.60)	.62 (.62)	.57 (.55)

Note. 13-day interval = 72-hour to day 15 assessment; 29-day interval = day 15 to day 45 assessment; 42-day interval = 72-hour to day 45 assessment; SCAT3 = Sport Concussion Assessment Tool—3; SRT = Simple reaction time; CDS = code substitution-learning; PRT = procedural reaction time; MTH = mathematical processing; M2S = matching to sample; CSD = code substitution-delayed; SR2 = simple reaction time 2; GNG = go no-go; SP = spatial processing; CDR = code substitution-recall; MS = memory search; SR2 = simple reaction time trial 2; VERM = verbal memory composite; VISM = visual memory composite; VMS = visual motor speed composite; RT = reaction time composite.

Sensitivity

Group (mTBI, control) × Time (72-hr, day 15, day 45) repeated-measures analyses of variance were computed for each CNT subscale. Greenhouse-Geisser correction was used where needed for violations of the sphericity assumption. Follow-up t tests were conducted where appropriate to examine between-group differences at each time point. Adjustment for multiple comparisons was performed using the false discovery rate method (Benjamini & Hochberg, 1995). Cohen’s d effect sizes of mTBI versus control group differences were computed across measures and time points. Next, receiver operating characteristic (ROC) curves were produced for each index to provide another clinically relevant metric regarding the value of each measure for individual decision making about group status.

RESULTS

Sample Characteristics

The demographic makeup and acute injury characteristics of the sample are presented in Table 1. The groups were closely matched on variables including gender, age, race, socioeconomic status, estimated verbal intellectual ability (WTAR score), mTBI history, and litigation related to injury. The mTBI group had somewhat higher representation of subjects with a self-reported history of psychiatric disorder (n = 8 vs. 1 in the control group; p = .031). Of these, four reported bipolar depression (two of whom had a comorbid anxiety disorder), two reported only unipolar depression, two reported an unspecified mood disorder, and three reported an anxiety disorder. Eight of these subjects were taking a psychiatric medication at the time of enrollment. Exclusion of subjects with psychiatric histories from analyses, however, had negligible effects on the main results of this study (e.g., mTBI vs. control group effect sizes reported Table 4 declined at a mean of d = .01).

Table 1.

Sample characteristics

	mTBI	Trauma control
	N= 94	N= 80
	M (SD) or %	M (SD) or %	p-Value
Gender (male)	60.6%	48.8%	.116
Age (years)	29.11 (7.58)	30.65 (7.83)	.189
Race			.230
Black	51.1%	56.3%
White	40.4%	41.3%
Other/unknown	8.5%	2.5%
ADHD	12.9%	6.3%	.143
Learning disability	4.3%	6.3%	.554
Psychiatric diagnosis	8.5%	1.3%	.031
WTAR standard score	93.88 (17.09)	95.28 (16.95)	.592
Household SES	41.24 (12.01)	42.99 (10.53)	.361
Health insurance type			.841
Commercial	46.8%	47.4%
Government	31.9%	32.1%
None	20.2%	20.5%
Other/unknown	1.1%	0.0%
Number of prior mTBIs			.762
0	64.9%	70.9%
1	28.7%	25.3%
2	3.2%	1.3%
3 +	3.2%	2.5%
Mechanism of injury			.039
Motor vehicle-traffic	59.6%	66.3%
Fall	25.5%	31.3%
Assault	4.3%	0.0%
Struck by/against	10.6%	2.5%
Litigation related to injury	27.3%	27.3%	.999
Worker’s compensation injury	15.9%	9.0%	.180

Note. ADHD = attention deficit-hyperactivity disorder; WTAR = Wechsler Test of Adult Reading standard score; SES = Hollingshead socioeconomic status. Missingness was n = 1 on number of prior mTBIs and ADHD, 9 on litigation status, and 8 on worker’s compensation status.

Table 4.

mTBI versus trauma control group effect sizes (Cohen’s d)

	<72 Hours	Day 15	Day 45
SCAT3 symptomsa	–.78	–.49	–.35
ANAM
Composite	–.17	–.29	–.40
SRT	.22	–.10	–.29
CDS	–.15	.05	–.35
PRO	–.06	–.27	–.26
MTH	-.60	–.21	–.17
M2S	-.26	–.11	–.11
CDD	.15	–.15	–.03
SR2	–.26	–.33	–.43
GNG	–.28	–.24	–.43
Mean	–.16	–.18	–.27
DANA
SRT	.00	–.23	–.03
CDS	.14	.11	.09
PRT	–.04	.15	–.04
SP	.15	.24	.33
GNG	–.05	.11	–.17
CDD	.15	.17	.11
M2S	.08	.32	.02
MS	–.04	–.12	–.06
SR2	–.19	.10	–.19
Mean	.02	.09	.01
ImPACT
VERM	–.05	.15	.19
VISM	.13	–.30	–.03
VMS	–.07	–.12	–.23
RT	.08	–.11	–.11
Mean	.00	.00	–.05

^aSCAT3 Day 8 d = –.60. Comparisons are all scaled such that negative values reflect worse performance in the mTBI group. Values are bolded where p< .05 after adjustment for multiple comparisons. SCAT3 = Sport Concussion Assessment Tool—3; SRT = Simple reaction time; CDS = code substitution-learning; PRT = procedural reaction time; MTH = mathematical processing; M2S = matching to sample; CSD = code substitution-delayed; SR2 = simple reaction time 2; GNG = go no-go; SP = spatial processing; CDR = code substitution-recall; MS = memory search; SR2 = simple reaction time trial 2; VERM = verbal memory composite; VISM = visual memory composite; VMS = visual motor speed composite; RT = reaction time composite.

Among the injured sample, 36.8% endorsed loss of consciousness, 15.8% posttraumatic amnesia, and 6.3% retrograde amnesia. We could not confirm loss of consciousness with witnesses and the report of loss of consciousness could simply represent posttraumatic amnesia. The overall distribution of CNT pairings across the sample evaluated in this manuscript was: 38.2% ANAM-DANA, 12.9% ANAM-ImPACT, and 48.9% DANA-ImPACT.

The percentage of the sample that completed day 15 and 45 follow-up assessments was 89.1% and 83.9%, respectively. Those who did versus did not complete the day 15 assessment were less likely to have attention deficit-hyperactivity disorder (ADHD) (7.8% vs. 26.34%; χ²[1] = 6.55; p= .010). Similarly, those who did versus did not complete the day 45 assessment were less likely to have ADHD (5.5% vs. 32.1%; χ²[1] = 18.78; p < .001). Otherwise, all other demographic, history, injury, and 72-hr assessment measures were equivalent between groups (p > .05).

Trauma control subjects self-reported injuries to a variety of bodily regions: back (61.3%), neck (50.0%), upper extremity (45.0%), lower extremity (45.0%), trunk (18.8%), and head (16.3%). The 13 controls coded positive for head trauma reported lacerations, bruises, or other injuries to the head but did not otherwise meet criteria for mTBI (i.e., did not endorse altered mental status or mTBI symptoms). Excluding these subjects from analyses did not impact the primary results (e.g., M change in Cohen’s d values reported in Table 4 was d= .01).

Finally, although litigation status was unrelated to injury group and, therefore, had no effect on the major study analyses, given the relatively high prevalence of the sample’s reported involvement in litigation and the relevance of this variable to the neuropsychology audience, data pertaining to the magnitude of litigation-related effects on the neurocognitive performance and mTBI symptoms are presented in the Supplementary Materials. To summarize the results briefly, subjects’ who endorsed any involvement in litigation related to their injuries tended to report more mTBI symptoms and perform more poorly on neurocognitive measures, with effects small-to-medium in magnitude. Given the relatively crude manner in which this variable was assessed,² however, these data should be interpreted cautiously.

Test–Retest Reliability of Symptom and Neurocognitive Indices

Table 2 displays the test-retest reliability of the symptom and neuropsychological measures obtained at varying intervals using data from the trauma control group. The SCAT3 symptom severity score had reliability coefficients over .70 for the briefer test-retest interval (Pearson vs. ICC = .78 vs. .74), with somewhat lower magnitudes for the longer test–retest intervals (e.g., Pearson r for 29-day and 42-day intervals = .60 and .55, respectively). For the CNTs, 44–63% of Pearson reliability coefficients were over .6 across all test-retest intervals (12 of 27 for ANAM, 17 of 27 for DANA, and 5 of 12 for ImPACT, respectively), and 25–33% of coefficients were over .7 (9 of 27, 8 of 27, and 3 of 12 across all test–retest intervals for ANAM, DANA, and ImPACT, respectively). Similar results were found using ICCs (33–63% of coefficients over .6 and 19–30% over .7 across CNTs and test–retest intervals). Excluding subjects who failed embedded effort indices for each CNT had minimal effect on test–retest reliability coefficients (e.g., across all time points, Pearson stability coefficients improved by a mean of .06, .01, and .01 for ANAM, DANA, and ImPACT, respectively).

Between-Group Differences and Effect Sizes on Symptom and Neurocognitive Measures

Table 3 provides descriptive statistics for the mTBI and control groups across measures and time points, while Table 4 provides the between-group differences expressed as Cohen’s d. The Supplementary Materials (Tables S1–S2) provides additional statistical output to support the information provided in Table 3. SCAT3 symptom severity score was the only variable that demonstrated statistically significant between-group differences at any of the acute and subacute time points (72-hr, day 8, and day 15 d = .78, .60, and .49, respectively, FDR-adjusted ps < .05; all in the expected direction of higher symptoms for mTBI vs. control subjects). The symptom score was not significantly different between the groups after correction for multiple comparisons at day 45 (d = .35). No CNT measure demonstrated statistically significant group differences at any of the post-injury time points.

Before adjustment for multiple comparisons, only one main effect of Group (of 22 CNT variables) was significant (ANAM simple reaction time 2 Group p = .022) and no Group × Time interactions were significant. In terms of magnitude or clinical meaningfulness of group differences, the neurocognitive measures demonstrated quite small effect sizes even at 72 hr (absolute value of 72-hr ANAM ds = .06 to .60 with all but one under .3; DANA ds = .00 to .19; ImPACT ds = .05 to .13; across these measures 15 were in the expected direction and 10 were not). Excluding subjects who failed embedded effort indices for each CNT had minimal effect on group differences (e.g., across all time points, d values became stronger in the expected direction by a mean of .05, .05, and .01 for ANAM, DANA, and ImPACT, respectively).

ROC Curves

Table 5 displays the area under the ROC curve (AUC) as another metric of the ability of these clinical measures to discriminate between the mTBI and orthopedically-injured control groups. The best performing measure (SCAT-3 symptom severity score) provided only fair discrimination between groups (AUC = .72 at 72-hr), whereas discrimination was poor for neurocognitive measures (M AUC for ANAM, DANA, and ImPACT at 72-hr = .56, 49, and .50, respectively).

Table 5.

Area under the ROC curve

	<72 Hours	Day 15	Day 45
SCAT3 symptomsa	.72	.63	.54
ANAM
Composite	.56	.58	.60
SRT	.48	.52	.56
CDS	.57	.47	.56
PRT	.56	.59	.60
MTH	.68	.52	.61
M2S	.56	.52	.55
CSD	.46	.53	.51
SR2	.59	.58	.62
GNG	.59	.58	.56
Mean	.56	.54	.57
DANA
SRT	.49	.55	.48
CDS	.46	.48	.46
PRO	.50	.47	.51
SP	.45	.42	.41
GNG	.51	.48	.52
CDD	.45	.46	.46
M2S	.48	.42	.48
MS	.53	.54	.49
SR2	.55	.51	.52
Mean	.49	.47	.48
ImPACT
VERM	.53	.46	.56
VISM	.48	.58	.53
VMS	.51	.52	.57
RT	.48	.55	.59
Mean	.50	.53	.56

^aSCAT3 Day 8 AUC = .68. Values are bolded where p< .05 after adjustment for multiple comparisons. SCAT3 = Sport Concussion Assessment Tool-3; SRT = Simple reaction time; CDS = code substitution-learning; PRT = procedural reaction time; MTH = mathematical processing; M2S = matching to sample; CSD = code substitution-delayed; SR2 = simple reaction time 2; GNG = go no-go; SP = spatial processing; CDR = code substitution-recall; MS = memory search; VERM = verbal memory composite; VISM = visual memory composite; VMS = visual motor speed composite; RT = reaction time composite.

DISCUSSION

In this prospective study of the utility of three computerized neurocognitive test batteries (ANAM, DANA, ImPACT) for the assessment of mTBI in patients recruited from an ED, none provided more than minimal discrimination between mTBI and trauma control groups within 72 hr of injury (M = 39 hr) or at later time points (days 15 and 45 post-injury). For example, the mean Cohen’s d (scaled with negative values in the direction of more impairment for mTBI vs. control groups) at 72 hr for ANAM, DANA, and ImPACT were –.16, .02, and .00, respectively. These findings contrast those previously published from the sports arm of this study, which found stronger (generally medium-to-large) between-group differences in performance on ANAM, Axon, and ImPACT within 24 hr (M = 19 hr) of injury in a sample of high school and collegiate athletes (Nelson, LaRoche, et al., 2016).

That effect sizes were much smaller for ED versus sport samples likely reflects one major difference between the two arms of the study—the use of injured controls in the ED sample. As nonspecific injury- and treatment-related factors present in trauma populations broadly can contribute to similar neuropsychological sequelae as mTBI (e.g., Babikian et al., 2011; Bruera, Macmillan, Hanson, & MacDonald, 1989; Macciocchi, Seel, & Thompson, 2013; Rabinowitz et al., 2015), the limited discrimination observed in neurocognitive measures between ED-recruited mTBI and trauma control groups is likely due to the nonspecific neurocognitive sequelae of recent traumatic injury (and other co-occurring factors common in civilian trauma populations that can contribute to neuropsychological impairment; e.g., Iverson, 2005). Of course, comparable data on an orthopedically-injured athlete sample would be needed to confirm this conjecture.

To put the overall level of performance of the two study samples into context, both the mTBI and trauma control participants in this ED-recruited sample scored, on average, 0.3 SDs below the acutely concussed athletes and 1.5 SDs below the healthy control athletes of the sports sample (using ImPACT metrics at the most acute post-injury time point for illustrative purposes; Nelson, LaRoche, et al., 2016). Thus, irrespective of injury type, trauma patients perform cognitively similar to (or slightly worse than) adolescent and young adult athletes with sport-related mTBI and significantly worse than healthy athletes.

Whereas CNT performance was equivalent between mTBI and trauma control groups, SCAT3 symptom severity differentiated better between groups (i.e., was higher in mTBI vs. control subjects; 72-hr d = –.78), with this effect diminished but still more robust than neurocognitive measures during the follow-up period (day 8, 15, and 45 d = –.60, –.49, and –.35, respectively). This is consistent with findings from the sports arm of the study of stronger sensitivity of self-reported symptoms than CNT performance to mTBI (24-hr symptom d = –1.53 vs. mean neurocognitive Cohen’s ds in the .5 to .7 range; Nelson, LaRoche, et al., 2016). Taken together, these findings underscore the relevance of measuring mTBI symptoms in the clinical assessment of mTBI across patient populations and injury settings.

That said, mTBI symptoms are also not specific to mTBI, a fact supported by prior studies (e.g., Landre, Poppe, Davis, Schmaus, & Hobbs, 2006; Smith-Seemiller, Fow, Kant, & Franzen, 2003) and these data [i.e., differences between mTBI and control groups in Project Head to Head were somewhat smaller in the ED (d = .76) vs. the sport samples (d = 1.53)]. Similar to neurocognitive measures, “mTBI” symptom ratings in our sample probably reflect both acute effects of injury in general as well as the high prevalence of mTBI-like symptoms in community samples (Iverson & Lange, 2003). Although the slightly longer time interval between injury and first assessment in the ED sample (<72 hr vs. <24 hr; M time from injury to first assessment = 39 vs. 19 hr) could also contribute to weaker effects sizes on clinical measures (see McCrea et al., 2003), the relatively slow recovery trajectory of this ED sample³ suggests that this procedural difference probably does not contribute substantially to the observed differences in findings across samples.

Finally, test–retest reliability of the neurocognitive measures in the trauma controls of this ED sample were similar to those reported on the healthy control athletes of the sport sample (Nelson, LaRoche, et al., 2016), with roughly half over .60 and a quarter over .70. That measurement stability was similar across samples suggests that the reduced sensitivity of neurocognitive measures to mTBI in the ED versus sport sample is not due to lower measurement stability in this population. Although most neurocognitive indices did not meet common standards for use in clinical practice, the similar stability of CNT measures across samples is nevertheless of interest given that the acutely injured nature of this sample was expected to diminish performance stability in comparison to what has been observed in healthy controls.

This investigation had several limitations. First, subjects were evaluated in a laboratory setting within 72 hr of injury; thus, it is possible that stronger group differences in clinical assessment measures would have been found had subjects been assessed more acutely (such as within the ED). Second, the study design (i.e., assignment of two of three CNTs to each subject) and presence of loss to follow-up (16% at 45 days post-injury) contributed to smaller sample sizes (<50) for some CNT measures and at some time points. Nevertheless, the findings are considered valuable since quantifying effect sizes was of greater interest than achieving statistical significance, because the acute (72-hr) post-injury time point (in which all subjects provided data) was of greatest interest in terms of hypothesized cognitive effects, and because there did not appear to be significant bias in the subset of the sample that completed day 15 and 45 follow-up appointments (i.e., completion of follow-up appointments was unrelated to major demographic and injury-related factors). Third, some participants were examined in the same room as other subjects, which could have affected their performance. In particular, group settings tend to negatively impact cognitive performance, although the magnitude of this effect depends on several factors such as group size and the particular cognitive domain being assessed (Eastvold, Belanger, & Vanderploeg, 2012; Lichtenstein, Moser, & Schatz, 2014). While group testing occurred relatively infrequently, in the future testing setting should be better controlled or documented so that the effect of environmental factors can be minimized or formally evaluated.

In summary, we found that among three CNTs (ANAM, DANA, ImPACT) administered to community patients recruited from an ED, none reliably differentiated between mTBI and trauma control groups within 72 hr of injury or later post-injury time points. While SCAT3 symptom scores were significantly elevated in mTBI versus control groups, effect sizes were modest in comparison to what has been reported in the sport-related mTBI literature, where concussed athletes have typically been compared to healthy controls. We interpret these attenuated findings as a consequence of the nonspecific factors common in trauma patients that mimic the clinical effects of mTBI. The implication of these findings is that these assessment tools, which are commonly used for assessment of sport-related mTBI, appear less valuable for the assessment of mTBI sequelae in this more complex and heterogeneous patient-based ED sample.

Alternatively, it is possible that more acute neuropsychological assessment or aggregation of neuropsychological measures with other emerging markers of mTBI (e.g., blood biomarkers, neuroimaging) would contribute to improved performance in the context of acute identification of the effects of mTBI specifically. Furthermore, these findings do not rule out the possibility that acute neuropsychological data could be informative for prognostic purposes in mTBI or other patient populations (Dischinger, Ryb, Kufera, & Auman, 2009; McCrea et al., 2013; Meehan, Mannix, Monuteaux, Stein, & Bachur, 2014; Nelson, Tarima, et al., 2016). At the present time, however, the findings suggest that clinicians can continue to apply standard clinical criteria to assess patients for mTBI; neurocognitive assessment may provide supporting data on patients’ presenting concerns but is not useful for diagnostic purposes.