A Computerized Test Battery Sensitive to Mild and Severe Brain Injury

Abstract

Objective

Computerized neurocognitive testing (CNT) appears to be suited to measure relatively mild degrees of neurocognitive impairment in circumstances where speed, efficiency, and low cost are important. Computerized tests are used in the evaluation and management of patients who have had mild brain injuries; the objective is to determine if computerized testing is equally reliable and valid in the evaluation of patients who have had more severe brain injuries.

Design

A cross-sectional, naturalistic study of brain injury patients compared with normal controls.

Setting

An outpatient neuropsychiatry clinic.

Participants

141 patients, aged 18–65 years, who had sustained traumatic brain injuries (TBIs): 13 patients with postconcussion syndrome; 15 who had recovered from mild brain injuries; 85 patients who had had severe brain injuries, but who had recovered, and were living independently; and 28 severe brain injury patients who were unable to live without assistance; compared with 145 normal controls.

Interventions

Not applicable.

Main Outcome Measures

The CNS Vital Signs (CNS VS) battery is a PC-based system that includes tests of verbal and visual memory, psychomotor speed, complex attention, reaction time, and cognitive flexibility.

Results

Performance on the CNS VS battery was related to severity of brain injury and degree of recovery. Tests of psychomotor speed and cognitive flexibility were the most relevant to TBI status. Patients who had recovered from mild brain injuries scored almost as well as normal controls. The Neurocognition Index (NCI), a summary score based on performance on all the tests in the battery, was 100 for normal controls and 98 for recovered mild brain injury patients. Postconcussive patients scored 82 on the MCI, and severe brain injury patients scored 66 on the NCI if they were living independently and 47 if they were not.

Conclusions

Computerized tests like CNS VS allow clinicians the advantage of precise neurocognitive measurement in the service of diagnosis and appropriate treatment. CNTs are never going to replace the flexibility or comprehensiveness of conventional neuropsychological testing, but they have a role to play in circumstances where a full test battery is not feasible, such as screening and serial assessment.

Introduction

Neuropsychological testing is the “gold standard” for evaluation of patients who have had brain injuries.[1–5] But there are hardly enough qualified neuropsychologists to keep up with the 2 million or so Americans who have brain injuries every year,[6] and even if there were, the cost of repeated evaluations as the patient recovers (or fails to recover, as the case may be) would be prohibitive. Neuropsychologists have tried to address this problem by introducing brief screening batteries,[7,8] rating scales,[9] or CNT batteries.[10,11]

Theoretically, at least, CNTs can increase productivity, efficiency, and knowledge. But like every technology, computerized testing has limitations. Many computerized batteries were, and still are, relatively stunted, in terms of their psychometric development. An unsupervised subject sitting in front of a monitor may not be an optimal testing environment. CNTs are capable of generating a mass of data that are unnervingly precise, but of dubious salience, especially in the hands of untrained clinicians.

On the other hand, CNTs have a few advantages compared with conventional psychological testing. These include consistency in administration and scoring, the ability to generate numerous alternative forms suitable for repeated testing, precise stimulus control, the ability to track various components of subjects' responses, increased cost efficiency in testing, and the ability to develop large and accurate databases.[12] Published reports tend to emphasize the feasibility of the technology, its acceptability to patients, and the reliability of the data thus generated.[13]

CNT technology is well suited to a new and developing arena for mental testing: measuring relatively mild degrees of neurocognitive impairment in circumstances where speed, efficiency, and low cost are important. An example is sports-related concussion management.[14–17] CNTs designed for concussion management tend to be very short (10–20 minutes) and measure specific neurocognitive functions that are relevant to the concussed state, like attention, reaction time, and working memory. Longer CNTs take about 30–60 minutes but address a wider range of neurocognitive functions. They are more appropriate for clinical settings and more relevant to the problems of patients who have sustained more extensive or severe brain injuries.

The authors developed CNS VS as a computerized battery that is reliable[18] and sensitive to cognitive impairment caused by different conditions. The battery has been used in studies of children with ADHD,[18] adults with depression,[19] and elderly people with mild cognitive impairment and dementia.[19] The question posed in this paper is how well CNS VS can evaluate the cognitive impairment of patients with TBI at different levels of severity. As a rule, CNTs used for concussion management have not been tested in patients with more severe brain injuries; conversely, most broad-spectrum CNTs have not been tested in patients with concussion.

Methods

Subjects and Procedure

The subjects were 141 patients who had sustained TBIs and who were evaluated at the North Carolina Neuropsychiatry Clinics in Chapel Hill and Charlotte during a 2-year period from March 2003 through April 2005. All new patients at the clinic are evaluated with a battery of tests, including CNS VS, and their CNT data are stored in a central database. Patients were selected if they met the following criteria:

1. Aged 18–65 years;

2. Closed head injury (not anoxic, electrical, toxic, or infectious brain injuries) uncomplicated by stroke, seizures, or severe psychiatric comorbidity (eg, psychosis);

3. No significant premorbid cognitive or psychiatric conditions (eg, mental retardation, alcoholism, substance abuse); and

4. No current treatment with sedating or cognitive-enhancing medications (eg, benzodiazepines, antispasticity agents, antipsychotics, psychostimulants, or cholinesterase inhibitors).

A total of 318 TBI subjects were identified by the database, of whom 141 met the criteria listed above. The database divided the patients into 4 groups, which we confirmed by chart review:

1. Patients with postconcussion syndrome (PCS);

2. Mild brain injury (MBI);

3. Severe traumatic brain injury (TBI-1); and

4. Severe, disabling traumatic brain injury (TBI-2).

The patients were classified on the basis of accepted criteria.[20] PCS and MBI patients had sustained injuries associated with loss of consciousness (LOC) (not more than 20 minutes) and/or posttraumatic amnesia (not more than 24 hours); no focal findings on neurologic examination; and normal imaging with computed tomography or magnetic resonance imaging. PCS patients were seen within 3 months of the injury and complained of typical postconcussive symptoms: headache, dizziness, fatigue, trouble sleeping, irritability, emotional lability, difficulty concentrating, memory impairment, mental slowing, and alcohol intolerance. MBI patients were at least 12 months postinjury and no longer complained of acute postconcussive symptoms. Some were symptomatic, some were not. Persistent symptoms included depression, headache, inattention, and poor memory.

Both groups of severe TBI patients had sustained severe injuries with Glasgow Coma Scale scores of 8 or below and more than 24 hours LOC. Patients in the TBI-1 group had made good recoveries and were able to live independently, to handle their own finances, and to drive. The TBI-2 patients had not been so successful. They were not able to live independently or to handle their own finances or to drive. (The database also generated a small number of patients who had sustained “moderate” TBI, but too few to include in this analysis.)

The sources of injury were motor vehicle accidents (52%), falls (13%), and blows or assaults (12%), with miscellaneous industrial and sporting accidents accounting for the rest. The different causes of injury were equally distributed among the 4 groups.

The database generated 145 normal controls to match the TBI patients in terms of age, race, and gender. (The match was close, but not perfect; there were more males in the severe TBI groups.) Normal controls are free of current or past neurologic or psychiatric disorders, in good health, and free of any regular medications. (Table 1)

Table 1.

141 TBI Patients and 145 Normal Controls

	NML	PCS	MBI	TBI-1	TBI-2	P <
N	145	13	15	85	28
Age	41.97	47.38	47.60	42.14	38.00	.063
LOC Duration		6.17 min	7.92 min	16.8 days	26.25 days
Education		14.42	14.62	13.33	11.43	.058
White	120	10	13	73	24
Nonwhite	25	3	2	12	4	.9
Female	47	8	6	25	3
Male	98	5	9	60	25	.02

NML = normal controls; PCS = patients with postconcussion syndrome; MBI = mild brain injury; TBI-1 = severe traumatic brain injury; TBI-2 = severe, disabling traumatic brain injury; LOC = loss of consciousness

The CNS Vital Signs Battery

The CNS VS battery contains 7 tests. The battery is composed of tests that are widely used by neuropsychologists and known to be reliable and valid. We chose tests that are known to be sensitive to most of the causes of mild cognitive dysfunction, and we tried to cover an appropriate span of cognitive domains.

The Verbal Memory Test (VBM) and Visual Memory Test (VIM) are adaptations of the Rey Auditory Verbal Learning Test and the Rey Visual Design Learning Test.[21,22] VBM and VIM are recognition tests, however, not tests of recall. Correct responses from VBM and VIM are summed to generate a composite memory or memory domain score.

The Finger Tapping Test (FTT) is one of the core tests of the Halstead-Reitan Battery. Symbol Digit Coding (SDC) is based on the symbol digit modalities test,[23] itself a variant of the Wechsler digit symbol substitution test. The total of right and left taps from the FTT and total correct responses on the SDC generates a composite score for “psychomotor speed.”

The Stroop Test (ST) in CNS VS has 3 parts that generate simple and complex reaction times. Averaging the 2 complex reaction time scores from the ST generates a domain score for “reaction time.” It might be more precise to refer to this domain as “information processing speed.”

The Shifting Attention Test (SAT) measures the subject's ability to shift from one instruction set to another quickly and accurately. Other computerized batteries, like the NES2, CogState, and CANTAB, have shifting attention tests. Color-shape tests like the SAT have been used in cognitive imaging studies.[24,25] A domain score for cognitive flexibility is generated by taking the number of correct responses on the SAT and subtracting the number of errors on the SAT and the ST.

The Continuous Performance Test (CPT) is a measure of vigilance or sustained attention.[26] A domain score for “complex attention” is generated by adding the number of errors committed in the CPT, the SAT, and the ST.

Because the presentation of stimuli is randomized, no 2 presentations of CNS VS are ever the same; thus, the test battery is appropriate for serial administration. Several of the tests draw stimuli from a “reservoir” of words or figures (VBM, VIM, SDC). Several tests record reaction times with millisecond accuracy (VBM, VIM, FTT, ST, SAT, CPT).

A medical office assistant can initiate the test, and a child with a fourth grade reading level can take the test battery unassisted. The visual arrays are simple and easy to read, even to someone who is color-blind. It does not use a mouse, a joystick, or a touchscreen, because those devices introduce an unacceptable level of instability to its millisecond accuracy. A minimum number of keys are in play, so keyboard skills have minimal influence on performance. The test is administered on an ordinary, Windows-based PC and takes about 30 minutes. A report is generated by the machine as soon as the test is completed.

A more complete description of the tests and scoring is presented in the Appendix.

Analysis

The tests in the CNS VS battery generate the following scores:

1. The 7 tests generate 17 primary scores (correct responses, taps, errors of omission and commission, reaction times).

2. The 7 tests generate 5 domain scores (memory, psychomotor speed, reaction time, complex attention, and cognitive flexibility). Domain scores are generated as raw scores and then computed as standard scores for age. Standard scores have a mean of 100 and a standard deviation of 15.

3. The Neurocognition Index (NCI) is the mean of the 5 domain scores, just as an IQ score is calculated from the mean of the subtest scores.

4. For correct responses and taps, higher scores mean better performance. For errors and reaction times, lower scores are better. Domain scores, presented as raw scores, reflect this pattern. For standard scores, higher is always better. A standard score of 100 represents the 50th percentile.

5. The shifting attention test is a speed-accuracy tradeoff test, so a faster reaction time is only better if the number of errors is lower and the number of correct responses is higher.

The SAT efficiency (SATq) score captures both elements, speed and accuracy, by this formula:

$$\text{SATq}=\left(\text{SAT reaction time/percent correct}\right)/1000)$$

A lower SATq represents faster, more accurate performance on the test.

The domain scores and primary scores of the 5 groups were compared by multiple analysis of variance (MANOVA), controlling for age, gender, race, and years of education. MANOVA indicated whether or not the 5 groups differed significantly. If the MANOVA showed that differences existed, paired t tests were done to determine which groups actually contributed to the difference. Discriminant function analysis was done to determine which domains were most sensitive to the effects of brain injury. Receiver operating characteristics (ROC) curves were analyzed to determine the specificity and sensitivity of the test battery.

Results

MANOVA indicated significant differences among the 5 groups on all 5 domain scores, whether represented as raw scores or standard scores, and in the NCI. Seven of 17 primary scores were significantly different (Table 2).

Table 2.

MANOVA of CNS VS Scores in Normals and Brain Injury Patients

TEST/DOMAIN	NML	PCS	MBI	TBI-1	TBI-2	F	P <
NEUROCOGNITION INDEX ss	99.80	81.55	97.65	66.34	46.77	5.38	.0002
MEMORY ss	99.69	89.90	96.31	76.71	51.92	2.51	.0292
PSYCHOMOTOR SPEED ss	101.06	85.66	100.46	71.10	44.48	4.33	.0011
REACTION TIME ss	98.59	76.64	96.58	64.07	64.98	3.38	.0059
COGNITIVE FLEXIBILITY ss	100.98	81.56	99.31	57.41	37.20	4.83	.0005
COMPLEX ATTENTION ss	100.41	83.05	102.70	57.03	40.21	3.92	.0023
MEMORY	98.43	90.70	96.60	86.69	74.52	2.47	.0318
PSYCHOMOTOR SPEED	172.08	139.07	167.07	124.78	83.87	3.92	.0023
REACTION TIME*	647.49	816.15	667.13	883.01	859.06	2.49	.0306
COGNITIVE FLEXIBILITY	44.53	21.38	42.13	10.60	−9.87	4.48	.0009
COMPLEX ATTENTION*	7.60	17.77	7.07	29.52	48.87	2.74	.0192
Verbal Memory, total correct	52.16	50.40	50.73	45.00	38.25	3.19	.0047
Visual Memory, total correct	46.28	43.75	45.87	41.84	36.42	1.27	.2780
Finger Tapping, right	59.15	48.92	58.21	44.96	32.42	2.71	.0131
Finger Tapping, left	57.80	47.69	55.14	43.98	31.88	2.19	.0412
Finger Tapping, total	116.95	96.62	113.36	88.99	64.29	2.69	.0138
Symbol Digit Coding, correct	54.97	43.31	51.60	35.65	21.40	3.73	.0015
Stroop, Simple Reaction Time*	285.48	404.38	309.73	471.62	427.78	1.64	.1349
Stroop, Complex Reaction Time*	598.29	729.15	623.27	823.39	847.04	1.98	.0645
Stroop Color-Word Reaction Time*	696.68	903.15	711.00	942.63	871.08	1.58	.1503
Stroop Test, errors*	1.37	3.54	2.80	3.37	5.33	0.30	.9712
Shifting Attention, correct	51.68	37.08	50.60	33.57	24.96	3.46	.0026
Shifting Attention, errors*	5.79	12.15	5.67	19.16	29.26	2.06	.0547
Shifting Attention, Reaction Time*	1055.40	1221.02	1077.87	1169.85	1062.99	0.58	.8051
Shifting Attention, Efficiency*	1.19	1.91	1.21	2.09	2.93	1.37	.2306
Continuous Performance, correct	39.85	39.15	40.00	38.09	34.85	2.68	.0140
Continuous Performance, errors*	0.68	2.08	0.50	7.01	14.08	1.84	.0876
Continuous Performance, Reaction Time*	403.09	514.69	433.14	507.98	591.04	0.69	.7104

NML = normal controls; PCS = patients with postconcussion syndrome; MBI = mild brain injury; TBI-1 = severe traumatic brain injury; TBI-2 = severe, disabling traumatic brain injury; F = the F statistic; ss = subjects

^*Lower is better

These differences are expressed graphically in Figures 1–6. Overall, the MBI patients performed as well as normal controls in the Neurocognitive Index, a summary score generated by averaging the 5 domain scores, and in each of the 5 domains. The PCS, TBI-1 and TBI-2 groups were progressively more impaired.

Figure 1.

Neurocognition Index in normal controls and brain injury patients.

Figure 2.

Complex attention in normal controls and brain injury patients.

Figure 3.

Cognitive flexibility in normal controls and brain injury patients.

Figure 4.

Reaction time in normal controls and brain injury patients.

Figure 5.

Psychomotor speed in normal controls and brain injury patients.

Figure 6.

Memory in normal controls and brain injury patients.

MANOVA indicates whether or not there are overall group differences for a particular test or domain. In order to determine exactly where those differences lay, it was necessary to perform paired t tests. In Table 3, it is apparent that normal subjects differ from PCS patients in the composite score (the NCI) and in 4 of the 5 domains: psychomotor speed, reaction time, cognitive flexibility, and complex attention. Normal controls did not differ from the MBI group, patients who recovered from concussion in any area. Normal controls differed from the 2 severe TBI groups in every domain.

Table 3.

Paired t Tests Between Controls and Brain Injury Patients

	NML vs PCS	NML vs MBI	NML vs TBI-1	NML vs TBI-2
	t	P <	t	P <	t	P <	t	P <
NEUROCOGNITION INDEX ss	5.46	.0000	0.82	.4156	17.50	.0000	22.18	.0000
MEMORY ss	1.65	.1004	0.76	.4495	8.90	.0000	11.34	.0000
PSYCHOMOTOR SPEED ss	3.34	.0011	0.16	.8763	13.54	.0000	16.94	.0000
REACTION TIME ss	3.85	.0002	0.46	.6441	12.41	.0000	8.74	.0000
COGNITIVE FLEXIBILITY ss	3.75	.0003	0.39	.6943	15.32	.0000	16.98	.0000
COMPLEX ATTENTION ss	3.34	.0011	-0.55	.5820	14.19	.0000	15.89	.0000

NML = normal controls; PCS = patients with postconcussion syndrome; MBI = mild brain injury; TBI-1 = severe traumatic brain injury; TBI-2 = severe, disabling traumatic brain injury; ss = subjects

PCS patients differed significantly from MBI patients in psychomotor speed, reaction time, and complex attention. PCS patients differed from the TBI-1 patients in 3 of 5 domains, and from the TBI-2 group in 4 of 5 domains (Table 4). The NCI was significantly different in all of these comparisons.

Table 4.

Paired t Tests Between PCS Patients and Other Brain Injury Patients

	PCS vs MBI	PCS vs TBI-1	PCS vs TBI-2
	t	P <	t	P <	t	P <
NEUROCOGNITION INDEX ss	−2.39	.0253	2.42	.017499	4.86	.0000
MEMORY ss	−0.71	.4880	1.66	.1011	3.19	.0034
PSYCHOMOTOR SPEED ss	−2.66	.0141	2.56	.012143	6.07	.0000
REACTION TIME ss	−2.07	.0491	1.49	.139726	1.18	.2474
COGNITIVE FLEXIBILITY ss	−2.05	.0512	2.86	.00537	5.01	.0000
COMPLEX ATTENTION ss	−2.43	.0232	2.79	.006662	4.52	.0001

PCS = patients with postconcussion syndrome; MBI = mild brain injury; TBI-1 = severe traumatic brain injury; TBI-2 = severe, disabling traumatic brain injury; ss = subjects

The MBI patients, like the normal controls, differed from the 2 severe TBI groups in every area. The 2 severe TBI groups also differed significantly, in most domains (Table 5).

Table 5.

Paired t Tests Between MBI and TBI Patients

	MBI vs TBI	MBI vs TBI-2	TBI-1 vs TBI-2
	t	P <	t	P <	t	P <
NEUROCOGNITION INDEX ss	6.07	.0000	9.17	.0000	4.62	.0000
MEMORY ss	3.27	.0015	4.90	.0000	4.34	.0000
PSYCHOMOTOR SPEED ss	6.26	.0000	11.18	.0000	6.39	.0000
REACTION TIME ss	4.97	.0000	4.96	.0000	−0.16	.8724
COGNITIVE FLEXIBILITY ss	5.97	.0000	9.03	.0000	3.35	.0012
COMPLEX ATTENTION ss	5.87	.0000	9.29	.0000	2.52	.0134

MBI = mild brain injury; TBI-1 = severe traumatic brain injury; TBI-2 = severe, disabling traumatic brain injury; ss = subjects

On the basis of these comparisons, it appears that the most consistent differences occurred in the composite NCI and in the domains of psychomotor speed and cognitive flexibility. In order to determine in a more systematic way which elements of the test battery best discriminated patients in the different groups, canonical discriminant function analysis was done. Since MBI patients did not differ from normal controls, they were excluded from this analysis. Comparing controls with PCS patients, TBI-1 patients, and TBI-2 patients, one factor was generated that explained 95.6% of the variance with an eigenvalue of 2.294. Contributing to this factor were the NCI (0.838) and domain scores for cognitive flexibility (0.777), psychomotor speed (0.748), attention (0.717), and memory (0.515).

To address the relative importance of different domains for the purpose of discriminating between normal controls and brain injury patients, we examined the issue of sensitivity and specificity by generating ROC curves. The clinically relevant comparison here would be between normal subjects and PCS patients. The better a test is at discriminating between controls and patients, the greater is the area under the ROC curve. This is a more rigorous test than MANOVA. ROC curves indicated significant differences in the domains of psychomotor speed and cognitive flexibility and in the composite NCI (Table 6).

Table 6.

Areas Under the ROC Curves, Controls vs PCS Patients

	AUC	Asymptotic Sig
PSYCHOMOTOR SPEED ss	0.752	0.0170
NEUROCOGNITION INDEX ss	0.747	0.0192
COGNITIVE FLEXIBILITY ss	0.708	0.0485
COMPLEX ATTENTION ss	0.643	0.1761
MEMORY ss	0.620	0.2567
REACTION TIME ss	0.618	0.2644

AUC = area under the curve; ss = subjects

Discussion

A study like this is usually designed to employ an assessment instrument (eg, a computerized test) to describe something new or important about a pathological condition (eg, brain injury). That was not the purpose of this investigation. We already know a great deal about the neuropsychological effects of brain injury, and our results simply restate what is known. TBIs impair cognition in every domain, and the severity of the injury determines the degree of cognitive impairment. Tests of psychomotor speed and executive function are particularly sensitive to the effects of TBI. Tests of memory are also sensitive to brain injury, and CNS VS demonstrated memory deficits in patients who had severe brain injuries (but not concussion – see below).

The purpose of this investigation was not to demonstrate that brain injury patients have cognitive impairment, but rather to evaluate the discriminant validity of the CNS VS battery in patients with a spectrum of TBIs. The results of the investigation indicate that the CNS VS battery does have discriminant validity. When administered in a clinical setting to brain injury patients, the test battery generates results that are similar to conventional neuropsychological tests and other computerized test batteries.

For example, it is known that the postconcussional state is associated with neurocognitive deficits. The effects of concussion on neuropsychological function have been described as “moderate,” on the basis of meta-analysis of 39 studies involving 1463 subjects. These effects, however, were observed only during the acute phase (ie, within 3 months of injury).[27] After more time passes, the neurocognitive status of patients who were concussed returns, in the majority of cases, to normal.[27–30]

On the CNS VS battery, patients with acute postconcussional symptoms were found to have measurable cognitive deficits. Patients who were more than 12 months post concussion performed normally on the test battery. The cross-sectional nature of this research did not allow the authors to track individual patients' courses of recovery, and that is a regrettable weakness. One of the advantages of computerized tests is that patients can be tested serially as they recover from brain injury, and their response to therapeutic interventions can be appraised. The data in this study indicate that CNS VS is sensitive to recovery from mild brain injury, but only on the basis of cross-sectional data, not serial data from individual patients.

Another weakness is that memory impairment, a common complaint in PCS patients, is not captured by the CNS VS battery. In fact, memory is the only cognitive domain in which CNS VS does not discriminate between controls and PCS patients. The failure of the memory domain to achieve statistical significance in a pairwise t test may simply be a function of the small cell size (N = 13). Alternatively, the recognition memory tasks in CNS VS may not be as sensitive to the mild memory deficits of the PCS patients as a recall tack would be.

We have presented data on a composite score, the Neurocognition Index (NCI), a summary score of the patient's cognitive status, generated by averaging the 5 domain scores. The idea of generating a summary score for a battery of neuropsychological tests is controversial, however. The Halstead-Reitan battery has an “impairment index” that is not unlike the NCI in CNS VS. But many neuropsychologists eschew the idea that a single number can capture the complexity of a full neuropsychological battery.[31] The value of neuropsychological testing, it is argued, is to identify particular areas of cognitive weakness, not to establish whether the patient is globally impaired. That is a defensible argument, and one with which we sympathize.

The NCI in CNS VS is currently employed for research purposes only. We are not, ourselves, convinced of its clinical utility. Although the data presented here are supportive of the NCI, at least statistically, it may only be useful as an illustrative device. For example, the columns in Table 2 indicate that PCS patients are functioning about 1 standard deviation lower than controls (standard score [ss] = 82); the TBI-1 patients about 2 standard deviations lower (ss = 66); and the TBI-2 patients more than 3 standard deviations lower (ss = 47). But only more research will decide the larger utility of the NCI.

Some authors assert that MBI patients have mild persistent deficits, persisting even after 12 months, but apparent only on very sensitive tests[32] or under circumstances of physiological stress.[33] In our sample, none of the composite or domain scores of MBI patients were significantly different from controls, although 2 primary scores were (Stroop errors and CPT reaction time). Correcting for multiple measures diminishes, but does not vitiate, the significance of these differences. It is possible that CNS VS can detect the very mild cognitive deficits that persist in some patients who have had a mild brain injury, but it is also possible that our findings are simply an artifact of small samples. In cases where the persistence of mild cognitive impairment is at issue, formal neuropsychological testing is preferred to any computerized test.

It is well-established that following brain injury, functional disability is largely the result of cognitive impairment. In accord with that principle, CNS VS successfully marked TBI patients at varying degrees of severity. Patients who had sustained severe TBIs, but who were functionally independent, performed better on the test battery than severe TBI patients who remained functionally dependent in major areas.

It is also known that deficits in psychomotor speed and executive control functions characterize patients who have had brain injuries. Reaction time tests have consistently indicated slowness of information processing, deficits in divided and sustained attention, and inconsistent performance following brain injury.[34,35] Analysis of the CNS VS data by discriminant function analysis and ROC curves reflects this pattern, although the relatively small number of subjects is a weakness for these kinds of analysis.

The use of a computerized battery of tests is given to advantages and disadvantages. Weber and colleagues,[36] for example, reported that patients' negative attitude toward computers was associated with “nervousness” during testing and poor results in certain tasks. On the other hand, Gur and colleagues[37] reported that patients performed equally well (or poorly) on traditional neuropsychological batteries and computerized tests, and that “patients tolerated the computerized scan well. In contrast to the traditional battery, which taxes patients' endurance, patients seemed to appreciate the brevity of the computerized scan. They did not have difficulties operating the computer and informally they appeared more relaxed being tested by a computer rather than a person” (p 781). Our clinical experience with CNTs in general, and CNS VS in particular, is in support of that view.

This investigation demonstrates that the performance of brain injury patients on the CNS VS battery is reflective of the existing neuropsychological literature. The results provide at least a degree of discriminant validation to the test battery. It appears that the test battery is equally valid in patients with PCS and MBI on the one hand, and in patients with severe TBIs on the other. It is likely that other broad-spectrum computerized batteries, like the NES or ANAM, would prove to be sensitive to the effects of mild and severe brain injuries, if they applied to the method adopted in this paper. We cannot speculate concerning the spectrum of sensitivity of narrower computerized batteries, like ImPact, CogState, or HeadMinder, which are advertised primarily for concussion management. The ideal investigation, of course, would involve direct comparison of different computerized batteries, compared with conventional neuropsychological tests.

CNTs like CNS VS and others may be valid indicators of brain injury. It is necessary, though, to consider how they may be used, or misused. First, our data do not establish that CNS VS is a “diagnostic” test. The diagnostic utility of any test requires an entirely different kind of analysis, and no computerized test has yet met that standard. CNTs do no more than indicate a patient's level of function at a point in time. They may do so precisely, and as reliably as any other test, but interpreting the meaning of that information is the responsibility of a skilled clinician. Diagnosis is a clinical exercise that relies on data from many different sources.

The results of computerized tests like CNS VS do not have the specificity necessary to qualify as diagnostic tools. They are highly sensitive to mild cognitive dysfunction, though, and that makes them suitable to be used as screening instruments.

A second caveat: CNTs like CNS VS generate massive amounts of precise data that can be misinterpreted or misused by poorly trained clinicians. In our communications with psychiatrists and neurologists who have used the test in their practices, we have not always been impressed by their facility at judging exactly what the test means and what to do next. The widespread use of computerized testing by unqualified professionals would be cause for concern. The purpose of a screening battery is to identify patients who need referral to a qualified specialist.

CNTs like CNS VS afford the clinician a tool that is more precise than patients' subjective complaints but less definitive than a diagnostic neuropsychological battery. Since they are easier to apply at frequent intervals than conventional testing, they are ideal for the purpose of serial testing of patients recovering from brain injury (or stroke), and for assessing the utility of treatments like cognitive remediation or medications. In our clinics, CNS VS is used to complement the neuropsychological assessment, and then serially, to evaluate the effects of treatment. The results of this investigation support the validity of this approach.

Appendix: The CNS VS Assessment Battery

Verbal Memory Test (VBM) and Visual Memory Test (VIM)

The CNS VS battery includes parallel tests of verbal memory (word list learning) and visual memory (figure learning). The tests are virtually identical, but one uses words as stimuli, the other, geometric shapes.

The VBM is an adaptation of the Rey Auditory Verbal Learning Test.[21,22] It is a recognition test, however, not a test of recall. In the CNS VS version, 15 words are presented, one by one, on the screen. A new word is presented every 2 seconds. The subject is asked to remember these words. Then a list of 30 words is presented. The 15 target words are mixed randomly among 15 new words. When the subject recognizes a word from the original list, he or she presses the space bar. After this trial of 30 stimuli, the subject goes on to do the next 6 tests. At the end of the battery, about 20 minutes later, the 15 target words appear again, mixed with 15 new nontarget words.

The VIM in CNS VS is based on the Rey Visual Design Learning Test; the latter is, in turn, a parallel to the Rey Auditory Verbal Learning Test, using geometric figures rather than words, and requiring the subject to draw the figures from memory. In CNS VS, the VIM is just like the VBM. Fifteen geometric figures are presented; the subject has to identify those figures nested among 15 new figures. Then, after 5 more tests, there is a delayed recognition trial.

The VBM draws from a reservoir of 120 words selected from word-frequency tables. The VIM draws from a reservoir of 120 simple geometric designs. The scoring is straightforward: correct hits and correct passes, immediate and delayed.

Correct responses from VBM and VIM are summed to generate a composite memory or memory domain score. The highest score one can attain is 120; the lowest is 60. Scores below 60 suggest willful exaggeration.

Finger Tapping Test (FTT)

The FTT is one of the most commonly used tests in neuropsychology, because of its simplicity and reliability, and because it generates relevant data about fine motor control, which is based on motor speed as well as kinesthetic and visual-motor ability.[38] It was one of the core tests of the Halstead-Reitan Battery, which dates to the 1940s, but similar tests were used by nineteenth century psychologists like Wundt, Galton, and Cattell. The FTT is believed to be one of the most sensitive neuropsychological tests for determining brain impairment.[38]

In CNS VS, the FTT is a very simple test. Subjects are asked to press the space bar with their right index finger as many times as they can in 10 seconds. They do this once for practice, and then there are 3 test trials. The test is repeated with the left hand. The score is the average number of taps, right and left.

Symbol Digit Coding (SDC)

Coding has been a component of the Wechsler Intelligence Scales since 1944 (Digit Symbol Substitution, DSST). The Symbol Digit Modalities Test (SDMT)[23] is a variant of the Wechsler DSST, but the position of symbols and digits is reversed. The clinical and psychometric properties of the SDMT are similar to those of the DSST. Although the SDMT may be a “harder” test, and thus more sensitive to neurotoxicity, performance on the SDMT and the DSST are highly correlated.[39] Smith maintained that the SDMT was “usually the most sensitive (test) to the presence of acute or chronic ‘organic’ cerebral dysfunction.”[23] A modification of the SDMT was used by Letz in the NES2 – the subject types in numbers from the keyboard corresponding to noniconic symbols in the key.

In the CNS VS SDC, the subject is given a training session to learn how to link numbers to symbols. The test itself consists of serial presentations of screens, each of which contains a bank of 8 symbols above and 8 empty boxes below. The subject types in the number that corresponds to the symbol that is highlighted. Only the digits from 2 through 9 are used; this to avoid the confusion between the number “1” and the letter “I” on the keyboard. The test lasts for 120 seconds. The goal is to type in as many correct numbers as one can in 120 seconds.

Neither the SDMT nor the DSST are suitable for repeated administration because subjects are able to remember the code and thus accelerate their performance.[40] Modifications in the test are necessary if it is to be used repeatedly; for example, changing the code in a random way on successive administrations. The SDC in CNS VS draws from a reservoir of 32 symbols. Each time the test is administered, the program randomly chooses 8 new symbols to match to the 8 digits.

Scoring is the number of correct responses generated in 2 minutes. The total of right and left taps from the FTT and total correct responses on the SDC generates a composite score for “psychomotor speed.”

The Stroop Test (ST)

In 1935, the psychologist J.R. Stroop demonstrated that naming is slowed when subjects are asked to name the ink color of an incongruous color word; for example, the word “blue” printed in red ink.[41] The incongruity of word color and word meaning generates an “interference” effect.

The ST is still used as part of standard neuropsychological batteries, and several computerized versions of the test have been developed. It is a favorite test in studies of the neurocognitive effects of CNS drugs, especially antiepileptic drugs.

There have been several versions of the ST over the years. The modification adopted for CNS VS uses only 4 colors/color words (red, green, yellow, blue), and only one key is in play, the space bar. The test has 3 parts. In the first, the words RED, YELLOW, BLUE, and GREEN (printed in black) appear at random on the screen, and the subject presses the space bar as soon as he or she sees the word. This generates a simple reaction time score.

In the second part, the words RED, YELLOW, BLUE, and GREEN appear on the screen, printed in color. The subject is asked to press the space bar when the color of the word matches what the word says. This generates a complex reaction time score.

In the third part, the words RED, YELLOW, BLUE, and GREEN appear on the screen, printed in color. The subject is asked to press the space bar when the color of the word does not match what the word says. This part also generates a complex reaction time score, called the “color-word reaction time.” The color-word reaction time is, on average, 120 msecs longer than the complex reaction time generated in part 2 of the test (range, 78–188 msecs) (the “Stroop effect”). Part 3 also generates an error score.

Averaging the 2 complex reaction time scores from the ST generates a domain score for “reaction time.” It would be more precise to refer to this domain score as “information processing speed in a test of executive function.”

The Shifting Attention Test (SAT)

The SAT measures the subject's ability to shift from one instruction set to another quickly and accurately. In the SAT, subjects are instructed to match geometric objects either by shape or by color. Three figures appear on the screen, 1 on top and 2 on the bottom. The top figure is either a square or a circle. The bottom figures are a square and a circle. The figures are either red or blue; the colors are mixed randomly. The subject is asked to match one of the bottom figures to the top figure. The rules change at random. For one presentation, the rule is to match the figures by shape; for another, by color. This goes on for 90 seconds. The goal is to make as many correct matches as one can in the time allotted. The scores generated by the SAT are: correct matches, errors, and response time in milliseconds.

There is not a precise parallel to the SAT in the compendium of conventional neuropsychological tests, although Trails B and the Wisconsin Card Sort are sometimes considered to be tests of shifting attention. Computerized tests, however, like the NES2, CogState, and CANTAB, have shifting attention tests that are not dissimilar to the SAT,[42–44] and color-shape tests like the SAT have been used in cognitive imaging studies.[24,25]

A domain score for cognitive flexibility is generated by taking the number of correct responses on the SAT and subtracting the number of errors on the SAT and the ST.

The Continuous Performance Test (CPT)

The CPT is a measure of vigilance or sustained attention or attention over time.[26] It has been a popular test because of its robust relationship to psychiatric disorders. Poor performance on the CPT has been reported in ADHD,[45,46] learning disabilities,[47–49] patients with epilepsy,[50] and schizophrenics.[51,52] It is sensitive to CNS dysfunction in general and is not specific to any particular condition.[53]

The CPT is also sensitive, for better or worse, to the effects of various drugs. In children with ADHD, performance on the CPT is reliably improved by stimulant medications.[54,55] Alcohol consumption[56] adversely affects performance on the CPT, but nicotine tends to improve performance on the test.[57] Certain anticonvulsant medications impair performance on the CPT.[58]

The CPT in VS is a conventional version of the test, although it is shorter than some other versions. In the VS CPT, the subject is asked to respond to target stimulus “B” but not to any other letter. In 5 minutes, the test presents 200 letters. Forty of the stimuli are targets (the letter “B”); 160 are nontargets (other letters). The stimuli are presented at random, although the target stimulus is “blocked” so it appears 8 times during each minute of the test.

Scoring is correct responses, commission errors (impulsive responding), and omission errors (inattention). The CPT also reports subjects' choice reaction time for each variable. A domain score for “complex attention” is generated by adding the number of errors committed in the CPT, the SAT, and the ST.