Monitoring Resolution of Postconcussion Symptoms in Athletes: Preliminary Results of a Web-Based Neuropsychological Test Protocol

Abstract

Objective:

A new Web-based neuropsychological test was field tested to determine usefulness in detecting and monitoring resolution of symptoms after sport-related concussions and in providing objective information for return-to-play decisions.

Design and Setting:

We obtained neuropsychological baseline data on all subjects. After concussion, subjects were administered alternate, equivalent follow-up tests until symptoms resolved. Follow-up testing typically occurred at 1- to 2-day intervals after the concussion.

Subjects:

Baseline testing was obtained for 834 athletes as part of ongoing field trials. Subsequently, 26 athletes sustained concussions and were studied.

Measurements:

We administered The Concussion Resolution Index (CRI) at baseline and alternate forms posttrauma. Follow-up tests included a self-report inventory of neurophysiologic symptoms.

Results:

A total of 88% of patients were identified as symptomatic on initial postconcussion testing. The CRI appeared relatively resistant to retest effects, and multiple administrations tracked resolution of symptoms over short and extended time periods.

Conclusions:

Although the CRI is still in field trials, preliminary data indicate that the CRI may be a useful method for athletic trainers and other professionals to expeditiously track resolution of symptoms after sport-related concussion.

Team physicians and athletic trainers increasingly are recognizing the importance of concussion management in athletes. Although consensus has not been reached regarding the specifics of return-to-play guidelines after a concussion, experts uniformly agree that athletes should not return to play until all symptoms have resolved. Concussive injuries require immediate evaluation in all athletes, typically by means of a short neurologic screening and mental status test.^1,2 After the initial assessment and characterization of an injury, subsequent follow-up tests may show persistence of symptoms for 3 to 10 days in uncomplicated cases of single concussion.³

Neuropsychological evaluation using comparison of preinjury baseline and postconcussion performances is considered the most sensitive objective method of detecting the presence and resolution of cognitive postconcussion symptoms.⁴ Typically, mild changes in cognitive functions such as attention and concentration, memory, information processing, and motor speed are identified.⁵ In practice, however, implementing traditional neuropsychological protocols is time consuming, inefficient, and expensive. Athletes must be tested individually in a face-to face format; testing requires 30 minutes to 2 hours per athlete; administration, scoring, and statistical analysis must be carried out by trained clinicians; and, in the event of a concussion, follow-up tests must be scheduled, administered, scored, and statistically analyzed. All of these factors impede the athletic trainer's ability to use the results in a timely fashion.

Another well-known and longstanding problem in neuropsychology is related to statistical analysis of serial assessments. How test-retest reliability, practice effects, and their interaction should be most accurately identified and accounted for in the interpretation of serial test results is not entirely clear. Test-retest reliability is rarely provided for many tests used in sport concussion assessment, and reported reliability estimates are derived from relatively long between-test time intervals. Echemendia et al⁶ reported test-retest reliabilities for the following commonly used tests: Controlled Oral Word Association Tests, r₁₂ = .77; Symbol Digit Modalities Test, r₁₂ = .70; Trail-Making Test Part A, r₁₂ =.43; Trail-Making Test Part B, r₁₂ = .54; VIGIL 1, r₁₂ = .49; Digit Span, r₁₂ = .52. While no “gold standards” in the field characterize short interval test-retest reliability, investigators recently examined test-retest changes and described reliability estimates from .70 to the low .90s as “generally good.”⁷ These estimates are consistent with standards usually applied in the field.

In addition, knowledge of the management of practice effects and their relationship to test-retest reliability in group analyses in particular is limited. Significant practice effects have been documented in frequently used tools for assessment of concussion, including the Paced Auditory Serial Addition Test, the Stroop Color Word Test, and the Trail-Making Test Parts A and B.⁸ Furthermore, practice effects seem to vary across tests, across test-retest intervals, and across subjects.^7,9 Although alternate test forms can mitigate these problems to a certain extent, such forms are not available for many frequently administered tests, and little has been published on the use of alternate forms for follow-up assessment of postconcussion cognitive symptoms.

These challenges have complicated the implementation of the baseline-posttrauma assessment model of sport concussion. In order to address and more precisely manage limitations due to test-retest reliability and assess the practice effects, a number of researchers have employed the Reliable Change Index (RCI).^6,10 This statistical technique allows for direct comparison of an athlete's postconcussion performance with his or her baseline performance while incorporating test reliability and practice effects into statistical computations and results. The RCI was designed specifically to assess intraindividual differences across time. In group analyses, on the other hand, means and their standard deviations are typically employed for comparison, and no built-in mechanism directly accounts for low reliability coefficients or practice effects. The development of computerized measures has been proposed in order to tackle many of the practical, psychometric, and statistical challenges in sport concussion assessment.^5,10,11 Recently, several computerized measures have been used in sports, including the Automated Neuro-Psychological Assessment Metrics (ANAM, Office of Military Performance Assessment Technology, Walter Reed Army Institute of Research, Washington, DC),¹¹ Immediate Post-Concussion Assessment and Cognitive Testing (ImPACT, Henry Ford Research Center, Detroit, MI),⁵ and Vigil Continuous Performance Test (VIGIL, The Psychological Corporation, San Antonio, TX).⁶ However, limited information is available regarding their suitability for baseline and posttrauma analyses.

The Concussion Resolution Index (CRI) (HeadMinder, Inc, New York, NY)¹² is a Web-based computerized neuropsychological assessment battery designed for athletic trainers and other professionals who manage and monitor resolution of symptoms due to sport-related concussion. The CRI was developed to address many concerns regarding the use of current assessment techniques, including issues of test-retest effects, practice effects, the need for alternate forms, ease of administration, time efficiency, and cost.

METHODS

Participants and Design

Participants were athletes recruited from 14 teams and institutions. Institutional review board approval was obtained for 12 institutions, including all high schools. Two institutions for adult athletes did not have an institutional review board, nor did they request such review. Informed consent was obtained from all participants; parental consent was obtained for all high school students. Baseline CRI evaluations were performed on 834 athletes. Assessments were conducted in institutional computer laboratories, with athletes in groups ranging in size from 4 to 20. Trained administrators supervised all assessments. Subsequently, athletes who sustained a concussion, as identified by their treating professional, received follow-up tests until all symptoms resolved. These follow-up tests were administered according to clinical judgment, typically at 1- to 2-day intervals. The test findings were not used as a return-to-play criterion.

Evaluation Measure

The CRI is a Web-based computerized neuropsychological assessment instrument that was developed specifically to compare an athlete's postconcussion performance with his or her own pretrauma baseline performance. The CRI can be administered from any computer with an Internet connection. The administrator views test results and reports from any Internet-connected computer using a secure password. CRI baseline assessments take less than 25 minutes, and postconcussion assessments take approximately 20 minutes. Statistical analyses of injured athletes' test performances are adjusted for test-retest reliability and for practice effects, and reports are generated automatically after the posttrauma test is completed. Alternate forms are available so that multiple follow-up assessments can be administered to track resolution of cognitive symptoms. Self-reported neurophysiologic symptoms are also incorporated into the results along with pertinent medical background and concussion history. This information is included to assist the treating professional in gathering a comprehensive assessment of postconcussion symptom pattern and resolution. All records, including injury and medical history, are secure, confidential, and available only to authorized professionals with a password.

The CRI includes measures of cognitive functions associated with postconcussion syndrome, such as memory, reaction time, speed of decision making, and speed of information processing.^10,13,14 Six subtests are administered at baseline and again at each posttrauma evaluation. These 6 subtests constitute 3 speeded test indices and 2 error scores (Figure 1). Animal Decoding, during which athletes are instructed to type in numbers keyed to pictures of animals, and Symbol Scanning, during which athletes are instructed to rapidly determine whether identified sets of symbols are present among a set of distractors, comprise the Processing Speed index. Reaction Time, during which athletes press the spacebar when a target shape appears on the screen, and Cued Reaction Time, during which athletes press the spacebar when a target shape appears immediately after a “cue” shape, comprise the Simple Reaction Time index. An error index is also calculated based on total false-positives and false-negatives in these 2 tests. Visual Recognition 1 and Visual Recognition 2 present series of pictures, some of which are repeated. Athletes are instructed to press the spacebar as quickly as possible whenever they recognize a picture from an earlier exposure. The response latency is measured. An error index is calculated based on total false-positive and false-negative responses on these 2 tests.

Figure 1.

Derivation of indices.

Because the CRI is a new instrument, we describe its principal psychometric characteristics. Norms were derived from performance data collected from 414 athletes in high school, college, and club settings during the initial test development phase.¹⁵ Of these, 216 were male (52%) and 198 female (48%). Roughly equal numbers of participants younger and older than 18 years were recruited to provide normative data suitable for use with both high school and adult-aged populations. Most (58%) were white, 12% were African American, 6% were Asian American, and 4% were Hispanic. Six percent identified their ethnicity as “other,” and 14% chose not to identify an ethnic group.

Concurrent validity analysis reveals that CRI indices and subtests are correlated with traditional neuropsychological measures. Correlation of the CRI Processing Speed Index with other measures of processing speed was 0.66, 0.60, 0.57, and 0.58 for the Symbol Digit Modality Test, the Grooved Pegboard Test (dominant and nondominant hands), and the WAIS-III Symbol Search subtest, respectively. The CRI Simple Reaction Time Index correlated with Grooved Pegboard performance at 0.46 and 0.60 for the dominant and nondominant hands, respectively, and with the Trail-Making Test Part A at 0.56. Correlations of the CRI Complex Reaction Time Index with Grooved Pegboard performance were 0.59 and 0.70 for the dominant and nondominant hands, respectively. These correlations are mostly in the moderate range¹⁶ and indicate that the CRI indices measure similar constructs in normative subjects. However, it is not clear at this time, given the differences in the interface between computerized and noncomputerized instruments, whether they actually measure identical constructs. Moreover, although 2 measures may be correlated in healthy subjects, we may not be able to assume that this is the case in a sample of concussed individuals. For instance, the Digit Symbol and Symbol Digit Modalities Tests have a correlation of 0.73 in normal subjects. However, reports of the clinical sensitivity of the Symbol Digit Modalities Test to sport-related concussion outnumber those for the Digit Symbol Test.⁴ One purpose of our study was to explore the clinical validity of the CRI indices.

For athletes in college or adult club teams, 2-week test-retest reliabilities for the CRI Indices were 0.90 for the Processing Speed Index, 0.73 for the Simple Reaction Time Index, and 0.72 for the Complex Reaction Time Index. For high school students, 2-week test-retest reliabilities were 0.79 for the Processing Speed Index, 0.72 for the Simple Reaction Time Index, and 0.65 for the Complex Reaction Time Index. These test-retest reliabilities compare reasonably well with those noted above for tests typically administered in the assessment of sport-related concussion.

Statistical Analysis

For statistical analyses, performances on the posttrauma tests are compared with baseline scores for each of the 3 indices and the 2 error scores. For the indices, which comprise normally distributed scores, the RCI¹⁷ is used. The RCI yields a standard score (z score) as follows: x₂−x₁/s_diff, where s_diff = √2(SE²), and SE is the standard error of measurement. The RCI is used to determine if changes in scores from baseline to posttrauma retest are significant. As stated above, this is a useful means of comparing preinjury and postinjury scores derived from traditional face-to-face neuropsychological measures.¹⁰ Analysis of normative data revealed a uniform significant practice effect on the Processing Speed Index. Therefore, in accordance with the Temkin et al^7,9 model, injured players' trauma scores were adjusted by −0.17 seconds on the Processing Speed Index. For the Simple Reaction Time and Complex Reaction Time Indices, mean practice effects of 0.004 and −0.001 seconds, respectively, were identified. Adjustments for these effects were not included due to their size. In keeping with previous use of the reliable change index in a sport setting,⁹ the CRI classifies P values of less than .05 (1-tailed test) as indicative of significantly worsened (ie, slowed) performance when compared with baseline scores. P values ranging from .05 to .15 are described as suggestive of borderline worsened performance due to concussion. Analyses of error scores, which are not normally distributed, are based on the frequency of additional errors on the postinjury test compared with the baseline levels. Cutoffs used for classification of performance are based on additional error values (rather than absolute number of errors) found in fewer than 15% and 5% of the normative sample, representing borderline and significant increases in errors, respectively.

Classification of Test Results

The overall performance of athletes on each posttrauma CRI administration is depicted visually by a traffic light and classified as red, yellow, or green. Red lights reflect statistically significant cognitive test results or 3 or more neurophysiologic symptoms, or both, and indicate that symptoms do not appear to have resolved. Yellow lights indicate borderline cognitive test results or 1 or 2 neurophysiologic symptoms, or both, signaling a need to examine results more closely in view of other factors. Green lights indicate nonsignificant cognitive test findings and no self-reported ongoing neurophysiologic symptoms. If the athlete reports any neurophysiologic symptoms, even if all postconcussion cognitive test scores are within normal limits, performance is classified as either yellow or red, depending on the number of symptoms. All results are identified as statistical test findings or self-reported test findings, or both; no interpretation accompanies these reports. Clinicians are urged to consider a range of factors not assessed by the CRI in any return-to-play decision.

RESULTS

Twenty-six athletes sustained concussions. Fourteen (54%) of these athletes underwent periodic follow-up CRI evaluations until all cognitive and all self-reported neurophysiologic symptoms resolved. Four athletes (15%) underwent follow-up CRI evaluations until cognitive symptoms resolved, but they remained classified as yellow or red due to ongoing self-reported symptoms only. Four other athletes (15%) underwent at least 2 postconcussion CRI exams and were identified with cognitive symptoms at the final posttrauma test. Four athletes (15%) underwent only one postconcussion evaluation with significant cognitive findings and no further CRI follow-up.

Twenty-three of the 26 concussed athletes (88%) were classified as either red (n=16) or yellow (n=7) on the first follow-up test (Figure 2). Three of these athletes (12%) had positive findings (2 red, 1 yellow) due to decreased cognitive performance alone, in comparison with their own baseline performances. These athletes would not have been identified as symptomatic based solely on self-report. At the second postconcussion test (Figure 3), 12 athletes still showed evidence of ongoing cognitive or neurophysiologic symptoms, or both, with 7 classified as red and 5 as yellow. Twenty-five percent of these athletes were identified as symptomatic based on their performance on cognitive measurements alone.

Figure 2.

First follow-up after trauma.

Figure 3.

Second follow-up after trauma.

Each athlete's performance on the indices contributed to the initial overall CRI classification status of red (impaired), yellow (borderline), or green (asymptomatic) (Table). Self-reported neurophysiologic symptoms were present in 20 of the 26 (77%) patients. In 18 (69%) of these subjects, at least one cognitive test index indicated borderline or significant decreases in cognitive test performance. Eleven of the athletes (42%) had more than one significant cognitive test finding. Fifteen athletes (58%) manifested both neurophysiologic and cognitive symptoms. Three (12%) had cognitive symptoms without neurophysiologic complaints, and 5 (19%) reported neurophysiologic symptoms but had no significant cognitive findings.

The most sensitive CRI cognitive index, Complex Reaction Time, was significant in 13 (50%) of the athletes. The mean effect size for this index was a z score of −1.44 and ranged from −5.76 to 2.13. The next most sensitive was the Simple Reaction Time Index, which showed results in the borderline to significant range in 11 (42%) of the subjects. The mean effect size for this index was a z score of −1.39 and ranged from −7.04 to 2.04. Finally, the Processing Speed Index showed significant results in 4 (15%) of the cases. The mean effect size for this index was a z score of 0.06 and ranged from −2.78 to 2.90. These large ranges are consistent with the variability seen in postconcussion studies, especially in light of evidence that symptoms may wax and wane over time.^14,19

One athlete (15), who tested within normal limits on follow up, obtained cognitive test scores below baseline on all 3 indices. Although this finding could be due to regression to the mean or other factors, it underscores the importance of interpretation and clinical supervision of any neuropsychological test results.

Figure 4 depicts the initial severity and course of symptom resolution for each athlete according to the time of test administration. The Cantu grading scale¹⁸ is used here because the severity and duration of symptoms are of primary interest. Accordingly, 12 concussions were grade 1 (46%), 6 (23%) were grade 2, and 8 (31%) were grade 3. All patients with concussions who were followed until asymptomatic had resolved their symptoms by day 15, with the grade 3 injuries generally requiring more time for symptom resolution.

Figure 4.

Course of Symptom Resolution.

¹Developed neurophysiologic symptoms and was retested.

²Due to neurophysiologic symptoms only. All cognitive test deficits resolved/athlete not retested.

³Follow-up not obtained.

The CRI appeared to be relatively resistant to retest effects, even when tests were administered at close intervals. Indeed, 3 athletes (14, 19, and 22) consistently showed borderline or impaired results across 3 or more tests within a 30-hour period. More generally, resolution of symptoms appeared to progress linearly for some of the athletes. However, others evidenced a waxing and waning course. For example, 2 athletes' status changed from nonsignificant to significant (6 and 10). When athlete 10 was first tested 2 days postinjury, he was symptomatic. Four days postinjury, all symptoms had resolved; however, he later developed neurophysiologic symptoms (headaches) and was reclassified as red. Ten days postinjury, he was tested again and classified as green. Similarly, when tested 2 days postinjury, athlete 6 had both neurophysiologic and cognitive symptoms. Four days postinjury, all symptoms had resolved; however, he later developed neurophysiologic symptoms (fatigue, difficulty sleeping, and memory problems) and was retested, displaying borderline performance on one of the cognitive indices.

DISCUSSION

Preliminary results indicate that the CRI is sensitive to the sequelae of concussion in athletes. Postconcussion symptoms were detected in 23 of 26 subjects (88%) studied on the initial testing. Of the 26 concussed athletes in our study, 5 athletes (19%) would have been identified by reported neurophysiologic symptoms alone, and 15 athletes (58%) were identified by both cognitive and reported neurophysiologic symptoms. However, 3 athletes (12%) were identified solely by cognitive symptoms, increasing the diagnosed group by 13%. At the second test instance, 12 of the athletes remained symptomatic. Although 6 (50%) were positive solely for reported neurophysiologic symptoms, 3 (25%) were identified purely by cognitive symptoms. These results indicate that adding neuropsychological assessment to a sports medicine protocol may increase diagnostic ability over and above symptom checklists, particularly with repeated follow-up assessments.

Overall, neuropsychological measures worsened in a sizable majority (69%) of the patients. Given that 46% of the subjects had grade 1 concussions, the CRI appeared to be sensitive to even very mild concussions. Moreover, several of the symptom resolution patterns suggest that the CRI remains sensitive to the effects of concussion across multiple follow-up tests. The CRI remained sensitive to grade 3 injuries for periods up to 15 days. In keeping with the findings of Barth et al,^3,13 most symptoms appeared to have resolved by day 10.

The most sensitive CRI neuropsychological measure was the Complex Reaction Time Index, which identified concussed athletes as taking significantly longer to recognize a picture from an earlier exposure trial in 13 (50%) of the cases. This finding is also consistent with the hypothesis of Barth et al,^3,13 that the general cognitive problem underlying worsened performance due to concussion is a decrease in the speed of information processing. Consistent with Bleiberg et al,¹⁴ a measure of simple reaction time was found to be sensitive in a number of patients (42%). Although further research is necessary, these findings provide strong construct validity that the CRI measures concussion as per its design.

Adopting computerized assessments may pose challenges to clinicians. Given an improved level of assessment precision facilitated by computers, clinicians may detect more cases of mild concussion. Greater detection may mitigate the potential for negative outcomes that have been repeatedly linked to early return to play, including increased risk for subsequent concussion,²⁰ development of chronic postconcussion syndrome,²¹ and second-impact syndrome.²² However, the potential increase in detected incidence of concussions may also lead to additional burdens in monitoring and managing a larger number of injured athletes until concussion-management parameters regarding diagnostic criteria, concussion grading scales, and criteria for recovery and return to play are agreed upon. Moreover, this potential increase underscores the importance of the use of clinical judgment by trained professionals along with the CRI data for monitoring resolution of symptoms.

Neuropsychological testing is currently considered the most sensitive objective method for detecting cognitive postconcussion symptoms.⁴ These preliminary results indicate that the CRI holds promise as a method for assessing and tracking resolution of postconcussive cognitive and neurophysiologic symptoms. Because of the advantages offered by being available on a Web-based platform and in light of its built-in statistical analysis, the CRI has the potential to be useful for both research and time-efficient management of sport-related concussion.

Many professional and university sports teams have already integrated neuropsychological assessment into their assessment protocols as their resources support professional neuropsychological consultation.⁵ Unfortunately, the greatest number of at-risk athletes remain in the college, high school, and grade school settings, where resources are scarce and individual neuropsychological evaluation is both impractical and unaffordable. The CRI shows promise as a cost-effective and efficient concussion-management tool that collects, compares, and reports longitudinal cognitive performance and neurophysiologic symptoms and pertinent medical and personal history. Athletic trainers who may oversee literally hundreds of athletes can rely on an objective reporting system, accessible from any Internet-connected location. Moreover, the CRI may facilitate cognitive screening for a large number of underserved young athletes who currently do not receive a complete multidimensional assessment. In light of the current study, as many as 11% to 25% of concussions in this population may go undiagnosed due to the lack of baseline-to-posttrauma cognitive assessment. This concern is especially critical for young athletes, whose safety may be enhanced by empirically derived return-to-play guidelines, especially in light of their greater potential risk for adverse outcomes such as second-impact syndrome.⁴

Although athletic trainers may not currently include neuropsychological assessment in their existing protocol for medical management of athletes, the CRI could easily become part of the routine preseason physical examination clearing process. This would provide baseline cognitive functioning data, pertinent medical history, and other data that could be useful in managing the athlete should a concussion occur. In the case of a possible or definite concussion, the athletic trainer would administer serial CRI posttrauma assessments to monitor symptoms and cognitive functioning. The CRI could also be implemented within an existing sports medicine program that includes neuropsychological assessments. Baseline and follow-up data could also be collected, as described above, in addition to any ongoing cognitive screening program. Optimally, injured athletes would undergo additional serial tests, such as the Hopkins Verbal Learning Task, to assess verbal memory deficits.

The current data represent preliminary findings on relatively few subjects. Further research is needed to determine optimal time frames for monitoring resolution of postconcussion symptoms and identifying symptoms and factors associated with negative outcomes and long-term consequences in order to obtain empirically derived return-to-play guidelines. Future research efforts should address weaknesses in the current data, including obtaining normative data for varying retest schedules and standardized sideline assessment of injury severity, and the use of matched controls to better describe the CRI's sensitivity and specificity. We hope that additional research will result in a more comprehensive understanding of clinical management of sport-related concussion.