Pilot Evaluation of a Novel Clinical Test of Reaction Time in National Collegiate Athletic Association Division I Football Players

Abstract

Context:

Evidence suggests that concussion prolongs reaction time (RT). We have developed a simple, reliable clinical tool for measuring reaction time that may be of value in the assessment of concussion in athletes.

Objective:

To compare baseline values of clinical RT (RT_clin) obtained using the new clinical reaction time apparatus with computerized RT (RT_comp) obtained using a validated computerized neuropsychological test battery.

Design:

Cross-sectional study.

Setting:

Data were collected during a National Collegiate Athletic Association Division I collegiate football team's preparticipation physical examination session.

Patients or Other Participants:

Ninety-four Division I collegiate football players.

Main Outcome Measure(s):

The RT_clin was measured using a 1.3-m measuring stick embedded in a weighted rubber disk that was released and caught as quickly as possible. The RT_comp was measured using the simple RT component of CogState Sport.

Results:

For the 68 athletes whose CogState Sport tests passed the program's integrity check, RT_clin and RT_comp were correlated (r  =  0.445, P < .001). Overall, mean RT_clin was shorter and less variable than mean RT_comp (203 ± 20 milliseconds versus 268 ± 44 milliseconds; P < .001). When RT_clin and RT_comp were compared between those athletes with (n  =  68) and those without (n  =  26) valid CogState Sport test sessions, mean RT_clin was similar (202 ± 19 milliseconds versus 207 ± 23 milliseconds; P  =  .390), but mean RT_comp was different (258 ± 35 milliseconds versus 290 ± 55 milliseconds; P  =  .009).

Conclusions:

The RT_clin was positively correlated with RT_comp and yielded more consistent reaction time values during baseline testing. Given that RT_clin is easy to measure using simple, inexpensive equipment, further prospective study is warranted to determine its clinical utility in the assessment of concussion in athletes.

Key Points

• In this sample of male collegiate football players, using a simple and inexpensive device, clinical reaction time was a valid measure of baseline reaction time.

• Clinical reaction time on the device was both positively correlated with and more consistent than reaction time obtained via computerized testing.

• Clinical reaction time assessment may become a valuable new tool in the multifaceted approach to diagnosis and management of concussion in athletes.

The use of neuropsychological testing for the assessment of sport concussion has rapidly grown in the United States since its introduction to the sports medicine community by Barth et al¹ in the 1980s. Over the past decade, computerized neuropsychological test batteries have become increasingly popular. Computerized batteries offer many benefits over traditional paper-and- pencil neuropsychological tests. These include standardization of stimulus presentation; shorter administration times; the ability to rapidly and accurately analyze and store data, facilitating comparison with prior test performance; the existence of multiple equivalent forms of the test to minimize practice effects; and the ability to simultaneously assess multiple athletes during a baseline testing session independent of a trained neuropsychologist.^2,3 Another advantage of computerized neuropsychological test batteries such as ImPACT (Immediate Post-Concussion Assessment and Cognitive Testing (ImPACT Applications Inc, Pittsburgh, PA),⁴ CogState Sport (CogState Ltd, Melbourne, Australia),⁵ and ANAM (Automated Neuropsychological Assessment Metrics; Defense and Veterans Brain Injury Center, Washington, DC)⁶ is that they can accurately measure reaction time to the millisecond level. This gives computerized testing a considerable advantage over more traditional test methods in detecting the subtle impairments in central processing speed that are typically present after a concussion.

A considerable body of literature^7–13 supports the paradigm of prolonged reaction time immediately after sport-related concussion, followed by gradual improvement with an eventual return to baseline. In addition to absolute prolongation of reaction time, response variability has been shown to increase after concussion.¹⁰ Prolongation of reaction time after concussion parallels the persistence of postconcussive symptoms.^12,13 Furthermore, prolonged reaction time may persist even after full resolution of self-reported symptoms and return to sport based on clinical evaluation.¹¹ These findings suggest that a reaction time measure adds sensitivity to the clinical assessment of concussion compared with self-reported symptoms and a general physical examination alone.

Despite the advantages of computerized neuropsychological testing, it has a limited role in the initial diagnosis of concussion, which must occur on the playing field at the time of injury. Concussion remains a clinical diagnosis that is made by the athletic trainer or team physician based on the athlete's presenting signs and symptoms. Furthermore, the total cost associated with computerized neuropsychological testing, which has been estimated at $669 to $677 per athlete,¹⁴ including the need for a computer platform on which to administer the test, restricts its accessibility. This limitation is most marked for high-school aged and younger athletes, who have fewer available sport-related health care resources than collegiate and professional athletes, and is problematic given that 65% of sport-related concussions occur in persons aged 5 to 18 years.¹⁵ Additional concern can arise when the postinjury neuropsychological test results are compared with the preseason baseline test. An athlete whose motivation is lacking during baseline testing may not put forth his or her best effort, resulting in poor baseline test performance. After injury, however, the athlete is likely to be highly motivated to perform well on neuropsychological tests. Thus, return-to-play decision making may be based on a flawed comparison.^16,17

A simple, inexpensive clinical measure of reaction time that can be used on the sideline or in the training room at the time of injury would be a valuable addition to the sports medicine practitioner's “diagnostic toolbox” for sport-related concussion. Furthermore, such a tool might be valuable for tracking recovery from concussion in young athletes who do not have access to formal computerized neuropsychological testing, and an intrinsically motivating test would reduce baseline inconsistencies in test results due to poor motivation. In order to give clinicians access to such a test of reaction time, we developed the clinical reaction time apparatus, which involves standardizing a simple experiment commonly performed in high school physics classrooms.¹⁸ The apparatus is a thin, rigid cylinder to which a weighted disk is attached. The weighted disk ensures verticality and consistency of hand position. The apparatus is released by the examiner and then caught as quickly as possible by the athlete being evaluated. The distance the apparatus falls before being arrested is measured and converted into clinical reaction time (RT_clin) using the formula for a free body falling under the influence of gravity. Pilot work in a population of healthy adults has shown RT_clin to be a reliable and valid measure of reaction time.^19,20

The purpose of our study was to compare, in a population of collegiate athletes, baseline RT_clin with computerized reaction time (RT_comp) obtained using an established computerized neuropsychological test battery. Our primary hypothesis was that RT_clin and RT_comp would be positively correlated in those athletes who put forth a valid effort on their baseline CogState Sport tests, as measured by passing the program's internal integrity check (see “Methods” section). Failure of the program's integrity check process may be a marker for poor or suspect motivation during the test session. Because reaction time performance is affected by motivation, a secondary hypothesis was that different relationships would be observed between RT_clin and RT_comp in those who passed versus those who failed the CogState Sport integrity check.

METHODS

We recruited 94 National Collegiate Athletic Association Division I collegiate football players at a single university during a preparticipation physical examination session. Before testing, all athletes provided informed written consent, and this research was approved by the institutional review board at our institution. All members of the university's football squad who were at least 18 years of age on the day of the preparticipation examination were eligible for inclusion. Exclusion criteria included self-report of an unresolved concussion or any active injury affecting the right hand or arm that might affect the athlete's ability to participate in RT_clin testing. We recorded each athlete's age and dominant hand at the time of testing.

All athletes completed a single baseline CogState Sport computerized neuropsychological test (version 5.6.4) session. Testing was conducted in groups of 8 athletes at a time on separate personal computers in a computer laboratory and was supervised by physicians familiar with the CogState Sport program. Each athlete wore headphones during the test battery to minimize outside distractions. The CogState Sport computerized cognitive assessment is based on the presentation of playing cards on a computer monitor and includes detection (simple reaction time), identification (choice reaction time), 1-card learning (a continuous visual recognition learning task), and 1-back tests designed to measure psychomotor function, visual attention, continuous visual recognition learning, visual recognition memory, working memory, and attention.²¹ The simple reaction time task involves the presentation of a single playing card face down in the middle of the computer screen. The athlete must press the k key as quickly as possible whenever the card turns face up. At least 35 trials are completed, with additional trials added if the athlete presses the k key before the card turns face up. When such an anticipatory button press occurs, a brief warning is sounded before the next trial to indicate the error. The program also includes a self-reported survey of clinical symptoms and concussion history.

The CogState Sport program undergoes several internal integrity checks (D. Darby, MBBS, PhD, FRACP, CogState Ltd, written communication, September 2009):

1. Simple reaction time accuracy (ie, nonanticipatory response rate) is 90% or greater.

2. Choice reaction time accuracy (ie, correctly indicating whether the card presented is red) is 80% or greater.

3. One-card learning accuracy (ie, correctly indicating whether the card presented has previously appeared during the task) is 53% or greater.

4. Mean simple reaction time is faster than mean choice reaction time.

5. Mean simple reaction time is faster than mean 1-back reaction time (ie, the reaction time to decide whether the card presented is identical to the immediately previous card).

The software then reports whether the athlete's performance passes or fails the integrity check process. The raw simple reaction time data for all nonanticipatory trials were extracted from each athlete's CogState Sport test session for analysis. In addition, we noted each athlete's concussion history, as self-reported in CogState Sport.

Each athlete participated in simple RT_clin testing using the clinical reaction time apparatus, administered by 1 of 3 examiners. These included 2 of the study investigators (J.T.E. and J.K.R.) and 1 member of the university's athletic training staff. The apparatus is a measuring stick, 1.3 m long, coated in high-friction tape and marked in 0.5-cm increments, that is embedded in a weighted rubber disk. The athlete sits with the right forearm resting comfortably on a horizontal desk surface, such that the proximal edge of the hypothenar eminence is positioned at the edge of the desk surface. The examiner suspends the apparatus vertically, with the weighted disk positioned inside the athlete's open hand, such that the superior surface of the weighted disk is aligned with the plane of the athlete's first 2 digits and no part of the athlete's hand is in contact with the weighted disk. The examiner releases the apparatus at predetermined, randomly assigned time intervals of between 2 and 5 seconds to prevent the athlete from anticipating the time of release. The athlete then catches the apparatus as quickly as possible after it begins to fall (Figure 1). In the event of an anticipatory grasp before the device is released, the examiner restarts the random-delay interval count before releasing the device. In the rare instances when an athlete drops the device, a “drop” trial is recorded and that trial is not used in the calculation of mean RT_clin. All athletes were tested using the right hand with a protocol of 2 practice trials followed by 8 data-acquisition trials. The fall distance was measured from the superior surface of the weighted disk to the most superior aspect of the athlete's hand and was converted into a reaction time (in milliseconds) using the formula for a body falling under the influence of gravity (d  =  ½gt²), where d is distance, g is acceleration due to gravity, and t is time.

Figure 1.

The authors demonstrate the clinical reaction time (RT_clin) testing procedure using the clinical reaction time apparatus. The examiner (standing) vertically suspends the apparatus, allowing the spacer portion of the device to rest inside the open hand of the test participant. When the examiner drops the apparatus, the test participant catches it as quickly as possible. The distance the apparatus falls is recorded and converted into a reaction time (in milliseconds) using the formula for a body falling under the influence of gravity.

We calculated means and SDs for RT_clin and RT_comp for each athlete. Mean RT_clin was compared with mean RT_comp in all athletes using paired t tests. We chose to define RT_clin and RT_comp variability for each participant as the SD of his individual clinical or computerized reaction time trials, respectively. We compared the variability in RT_clin with the variability in RT_comp in all 94 athletes using paired t tests. Additional analysis was performed to compare mean RT_clin with mean RT_comp and the variability in RT_clin with the variability in RT_comp using paired t tests in 2 subgroups: athletes whose performance on the CogState Sport test battery passed the program's internal integrity checks and those whose performance did not pass. Mean RT_clin and mean RT_comp were correlated using a Pearson correlation coefficient, r, in both subgroups.

Additional subgroup analyses were conducted using independent-samples t tests to compare both mean RT_clin and mean RT_comp in those athletes who passed versus those who failed the CogState Sport program's integrity checks, those athletes with a self-reported history of concussion versus those without, and left-handed versus right-handed athletes. To identify a possible learning effect for RT_clin and RT_comp, we performed linear regressions of RT_clin and RT_comp by trial number and compared the mean slopes of the 94 least-squares regression lines for each measure with zero using 1-sample t tests. The null hypothesis of zero slope would indicate that mean RT_clin and mean RT_comp did not change over trials and, therefore, would argue against a learning effect. Statistical analyses were conducted using either SPSS (version 14.0; SPSS Inc, Chicago, IL) or SAS (version 9.1; SAS Institute Inc, Cary, NC) for Windows.

RESULTS

A total of 94 male collegiate football players, with a mean age of 19.9 ± 1.5 years (range, 18 to 23 years), participated in this study. Of those, 68 passed the CogState Sport program's internal integrity check. In this subset of athletes, a correlation was seen between mean RT_clin and mean RT_comp (r  =  0.445, P < .001). This relationship was not found in the 26 athletes who did not pass the CogState Sport program's internal integrity check (r  =  −0.080, P  =  .698). In the overall sample of 94 athletes, mean RT_clin was shorter than mean RT_comp (203 ± 20 milliseconds versus 268 ± 44 milliseconds, respectively; P < .001). Furthermore, RT_clin had less variability, than RT_comp (26 ± 13 milliseconds versus 99 ± 58 milliseconds, respectively; P < .001). When mean RT_clin and mean RT_comp were compared between athletes with and without valid CogState Sport sessions, group differences were found in mean RT_comp but not in mean RT_clin (Table 1). Similarly, there was no difference in RT_clin variability between athletes with and without valid CogState Sport test sessions, whereas variability in RT_comp was greater in athletes who did not pass the CogState Sport integrity checks (Table 2).

Table 1.

Mean Clinical (RT_clin) and Computerized (RT_comp) Reaction Time Data for Athletes With Valid and Invalid CogState Sport Sessions (Mean ± SD), Milliseconds

Table 2.

Variability in Clinical (RT_clin) and Computerized (RT_comp) Reaction Time Data for Athletes With Valid and Invalid CogState Sport Sessions (Mean ± SD), Milliseconds

Of the 94 athletes studied, 37 reported a history of concussion. One athlete reported a history of 8 concussions, 2 reported 4 concussions, 2 reported 3 concussions, 9 reported 2 concussions, and 23 athletes reported 1 concussion. Of the 63 concussions reported, 23 involved loss of consciousness and 12 involved amnesia. No information was available regarding how much time had passed since these injuries; however, all athletes had fully recovered and had no concussion symptoms at the time of testing. When comparing the 37 athletes with a history of concussion to the 57 without such a history, no differences were detected in mean RT_clin (201 ± 15 milliseconds versus 205 ± 23 milliseconds; P  =  .225) or mean RT_comp (269 ± 48 milliseconds versus 265 ± 41 milliseconds; P  =  .735). The 11 left-handed athletes had a shorter mean RT_clin than the 83 right-handed athletes (188 ± 12 milliseconds versus 205 ± 20 milliseconds; P  =  .001), but no difference was detected in mean RT_comp between left-handed and right-handed athletes (265 ± 40 milliseconds versus 267 ± 44 milliseconds; P  =  .907).

With respect to a possible learning effect, the mean RT_clin value by trial is illustrated in Figure 2. The mean slope of the least-squares linear regression lines for RT_clin by trial number was −2.4 ± 5.4 milliseconds per trial. The mean slope of the least-squares linear regression lines for RT_comp was −0.92 ± 2.5 milliseconds per trial. Both slopes differed from zero (each P < .001), indicating that RT_clin and RT_comp both changed across trials. Based on this regression analysis, the athletes involved in this study had an average improvement of 2.4 milliseconds per trial for RT_clin and 0.92 milliseconds per trial for RT_comp.

Figure 2.

Clinical reaction time (RT_clin) by trial averaged across all 94 athletes. Error bars represent 95% confidence intervals.

DISCUSSION

The RT_clin and RT_comp were positively correlated in those athletes who put forth a valid effort on their baseline CogState Sport test, as measured by passing the program's internal integrity check, but not in those athletes whose performance did not pass. Although this is not the only possible reason for the difference, failure to pass the CogState Sport integrity check may reflect poor effort or lack of motivation on the part of the athlete taking the test. We believe that this is the most likely explanation for integrity-check failure in our sample of athletes. Thus, RT_clin appears to be a valid measure of reaction time in this population if RT_comp data that have passed the CogState Sport integrity check are the criterion standard for comparison. Similar results were obtained^19,20 when RT_clin was measured in a general population of healthy adults and compared with a different computerized test of reaction time that conformed to widely validated and accepted techniques for measuring reaction time and served as a criterion standard for comparison. The validity of RT_clin is also supported by the finding that RT_clin in left-handed athletes was faster than in their right-handed counterparts, a result consistent with previous reaction time research involving athletes.²² The presence of similar learning effects for RT_clin and RT_comp further supports the validity of RT_clin.

In this sample of 94 male football players, baseline performance on the clinical reaction time task appeared to be more consistent (ie, more uniform) than performance on the computerized reaction time task. The data support this finding in 2 ways. First, RT_clin measurements showed less variability than RT_comp measurements, as determined by group SDs for the measurements. Second, RT_comp was increased in those athletes whose CogState Sport performance did not pass the program's internal integrity check compared with those who had valid CogState Sport test sessions. In contrast, athletes with and athletes without valid CogState Sport test performance demonstrated nearly identical RT_clin measurements. One possible explanation for the more consistent performances on the clinical reaction time task is that this task is more intrinsically motivating than the computerized task and, therefore, athletes were more uniformly motivated during RT_clin testing than during RT_comp testing. This possibility is consistent with the subjective impression of the study investigators, who noted that the athletes were clearly aware of the distance the clinical reaction time apparatus fell on each trial and became competitive with their previous performances during testing. These observations contrasted with those of the athletes' demeanors while taking the computerized test battery, in that they quietly performed the tasks without showing any outward signs of motivation to perform at their best. Moreover, in a nonathletic population (N  =  31), RT_clin was more motivating than a computerized measure of RT, although that computerized measure was not associated with CogState Sport.²³

This clinical test of reaction time has potential clinical utility and may prove to be a valuable tool for athletic trainers and team physicians. It could be used in conjunction with a standard neurologic examination; standardized symptom checklists, such as the Standardized Assessment of Concussion^24,25 or the Sport Concussion Assessment Tool²⁶; or an assessment of postural stability, such as the Balance Error Scoring System,^27,28 to improve the sports medicine practitioner's evaluation of athletes suspected of having sustained a concussion. The RT_clin is not intended to serve as a replacement for more rigorous computerized neuropsychological test batteries for those athletes who have access to such evaluation techniques. However, when combined with sound clinical judgment, RT_clin may be of value in monitoring recovery from a concussion in younger athletes who play for teams that do not have access to the more expensive computerized test batteries. Further study is warranted to investigate these possibilities.

This pilot study has several limitations. It was designed only to compare reaction times measured using the clinical reaction time apparatus with those measured via CogState Sport. Also, this study was not designed to assess the athletes' motivation. Further research should be conducted in athletic populations before firm conclusions are drawn regarding the role of motivation on baseline clinical reaction time assessment. In addition, we investigated only collegiate male football players. The study should be replicated in a more diverse group of athletes before generalizations to other athletic populations are made. Another limitation is that 2 of the 3 administrators of the clinical reaction time test were study authors; therefore, they were aware of the study hypothesis and were not blinded. It is unclear how this may have influenced the athletes' performance on such a rapid task, particularly given that the evaluators were not aware of the athletes' CogState Sport test results at the time of RT_clin determination, but bias is possible. A further limitation is that all athletes were tested only once, including those athletes whose CogState Sport performance did not pass the program's internal integrity checks. Straume-Naesheim et al²⁹ reported a slight learning effect for CogState Sport and advocated a double-baseline test protocol to eliminate this effect. Our data do, in fact, indicate a learning effect for the simple reaction time component of CogState Sport. Also, although our pilot reliability and validity RT_clin testing did not identify a significant learning effect,^19,20 a learning effect did appear to be present for RT_clin. It would be valuable to determine the relationship between RT_clin and RT_comp after testing both with a double-baseline protocol, including obtaining RT_comp values during subsequent valid CogState Sport test sessions for those athletes whose initial performance failed the integrity check process.

In summary, RT_clin appears be a valid means of measuring baseline reaction time in a population of male collegiate football players and can be tested in an entire team as part of a routine preparticipation physical examination session. The athletes tested appeared to be intrinsically motivated by the clinical reaction time task, and this may have led to the observed improved consistency in RT_clin measures as compared with RT_comp measures. The more consistent baseline values would be useful when making comparisons with postinjury values after a concussion. Further study is needed to determine whether RT_clin is sensitive to the known effects of concussion on RT^7–13 and whether this test method is feasible for concussed athletes on the sideline or in the training room immediately after an injury.