Between-Seasons Test-Retest Reliability of Clinically Measured Reaction Time in National Collegiate Athletic Association Division I Athletes

Abstract

Context:

Reaction time is typically impaired after concussion. A clinical test of reaction time (RT_clin) that does not require a computer to administer may be a valuable tool to assist in concussion diagnosis and management.

Objective:

To determine the test-retest reliability of RT_clin measured over successive seasons in competitive collegiate athletes and to compare these results with a computerized measure of reaction time (RT_comp).

Design:

Case series with repeated measures.

Setting:

Preparticipation physical examinations for the football, women's soccer, and wrestling teams at a single university.

Patients or Other Participants:

102 National Collegiate Athletic Association Division I athletes.

Interventions :

The RT_clin was measured using a measuring stick embedded in a weighted rubber disk that was released and caught as quickly as possible. The RT_comp was measured using the simple reaction time component of CogState Sport.

Main Outcome Measure(s):

Data were collected at 2 time points, 1 season apart, during preparticipation physical examinations. Outcomes were mean simple RT_clin and RT_comp.

Results:

The intraclass correlation coefficient estimates from season 1 to season 2 were 0.645 for RT_clin (n = 102, entire sample) and 0.512 for RT_comp (n = 62 athletes who had 2 consecutive valid baseline CogState Sport test sessions).

Conclusions:

The test-retest reliability of RT_clin over consecutive seasons compared favorably with that of a concurrently tested computerized measure of reaction time and with literature-based estimates of computerized reaction time measures. This finding supports the potential use of RT_clin as part of a multifaceted concussion assessment battery. Further prospective study is warranted.

Key Points.

• Reaction time is typically prolonged after sport-related concussion and can be measured clinically.

• Test-retest reliability of the clinical measure of reaction time over consecutive seasons compared favorably with a computerized measure of reaction time.

• The potential use of the clinical measure of reaction time as part of the sports medicine practitioner's multifaceted concussion assessment battery is supported.

Concussion is a common and potentially serious injury that frequently results from sport participation. Recent management guidelines^1,2 emphasize a multifaceted approach to concussion assessment with the goal of determining that an athlete has fully recovered before returning to play. Full recovery from a concussion includes subjective symptom resolution and complete normalization of the physical examination, including postural stability and cognitive assessment. Over the past 2 decades, the use of computerized concussion assessment batteries such as the Immediate Post-Concussion Assessment and Cognitive Test (ImPACT; ImPACT Applications, Inc, Pittsburgh, PA),³ CogState Sport (CogState Ltd, Melbourne, Australia),⁴ and the Automated Neuropsychological Assessment Metrics (ANAM; Defense and Veterans Brain Injury Center, Washington, DC)⁵ has grown substantially to assist sports medicine professionals in assessing a concussed athlete's cognitive recovery. One measure that is included in all widely used computerized concussion assessment batteries is reaction time.

Impaired reaction time after sport-related concussion has been demonstrated repeatedly.^6–10 Reaction time measures provide one of the most sensitive indices of cognitive changes after concussion in both athletic and general head injury populations.^11,12 Decreased speed of information processing is thought to account for the cognitive performance deficits seen after concussion.¹¹ Impaired reaction time after concussion parallels the persistence of postconcussive symptoms^7,9 and may persist beyond resolution of self-reported symptoms and clinical findings.⁸ Reaction time assessment can add sensitivity to the sports medicine practitioner's concussion assessment battery. However, the measurement of reaction time relies on specialized computer programs, which limits its accessibility to many athletes, especially younger athletes, who are traditionally underserved by the sports medicine community as compared with athletes at the collegiate and professional levels. This point is especially concerning given that people aged 5–18 years account for 65% of all sport-related concussions.¹³

To provide clinicians with a reaction time measure that does not rely on computers, we¹⁴ developed a simple clinical method for measuring reaction time (RT_clin). Pilot reliability and validity studies¹⁴ in a small sample of healthy adult volunteers demonstrated excellent test-retest and intertester reliabilities for this technique and conformity to known reaction time characteristics, such as slowing with age and dual task, as well as correlation with a computerized reaction time measure. We¹⁵ have also demonstrated a significant positive correlation between RT_clin and the simple reaction time component of the CogState Sport computerized neuropsychological test battery in a sample of collegiate football players during preparticipation physical examinations. Furthermore, we¹⁶ have shown RT_clin to strongly correlate with a test participant's ability to protect the head using the hands during a laboratory task designed to simulate a sport-specific protective response. In ongoing research, we¹⁷ are investigating the effect of concussion on RT_clin in collegiate athletes using a postinjury comparison with baseline design.

It is critical to determine the stability of any measure intended to be used serially, as when comparing postinjury results with a preseason baseline. Only a small body of literature describes the test-retest reliability over various time intervals of currently available computerized sport concussion assessment batteries, each including a computerized measure of reaction time. Generally speaking, such studies have demonstrated the greatest stability over short test-retest intervals, on the order of 1–2 weeks, and decreasing reliability over intervals up to 2 months. Season-to-season comparisons are rare, despite these being common retest intervals in practice.

In pilot work, we¹⁴ reported a test-retest intraclass correlation coefficient (ICC) of 0.860 (P = .004) for RT_clin in a sample of 10 healthy adult volunteers tested by a single examiner on 2 occasions within 1 month. However, little can be inferred from these data with respect to the test's long-term stability in athletes given the small sample size, the lack of inclusion of athletes, and the short between-tests time interval. Therefore, the purpose of our current study was to measure the 1-year test-retest reliability of RT_clin in a larger sample of National Collegiate Athletic Association Division I athletes. To accomplish this, we compared year 1 and year 2 mean RT_clin values in athletes who completed preseason RT_clin testing over 2 consecutive seasons. This testing was performed as part of an ongoing concussion monitoring program. A secondary goal of the study was to compare the stability of RT_clin over seasons with that of a concurrently administered computerized measure of reaction time (RT_comp), namely the simple reaction time component of the CogState Sport test battery.

METHODS

Study Participants

We recruited student-athletes from the football, women's soccer, and wrestling teams from a single university during their preparticipation physical examination sessions. Recruitment of football players began before the 2007–2008 season, whereas recruitment of wrestlers and soccer players began the next year. Recruitment continued through the 2009–2010 season for all 3 teams. Before testing, all student-athletes provided informed written consent. This research was approved by the University of Michigan Institutional Review Board. All members of the football, women's soccer, and wrestling teams who were at least 18 years of age at the time of recruitment were eligible to participate. Athletes were excluded if they were recovering from a concussion or had an acute upper extremity injury affecting their ability to complete the clinical reaction time task at the time of testing. The order of RT_clin and RT_comp testing was determined by convenience during the preparticipation examination and was not strictly controlled or counterbalanced.

Measurement of RT_clin

Clinical reaction time is determined using a simple, manual visuomotor task: the time needed to catch a suspended vertical shaft by hand closure. The device is equipped with a weighted spacer at the lower end to ensure near verticality and standardize finger closure distance. Testing took place in an isolated room when one was available and otherwise in the corner of a larger room where the athletes waited as they moved between components of the preparticipation examinations. The RT_clin test protocol used has been previously described.^14,15 In brief, the athlete sat with the forearm resting on a horizontal desk or table surface with the hand positioned at the edge of the surface. The athlete held the hand sufficiently open to fit around, but not touch, the weighted disk portion of the clinical reaction time apparatus. The examiner suspended the apparatus and released it after randomly determined time delays between 2 and 5 seconds so as to minimize the athlete's ability to anticipate release of the device. Upon release, the athlete caught the device as rapidly as possible by hand closure (Figure). Clinical reaction time was calculated from the fall distance of the device using the formula for a body falling under the influence of gravity (d = 1/2gt²), with fall distance measured from the most superior aspect of the athlete's hand after catching the device. Anticipatory grasps and “drop” trials were excluded, as previously described.¹⁵ Each athlete performed 2 practice trials, immediately followed by 8 data acquisition trials.

Figure.

The clinical measure of reaction time (RT_clin) testing procedure and apparatus.

Measurement of RT_comp

During the same preparticipation physical examination session, each athlete completed a single baseline CogState Sport (version 5.6.4) computerized neuropsychological test session. Testing was completed as previously described,¹⁵ in maximum groups of 8 athletes seated at separate personal computers in a computer laboratory supervised by physicians familiar with the program. Each athlete wore noise-blocking headphones. The simple reaction time component of the test battery involves depressing the “k” key as rapidly as possible when a playing card presented on the computer screen turns face up. The raw simple reaction time data for all nonanticipatory trials (all trials that were not preceded by a keystroke before the card turned face up) were extracted for analysis. The RT_comp data were included for all athletes who had 2 valid CogState

Statistical Analyses

Means and SDs of RT_clin and RT_comp were calculated for each athlete during each test session. Test-retest reliability for RT_clin between the first and second test sessions was characterized by ICC (2,8), determined by a 2-way random-effects analysis-of-variance model, with a corresponding ICC (2,35) for RT_comp.^18,19 Test-retest ICCs are interpreted from 0 to 1, with a value of zero representing no, or random, consistency and a value of 1 representing perfect consistency between test sessions.²⁰ In general, higher ICC values indicate less error variance and better test-retest reliability.²¹ Standard error of measurement (SEM) values²² were also calculated for RT_clin and RT_comp at year 1 and year 2.

We compared RT_clin directly with RT_comp across athletes at year 1 and year 2 using paired t tests. To investigate a systematic change from year 1 to year 2 that would suggest a learning effect, we compared mean RT_clin and RT_comp values within each athlete over the 2 test sessions using paired t tests. The RT_clin was also analyzed after dividing the overall population into athletes who had 2 valid CogState Sport baseline test sessions and those who did not in order to determine whether RT_clin differences were present between these subgroups. Independent-samples t tests were used to compare mean RT_clin between groups. All statistical analyses were conducted using SPSS for Windows (version 16.0; SPSS Inc, Chicago, IL).

RESULTS

Of 251 student-athletes who participated in the study, 102 athletes completed baseline RT_clin test sessions during 2 consecutive seasons and were included in the data analysis. Of these, 78 played football, 14 wrestled, and 10 played women's soccer. Sixty-two of these athletes had 2 corresponding valid RT_comp data points and were included in the RT_comp analysis. Of the 40 athletes excluded from RT_comp analysis, 27 had at least 1 invalid CogState Sport session and 13 had missing CogState Sport data from athlete identifier coding errors.

The overall mean RT_clin and RT_comp values and their SDs, as well as ICCs and SEMs for RT_clin and RT_comp, are presented in Table 1. The RT_clin had a higher overall test-retest ICC than did RT_comp, and the SEM was lower for RT_clin than for RT_comp. In athletes with valid CogState Sport data for comparison, mean RT_clin was 50 milliseconds shorter than mean RT_comp in year 1 (t = −12.01, P < .001) and 62 milliseconds shorter in year 2 (t = −12.02, P < .001). A learning effect occurred in RT_clin (year 1 to year 2: 11 milliseconds, t = 4.66, P < .001), whereas RT_comp did not differ between the years (t = −0.16, P = .875). The RT_clin did not differ between athletes with and without valid RT_comp data for comparison in year 1 (205 ± 22 milliseconds versus 209 ± 25 milliseconds, respectively; t = −0.894, P = .373) or year 2 (194 ± 23 milliseconds versus 198 ± 25 milliseconds, respectively; t = −0.834, P = .406). Furthermore, the ICCs for RT_clin in athletes with and those without 2 valid CogState Sport computerized test sessions (ICC = 0.548 and ICC = 0.746, respectively) both fall within the overall 95% confidence interval for RT_clin (0.422, 0.775).

Table 1.

Overall Group Results for Clinical and Computerized Measures of Reaction Time

Measure	n	Mean ± SD, ms		Standard Error of Measurementa		Intraclass Correlation Coefficient	95% Confidence Interval for Intraclass Correlation Coefficient
		Year 1	Year 2	Year 1	Year 2
Clinical measure of reaction time	102	207 ± 23	196 ± 24	16.6	16.6	0.645	0.422, 0.775
Computerized measure of reaction time	62	255 ± 29	256 ± 34	23.5	27.5	0.512	0.186, 0.707

^aCalculated as SD * .

DISCUSSION

The results of this study demonstrate that RT_clin has sufficient long-term stability, as measured by its test-retest reliability over successive seasons, to be used serially as a concussion assessment tool for preseason baseline to postinjury comparisons. The 1-year reliability estimate for RT_clin (ICC = 0.645) compares favorably with that of concurrently measured RT_comp. Although the ICC for RT_clin obtained in this study is somewhat lower than ICCs in the literature for computerized reaction time measures over retest intervals of 1 hour to 2 weeks,^3,11,12 it is comparable with or greater than published values for time intervals ranging from 45 days to 2 years.^21,23 Table 2 summarizes literature-based reliability estimates for the reaction time components of ImPACT, CogState Sport, Concussion Resolution Index (Head-Minder, Inc, New York, NY), and Concussion Sentinel (Cog-State Ltd). Overall, the 1-year stability of RT_clin falls within the range of values reported for computerized reaction time measures that are currently in widespread use.

Table 2.

Summary of Studies Reporting Test-Retest Reliability for Computerized Reaction Time Measures

Study	Sample Tested	Computerized Test Battery Used	Retest Interval	Reliability
Iverson et al3 2005	56 healthy adolescents and young adults	ImPACTa	1–13 d	Pearson correlation coefficient r = 0.79
Collie et al12 2003	60 healthy young adults	CogState Sportb	1 h1 wk	ICC = 0.90ICC = 0.76
Erlanger et al11 2001c	High school, collegiate, and adult club athletes	Concussion Resolution Indexd	2 wk	0.73 (collegiate and adult club athletes) 0.72 (high school students)
Broglio et al21 2007	118 college students	ImPACT	0–45 d45–50 d	ICC = 0.39ICC = 0.51
		Concussion Sentinele	0–45 d45–50 d	ICC = 0.60ICC = 0.55
		Concussion Resolution Index	0–45 d 45–50 d	ICC = 0.65ICC = 0.36
Schatz23 (2010)	95 collegiate athletes	ImPACT	1.9 ± 0.6 y	ICC = 0.676

Abbreviation: ICC, intraclass correlation coefficient.

^a ImPACT Applications, Inc, Pittsburgh, PA.

^b CogState Ltd, Melbourne, Australia.

^c The number of athletes tested in each group and the reliability index used were not reported.

^d HeadMinder, Inc, New York, NY.

^e CogState Ltd.

The level of test-retest agreement thought to be acceptable for clinical use varies among authors. The 1-year RT_clin ICC falls above the minimum acceptable ICC value of 0.60 proposed by Anastasi²⁴ but below the 0.90 ICC value cited by Randolph et al²⁵ as desirable for making decisions about individual change in the context of sport-related concussion. In reality, it is rare for a clinical test to achieve test-retest reliability greater than 0.90, and none of the currently available and widely used computerized concussion batteries have yielded ICCs for reaction time or any other test measure that approach this level for re-test intervals longer than 1 hour. Although reliability estimates of ICC > 0.90 would be ideal, this appears to be an unrealistic benchmark for a test assessing something as complex as the brain's processing speed over prolonged time periods. Tests with lower reliability indices, as demonstrated for RT_clin and the numerous computerized reaction time measures detailed here, can still be of clinical value.

An important contributor to a test's reliability is the learning effect associated with repeated administration of the test. Even though the pilot RT_clin reliability and validity study did not suggest a learning effect in nonathletes,¹⁴ a learning effect over 8 RT_clin trials appeared to be present during our initial baseline assessment¹⁵ of RT_clin in 94 collegiate football players. The present study further suggests the possibility of a learning effect in athletes by demonstrating an average RT_clin decrease of 11 milliseconds from year 1 to year 2. The test protocol for determining RT_clin in athletes has been consistent across studies, with participants given 2 practice trials followed by 8 data acquisition trials. Investigation into the effect of more practice trials before data collection is warranted.

It is noteworthy that almost one-third of the athletes who participated in this study had one or more invalid CogState Sport test sessions. Although a variety of reasons for an invalid computerized test session exist, poor effort or motivation on the part of the athlete is probably one of the most common. The finding that RT_clin did not differ between athletes with and without valid RT_comp data for comparison parallels our prior findings¹⁵ and suggests that RT_clin may be immune to this concern. Furthermore, RT_clin may have actually been more stable across seasons in the athletes without valid RT_comp data for comparison, that is, primarily athletes excluded from RT_comp analysis due to invalid CogState Sport sessions. A possible explanation is that RT_clin may be more intrinsically motivating than computerized reaction time tasks. This could be because of several factors, including the more physical nature of the RT_clin task and the direct one-on-one interaction inherent in RT_clin testing that is absent from computerized test batteries. Less than full effort and motivation on cognitive test performance has a potent influence on outcome, as demonstrated by Green et al,²⁶ who showed that suboptimal effort suppressed overall test battery performance 4.5 times more than did moderate to severe brain injury. Ensuring optimal effort is critical when preseason baseline tests are the basis for comparison with postinjury testing, when an athlete is often highly motivated to perform well so as to return to play. Therefore, an intrinsically motivating test is likely to provide higher-quality baseline test results. Additional study is necessary to specifically address the role of motivation in baseline reaction time testing.

The strength of the conclusions in this study must be tempered by the limitations. We investigated only Division I athletes, 76% (78/102) of whom were football players and 90% (92/102) of whom were male. Therefore, the results are applicable primarily to this population and may not generalize to female athletes or athletes participating at high school or youth levels. We would not expect test-retest reliability to differ in these populations, but additional study in females and younger populations is warranted. This is especially true given that the simplicity and anticipated low cost of RT_clin may make it most valuable in younger athletes who do not have access to computerized neuropsychological test batteries. To our knowledge, Erlanger et al¹¹ provided the only test-retest reliability estimates for computerized concussion batteries that allowed comparison of athletes across age groups. The reliability estimates reported do not differ greatly between high school students and collegiate or adult club athletes on processing speed (0.79 and 0.90, respectively), simple reaction time (0.72 and 0.73, respectively), or complex reaction time (0.65 and 0.72, respectively). In addition, the results may not apply to the shorter time periods that more closely approximate the typical timeframe seen in sport-related concussion assessment and management.²¹ An additional study limitation is the lack of counterbalancing in the order of RT_clin and RT_comp testing. This lack of control was pragmatic because of the logistics of testing large numbers of athletes during their preparticipation physical examinations. We do not hypothesize any order effect, but the true effect of test order is unknown. A final limitation of this study is that the RT_comp data used for comparison with RT_clin were not of optimal quality. All athletes completed only a single CogState Sport test session each season, and 40 athletes had at least 1 missing or invalid CogState Sport test session. With regard to the 13 athletes with missing RT_comp data, a systematic problem in the way athlete identifiers were coded within the CogState system in the first year of testing, which was subsequently fixed, is to blame. Regarding the large number of invalid CogState Sport test sessions with unusable data, authors of future prospective studies should consider a double-baseline CogState Sport test protocol to reduce the learning effect associated with the test and improve data quality.²⁷ Although it would not affect the primary outcome variable of this study (RT_clin stability), this change in protocol may allow a better comparison of RT_clin with RT_comp.

In summary, the test-retest reliability of RT_clin over consecutive seasons compares favorably with computerized reaction time measures. This suggests that it is a stable measure across seasons and, taken in the context of our previous work,^14–17 supports its potential use as part of the sports medicine practitioner's multifaceted concussion assessment battery. We caution that impaired simple reaction time is only one of many typical signs indicating that an athlete has sustained a concussion. Therefore, it must be interpreted within the greater clinical context of the concussed athlete. Furthermore, in athletes who have access to computerized neuropsychological testing, RT_clin should not be considered a replacement for computerized tests, which measure multiple indices of concussion in addition to simple reaction time. However, the simplicity and low cost of RT_clin could facilitate its use in youth athletes and others who do not have access to computerized concussion assessment programs. In athletes who do have access to computerized testing, RT_clin may serve a complementary role as a true sideline tool in the initial concussion diagnosis, when use of computerized test batteries is impractical. Further study is warranted to examine the influence of motivation on baseline reaction time assessment, evaluate the use of RT_clin in younger populations, and prospectively investigate the effect of concussion on RT_clin. Controlled research involving concussed athletes will also need to define a clinically meaningful change in RT_clin from baseline on which management decisions can be based.