Letter to the Editor

Dear Editor,

The Journal of Athletic Training ( JAT) recently published an article entitled “Is Neuropsychological Testing Useful in the Management of Sport-Related Concussion?” ¹ This article was published as a literature review and purports to evaluate studies using neuropsychological testing programs in the management of sport-related concussion, including the ImPACT test battery. Unfortunately, the article does not present a balanced and representative review of the literature. Specifically, the authors have omitted articles that did not support their supposition that neuropsychological testing is of little value in evaluating sport-related concussion. This lack of inclusion of data appears to be particularly structured to criticize computer-based programs such as ImPACT. Although I do not agree with some of their assertions regarding more traditional “paper-and-pencil” testing, this response will focus specifically on ImPACT.

Regarding the scientific basis of ImPACT, there have now been 18 peer-reviewed articles in highly regarded journals, including the Journal of Neurosurgery, Archives of Clinical Neuropsychology, Neurosurgery, American Journal of Sports Medicine, Clinical Journal of Sport Medicine, Journal of the International Neuropsychological Society, and Clinical Neuropsychologist. In addition, approximately 30 published abstracts and numerous papers have been presented at national and international conferences. There is also extensive normative data on the Web site www.impacttest.com. All of this information has been readily available for public scrutiny on the Web site but was either selectively reviewed or completely ignored by the authors of the JAT article.

Below we address a few of the many erroneous assertions and conclusions drawn by the JAT article.

The authors of the article correctly emphasize the need for studies to evaluate both the reliability and validity of test procedures, yet they make broad statements regarding the requirements for a specific level of test reliability without acknowledging published studies that address reliability. ² The article by Iverson and his colleagues demonstrated minimal practice effects for athletes taking the test twice. In addition, reliable change index change scores have been published and available to aid in the clinical interpretation of ImPACT for several years. Curiously, even though the authors cite the Iverson et al article elsewhere in the JAT paper, they fail to cite this article when discussing the reliability of ImPACT. Instead they state, “we were unable to find any peer-reviewed paper reporting reliability data on ImPACT.”

The authors also attempt to establish standards for the reliability of neuropsychological tests. For instance, in the JAT article, they state that “a reliability of close to .90 is necessary for this purpose.” Much of the authors' research has employed the Standardized Assessment of Concussion (SAC). ³ The authors' own research using the SAC found a test-retest reliability of only .55. Furthermore, an independent study of the SAC found poor interrater reliability and statistically significant practice effects in non-concussed collegiate football players. ⁴ Despite what they would consider “unacceptable” reliability, the authors have widely touted the use of the SAC for making return-to-play decisions and have frequently stated that the SAC is both reliable and valid. ⁵ Therefore, the position taken in their critique of neuropsychological testing is inconsistent with their position in past publications and is difficult to comprehend.

The authors' assertion that there has been “only one peer-reviewed article involving a prospective controlled study using ImPACT” is again incorrect. Multiple studies have been completed that have demonstrated the ability of the ImPACT test battery to separate concussed from non-concussed athletes, even in relatively mildly injured athletes. The published statistical differences between concussed and age-matched control subjects were very robust, with a very large effect size of 1.21 being found for the memory composite score. ⁶ They also criticize the article based on our focus on the memory composite indicator of ImPACT. However, they fail to mention that the article was specifically designed to evaluate the potential relationship of on-field amnesia and memory recovery in athletes. Contrary to their criticism of the Journal of Neurosurgery article, this study supports the sensitivity of ImPACT, even when employed with mildly injured athletes. The ImPACT was able to detect significant differences over time even in a group of athletes who demonstrated only 5 minutes of mental status changes on the field.

Regarding the sensitivity and specificity of the ImPACT test battery, these have also been studied. Although a recent study ⁷ was not available at the time the JAT article was originally written, anyone attempting to conduct a fair scientific review could be reasonably expected to check for new research before publication of an article that makes such sweeping generalizations as in their paper. This paper was also previously presented as an abstract and was available on the ImPACT Web site. Schatz et al ⁷ found that even conducting a “blind” (eg, purely statistical) classification (discriminant function analysis), ImPACT correctly classified 82% of the concussed group and 89% of the non-concussed group.

Another position taken by the JAT article appears to be that current neuropsychological assessment approaches are of little value in the management of concussion and that decisions should be made primarily on the basis of player symptoms. In my opinion, this represents a naïve position and assumes that athletes are able or willing to accurately report their symptoms. On any given week during football season, we evaluate upwards of 80 concussed athletes in our clinic at the University of Pittsburgh Medical Center. Many of these athletes flatly deny symptoms, even when we are certain that they are still experiencing them (based on reports from parents, friends, and spouses). Presumably this occurs because of internal and external pressures to return to play. This is a very well-known phenomenon in sports medicine and is nothing new. We believe that returning these athletes to play based on self-report of symptoms alone is an extremely dangerous practice.

A major conclusion of the JAT article is that neuropsychological testing should only be undertaken when the athlete becomes symptom free. We could not disagree with this position more. Most of our athletes are high school students and, therefore, are frequently under pressure to return to the classroom, often within days of injury. Our neuropsychological test data provide useful and practical information to athletes, their parents, and teachers regarding the capacity of the injured student to perform school work, particularly during the early recovery process. Furthermore, we have recently completed a study ⁸ that clearly establishes the added value of the ImPACT test battery in separating concussed athletes from age-matched controls. In this study, the use of ImPACT led to a 26% better diagnostic yield compared with the evaluation of symptoms alone.

Finally, the recently published article in JAT concludes that neuropsychological testing should be interpreted conservatively and that return-to-play decisions should also take into account player symptoms. Although I agree with this statement, the authors appear to suggest that neuropsychologists frequently advocate an “either/or” approach to return-to-play decisions. This could not be farther from the truth. The authors of the ImPACT test battery, as well as the vast majority of other neuropsychologists, recognize that neuropsychology should never be used in and of itself in determining the overall status of a concussed athlete. The fact that the ImPACT test battery has always contained a symptom inventory would appear to emphasize this point. I have never suggested that neuropsychological testing be used without paying attention of other aspects of functioning, including self-reported symptoms. ^{8 9} In fact, both the NFL and NHL neuropsychology programs, which I oversee, are based on the premise that neuropsychology should never be used in a vacuum. ^{10 11}

I thank the Journal of Athletic Training for taking an active role in the education of athletic trainers. Athletic trainers are extremely important team members in the management of concussion. Unfortunately, the JAT article provides a very biased review of the literature and does nothing to move our understanding of concussion forward. In addition, the authors' opinions are inconsistent with current thought within the field of neuropsychology.