Discriminant Construct Validity of ImPACT™: A Companion Study

Abstract

In our previous analysis of ImPACT™ scores relative to traditional neuropsychological tests (NP) and experimental tasks, we demonstrated good convergent construct validity for the primary ImPACT™ test-score composites. Adequate discriminant validity was also reported, but complete analysis was not undertaken. Here, test scores from the 54 collegiate football and hockey players were re-analyzed to specifically address the discriminant validity of the I ImPACT™ composite scores using a multiply operationalized correlation matrix of multi-trait multi-method data. By combining the different trait scores obtained with the same method, it is possible to assess the degree of shared method variance that serves as error in score interpretation. Discriminant validity is determined by non-significant correlations between a target composite and average of the other traits. Results showed that the ImPACT™ Verbal Memory (p = .002), Visual Memory (p = .017) and Visual Motor Speed (p = .010) were highly correlated with composites of the other scores, while the Reaction Time composite demonstrated adequate discriminant validity (p = .411). In comparison, all of the NP composites showed good discrimination (all p-values >.05, except for Reaction Time p = .05). Thus, three of four ImPACT composite scores were not sufficiently distinct to support specific construct-oriented interpretations.

Introduction

Campbell and Fiske (1959) recommended that construct validity demonstrate both high correlations with tests of supposed similar constructs and low correlations with tests from which it should differ. These processes were described as ‘convergent’ and ‘discriminant’ validity, respectively. Convergent validity is thus a type of construct validity that examines the degree to which the operationalization of specific constructs (i.e., test scores) are similar to (converge on) test scores that they would be expected to be related to. In contrast, discriminant validity demonstrates that different or unique traits do not correlate with each other. Campbell and Fiske (1959) went on to articulate the process of multi-trait multi-method analysis of construct validity, by which constructs of interest (traits) are measured by multiple means (methods). In this way, better construct specification can be obtained while controlling for shared method variance.

Discriminant validity is an index of difference. That is, an index of the difference between the test/construct of interest and some other test/construct. In this approach, the constructs that are assessed within each method are compared (correlated) with the mean of the other constructs to show how different the target construct is from other constructs (Figure 1). For example, if a visual memory tests correlates highly with a verbal memory test, there is not good disciminant validity as it is expected that these tests assess different underlying constructs. In the multiple-operationalization, the visual memory test is correlated with the mean of verbal memory, processing speed and reaction time tests. In this example, the verbal memory test’s usefulness for specific identification of visual memory deficits is called into question. Thus, while convergent validity seeks high correlations among similar constructs, discriminant validity expects low correlations between dissimilar constructs.

Figure 1.

Hetero-Trait Mono-Method Multiple Operationalization

In our previous analysis of ImPACT™ relative to paper and pencil neuropsychological tests and experimental tasks, we demonstrated good convergent validity for the primary ImPACT™ composite scores (Maerlender, Flashman, Kessler et al, 2010). We noted the likely presence of adequate discriminant validity based on score patterns, without more formal analysis. Here use a multiply-operationalized procedure for analyzing the correlation matrix of z-transformed scores to demonstrate which ImPACT™ tests are significantly different from the others as an indicator of discriminant validity and construct specificity (Cole, Howard & Maxwell, 1981).

By combining multiple traits that were assessed using a single method, it is possible to compare (for instance) the ImPACT™ Verbal Memory composite to the mean of the verbal memory scores from the other methods (paper and pencil and experimental measure).

Procedure

The same data as considered in Maerlender et al 2010 was analyzed. Fifty-four collegiate football players were administered preseason ImPACT™ computerized tests and a battery of more traditional neuropsychological (paper and pencil: NP) tests. The test batteries appear in Table 1.

Table 1.

Composition of domain scores for ImPACT, neuropsychological tests (NP) and experimental tests (adapted from Maerlender et al 2010)

	ImPACT Composite Scores	NP Composite Scores	Experimental Composite Scores
Verbal Memory	Word Memory test (% correct), Symbol Match (hidden symbols scores), Three Letters (total correct).	CVLT (trials 1–5 total; long delay total recognition discriminability).	Verbal Continuous Memory Task (number correct, long delay)
Visual Memory	X’s and 0’s (total correct memory), Design Memory (total % correct.	BVMT-R (trial 1, total learning trials 1–3, delayed recall).
Reaction Time	X’s and 0’s (average correct RT), Symbol Match (average correct RT), Color Match (average correct RT).	CPT (Simple Reaction Time, Vigilance & Distractibility average reaction times).	N-Back (mean reaction times 0-back), Verbal Continuous Memory (mean reaction times, new condition).
Visual Motor Speed/ Processing Speed	X’s and 0’s (total correct interference score), Three Letters (average counted correctly).	DKEFS Trail Making (sum of trials 1–3 and 5), DKEFS Verbal Fluency (sum of Category & Letter Fluency), DKEFS Color Word Interference Test (sum of 4 conditions).	N-Back (sum of reaction times of the 1-, 2- and 3-back).

BVMT-R = Brief Visual Memory Test, Revised (Benedict, 1997); CVLT = California Verbal learning test, 2^nd Ed (Delis, Kramer, Kaplan, & Ober, 2000); Gordon, 1986; DKEFS = Delis Kaplan Executive Function System (Delis, Kaplan, & Kramer, 2001); PASAT = Paced Auditory Serial Attention Test, Gronwall, 1977.

To determine discriminant validity, multi-trait mono-method score combinations were used. First, for each ImPACT™, paper-pencil and experimental composite score generated, a z-score was calculated based on the sample’s score distribution (N = 54¹). Then, combinations of the scores were created so that each composite z-score could be correlated with the averaged linear combination of the other composites. For instance, the z-score for ImPACT™ Verbal Memory composite was correlated with the average of the z-scores for the ImPACT™ Visual Memory, ImPACT™ Visual Motor Speed (PS for processing speed), and ImPACT™ Reaction Time. The result is a multiply operationalized discriminant validity coefficient for ImPACT™ Verbal Memory.

For example:

• z1 r ((z2+z3+z4)/3)

where z1 is the z-score for the Verbal Memory composite, z2 is the z-score for the Visual Memory composite, z3 is the z-score for the Visual Motor Speed (processing speed), and z4 is the z-score for the Reaction Time composite.

After computing the multiply-operationalized correlations, the effect of anxiety was considered. In our previous study we found that state anxiety correlated significantly with the ImPACT™ Visual Motor Speed composite. Thus, accounting for that systematic variance was seen as important. The NP multi-operationalized correlations were also calculated to serve as a comparison. Anxiety was not related to NP composites. The means and standard deviations for the composites appear in Table 2.

Table 2.

Means and standard deviations (s.d.) of transformed composite scores

Z-scores	N	Mean	s.d.
ImPACT Verbal Memory	54	−0.013	1.005
ImPACT Visual Memory	54	−0.029	0.986
ImPACT Visual Motor
Speed	54	0.001	1.009
ImPACT Reaction Time	54	−0.013	1.005
NP Verbal Memory	54	0.004	0.856
NP Visual Memory	33	0.094	0.770
NP Processing Speed	54	0.164	0.517
NP Reaction Time	54	0.305	0.638

Results

These data show that in the current sample, three of the four ImPACT™ composites shared significant method variance with each other (Visual Memory, Verbal Memory and Visual Motor Speed), while Reaction Time did not (p<.05: see Table 3). State anxiety accounted for about 10% of the variance in Visual Motor Speed scores (β = −.286, R² =.10, p = .02).

Table 3.

Point-biserial correlations (p-values) of multiply-operationalized ImPACT and NP composite scores (multi-trait, mono-method)

	VrMem v Others	VsMem v Others	PS v Others	RT v Others
ImPACT*
(N – 54)	.423 (.002)	.328 (.017)	.354 (.010)	.117 (.411)
NP (N = 33)	.243 (.172)	.212 (.237)	.200 (.264)	.346 (.048)

^*partial correlation adjusting for state anxiety

The NP correlations all showed good disciminability, as expected, with the exception of Reaction Time. The Reaction Time correlation was marginally significant.

Discussion

This analysis further supports the findings from Maerlender et al, 2010 in which the specificity of the ImPACT™ composites was questioned. Using the same sample, we demonstrate that only the Reaction Time composite score of ImPACT™ demonstrated adequate construct discriminability, while other composite scores were too highly interrelated to provide meaningful information about specific cognitive functions, despite their labels (e.g., Verbal Memory).

In the NP sample, good discriminability was found for all constructs except Reaction Time. The trend towards significance in the NP Reaction Time score was unexpected and likely reflects the limited task selection.

These findings extend and compliment the convergent validity findings from our previous study (Maerlender, Flashman, Kessler, et al, 2010). Together, they provide a more complete picture of ImPACT™ construct validity at baseline. Although this is a limited sample, until proven otherwise, the use of ImPACT™ for clinical differentiation of specific neuropsychological problems is not warranted.