The Effect of Varied Test Instructions on Neuropsychological Performance following Mild Traumatic Brain Injury: An Investigation of “Diagnosis Threat”

Abstract

Diagnosis threat is a psychosocial factor that has been proposed to contribute to poor outcomes following mild traumatic brain injury (mTBI). This threat is thought to impair the cognitive test performance of individuals with mTBI because of negative injury stereotypes. University students (N=45, 62.2% female) with a history of mTBI were randomly allocated to a diagnosis threat (DT; n=15), reduced threat (DT-reduced; n=15), or neutral (n=15) group. The reduced threat condition invoked a positive stereotype (i.e., that people with mTBI can perform well on cognitive tests). All participants were given neutral instructions before they completed baseline tests of objective cognitive function across a number of domains, psychological symptoms, and PCS symptoms, including self-reported cognitive and emotional difficulties. Participants then received either neutral, DT, or DT-reduced instructions before repeating the tests. Results were analyzed using separate mixed model analysis of variances (ANOVAs); one for each dependent measure. The only significant result was for the 2 × 3 ANOVA on an objective test of attention/working memory, Digit Span (p<0.05), such that the DT-reduced group performed better than the other groups, which were not different from each other. Although not consistent with predictions or earlier DT studies, the absence of group differences on most tests fits with several recent DT findings. The results of this study suggest that it is timely to reconsider the role of DT as a unique contributor to poor mTBI outcome.

Introduction

The debilitating nature of post-concussion syndrome (PCS) is well documented.¹ However, the debate surrounding the etiology and maintenance of poor outcomes following mild traumatic brain injury (mTBI) continues.² This controversy is centered on whether the pattern of persisting symptoms is the result of direct changes to the neural substrate or the result of interacting neurological and psychological factors.³ A biopsychosocial model has been offered to capture the complex nature of poor mTBI outcome, which can include PCS.⁴

The biopsychosocial model of poor mTBI outcomes includes several social psychological factors, one of which is diagnosis threat (DT).⁴ Several studies have investigated other social psychological factors in this model (e.g., good-old-days bias; expectation as etiology), but DT is arguably not as well researched as some of these other factors.^5,6 DT may warrant further investigation because this factor is one that clinicians may be able to influence, unlike many of the other contributors to poor mTBI outcomes.^7–10

The term “diagnosis threat” was first published in a mTBI study more than 10 years ago.¹¹ This term was based on a broader and earlier established concept, “stereotype threat.”^12,13 The stereotype threat literature has shown that threats based on racial¹⁴ and gender stereotypes can change cognitive test performance.¹⁵ A wide range of other outcomes have also been found to be susceptible to stereotype threats, including psychophysiological measures,^16,17 cerebral activation assessed using functional magnetic resonance imaging,¹⁸ and automatic performance tasks.¹⁹ While some of the effects on cognitive tests are selective rather than global,¹⁴ taken together, these findings suggest good support for the stereotype threat effect. Although the same level of support for mTBI DT has not yet been demonstrated given its relationship to stereotype threat, it is possible that DT effects might be similarly potent.

In the mTBI context, DT is said to occur when a mTBI patient performs poorly on tests of cognitive functioning because he/she is made aware before testing that such patients generally perform such tasks poorly.¹¹ In some tests of the stereotype threat it is also made clear to examinees that their test results will be compared to the stereotype reference group.^14,20 However, even without the latter instruction, in two very influential papers Suhr and Gunstad clearly demonstrated an mTBI DT effect.^11,21 These seminal studies demonstrated the mTBI DT effect across a range of cognitive tasks in a sample of university students with a self-reported history of mTBI.^11,21 In both studies, the DT group demonstrated significantly worse performance on cognitive tests, compared with those given neutral instructions.^11,21 While Suhr and Gunstad only found changes on tests of general intellect and memory,¹¹ in a subsequent study they extended these findings to tests of attention, working memory, psychomotor speed and memory.²¹ Of clinical importance, these studies demonstrated that approximately 37.5% and 46% of the DT groups had at least one impaired score, compared with only 6.6% and 12% of those given neutral instructions.^11,21

Since Suhr and Gunstad's most recent study,²¹ there have been a handful of mTBI DT studies; however, this research has been unable to replicate the earlier cognitive test findings.^22–24 Several methodological differences between the Suhr and Gunstad study and subsequent studies have been suggested to account for these inconsistent findings. For example, Kinkela delivered the stereotyped message through a negatively framed video about mTBI,²² as opposed to the written instructions used in other studies.^21–24 This difference is unlikely to account for the failure to replicate findings because even with the same instruction format, contrary findings occur.²⁴ Another possible explanation for the inconsistent mTBI DT findings might be sample differences. For example, Kit did not find an effect using a community dwelling sample with mild and moderate TBI,²³ whereas Suhr and Gunstad did find an effect using a mild head injury sample.^11,21 This factor, too, is unlikely to explain the inconsistent effects; even when injury severity is matched across studies, findings are mixed.^11,24 A third explanation might be variation in the detail of the instructions used to induce the threat (e.g., whether it includes comparison with a reference group or not);^11,21,23 yet, even with the same instructions, the effect has not been robust.^23,24 A related, and to the best of our knowledge, not previously articulated possible explanation is that the groups with whom participants are asked to identify are different; for example, some threat instructions require participants to identify with the stereotype of a head injured²⁴ or concussed individual;²¹ yet, the negative expectations associated with these terms have been shown to be significantly different.^25–27

While it remains difficult to identify the reasons for these discrepancies in the DT literature, this research could be advanced by further studies that extend the test of this concept and address the methodological limitations of past studies. For example, none of the past DT studies have controlled for pre-existing injury expectations (or stereotypes), yet this factor is the subject of the experimental manipulation. Although several studies have used random group allocation which could control this variable, it is possible that conflicting results could be due to a failure to ensure adequately matched groups. A study with a within-subjects design that would minimize the effects of this and other individual differences has not yet been attempted.

Further, as suggested previously,²¹ it may be possible to extend the test of the DT effect by exploring whether threat-minimizing instructions are effective.²¹ For example, the broader stereotype threat literature recommends applying a “mitigating factors” approach whereby the threat is reduced through education and test performance is not impaired.^7,8 Whether or not this approach would work in the mTBI DT context has yet to be demonstrated. A second test of the limits of the DT effect is to determine whether threats influence outcomes other than cognitive performance. Suhr and Gunstad included affective measures, but did not find a DT effect.²¹ The study by Ozen and Fernades did show a mTBI DT effect on some measures of subjective cognitive complaints but this recent finding requires replication.²⁴ It is also possible that novel outcomes could be explored, such as the effect of DT on PCS symptom report. Given that lay people attribute a range of negative characteristics to people with mTBI beyond cognitive difficulties,²⁵ effects beyond these outcomes would be expected.

The primary aim of this study was to determine whether a broad range of outcomes would be different for participants exposed to instructions aimed at inducing or reducing a DT. A neutral instruction condition also was included as a control, and three outcomes were assessed: cognitive performance, affective functioning, and PCS symptom report. In contrast to previous research,^23,24 and in an attempt to provide better control over individual differences, the current study employed a within-subjects design. That is, we compared participants' baseline performance to their performance post-instruction (i.e., neutral, DT, or DT-reduced). Based on previous findings,^11,21,24 it was hypothesized that those exposed to the negatively stereotyped instructions (DT) would perform worse on tests of cognitive functioning and report more psychological distress and PCS symptoms than those exposed to the DT-reduced or neutral instructions and, that those exposed to the DT-reduced or neutral instructions would perform similarly on the three outcome measures. In line with Suhr and Gunstad,^11,21 it was further hypothesized that for the DT group there would be an increase in the percentage of participants performing at a level of clinical impairment on at least one outcome after receiving DT instructions, whereas this percentage was not expected to change after DT-reduced or neutral instructions.

Method

Participants

Seven hundred nine undergraduate students completed an online pre-screening questionnaire to determine their eligibility to participate. Ninety-one of these individuals passed this screening test, which included several inclusion and exclusion criteria (Table 1). Approximately half of those who passed this pre-screen (n=45) agreed to proceed to a one hour session involving face to face testing (mean age [M_age]=24.08 years; standard deviation [SD]=5.56; 62.2% female). There were no significant differences between those participants who met eligibility criteria and did or did not undertake face to face testing. [(Group comparisons of age, gender and education were conducted to assess for sampling bias between those participants who participated (n=45) in the cognitive testing and those who did not (n=46). Analysis of variance and Chi-square analyses revealed no significant main effects for age (p=0.443; continuing=24.81 years [standard deviation (SD)=5.92], drop out=23.50 years [SD=8.70]); gender (p=731; female: continuing=62.2%, drop out=58.7%); or education (p=0.381; secondary school: continuing=60.0%, drop out=69.6%).] Continuing participants were randomly allocated to one of three conditions (neutral, DT, or DT-reduced; n=15 per condition). The demographic characteristics of the sample by group are shown in Table 2. There were no significant differences between the three groups on key demographic variables (Table 2).

Table 1.

Study Inclusion and Exclusion Criteria

	Details
Inclusion criteria mTBI	Defined as a forceful blow to the head or any acceleration or deceleration force (i.e. whiplash) that resulted in one of the following: confusion or disorientation, loss of consciousness (<30 minutes), posttraumatic amnesia not exceeding 24 hours, and/or other temporary neurological abnormalities (i.e. intracranial lesion not needing surgery).
English language proficiency	Assumed on the basis of current enrolment at an English language speaking university.
Passed effort test	Passed Test of Memory and Malingering (TOMM)
Exclusion criteria
Recent injury	In the past three months
History of psychological or neurological disorder	Diagnosed or receiving treatment in the past twelve months
Previous neuropsychological assessment	In the past twelve months

mTBI (mild traumatic brain injury) was defined according to The Mild Traumatic Brain Injury Committee of the Head Injury Interdisciplinary Special Interest Group of the American Congress of Rehabilitation Medicine.⁵⁷ Apart from the TOMM, responses were based on self-report. Participants achieved a score of 45 or greater on the TOMM Trial 2 and the Retention Trial.⁵⁰

Table 2.

Participant Demographic Characteristics in Each Experimental Condition and the Statistical Significance of Cross-Condition Comparisons

	Experimental condition
	Neutral (n=15)	DT (n=15)	DT-reduced (n=15)	p
Age (in years)
M	22.00	26.73	23.53	0.060
SD	3.00	7.04	5.11
Gender
Male	33.3%	40.0%	40.0%	0.910
Female	66.7%	60.0%	60.0%
Time since mTBI (in years)
M	2.52	3.76	5.88	0.079
SD	2.83	3.24	5.21
Years of Education (Completed)
Secondary School	73.3%	46.7%	60.0%	0.331
Bachelor Degree	13.3%	20.0%	20.0%
Diploma	13.3%	26.7%	6.7%
Masters	0.00%	0.00%	13.3%
PhD	0.00%	6.7%	0.00%
Injury cause
Sport-related	66.7	66.7	40.0	0.161
Non sport-related	33.3	33.3	60.0
Pre-threat (IPQ-R subscales)
Consequences
M	2.03	1.78	1.72	0.373
SD	0.57	0.73	0.62
Recovery timeline
M	2.09	2.10	1.74	0.276
SD	0.63	0.83	0.55
Effort (TOMM) Trial 2 score
M	50.00	50.00	49.93	0.165
SD	0.00	0.00	0.26
Understood test instructions (% passing)	-	46.67%	66.67%	0.269

N=45.

Cross-condition comparisons were performed using ANOVA tests for continuous variables and Chi-square tests for categorical tests. Significance evaluated at p=0.05 (2-tailed). Pre-induced threat (or participants understanding of the injury they had experienced) was assessed using the Illness Perception Questionnaire-Revised, IPQ-R⁴⁹, timeline and consequences subscales. The 6-item subscales are rated on a 5 point Likert scale, ranging from 1 (strongly disagree) to 5 (strongly agree); higher scores indicate more negative illness perceptions. Effort was assessed using the Test of Memory and Malingering (TOMM⁵⁰) and comprehension of instructions was assessed by determining whether the participants had read and understood the experimental instructions (i.e., summarize the instructions (qualitative; plus did you respond as per the instructions (yes/no).

DT, diagnosis threat; DT-reduced, diagnosis threat reduced; M, mean; SD, standard deviation; mTBI, mild traumatic brain injury; IPQ-R, Illness Perception Questionnaire-Revised.

Instruction sets and pilot test

The instructions for this study included a) neutral instructions, which were devoid of diagnostic terminology, did not convey any expectation of test performance, and were of the type that might be used as a general introduction to testing; b) DT instructions, which were closely modeled on previous research;²⁴ and, c) DT-reduced instructions, which were developed for this study to try to emphasize individual control over group effects. The neutral instructions were as follows:

The following tests are designed to assess cognitive functioning skills such as working memory, attention, information processing etc., and emotional health through tests of psychological functioning. While you may find some tests to be quite easy, there will be others that will be more difficult. Please give your best effort. You will be given a break in between tests.

The DT and DT-reduced instructions had the same introductory and concluding statement as each other, with variation introduced in between these parts. These instructions provided different advice about a) whether cognitive test performance following mTBI returns to normal; b) whether individuals can use compensatory strategies, such as increased effort, to assist their performance post injury; and c) whether test performance is subject to individual control. These instructions referenced mTBI, as opposed to other diagnoses because this term is regarded as more “alarming” to patients and their families than other diagnostic terms.²⁵

Because of the possibility that the difference in the findings in the DT literature could be due to instruction variation, the absence of a precedent for DT-reduced instructions, and our desire to match these instructions on secondary parameters, a pilot process was used to refine the DT and DT-reduced instructions and match these instructions on their perceived effectiveness. This piloting process involved 100 participants (84% female) between ages 17 and 70 years (M_age=31.59 years; SD=11.77) drawn from those individuals who were ineligible to continue from the initial recruitment pool. All participants first read the DT instructions and then responded to four statements using a 4-point Likert Scale (1=strongly disagree to 4=strongly agree; Table 3). All participants were then required to read the DT-reduced instructions and then respond to another four statements on a 4-point Likert scale (Table 3). The statements were used to gauge the likely effectiveness of the instructions (e.g., DT-reduced item: “Based on these instructions I expect that an individual's performance on tests of cognitive functioning would not be influenced by their previous mild traumatic brain injury”). A pre-determined arbitrary criterion of agreement by at least 70% of the sample was used to indicate that no modification of the instructions was necessary. On average, 82% (range 81–85%) and 80% (range 71–93%) of the pilot sample that evaluated the DT or DT-reduced instructions agreed or strongly agreed with the statements, respectively; therefore, no changes to the instructions were made. The instructions for DT and DT-reduced conditions are presented in Table 4.

Table 3.

Questions Used in the Pilot Phase to Develop Experimental DT and DT-Reduced Instructions. These Prompts Were Presented after the following Stem: “Based on These Instructions I Would Expect That…”

Instruction type
DT-reduced	Diagnosis threat
…individuals will not experience symptoms of mild traumatic brain injury long after the initial injury.	…individuals will experience symptoms of mild traumatic brain injury long after the initial injury.
…an individual's performance on tests of cognitive functioning will not be influenced by their previous mild traumatic brain injury.	…an individual's performance on tests of cognitive functioning will be influenced by their previously sustained mild traumatic brain injury.
…individuals who have sustained a mild traumatic brain injury have control over their performance on tests of cognitive functioning, such as working memory, attention and processing speed.	…individuals who has sustained a prior mild traumatic brain injury have no control over their performance on tests of cognitive functioning, such as working memory, attention and processing speed.*
…individuals who have sustained a mild traumatic brain injury can perform to the level of individuals who do not have a history of mild traumatic brain injury of on tests of cognitive functioning.	…individuals who have sustained a mild traumatic brain injury cannot perform to the level of individuals who do not have a history of mild traumatic brain injury on tests of cognitive functioning.*

Note: All participants provided responses to these items using a 4 point likert scale, where 1=strongly disagree and 4=strongly agree. ^*=All items achieved a mean rating of 3 or greater apart from these two items.

DT, diagnosis threat.

Table 4.

Instructions Used for Experimental (DT or DT-reduced) Groups. The Introduction and Concluding Part of the Threat and Benefit Instructions Was Common across These Two Conditions

	Instruction type
	Diagnosis threat*	DT-reduced
Introduction	This study entitled ‘Cognitive and Affective Functioning in Adults’ is actually assessing performance of cognitive functioning skills such as working memory, attention and information processing in individuals who have experienced a mild traumatic brain injury in their past (at least 3 months ago) that was a result of any contact forces (i.e. hit or fall) or acceleration/deceleration trauma (i.e. vehicle accident). You were asked to participate further in this study due to your history of sustaining a prior mild traumatic brain injury.
Middle	Past research has shown that individuals who have had a mild traumatic brain injury do not perform as well as individuals who have not had a mild traumatic brain injury on tests of memory, and attention, even though the injury was sustained quite some time ago. Research has also shown that memory and attention abilities are not under the personal control of the individual. Therefore it is suggested that cognitive functioning is permanently affected, as a result of the mild traumatic brain injury, and despite considerable effort those with mTBI will not outperform people who have not sustained a mild traumatic brain injury.	Past research has shown that individuals who have had a mild traumatic brain injury typically fully recover within a couple of months post-injury, and in turn return to normal cognitive functioning. Following this period of recovery individuals perform just as well as individuals who have not had a mild traumatic brain injury on tests of memory, and attention, and can even outperform such individuals. Research has also shown that memory and attention abilities are under personal control of the individual. Therefore it is suggested that cognitive functioning is not influenced as a result of the mild traumatic brain injury, and memory and attention abilities can improve with effort.
Conclusion	It is the aim of the present study to confirm the above findings. As such you will be required to complete the battery of tests again. Please give your best effort. You will be given a break in between tests.

^* The DT instructions for this study were based on those used by Ozen and Fernandes.²⁴ Underlined words were changed from Ozen and Fernandes²⁴ for this study. The DT and DT-reduced instructions used the same introductory and concluding statements, but the central part of these instructions varied according to condition.

DT, diagnosis threat.

Measures

Cognitive performance measures

The following cognitive tests were used because they are recommended for mTBI research.²⁸

Scanning and executive functioning

The Trail Making Test (TMT) consists of two parts.²⁹ Part A assesses scanning and motor speed skills, and Part B assesses higher order cognitive functions, such as mental flexibility.³⁰ Individuals are required to connect circles in ascending order as quickly as possible from (1–25) in Part A, and in a number-letter sequence (1, A, 2, B etc) in Part B.²⁹ The TMT is susceptible to practice effects; however, alternate forms have demonstrated strong test–retest reliabilities.^31,32

Immediate memory and learning

In the Rey Auditory Verbal Learning Test (RAVLT), individuals are read a list of fifteen unrelated words five times, and at the end of each reading they recall as many of the list words as they can, in any order.^33,34 Although susceptible to practice effects, alternate versions have strong test-retest reliability over a one month interval.³⁵

Processing speed

The Symbol Coding and Symbol Search subtests of the Wechsler Intelligence Scale-III (WAIS-III) are timed tests with a maximum response time of 120 seconds.³⁶ For the Symbol Coding subtest, individuals fill in a grid of blank boxes with symbols, as they correspond to the numbers provided in a legend. In the Symbol Search subtest, individuals identify whether target symbols are present in a group of symbols by responding yes or no. Strong test-retest reliability has been shown for both subtests; however, they are prone to increases of approximately 2.5 to 8.3 points when retested.³⁷

Verbal fluency

The Controlled Oral Word Association Test (COWAT) requires individuals to produce as many words as possible in a one minute period that begin with a specified letter.^38,39 Standard administration with the letters F, A and S was used, along with the alternate version R, W and T.³⁴ The COWAT has demonstrated strong internal consistency and test–retest reliability with the use of alternate forms.⁴⁰

Attention and working memory

The Digit Span subtest of the WAIS-III has two parts; Digit Span Forwards and Digit Span Backwards.³⁶ The examiner reads out a sequence of numbers. In Digit Span Forwards the individual is required to repeat the numbers in a forward sequence, and in Digit Span Backwards the numbers are repeated in a backward sequence.⁴¹ The subtest has shown strong test-retest reliability and minimal practice effects.⁴²

Affective functioning

The Brief Symptom Inventory 18 (BSI-18) is an 18-item self-report questionnaire that examines current levels of psychological symptomatology on a 5-point Likert scale, ranging from not at all distressed (0) to extremely distressed (4).⁴³ Three six-item dimensions of general distress, somatization (e.g., faintness or dizziness), depression (e.g., thoughts of ending your life), and anxiety (e.g., nervousness or shakiness inside) are assessed, and totalling the subscale scores gives the Global Severity Index (GSI).⁴⁴ The BSI-18 demonstrates strong test-retest reliability and internal consistency.⁴⁵

Self-reported PCS symptoms

The Neurobehavioural Symptom Inventory (NSI) assesses PCS symptoms and is a recommended supplemental outcome measure for brain injury research.^28,46 The NSI has 22-items that assess sensory, somatic/physical, cognitive, and affective/psychological symptoms, respectively. This study embedded the NSI within a larger measure, which modified the original 5-point rating scale that measured the presence/severity of each symptom to instead measure the extent of disturbance that the symptoms caused in the previous two weeks ranging from 1 (not at all) to 5 (extremely). Total scores were calculated by summing across all 22 items, and cluster scores were calculated as per Kennedy and colleagues.⁴⁷ Higher scores on the NSI represent greater symptomatology. A cut-score of 17.5, described by King, was used to indicate clinical significance of total scores.⁴⁸ Previous studies indicate that the NSI has good psychometric properties.²⁸

Other measures

Three additional measures were administered to control for pre-induced expectations of mTBI, suboptimal effort and comprehension of instructions. Pre-induced mTBI expectations. The timeline and consequences subscales of the Illness Perception Questionnaire Revised (IPQ-R) were administered to examine participants' pre-existing perceptions of mTBI.⁴⁹ A minor modification was implemented to ensure IPQ-R items were applicable to mTBI (e.g., “my mTBI [instead of illness] will last a long time”). High IPQ-R scores reflect strongly held beliefs about the chronic nature and negative consequences of mTBI.

Effort

The Test of Memory Malingering (TOMM) was used to control for response bias.⁵⁰ The TOMM has shown strong diagnostic accuracy in identifying individuals who are giving suboptimal effort (i.e., false positive rate 2%),⁵¹ and is not influenced by level of affective functioning, age, education, or cognitive dysfunction.⁵² A score of 45 or less on Trial 2 and the Retention Trial was used as the criterion to identify suboptimal effort.⁵⁰

Post-experimental questionnaire

A post-experimental questionnaire was used to assess whether the participants had read and understood the experimental instructions. Participants were instructed to respond honestly to this questionnaire and completed the following items: “Acted in a way that was consistent with the instructions provided” (Response: Yes or No) and “In your own words, briefly explain what the second set of instructions informed you” (Open ended response; coded by one rater as correct or incorrect).

Procedure

Prior to data collection, randomization to condition was achieved with the assistance of a person who did not conduct the tests. A random numbers table was used to determine to which condition participants would be assigned and which test order they would receive. As participants enrolled, the examiner selected the pre-determined test package for that participant. To achieve a single blind design, the experimental instructions were included in the kits, conveyed in writing, and not disclosed to the examiner.

The face-to-face testing consisted of two parts. In part 1, all participants completed an assessment of baseline cognitive performance, affective functioning and PCS symptoms, unaware that the study was about mTBI. In part 2, which occurred approximately five minutes after the baseline cognitive assessment, participants were exposed to the instructions (i.e., neutral, DT, or DT-reduced: see Instruction Sets and Pilot Test) and then again completed an assessment of cognitive performance (using alternate forms where available), affective functioning, and PCS symptoms. While previous studies administered the battery of tests in a set order,²⁴ we employed partial counterbalancing using a Latin Squares method to control for order effects and ensure that when alternate forms were available these were balanced within and across conditions. The RAVLT and TOMM were always administered first and second respectively because these tests require time between the initial and retention trial. The IPQ-R and post-experimental questionnaire were administered after part 2 testing to assess for pre-induced mTBI expectations and whether participants in the DT-reduced and DT groups remembered and behaved in accordance with their instructions, respectively.

Testing was conducted during the mid-year university holiday break and in the first few weeks of semester to ensure that any group differences were not due to the stress associated with the end of semester assessment. At the end of testing, participants were debriefed. This debriefing included explicit instruction that, in general, individuals do not experience long-term effects of mTBI.⁵³ Participants either received course credit or were entered into a draw to win a $100 Coles/Meyer gift voucher.

Results

Preliminary analyses

The data were analyzed using SPSS version 18. The data were screened for entry errors, missing values and breaches of assumptions. There were no missing data recorded. All assumptions were met unless otherwise stated. An alpha level of 0.05 was applied to determine statistical significance, unless otherwise specified.

Due to the study's small sample size, Levene's test homogeneity of variance was breached on a number of measures. As recommended by Brown and Forsythe, the Brown-Forsythe Test of Equality of Means was used as a more robust measure for small sample sizes.⁵⁴ Preliminary analyses were undertaken to test for order effects and determine if randomization was effective. Group comparisons using Chi-square revealed no significant differences due to test order or version (test order, p=0.928; e.g., digit span subtest first: neutral=37.5%, DT-reduced=25.0%, DT=37.5%; and, test version, p=0.537; Alternate version first: neutral=26.1%, DT-reduced=34.8%, DT=39.1%). Further, group comparisons (using ANOVA) revealed no significant differences in baseline performance, with the exception of Symbol Coding. Due to the heterogeneity of variance on the Symbol Coding subtest the Kruskal-Wallis test was used as the non-parametric equivalent to a one-way ANOVA, and revealed significant differences in performance on this measure across the three groups, (H(2)=9.26; p=0.01). Using a Bonferonni adjusted alpha (p<0.0167), Mann-Whitney tests showed that while participants in the DT-reduced (median [Mdn]=45.0) and DT (Mdn=48.0) groups had similar baseline scores (U=102.5; z=−0.416; p=0.677), participants who received the neutral instructions (Mdn=71.0) performed significantly better than those who received the DT-reduced (U=48.0, z=−2.679, p=0.007), or DT instructions (U=51.5, z=−2.534, p=0.011).

Influence of mTBI diagnosis threat instructions on cognitive performance, affective functioning, and PCS symptom complaints

Descriptive statistics for all measures by group for each testing session are shown in Table 5. On visual inspection of these data it appeared that compared to baseline, performance on most measures improved following DT-reduced instructions, whereas the DT instructions did not change (or decrease) performance. Minimal practice effects were shown in the neutral instruction condition. All three groups reported similarly low levels of psychological distress and PCS symptoms before and after instructions.

Table 5.

Performance on CDE-Recommended Neuropsychological Tests for the Three Experimental Groups across Time

	Neutral M(SD)		DT-reduced M(SD)		Diagnosis Threat M(SD)
Neuropsychological tests by cognitive domain	Baseline	Post-instructions	Baseline	Post-instructions	Baseline	Post-instructions	F	p	ηp2
Memory
RAVLT (Total Trials I-V)	54.80 (9.09)	54.73 (10.58)	50.40 (5.12)	54.13 (11.35)	49.33 (6.92)	50.13 (9.26)	0.84	.438	.039
RAVLT Retention	23.47 (4.76)	23.20 (5.02)	21.73 (4.79)	22.80 (6.09)	21.47 (4.76)	20.27 (5.66)	0.91	.411	.041
Psychomotor Speed
Trail Making Test A	18.81 (3.86)	16.07 (4.49)	16.58 (2.67)	16.15 (9.74)	19.62 (7.21)	18.04 (5.28)	0.40	.141	.019
Symbol Coding1	31.36 (13.96)	26.23 (12.82)	18.03 (11.31)	22.70 (12.14)	19.60 (10.21)	20.07 (14.43)	2.09	.136	.091
Symbol Search	42.27 (8.58)	45.00 (12.52)	40.20 (9.64)	45.53 (10.90)	37.87 (11.59)	39.47 (11.27)	1.84	.072	.080
Attention/Working Memory
Digit Span subtest*	18.93 (4.37)	20.27 (3.99)	18.73 (3.01)	21.07 (4.33)	18.60 (3.85)	18.60 (4.08)	3.71	.033	.150
Executive Functioning
Trail Making Test B	37.09 (9.23)	31.87 (11.35)	35.38 (13.13)	29.43 (7.29)	34.57 (6.55)	39.81 (18.15)	2.85	.069	.119
Verbal Fluency
COWAT	38.73 (10.49)	39.60 (11.86)	37.87 (11.59)	39.47 (11.27)	38.6 (7.05)	43.80 (7.12)	2.44	.099	.104
Affective Functioning
BSI-18 Global Index Severity	25.93 (8.04)	25.00 (7.59)	28.33 (13.23)	27.80 (13.63)	22.33 (4.56)	22.27 (6.96)	.270	.765	.013
Subjective complaints
NSI-Somatic	1.48 (0.44)	1.36 (0.42)	1.56 (0.66)	1.44 (0.65)	1.33 (0.38)	1.31 (0.38)	0.79	.459	.036
NSI-Cognitive	1.59 (0.48)	1.59 (0.53)	1.67 (0.63)	1.64 (0.70)	1.45 (0.39)	1.51 (0.51)	0.48	.623	.022
NSI-Affective	2.00 (0.80)	1.87 (0.65)	1.97 (0.84)	1.82 (0.86)	1.73 (0.64)	1.63 (0.63)	0.09	.912	.004
NSI-Sensory	2.05 (0.66)	1.86 (0.59)	2.09 (.95)	1.91 (0.94)	1.98 (0.73)	1.71 (0.61)	0.43	.652	.020
NSI-Total	17.13 (11.17)	14.53 (9.76)	18.20 (15.39)	15.47 (16.29)	14.07 (10.75)	11.93 (10.67)	.158	.855	.007

N=45, n=15 per condition.

η_p²=partial eta-squared. ^*p<.05 (two-tailed).

¹Due to the heterogeneity of variance on this measure a non-parametric mixed model ANOVA was run, via the creation of ranked variables.⁵⁷ Therefore means and standard deviations on this measure are reflective of ranked terms not the original test scale. Tests are grouped into cognitive domains as per Wilde et al.²⁸

CDE, common data elements; M, mean; SD, standard deviation; DT-reduced, diagnosis threat reduced; RAVLT, Rey Auditory Verbal Learning Test; COWAT, Controlled Oral Word Association Test; BSI-18 GSI, Brief Symptom Inventory 18 Global Severity Index; NSI, Neurobehavioural Symptom Inventory.

A series of 2×3 mixed model ANOVAs was conducted to assess whether the instruction type affected performance over time. For each of these analyses, there was a) one independent between groups variable, Instruction, with three levels (neutral, DT, or DT-reduced); and, b) one independent within groups variable, Time, with two levels (pre and post instructions). The dependent variable for these analyses was a score from one of three outcome types: cognitive test performance, with functioning assessed across several domains; PCS symptoms; and, affective functioning. Table 5 displays the results of these analyses, most of which did not reveal statistically significant effects.

There was one exception to this trend in the 2×3 ANOVA results. Digit Span yielded a significant Time x Instruction interaction, F(2,42)=3.711; p=.033, η_p²=0.150; see Table 5. Using a Bonferroni adjusted alpha (p=0.0167), follow up one-way repeated measures ANOVAs revealed that Digit Span performance improved significantly over time in one of the three groups (DT-reduced, F(1,14)=10.269; p=.006; d=0.63; 95% CI [18.67, 23.47]). The neutral (F([1,14]=6.087, p=.027, d=0.32, 95% CI [18.06, 22.48]) and DT groups, (F[1,1]=0.000, p=1.00, d=0.00, 95% CI [16.34, 20.86]) did not change significantly over time.

Clinical significance of findings

An analysis of pre and post scores on cognitive tests by a clinical standard was also conducted in line with previous research.²¹ Note that we intended to extend this approach to other outcomes; however, none of the participants were above the clinical cut-off for the BSI-18 Global Severity Index or the NSI-Total score at baseline or follow up. For the cognitive tasks, performance was re-coded as either unimpaired or clinically impaired (i.e., scores that were more than 1.5 standard deviations below the mean based on the published norms).⁵⁵

A series of McNemar's tests was conducted to assess the effect of instructions on the number of impaired performances before and after instructions. The results of these analyses indicated that there was no statistically significant difference before and after instructions in the percentage of the sample with at least one cognitive score in the clinically impaired performance range for any group. Specifically, the percentage of the sample with at least one score in the clinically impaired range was as follows: Neutral group, before (33.33%) and after (33.33%) instructions (χ²[4]=1.00, p=0.930); DT-reduced group, before (80%) and after (80%) instructions (χ²([4]=1.00, p=0.910); and, DT group, before (60%) and after (53.33%) instructions (χ²[4]=3.00, p=0.558).

Discussion

The primary purpose of this study was to investigate whether DT instructions would influence performance on cognitive, affective and PCS symptom report measures in people with a history of mTBI. Despite the extensive evidence supporting the notion of stereotype threat,^12,56 the findings for mTBI DT have been less conclusive. Further, the idea of testing some theoretically plausible effects was pursued in this study by using instructions that should mitigate the DT effect (DT-reduced) and testing a broad range of outcomes. Overall, the results of the study did not support the first hypothesis. That is, we were unable to show that performance on any outcome was adversely impacted by DT instructions, relative to other conditions. The second hypothesis, that the DT-reduced group would perform like controls, was supported but in the absence of a DT effect, this finding is not meaningful. Only on Digit Span did some of the predicted group differences emerge (ie. DT-reduced performed better than the DT group); however, even on this test a diagnosis threat, per se, was not observed.

While these findings for cognitive test outcomes are not consistent with the hypotheses, these findings are consistent with the most recent DT studies^22–24 and are similar to those of a recent racial stereotype threat study, where only one of several cognitive domains (memory, assessed using a composite cross modal score) succumbed to a significant but small threat effect.¹⁴ Whereas Suhr and Gunstad demonstrated a mTBI DT effect across a number of cognitive domains (visuo-spatial ability, verbal comprehension, psychomotor speed, attention and working memory),^11,21 such a generalized effect has yet to be replicated. Further, while both Suhr and Gunstad and Ozen and Fernandes studied outcomes additional to cognitive performance (i.e., cognitive or affective complaints, respectively),^21,24 our subjective test results for both measures of this type, and a third novel outcome, PCS symptoms, did not show a threat effect. Our findings on those subjective outcomes that have been studied previously are consistent with the previous literature.^21,24

This study also failed to find “indirect” support for the concept of mTBI DT by testing for theoretically plausible effects across a range of outcomes and using instructions that might mitigate negative effects. Unlike Ozen and Fernandes, who found that the DT increased cognitive complaints,²⁴ we did not find that the DT group reported significantly more PCS or psychological distress symptoms relative to participants in the DT-reduced or neutral groups. Consistent with Suhr and Gunstad, we did not find a DT effect on affective complaints;²¹ however, this finding itself is arguably inconsistent with the broader stereotype threat literature in which affective function has been shown to be susceptible to threat effects.⁸

There are several possible explanations for the discrepancy between findings across studies, including the idea that the threat was not sufficiently activated in our study. Only 46.67% and 66.67% of the experimental groups remembered the DT and DT-reduced instructions, respectively, but this difference was not significant across these conditions (χ²(1)=1.22; p=0.269). Whether the participants in other DT studies recalled their instructions cannot be determined because this study was the first to employ a post-experimental check of this type; however, in the broader stereotype threat literature, a similar post-experimental instruction recall percentage was noted,¹⁴ suggesting that our result may not be atypical. It is also possible that the wording of our instructions did not sufficiently activate the threat because they did not include a specific instruction that the examinee's performance would be compared to those of the diagnostic group. Peer comparisons have been suggested as an important component of stereotype threat instructions;¹⁴ however, mTBI DT studies have typically not included this element (for an exception see Ozen and Fernandes)²⁴ and our DT instructions were almost identical to those used by Suhr and Gunstad.^11,21 A third possibility is that the threat effect was not sufficiently activated because of the sample characteristics. Participants in this study were, on average, four years post-injury. Although the influence of stereotypes is expected to be elicited in individuals regardless of the time since injury, the threat may have been more personally relevant to the participant if the injury had occurred recently.¹² In our study, the gap between injury and testing may have allowed participants the opportunity to challenge the stereotype, especially given that they were undergraduate students. Suhr and Gunstad have suggested that university students are less likely to strongly identify themselves as brain injured because they may not have strong feelings of inferiority about their test performance.¹¹ A further possibility is that the threat was not sufficiently activated because the majority of our participants sustained their injury via a mechanism that is publicly perceived as less negative (i.e., sport-concussion rather than motor vehicle accident-mTBI).²⁶ Our sample consisted of more than half of sport-related injuries. An athlete's prior experience of concussion may be in contrast to the stereotypes that we were attempting to elicit.²³

In addition to the possibility that this study did not sufficiently activate mTBI stereotypes, this study has a number of limitations. For example, the neutral instructions were not representative of the full instructions that would be used in routine clinical practice. Second, this study used a sample with a history of self-reported mTBI; we did not verify injury details. Third, our sample size was very small, a consideration that may be at least partly offset by our use of a within-subjects design.

As noted previously, the aim of this study was to focus on a factor that has been regarded as one of several biopsychosocial contributors to the significant and chronic distress and disability that some individuals experience post mTBI.⁴ However, taken together with the majority of other DT studies and given the direction of most of the current study's results, it may be timely to question whether this factor should remain in the model as a separate or specific contributor to poor outcome. We raise this question tentatively because we also point out that the mTBI DT studies are perhaps analogues to simulator malingering studies given their experimental nature, the samples used, and the assessment context (i.e., at the university), and it is possible that patients who are being assessed for genuine diagnostic purposes are impacted by a range of additional cues from the clinical environment. For example, if the assessment occurs in a hospital setting where the examinee has already accepted a patient identity, and other factors from the mTBI outcome model are at play such as the good-old-days bias, these conditions might increase susceptibility to DT effects; a question that this study does not address. Future research efforts may be needed to further explore this effect, in particular focusing on whether variables that have not yet been measured in mTBI DT studies such as suggestibility, or by studying this effect in combination with other factors from the biopsychosocial model to refine our understanding of its potential as a unique contributor to this conceptual model.⁴

Acknowledgments

The Human Research Ethics Committee of Queensland University of Technology (QUT- QUT-HREC #1200000165) approved this research. This project was granted an occupational workplace health and safety clearance. Funding for this project was provided by the School of Psychology and Counselling, Queensland University of Technology. The authors acknowledge Chloe Kempe for assistance with randomization and cross scoring, Sherrie-Anne Kaye for advice on the pilot study, and Dr. April Thames whose correspondence with us influenced our writing of this paper.

Author Disclosure Statement

No competing financial interests exist.