Correspondence of the Boston Assessment of Traumatic Brain Injury-Lifetime (BAT-L) Clinical Interview and the VA TBI Screen

Abstract

Objective

Mild traumatic brain injury is the signature injury of Operation Enduring Freedom (OEF), Operation Iraqi Freedom (OIF), and Operation New Dawn (OND), yet its identification and diagnosis is controversial and fraught with challenges.

Setting

In 2007, the Department of Veterans Affairs (VA) implemented a policy requiring traumatic brain injury (TBI) screening on all individuals returning from deployment in the OEF/OIF/OND theaters of operation that lead to the rapid and widespread use of the VA TBI screen. The Boston Assessment of TBI-Lifetime (BAT-L) is the first validated, postcombat semistructured clinical interview to characterize head injuries and diagnose TBIs throughout the life span, including prior to, during, and post–military service.

Participants

Community-dwelling convenience sample of 179 OEF/OIF/OND veterans.

Main measures

BAT-L, VA TBI screen.

Results

Based on BAT-L diagnosis of military TBI, the VA TBI screen demonstrated similar sensitivity (0.85) and specificity (0.82) when administered by research staff. When BAT-L diagnosis was compared with historical clinician-administered VA TBI screen in a subset of participants, sensitivity was reduced.

Conclusions

The specificity of the research-administered VA TBI screen was more than adequate. The sensitivity of the VA TBI screen, although relatively high, suggests that it does not oversample or “catch all” possible military TBIs. Traumatic brain injuries identified by the BAT-L, but not identified by the VA TBI screen, were predominantly noncombat military injuries. There is potential concern regarding the validity and reliability of the clinician administered VA TBI screen, as we found poor correspondence between it and the BAT-L, as well as low interrater reliability between the clinician-administered and research-administered screen.

Mild traumatic brain injury (mTBI) is the signature injury of Operation Enduring Freedom (OEF), Operation Iraqi Freedom (OIF), and Operation New Dawn (OND), yet its identification and diagnosis is controversial and fraught with challenges. Congress, the Department of Defense, and the Department of Veterans Affairs (VA) have called for urgent identification and treatment of TBI in this cohort. As a result, screening tools and diagnostic instruments were developed and applied rapidly.¹ We recently published the Boston Assessment of TBI-Lifetime (BAT-L), the first validated, postcombat semistructured clinical interview to characterize head injuries and diagnose TBIs throughout the life span including prior to, during, and post–military service.² We report here on the correspondence of TBI diagnosis from the BAT-L clinical interview with that of the VA 4-item TBI screen.

THE VA TBI SCREENING PROCESS

In 2007, the VA implemented a policy requiring TBI screening on all individuals returning from deployment in the OEF/OIF/OND theaters of operation (VHA Directive 2007–2013). In less than 2 years (May 2007 to February 2009), the VA TBI screened 281 607 OEF/OIF/OND veterans nationally, underscoring the rapid and widespread use of this tool. The VA TBI screen/Clinical Reminder is a computer-based screen designed to be more sensitive than specific and identify individuals at risk for having sustained TBI during deployment that are currently symptomatic and have not previously been identified. Individuals who screen positive are then referred to polytrauma clinics for the comprehensive TBI evaluation in order to determine the presence or absence of a military TBI. The VA TBI screen consists of questions asked to all OEF/OIF/OND veterans who present for care in any clinic in a VA facility, including primary care and specialty care clinics (see Table 1). It is critical to recognize that screening for possible TBI cannot replace the detailed TBI diagnostic process. The VA’s system for identifying individuals at risk of having sustained a TBI has frequently been confused with the actual procedures for documenting a history of a deployment TBI.

TABLE 1.

VA TBI screen^a

VA TBI screen Clinical Reminder

Did you have any injury(ies) during your deployment from any of the following? (Blast or explosion, vehicular accident, fragment wound above the shoulders, fall) Did any injury you received while deployed result in any of the following? (Losing consciousness, being dazed or confused, not remembering the event, concussion, head injury) Did any of these begin or get worse afterwards? (Memory problems, balance problems, sensitivity to bright light, irritability, headaches, sleep problems) In the past week, have you had any of the above symptoms? (The same symptoms as question 3 are queried).

^aA veteran must respond affirmatively to all 4 questions to produce a positive screen and be referred on for further “second-level” traumatic brain injury evaluation.

Preliminary examinations of the sensitivity and specificity of the 4 screening questions, when compared with clinician-diagnosed deployment-related TBI in the context of a full diagnostic interview, have resulted in inconsistent findings and continue to be investigated. Terrio and colleagues³ reported poorer sensitivity (60%) than specificity (96%). However, the sensitivity increased to 80%, with a slight decrease in specificity to 93%, for positive TBI screening when affirmative responses to only questions 1 and 2 were included. Donnelly and colleagues¹ observed that the 4 screening questions had better sensitivity than specificity (high sensitivity 94%, moderate specificity 59%). In a large database study utilizing the VA’s centralized database Patient Care Services on the TBI Clinical Reminder Screen and Comprehensive TBI Evaluation results of veterans (N = 48 175), Belanger and colleagues⁴ showed that TBI Clinical Reminder Screen has generally good sensitivity (0.87–0.90) but poor specificity (0.13–0.18). Sayer⁵ reported that as of April 2011, approximately 20% of OEF/OIF/OND veterans who received VA medical care since the TBI screen was implemented have screened positive for TBI, and of those, 55% went on to be diagnosed with a TBI after evaluation.

Documenting that the Department of Veterans Affairs traumatic brain injury (VA TBI) screening process is more sensitive than specific is critical, given that this is the primary method by which returning OEF/OIF/OND service members are identified to receive a TBI evaluation. Comparison of TBI diagnosis correspondence between the BAT-L with the VA TBI screen will add to our understanding of the utility of the VA TBI screening procedures.

METHODS

Participants

One hundred seventy-nine consecutive participants (158 men/21 women), who were deployed veterans of OEF/OIF/OND and enrolled in the VA Rehabilitation Research and Development–supported Traumatic Brain Injury Center of Excellence at VA Boston Healthcare System: The Translational Research Center for TBI and Stress Disorders (TRACTS), participated in this study. See Table 2 for demographics. Participants completed an extensive test battery including assessment of TBI, cognition, psychiatric status and posttraumatic stress disorder (PTSD), biologics, and neuroimaging.

TABLE 2.

Mean, standard deviation, and range are provided for the basic demographics of the sample^a

	Mean (SD)
Age, y	33 (8.25), range = 20–62
Education, y	13.9 (1.97)a, range = 12–20
Gender
Female	12%
Male	88%
Ethnicity
White	66.5%
African American	11.7%
Asian	2.8%
American Indian	0.6%
Hispanic	16.8%
Unknown	1.7%
Number of OEF/OIF/OND deployments	1.37 (0.66), range = 1–5
OEF/OIF/OND deployment duration, mo	14.6 (8.82), range = 3–56
Time since last OEF/OIF/OND deployment, mo	34.0 (27.3), range = 1–99

Abbreviations: OEF, Operation Enduring Freedom; OIF, Operation Iraqi Freedom; OND, Operation New Dawn.

^aThere were 7 service members who received a general education degree and were assigned 12 years of education.

VA TBI screen

The VA TBI screen was administered to all veterans in our sample by a member of the TRACTS research staff during the same evaluation in which the BAT-L was administered. Positive TBI screen on the VA TBI screen is defined in Table 1. In addition, TBI screens were obtained from veterans’ VA medical records for subset of our sample (n = 87). These included all veterans who were registered, local VA patients (computerized patient record system for VA Boston Healthcare System) and received a clinician-administered TBI screen as part of their clinical care. Note that the clinician-administered VA TBI screen may have been administered at various time points, including days, months, or even years apart from the BAT-L and research-administered VA TBI screen.

The Boston Assessment of TBI-Lifetime (BAT-L)

The BAT-L was administered to assess blast exposure, blast-related TBI, and other TBIs occurring during military service, as well as TBIs acquired throughout a veteran’s lifetime (including injuries both prior to and post–military service). The BAT-L is a comprehensive clinical interview process that includes the use of specific probes targeting the unique experiences of OEF/OIF/OND veterans to assess the physiological disruption of consciousness in the context of co-occurring traumatic events. Traumatic brain injury was assessed during 3 time epochs: (1) prior to military service (pre-military), (2) during active military training and duties (military: blast-related and other mechanism(s) during combat, training, or other activities during active duty), and (3) after returning stateside (postdeployment). The 3 most severe injuries in each epoch were evaluated. We were careful to query for falls, accidents (motor vehicle and other), training injuries, assault, and sports-related military injuries, as these are the most common mechanisms of injuries reported in addition to blast events. The BAT-L guidelines were followed to establish a time-line for altered mental status, posttraumatic amnesia, and loss of consciousness.

Rather than focusing on just the detection of TBI, the BAT-L establishes specific grading of severity of mild TBIs that are most common in this cohort. Traumatic brain injury severity was rated according to the Department of Defense criteria (see Table 3). Mild traumatic brain injury, or concussion, was further subdivided into grade I, II, or III injuries according to the BAT-L’s hybrid classification system (see Table 3). Diagnosis included review at a weekly TBI diagnostic consensus meeting consisting of at least 3 doctoral-level psychologists. To be comparable with the deployment-focused VA TBI screen, positive TBI screen on the BAT-L was defined as any military-related TBI (blast or nonblast etiology) involving altered mental status as well as those involving a loss of consciousness that occurred in theater. For a full description of the instrument and its reliability and validity, see the study by Fortier et al.²

TABLE 3.

Boston Assessment of TBI-Lifetime hybrid classification system for the diagnosis of mild traumatic brain injury into mild grade I, II, and III injuries^a and Department of Veterans Affairs and Department of Defense consensus criteria for traumatic brain injury severity as defined in the clinical practice guidelines: management of concussion—mild traumatic brain injury^b

Criteria	Mild			Moderate	Severe
	Grade I	Grade II	Grade III
Loss of consciousness	None	<5 min	>5 min and <30 min	>30 min and <24 h	>24 h
Alteration of mental status	0–15 min	>15 min and <24 h	>24 h	>24 h; severity based on other criteria
Posttraumatic amnesia	0–15 min	>15 min and <24 h	>24 h	>1 d and <7 d	>7 d
Glasgow Coma Scale	13–15	13–15	13–15	9–12	<9

^aAdapted from Bailes and Cantu.⁶

^bDepartment of Veterans Affairs and Department of Defense.⁷

RESULTS

All of the comparisons reported below reference the VA TBI screen relative to the BAT-L clinical interview as the criterion standard. Thus, as shown in Table 4, a false negative refers to an instance whereby the BAT-L determined that a TBI occurred but the VA TBI screen determined that a TBI did not occur. Likewise, a false negative refers to an instance whereby the BAT-L determined that a TBI did not occur but the VA TBI screen determined that a TBI did occur.

TABLE 4.

Identification of possible TBI according to VA TBI screen compared with BAT-L in 179 OEF/OIF/OND service members^a

VA TBI Screen	BAT-L for military TBI during OEF/OIF/OND deployment(s)
	Positive	Negative	Total
Positive	55 (true positive)	20 (false positive)	75
Negative	10 (false negative)	94 (true negative)	104
Total	65	114	179

Abbreviations: BAT-L, Boston Assessment of TBI-Lifetime; OEF, Operation Enduring Freedom; OIF, Operation Iraqi Freedom; OND, Operation New Dawn; TBI, traumatic brain injury; VA TBI, Department of Veterans Affairs Traumatic Brain Injury.

^aKappa = 0.65, Kendall τ-b = 0.65; sensitivity (0.85) and specificity (0.82).

Correspondence between the BAT-L and the research-administered VA TBI screen

There was moderate correspondence between the BAT-L and the VA TBI screen (κ = 0.65; Kendall τ-b = 0.65), which is not surprising given that the VA TBI screen is designed to be more sensitive than specific. Based on our diagnosis of military TBI occurring during OEF/OIF/OND deployment from the BAT-L, the VA TBI screen demonstrates sensitivity of 0.85 and specificity of 0.82 (see Table 4).

Identifying false negatives on VA TBI screen

Among the individuals diagnosed with a military-related TBI during OEF/OIF/OND deployment on the BAT-L (n = 65), 15% screened negative for TBI on the VA 4-item screening instrument (n = 10) (see Table 4). This is 6% of the entire sample. Further analysis of these veterans (false negatives) revealed that the VA TBI screen missed identification of one particular subset of military injuries, which we identified as noncombat military injuries. Of these 10 “missed” TBIs via VA TBI screen, 7 occurred during OEF/OIF/OND deployment but were unrelated to combat. These included accidents while performing official duties (eg, fall from truck while unloading materials) and injuries while deployed but not while performing official duties (eg, sports injuries, fights, falls). The remaining 3 injuries did, in fact, occur during combat duty (motor vehicle accident during combat patrol, 2 blast injuries) and were simply not captured by the VA TBI screen. Of note, there were 7 additional military training injuries captured by the BAT-L as military-related mild TBIs, but not identified by the VA TBI screen, as they occurred state-side (not while officially deployed to OEF/OIF/OND). Given that such stateside injuries were not intended to be captured by the VA TBI screen, we adjusted the BAT-L classification and considered these true negatives.

Current symptoms and false negatives on the VA TBI screen

The VA TBI screen was designed to identify individuals with a possible TBI who also report current symptoms (ie, item 4, report of current symptoms is necessary for a positive screen). Thus, it is important to consider whether the false negatives on the VA TBI screen (veterans not identified or “missed” relative to the BAT-L assessment) were individuals who had a TBI (and should have answered positive to items 1 and 2) but did not endorse current symptoms (item 4). If the veterans being diagnosed with a military TBI by the BAT-L but not by the VA TBI screen were missed by the screen because they were not currently symptomatic, they would in fact be true negatives rather than false negatives (according to VA TBI screen criteria). However, only 4 of the 10 false negatives on the VA TBI screen fall into this category and denied current symptoms (negative response to item 4 of the VA TBI screen; see Table 1). Importantly, denial of current symptoms was not the only reason these veterans were not identified by the VA TBI screen. Veterans denied other items as well (item 1 was not endorsed by 2 of the 10 false negatives, item 2 was not endorsed by 9, item 3 was not endorsed by 4, and item 4 was not endorsed by 4).

Because we have the luxury of a rich data set for these individuals, we can independently determine whether these 4 individuals were symptomatic at the time of the TRACTS evaluation. As evidenced in multiple self-report and interview measures, these 4 veterans did, in fact, endorse current difficulties with memory, balance, sensitivity to bright light, irritability, headaches, and/or sleep at the time of their evaluation. It is for these reasons that we keep them in our primary analyses as false negatives (see Table 4).

Identifying false positives on VA TBI screen

Twenty veterans screened positive for TBI on the basis of the VA 4-item screen but did not receive a diagnosis of military TBI on the BAT-L (see Table 4). Using the extensive TRACTS data set, we found that the majority of these veterans met Diagnostic and Statistical Manual of Mental Disorders (Fourth Edition) criteria for PTSD (70%), as diagnosed by the Clinician-Administered PTSD Scale^8,9 either at the time of the evaluation or postdeployment. Sixty percent of false-positive veterans met for current PTSD and 70% met for postdeployment PTSD. This finding may reflect the fact that the VA TBI screen, but not the BAT-L, requires the affirmation of current symptoms that are consistent with either TBI or PTSD (such as concentration difficulties).

Correspondence between the BAT-L and items 1 and 2 only from the research-administered VA TBI screen

As suggested by Terrio et al,³ we also investigated the incidence of a positive TBI screen when the assessment was limited to only items 1 and 2 on the screen. Results were similar to the 4-item screen, with only a slight increase in sensitivity to 0.86 and a decrease in specificity to 0.75 (see Table 5). Using the BAT-L as our criterion standard measure, we found that the 4-item screen resulted in 10 false negatives (missed on VA TBI screen), whereas the 2-item screen resulted in 9 false negatives (missed on VA TBI screen). This suggests that there may be a slight advantage of using only the first 2 items on the screen with regard to detecting the presence of possible TBI.

TABLE 5.

Identification of possible traumatic brain injury according to VA TBI screen items 1 and 2 compared with BAT-L in 179 Operation Enduring Freedom/Operation Iraqi Freedom service members^a

VA TBI screen: Items 1 and 2 only	BAT-L for military TBI during deployment(s)
	Negative	Positive	Total
Negative	86	9	95
Positive	28	56	84
Total	114	65	179

Abbreviations: BAT-L, Boston Assessment of TBI-Lifetime; VA TBI, Department of Veterans Affairs Traumatic Brain Injury.

^aKappa = 0.58, Kendall τ-b = 0.59; sensitivity (0.86) and specificity (0.75).

Correspondence between the BAT-L and the clinician-administered VA TBI screen

There was poor correspondence between the BAT-L military TBI diagnosis and historical clinician-administered VA TBI screen (κ = 0.31; Kendall τ-b = 0.32). Based on our diagnosis of military TBI during OEF/OIF/OND deployment from the BAT-L, the sensitivity of the clinician-administered VA TBI screen was greatly reduced (sensitivity, 0.48) while specificity was similar (specificity, 0.82). More than half of individuals who were diagnosed with a military-related TBI during deployment on the BAT-L (n = 33) screened negative for TBI (were “missed”) on the basis of the clinician-administered VA TBI screen (false negatives = 17).

Interrater reliability of research-administered versus clinician-administered VA TBI screen

Interrater reliability between historical clinician-administered VA TBI screens and research-administered (TRACTS staff) was also low (κ= 0.30; Kendall τ-b = 0.32).

DISCUSSION

The BAT-L and the VA TBI screen have very different purposes, so the observed moderate agreement was anticipated. Based on BAT-L diagnosis of military TBI in theater during OEF/OIF/OND deployment, the VA TBI screen (administered by research staff) demonstrated similar sensitivity (0.85) and specificity (0.82). The specificity of the VA TBI screen was more than adequate. The sensitivity of the VA TBI screen, although quite high, may be lacking as the data suggest that it does not oversample possible TBIs, as was the original intent of casting a broad safety net to catch potential veterans with TBI. Of note, the VA TBI screen missed 15% of veterans who were diagnosed with a military TBI during OEF/OIF/OND deployment on the BAT-L. Given that the goal of the VA TBI screen is to identify those with current symptoms who have not previously been diagnosed with TBI for further evaluation, this reinforces the finding of Terrio and colleagues³ that the VA TBI screen may not be as sensitive to TBIs as initially intended. However, at least in this sample, this does not appear to be a result of the focus of the screening questions on current postconcussive symptoms (eg, item 4). Item analysis of the VA TBI screen revealed that veterans with a TBI were not missed solely because of denial of current symptoms. In fact, sensitivity and specificity were comparable when comparing the first 2 items with the full VA TBI screen. These results suggest that the VA TBI screen could be condensed by administering only the first 2 TBI-specific items without compromising sensitivity or specificity.

The BAT-L is designed to diagnose all military TBIs, not just those individuals experiencing current symptoms, whereas the goal of the VA TBI screen is to identify individuals with possible TBI experiencing current symptoms. The majority of false negatives on the VA TBI screen endorsed current symptoms on the VA TBI screen. The remaining false negatives were noted to endorse current symptoms in the same domains queried by the VA TBI screen on the basis of additional information on other assessment tools that was available to these investigators (eg, memory, headaches). Therefore, the false negatives found in this study do endorse current symptoms and are of concern to the VA, given the VA’s goal of identifying possible TBI with active symptoms. Consideration should be made as to how the VA TBI screen may be adapted to identify such cases, such as lengthening the time frame for which current symptoms are queried (eg, from a week to a month).

A critical finding of this study was that the majority of those screening positive on the VA TBI screen with no history of military TBI (ie, false positives) in this sample carried the Diagnostic and Statistical Manual of Mental Disorders (Fourth Edition) diagnosis of PTSD. This suggests that self-reported persisting symptoms may be due to the presence of psychiatric comorbidities (eg, PTSD), rather than an actual history of TBI. Furthermore, this group of veterans was in close proximity to blasts while deployed, although they did not experience a blast or military TBI (most likely leading to their endorsement of VA TBI screen items 1 and 2). Therefore, the false positives observed on the VA TBI screen are largely represented by veterans who were in close proximity to blasts during deployment and who meet criteria for PTSD. Referral of patients for a comprehensive TBI evaluation who do not have a history of TBI could undermine acceptance of psychological diagnoses and appropriate treatment plan.

Further analysis of veterans who were diagnosed with military TBI during OEF/OIF/OND deployment on the BAT-L but were not identified by the VA TBI screen highlights that the VA TBI screen is missing one particular subset of military injuries: noncombat military injuries (eg, fall from truck while unloading materials, sports injuries, fights). To better identify all veterans who may have incurred a deployment-related TBI, cues for item 1 should be expanded to include this subset of military injuries that often lead to TBI.

Injuries that did not occur during OEF/OIF/OND deployment in theater, but did occur during OEF/OIF/OND military service while stateside, were not counted as false negatives. Rather, they were considered true negatives, because the VA TBI screen was specifically designed to identify possible TBIs occurring while deployed to OEF/OIF/OND with current symptoms. In our sample, there were 7 injuries that occurred during training exercises (eg, blow to the head with a pugil stick) while stateside/outside of combat theater that the VA TBI screen did not capture. We feel that military-related injuries should be flagged for further evaluation if occurring during OEF/OIF/OND-related service regardless of location of injury. Outside of combat theater, training injuries were fairly common, accounting for 10% of our military-related TBIs diagnosed on the BAT-L.

While there was good correspondence between the BAT-L and the research-administered VA TBI screen, there was poor correspondence between the BAT-L military TBI diagnosis and historical clinician-administered VA TBI screen (VA medical records). Interrater reliability between clinician-administered VA TBI screen and research-administered (TRACTS staff) VA TBI screens was also low. There may be multiple reasons for these discrepancies. First, there was a time disparity between the administrations of the clinician-administered VA TBI screen and the BAT-L/research-administered VA TBI screen. In all cases, the clinician-administered VA TBI screen was obtained days, months, or even years before the BAT-L and research-administered VA TBI screen. Second, there was a significant contextual disparity between the administrations of the instruments. It is possible that in a research context, veterans adopted a relatively more reflective mindset and were able to answer the VA TBI screen items more accurately than when confronted with the VA TBI screen at the time of initial contact with VA (eg, making an appointment with audiology). Third, interviewer style and experience with TBI may affect administration, particularly if the interviewer provides additional information or context when administering screening questions.

Possible low interrater reliability may have implications for the utility of the VA TBI screen when administered across the healthcare system by a range of health-care providers. Clinician-administered VA TBI screens may be conducted in any clinical department by staff with very little knowledge and experience with assessment of TBI. In light of the current findings, this may not be appropriate for accurate administration. While the administration of the VA TBI screen by a broad band of healthcare support staff and providers was a laudable goal, it does appear that staff experience with TBI and OEF/OIF/OND-specific issues may impact the tool’s administration. The type of clinical appointment in which the tool is administered may also impact results. For example, asking a veteran the 4-item screen as part of a TBI or neurological evaluation or a research study may be very different than asking during an unrelated clinical visit when the questions may be very discrepant from the visit’s purpose (ie, dental visit). The low interrater reliability of the VA TBI screen when administered by different examiners and low sensitivity of the VA TBI screen as administered by clinical staff are of concern for the VA’s continued use of this instrument. One way to address this pitfall is to add further training in administration for VA staff or to add administration prompts to the VA TBI screen within the computerized patient record system VA medical record system that remind TBI-naive clinicians to specific cues needed to reliably administer this screen.

SUMMARY

Based on BAT-L diagnosis of military TBI, the VA TBI screen demonstrated similar sensitivity (0.85) and specificity (0.82). The specificity of the VA TBI screen was more than adequate. However, sensitivity may be lacking as the data suggest that the VA TBI screen may not oversample or “catch all” possible TBIs. Traumatic brain injuries not identified by the screen were largely noncombat military injuries (70%), although some combat injuries (30%) were also not identified. The poor correspondence between the BAT-L military TBI diagnosis and the clinician-administered VA TBI screen and low interrater reliability of the VA TBI screen when administered by different examiners raise potential concern about the continued use of this instrument across the healthcare system by a range of healthcare providers.