Sensitivity and Specificity of the Beck Depression Inventory-II in Persons With Traumatic Brain Injury

Abstract

Objectives

Our objective was to examine the Beck Depression Inventory-II (BDI-II) in a traumatic brain injury (TBI) sample using a receiver operating characteristic (ROC) curve to determine how well the BDI-II identifies depression. An ROC curve allows for analysis of the sensitivity and specificity of a diagnostic test using various cutoff points to determine the number of true positives, true negatives, false positives, and false negatives.

Design

This was a secondary analysis of data gathered from an observational study. We examined BDI-II scores in a sample of 52 veterans with remote histories of TBI.

Setting

This study was completed at a Veterans Affairs (VA) Medical Center.

Participants

Participants were veterans eligible to receive VA health care services.

Interventions

Not applicable.

Main Outcome Measures

Outcome measures included the BDI-II and the Structured Clinical Interview for DSM-IV Axis I Disorders (SCID-IV).

Results

We generated an ROC curve to determine how well the BDI-II identifies depression using the SCID-IV as the criterion standard for diagnosing depression, defined here as a diagnosis of major depressive disorder. Results indicated a cutoff score of at least 19 if one has a mild TBI or at least 35 if one has a moderate or severe TBI. These scores maximize sensitivity (87%) and specificity (79%).

Conclusions

Clinicians working with persons with TBI can use the BDI-II to determine whether depressive symptoms warrant further assessment.

The clinical demands of managing persons with TBI both postacutely and throughout the course of a lifetime are significant. This is due, in part, to wide-ranging sequelae from health conditions and cognitive changes to persisting affective problems such as depression.^1–5 Within the first few years of a TBI, symptoms of depression are frequently reported, with prevalence ranging from 33% to 42%.^6,7 Furthermore, although one study found that the risk of depression drops significantly over the first 3 years after injury, estimates of subsequent prevalence remain higher than in non-TBI populations.⁸ Data from up to 8 years after TBI indicate a prevalence of 61%; however, this figure drops to 48% after removing the effects of preinjury mood disorders.⁹ Additional research extends this finding by showing that the presence of a preinjury mood disorder increases the likelihood of major depression after TBI.⁶ Long-term follow-up studies and cross-sectional data indicate that persisting psychologic problems including depression and social isolation can occur up to 25 years after TBI.^1,4,10–12 In the longest follow-up study to date, Koponen et al¹³ found that 26.7% of TBI survivors met criteria for major depression 30 years after their injuries.

Not only is depression an issue potentially requiring significant clinical attention after TBI, but this problem is exacerbated by the growing number of people living with TBI. The number of people who survive TBI has been increasing due to improvements in emergency medicine and technology and to greater enforcement of seat belt laws.^14,15 Furthermore, two thirds of those who sustain a TBI before age 30 will likely live for another 30 to 40 years.¹⁶ Moreover, Harrison-Felix et al¹⁷ found that sustaining a TBI resulted in an average reduction of life expectancy of only 7 years. Hence, clinicians need efficient screening and diagnostic tools, particularly for problems such as depression. With this in mind, those working with survivors of TBI are encouraged to consider the measurement issues of tools they use in clinical practice.

To assess for psychiatric disorders in this population, semi-structured interviews such as the DSM-IV¹⁸ SCID-IV¹⁹ are encouraged. Robinson and Jorge²⁰ conclude that using diagnostic criteria based on such standards provides the soundest diagnosis of depression for people in this population. However, because semi-structured interviews are time consuming, their use is often not feasible in settings that present significant time constraints. The use of less time consuming self-report tools, such as the BDI-II,²¹ can facilitate the diagnostic process as a screen for depressive symptoms, determining the necessity of engaging in a lengthy, semi-structured interview.

The previous version of the BDI-II,²¹ the BDI,²² is widely used in rehabilitation settings.²³ Clinicians working in such settings should consider the psychometric properties of this instrument to ensure that they have selected the most appropriate tool for assessing depressive symptoms, bearing in mind the diagnostic challenges in assessing persons with brain injury. For example, one challenge of diagnosing depression post-TBI is that somatic symptoms such as changes in sleep, appetite, or libido may also occur as a result of injury to the brain, thus leading to potential overdiagnosis of depression.²⁰ However, researchers studying the BDI in a TBI sample found no overendorsement of somatic symptoms of depression.²⁴ Additionally, they noted that the BDI is an effective screening tool with high reliability estimates and a factor structure similar to that found in the general population.²⁴

Conversely, despite its sound factor structure and high reliability, previous literature questions the use of the BDI to identify depression in this population because of its low sensitivity in discriminating persons who are depressed from those who are not depressed.²⁵ With regard to the BDI-II,²¹ recent factor analysis findings indicate that it is useful for identifying symptoms of depression during acute TBI recovery,²⁶ but to date, there are no published data regarding the sensitivity and specificity of this measure in a TBI population. Therefore, our aim for this study was to examine these particular psychometric properties of the BDI-II²¹ in a TBI sample. We generated an ROC curve to determine how well the BDI-II²¹ identifies depression and to determine its maximum sensitivity and specificity, using the SCID-IV¹⁹ as the criterion standard for diagnosing depression. An ROC curve allows for analysis of the sensitivity and specificity of a diagnostic test using various cut-off points to determine the number of true positives, true negatives, false positives, and false negatives.

METHODS

Procedures

This is a secondary analysis of data gathered during an observational study of veterans with TBI and/or PTSD receiving care at a western Veterans Affairs Health Care System facility.²⁷ Seventy-two subjects participated in the observational study, for which institutional review board approval was obtained. These individuals completed the SCID-IV and the BDI-II, among other assessments, including neuroimaging, neurologic, and neuropsychologic examinations. All instruments were administered by psychologists or trained research assistants.

For inclusion in the observational study, we chose veterans receiving outpatient and inpatient services for treatment related to TBI and/or PTSD. Exclusion criteria were: (1) a history of neurologic disease other than TBI; (2) sleep apnea; (3) presence of the following psychiatric disorders: schizophrenia, psychosis, or bipolar I disorder; (4) scores on the BDI-II that were more than 2 SDs higher than a historical mean of scores obtained by persons participating in this VA facility’s PTSD Residential Rehabilitation Program; (5) a Computerized Assessment of Response Bias²⁸ performance of type III (very poor effort) or IV (extreme exaggeration or response bias); (6) active substance abuse responses on the SCID-IV that suggested abuse in the 7 days prior to participation; (7) legal blindness; (8) current guardianship or mention of recommendation for guardianship in medical chart within the prior 6 months; or (9) inability to complete the process of informed consent or magnetic resonance imaging.

Of all participants who signed consent forms for the observational study, 21 were excluded based on the above criteria and 14 withdrew from the study (eg, left the area or the PTSD program), and therefore, the remaining 72 comprised the original sample. Of the 72 individuals, 59 had a remote history of TBI. Of the 59 with TBI, 7 individuals had incomplete SCID-IV interviews; therefore, the remaining 52 subjects represent the sample used in the present study. Characteristics of the sample used for the current analyses are presented in table 1.

Table 1.

Characteristics of the Sample (N=52)

Characteristic	Value
Age
Mean ± SD	51.7±10.3
Median	54
Range	23–74
Years since injury
Mean ± SD	23±15
Median	21
Range	1–51
Sex
Male	47 (90)
Female	5 (10)
Race/ethnicity
White	38 (73.1)
Hispanic	10 (19.2)
Black	4 (7.7)
Education
< High school	5 (9.6)
High school	15 (28.9)
> High school	32 (61.5)
TBI severity
Mild	25 (48.1)
Moderate	9 (17.3)
Severe	18 (34.6)

NOTE. Values are n (%) or as otherwise noted. Percentages are rounded to the nearest tenth.

Measures

We confirmed a history of TBI via semi-structured interview and chart reviews using criteria set forth by the Centers for Disease Control.²⁹ Time since injury ranged from 1 to 51 years. Injury severity (mild, moderate, severe) was assessed as advocated by a continuing medical education program developed by the Department of Veterans Affairs.³⁰ Specific criteria are as follows: mild TBI (altered or LOC <30min with negative imaging, GCS of 13–15, or PTA <24h); moderate TBI (LOC <6h with positive imaging, GCS of 9 –12, or PTA of <7d ; and severe TBI (LOC >6h with abnormal imaging, GCS of <9, or PTA of >7d).³⁰ If a participant had more than 1 TBI, the severity of injury was categorized based on their most significant injury. For these individuals, time since injury was also based on the date of their most significant injury.

The BDI-II,²¹ the most recent version of the BDI,²² was designed to measure the presence and severity of depression symptoms as noted in the DSM-IV.¹⁸ The BDI-II is a 21-item self-report Likert scaled measure that provides a choice of 4 responses per question. The 21 items are summed to provide a single score. Recommended cut-off scores are 0 to 13 (minimal depressive symptoms), 14 to 19 (mild depressive symptoms), 20 to 28 (moderate depressive symptoms), and 29 to 63 (severe depressive symptoms). This instrument has been shown to have sound psychometric properties in general populations²¹ and a sound factor structure for use with survivors of TBI.²⁶

The SCID-IV is a semi-structured interview that enables researchers to make DSM-IV Axis I diagnoses.¹⁹ In the original study of 72 veterans, the SCID-IV was used to identify DSM-IV Axis I disorders in order to satisfy inclusion and exclusion criteria. In the current analysis, data regarding the presence or absence of MDD were used to conduct the ROC curve analysis.

Analyses

Because previous literature questions the usefulness of the BDI to identify depression in a TBI population,²⁵ we investigated the usefulness of the BDI-II for the same purpose. Using logistic regression with depression as the outcome (as determined by a diagnosis of MDD on the SCID-IV) and BDI-II scores as well as TBI severity (categorized as mild and moderate/severe) as the predictors, an ROC curve was generated. The area under the ROC curve (C-index) was calculated along with the 95% CI. A perfect test would have a C-index of 100, whereas an uninformative test would have a C-index of 50. Additionally, optimal cutoff values were determined, thereby maximizing sensitivity and specificity. All analyses were run in SAS v9.1^a and assumed a 2-sided test of hypothesis with alpha set at .05.

RESULTS

The mean ± SD BDI-II score was 25.0±14.6 and a range of 0 to 53. Twenty-three participants met criteria for MDD, and 18 met criteria for PTSD according to the SCID-IV. Ten of the participants (19%) had a remote history of multiple TBIs. Additional sample characteristics are presented in table 1.

The ROC curve for the logistic model of SCID-IV depression as a function of BDI-II scores and TBI severity is displayed in figure 1. BDI-II scores and TBI severity were significant predictors of depression in the logistic model (P=.004 and P=.02, respectively). For each 5-point increase in BDI-II score there is a 70% increase in the predicted OR of depression (OR=1.7; 95% CI, 1.2–2.4), and mild TBI is associated with a 460% increase in the predicted odds of depression as compared with moderate or severe TBI (OR=5.6; 95% CI, 1.3–25.0). The C-index is 88.8 (95% CI, 80.2–97.3) indicating a reasonable test. The cutoff scores that maximize sensitivity (87%) and specificity (79%) are different for those with mild TBIs and those with more severe injuries. If a person has a mild TBI, a BDI-II score less than 19 indicates no depression whereas a score of 19 or higher does indicate depression. However, if a person has a moderate or severe TBI, the BDI-II score indicating depression is 35 or higher. In order to identify all true positives (ie, those with depression correctly identified as such by their BDI-II score) when assessing those with mild injuries, a score of 6 or higher is necessary in order to obtain 100% sensitivity, whereas a score of 23 or higher is necessary in order to obtain 100% sensitivity in those with moderate or severe injuries. In either case, the specificity decreases to 59% indicating a decrease in identification of true negatives (those with no depression correctly identified as such by their BDI-II score).

Fig 1.

ROC curve: sensitivity and specificity of the BDI-II plus TBI severity using the SCID-IV as a criterion standard.

DISCUSSION

The goal of our analysis was to determine the sensitivity and specificity of BDI-II scores and TBI severity in those with a history of TBI using the SCID-IV to establish a diagnosis of depression. Our results show that the BDI-II is an appropriate gauge of depression in a TBI population. The ROC curve indicates that when assessing those with mild TBI, the optimal combination of sensitivity and specificity occurs when a BDI-II score of 19 or higher is used to determine whether depressive symptoms warrant further assessment. When assessing those with moderate or severe TBI, the optimal combination of sensitivity and specificity occurs when a BDI-II score of 35 or higher is used. No published studies are available with which to compare these findings. However, one study examined the sensitivity and specificity of the earlier version of this instrument, the BDI,²² using an earlier version of the SCID-IV¹⁹ as the criterion standard to establish a diagnosis of depression in a TBI population. Researchers calculated sensitivity at various levels of specificity and found low sensitivity (36%) when specificity was set at 80%, questioning the use of the BDI as a means of detecting depression in persons with TBI.²⁵

Taking an over-inclusive approach for those with mild TBI, a score of 6 or higher on the BDI-II would correctly identify all individuals with depression (sensitivity of 100%), as would a score of 23 or higher for those with moderate or severe TBI; however, these cutoffs lead to substantial over-diagnosis of depression (specificity of 59%). There may, however, be some benefit in using the lower cutoffs in settings where services are provided to a large number of persons. In such settings, the BDI-II may serve as a highly sensitive and convenient screening tool during an initial intake appointment. However, further assessment remains necessary for diagnostic purposes. It is important to note that if an individual scores lower than 6 or 23, respectively, it is possible that he or she could have dysthymia or an adjustment disorder with depressed mood.

Those with mild TBI are more likely to be depressed; therefore, their BDI-II score can be lower and still indicate depression. With regard to the relationship between TBI severity and symptoms of depression, one study has shown similar findings up to 2 years postinjury, with persons who have severe TBIs reporting lower levels of depressive symptoms than those with mild injuries.³¹ Malec et al³¹ suggest that the difference between self-reported depression in those with mild as compared with severe injuries may be due, in part, to one’s self-awareness. They found that those with mild TBI had some amount of self-awareness, suggesting that they were more likely to report symptoms of depression, whereas those with severe TBI had impaired self-awareness, suggesting that they were less likely to report symptoms of depression.³¹ Others have questioned the relationship between TBI severity and deficit awareness³² or have found the opposite relationship in that those who were the most self-aware had the best emotional adjustment.³³

Study Limitations

With regard to limitations, SCID-IV interviewers were not masked to the BDI-II results, thus introducing an element of possible bias. However, at the time of data collection, determining the sensitivity and specificity of the BDI-II was not the purpose of the study. Therefore, we assume that this possible bias was minimized. The sample size of this study is not large enough to establish diagnostic validity, and replication with a larger and more diverse TBI sample is warranted. Additionally, the predominantly older male veteran sample limits the generalizability of the findings. Furthermore, veterans with BDI-II scores above 54 were excluded from the study, also limiting the generalizability of the results. Moreover, the veterans in this sample were not self-referred to the outpatient and inpatient services from which they were recruited, also limiting the external validity of the findings. Finally, generalizability to civilians may be limited.

Future studies should examine the relationship of depressive symptoms as identified by the BDI-II with regard to poor psychiatric outcomes in TBI populations such as suicide. This is particularly important given that persons with a history of TBI have higher rates of suicide attempts and death by suicide than those without a history of TBI.^34,35 In this vein, ongoing and accurate screening is particularly important because increased risk of suicide persists for years after injury.³⁵

CONCLUSIONS

Our results support the use of the BDI-II as a screening tool for depression in this population. However, clinicians should use multiple means of assessment to improve the accuracy of diagnosing depression. If an over-inclusive approach is taken, those with mild TBI who score at least a 6 and those with moderate or severe TBI who score at least a 23 on the BDI-II should be interviewed further using semi-structured interviews to determine whether depression is present. A less cautious approach that maximizes both sensitivity and specificity entails using a score of at least 19 if a person has a mild TBI or a score of at least 35 if an individual has a moderate or severe TBI, but multiple means of assessment should still be used in this scenario.

List of Abbreviations

BDI

Beck Depression Inventory

BDI-II

Beck Depression Inventory-II

CI

confidence interval

DSM-IV

Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition

GCS

Glasgow Coma Scale

LOC

loss of consciousness

MDD

major depressive disorder

OR

odds ratio

PTA

posttraumatic amnesia

PTSD

posttraumatic stress disorder

ROC

receiver operating characteristic

SCID-IV

Structured Clinical Interview for DSM-IV Axis I Disorders

TBI

traumatic brain injury