Discriminant validity of the Hospital Anxiety and Depression Scale, Beck Depression Inventory (II) and Beck Anxiety Inventory to confirmed clinical diagnosis of depression and anxiety in patients with chronic obstructive pulmonary disease

Abstract

The objective of this study was to investigate the discriminant validity of commonly used depression and anxiety screening tools in order to determine the most suitable tool for patients with chronic obstructive pulmonary disease (COPD). COPD patients (n = 56) completed the Hospital Anxiety and Depression Scale (HADS), Beck Depression Inventory (BDI-II) and Beck Anxiety Inventory (BAI). These scores were compared to confirmed clinical diagnoses of depression and anxiety using the Mini Neuropsychiatric Interview. HADS depression subscale (HADS-D) sensitivity/specificity was 78/81%; BDI-II 89/77%; HADS anxiety subscale (HADS-A) 71/81%; and BAI 89/62%. HADS-D sensitivity/specificity was improved (100/83%) with the removal of Q4 ‘I feel as if I am slowed down’ and adjusted cut-off (≥5). Removal of BDI-II Q21 ‘Loss of interest in sex’ with adjusted cut-off ≥12 resulted in similar improvement (100/79%). No problematic items were identified for HADS-A or BAI. Previously reported low sensitivity/specificity of the HADS for COPD patients was not replicated. Furthermore, simple modifications of the HADS-D markedly improved sensitivity/specificity for depression.

BDI-II, HADS-A and BAI produced acceptable sensitivity/specificity unmodified. Pending further research for COPD patients we recommend continued use of the HADS-A with standard cut-off (≥8) and removal of Q4 of the HADS-D with lower cut-off ≥5.

Introduction

Depression and anxiety are common comorbidities in patients diagnosed with chronic obstructive pulmonary disease (COPD), with benchmark prevalence estimates of 40% for depression and 36% for anxiety.¹ Screening COPD patients for concomitant psychological distress is important as it has been found to contribute to poorer health outcomes across a number of domains, including: increased exacerbation rates, diminished exercise performance and functional mobility, reduced health-related quality of life, increased number of emergency hospital visits, increased hospitalizations, increased length of stay as an admitted patient, increased mortality rates and in general greater economic burden.^2–8 Consequently, European and Australasian COPD management guidelines recommend routine screening for depression and anxiety in COPD patients using the Hospital Anxiety and Depression Scale (HADS).^9,10 Use of the HADS with COPD patients is widespread; being reported in at least 17 published studies.^5,6,11–25 However, only two previous studies have investigated the discriminant validity of the HADS with COPD populations and both cast doubt about its usefulness. Cheung et al.¹² confirmed clinical diagnoses of anxiety disorders with 55 elderly New Zealand COPD patients using the Mini Neuropsychiatric Interview (MINI) and reported at the standard cut-off ≥8, the HADS anxiety subscale (HADS-A) provided sensitivity/specificity of 36/90% and area under curve (AUC) of 79%. They recommended a lower cut-off ≥4 that yielded an improved sensitivity/specificity of 79/71%. Nowak et al.²⁶ used the German version of the HADS depression subscale (HADS-D) to assess 259 Swiss COPD patients for depression, cross-referenced with diagnoses of depression co-morbidity noted in patients’ clinical records, and also reported low discriminant validity at the standard cut-off ≥8 (sensitivity/specificity 25/84%; AUC 66%). They recommended a lower optimal cut-off score of ≥5 but this still yielded fairly low sensitivity/specificity 62/63%. These results cast doubt over the appropriateness of using the HADS for patients with COPD. However, these results demand replication before any firm conclusions can be drawn.

Potential alternatives to the HADS for COPD patients include the Beck Depression Inventory (BDI-II), also a popular research measure to screen COPD patients for depression, and the counterpart for anxiety, the Beck Anxiety Inventory (BAI). These tests are longer than the HADS, with 21 items each, compared to the 14 items in total for the HADS, but are well established and popular for screening COPD patients for studies involving cognitive behavioural therapy.^27–30 However, we are unaware of any studies that have established the discriminant validity of either the BDI-II or BAI in COPD patients. Therefore, the aims of the present study were to investigate the discriminant validity of the HADS with patients diagnosed with COPD, and to examine the discriminant validity of the BDI-II and BAI as potential alternatives for this population.

Method

Sample

Participants were outpatients attending community-based COPD clinics in Perth, Western Australia. Participants were excluded if they had a life expectancy of less than 6 months, were currently involved in another research study, had an illness exacerbation resulting in hospitalization within the previous month, were not fluent in English or were blind, deaf or diagnosed with dementia or Alzheimer’s disease. Ethics approval for the study was granted by Royal Perth Hospital, Edith Cowan University and the South Metropolitan Health Service Human Research ethics committees. We attempted to contact a list of 164 outpatients. Thirty-five individuals were uncontactable because they were either deceased, discharged from the clinics, had disconnected telephones or we were unable to contact them after five separate attempts. Of the 129 successfully contacted, 27 did not meet our inclusion criteria and 46 declined to participate, citing ill-health or ‘lack of time’. The final sample consisted of 56 patients with confirmed diagnosis of COPD by respiratory physicians who gave their informed consent to participate in the study, representing 34% of our original sampling pool, and a consent rate of 55% of contactable and eligible participants. The mean age of participants was 73.3 years (SD: 8.9; range 50–91). There were more females (58.9%) than males (41.1%) and the majority of participants were retired (73.2%) and married or in a de facto relationship (69.7%; Table 1).

Table 1.

Participant characteristics.

Characteristic	N	%
Males	23	41.1
Females	33	58.9
Aboriginal or Torres Strait Islander	1	1.8
Education
Up to 10 years of school	26	46.4
11 years and above	27	48.2
Other	3	5.4
Marital status
Married/de facto	39	69.7
Not in a current relationship	17	30.3
Has a carer	41	73.2
Occupation
Employed	4	7.2
Retired	41	73.2
Other	11	19.6
GOLD stage
Mild	2	3.6
Moderate	18	32.1
Severe	25	44.6
Very severe	3	5.4
Missing	8	14.3

GOLD: Global Initiative for Chronic Obstructive Lung Disease Classification.

Instruments

Participants were asked to self-complete the HADS, BDI-II and BAI, followed immediately by a structured clinical interview with a provisional psychologist conducting the MINI. The MINI (v6.0.0) is considered a ‘gold standard’ in determining incidence of Axis I psychiatric disorders as per DSM-IV and ICD-10 criteria.³¹

Statistical analyses

Aggregated and item-by-item scores were examined for the HADS, BDI-II and BAI and compared to clinically confirmed current major depression and any anxiety disorders (panic disorder, agoraphobia and generalized anxiety disorder) based upon MINI diagnostic criteria. The sensitivity, specificity, Youden’s index J, positive predictive value, negative predictive value, κ coefficient and AUC values were calculated using the MINI as the clinical standard for the presence or absence of psychological comorbidity. Independent samples t-tests were also used to compare the mean HADS, BDI-II and BAI scores of participants in various groups. IBM SPSS (v22) was used for all statistical analyses.

Results

Depression

Nine of 56 patients (16.1%) met the clinical diagnosis for major depression according to the MINI. The mean HADS-D and BDI-II scores for these patients are compared to others in Table 2.

Table 2.

Proportions and averages of depression and anxiety scores for the MINI, HADS-D, HADS-A, BDI-II and BAI.

Depression measure	MINI	HADS-D ≥8	BDI-II ≥14
Prevalence	16% (95% CI ± 10)	29% (95% CI ± 11.8)	34% (95% CI ± 12.4)
Average scores with depression	–	12.33 (SD = 4.15)	32.44 (SD = 12.04)
Average scores without depression	–	4.34 (SD = 2.83)a	9.40 (SD = 8.21)a
Anxiety measure	MINI	HADS-A ≥8	BAI ≥8
Prevalence	25% (95% CI 14–36%)	32% (95% CI ± 12.2)	48% (95% CI ± 13.1)
Average scores with anxiety	–	9.14 (SD = 4.07)	20.79 (SD = 12.63)
Average scores without anxiety	–	4.36 (SD = 4.03)a	8.60 (SD = 10.39)a

MINI: Mini Neuropsychiatric Interview; HADS-D: Hospital Anxiety and Depression Scale depression subscale; HADS-A: Hospital Anxiety and Depression Scale anxiety subscale; BDI-II: Beck Depression Inventory; BAI: Beck Anxiety Inventory.

^aDenotes statistically significant difference to participants with depression at p < 0.001.

Both HADS-D and BDI-II scores were significantly different between participants with clinically confirmed depression compared to those without. AUC statistics for the HADS-D and BDI-II were both close to perfect (94.8%, p < 0.001; 94.9%, p < 0.001 respectively). As can be seen in Table 3, the recommended HADS-D cut-off of ≥8 identified 7 of 9 (77.8%) true positive cases and 38 of 47 (80.9%) true negative cases. Youden’s index J suggested a similar optimal cut-off of ≥7. The mean score of false positive cases was significantly lower (M = 9.38, SD = 1.30) than true positives (M = 13.86, SD = 3.29; t(8.36) = −3.571, p = 0.003). The recommended BDI-II cut-off ≥14 identified 8 of 9 (88.9%) true positive cases and 36 of 47 (76.6%) true negative cases. Youden’s index J suggested a similar optimal cut-off of ≥13. BDI-II true positives cases had a significantly higher mean score (M = 32.4, SD = 12.0) than the false negative cases (M = 22.5, SD = 5.6; t(18) = −2.454, p = 0.025).

Table 3.

The sensitivity and specificity of the HADS-D, HADS-D excluding Question 4, BDI-II and BDI-II excluding Question 21 for detecting major depression in COPD patients (%).

Cut-off points	Sensitivity	Specificity	Youden’s index J	PPV	NPV
HADS-D
3	100.0	31.9	0.32	22.0	100
4	100.0	46.8	0.47	26.5	100
5	100.0	57.4	0.57	31.0	100
6	100.0	74.5	0.75	42.9	100
7a	100.0	78.7	0.79	47.4	100
8	77.8	80.9	0.59	43.8	95.0
9	77.8	85.1	0.63	50.0	95.2
10	77.8	91.5	0.69	63.6	95.5
11	55.6	95.7	0.51	71.4	91.8
HADS-D ex Qu 4
1	100.0	28.3	0.28	21.4	100
2	100.0	47.8	0.48	27.3	100
3	100.0	69.6	0.70	39.1	100
4	100.0	78.3	0.78	47.4	100
5a	100.0	82.6	0.83	52.9	100
6	88.9	87.0	0.76	57.1	97.6
7	77.8	93.5	0.71	70.0	95.7
8	55.6	95.7	0.51	71.4	91.8
9	55.6	97.8	0.54	83.3	92.0
BDI-II
9	100.0	63.8	0.64	34.6	100
10	100.0	70.2	0.70	39.1	100
11	100.0	70.2	0.70	39.1	100
12	100.0	76.6	0.77	45.0	100
13a	100.0	76.6	0.77	45.0	100
14	88.9	76.6	0.66	42.1	97.3
15	88.9	78.7	0.68	44.4	97.4
16	88.9	78.7	0.68	44.4	97.4
17	88.9	78.7	0.68	44.4	97.4
BDI-II ex Qu 21
8	100.0	61.7	0.62	33.3	100.0
9	100.0	70.2	0.70	39.1	100.0
10	100.0	74.5	0.75	42.9	100.0
11	100.0	74.5	0.75	42.9	100.0
12a	100.0	78.7	0.79	47.4	100.0
13	88.9	78.7	0.68	44.4	97.4
14	88.9	78.7	0.68	44.4	97.4
15	88.9	80.9	0.70	47.1	97.4
16	88.9	80.9	0.70	47.1	97.4

HADS-D: Hospital Anxiety and Depression Scale depression subscale; BDI-II: Beck Depression Inventory; PPV: positive predictive value; NPV: negative predictive value.

^aOptimal cut-off.

An examination of each item of the HADS-D using independent samples t-tests identified the mean score for Question 4 ‘I feel as if I am slowed down’ as substantially higher than all other items. This was the only item of the HADS-D for which no participant scored zero, and the only item for which there was no statistically significant difference in mean scores between those with clinically diagnosed depression (M = 2.56, SD = 0.73) and those without (M = 2.09, SD = 0.86; t(54) = −1.544, p = 0.128). The sensitivity and specificity of the HADS-D was therefore recalculated when excluding this item. With Question 4 excluded, Youden’s index J suggested an optimal cut-off of ≥5, which detected all true positive cases (100.0%) and 39 of 47 true negative cases (83.0%) within our sample (Table 3). True positive cases still had a significantly higher mean HADS-D score (M = 11.0, SD = 3.22) than false positive cases (M = 6.33, SD = 1.80; t(14) = −3.694, p = 0.002).

Independent samples t-tests for each item of the BDI-II also highlighted Question 21. The mean score for Question 21 ‘Loss of interest in sex’ did not differ significantly between those with a clinical diagnosis of depression (M = 1.30, SD = 0.43) and those without (M = 1.32, SD = 1.92; t(54) = −1.181, p = 0.243). Therefore, the sensitivity and specificity of the BDI-II was recalculated to exclude Question 21. Upon removal of Question 21, Youden’s index J suggested an optimal cut-off of ≥12 that detected all true positive cases (100.0%) and 37 of 47 true negative cases (78.7%) within our sample (Table 3).

Anxiety

Fourteen of 56 patients (25.0%) met the MINI criteria for an anxiety disorder. This included six patients diagnosed with panic disorder, five with agoraphobia and three with generalized anxiety disorder. Table 2 shows the proportion and average scores of patients identified as meeting the diagnostic criteria for any anxiety disorder via the MINI and those meeting clinically relevant anxiety symptomatology via the HADS-A and BAI.

Independent samples t-tests suggested both HADS-A and BAI scores were significantly different between participants with clinically confirmed anxiety disorders compared to those without. AUC statistics for the HADS-A and BAI were both in the ‘fair’ range (78.4%, p < 0.001; and 78.5%, p < 0.001, respectively). At the recommended cut-off ≥8 the HADS-A identified 10 out of 14 true positive cases (71.4%) and 34 of 42 true negative cases (81.0%). With our sample Youden’s index J suggested an optimal cut-off of ≥9. There were no significant differences between the mean scores of true positives (M = 11.20, SD = 2.44) versus false positives (M = 11.38, SD = 2.62; t(16) = 0.146, p = 0.885). With the recommended cut-off ≥8 the BAI identified 11 of 14 true positive cases (78.6%) and 26 of 42 true negative cases (61.9%). For our sample, the optimal cut-off was ≥12, which maintained true positives at 78.6% but improved false negatives to 76.2% (Table 4). No significant differences in BAI score were found between true positive cases (M = 25.10, SD = 10.23) and false positive cases either (M = 18.44, SD = 11.01; t(16) = −1.542, p = 0.136).

Table 4.

The sensitivity and specificity of the HADS-A and BAI for detecting anxiety in COPD participants.

Cut-off points	Sensitivity	Specificity	Youden’s index J	PPV	NPV
HADS-A
5	85.7	64.3	0.50	44.4	93.1
6	78.6	71.4	0.50	47.8	90.9
7	71.4	78.6	0.50	52.6	89.2
8	71.4	81.0	0.52	55.6	89.5
9a	71.4	85.7	0.57	62.5	90.0
10	64.3	85.7	0.50	60.0	87.8
11	28.6	88.1	0.17	44.4	78.7
12	14.3	90.5	0.05	33.3	76.0
13	14.3	92.9	0.07	40.0	76.5
BAI
8	78.6	61.9	0.41	40.7	89.7
9	78.6	71.4	0.50	47.8	90.9
10	78.6	76.2	0.55	52.4	91.4
11	78.6	76.2	0.55	52.4	91.4
12a	78.6	76.2	0.55	52.4	91.4
13	71.4	78.6	0.50	52.6	89.2
14	71.4	81.0	0.52	55.6	89.5
15	71.4	81.0	0.52	55.6	89.5
16	71.4	81.0	0.52	55.6	89.5

HADS-A: Hospital Anxiety and Depression Scale anxiety subscale; BAI: Beck Anxiety Inventory; PPV: positive predictive value; NPV: negative predictive value.

^aOptimal cut-off.

An examination of the mean response to each question within the HADS-A revealed no conspicuously inflated items for COPD patients.

Examining the mean response to each question within the BAI revealed Question 15 ‘Difficulty breathing’ was substantially higher than other items. However, the mean score for this question remained statistically different between those with clinically diagnosed anxiety (M = 1.71, SD = 1.07) and those without (M = 1.0, SD = 1.12; t(54) = −2.05, p = 0.045), suggesting discriminant validity still existed and removal of this item was therefore not warranted.

Discussion

The first aim of this study was to investigate the discriminant validity of the HADS with patients diagnosed with COPD. We were unable to replicate the findings of Cheung et al. and Nowak et al.^12,26 In contrast to Cheung et al., who found a much lower optimal cut-off for the HADS-A of ≥4, our optimal cut-off (≥9) was a very close approximation to the standard recommendation of ≥8. The reason for the different results is far from clear. The proportion of our sample diagnosed with anxiety disorders (25.0%) corresponded very closely with the sample of Cheung et al. (25.5%) using the same diagnostic test (MINI) and extremely similar patient profiles (Anglo-Saxon dominant cultures, i.e., New Zealand vs. Australia). So too, both studies had similar sample sizes (n = 55 vs 56), demonstrated similar levels of discriminant validity (AUC 79% vs 78.4%), and shared fair sensitivity (79% vs 71%) and specificity (71% vs 86%) at their optimal cut-offs. However, the COPD severity in the population studied by Cheung et al. was not reported, raising the question as to whether this may have affected prevalence rates which would ultimately impact on sensitivity and specificity scores. Cheung et al. acknowledged their optimal cut-off was ‘unusually low’ and further replication of their results was therefore warranted. The fact that we were unable to replicate their results now casts doubt over their suggestion of a HADS-A cut-off of ≥4 for COPD patients. Further attempts to replicate the results of Cheung et al. are needed to clarify this matter.

Contrary to Nowak et al. who suggested the HADS-D provided poor discriminant validity for COPD patients (AUC 66.2%), our results suggested its discrimination was excellent (AUC 94.8%). Even with their optimal cut-off ≥6, Nowak et al. observed a borderline sensitivity/specificity of 62.1/62.6% contrasting with our optimal cut-off ≥7 achieving 100.0/78.7%, and improving even further with removal of Question 4 and lowering the cut-off to ≥5 yielding 100.0/83.0%. There are many possibilities why we were unable to replicate the results of Nowak et al., including vastly differing sample sizes (n = 259 vs 56), differing prevalence of major depression (16.1% vs 11.2%), use of the German versus English version of the HADS, differing cultures and use of pre-diagnosed depression based on medical records subsequently confirmed by the patient’s physician versus current diagnosis confirmed by a psychologist using a structured clinical interview.

Despite the HADS being created to avoid somatic symptom overlap with anxiety and depression in medical patients, the continued use of HADS with chronic diseases remains contentious as a variety of cut-offs have been suggested differing from optimal cut-off scores suggested for the general patient population, for example, HADS-D ≥4 in coronary heart disease³² and HADS-D ≥11 in end-stage renal disease.³³ Given that the discriminant validity of the HADS has only been tested in COPD populations three times with divergent results, the recommendation of using this screening tool in international management guidelines remains contentious. More population-specific tools, such as the Geriatric Anxiety Inventory and COPD Anxiety Questionnaire (German: CAF), may be preferable.^34,35 However, Cheung et al. investigated the Geriatric Anxiety Inventory as another alternative measure for use with COPD patients but found it no more useful than the HADS, and to the best of our knowledge the CAF has yet to be translated and validated for use in English and we are unaware of any others for COPD.

The second aim of our study was to examine the discriminant validity of the BDI-II and BAI as viable alternatives to the HADS for use with patients diagnosed with COPD. The BDI-II demonstrated excellent discriminant validity and very similar sensitivity/specificity to the HADS-D. Question 21 did not discriminate between those with and without clinical depression. However, its removal only marginally improved the validity of optimal cut-offs (J = 0.77 vs 0.79). Therefore, we see little value in unnecessarily modifying the BDI-II for use with COPD patients. Likewise, the BAI demonstrated similar discriminant validity to the HADS-A. However, given the shorter length of the HADS (14 items combined) compared to the BDI-II and BAI (42 items combined) and the free use of the former, versus cost per test of the latter, there may be limited advantage of using the Beck inventories over the HADS.

The relatively modest sample size of stable COPD patients in our study increases the probable error of our prevalence estimates. However, they were highly consistent with previous estimates, giving us some reassurance that they are reasonably representative. It is possible our study suffered selection bias; patients suffering poorer mental health may have been more motivated to participate, thus inflating our prevalence estimates. Fortunately, we achieved a consent rate of 55% implying our data represent the majority of our sample, thereby reducing this potential for sampling error. Another potential limitation is that we did not exclude participation in pulmonary rehabilitation as a possible confounder, which previous evidence suggests can be effective for reducing anxiety and depression in COPD patients.^36,37 Cheung et al. and Nowak et al. also failed to control for this, as such future studies should ensure that pulmonary rehabilitation is treated as a covariate.

Clinical implications

Since Question 4 from the HADS-D appears to be a universal symptom of COPD patients and our data suggests the removal of this question and lowering the cut-off to ≥5 provides superior sensitivity and specificity, we recommend this course of action when using the HADS-D specifically with people diagnosed with COPD. As the HADS-A provided fair sensitivity and specificity and no items appeared to overlap with the symptomatology of COPD, we recommend retaining the standard cut-off point of ≥8 for the HADS-A until such time as future studies suggest otherwise. Our results therefore support current guidelines of routine use of the HADS as a screening instrument for COPD patients, retaining the traditional HADS-A cut-off score of ≥8 whilst removing Question 4 for the HADS-D and using a lower cut-off score of ≥5.

Our investigation into the validity of screening tools for use in a clinical setting further highlights the importance of appropriate follow-up measures for those that screen positive. However, due to the restrictive time frame of these measures, it is possible that patients may feel depressed without being flagged as suffering from clinical levels and should therefore also be considered for follow-up. Our study found that those clinically diagnosed with depression via the MINI and also screened as having depression symptomatology in both the HADS-D and BDI-II had significantly higher mean scores than those that did not screen positive. Whilst the proportion of true positive to false positive cases was too small to be able to draw any definitive conclusions, another avenue for future research which has very important implications for targeted screening and treatment may be to investigate the demographic and physiological differences (e.g. forced expired volume in one second, six-minute walk test, number of exacerbations) in COPD patients that may help to distinguish between these two subgroups.