Validity and Reliability of Screening Measures for Depression and Anxiety Disorders in Rheumatoid Arthritis

Abstract

Objective

To test the validity and reliability of screening instruments for depression and anxiety in rheumatoid arthritis (RA).

Methods

Participants with RA completed the Patient Health Questionnaire (PHQ‐2 or PHQ‐9), the Patient Reported Outcomes Measurement Information System depression short form 8a and anxiety short form 8a, the Hospital Anxiety and Depression Scale anxiety score (HADS‐A) and depression score (HADS‐D), the Overall Anxiety Severity and Impairment Scale, the Generalized Anxiety Disorder 2‐ and 7‐item scales, and the Kessler‐6 scale. Clinical depression and anxiety disorders were confirmed using the Structured Clinical Interview for Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition, Axis I Disorders (SCID‐1) research version. We reported sensitivity, specificity, positive predictive value, and negative predictive value using SCID‐1 diagnoses as the criterion standard. Test–retest reliability was assessed with the intraclass correlation coefficient.

Results

Of 150 participants, 11.3% had SCID‐1–diagnosed depression, 7.3% had SCID‐1–diagnosed generalized anxiety disorder, and 19.3% had any SCID‐1–diagnosed anxiety disorder. For depression, sensitivity ranged from HADS‐D (cut point 11; 35%) to PHQ‐2 (88%) and PHQ‐9 (87%). Specificity ranged from PHQ‐9 (77%) and PHQ‐2 (84%) to HADS‐D (cut point 11; 94%). Positive predictive value ranged from 30% to 43%. Negative predictive value ranged from 92% to 98%. For generalized anxiety disorder, sensitivity ranged from HADS‐A (cut point 11; 45%) to HADS‐A (cut point 8; 91%). Specificity ranged from 81% to 89% for all measures except the HADS‐A (cut point 8; 63%). Intraclass correlation coefficient estimates ranging from 0.69 to 0.88 confirmed good test–retest reliability.

Conclusion

Depression screening instruments had good diagnostic performance; anxiety instruments were more variable. Identified depression and anxiety require clinical confirmation.

INTRODUCTION

Rheumatoid arthritis (RA) is a chronic immune‐mediated inflammatory disease. Despite improvements in therapy, affected individuals often experience persistent disease activity and functional limitations that adversely impact social participation and quality of life. The psychological impacts of chronic disease are significant. Several studies have reported rates of depression and anxiety in RA exceeding those in the general population 1, 2, 3. Mental illness is common at RA onset 4 and may precede diagnosis 5. Comorbid depression and anxiety complicate the assessment and management of RA due to overlapping physical symptoms and are associated with poor arthritis outcomes, including increased hospitalization and increased mortality 4, 6, 7, 8, 9, 10. Achieving disease control in RA requires early assessment and management 11, including management of comorbidities such as mental health disorders. To identify individuals potentially affected by mental health disorders, clinicians need access to valid, reliable, and brief screening instruments for these conditions.

SIGNIFICANCE & INNOVATIONS.

• Depression and anxiety are common in rheumatoid arthritis and contribute to adverse health outcomes, yet are often under‐recognized.

• The screening instruments for depression had good diagnostic performance; the instruments for anxiety were more variable.

• Compared to diagnoses determined by the Structured Clinical Interview for Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition, Axis I Disorders, some patient‐report instruments overestimated the prevalence of depression and anxiety.

Several screening instruments for depression and anxiety have been validated in the general population 12, 13, 14, 15, 16, 17, 18, 19. However, these scales often include somatic symptoms of depression, such as fatigue, pain, difficulty sleeping, and “getting going” in the morning, which are also common symptoms of RA. Similar issues arise when screening for anxiety. The misattribution of physical symptoms of RA to anxiety or depression may lead to overestimation and misclassification of these mental disorders, that is, to criterion contamination 20. Similarly, the physical symptoms of depression and anxiety may be incorrectly attributed to RA. Although some measures, such as the Hospital Anxiety and Depression Scale (HADS), were designed to avoid this misattribution, their performance in clinical settings still requires testing, because the strategy of avoiding potentially contaminated items may not improve performance.

Studies examining the psychometric performance of instruments for assessing depression and anxiety in RA are limited 21, 22, 23, 24. Most have compared different self‐administered instruments to each other or to measures of RA activity, but not to valid and reliable structured clinical assessments of psychiatric disorders. One study compared self‐reported questionnaires for depression to an interviewer‐administered symptom severity scale 21, 25. No studies have systematically assessed anxiety instruments as compared to clinical assessment based on a structured interview in RA, although anxiety is more prevalent in RA than depression 1, 23, 26. We aimed to evaluate the validity, reliability, and optimal cut point of multiple, commonly used, self‐report screening instruments for depression and anxiety for individuals with RA. We used the Structured Clinical Interview for Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition (DSM‐IV) Axis I Disorders (SCID‐1) as the criterion standard for assigning a diagnosis of a mental health disorder because it reliably assesses both depression and anxiety.

PATIENTS AND METHODS

As described elsewhere 8, from November 2014 through July 2016 we recruited individuals with physician‐confirmed RA, based on medical records review 27, who were ages ≥18 years, willing to participate for 3 years, and able to complete the questionnaires and interviews conducted in English. Participants were recruited through the Arthritis Centre clinic in Winnipeg, Manitoba and through community rheumatology and family medicine clinics. The University of Manitoba Health Research Ethics Board approved the study. All participants provided informed consent. Members of the Canadian Institutes of Health Research Team in Defining the Burden and Managing the Effects of Psychiatric Comorbidity in Chronic Immunoinflammatory Disease can be found in Appendix A.

At enrollment, participants completed paper questionnaires addressing their arthritis and mental health, trained assessors conducted a joint examination (28 tender and swollen joint counts), and medical records were reviewed for measures of arthritis severity. The trained interviewers administered the SCID‐1 28 the same day or within 2 weeks of enrollment. A convenience subgroup of 115 participants completed the screening instruments again within 2 weeks of initial administration for assessment of test–retest reliability.

Sociodemographic and arthritis characteristics

Participants self‐reported their sex, date of birth, ethnicity, and highest level of education attained. Race was categorized as white or visible minority status. Education was categorized as less than high school, high school, college, technical/trade school, and Bachelor's degree or higher. Arthritis‐specific measures used for descriptive purposes included age at RA diagnosis, physical function as measured by the modified Health Assessment Questionnaire, arthritis disease activity as measured by the Clinical Disease Activity Index 29, and current RA treatment with disease‐modifying antirheumatic drugs, biologics, or glucocorticoids.

Depression and anxiety screening instruments

Each participant completed the Patient Health Questionnaire (PHQ‐9) 12, from which we derived scores for the PHQ‐9 and PHQ‐2 13, the HADS 14, the Kessler‐6 distress scale 15, 16, the Patient Reported Outcomes Measurement Information System (PROMIS) depression short form 8a (PROMIS depression) 19, the Generalized Anxiety Disorder 7‐item scale, from which we derived scores for the GAD‐7 and GAD‐2 17, the Overall Anxiety and Severity Impairment Scale (OASIS) 18, and the PROMIS anxiety short form 8a (PROMIS anxiety) 19. We selected these measures based on their ease of use, availability for self‐administration, and lack of copyright restrictions for clinical use. The instruments all had acceptable face validity, as judged by the mental health clinicians on the team; these measures were considered to cover the concepts they purported to measure. Table 1 shows a summary of the scales studied. For each measure, higher scores indicate more severe symptoms.

Table 1.

Self‐administered instruments for depression and anxiety^a

Measure	Time period assessed	No. of items	Item response range	Standard cut point
Depression
PHQ‐2	Last 2 weeks	2	0–3	3
PHQ‐9	Last 2 weeks	9	0–3	10
HADS‐D	Last week	7	0–3	8 or 11
PROMIS depression	Last week	8	1–5	T score 60
Kessler‐6 scaleb	30 days	6	1–5	19
Anxiety
GAD‐2	Last 2 weeks	2	0–3	3
GAD‐7	Last 2 weeks	7	0–3	10
HADS‐A	Last week	7	0–3	8 or 11
OASIS	Last week	5	0–4	8
PROMIS anxiety	Last week	8	1–5	T score 60

^aPHQ‐2 = Patient Health Questionnaire 2; HADS‐D = Hospital Anxiety and Depression Scale depression score; PROMIS depression = Patient Reported Outcomes Measurement Information System depression short form 8a; GAD‐2 = Generalized Anxiety Disorder 2; HADS‐A = HADS anxiety score; OASIS = Overall Anxiety and Severity Impairment Scale; PROMIS anxiety = PROMIS anxiety short form 8a.

^bKessler‐6 scale is a measure of general psychological distress.

Assessment of pain and fatigue

Participants also completed the Fatigue Impact Scale for Daily Use, a validated 8‐item fatigue instrument, scored from 0 (no problem) to 4 (extreme problem) for each item 30. Pain was assessed using the Medical Outcomes Study modified pain effects scale, a reliable and valid instrument with scores ranging from 6 to 30, with higher scores indicating greater pain 31.

Diagnostic interview

The SCID‐1 is a semistructured interview to identify anxiety, mood, and substance use disorders using DSM‐IV criteria 25. Clinical diagnoses of major depression and generalized anxiety disorder based on the SCID‐1 were considered the reference standards for assessing criterion validity of the self‐reported instruments. Complementary analysis used any anxiety disorder (including generalized anxiety disorder, panic disorder, social phobia, specific phobia, anxiety disorder due to general medical condition, or anxiety disorder due to substance use) as the reference standard. SCID‐1 interviews were conducted by graduate students in clinical psychology, nurses, and research coordinators. Interviewers were trained and supervised by a clinical psychologist with extensive expertise in the SCID‐1. Training consisted of a detailed review of the SCID‐1 users guide and modules, watching example SCID‐1 interviews, role‐playing SCID‐1 interviews, and being monitored while conducting SCID‐1 interviews. Interviewers met regularly to review interviews and ensure consistency of conducting interviews. The clinical psychologist met with the interviewers, periodically reviewed the SCID‐1 scoring documents, and was available to discuss any responses that raised immediate health concerns 8. Interviewers were blinded to the results of the screening instruments.

Statistical analysis

Characteristics of participants were summarized using frequency (%) for categorical variables, and mean ± SD or median (interquartile range) for continuous variables. Pairwise deletion was used for missing data since the amount was minimal. According to the taxonomy proposed by the Consensus‐Based Standards for the Selection of Health Measurement Instruments 32, we assessed criterion validity, construct validity (through hypothesis testing), content validity, internal consistency reliability, and test–retest reliability of the selected screening instruments.

Criterion validity designates how well the scores of the screening instruments reflect the reference (criterion) standard 32. First, we reported depression and anxiety status based on the SCID‐1 (criterion standard for depression and generalized anxiety disorder) and then based on self‐reported screening instruments. Based on published cut points for depression/anxiety for these scales (Table 1), we computed sensitivity, specificity, positive predictive value (PPV), negative predictive value, and accuracy with 95% confidence intervals (95% CIs) for the screening instruments as compared to the criterion standard. Accuracy is the instrument's ability to distinguish between affected and nonaffected individuals, estimated as (true positive + true negative)/(true positive + true negative + false positive + false negative).

Second, we identified the best cut point for predicting depression or anxiety (or general psychological distress) using receiver operating curve (ROC) analysis, because these cut points could be different for the RA population than for the general population 33. We calculated the optimal cut point by maximizing Youden's J index (sensitivity + specificity – 1) 34. We compared the area under the curve (AUC) between screening instruments using binary logistic regression, separately for depression and anxiety scales.

Construct validity measures the degree to which the measure of interest correlates with measures of other variables in hypothesized ways. We calculated Spearman's rank correlations (with 95% CIs) between the total scores on the screening instruments with participant‐reported fatigue and pain, expecting to find moderate correlations (convergent validity). Very strong correlations would suggest criterion contamination. We calculated Spearman's rank correlations between the total scores on the screening instruments and age, expecting weaker correlations (divergent validity). Correlations (positive or negative) of 0.10–0.29 were considered weak, 0.30–0.49 moderate, and 0.50–1.00 strong 35.

Internal consistency reliability is the degree to which items are interrelated and was assessed for each screening instrument using Cronbach's alpha 36. Test–retest reliability refers to the reproducibility of the scores over time among respondents who are stable. We measured test–retest reliability for the 2 administrations of the instruments using the intraclass correlation coefficient (ICC) with 95% CI. Correlation coefficient values were classified as poor (<0.50), moderate (0.50–0.75), good (>0.75–0.90), or excellent (>0.90) 37.

Complementary analysis

In a complementary analysis, we repeated the criterion validity analysis for the anxiety scales using any SCID‐1–diagnosed anxiety disorder as the criterion standard. Statistical analyses were performed with SAS software, version 9.4.

RESULTS

We enrolled 154 individuals with RA, of whom 153 completed the SCID‐1. We included the 150 participants who completed the SCID‐1 within 2 weeks of enrollment, of whom 42 (28.0%) completed the SCID‐1 the day of enrollment. Most participants were women with education of high school or above and with moderate RA activity (Table 2). Of the 150 participants, 115 (76.7%) repeated the clinical screening instruments of interest 2 weeks later to assess test–retest reliability.

Table 2.

Demographic and clinical characteristics of study participants (n = 150)^a

Characteristic	Value
Sex, no. (%)
Female	127 (84.7)
Male	23 (15.3)
Age at enrollment, years	59.8 ± 11.7
Race, no. (%)
White	114 (76.0)
Other	36 (24.0)
Highest level of education, no. (%)
Less than high school	13 (8.7)
High school/GED	35 (23.3)
College	45 (30.0)
Technical/trade	19 (12.7)
University Bachelor's degree or higher	38 (25.3)
Time from enrollment to SCID‐1 completion, weeks	0.11 ± 0.15
Age at RA diagnosis (self‐reported), years	41.7 ± 14.9
RA features
Swollen joints	1.30 ± 2.55
Tender joints	4.65 ± 6.83
CDAI	10.2 ± 10.6
M‐HAQ	0.51 ± 0.51
Any RA drug use, no. (%)b	138 (92.0)
DMARD	126 (84.0)
Biologic	50 (33.3)
Prednisone	20 (13.3)

^aValues are the mean ± SD unless indicated otherwise. GED = General Education Diploma; SCID‐1 = Structured Clinical Interview for Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition, Axis I Disorders; RA = rheumatoid arthritis; CDAI = Clinical Disease Activity Index; M‐HAQ = modified Health Assessment Questionnaire; DMARD = disease‐modifying antirheumatic drug.

^bDMARD, biologic, or prednisone. Without prednisone the value is 136 (90.7).

Using the SCID‐1, 11.3% of participants were classified as currently depressed. The proportion classified as having clinically significant depressive symptoms by the depression screening instruments varied and exceeded the proportion identified as depressed on the SCID‐1, except for the HADS‐D when using a cut point of 11 (Table 3). The HADS‐D (cut point 11) most closely approximated the prevalence of SCID‐diagnosed major depression at 9.3%. Using the SCID‐1, 19.3% of participants were classified as having any current anxiety disorder, while 7.3% were classified as having a generalized anxiety disorder. The proportion classified as having clinically significant anxiety by the anxiety screening instruments varied and exceeded the proportion identified as anxious on the SCID‐1, except for the HADS‐A when using a cut point of 11 (Table 3). The HADS‐A at a cut point of 11 most closely approximated the prevalence of SCID‐1–diagnosed anxiety at 13.4%.

Table 3.

Test characteristics for previously defined cut points for depression and anxiety screening measures^a

Instrument	Cut point ≥	Affected, no. (%)	Sensitivity (95% CI)	Specificity (95% CI)	PPV (95% CI)	NPV (95% CI)	Accuracy (95% CI)
Depression
PHQ‐2	3	36 (24.2)b	0.88 (0.63, 0.98)	0.84 (0.77, 0.90)	0.42 (0.25, 0.59)	0.98 (0.94, 0.98)	0.84 (0.78, 0.90)
PHQ‐9	10	43 (29.7)c	0.87 (0.62, 0.98)	0.77 (0.69, 0.84)	0.32 (0.19, 0.48)	0.98 (0.93, 0.99)	0.79 (0.71, 0.85)
HADS‐D	8	33 (22.0)	0.59 (0.33, 0.82)	0.83 (0.75, 0.89)	0.30 (0.16, 0.49)	0.94 (0.88, 0.98)	0.80 (0.73, 0.86)
HADS‐D	11	14 (9.3)	0.35 (0.14, 0.62)	0.94 (0.88, 0.97)	0.43 (0.18, 0.71)	0.92 (0.86, 0.96)	0.87 (0.81, 0.92)
PROMIS depression	T score 60	36 (24.0)	0.76 (0.50, 0.93)	0.83 (0.75, 0.89)	0.36 (0.20, 0.54)	0.96 (0.91, 0.99)	0.82 (0.75, 0.88)
Kessler‐6 d	19	133 (88.9)	0.35 (0.14, 0.62)	0.96 (0.91, 0.99)	0.55 (0.23, 0.83)	0.92 (0.86, 0.96)	0.89 (0.83, 0.94)
Anxiety
GAD‐2	3	31 (20.7)b	0.73 (0.39, 0.94)	0.83 (0.76, 0.89)	0.26 (0.12, 0.45)	0.97 (0.93, 0.99)	0.82 (0.75, 0.88)
GAD‐7	10	27 (18.1)b	0.64 (0.31, 0.89)	0.86 (0.78, 0.91)	0.26 (0.11, 0.46)	0.97 (0.92, 0.99)	0.84 (0.77, 0.89)
OASIS	8	34 (22.7)	0.82 (0.48, 0.98)	0.82 (0.75, 0.88)	0.26 (0.13, 0.44)	0.98 (0.94, 0.99)	0.82 (0.75, 0.88)
HADS‐A	8	61 (40.9)b	0.91 (0.59, 0.99)	0.63 (0.54, 0.71)	0.16 (0.08, 0.28)	0.99 (0.94, 1.0)	0.65 (0.57, 0.73)
HADS‐A	11	20 (13.4)b	0.45 (0.17, 0.77)	0.89 (0.83, 0.94)	0.25 (0.09, 0.49)	0.95 (0.90, 0.98)	0.86 (0.79, 0.91)
PROMIS anxiety	T score 60	34 (22.7)	0.73 (0.39, 0.94)	0.81 (0.74, 0.87)	0.24 (0.11, 0.41)	0.97 (0.93, 0.99)	0.81 (0.73, 0.87)

^aThe criterion standard is the generalized anxiety disorder based on the Structured Clinical Interview for Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition, Axis I Disorders. 95% CI = 95% confidence interval; PPV = positive predictive value; NPV = negative predictive value; PHQ‐2 = Patient Health Questionnaire 2; HADS‐D = Hospital Anxiety and Depression Scale depression score; PROMIS depression = Patient Reported Outcomes Measurement Information System depression short form 8a; GAD‐2 = Generalized Anxiety Disorder 2; OASIS = Overall Anxiety and Severity Impairment Scale; HADS‐A = HADS anxiety score; PROMIS anxiety = PROMIS anxiety short form 8a.

^b1 patient missing.

^c5 patients missing.

^dKessler‐6 scale is a measure of general psychological distress.

Criterion validity

Table 3 shows performance of the depression measures based on the typically recommended cut points. Sensitivity was highest for the PHQ‐2 (88%) and the PHQ‐9 (87%) and lowest for the HADS‐D with a cut point of 11 (35%). Specificity was highest for the HADS‐D (cut point 11) at 94%, followed by the PHQ‐2, HADS‐D (cut point 8), and PROMIS depression (all 83–84%). Specificity was lowest for the Kessler‐6 (5%). Figure 1A shows the ROC curves for depression instruments. The AUC did not differ between the PHQ‐2 (AUC 0.93 [95% CI 0.87, 0.98]) and the PHQ‐9 (AUC 0.90 [95% CI 0.84, 0.97]; P = 0.34), PROMIS depression (AUC 0.92 [95% CI 0.87, 0.97]; P = 0.83), or Kessler‐6 (AUC 0.90 [95% CI 0.83, 0.97]; P = 0.46). However, the AUC of the PHQ‐2 was significantly higher than that of the HADS‐D (AUC 0.84 [95% CI 0.75, 0.93]; P = 0.008). The AUC of the PHQ‐9 (P = 0.02) and the PROMIS depression (P = 0.04) were also significantly higher than that of the HADS‐D.

Figure 1.

Receiver operating characteristic curves for screening measures as compared to the Structured Clinical Interview for Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition, Axis I Disorders. A, Depression; B, Generalized anxiety disorder. PHQ‐2 = Patient Health Questionnaire 2; AUC = area under the curve; PROMIS = Patient Reported Outcomes Measurement Information System; Kessler‐6 = Kessler‐6 scale (a measure of general psychologic distress); HADS‐D = Hospital Anxiety and Depression Scale depression score; OASIS = Overall Anxiety and Severity Impairment Scale; GAD‐2 = Generalized Anxiety Disorder 2; HADS‐A = HADS anxiety score.

Based on the ROC analysis, the optimal cut points for some of the depression screening instruments differed from those routinely recommended (see Supplementary Table 1, available on the Arthritis Care & Research website at http://onlin​elibr​ary.wiley.com/doi/10.1002/acr.24011/​abstract). Specifically, optimal cut points were 7 for the HADS‐D, 57.7 for the PROMIS depression, and 11 for the Kessler‐6.

Table 3 shows performance of the anxiety scales based on the typically recommended cut points and using generalized anxiety disorder as the criterion standard. Sensitivity was highest for the HADS‐A (cut point 8) at 91% and lowest for the HADS‐A (cut point 11) at 45%. Specificity ranged from 81% to 89% for all measures except the HADS‐A (cut point 8) at 63%. The ROC curves for anxiety instruments are shown in Figure 1B. The AUC did not differ between the HADS‐A (AUC 0.82 [95% CI 0.69, 0.94]) and the GAD‐7 (AUC 0.84 [95% CI 0.75, 0.94]; P = 0.59), GAD‐2 (AUC 0.84 [95% CI 0.73, 0.94]), PROMIS anxiety (AUC 0.86 [95% CI 0.74, 0.97]; P = 0.54), or OASIS (AUC 0.88 [95% CI 0.77, 0.99]; P = 0.42).

For generalized anxiety disorder, the ROC curve analysis suggested that the optimal cut points for some of the anxiety screening instruments differed from those recommended (see Supplementary Table 1, available on the Arthritis Care & Research website at http://onlin​elibr​ary.wiley.com/doi/10.1002/acr.24011/​abstract). The optimal cut points were 9 for the GAD‐7, 9 for the HADS‐A, and 55.4 for the PROMIS anxiety scale.

In the complementary analysis for anxiety disorders using any anxiety disorder as the criterion standard, sensitivities were lower than for the analysis using generalized anxiety disorder as the criterion standard, as were the AUC estimates (see Supplementary Tables 2 and 3, available at http://onlin​elibr​ary.wiley.com/doi/10.1002/acr.24011/​abstract). The optimal cut points for any anxiety disorder were 6 for the GAD‐7, 6 for the OASIS, 9 for the HADS‐A and a T score of 56.4 for the PROMIS anxiety scale (see Supplementary Table 2, available at http://onlin​elibr​ary.wiley.com/doi/10.1002/acr.24011/​abstract).

Construct validity

Higher scores on all depression instruments except the Kessler‐6, and on all anxiety instruments, were moderately associated with higher pain and fatigue scores (Table 4). As expected, age was not strongly associated with these measures.

Table 4.

Construct validity: Spearman's correlation coefficients for anxiety and depression measures with pain, fatigue, and age (n = 141)^a

Measure	Pain	Fatigue	Age
Depression
PHQ‐2	0.58 (0.46, 0.68)	0.49 (0.36, 0.61)	–0.17 (–0.32, –0.0006)
PHQ‐9	0.71 (0.62, 0.78)	0.67 (0.57, 0.75)	–0.24 (–0.39, –0.08)
PROMIS depression	0.62 (0.51, 0.72)	0.63 (0.52, 0.72)	–0.22 (–0.37, –0.05)
HADS‐D	0.71 (0.62, 0.78)	0.68 (0.59, 0.76)	–0.19 (–0.35, –0.029)
Kessler‐6 scale b	0.66 (0.56, 0.75)	0.64 (0.53, 0.73)	–0.23 (–0.38, –0.067)
Anxiety
OASIS	0.61 (0.50, 0.71)	0.64 (0.53, 0.73)	–0.25 (–0.40, –0.085)
GAD‐2	0.49 (0.36, 0.60)	0.51 (0.38, 0.62)	–0.18 (–0.33, –0.014)
GAD‐7	0.58 (0.46, 0.68)	0.60 (0.49, 0.70)	–0.23 (–0.38, –0.068)
PROMIS anxiety	0.61 (0.49, 0.70)	0.63 (0.53, 0.72)	–0.23 (–0.38, –0.064)
HADS‐A	0.51 (0.38, 0.62)	0.60 (0.48, 0.70)	–0.21 (–0.37, –0.05)

^aValues are the correlation coefficient (95% confidence interval). PHQ‐2 = Patient Health Questionnaire‐2; PROMIS depression = Patient Reported Outcomes Measurement Information System depression short form 8a; HADS‐D = Hospital Anxiety and Depression Scale depression score; OASIS = Overall Anxiety and Severity Impairment Scale; GAD‐2 = Generalized Anxiety Disorder 2; PROMIS anxiety = PROMIS anxiety short form 8a; HADS‐A = HADS anxiety score.

^bKessler‐6 scale is a measure of general psychological distress.

Reliability

All depression and anxiety instruments had acceptable internal consistency and reliability as measured by Cronbach's alpha (Table 5). Of the depression instruments, the PROMIS depression scale had the highest internal consistency reliability (0.97), while the HADS‐D had the lowest (0.84). Of the anxiety instruments, the PROMIS anxiety tool and the OASIS (0.93) had the highest internal consistency reliability, while the GAD‐2 (0.69) had the lowest. Test–retest reliability, as measured by the ICC, ranged from 0.84 (HADS‐D) to 0.88 (PROMIS depression) for the depression scales. For the anxiety scales, the ICC ranged from 0.69 (GAD‐2) to 0.83 (OASIS) (Table 5).

Table 5.

Reliability of anxiety and depression measures^a

Instrument	Internal consistency reliability (95% CI)	Test–retest reliability ICC (95% CI)
Depression
PHQ‐2	0.86 (0.63, 1.0)	0.86 (0.81, 0.90)
PHQ‐9	0.89 (0.80, 0.97)	0.86 (0.80, 0.90)
PROMIS depression	0.97 (0.88, 1.0)	0.88 (0.83, 0.92)
HADS‐D	0.84 (0.74, 0.94)	0.85 (0.79, 0.90)
Kessler‐6 scale b	0.88 (0.77, 0.99)	0.84 (0.77, 0.89)
Anxiety
OASIS	0.93 (0.82, 1.0)	0.83 (0.76, 0.88)
GAD‐2	0.69 (0.58, 1.0)	0.69 (0.61, 0.77)
GAD‐7	0.90 (0.81, 1.0)	0.81 (0.71, 0.86)
PROMIS anxiety	0.93 (0.84, 1.0)	0.82 (0.75, 0.87)
HADS‐A	0.84 (0.74, 0.94)	0.79 (0.71, 0.85)

^a95% CI = 95% confidence interval; ICC = intraclass correlation coefficient; PHQ‐2 = Patient Health Questionnaire 2; PROMIS depression = Patient Reported Outcomes Measurement Information System depression short form 8a; HADS‐D = Hospital Anxiety and Depression Scale depression score; OASIS = Overall Anxiety and Severity Impairment Scale; GAD‐2 = Generalized Anxiety Disorder 2; PROMIS anxiety = PROMIS anxiety short form 8a; HADS‐A = HADS anxiety score.

^bKessler‐6 scale is a measure of general psychological distress.

DISCUSSION

In a cohort of individuals with established RA, we examined the validity and reliability of several commonly used self‐report instruments for anxiety and depression. Using the criterion standard of the SCID‐1 to confirm clinical diagnosis, the prevalence of current depression was 11.3%, generalized anxiety disorder was 7.3%, and any anxiety disorder was 19.3%. The diagnostic performance of all the depression scales was remarkably similar and generally good based on ROC analysis, but not excellent (AUC <0.90). The diagnostic performance of the anxiety scales was also similar across instruments, but not as good as that of the depression scales; performance was better for identifying generalized anxiety disorder than for any anxiety disorder. This finding may reflect the fact that any anxiety disorder includes panic disorder or phobias in which symptoms are limited to specific circumstances. Internal consistency reliability, construct validty, and test–retest reliability were acceptable for all screening instruments.

Our study complements and expands work from Englbrecht et al 21, which validated screening instruments for depression against the Montgomery‐Asberg Depression Rating Scale (MADRS) severity rating scale in individuals with RA 21, 25. The MADRS, a clinician‐rated severity scale of patient symptoms, minimizes the assessment of somatic symptoms and has been proposed as a useful tool for conditions such as RA. In contrast, we used the SCID‐1 as the criterion standard. The SCID‐1 is a semistructured interview, and although it requires more time and greater expertise to administer, it has the advantage of assessing anxiety and depression and is the best approximation of a gold standard 38. In these 2 studies, the cohorts had similar demographics and RA disease activity indicators, and the prevalence of depression identified using the MADRS and the SCID‐1 in these studies was similar, albeit slightly lower in our cohort (MADRS score >12 18.5%; SCID‐1 11.3%) 21, 25. The PHQ‐9 was the only self‐reported depression instrument assessed in both studies, and despite the differences in the criterion standard used to clinically diagnose depression, it had similar sensitivity and specificity.

We specifically validated the PROMIS scales for depression and anxiety against formally assessed mental health diagnoses in RA patients, complementing previous work that evaluated the convergent validity and reliability of the PROMIS scales in RA 22. The PROMIS scales were developed using contemporary rigorous psychometric methods 39, and normative population data are available; therefore these scales are increasingly employed in research and clinical settings. We found that the PROMIS scales had the highest internal consistency reliability, and high test–retest reliability, but their optimal cut points for identifying clinically significant depression or anxiety in individuals with RA were lower than those of the general population. Regardless, the PROMIS scales for depression and anxiety were valid for use in RA.

Since anxiety and depression commonly coexist, we also assessed the performance of the Kessler‐6 distress scale, a general measure of psychological distress due to depression, anxiety, or other mental illness. The Kessler‐6 has been validated in populations seeking mental health treatment but did not perform as well in this RA population. Targeted instruments for depression and anxiety appear better suited for screening in the RA population, as management approaches differ greatly between anxiety and depression diagnoses. Distinguishing depression from anxiety could be particularly relevant in environments with limited mental health resources if clinicians opt to direct patients to online educational resources while awaiting formal assessment with a mental health provider.

The performance of the depression and anxiety screening instruments that we found in RA is similar to that found in other chronic immune‐mediated conditions, such as multiple sclerosis (MS) 40 and inflammatory bowel disease (IBD) 41. Like RA, MS and IBD also have increased rates of depression and anxiety compared to the general population 42, 43, 44, and similar concerns exist with screening instruments for identifying mental health disorders. As we found in RA, the optimal cut points for depression and anxiety screening instruments found in IBD 41, MS 40, and other chronic conditions 45 have differed from those for the general population; this finding suggests that adjustments to the cut points used for standardized screening instruments are required when these scales are applied to populations with specific chronic health conditions. However, these optimal cut points were determined on the basis of both sensitivity and specificity, while the risks of false negatives (missed cases) and false positives (leading to unnecessary assessments) must also be considered when choosing the most suitable instrument and cut point to use in each specific clinical context.

Compared to semistructured interviews such as the SCID‐1, which requires highly trained personnel to administer, self‐reported symptom questionnaires are more feasible for a rheumatology practice or research study because they can be completed by patients during the clinic visit, report a range of clinical severity, and when administered over repeated visits may be sensitive to the effect of interventions. The optimal choice of a screening instrument to use in the clinic or in research studies is determined by its reliability and validity in RA patients, its ease of use by patients and clinicians, and by its availability, with minimal or no cost. Of the depression instruments studied, the PHQ‐2, which has only 2 questions, may be the easiest to implement clinically. However, in this study, the PHQ‐2 was extracted from the PQH‐9, and we have not confirmed how well it performs independent of administering the remaining items. The self‐report instruments for anxiety were less accurate than those for depression but still performed adequately. Anxiety was most accurately assessed using the HADS‐A at a cut point of 11. However, at cut points where the HADS‐A is highly sensitive (cut point 11), it did not preserve the high specificity that would normally justify its use, given associated usage fees. The overall performance of the HADS‐A was similar to the other instruments tested that are in the public domain, such as the GAD‐7, OASIS, and PROMIS anxiety. The 2‐question GAD‐2 extracted from the GAD‐7 is easy to administer in clinical practice, but relatively poor internal consistency and reliability may limit its utility.

While any of the instruments assessed could be used in a rheumatology clinic to identify elevated depression or anxiety symptoms, the PHQ‐9 or PHQ‐2 (depression) and OASIS, GAD‐7 or GAD‐2 (general anxiety disorder), are briefest. Although the PHQ‐2 and GAD‐2 were not validated in isolation, clinicians can ask these brief questions as a quick clinical screen, thereby incorporating depression and anxiety screening into routine clinical practice without the requirement of extensive additional documentation. Further, for generalists who see patients with a range of chronic immune‐mediated inflammatory diseases, these measures also performed well in individuals with MS and IBD.

Our study has limitations. Participants were recruited from academic outpatient rheumatology, community rheumatology, and family medicine clinics located in the same region. Although the demographic characteristics of our sample are representative of most general RA populations as discussed previously 8, our findings may not apply to other settings. For instance, the participants were predominantly female, white, and highly educated, with moderate disease activity and relatively preserved physical function. Most were receiving RA treatment, but publicly funded health care in our region affects access to biologic therapies and mental health supports. We cannot exclude participant bias, and because SCID‐1 evaluations were conducted only once, we cannot assess sensitivity to change. A subgroup completed questionnaires again within 2 weeks to assess test–retest reliability, and because no interventions were conducted between assessments, significant interval changes in mental health were unlikely. Whether the administration of multiple instruments at the same visit influenced participant responses is unknown. Our choice of instruments was not exhaustive but was based on ease of administration, applicability across other chronic diseases, and availability; other instruments may perform better. Finally, despite good overall accuracy of the screening instruments, low PPV highlights the need for clinical confirmation when elevated scores are identified and the need for caution when interpreting rates of depression and anxiety based solely on self‐report symptom scales.

Depression and anxiety are common in RA yet often under‐recognized. The adverse impact of psychiatric comorbidity on arthritis outcomes is increasing recognized 4, 6, highlighting the need for feasible screening instruments to identify individuals at risk for mental health disorders or who have high levels of anxiety or depression symptoms that warrant attention and that may affect outcomes, regardless of whether or not the level of symptoms meet criteria for a disorder 46. Clinical expertise is still required for formal diagnosis of mental health disorders, but our findings suggest that self‐administered screening questionnaires for depression and anxiety can identify individuals who may benefit from referral to a mental health specialist for further assessment and management. However, to be effective, screening must translate into better outcomes in a way that offsets the investment of resources required to confirm the diagnoses. If screening is introduced into settings where resources are strained, the extra burden of assessing individuals who do not need additional intervention (false positives) can divert resources away from those who need them more. Thus, the optimal choice of screening instrument, and optimal cut point, may vary depending on the situation and purpose of administration. Regardless, incorporation of screening tools for depression and anxiety into clinical practice may improve outcomes for patients with RA.

AUTHOR CONTRIBUTIONS

All authors were involved in drafting the article or revising it critically for important intellectual content, and all authors approved the final version to be submitted for publication. Dr. Hitchon had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

Study conception and design

Hitchon, Lix, Graff, Fisk, Patten, Bolton, Sareen, El‐Gabalawy, Marriott, Bernstein, Marrie.

Acquisition of data

Hitchon, Graff, Bolton, Sareen, Bernstein, Marrie.

Analysis and interpretation of data

Hitchon, Zhang, Peschken, Lix, Graff, Fisk, Patten, Bolton, Sareen, El‐Gabalawy, Marriott, Bernstein, Marrie.

Supporting information

 

APPENDIX A. CANADIAN INSTITUTES OF HEALTH RESEARCH TEAM IN DEFINING THE BURDEN AND MANAGING THE EFFECTS OF PSYCHIATRIC COMORBIDITY IN CHRONIC IMMUNOINFLAMMATORY DISEASE

Members of the Canadian Institutes of Health Research Team in Defining the Burden and Managing the Effects of Psychiatric Comorbidity in Chronic Immunoinflammatory Disease are as follows: Ruth Ann Marrie, Charles N. Bernstein, Lindsay Berrigan, James M. Bolton, Renée El‐Gabalawy, John D. Fisk, Lesley A. Graff, Carol A. Hitchon, Alan Katz, Lisa M. Lix, James Marriott, Scott B. Patten, Christine Peschken, Jitender Sareen, Scott B. Patten, Alexander Singer, John R. Walker, and Ryan Zarychanski.