Abstract
The focus on recovery, not just symptom reduction, in mental health care brings a need for psychometrically sound measures of recovery. This study examined the factor structure and sensitivity to change of a common measure of mental health recovery, the Recovery Assessment Scale (RAS). We conducted a secondary data analysis from a randomized clinical trial of self-management for depression (n=302). We tested both bifactor and the previously found five factor model. Sensitivity to change was examined three ways: 1) between the intervention and control group; 2) across time in the intervention group; and 3) in those whose depression remitted. The previous five factor model was supported. One subscale, no domination by symptoms, was particularly sensitive to change and showed sensitivity to change whereas the subscale reliance on others did not show change in any of the comparisons. Results suggest the subscales of the RAS should be examined separately in future studies of recovery.
Keywords: recovery, mental health services, outcomes, depressive symptoms, major depression
Introduction
Historically, the treatment of depression and other mental disorders has focused on reducing symptoms and decreasing the negative experiences of mental illness. However, there is increasing recognition of the importance of incorporating a recovery model of care into both treatment and research.1,2 Recovery models focus on increasing positive experiences and helping clients live a meaningful life instead of a sole focus on reducing symptoms.2 However, in order to be more recovery focused, psychometrically sound measures of recovery are required.
One of the most commonly used measures of recovery in mental illness research is the Recovery Assessment Scale.3 The scale consists of 41 statements, each positively worded, and participants rate their agreement with each item. Factor analyses showed a five factor structure with only 24 of the items loading on the factors,4 so only those 24 items are scored in subscale analysis. The five factors or subscales are personal confidence and hope, willingness to ask for help, having a goal and success orientation, better reliance on others and not being dominated by the symptoms of the mental illness. Although the five factor structure of the RAS is well supported, a bifactor model has not been examined. A bifactor model includes a general factor that affects all items and accounts for the correlations between factors.5 As depicted in Figure 1a, the bifactor model includes a general factor of all scored items. Additional factors are included for each of the subscales, the five RAS subscales in this case. No correlation is included between the subscales because the general factor accounts for any bivariate correlation between subscales. This differs from the five factor model, depicted in Figure 1b, that models the associations between factors directly without a general factor. An advantage of knowing whether a five factor or bifactor model fits the RAS is that the five factor model supports use of the subscales while the bifactor model supports the use of both subscales and total scores. Although total scores that include all the subscales are often used, rarely is a bifactor model investigated to determine if the total score is appropriate to use or if the subscales should be the main focus in research and clinical settings.
Figure 1. Diagram of the bifactor (1a) and five factor (1b) models.
The bifactor model includes a general factor along with an additional factor for each subscale. The general factor accounts for the correlation between subscales. The five factor model only includes the five subscales with correlations between subscales.
The RAS has shown good reliability and validity further supporting its continued use in mental health research. Fourteen-day test-retest reliability was high (0.88) as was internal consistency (0.93) in one of the original evaluation studies.6 The RAS also correlates positively with other measures of positive outcomes such as self-esteem and quality of life6 as well as other recovery-based measures,7 supporting its validity. While the RAS has excellent psychometric properties overall, one omission from the literature is the RAS’s sensitivity to change. Sensitivity to change is the ability of a measure to show change after an intervention, across time or between groups that should theoretically differ on the measure.8 Being sensitive to change is particularly important for measures used in clinical trials and treatment settings as they may reflect measurement problems rather than true success or failure of the treatment if the measure is not sensitive to change.
The current study sought to expand our knowledge of the psychometric properties of the RAS. Secondary analysis of data from a randomized controlled trial of a self-management support services intervention for chronic depression was used.9 The randomized trial showed the intervention improved the total RAS score and reduced depressive symptoms and major depressive episodes. For this study, we examined both factor structure and sensitivity to change of the RAS. For factor structure, we tested the five factor model and the bifactor model. We examined sensitivity to change three ways: comparing the intervention and control group; comparing baseline and follow-up scores from people with a diagnosis of depression at baseline but no diagnosis at follow-up; and comparing baseline and follow-up scores in the intervention group alone. Both the total score and the subscales were analyzed for sensitivity to change.
Methods
Participants and procedures
This study was a secondary analysis of a clinical trial testing a depression self-management support service intervention against a treatment as usual control group.9 To be eligible, potential participants had to be 18 years of age or older, have seen a doctor for antidepressants at least twice in the past year, be diagnosed with recurrent depression or dysthymia and have residual symptoms of depression. Exclusion criteria were current participation in any other depression treatment trial, plans to move out of the study area and diagnosis of bipolar disorder or psychotic disorder. Participants were not excluded if they had comorbid anxiety, substance use or personality disorders. Participants (n=302) were recruited from four primary care clinics and a local medical center using two recruitment methods. The first method identified potentially eligible participants through electronic health records and sent an invitation letter to the potential participants. The second method gave medical providers information and brochures about the study so they could refer patients. All participants provided informed consent and the institutional review boards of the study centers approved the procedures before the study was conducted.
After participants were accrued, they completed a baseline assessment. They were then randomized to either the self-management intervention or the treatment-as-usual control group. The intervention lasted for 18 months and consisted of regular contacts with a mental health professional functioning as a care manager and skills groups that included both recovery-oriented and cognitive-behavioral strategies as well as strategies to improve motivation and engagement in treatment. Groups met weekly for the first two and a half months, twice monthly for the next two months and then monthly during the maintenance phase or the remaining months. A mental health profession and peer specialist led the groups and the peer specialists also provided brief contacts, as needed, that focused on recovery goals. The care manager also contacted participants at least monthly during the initial year of study participation to support participants’ practice of self-care strategies and assess symptoms and progress in meeting their treatment goals. Contacts after the first year depended on symptom level, and engagement with other providers. Care managers also provided reports to other providers and coordinated follow-up care. Assessments were completed at three, six, 12 and 18 months after the baseline assessment.
Measures
Recovery Assessment Scale (RAS)
The RAS consists of 41 items, each answered on a 5-point Likert scale ranging from 1 (strongly disagree) to 5 (strongly agree). Only 24 of the items are included in scoring and averaging the items creates the scores for the total scale and the subscales: confidence and hope; willingness to ask for help; goal and success orientation; reliance on others; and no domination by symptoms.
Structured Clinical Interview for the DSM (SCID)
The SCID10 is an interviewer-administered measure used to make diagnoses of psychiatric disorders from the Diagnostic and Statistical Manual of Mental Disorders.11 Interviewers completed training in administering the SCID before interviewing participants. For this analysis, only the diagnosis of Major Depression was used.
Statistical analysis
Factor analysis
Data from the baseline assessment was used for the factor analyses. A confirmatory approach was used for the factor analyses as the factor structure of the RAS has already been investigated. However, to add to the literature on the RAS, we tested a bifactor model as well as the previously supported five factor model. A bifactor uses a general factor, which affects all items, to explain the correlation between the other factors.5 A bifactor model can support the use of both the total scale score as well as the subscale scores. However, if the five factor model fits better, this would suggest a focus on the subscales instead of the total score. We therefore tested a bifactor model and the five factor model using Lisrel 8.8. The analyses used a poychoric correlation matrix with diagonally weighted least squares for the estimation method. Models were compared on the tests of perfect and close fit, root mean square error of approximation (RMSEA),12 comparative fit index (CFI),13 root mean square of the residuals (RMSR), expected cross validation index (ECVI) and the corrected Akaike Information Criterion (AIC). For the RMSEA, RMSR, ECVI and AIC, lower values indicate better fit. For the CFI, higher values indicate better fit. Although the RAS has 41 items, we only used the 24 scored items in the bifactor model and did not include the unscored 17 items in the model.
Sensitivity to change
We examined sensitivity to change in three comparisons. First, the intervention and control groups were compared at each follow-up assessment (three, six, 12 and 18 months). Second, people who met criteria for Major Depression on the SCID at baseline were classified as improved at each follow-up if the SCID showed they did not meet criteria anymore. For this comparison, RAS values at the follow-up assessments were compared to baseline. Lastly, the baseline values were compared to the follow-up values for the intervention group only. For these analyses, we used the effect size measure of sensitivity to change.8,14 While other measures are available, they could not be conducted for the first comparison (intervention and control groups) as the measures assume a correlation between the groups and the groups were created by random assignment. Although we use Cohen’s14 original guidelines of interpreting effect size (0.2=small, 0.5=moderate, 0.8=large), these are not strict cutoffs and effect sizes should be considered within the context of the population and study design.
Results
The sample recruited was consistent with the population of the clinics and medical center through which the study was conducted (see Table 1 for descriptive statistics). The sample was predominantly female, unsurprising given the gender difference in depression.15 The average participant was middle-aged and had not completed a bachelor’s degree. Attrition was low as 90% of the sample was retained by the last (18 month) follow-up.
Table 1.
Demographic characteristics (n=302)
| Variable | Mean (SD) or Percent (n) |
|---|---|
| Age, years | 49.9 (14.0) |
| Female | 58.9% (178) |
| Bachelor’s degree or higher | 31.1% (94) |
| Race and Ethnicity | |
| Caucasian/White | 79.5% (240) |
| African American | 17.2% (52) |
| Hispanic | 4.3% (13) |
| Asian | 4.0% (12) |
| Native American | 5.0% (15) |
| Other group | 2.3% (7) |
| Intervention group | 49.7% (150) |
| Baseline RAS Total Score | 3.46 (0.43) |
| Baseline RAS Subscale Scores | |
| Confidence and Hope | 3.25 (0.59) |
| Willingness to ask for Help | 3.45 (0.85) |
| Goal/Success Orientation | 3.48 (0.68) |
| Reliance on Others | 4.06 (0.63) |
| No Domination by Symptoms | 2.58 (0.81) |
| Met Criteria for Major Depression at Baseline | 70.9% (214) |
RAS=Recovery Assessment Scale, SD=Standard Deviation
Factor analysis
The confirmatory factor analyses supported the five factor model. The fit indices, reported in Table 2, showed both the five factor and bifactor models fit the data well. However, the AIC did not substantially improve between the five factor and bifactor models.16 Also, factor loadings, in Table 3, showed that some loadings on the general factor of the bifactor model were low, particularly for items 30 through 32 and item 40. As the five factor model is more parsimonious and fit did not substantially improve with the addition of the general factor, the original five factor structure was supported.
Table 2.
Fit indices for factor analyses.
Five factor model was taken from previous studies while the bifactor model is the five factor model with an additional general factor that all items load on.
| Five factor | Bifactor | |
|---|---|---|
| Perfect Fit | χ2=436.10, p<.001 | χ2=341.62, p<.001 |
| Close fit | p=0.31 | p=0.95 |
| RMSEA | 0.052 (0.044, 0.060) | 0.041 (0.032, 0.050) |
| CFI | 0.97 | 0.98 |
| RMSR | 0.07 | 0.067 |
| ECVI | 1.88 | 1.65 |
| Corrected AIC | 823.95 | 823.08 |
AIC=Akaike Information Criterion, CFI=Comparative Fit Index, ECVI=Expected Cross Validation Index, RMSEA=Root Mean Square Error of Approximation, RMSR=Root Mean Square Residual
Table 3.
Factor loadings from factor analyses.
| Five Factor Model | Bifactor Model | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
|
| |||||||||||
| Goals | Hope | Help | Others | Symptoms | Goals | Hope | Help | Others | Symptoms | General | |
| Item 01 | 0.45 | 0.43 | 0.33 | ||||||||
| Item 02 | 0.65 | 0.21 | 0.57 | ||||||||
| Item 03 | 0.61 | 0.93 | 0.45 | ||||||||
| Item 04 | 0.74 | 0.16 | 0.65 | ||||||||
| Item 05 | 0.73 | 0.17 | 0.65 | ||||||||
| Item 06 | 0.84 | 0.81 | 0.36 | ||||||||
| Item 11 | 0.48 | 0.19 | 0.48 | ||||||||
| Item 14 | 0.53 | −0.09 | 0.54 | ||||||||
| Item 15 | 0.68 | 0.39 | 0.67 | ||||||||
| Item 16 | 0.54 | 0.64 | 0.53 | ||||||||
| Item 20 | 0.58 | −0.18 | 0.59 | ||||||||
| Item 22 | 0.64 | −0.21 | 0.65 | ||||||||
| Item 24 | 0.72 | −0.20 | 0.74 | ||||||||
| Item 25 | 0.59 | −0.04 | 0.60 | ||||||||
| Item 27 | 0.65 | 0.37 | 0.43 | ||||||||
| Item 28 | 0.76 | 0.87 | 0.47 | ||||||||
| Item 29 | 0.69 | 0.40 | 0.45 | ||||||||
| Item 30 | 0.84 | 0.75 | 0.30 | ||||||||
| Item 31 | 0.81 | 0.69 | 0.32 | ||||||||
| Item 32 | 0.85 | 0.88 | 0.27 | ||||||||
| Item 36 | 0.35 | −0.03 | 0.35 | ||||||||
| Item 37 | 0.84 | 0.72 | 0.40 | ||||||||
| Item 39 | 0.86 | 0.78 | 0.39 | ||||||||
| Item 40 | 0.48 | 0.27 | 0.32 | ||||||||
Sensitivity to change
Effect sizes are reported in Table 4. The RAS showed moderate to large changes over time or when symptoms improved but only showed a small change between the two treatment groups. When comparing the control and intervention groups, three subscales (Confidence and Hope, Willingness to ask for Help and Goal/Success Orientation) and the total score only showed small differences between the two groups by the 18 month follow-up. No Domination by Symptoms showed small to medium differences between the control and intervention groups. In contrast to the comparisons between treatment groups, nearly all subscales of the RAS and the total score showed small to moderate changes at three months, increasing to moderate and large changes by 18 months when comparing baseline RAS scores to follow-up scores for those whose depression remitted. The one exception was the Reliance on Others subscale that practically did not change across all three comparisons. The effect sizes comparing baseline to follow-up RAS scores in the intervention group showed a similar pattern to the analyses for improvement by the SCID with the total RAS score and all subscales except Reliance on Others showing small changes at 3 months and moderate to large changes by 18 months.
Table 4. Sensitivity to Change Results.
Higher effect size (ES) indicates that the subscale or total scale showed more change compared to the baseline for ‘Improved by SCID’ and ‘Over time, Intervention group’ or compared to the control group for ‘Control to intervention’.
| Subscale | Month of assessment | Control to intervention | Improved by SCID | Over time, Intervention group | |||
|---|---|---|---|---|---|---|---|
|
| |||||||
| N | ES | N | ES | N | ES | ||
| Confidence & Hope | 3 | 282 | 0.041 | 103 | 0.492 | 137 | 0.336 |
| Confidence & Hope | 6 | 277 | 0.089 | 114 | 0.666 | 137 | 0.500 |
| Confidence & Hope | 12 | 273 | 0.070 | 113 | 0.905 | 135 | 0.577 |
| Confidence & Hope | 18 | 274 | 0.282 | 124 | 1.026 | 134 | 0.877 |
| Willingness to ask for Help | 3 | 280 | 0.152 | 103 | 0.224 | 136 | 0.298 |
| Willingness to ask for Help | 6 | 277 | 0.163 | 114 | 0.443 | 137 | 0.392 |
| Willingness to ask for Help | 12 | 273 | 0.101 | 113 | 0.506 | 135 | 0.428 |
| Willingness to ask for Help | 18 | 274 | 0.234 | 124 | 0.607 | 134 | 0.556 |
| Goal/Success Orientation | 3 | 282 | 0.096 | 103 | 0.377 | 137 | 0.230 |
| Goal/Success Orientation | 6 | 277 | −0.036 | 114 | 0.470 | 137 | 0.257 |
| Goal/Success Orientation | 12 | 273 | 0.144 | 113 | 0.603 | 135 | 0.408 |
| Goal/Success Orientation | 18 | 274 | 0.245 | 124 | 0.751 | 134 | 0.595 |
| Reliance on Others | 3 | 282 | 0.101 | 103 | 0.113 | 137 | 0.118 |
| Reliance on Others | 6 | 277 | −0.043 | 114 | 0.237 | 137 | 0.139 |
| Reliance on Others | 12 | 273 | −0.073 | 113 | 0.134 | 135 | 0.088 |
| Reliance on Others | 18 | 274 | −0.038 | 124 | 0.226 | 134 | 0.197 |
| No Domination by Symptoms | 3 | 281 | 0.123 | 103 | 0.944 | 137 | 0.490 |
| No Domination by Symptoms | 6 | 277 | 0.248 | 114 | 1.111 | 137 | 0.769 |
| No Domination by Symptoms | 12 | 273 | 0.337 | 113 | 1.247 | 135 | 0.947 |
| No Domination by Symptoms | 18 | 273 | 0.428 | 123 | 1.428 | 134 | 1.177 |
| Overall RAS Scale | 3 | 282 | 0.192 | 103 | 0.701 | 137 | 0.474 |
| Overall RAS Scale | 6 | 277 | 0.185 | 114 | 0.924 | 137 | 0.678 |
| Overall RAS Scale | 12 | 273 | 0.197 | 113 | 1.179 | 135 | 0.765 |
| Overall RAS Scale | 18 | 274 | 0.341 | 124 | 1.377 | 134 | 1.084 |
ES=Effect size, RAS=Recovery Assessment Scale, SCID=Structured Clinical Interview for the DSM
Discussion
The results of this study showed the RAS is a psychometrically sound measure confirming previous research.4,6 The five factor structure shown in other studies was supported and a bifactor model did not fit the data better. While the overall RAS score showed sensitivity to change across treatment groups, changes in depression diagnoses and across time in the intervention group, not all subscales showed equal sensitivity to change. The no domination by symptoms subscale showed excellent sensitivity to change while the confidence and hope, goal and success orientation and willingness to ask for help subscales were sensitive to change in depression diagnosis and across time in the intervention group. The reliance on others subscale was not particularly sensitive to change and most subscales only showed small differences between the control and intervention groups on follow-up assessments.
That the symptom subscale of the RAS showed change particularly well is not surprising as most treatments are focused on alleviating symptoms.2 The intervention in this study partially focused on helping participants engage more in traditional mental health care and on CBT skills.9 Previous studies have highlighted how the RAS correlates with more traditional symptom-based measures.17
Implications for Behavioral Health
The results imply the subscales of the RAS should ideally be used to measure response to treatment, whether in research or clinically, instead of just the total score. Results of the factor analysis suggested that each subscale measures a distinct, but related, aspect of recovery rather than just facets of a single construct of recovery. The sensitivity to change results also showed that certain subscales, no domination by symptoms in particular, are more sensitive to change than other subscales. The RAS was particularly sensitive to change over time, as shown by comparisons of changes in depression diagnosis and changes over time in the intervention group. However, effect sizes were smaller when comparing the intervention group to the control group suggesting that the RAS may be more useful for longitudinal comparisons but less likely to show results for symptom-focused treatments.
The implications should be considered within the strengths and limitations of the study. Participants were included regardless of comorbidity status, improving generalizability of the results but possibly reducing effect sizes. Unlike most sensitivity to change studies, we were able to compare an intervention and control group as well as changes over time. This intervention may not have affected reliance on others, possibly explaining why this subscale did not show sensitivity to change from the intervention. Also, the subscales that did show sensitivity to change may not show sensitivity to other interventions. For example, an intervention focused on individual therapy or medication treatment would differ significantly from the intervention described in this paper and the sensitivity to change results may not apply. Further trials of different interventions would be needed to test whether reliance on others is sensitive to other interventions. These results might not translate to populations with other mental illnesses as their primary diagnosis or to populations outside the Northwest region of the United States. Further studies are also needed to determine whether the 17 unscored items of the RAS have to be administered or whether only the 24 scored items can be used. Nevertheless, results support the continued use of the RAS and gives guidance on ways to effectively use the measure in clinical use or future research studies.
Acknowledgments
This study was supported by the National Institute of Mental Health (grant number MH065530). Registration number at clinicaltrials.gov is NCT01139060.
Footnotes
Conflict of Interest: The authors do not have any conflicts of interest to disclose.
References
- 1.President’s New Freedom Commission on Mental Health; Services HaH, editor. Achieving the Promise: Transforming Mental Health Care in America. 2003. [Google Scholar]
- 2.Schrank B, Slade M. Recovery in psychiatry. Psychiatric Bulletin. 2007;31:321–325. [Google Scholar]
- 3.Giffort D, Schmook A, Woody C, et al. Construction of a scale to measure consumer recovery. Springfield, IL: 1995. [Google Scholar]
- 4.Corrigan PW, Salzer M, Ralph RO, et al. Examining the factor structure of the recovery assessment scale. Schizophrenia Bulletin. 2004;30(4):1035–1041. doi: 10.1093/oxfordjournals.schbul.a007118. [DOI] [PubMed] [Google Scholar]
- 5.Holzinger KJ, Swineford F. The bifactor method. Psychometrika. 1937;2:41–54. [Google Scholar]
- 6.Corrigan PW, Giffort D, Rashid F, et al. Recovery as a psychological construct. Community Mental Health Journal. 1999;35(3):231–239. doi: 10.1023/a:1018741302682. [DOI] [PubMed] [Google Scholar]
- 7.McNaught M, Caputi P, Oades LG, et al. Testing the validity of the Recovery Assessment Scale using an Australian sample. Australian and New Zealand Journal Psychiatry. 2007;41(5):450–457. doi: 10.1080/00048670701264792. [DOI] [PubMed] [Google Scholar]
- 8.Streiner DL, Norman GR. Health Measurement Scales: A practical guide to their development and use. Oxford University Press; 2008. [Google Scholar]
- 9.Ludman EJ, Simon GE, Grothaus LC, et al. Organized Self-Management Support Services for Chronic Depressive Symptoms: A Randomized Controlled Trial. Psychiatric Services. 2015:appips201400295. doi: 10.1176/appi.ps.201400295. [DOI] [PubMed] [Google Scholar]
- 10.First MB, Spitzer RL, Gibbon M, et al. Structured clinical interview for DSM-IV Axis I disorders, research version, non-patient edition (SCID-I/NP) New York: Biometrics Research, New York State Psychiatric Institute; 2002. [Google Scholar]
- 11.American Psychiatric Association. Diagnostic and statistical manual of mental disorders: DSM-IV-TR. 4. Washington, DC: American Psychiatric Association; 2000. text revision ed. [Google Scholar]
- 12.Browne M, Cudeck R. Alternative ways of assessing model fit. London, England: Sage; 1993. [Google Scholar]
- 13.Hu L, Bentler PM. Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal. 1999;6(1):1–55. [Google Scholar]
- 14.Cohen J. Statistical power analysis for the behavioral sciences. 2. Hillsdale, NJ: Lawrence Earlbaum Associates; 1988. [Google Scholar]
- 15.Kessler RC, Chiu WT, Demler O, et al. Prevalence, severity, and comorbidity of 12-month DSM-IV disorders in the National Comorbidity Survey Replication. Archives of General Psychiatry. 2005;62(6):617–627. doi: 10.1001/archpsyc.62.6.617. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Reise SP, Morizot J, Hays RD. The role of the bifactor model in resolving dimensionality issues in health outcomes measures. Quality of Life Research. 2007;16(Suppl 1):19–31. doi: 10.1007/s11136-007-9183-7. [DOI] [PubMed] [Google Scholar]
- 17.Drapalski AL, Medoff D, Unick GJ, et al. Assessing recovery of people with serious mental illness: development of a new scale. Psychiatric Services. 2012;63(1):48–53. doi: 10.1176/appi.ps.201100109. [DOI] [PubMed] [Google Scholar]