Does the Recovery Assessment Scale measure the same recovery construct across time?

Abstract

Objective:

The Recovery Assessment Scale (RAS) is one of the most used recovery measures in recovery-oriented practice evaluation for people with mental health conditions. Although the psychometric properties have been extensively studied, one critical piece of information that is missing in the literature is evidence of longitudinal factorial invariance -- whether the RAS measures the same recovery construct across time. The current study empirically tested the longitudinal factorial invariance assumption for RAS.

Methods:

Structural equation modeling tested the longitudinal factorial invariance of RAS using data obtained from 167 people with severe mental illness at three time points longitudinally.

Results:

The longitudinal factorial invariance assumption was supported (i.e., configural, metric, partial scalar, and factor variance and covariance invariance).

Conclusions:

The study found the empirical evidence that RAS can measure the same recovery construct over time, which meets one of the important prerequisites for longitudinal assessment.

Recovery has become a central tenet of mental health services fostering “a process of change through which individuals improve their health and wellness, live a self-directed life, and strive to reach their full potential”(1). Under this guiding vision, recovery-oriented services need clear operationalization of the recovery construct and a set of measurable milestones to evaluate service effectiveness. Without reliable and valid recovery measures, effectiveness of recovery-oriented practices cannot be fully tested. Thus, establishing psychometrically-sound recovery measures is essential for advancing mental health practice, research, and policy.

Several recovery measures have been developed, and systematically reviewed to evaluate their utility (2–4). Among the existing recovery measures, the Recovery Assessment Scale (RAS) (5) has been identified as one of the most used measures in recovery-oriented practice evaluation, and the psychometric properties have been extensively studied (6, 7). The RAS has good reliability (i.e., internal consistency, test-retest reliability, interrater reliability), validity (i.e., consistent factor structures across different samples, expected associations with relevant constructs), and utility for intervention (6).

However, one critical piece of information that is missing includes the longitudinal factorial invariance of RAS, showing that the recovery construct measured by RAS is consistent over time. If this assumption fails, the observed score changes do not represent the true changes in the same recovery construct. For example, if the recovery experience has different meanings across time, the measurement score changes may not be comparable in the same structure or metric. Therefore, longitudinal factorial invariance is required to assure that the RAS measures the same recovery construct over time, which is a prerequisite for longitudinal assessments of change (8). The current study aims to test longitudinal factorial invariance of RAS with data collected from people with severe mental illness at three time points longitudinally. Given the growing emphasis of measurement-based practices (9), this study could increase our confidence in the RAS as a tool for longitudinal evaluation of recovery-oriented practice.

Methods

We conducted secondary data analyses with data originally collected from a study on CommonGround, an intervention designed to increase shared decision-making in psychiatric treatment (10) (see the original study for the details). Consumers were recruited during their psychiatric visits at a community mental health center. A total of 167 consumers participated in the study (pre-post with follow-up experimental group-only design). The participants were mostly male (56.9%), Black or African American (59.9%), and had completed high school or some college (58.1%). The data were collected prior to the intervention, 12 months, and 18 months after the intervention. Due to study dropout, 167 (baseline), 105 (12 months), and 83 (18 months) cases were available for the analyses. All procedures for the original study were approved by the [university] Institutional Review Board.

The Recovery Assessment Scale (RAS) was developed through consumer narratives and item reviews to capture self-perceptions of a sense of recovery for people with psychiatric disabilities (5). Five factors with 24 items were identified by exploratory and confirmatory factor analyses (4, 6). The five factors tap distinct domains and are correlated with psychosocial (e.g., empowerment, hope, quality of life) and symptom variables. The items use a five-point scale ranging from “strongly disagree (=1)” to “strongly agree (=5).” Cronbach alphas of each RAS domain for the current study were .86–.89 (personal confidence and hope), .86–.88 (willingness to ask for help), .79–.83 (goal and success orientation) .75–.77 (reliance on others), and .54–.75 (no domination by symptoms), which vary slightly across different time points.

Measurement and structural invariance testing (11) of RAS was conducted to evaluate whether the recovery concept measured by RAS is consistent over time (time 1[baseline], time 2[12 months], and time 3[18 months]). We created parcel-level indicators (i.e., averaging item scores within each domain) which represent the recovery construct (i.e., five indicators under the recovery construct). A parceling method can establish a parsimonious model and reduce the chance of residual correlations, dual loadings, or sampling errors that are often present in item-level data (12). Invariance testing was conducted sequentially (from the least to the most restrictive models) by the following steps: (a) configural longitudinal invariance (the factor structure is identical over time); (b) metric invariance (corresponding factor loadings are equal: the common factors have the same meaning over time); (c) scalar invariance (corresponding parcel-level indicator intercepts are equal: no different additive influences exist due to time which systematically act to raise or lower response patterns over time); (d) factor variance and covariance invariance (corresponding factor variance and covariance are equal: the recovery construct is structurally comparable over time); and (e) factor mean invariance (latent factor mean is equal: mean of the recovery construct is consistent over time).

Each model was evaluated using the multi-index approach (13), based on the Comparative Fit Index (CFI; values > .95 are preferred), Tucker-Lewis Index (TLI; values > .95 are preferred), and Root Mean Square Error of Approximation (RMSEA; values <.05 are preferred). The appropriateness of parameter constraints of RAS over time was assessed by means of a Chi-square difference test between nested models (the Likelihood Ratio Test), sequentially testing the least to most restrictive models of invariance. If the difference is statistically significant (a significant drop in model fit), the model constraints are not considered tenable (the corresponding parameters are not equal across time). When the constraints were not tenable, we used modification indices to obtain partial invariance. Analyses were performed using Mplus version 7.11. The full information maximum likelihood estimation method was used to accommodate missing data.

Results

The results are summarized in Figure 1. First, the configural invariance model revealed good model fit (the factor structure of the recovery construct was the same over time). Second, the metric invariance model revealed good model fit. A significant drop in model fit from the configural invariance model was not observed (the corresponding factor loadings were equal over time). Third, the scalar invariance model showed significant worsening of fit compared to the metric invariance model (p<.01). Based on the modification indices, the invariance constraint on the intercept for “no symptom domination” at time 1 was relaxed. The parcel-level scores at times 2 and 3 were significantly higher than at time 1. Re-examining fit with the partial scalar invariance model showed good model fit, with no significant drop in model fit from the scalar invariance model (p=.57). Fourth, the factor variance and covariance invariance model revealed good model fit, with no significant drop in model fit from the partial scalar invariance model (p=.48). Finally, the factor mean invariance model showed that the mean of the recovery latent construct (α) was higher at time 3 (α =.20) than at time 1 (α =0).

Figure 1.

The final longitudinal factorial invariance model of the Recovery Assessment Scale and the fit indices across models

Note: * The invariance constraint on the intercept was relaxed for partial scalar invariance

Discussion

The current study tested the longitudinal factorial invariance of RAS using data collected at three time points longitudinally. We confirmed the viability of the scale for longitudinal assessment, with supporting evidence of several indices.

The configural and metric invariance indicate that the recovery construct measured by RAS is consistent in terms of the components (five factors): the RAS captures the same conceptual meaning of the recovery construct over time. The finding is similar to prior research testing the factor structures of RAS across different samples that show the consistency of components across groups of people (6). Our analysis extends this consistency across time – that the RAS measures the same recovery construct longitudinally.

We confirmed the scalar invariance for four of the subscales (people’s response patterns for these domains are equal across time) except for “no symptom domination” (which met criteria for partial scalar invariance). People may evaluate “no symptom domination” differently across time (the domain items may become easier or more difficult to score high on the scale at different time points). Interestingly, “no symptom domination” is the only domain that uses time-relevant wordings (e.g., “My symptoms interfere less and less with my life”). People may weigh their symptom experiences differently across different time periods of the recovery journey. Although we cannot make direct comparisons, previous research also found that this domain functioned a little differently among the RAS domains (e.g., more sensitive to change (14)). Another possibility includes the potential lack of unidimensionality. The symptom domain consists of only three items, and the internal consistency is lower than the other four RAS domains, including in our sample. This might introduce inconsistent item response patterns within the domain.

Finally, we confirmed the factor variance and covariance invariance of RAS, showing that the variance of RAS does not significantly vary across time. All together, the series of invariance testing indicate that the recovery construct measured by RAS is comparable across time, which allows for rigorous tests in longitudinal assessment. In our data, the mean recovery scores increased from baseline (Time 1) to 18 months (Time 3) at the construct level, which is consistent with the manifest level outcome assessment in the original study (10).

The current study has some limitations. First, the study used a sample participating in an intervention study from one community mental health center. This limits the generalizability as well as some conclusions that might be drawn. For example, we cannot determine whether the partial scalar invariance of “no symptom domination” was due to the intervention (which might change the item response patterns after the intervention), or the nature of the domain itself (e.g., symptom experiences in recovery are influenced by time factors). Second, the available sample size was limited, and along with attrition, might compromise the power and parameter estimates. Although the RAS five domains are correlated to one another, each domain also represents a unique construct (15). Securing larger samples will permit additional assessments for each subconstructs in the future study.

Despite the limitations, the current study has important implications for the field of mental health recovery evaluation. The RAS has been the most used measure in recovery-oriented research and practice evaluation, with the presumption that it can capture the same recovery construct across time. Our study empirically tested this assumption, providing evidence that we can compare RAS scores over time with consideration that the symptom domain may function a little differently. We are not aware of any other recovery measures with longitudinal factorial invariance being tested. The importance of measurement-based practices has been emphasized, especially for routine use of symptom measures (9). Recovery is personal and individualized phenomenon, yet our findings suggest that the RAS can also reliably measure recovery over time, which is an important measurement trait that can inform recovery-oriented practices at the client, provider, agency, and policy levels.

Conclusions

Our study adds important validation, empirically confirming that RAS can measure the same recovery construct over time, which is an important prerequisite for longitudinal assessment. Personal recovery and clinical recovery are both important constructs, and the symptom domain may be an important domain in RAS that bridges these constructs; thus further examinations on this domain in longitudinal assessment may be important.

Highlights.

• Longitudinal factorial invariance of the Recovery Assessment Scale (RAS) was confirmed.

• RAS can measure the same recovery construct across time.

• The RAS symptom domain may function differently among the five subdomains.