CHoice of Outcome In Cbt for psychosEs (CHOICE): The Development of a New Service User–Led Outcome Measure of CBT for Psychosis

Abstract

Outcome measures for cognitive behavior therapy for psychosis (CBTp) have been derived from pharmacological studies, focusing on symptom change rather than outcomes such as distress or fulfilment. This study presents the development and psychometric properties of a new outcome measure (CHoice of Outcome In Cbt for psychosEs [CHOICE]), which reflects more strongly the aims of CBTp and the priorities of service users. Service users who had received CBTp participated in focus groups to discuss their outcome priorities, using a topic guide generated by a panel of experts in CBTp. A qualitative thematic analysis was undertaken to reach consensus on themes and generate items. Response scales were constructed for 3 dimensions: severity, satisfaction, and importance. The resulting questionnaire was piloted with service users who had not received CBTp, stratified by service type, ethnicity, and first language to ensure that it was user friendly and applicable prior to CBTp. The psychometric properties of the measure were then examined in a sample of 152 service users. Twenty-four items, and 2 of the dimensions (severity and satisfaction), were retained in the final measure. A factor analysis revealed a single psychological recovery factor interspersed throughout with both CBTp and recovery items. Test-retest reliability, construct validity, and sensitivity to change following CBTp were confirmed. The CHOICE measure is unique in being the first psychometrically adequate service user–led outcome measure of CBTp. It provides the opportunity to examine the evidence base for CBTp with an assessment approach that prioritizes service user definitions of recovery and CBT aims.

Introduction

Measurement of outcome in cognitive behavior therapy for psychosis (CBTp) has so far focused on psychotic symptom reduction and improved function as primary outcomes. Indeed, all the randomized controlled trials (RCTs) of CBTp selected in recent meta-analyses reported symptom reduction, and most reported global function as primary outcomes.^1–6 This strategy was partly chosen to be compatible with RCTs in medication outcome studies and has proved worthwhile in demonstrating the efficacy of CBTp in informing recent national and international guidelines for the treatment of psychosis.^7–9

The influential cognitive models of psychosis of Garety and colleagues,^10–13 Chadwick and Birchwood,^14–17 and Morrison,¹⁸ however, identify a variety of cognitive and emotional processes associated with the development and maintenance of psychotic symptoms: distressing emotions and, specifically, anxiety^11,19; beliefs about voices²⁰; distress and emotional and behavioral dysfunction that arise directly from beliefs; and the content of distressing appraisals and beliefs themselves. Additionally, focused CBT approaches have addressed relapse prevention, negative beliefs about psychosis, self-esteem, and stigma.^21–24 The resulting manuals emphasize a broad range of CBTp targets, which in turn influence the selection of outcome measures.

The suggestion here is that CBTp focuses on something other than, or in addition to, symptom reduction, and that the psychological model used informs both CBT target and outcome measurement. RCTs have tended to accommodate the potential broad focus of CBTp by adding a number of secondary outcomes. Evaluation of secondary outcomes has thus involved dimensional measures of symptoms, such as the Psychotic Symptoms Ratings Scales (PSYRATS)²⁵ to identify conviction, preoccupation, distress, and disability, in addition to an array of specific measures, such as the Beck Depression Inventory (BDI),²⁶ Beck Anxiety Inventory (BAI),²⁷ and Beliefs About Voices Questionnaire.^28,29 Voices Compliance Scale,³⁰ Voices Power Differential,²⁰ Omniscience Scale,²⁰ Personal Beliefs about Illness Questionnaire,³¹ and Rosenberg Self-esteem Scale.³² However, this use of multiple specific measures presents both statistical and methodological problems for RCTs. The advancement of the evidence base for CBTp requires outcome measures that are both specific to CBTp but sufficiently generic to be of relevance to therapy derived from all psychological models.

In addition, CBTp is a collaborative process, and the targets of CBTp on an individual basis are informed by the needs and wishes of the service user. Service user views and the concept of recovery are becoming increasingly central to the organization of mental health service provision.^7,8,33 Yet, consideration of service users’ views suggests that both their targets for intervention and their concepts of recovery differ from those commonly held by clinicians. Service users report that symptoms can be less debilitating than, for example, social exclusion and stigma, emotional problems, and difficulties with relationships.³⁴ Symptom reduction is not necessarily sufficient to improve quality of life, with issues such as empowerment, choice, control, and personal fulfilment being of central concern.³⁵ Therefore, service users’ views of recovery can diverge from the clinician model of symptom reduction and return of function,^36,37 in promoting a different course where the return to a previous functional level may be seen as a negative outcome, and the development of a new direction in life is paramount.³³ There are few measures within psychosis that are designed to capture these needs, although some do address service user priorities such as quality of life and empowerment^38,39 and define outcomes such as quality of life from a service user perspective.⁴⁰

From the foregoing, we conclude that a measure of CBTp suitable for assessing outcome in RCTs should incorporate both CBTp priorities and those of service users. It should be reliable, valid, and suitable for administration before and after therapy; specific enough to capture change; but sufficiently generic to apply across different CBTp approaches and models. Brevity and ease of administration are also important.

The goal of this study was to develop a new outcome measure of CBTp suitable for use in RCTs as well as in individual practice, in collaboration with service users. A self-report format was chosen to assess the service user perspective. This format affords greater flexibility for self or researcher administration, without the need for lengthy training. In keeping with the notion that there is more to outcome than problem reduction, a dimensional structure was incorporated with separate dimensions for severity, satisfaction, and importance for each concern. While severity and satisfaction are commonly assessed, a dimension of importance to the service user was included after consideration of the literature on “utility values” or relative values to an individual of different health states.^41–43 This dimension aimed to capture the current relative health value (importance) of problem areas to the individual. See online supplementary material for a copy of the measure.

Methods

Design

The study was designed to identify outcomes of CBT for psychosis (CBTp) considered important by both service users and cognitive behavioral therapists and to develop a reliable and valid questionnaire to assess these outcomes. Initially, a consultation among experts in CBTp generated broad probe areas for discussion of CBTp outcomes. The measure development was then conducted with service user researcher involvement (A.S.) through 7 stages of design, implementation, and analysis: (1) qualitative analysis of service user focus group discourses, (2) Delphi analysis and measure adaptation after 2 follow-up interviews with each focus group member, (3) initial piloting and adaptation of the complete measure, (4) factor analysis in a large sample, (5) test-retest reliability, (6) construct validity against traditional outcome measures, and (7) sensitivity to change.

Initial Consultation to Form Focus Group Topic Guide

A consultation among experts in the field of CBTp (n = 5: P.G., E.K., E.P., J.S., K.E.G.) yielded a list of broad probe areas for discussion in the focus groups, which were finalized following further written feedback from 3 of the original experts (P.G., E.K., J.S.) and 8 CBTp therapists working in a specialist CBTp service, the Psychological Interventions Clinic for oUtpatients with Psychosis (PICuP) Clinic (South London and Maudsley NHS Foundation Trust). These probes were developed to facilitate initial generation of a broad range of items for the questionnaire. The 6 probes were (1) mood/emotions, (2) empowerment/self-confidence/self-esteem, (3) coping with/understanding of/approach to problems, (4) day-to-day functioning/quality of life, (5) psychotic experiences/symptoms, and (6) relationships with people.

Development of the Measure

Participant characteristics for each stage are summarized in table 1.

Table 1.

Participant Characteristics for Stages 1–7

	n	Age, y (Mean)	Gender (% male)	Ethnicity (%Black/Minority Ethnic)	Marital Status (% Single/Divorced
Stage 1 and 2 focus groups/Delphi process	12	41.5 (7.4); 31–56	75	25	—
Stage 3 pilot interview	15	36.7 (8.8); 26–56	73.3	66.6	100
Stage 4 factor analysis	152	36.2 (8.9); 20–60	60.5	46.5 (n =142)	88.6 (n = 140)
Stage 5 construct validitya	120	35.9 (8.7); 20–60	64.2	47.4 (n=116)	89.4 (n = 114)
Stage 6 test-retest reliabilitya	30	34.3 (7.7); 20–50	57.1	39.3	89.3
Stage 7 sensitivity to changea	33	37.7 (7.0); 22–56	48.5	39.4	84.9

^aA subset of the total participant group involved in the factor analysis.

Stage 1—Focus Groups

Potential focus group participants were all service users who received therapy with the PICuP Clinic between January 2003 and December 2004 (n = 76). People were excluded if they were under 18 years, withdrew from therapy, received therapy from the group facilitator, or were too unwell to take part. Fifty-one participants were invited to take part, of whom 12 agreed. All interested participants (24% of all eligible users) took part in either the pilot (n = 3 participants) to inform the final format and topic guide or the focus group (n = 9 participants) to discuss possible and desired outcomes. They were audiotaped for transcribing and were facilitated by a service user researcher with first-hand CBT experience (A.S.) and a CBTp research clinician (K.G.). A research assistant (S.W.) made additional notes.

Focus groups followed a preprepared semistructured interview format, with the relative importance and nature of change being explored for all emerging outcomes. The 6 probe areas identified by CBTp experts were offered to facilitate item generation only when these were not covered spontaneously. Written feedback was obtained from each group member in the closing stages of the group on the outcomes considered most important.

A combined thematic and content analysis was applied to the transcribed discourses and written feedback using a multiple coding approach: Three researchers (research clinician, service user researcher, research assistant) all analyzed the full transcript of each group and produced independent theme structures. Potential items were either mapped on to the broad predetermined outcome areas (content analysis) identified by CBTp therapists or were new and emerging from the discourses (thematic analysis). The validity of themes was ascertained through a process of triangulation whereby independent analyses were subjected to a consensus between the 3 researchers to identify common themes and through respondent validation of themes during the follow-up interviews as described below.

Stage 2—Follow-up Interviews and Delphi Process

To capture outcomes that were held consistently over time and to reduce the impact of group pressure, the preliminary outcomes were fed back in 2 additional semistructured interviews with each focus group member. Focus groups and first and second interviews were held at approximately 1-month intervals to allow for interim analyses and preparation of the next interview document.

Participants were asked in follow-up interviews whether the derived outcomes made sense given the discussions in the focus groups and whether they were important and achievable through CBT. They were also asked to comment on the language and the format of the measure, preferences over the measurement scale, and inclusion of outcome descriptors. The pooling of all participant responses to all items revealed that 97% of outcomes made sense, 85% were deemed achievable through CBT, and the mean importance rating was 6.0 (range of 5.4–6.6) on a 7-point scale.

Following Delphi procedures, and in order to move toward a consensus, participants rated the importance of items on a 1- to 7- point scale for each outcome. At the second interview, participants were presented for each item with their own and the mean importance rating for the group from the first interview. They were then asked to rerate the importance of each item. The Delphi analysis compared the mean discrepancies between the individual and group means at time 1 compared with those at time 2. A paired samples t test demonstrated progression toward a consensus with significantly smaller discrepancies between individual and group ratings in the second, compared with the first, interviews (t = 3.34, df = 27, P = .002, 95% confidence interval [CI] = 0.06–0.23).

Stage 3—Pilot Interviews

The pilot measure, entitled CHOICE (CHoice of Outcome In Cbt for psychosEs) in recognition of the tailoring to service user priorities and CBT outcome principles, comprised 26 items, one of which, “coping,” was divided into 6 subsections to provide 31 items in total. As a result of service users’ emphasis on the individual nature of the psychosis experience and of goals, an additional 2 items were left blank for service users to complete.

A new group of people with psychosis (n = 15), who had not previously received CBT, completed the pilot measure. This group was selected to encompass service users in a range of psychiatric settings, ethnicity, and first language to ensure representativeness across the diverse population of people with psychosis. The psychiatric settings consisted of an inpatient ward, a local outpatient community team, and the PICuP clinic (n = 5 per setting). Within each setting, there were 2 individuals from black and minority ethnic populations and 1 person with English as their second language.

Participants also provided feedback on the language and clarity by underlining and commenting on problem areas and on the user friendliness by completing multiple choice questions relating to ease of completion, length, clarity, and emotions elicited. Sixty-seven percent of service users found the questionnaire intermediate or easy to complete, 80% found it intermediate or enjoyable, and 93% found it acceptable in length. The majority of service users (67%) thought that the outcomes were clearer without a description. Descriptors were therefore delegated to a supplementary booklet to be used by the administrator of the measure in the event that further explanation was required.

Factor Analysis

A factor analysis was undertaken to reduce item redundancy and to elucidate the factor structure. A 2-factor solution was predicted based on the dual goals of capturing both the aims of CBTp and the recovery priorities of service users. The analysis was based on data from 152 participants who completed the outcome measure as part of consecutive standard assessments for the PICuP clinic (baseline, pre-, mid-, or posttherapy assessments) and who gave informed consent for their data to be used for research. A principal axis factoring model was employed and compared using direct oblique and varimax rotations. Of the initial 31 outcomes, items entered into the final model were determined following an iterative process involving investigation of Pearson interitem correlations, corrected item to total correlations, and stepwise omission of highly correlated redundant items, items with low intercorrelations (<0.3), and those with low item-total correlations. Personal goal items were excluded from the factor analysis due to the smaller number of participants (n = 91) who identified individual goals. The factor solution was decided through consideration of the scree plot, eigenvalues, and predicted eigenvalues derived from a Monte Carlo parallel analysis of 100 randomly generated solutions with the same number of participants and items. Of the 3 dimensions (severity, satisfaction, and importance), those entered into the final model were decided after consideration of sensitivity to change and intercorrelations.

Test-Retest Reliability

Test-retest reliability was undertaken for each dimension, using the intraclass correlation (ICC) coefficient, with subject as a 1-way random effect.⁴⁴ The analysis incorporated the first 30 participants who either completed the measure as part of a standard clinic assessment and subsequently returned a postal retest copy or who completed the questionnaire at baseline and again at the second pretherapy assessment. The postal return rate was 30% (n = 19), and 11 people completed the questionnaire at both their initial (baseline) and second (pretherapy) assessment.

Construct Validity

The cross-sectional validation study comprised the first 120 people, involved also in the factor analysis, who completed the measure alongside their standard baseline or posttherapy assessments for the PICuP clinic. Construct validity was investigated for each dimension against 5 standard self-report outcome measures of psychotic symptoms, emotional problems, quality of life, and “cognitive insight,” described below.

Psychosis Symptoms

The PSYRATS²⁵—the PSYRATS comprises 2 multidimensional assessment scales: one each for delusions and hallucinations. The delusion scale comprises 6 items that form a 2-factor structure of distress/disruption and preoccupation/conviction.^24,44 The Hallucinations Scale has 11 items but a less clear structure with both 3- and 4-factor solutions reported, both of which include a distress/disruption/negative content factor.^25,45,46 For the purposes of the current study, validity was investigated against the factors reported by Steel et al.⁴⁵

Emotional Problems

1. BDI-II²⁶—a 21-item measure of depressive symptoms and cognitions

2. BAI²⁷—a 21-item measure of physiological anxiety symptoms

Quality of Life

The Manchester Short Assessment of Quality of Life Scale (MANSA)⁴⁷—a 16-item measure that assesses satisfaction with life across 12 subjective measures and 4 objective measures including areas such as employment, finances, leisure, friendships, relationships, personal safety, accommodation, and physical and mental health. Each item is rated on a 7-point scale with extremes labeled “couldn't be worse” and “couldn't be better.”

Cognitive Insight

The Beck Cognitive Insight Scale (BCIS)⁴⁸ is a 15-item scale that evaluates patients’ self-reflection and overconfidence in their interpretations of their experiences. The scale contains 2 subscales of self-certainty and self-reflexivity.

Sensitivity to Change

Finally, the sensitivity of the measure to change over a course of therapy was investigated for each dimension for the first 33 participants who completed therapy with the PICuP clinic (average 19 sessions [range = 5–47]) and who completed the measure before and after CBTp with a psychology assistant, independent of therapy delivery.

Results

Factor Analysis

Preliminary analysis revealed high intercorrelations between the dimensions of severity and satisfaction (r = 0.82 for the final scale) and a similar factor structure for each of these dimensions when conducted separately. Ratings of item importance were high (mean = 8.51 [range = 5.06–10] on a 10-point scale), but this dimension did not correlate significantly with either severity or satisfaction and did not change significantly with CBT, as discussed below. In the interests of brevity and simplicity, it was therefore decided to delete the importance dimension from the measure. The ratings of severity and satisfaction were then combined into mean scores across dimensions for the factor analysis. Following stepwise elimination of redundant, highly correlated items and those with low interitem and item-total correlations, a total of 24 items, 5 of which were subcategories of coping, were retained (see table 2). The majority of correlation coefficients between these items were 0.3 and above, the Kaiser-Meyer-Olkin value was 0.92, and the Bartlett's Test of Sphericity reached significance, thus supporting the suitability of the data for factor analysis.

Table 2.

Unrotated Factor Structure Item Loadings for 1-Factor Solution

1. Feeling happy	0.799
2. A sense of being in control of my life	0.791
3. Ways of dealing with unpleasant feelings and emotions (eg, depression, worry, anger)	0.784
4. Positive ways of thinking	0.778
5. Peace of mind	0.773
6. A positive purpose and direction in life	0.740
7. Ways of dealing with distressing experiences (eg, beliefs, thoughts, voices)	0.736
8. Feeling overwhelmed by negative feelings (eg, fear, depression, anger)	0.732
9. Positive ways of relating to people	0.709
10. Facing my own upsetting thoughts and feelings	0.704
11. Ways of dealing with group situations	0.699
12. Ways of dealing with everyday life stresses	0.689
13. Self-confidence	0.675
14. Ways of dealing with a crisis	0.672
15. The ability to relax	0.656
16. The ability to see things from another point of view	0.639
17. Understanding my experiences (eg, beliefs, thoughts, voices, and related feelings)	0.622
18. The ability to approach problems in a variety of ways	0.619
19. Feeling safe and secure	0.618
20. Understanding myself and my past	0.579
21. The effect of unpleasant experiences (eg, beliefs, thoughts, voices, feelings) on my life	0.567
22. Feeling that there is someone who understands and listens to me	0.458
23. Knowing I am not the only person who has unusual experiences	0.439
24. The ability to question the way I look at things	0.371

A principal axis factoring analysis of the 24 items, using SPSS Version 12, revealed the presence of 4 factors with eigenvalues exceeding 1, which explained 47.0%, 5.4%, 5.1%, and 4.6% of the variance, respectively. However, the scree plot revealed a clear break after the first factor. A single-factor structure was supported by the Monte Carlo parallel analysis in which only one factor exceeded the predicted criterion eigenvalue (actual eigenvalue 11.28 compared with a criterion eigenvalue of 1.812), compared with 100 randomly generated datasets with the same sample size (24 variables × 152 participants). The single factor, several high loading items, and communalities in the 0.5 range indicated that the 152 participants and 24 items provide sufficient power and a robust factor structure.^49,50 Hence, the final single-factor solution explained 47% of the variance. Inspection of the histogram of mean factor scores revealed that these were normally distributed across the full range of possible scores, with a mean of 4.52 and an SD of 1.93 (See figure 1). The unrotated factor loadings are presented in table 2.

Fig. 1.

Distribution of mean CHoice of Outcome In Cbt for psychosEs factor scores

Test-Retest Reliability

The dimensions had high internal consistency (Cronbach α = .83 for severity and .88 for satisfaction) and were reliable over time (test-retest ICC = 0.73, 95% CI = 0.51–0.86, P < .001 for severity, and ICC = 0.79, 95% CI = 0.61–0.90, P < .001 for satisfaction). Preliminary analysis also suggested good internal consistency and reliability for mean severity and satisfaction with individual goals (n = 14 participants).

Construct Validity

Construct validity was investigated using Pearson correlation analyses between the severity and satisfaction symptom dimensions and the appropriate total and factor scores derived from the measures of psychotic symptoms, emotion problems, quality of life, and cognitive insight. The PSYRATS delusions scale was completed by 83 of the 120 participants, and the PSYRATS voices scale by 69 participants. In all correlation analyses, α was set at the 1% level to control for multiple comparisons. The results are summarized in table 3. Of note, both the severity and satisfaction dimensions correlated positively with Quality of Life (MANSA) and negatively with emotional measures (BDI/BAI), with trends for negative correlations with PSYRATS delusions distress, voices distress (CHOICE severity), and disruption/control (CHOICE severity). Construct validity analysis for mean severity and mean satisfaction with personal goals revealed a similar pattern of correlations for the 91 participants who identified one or more individual goals.

Table 3.

Pearson Correlation Coefficients Between CHOICE Dimensions and Measures of Psychotic Symptoms, Emotional Problems, Quality of Life and Cognitive Insight

	CHOICE Mean Severity Score	CHOICE Mean Satisfaction Score
PSYRATS-delusions
Distress	r = −0.26, P = .018*	r = −0.24, P = .03*
Preoccupation/conviction	r = 0.11, P = .31	r = −0.07, P = .56
PSYRATS-voices
Distress/negative content/disruption	r = −0.28, P = .022*	r = −0.11, P = .39
Frequency/duration	r = 0.01, P = .92	r = 0.09, P = .47
Disruption/volume/control	r = −0.28, P = .02*	r = −0.15, P = .24
Location/beliefs about origin	r = 0.10, P = .43	r = 0.18, P = .15
BDI-II	r = −0.58, P < .001**	r = −0.45, P < .001**
BAI	r = −0.48, P < .001**	r = −0.35, P < .001**
MANSA	r = 0.52, P < .001**	r = 0.45, P < .001**
BCIS self-reflectivity	r = 0.02, P = .86	r = −0.04, P = .66
BCIS self-certainty	r = 0.03, P = .73	r = 0.14, P = .14

Note: CHOICE, CHoice of Outcome In Cbt for psychosEs; PSYRATS, Psychotic Symptoms Ratings Scales; BDI, Beck Depression Inventory; BAI, Beck Anxiety Inventory; MANSA; BCIS, Beck Cognitive Insight Scale.

*Significant at 5% level; **significant at 1% level.

Sensitivity to Change Following CBT

Thirty-three people completed the CHOICE measure at baseline and immediately following CBT (see figure). Paired samples t tests revealed a significant improvement on the CHOICE severity and satisfaction dimension scores from before to after therapy (t = −4.1, df = 32, 95% CI = −2.57 to −0.87, P < .001, for severity and t = −4.3, df = 32, 95% CI = −2.97 to −1.06, P < .001, for satisfaction).

Fig. 2.

Improvement in Mean Severity, Satisfaction, and Personal Goal Score Following Cognitive Behavior Therapy for Psychosis.

Sixteen people additionally recorded either 1 or 2 personal goals. These included, eg, “learning to deal with panic attacks,” “commencing guitar playing, artwork and tai chi,” “understanding how my past relates to the present,” “thinking with optimism and hope,” and “finding peace and happiness within myself.” Where an individual recorded 2 personal goals, for the purposes of investigating change following CBT, the mean score was taken across the 2 goals at baseline and again at follow-up for that individual. The significant improvement following CBT was maintained for personal goals (t = −3.12, df = 15, 95% CI = −5.13 to −0.96, P = .007), where the magnitude of change was even greater (mean improvement = 3.05 points).

Discussion

This study sets out to design, in collaboration with service users, a new measure of outcome in CBTp that reflects more accurately than existing assessments both the targets of CBTp and the priorities of service users.

The Psychometric Properties of the Measure

The final measure was a 2-dimensional 24-item self-report questionnaire, which provides mean scores for (1) severity and (2) satisfaction with a range of difficulties. An additional item, left blank, was included to record and evaluate individual intervention targets that were not included elsewhere.

The measure had good test-retest reliability, face and construct validity, and sensitivity to change. It was designed in full consultation with service users to capture the outcomes that they viewed as the most important and realistic to obtain through CBTp. The outcomes were presented and phrased in language deemed appropriate by service users and were acceptable and understandable prior to the start of therapy.

The Dimensions of the Measure

Severity and satisfaction were agreed, a priori, to be distinct dimensions as they have been demonstrated empirically in previous studies to relate to different factors. Subjective ratings of problem severity and problem change are influenced by the severity of delusions and hallucinations⁵¹ and the ability to subjectively appraise experiences.⁵² Subjective quality of life, which is comparable to satisfaction, is associated with psychosocial factors and depression,^53,54 while satisfaction specifically is influenced by age, service user expectations, and gratitude.⁵⁵ Although these 2 dimensions were closely related in the current study, they were maintained as separate dimensions for the reasons described. In light of the relationship between psychotic symptom severity and subjective rating of problems, it is of relevance that test-retest reliability was good for both dimensions, thus validating the reliability of service user responding.

Importance ratings in the current study were high but were not correlated with either severity or satisfaction and were not sensitive to change with CBT. Furthermore, the large number of items, each scored with separate scales and without explicit instructions to compare across items, suggests that this dimension captured absolute and not relative ratings of importance. The high and stable importance ratings added further evidence that the measure captured the important health goals of service users, but the insensitivity of this dimension to change precluded its inclusion within the outcome measure.

The Content of the Measure

The measure is interspersed throughout with both generic recovery–related items and CBT-specific items (see Supplementary Material). Recent service user–led research⁵⁶ has identified a number of important factors in recovery from psychosis, which included “rebuilding the self,” “rebuilding life,” and “hope for a better future.” Subthemes of these factors overlap with CHOICE items such as “understanding myself and my past,” “self-esteem,” “self-confidence,” “positive ways of relating to people,” “a positive purpose and direction in life,” “a sense of being in control of life,” and “hope and optimism.” Similar themes have been echoed in other studies that emphasize internal aspects of recovery such as self-direction, hope, empowerment, coping, and establishing the self as separate from symptoms.⁵⁷ Bellack33(p437) recently stated that recovery should not be “defined so broadly as to make its achievement unimportant.” At every stage in the development of the measure, we aimed to reconcile the capture of important broad recovery themes with the need for items that were specific and measurable.

In addition to capturing outcomes related to recovery from psychosis, the CHOICE measure includes outcomes that seem to be more specifically related to targets of CBTp. Relevant outcomes include “The ability to approach problems in a variety of ways,” “The ability to step back from overwhelming experiences (e.g. thoughts or voices),” “The ability to question the way I look at things,” and “Knowing I am not the only person who has unusual experiences,” the latter reflecting both the CBT focus on normalizing experiences and the recovery process of moving from social exclusion to inclusion.

The construct validity findings emphasize the close relationship between the CHOICE measure scores and both mood (depression and anxiety) and quality of life. Near significant trends were also found between both CHOICE dimensions and the delusions distress factor, as well as between the CHOICE severity dimension and the two voices factors of distress and disruption/volume/control. Hence, the CHOICE measure appears to capture general distress, psychosis-specific distress and quality of life outcomes. The closer relationship to general anxiety, distress and quality of life than to psychosis specific distress emphasises the breadth of treatment priorities for CBTp service users. This is consistent with the nature of CBTp as more than a quasi-neuroleptic¹⁴ aimed at symptom change. CHOICE therefore seems to capture psychological recovery in terms of therapy success or failure from the subjective perspective of the service user.

The Structure of the Measure

Despite the fact that the items were predicted to belong to 2 broad general themes, (ie, recovery and CBTp items), only one main factor was found. This finding may reflect the fact that CBT and recovery outcomes are closely enmeshed with each other. An exploration of the factor structure revealed that the items with the highest loadings could be conceptualized as those likely to yield more immediate changes and the goals of a more recovery and behavioral (coping) nature (eg, A sense of being in control of my life, ways of dealing with unpleasant feelings and emotions, a positive purpose and direction in life), whereas those with the lower loadings (with the exception of feeling safe and secure) were those more reminiscent of cognitive outcomes and processes of change in CBTp⁵⁸ (eg, the ability to see things from another point of view, the ability to approach problems in a variety of ways, understanding my experiences, understanding myself and my past, and the ability to question the way I look at things.) This factor structure may have been affected by the timing of the sampling, which mainly included people prior to the start of therapy (74%) for whom recovery and coping may have been a more relevant and cohesive set of outcomes than cognitive and process goals.

Limitations

The focus group members were self-selecting volunteer service users from a specialist CBTp clinic. In addition, the factor analysis was based on data provided by a single client group, each recruited from a specialist CBTp clinic. This sample may not be generalizable to other individuals with psychosis, and the factor analysis may require replication with a new sample. Furthermore, the service users involved in the measure development were all longer term service users, and the outcomes generated may not necessarily capture the phase-specific needs and priorities from people in their first episode. It is possible that the test-retest reliability may have been lower had the 70% who did not return their postal retest data been included, although test-retest data on approximately 20% of these responders were subsequently collected at their next follow-up interview. Reliability remained good despite the longer time to retest in this latter group.

The validity of the distinction between severity and satisfaction was not confirmed in the current study, despite a clear a priori distinction on theoretical grounds. This may have reflected some conceptual overlap between severity and satisfaction ratings for items tapping psychological recovery that are subjectively rated and that are compared with similarly subjective ratings of mood and quality of life. This distinction warrants further investigation.

The CHOICE questionnaire is clearly sensitive to change over time but requires further investigation using a RCT design to demonstrate sensitivity to change specifically with therapy.

On a related point regarding outcome measurement in RCTs, the limited correlation with psychosis-specific distress and lack of correlation with self-reflection and self-certainty on the BCIS demonstrate that the CHOICE measure complements, rather than replaces, the need for other psychosis symptom measures. In addition, validation against objective measures of function such as employment, education, and independent living should be investigated further in future research.

The Utility of CHOICE in Research and Therapy

The CHOICE measure is unique in being the first psychometrically adequate service user–led outcome measure of CBTp. The psychological recovery focused outcomes are broad enough in scope to capture the essence of CBTp across models but specific enough to be sensitive to change. The opportunity is thus provided to advance the evidence base for CBTp across models with an assessment approach that places service user definitions of recovery at the fore. In light of the structure of the measure, CHOICE may also provide useful information on the process of change in CBTp. Therapeutically, the CHOICE measure has value as a clinical tool for facilitating the development of shared goals and for evaluating progress and outcome. The recovery focus may place the measure in an ideal position to fill the need identified for additional instruments to facilitate and assess service users’ experiences of change and recovery during CBTp.⁵¹

Supplementary Material

Supplementary material is available at http://schizophreniabulletin.oxfordjournals.org.

Funding

South London and Maudsley NHS Foundation Trust (Research and Development grant).