Development and evaluation of an Individualized Outcome Measure (IOM) for randomized controlled trials in mental health

Abstract

Pre‐defined, researcher‐selected outcomes are routinely used as the clinical end‐point in randomized controlled trials (RCTs); however, individualized approaches may be an effective way to assess outcome in mental health research. The present study describes the development and evaluation of the Individualized Outcome Measure (IOM), which is a patient‐specific outcome measure to be used for RCTs of complex interventions. IOM was developed using a narrative review, expert consultation and piloting with mental health service users (n = 20). The final version of IOM comprises two components: Goal Attainment (GA) and Personalized Primary Outcome (PPO). For GA, patients identify one relevant goal at baseline and rate its attainment at follow‐up. For PPO, patients choose an outcome domain related to their goal from a pre‐defined list at baseline, and complete a standardized questionnaire assessing the chosen outcome domain at baseline and follow‐up. A feasibility study indicated that IOM had adequate completion (89%) and acceptability (96%) rates in a clinical sample (n = 84). IOM was then evaluated in a RCT (ISRCTN02507940). GA and PPO components were associated with each other and with the trial primary outcome. The use of the PPO component of IOM as the primary outcome could be considered in future RCTs. Copyright © 2015 John Wiley & Sons, Ltd.

Introduction

Randomized controlled trials (RCTs) evaluate the effectiveness of an intervention in relation to a pre‐specified primary outcome (Schulz et al., 2010). Pre‐defining a standardized measure as a primary outcome has several advantages, including informing the power calculation for the sample size, reducing post hoc “data fishing”, and supporting meta‐analytic aggregation of findings across studies. However, the use of a single pre‐defined primary outcome to evaluate complex interventions raises four issues.

First, a range of outcome domains exist (Slade, 2002), and as complex interventions may impact across this range (Craig et al., 2008), a single pre‐defined outcome measure may not capture all changes. Second, some measures may become widely used primarily because they have been used in other RCTs, rather than because of any inherent superiority (Everitt and Wessely, 2004). Third, pre‐specifying the primary outcome for a RCT may not be clinically informative, as recognized in the developing field of clinimetrics (Feinstein, 1983). Finally, patients may vary in the importance they attach to different types of outcomes. For example, the aim of treatment for some people may be symptomatic improvement while others have social goals (e.g. employment) or personal goals (e.g. to find meaning in their life) (Slade, 2009). As a result, a pre‐defined outcome measure may not be relevant to the person receiving the intervention.

A number of patient‐centred measures and approaches have been developed to address these issues, such as Goal Attainment Scaling (GAS) and Target Complaint Approach (Donnelly and Carswell, 2002). The GAS process was originally developed for mental health programmes (Kiresuk and Sherman, 1968). It requires the patient, together with the clinician, prospectively identifying a number of relevant goals, and then rating their progress towards each goal along a five‐point scale at follow‐up. Although each respondent has different goals, the overall GAS score is standardized so that data can be aggregated and compared across individuals (McDougall and King, 2007; Turner‐Stokes, 2009). GAS has been effectively used to identify and evaluate outcomes in RCTs in rehabilitation (Graven et al., 2011; Turner‐Stokes et al., 2010) and studies with older adults (Rockwood et al., 2007; Rockwood et al., 2003). Systematic reviews conclude that GAS shows high sensitivity to change, reliability and validity when used in chronic health care and physical and neurological rehabilitation (Hurn et al., 2006; Stevens et al., 2013). A systematic review of research evaluating the implementation of GAS in occupational health programmes found that goal setting could be implemented in mental health settings and that goal attainment was associated with positive change in clinical outcomes (Scott and Haggarty, 1984). GAS was also found to be an effective clinical tool among patients in psychiatric day care, with patients with schizophrenia having the highest success rate with 75% achieving a good outcome (Falloon and Talbot, 1982).

Despite this evidence, fully individualized goal setting is not implemented in RCTs in the mental health field. This may be due to operational limitations associated with GAS (Ertzgaard et al., 2011; King et al., 2000). In particular, goal scaling relies on a complex formula, making calculations complex and interpretation complicated (Stevens et al., 2013). Moreover, GAS is based on scaling at baseline and follow‐up which may not fully capture the actual level of goal attainment. Finally, researchers would most likely be involved in goal selection and scaling in RCTs, and they may not be able to assess whether the goal is realistic at baseline or rate attainment at follow‐up.

The aim of this study was to develop and evaluate a new trans‐diagnostic and patient‐valued approach to assessing change in RCTs of complex interventions in mental health, by using individualized and positively‐valued outcome domains. In line with recovery research (Leamy et al., 2011), the focus was on subjective experimental outcomes rather than clinical outcomes. The objectives were to develop, pilot, and assess the feasibility of the Individualized Outcome Measure (IOM), and then to evaluate IOM in a RCT by assessing whether IOM components (1) capture change over time, (2) co‐vary, and (3) are associated with the pre‐defined primary outcome.

Method

Participants

Participants were recruited from community‐based mental health teams in the South London and Maudsley National Health Service (NHS) Foundation Trust (pilot, feasibility, RCT) and 2Gether NHS Foundation Trust (RCT). Inclusion criteria were being aged 18–65 years, being able to speak and understand English, and being sufficiently well to participate in the opinion of the clinician providing community support. In the RCT, patients were participating in the REFOCUS Trial (ISRCTN02507940) which had additional inclusion criteria. These were having a primary clinical diagnosis of psychosis, having no immediate plans for discharge or transfer, not currently receiving in‐patient care, not being in prison, not participating in another substantial study, and being in regular contact with at least one worker in the team. Ethical approval was obtained (10/H0807/4).

Measures

The final version of the IOM (Version 3) comprised two components to index individual outcomes: Goal Attainment (GA) and Personalized Primary Outcome (PPO). The GA component is a variation of GAS, requiring patients to identify one personally relevant goal which mattered to them, that they thought they could achieve and which services could help with, at baseline. Patients were asked to identify only one goal as evidence from mental health research showed respondents can focus on one goal at a time (Scott and Haggarty, 1984). Having one goal also simplifies assessment of change and aids interpretability of the findings. At follow‐up, the patient rated how successful they were at reaching their goal on a five‐point Likert scale from zero (I am further away from my goal) to four (I did even better than expected).

After identifying their goal, patients were invited to complete the PPO component, which is a novel approach. It requires patients to choose the outcome domain which most closely maps onto their selected goal from a pre‐defined list. Each domain is associated with one standardized measure, which patients complete at baseline and again at follow‐up.

In IOM Version 1, 13 outcome domains and associated patient‐rated measures were included in the PPO component. The Activity and Participation Questionnaire (APQ) consists of 6 questions which assess vocational, educational and social participation in the past week (Coombs et al., 2013). The Short Form Health Survey (SF‐36) is a 36‐item measure of functional health and well‐being, consisting of eight scaled scores each with a score range from 0 to 100 (Ware and Sherbourne, 1992). The Short Health Survey Questionnaire (SF‐12) is a 12‐item measure of functional health and well‐being, consisting of two scaled scores with a score range from 0 to 100 (Jenkinson and Layte, 1997). The Empowerment Scale (ES) consists of 28 items rated on a 4‐point Likert scale (Rogers et al., 1997). The Herth Hope Index (HHI) consists of 12 items rated on a 4‐point Likert scale (Herth, 1992). The Rosenberg Self‐esteem Scale (RSES) consists of 10 items rated on a 4‐point Likert scale (Rosenberg, 1965), items were recoded so that high scores indicated higher self‐esteem levels. The Stigma Scale (SS) consist of 28 items rated on a 5‐point Likert scale, with high scores indicating low stigma (King et al., 2007). The Meaning of Life Questionnaire (MLQ) consists of 10 items rated on a 7‐point Likert scale (Steger et al., 2006). MLQ comprises two subscales: Search for and Presence of meaning of life, and only the Presence subscale was used. The MOS Social Support Survey (MOS) scale consists of 21 items rated on a 5‐point Likert scale (Sherbourne and Stewart, 1991). The Community Integration Measure (CIM) consists of 10 items rated on a 5‐point Likert scale (McColl et al., 2001). The Warwick–Edinburgh Mental Well‐being Scale (WEMWBS) consists of 14 items rated on a 5‐point Likert scale (Tennant et al., 2007). The Service User Perception of Functioning Scale (PPFS) consists of 5 items rated on a 5‐point Likert scale, scores range between 1 and 30 (Horn et al., 2012). The Manchester Short Assessment of Quality of Life (MANSA) consists of 12 items rated on a 7‐point Likert scale (Priebe et al., 1999).

Two additional measures were used. The Mental Health Confidence Scale (MHCS) consists of 16 items rated on a 16‐point Likert scale. The scale has good psychometric properties (Carpinello et al., 2000; Castelein et al., 2008). The Questionnaire about the Process of Recovery (QPR) consists of 22‐items rated on a five‐point Likert scale. This was the primary outcome of the REFOCUS Trial (Neil et al., 2009; Slade et al., 2011). An overall score was extrapolated by summing 15 of the original 22 items (Law et al., 2014). All scales were coded so that a high score indicated a positive outcome.

The two‐stage development process

Development and piloting

The PPO was developed using a narrative review of outcome domains in mental health (Bjorkman and Svensson, 2005; Myers et al., 2005; National Institute for Mental Health in England, 2008; Rush Jr, 2008; Slade, 2009) with specific emphasis on domains valued by patients (Faulkner and Layzell, 2000; Perkins and Repper, 2003; Rosenheck et al., 2005). For each domain, a related outcome measure was sought. This produced the initial version of IOM, Version 1, which was consulted on with the REFOCUS advisory boards, comprising a Lived Experience Advisory Panel (LEAP) of service users and carers, topic‐specific experts, a virtual advisory panel of service users, researchers and other stakeholders with an interest in Black and minority ethnic mental health, and an International Advisory Board of international experts. The consultation focused on identifying outcome domain omissions, measures for outcome domains, which did not have identified measures as well as to improve the content, order and format of the PPO. Following this consultation, IOM Version 2 was finalized.

A pilot study was, subsequently, undertaken with a convenience sample of 20 patients, to assess their experience of completing IOM Version 2. Characteristics of pilot study participants are shown in Table 1.

Table 1.

Socio‐demographic characteristics of participants

	Pilot study	Feasibility study	RCT evaluation
N (complete data)	20	75	340
Age (years)	40.9 years	42.3 (10.7)	43.5 (10.8)
Gender
Male	14 (70)	51 (68)	214 (63)
Female	6 (30)	24 (32)	125 (37)
Ethnicity
White	10 (50)	35 (47)	203 (60)
Black	6 (30)	32 (43)	85 (25)
Other	4 (20)	8 (11)	50 (15)
Diagnosis
Psychosis			340 (100)
Schizophrenia	5 (26)	35 (50)
Bipolar disorder	5 (26)	15 (21)
Depression	1 (5)	1 (1)
Mixed	4 (21)	14 (20)
Other	4 (21)	5 (7)

The pilot was conducted to identify any necessary change which may improve IOM. Informed consent was obtained and patients were paid £10 for their participation. All patients were able to complete and provide information on the IOM. Following the pilot, three measures of the PPO (APQ, SF‐36 and SF‐12) were excluded as they were not chosen by any respondent. Additionally, the ES was replaced with the MHCS. No changes were necessary for the GA component. Table 2 shows the list of 10 outcome domains and their associated measures which constituted IOM Version 3.

Table 2.

Personalized Primary Outcome (PPO) domains and baseline completion rates for feasibility study (n = 75) and evaluation in a RCT Version 3 (n = 340)

Outcome domain descriptions given to participants	Outcome domain being described	Associated measure	Feasibility study	Evaluation in a RCT
			N (%)	N (%)	Raw mean (standard deviation)
1. Feeling more hopeful about the future	Hope	Herth Hope Index	7 (9)	47 (14)	36.14 (5.1)
2. Feeling more in control of my life	Empowerment	Mental Health Confidence Scale	6 (8)	26 (8)	66.95 (17.1)
3. Feeling more positive about myself	Self‐esteem	Rosenberg Self‐esteem Scale	11 (15)	61 (18)	26.32 (4.4)
4. Feeling better treated by other people	Stigma	Stigma Scale	2 (3)	7 (2)	51.27 (13.2)
5. Feeling like you have meaning in your life	Meaning of life	Meaning of Life Questionnaire	7 (9)	42 (12)	24.45 (7.1)
6. Feeling supported by other people	Social support	MOS Social Support Survey	3 (4)	8 (2)	71.53 (13.9)
7. Feeling part of the community	Community integration	Community Integration Measure	3 (4)	11 (3)	30.46 (8.1)
8. Feeling better about your life	Well‐being	Warwick–Edinburgh Mental Well‐being Scale	8 (11)	39 (12)	45.26 (9.7)
9. Being able to manage day‐to‐day life	Daily functioning	Service User Perception of Functioning Scale	11 (15)	28 (8)	16.71 (3.6)
10. Having a better quality of life	Quality of life	Manchester Short Assessment of Quality of Life	17 (23)	71 (21)	55.90 (10.2)

Feasibility

We then assessed the feasibility of IOM Version 3 using a convenience sample of 84 current mental health patients, whose characteristics are shown in Table 1. They completed IOM at baseline and at three‐month follow‐up, and rated their experience of completing IOM. Participants were paid £10 for each round of assessment.

A total of 75 (89%) patients were able to identify a goal, a relevant outcome domain, and complete the associated PPO measure. The time to identify a goal ranged between one and five minutes [median = 1, interquartile range (IQR) = 1]. We found that 64 (85%) patients reported the goal was “very” or “extremely” important to them while only three (4%) were not happy about being asked. All 10 PPO domains and their associated measures were selected and completed (Table 2, column four). At follow‐up, 55 (73%) patients completed both IOM components. There was no association between attrition and any of the socio‐demographic variables. For the GA component, eight (15%) rated the goal as Fully Achieved, 28 (51%) as Partly Achieved and 19 (35%) as Not Achieved. Finally, 36 (65%) individuals rated their original goal as still Very or Extremely important.

Evaluation in a RCT

We then evaluated IOM Version 3 as an outcome measure in the REFOCUS Trial. The trial evaluated a team‐based intervention intended to impact on the content and process of care (Slade et al., 2011). The intervention was intended to lead to more collaborative staff–patient relationships, and greater staff‐focus on patient values, strengths, and goal‐striving. The intended benefits were more patient‐centred and recovery‐oriented care, and this emphasis on individualized care in this team‐based and trans‐diagnostic intervention meant that the use of a pre‐defined and invariant primary outcome was problematic in capturing the outcome domain relevant to each patient. The QPR was the pre‐defined primary outcome.

Participating teams (n = 27) were randomly allocated to control (n = 13, treatment as usual) or intervention (n = 14, treatment as usual plus REFOCUS intervention). A random sample of 15 patients was recruited from each participating team, and after giving informed consent, they completed a set of measures including QPR and IOM Version 3 at baseline and at one‐year follow‐up. Participants were paid £10 for each round of assessment.

Data analysis

The effectiveness of the intervention did not differ between arms (Slade et al., 2015), so data from both trial arms were pooled to address our hypotheses. We explored the relationship between attrition (missing versus present) and socio‐economic variables of gender age, ethnicity, marital status, education and accommodation type for the feasibility study and the trial. Missing data across the outcome scales were pro‐rated when less than 20% of items were left blank.

Prior to conducting our analysis, we recoded the IOM components to facilitate analysis. To avoid small cell counts, GA was recoded so that “successful” and “even better than expected” were coded as Fully Achieved, “some progress” was coded as Partly Achieved, and “no progress” and “further away from the goal” were coded as Not Achieved. To combine scores of the PPO, we standardized each measure using z‐scores based on population norms from previous research.

Objective 1 – change over time

In order to test if the difference was significantly different from zero, we conducted a regression analysis with no predictors on the PPO difference score between baseline and follow‐up. Subsequently, we assessed whether change on the PPO was clinically meaningful by implementing the Reliable Change Index (RCI) approach following the Jacobson and Truax's (1991) guidelines:

$$\mathrm{R}\mathrm{C}\mathrm{I}=\frac{x_2-x_1}{\sqrt{2{\left(s_1\sqrt{1-r_{\mathrm{x}\mathrm{x}}}\right)}^2}}$$

where r _xx is the Cohen's alpha (i.e. reliability) value and s is the scale standard deviation based on population values. RCI is a standardized measure of change and scores greater than the critical value (i.e. ± 1.96) correspond to reliable change (Zahra and Hedge, 2010). Using these cutoff points, we could group patients as having “Improved”, “Not changed”, or “Declined”.

Objective 2 – co‐variation

To assess whether the two IOM components are associated and tap onto the same construct, we regressed the PPO follow‐up score on GA while adjusting for baseline standardized scores.

Objective 3 – relationship with primary outcome

We assessed the relationship of the two IOM component with the REFOCUS Trial pre‐defined primary outcome of QPR by regressing the QPR change score separately onto the PPO change scores and GA.

In all regression analyses, we accounted for clustering at the team level by conducting random effects regression analyses with maximum likelihood estimation using the “xtmixed” command in Stata11. Site and trial arm were entered as co‐variates in the model in order to reflect the study design. Bonferroni corrections were implemented to adjust for multiple pairwise comparisons when appropriate.

Results

At baseline, 340 (86%) patient participants in the RCT completed both components of the IOM. Table 1 summarizes their socio‐demographic characteristics. Patients with complete IOM at baseline were more likely to be White British (versus other; χ ² ₍₁₎ = 8.5, p = 0.004), to live in private accommodation (versus supported; χ ² ₍₁₎ = 4.6, p = 0.03) and to have formal qualifications (versus no qualification; χ ² ₍₁₎ = 20.9, p < 0.001).

At one‐year follow‐up, 239 (of 340) patients completed both IOM components. Patients with complete information at follow‐up were more likely to have formal qualifications (versus no qualification; χ ² ₍₁₎ = 6.7, p = 0.01) and to be younger (t ₍₃₇₄₎ = 2.4, p = .02). All ten PPO domains and their associated measures were selected and completed (Table 1, column 5). The goal was rated as Fully Achieved by 63 (26%) patients, 113 (47%) rated it as Partly Achieved while 63 (26%) reported it as Not Achieved.

Objective 1: change over time

The regression analysis (average cluster size = 9, range: 1–13) indicated that the change score on the PPO between baseline and follow‐up was not different from zero (z = 1.3, p = 0.18, n = 239). The RCI approach indicated that 79% patients did not change over time (Table 3).

Table 3.

Reliable Change Index (RCI) at one‐year RCT follow‐up (n = 239)

	Cutoff (RCI = 1.96)	Change status n (%)
		Declined	No change	Improved
HHI	6.78	2 (6)	26 (79)	5 (15)
MHCS	8.69	5 (26)	8 (42)	6 (32)
RSES	7.30	1 (2)	42 (89)	4 (9)
SS	15.39	0 (0)	3 (100)	0 (0)
MLQ presence	8.34	1 (4)	24 (86)	3 (11)
MOS	11.62	2 (33)	4 (67)	0 (0)
CIM	7.70	0 (0)	3 (75)	1 (25)
WEMWBS	8.64	3 (12)	21 (84)	1 (4)
PPFS	2.36	2 (9)	14 (64)	6 (27)
MANSA	11.03	4 (8)	44 (85)	4 (8)
	Total	20 (8)	189 (79)	30 (13)

Note: HHI, Herth Hope Index; MHCS, Mental Health Confidence Scale; RSES, Rosenberg Self‐esteem Scale; SS, Stigma Scale; MLQ, Meaning of Life Questionnaire; MOS, MOS Social Support Survey; CIM, Community Integration Measure; WEMWBS, Warwick–Edinburgh Mental Well‐being Scale; PPFS, Service User Perception of Functioning Scale; MANSA, Manchester Short Assessment of Quality of Life

Objective 2: co‐variation

Analyses were conducted on all 239 participants with complete information on the IOM. A regression analysis across all 27 clusters (average cluster size = 9, range 1–13) showed an association between GA and PPO (χ ² ₍₂₎=13.8, p = 0.001; n = 239). Compared to patients who had Not Achieved, those who had Partly Achieved [b = 0.32, 95% confidence interval (CI) 0.09–0.56, z = 2.7, p = 0.007] or Fully Achieved their goal [b = 0.50, 95%CI 0.23–0.77, z = 3.7, p < 0.001] had higher PPO scores at follow‐up, while adjusting for baseline scores and accounting for multiple testing.

Objective 3: relationship with primary outcome

Regression analysis on all 237 patients with complete information on IOM and QPR (average cluster size = 9, range 1–13) showed that the PPO and QPR change scores were positively associated (b = 3.3, 95%CI 2.3–4.4, z = 6.2, p < 0.001). Similarly, regression analysis across all 27 clusters (average cluster size = 9, range 1–13) showed that GA was associated with follow‐up QPR scores (χ ² ₍₂₎ = 12.4, p = 0.002). In particular, patients who had Partly Achieved (b = 3.0, 95%CI 0.8–5.2, z = 2.6, p = 0.008) or Fully Achieved their goal (b = 4.3, 95%CI 1.9–6.8, z = 3.4, p = 0.001) reported greater recovery scores than those who had Not Achieved their goals, even after adjusting for multiple testing.

Discussion

Following growing consensus that individualized approaches may be an effective way to assess outcome in mental health research (Slade, 2009; Williams et al., 2015), a new measure of individualized outcome for use as a clinical end‐point in RCTs was developed, by literature review, expert consultation, piloting, assessment of feasibility and then evaluation in the context of a RCT. The IOM has two components, both of which had adequate acceptability and completion rates. Evaluation in a RCT showed that IOM could be used to differentiate between patients who changed versus those who did not change, that the two components (GA and PPO) co‐vary, and that both were associated with change in the pre‐defined primary outcome for the trial.

For GA, the majority of respondents (66% feasibility study, 72% RCT) reported having made at least some progress on their goal at follow‐up. These figures are in line with those observed in previous studies in mental health and rehabilitation (Hurn et al., 2006; Scott and Haggarty, 1984; Stevens et al., 2013). For example, 63% of patients with schizophrenia were found to have made good progress on their goal at six‐month follow‐up (Falloon and Talbot, 1982). These results suggest that the GA component, which is a variation of the GAS, is as effective as the original version at capturing achievement.

The PPO component did not capture any statistical or clinically meaningful change across time (Objective 1). Nonetheless, no change was observed on the REFOCUS Trial primary outcome (Slade et al., 2015), which suggests that change did not occur in the 12‐month period. Hence, implementation and replication in a future trial is needed. PPO scores at follow‐up were associated with GA (Objective 2), which indicates the two IOM components tap onto overlapping domains as originally intended. The PPO change scores were positively associated with change scores on QPR (Objective 3). Thus, it appears to capture change levels in similar fashion to that exhibited by a standardized questionnaire.

Finally, patients who had Fully or Partly Achieved their goal were more likely to report improved QPR scores than those who had not achieved their goal (Objective 3). These results are in line with evidence that GAS scores were positively associated with standardized measures of progress in rehabilitation (de Beurs et al., 1993; Hurn et al., 2006).

One limitation is the fact that we cannot ascertain whether goal attainment leads to a greater sense of recovery or whether increased recovery leads to GA as GA was assessed at follow‐up only. Previous research indicates that GA predicts progression in recovery (Clarke et al., 2009), and future studies implementing a mixed‐method design may help us to further understand this association. Patients were assumed to be failing at baseline but this could not be corroborated which represents a limitation. Future studies may incorporate baseline assessment to index change on this component, although this would require more complex calculations.

Combining the different measures of the PPO list may be criticized for potentially leading to biased results; however, we used a norm‐based standardization process which has been found to overcome these issues (Van Cleave et al., 2011). Finally, even though variations of the original RCI exists which take into account the regression to the mean phenomenon (Wise, 2004), research showed that the original RCI approach represents an optimal measure of change (Marsden et al., 2011; McGlinchey et al., 2002). The main strength of the study was the large and clinically representative sample recruited for a RCT in mental health, which renders our findings generalizable.

In summary, analysis of feasibility and evaluation of the IOM indicated that this approach can be implemented as an individualized outcome in RCTs for complex interventions in mental health. As the PPO provides an ecologically valid, standardized and more easily comparable approach to outcome measurement, we recommend the PPO component be retained, with the addition of the goal‐identification element of GA as selecting a goal facilitates identifying a relevant outcome domain. Future research should further assess the psychometric properties (e.g. internal consistency, construct validity and test–retest reliability) of the PPO and establish whether it could be implemented in different types of intervention in mental health research.

Declaration of interest statement

The authors have no competing interests.

Pesola, F. , Williams, J. , Bird, V. , Freidl, M. , Le Boutillier, C. , Leamy, M. , Macpherson, R. , and Slade, M. (2015) Development and evaluation of an Individualized Outcome Measure (IOM) for randomized controlled trials in mental health. Int. J. Methods Psychiatr. Res., 24: 257–265. doi: 10.1002/mpr.1480.