Outcomes From Consumer-Operated and Community Mental Health Services: A Randomized Controlled Trial

Abstract

Objective

Hierarchically organized board-and-staff–run consumer-operated service programs (COSPs) are viewed as organizations that promote recovery while working in concert with community mental health agencies (CMHAs). This study’s objective was to determine the effectiveness of such combined services for people with serious mental illness.

Methods

A board-and-staff–run consumer-operated drop-in center and colocated CMHA provided the context for the randomized clinical trial. In a weighted sample, 139 new clients seeking help from the CMHA were randomly assigned to agency-only service or to a combination of COSP and CMHA services. Client-members were assessed at baseline and eight months on a measure of symptom severity and on four recovery-focused outcome measures: personal empowerment, self-efficacy, independent social integration, and hopelessness. All scales used have high reliability and well-established validity. Differences in outcome by service condition were evaluated with multivariate analysis of covariance via dummy variable regression. Change scores on the five outcomes were the dependent variables. The covariates for the multivariate analysis included baseline status on each outcome measure and service condition between-group demographic differences.

Results

Results indicated that significant changes in three recovery-focused outcomes were associated with service condition across time: social integration (p<.001), personal empowerment (p<.006), and self-efficacy (p<.001). All changes favored the CMHA-only condition. Neither symptomology nor hopelessness differed by service condition across time.

Conclusions

Hierarchically organized board-and-staff–run COSPs combined with CMHA service may be less helpful than CMHA service alone.

In the past 25 years, the number of consumer-operated service programs (COSPs) for persons with serious mental illness has greatly increased. COSPs have become a major component of the mental health system and have been recognized as effective in fostering recovery (1). Government agencies (such as the Community Support Program of the Center for Mental Health Services of the Substance Abuse and Mental Health Services Administration) and advocacy organizations (such as the National Empowerment Center and the National Coalition of Mental Health Consumer-Survivor Organizations) promote a wide range of COSPs, including drop-in centers, clubhouses, independent living centers, case management services, employment agencies, and supported housing (2–8). When surveyed, almost half of a national sample of community mental health agencies (CMHAs) reported that they have referred clients to COSPs. CMHAs at which people with serious mental illness made up more than 50% of the caseload were almost 2.5 times more likely to make referrals; those where the CMHAs maintained a formal collaboration with a COSP were 7.5 times more likely to refer (9). These findings validate the growing and systematically supported role of COSPs in the provision of service to people with serious mental illness. Given their sanction as a future priority for funding, for example in California (10), COSPs are likely to play an increasing role in working with CMHAs to develop and implement treatment plans for individuals seeking services in public mental health systems. The objective of this study was to determine the effectiveness of such combined services for people with serious mental illness.

COSPs have certain key elements or “common ingredients” of program structure: Johnsen and colleagues (11) listed client involvement in hiring decisions, governance, and budget control; Mowbray and colleagues (12) cited client control in determining policies and procedures and a nonhierarchical program structure. However, COSPs have faced challenges in adhering to nonhierarchical structures. The COSP literature distinguishes two types of consumer-run organizations on the basis of their governance structures: the participatory, democratic self-help agency and the board-and-staff–run program (13,14). Self-help agency participant democracies give consumer-members a direct voice in major organizational decisions through participatory processes such as community meetings. Board-and-staff–run COSPs, like other COSPs, are primarily non-profit organizations, with a director who is a consumer, at least 50% consumer representation on the board of directors, and consumer authority to hire and fire professionals employed within the organization (15). In the board-and-staff–run COSP, however, this authority resides in hierarchically structured consumer leadership rather than collective decision-making processes. Leaders tend to be selected for their vision and ability to get things done (13). Given these characteristics, board-and-staff–run COSPs have been easier to develop because they do not require the creation and time-intensive implementation of a collective governance process. They are also more widespread and are thus more likely to be the type of organizational form chosen for future expansion of COSPs (16); their challenge is to ensure that leadership is accountable to the membership (16–18).

Although enthusiasm for COSPs is widespread, few consumer-managed interventions have been subjected to controlled trials (19–21). There also is limited understanding of how differential forms of COSPs contribute to achieving recovery objectives. One randomized controlled trial has demonstrated the effectiveness of COSPs organized as self-help agency participant democracies working in concert with CMHAs to promote recovery objectives (22). In this study, using similar methodology, we considered the effectiveness of board-and-staff–run COSPs in working with CMHAs to achieve recovery objectives.

Methods

Study setting and participants

The study design is a two-group randomized clinical trial, with a board-and-staff–run COSP and a CMHA serving a midsize urban area. The COSP we used includes peer support groups, material resources, drop-in socialization, and direct services. It provides help with money management and payeeship services, counseling, case management, peer counseling, assistance in obtaining survival resources (such as food, shelter, and clothing), and provision of information or referral. Its drop-in center program is open six days a week approximately eight hours a day with an average daily attendance of 50 individuals. It promotes mutual support between members. All members participate in a community meeting where program ideas are discussed and decided by vote. Major organizational decisions, such as budget and personnel actions, are delegated to staff, administrators, and the organization’s governing board. A former mental health care consumer serves as the COSP director, and the governing board’s membership comprises a majority of consumers; this leadership has control over personnel and budgetary decisions. Its organizational model is that of a “top-down” board-and-staff–run agency (13). The high degree of staff control distinguishes this program from five other COSPs in the region (23).

The CMHA was a county mental health organization that provides outpatient mental health services for people with mental illness. These services include assessment, medication review, individual and group therapy, case management, and referral. In addition to professional providers, several consumer case managers staff the CMHA, an increasingly common practice in CMHA service provision.

Study design

New CMHA clients, seen at a time when a researcher was available (N=189) between mid-1999 and 2001, were invited to enroll in the study; 162 (86%) agreed to participate. No significant differences in gender, race-ethnicity, and housing status were found between clients who agreed to participate and those who refused.

The CMHA and COSP programs were chosen for their proximity to each other (distance between the sites was 2.5 miles), which facilitated successful referrals and utilization comparisons. After entry into the CMHA and agreement to participate in the study, clients were randomly assigned to continued CMHA outpatient treatment or to a combination of COSP and CMHA services at a ratio of approximately 2:3, respectively, favoring the referral condition of COSP-CMHA combined services). The proportion assigned to CMHA-only service in the randomization process was used to ensure that an appropriate number of referrals were completed. This did not prove to be a problem.

Random assignment of clients accepted for CMHA services by a clinician occurred at the CMHA immediately after clients consented to participate in the study. After the decision to admit the client and the client’s agreement to participate in the study, the clinician opened an envelope that contained the client’s service assignment. The clinician told the client that he or she “would be assigned to a provider at the clinic and asked to attend a consumer-operated drop-in program” or simply that he or she would be “assigned to a provider.” Thus the group referred to COSP received both CMHA and COSP services, enabling a comparison of outcomes from receipt of both services with those from CMHA-only service.

Institutional review board approval was obtained for all study procedures, and all study participants provided informed consent.

Assessment and measurement

Interview schedules included questions on demographic characteristics and recovery indicators as well as the Diagnostic Interview Schedule for DSM-IV (DIS-IV) (24). These were pretested with a sample of 310 long-term users of self-help agencies in northern California (25) and with 30 CMHA clients. Interviews were conducted by former mental health clients and professionals trained by the Center for Self Help Research in Berkeley, California.

Participants were assessed on five outcome measures. The Personal Empowerment Scale (26) and the Self-Efficacy Scale (27), respectively, measure individuals’ control of and efficacy in their everyday lives. Individual social integration was assessed with the Independent Social Integration Scale (ISIS) (28,29), which measures five dimensions: social presence, access, participation, production, and consumption behaviors. Psychological functioning was assessed with the Brief Psychiatric Rating Scale (BPRS) (30). The Hopelessness Scale (31) assessed for a lack of hope. Higher scores on all scales indicated a higher degree of the measured characteristic. All scales possessed high internal consistency (α=.83–.95). Stability scores, measured across a six-month period, were in the moderate range (r=.48–.69). Each scale has independently established validity (25). BPRS interrater reliability was maintained throughout the study at a minimum of .80 with training film assessments.

Analyses

Analyses were completed with SPSS Version 18.0.0 (32). Frequencies and means were computed for the sample’s descriptive characteristics. Differences in individual characteristics between the COSP-CMHA and CMHA-only samples’ baseline characteristics observed in the follow-up samples were evaluated with chi square and analysis of variance tests. When between-condition differences were significant, the variable was selected for inclusion as a covariate control in the multivariate model evaluating the significance of outcome change attributable to the study condition.

Drop-in programs have individuals who attend with varying degrees of commitment; some individuals simply drop in for a cup of coffee, some to get out of the rain. Early work on such programs has emphasized the positive experience of “long-term users” of these services (25). To assess the experience of those fully participating in the service effort, we used SPSS’s generalized linear model (GLM) [weighting option to weight the samples by the inverse of the probability of reaching eight months of service in the client’s assigned service condition—that is, the number of people enrolled in the condition divided by the number who reached eight months of service. Data for those who did not reach eight months of service were given a weight of zero. This process gave greater importance to those completing the eight months of service and discounted those who did not. Results (see below) indicated virtually no difference in the probability of attaining the eight-month goal across conditions.

Using SPSS’s GLM multivariate procedure, we evaluated the change in recovery-focused outcomes associated with service condition assignment over an eight-month period. The GLM multivariate enabled an analysis of covariance with dummy variable regression that demonstrated the impact of service condition while taking into account baseline status on each outcome measure and observed between-group differences. Change scores (eight-month outcome score minus baseline) for each of the five recovery-focused outcomes were the dependent measures. Service condition (combined service, coded 1, versus CMHA only, coded 0) was the fixed factor. Baseline scores on all five outcome measures and the demographic characteristics of gender, education (less than high school, coded 0; high school completion or some postsecondary, coded 1), employment (employed, coded 1; not employed, coded 0), and receipt of Supplemental Security Income (yes=1, no=0) were the model covariates. The model was run twice. First, only the baseline scores on the outcome measures were used as covariates along with service condition to predict change outcomes; second, we entered the additional demographic covariates and reran the model.

Results

Of the 162 consenting participants in the weighted sample at baseline, 139 (86%) continued eight months of service: 86 (86%) in the combined-service condition and 53 (85%) in the CMHA-only condition. No one in the latter group self-referred to the COSP during the study. The mean± SD number of visits per month to the COSP was 4.24±.43 (median=4; mode=4); to the CMHA it was 2.1± 1.7 (median=1; mode=1). The sample’s mean age was 37.0±9.8, 75 (54%) were male, 121 (87%) were Caucasian, and 57 (41%) had a DIS-IV–assessed diagnosis of schizophrenia or schizoaffective disorder. Demographic and diagnostic information is listed in Table 1.

Table 1.

Characteristics of community mental health agency (CMHA) clients with serious mental illness who were referred or not referred to a consumer-operated services program^a

	Total sample (N=139)		Referral condition (N=86)		CMHA-only condition (N=53)
Characteristic	N	%	N	%	N	%	Statistic	df	p
Age (M±SD)	37±9.8		37±9.7		38±9.9		F=.21	1, 136	.650
Gender							χ2=6.40	1	.011
Male	63	46	46	54	17	32
Female	75	54	39	46	36	68
Marital status							χ2=1.05	2	.592
Married, separated, or widowed	12	9	6	7	6	11
Divorced	71	51	46	54	25	47
Never married	55	40	33	39	22	41
Race-ethnicity							χ2=1.08	2	.582
African American	12	8	6	7	6	11
White	121	87	76	89	45	83
Other	6	6	3	3	3	6
Housing status							—	—	—
Homeless	0	—	0	—	0	—
Not homeless	139	100	86	100	53	100
Supplemental Security Income
Beneficiary	68	49	37	56	34	63	χ2=4.99	1	.025
Not a beneficiary	71	51	48	43	20	37
Education							χ2=19.30	2	<.001
Less than high school	73	52	56	66	17	31
High school	29	21	9	11	20	37
Technical, some college, bachelor’s degree or more	37	27	20	23	17	31
Employment							χ2=9.87	1	.002
Not working	133	96	85	100	48	89
Working	6	4	0	0	6	11
Primary psychiatric diagnosis							χ2=.57	1	.448
Schizophrenia or schizoaffective disorder	57	41	37	43	20	37
Major affective disorder	82	59	48	56	34	63
Substance use disorder							χ2=.60	1	.433
Substance dependence	87	63	56	65	31	58
No substance dependence	52	37	30	35	22	41

^aWeighted sample

The multivariate analyses indicated that the two service condition groups changed differently across time with and without controls for the demographic differences. The initial model, including the service condition plus the five baseline outcome scores as covariates (service condition Wilks’ λ=.78, F=6.92, df=5 and 123, p< .001), indicated that this change was accounted for by two of the five recovery-focused outcomes: independent social integration (F=30.19, df=1 and 127, p<.001) and personal empowerment (F=4.32, df=1 and 127, p<.04). When the demographic controls were added to the final model, results remained significant (service condition Wilks’ λ=.81, F=5.72, df=5 and 119, p<.001), and three of the recovery-focused outcomes significantly contributed to the change: independent social integration (F=16.94, df=1 and 123, p<.001), personal empowerment (F=7.98, df=1 and 123, p<.006), and self-efficacy (F=12.16, df= 1 and 123, p<.001) (Table 2). All such changes favored the CMHA-only service condition. Scores for participants in the CMHA-only condition increased for social integration (mean increase=20.34±5.36 out of a possible 261), personal empowerment (mean increase=8.21±1.02 out of a possible 57), and self-efficacy (mean increase=5.04±1.69 out of a possible 56). In the combined-service condition, however, there were notable decreases in social integration (−9.29±4.05) and self-efficacy (−2.89± 1.28), and there was a modest increase in personal empowerment (4.34±.77). Symptoms, assessed with the BPRS, and hopelessness showed no difference in change between groups over the eight months.

Table 2.

Factors of change in social integration, empowerment, and self-efficacy after eight months of service with a consumer mental health agency (CMHA) or CMHA plus consumer-operated services

	Change in Independent Social Integration Scale (ISIS)a					Change in Personal Empowerment Scale (PES)b					Change in Self-Efficacy Scale (SES)c
Parameter	B	SE	td	p	ηp2	B	SE	td	p	ηp2	B	SE	td	p	ηp2
Service condition
CMHA only=.00	29.63	7.20	4.12	<.001	.121	3.87	1.37	2.82	.006	.061	7.92	2.27	3.49	.001	.090
Referral=1.00e	0	—	—	—	—	0	—	—	—	—	0	—	—	—	—
Covariate control at baseline
ISIS score	−.77	.10	−7.45	<.001	.311	.01	.02	.65	.518	.003	.05	.03	1.49	.140	.018
SES score	.82	.22	3.75	<.001	.103	.02	.04	.41	.679	.001	−.53	.07	−7.75	<.001	.328
PES score	1.36	.45	3.04	.003	.070	−.94	.08	−11.02	<.001	.497	−.37	.14	−2.61	.010	.052
BPRS scoref	5.86	1.09	5.37	<.001	.190	.36	.21	1.61	.109	.021	.12	.34	.35	.727	.001
Hopelessness score	−.52	.25	−2.05	.042	.033	−.15	.05	−3.18	.002	.076	−.44	.08	−5.55	<.001	.200
Gender	12.33	7.80	1.58	.116	.020	−2.15	1.48	−1.45	.150	.017	2.01	2.46	.82	.415	.005
Less than high school	−11.29	7.24	−1.56	.122	.019	−2.04	1.38	−1.48	.141	.018	7.21	2.29	3.15	.002	.075
Not working	9.16	15.67	.58	.560	.003	1.19	2.98	.40	.690	.001	25.02	4.95	5.06	<.001	.172
Receives Supplemental Security Income	−5.61	6.93	−.81	.420	.005	5.78	1.32	4.38	<.001	.135	−1.60	2.19	−.73	.467	.004

^aBetween subjects R²=.45; adjusted R²=.41; F=10.23, df=10 and 124, p<.001

^bBetween subjects R²=.75; adjusted R²=.73; F=37.18, df=10 and 124, p<.001

^cBetween subjects R²=.59; adjusted R²=.56; F=17.89, df=10 and 124, p<.001

^dRegression program assumes one degree for each variable in the model; thus specific degrees of freedom are not reported.

^eThe beta parameter was set to zero because of redundancy.

^fBrief Psychiatric Rating Scale

The overall effect size for service condition, measured by partial eta squared, was 19% of the variance. The individual outcomes yielded a partial eta squared of 12% for independent social integration, 6% for personal empowerment, and 1% for self-efficacy. The overall effect size can be characterized as large (>14%); the effect size for independent social integration and personal empowerment was medium (6% to 14%), and it was small for self-efficacy (<6%) (33,34).

The difference in average improvement between the CMHA-only and the combined-service conditions (that is, the absolute-risk reduction) was 15% for social integration, 7% for personal empowerment, and 14% for self-efficacy. The “number needed to harm,” by making a referral for each outcome, was, respectively, 6.6 for social integration, 14.7 for personal empowerment, and 7.1 for self-efficacy.

Discussion

Adding consumer-run helping efforts to CMHA services usually produces better outcomes. Our findings indicate, however, that joint board-and-staff–run COSP and CMHA services underperformed CMHA services alone. Independent social integration, personal empowerment, and self-efficacy improved to a greater extent in the CMHA-only condition than in the combined-service condition. On average, taking the group demographic differences into account, positive change in the CMHA-only sample was approximately three times that of the COSP-CMHA sample for social integration and self-efficacy, and twice that for personal empowerment.

These findings are best understood with reference to a similar trial (using the same methodology and outcome measures) in which the combined efforts of democratically run consumer self-help agencies and CMHAs substantially outperformed the CMHA-only condition (22). As noted earlier, self-help agencies and the board-and-staff–run COSPs differ. The former allow members a direct voice in major organizational decisions; the latter assign such decisions to staff. Was the CMHA-only condition more successful in this trial of a combined board-and-staff–run COSP and CMHA effort because the CMHA in this trial was superior to the CMHAs in the trial with self-help agencies? Unfortunately no; the participants in the CMHA-only condition in the self-help agencies trial had independent social integration, personal empowerment, and self-efficacy outcomes that were not significantly different from those in the CMHA-only condition in this board-and-staff–run trial (22). [An appendix providing details on those results is available in an online supplement to this article at ps.psychiatryonline.org.] Given this information, the performance of participants in the combined condition cannot be attributed to superior performance by the CMHA in this study. In the study of self-help agencies the referral group did better than the CMHA-only group, whereas in this study of board-and-staff–run COSPs the referred group did worse. In both studies CMHA-only participants performed similarly, falling between the high-performing referrals in the former and the low-performing referrals in the latter. The fact that the CMHA in this study had consumers working alongside professionals as case managers apparently made no difference in the performance of the CMHA participants in this trial.

Although there were baseline differences in the referral and CMHA-only sample characteristics in this study, these differences were taken into account in the analyses and thus are not likely to explain the underperformance of the joint effort herein. Although it is difficult to attribute the referral population’s poorer experience to a particular aspect of the COSP, its top-down decision-making procedures stand out as a probable contributing factor. The founding principles of consumer-led organizations emphasize the central importance of bottom-up as opposed to top-down administration—in the consumer movement’s terms, democracy-controlled decision making among self-help participants versus board-and-staff–run decision making in the COSP (17). From the outset of the movement, consumer leadership expressed concern about the possible failure of the latter to achieve the outcome potentials presumed inherent in the former. The consumer-led organization in this study was described by 49 consumer self-referrals as emphasizing organizational efficiency and staff control to the exclusion of membership participation in agency decision making, with the consequence of lower levels of membership organizational empowerment (23).

Historically, mental health consumers developed their own programs as alternatives to disempowering, professionally run services that limited participant self-determination. Such programs were based on the principle that participants can help themselves and each other through peer support, power sharing, and member control of services (17). Through these empowering processes, members were to regain hope, self-esteem, and self-confidence lost through stigmatization as persons labeled “mentally ill” (2,35–37). Finding oneself in a COSP that fails to organizationally empower may compromise recovery outcomes. McLean (38) documented the unfulfilled promise of a COSP drop-in center embedded in the community mental health system. Her findings associated the combined-services approach with the compromise of the key COSP principles of consumer participation in empowering decision making and the sacrifice of these goals to bureaucratic and funding pressures.

Early research indicated that “institutional neurosis” developed as an iatrogenic effect associated with the hierarchical structures of psychiatric hospitals, documenting the link between organizational structure and behavioral outcomes (39). This link has been made in community-based residential care homes with documentation of the behavioral impacts of management- versus client-centered organizations (40). The results herein seem to demonstrate the risks of hierarchical structures in a community-based organization.

This study has its limitations. It involved a single board-and-staff–run COSP. Whether the hierarchical structure that characterized this organization characterizes other board-and-staff–run COSPs is an empirical question yet to be addressed. This study represented only one case in a large number of possible organizational referral arrangements between board-and-staff–run COSPs and CMHAs. The study sample included a high proportion of individuals with major depression and none who were homeless. Although these sample characteristics were present in both study conditions, they may be less common in other settings. Statistically, the results are not generalizable beyond the participating programs. Yet the findings of this study are consistent with the consumer literature on the importance of organizational empowerment in the success of consumer-run program efforts, and they mirror the type of negative behavioral consequences seen in other disempowering mental health organizations. Further, the importance of any single-site randomized clinical trial should not be minimized, particularly when the findings represent a possible distinction between two interventions: a very helpful consumer-operated, democratically run self-help service and what appears to be an unhelpful consumer-operated board-and-staff–run service. These interventions now seem to be conflated into a single approach called consumer-operated services. Like all such findings, the findings of this study need confirmation through further research.

Conclusions

COSPs have made positive contributions to the alleviation of past stigmatization and disempowerment. Yet without a clearly defined process at the COSP for empowering members, referral from a CMHA to a COSP should be supported with caution, in that it may be less helpful than CMHA services alone.