Satisfaction with and Benefits of a Psych-social Club: Development of a Mixed Method Evaluation Instrument

Abstract

Deinstitutionalization of psychiatric mental health hospitals began in the 1960’s; thus, the delivery of mental health services became dependent on regional centers and community based, contracted services. One such service model is the psych-social club. Such clubs provide mental health services on the lower end of the cost continuum and reduce the utilization of higher cost services (Yurkovich, Labun, Cook & Lattergrass, 2005). This paper fills a gap by reporting on the development of a mixed method instrument entitled Benefits & Satisfaction Tool for Members of a Psych-social Club (B&ST-MPC). It also reports on the psychometric properties of the instrument, and measures members’ perception of satisfaction with and benefits of their psych-social club utilization, thus providing evaluative information for developing client centered continuums of care (Yurkovich et al., 2005). The paper includes a brief overview of the first mixed method study, background information on the rationale for instrument development, including methods used, and a discussion of the instrument’s psychometric properties.

Background and Framework

Deinstitutionalization of mental health care and hopes of lowering costs drove the development of service delivery models, which are currently integrated into local communities. However, reduced funding began undermining the true vision for this community delivery system and the continuum of care required by this population. Today, cuts to human service budgets continue on an almost annual basis at federal and state levels, with parity remaining a significant problem. In spite of these negative factors, elements of community-based services for persons with persistent mental illness (PMI) exist. One such service is the psych-social club model, which has the specific goal of assisting individuals experiencing PMI to maintain their wellness state (Yurkovich et al., 2005). Research indicates that a psych-social club’s mission consists of 1) providing a place to socialize safely with peers and staff, 2) offering opportunities to establish healthy peer relationships, 3) having supportive staff who give members feedback on their behavior within the club’s milieu, 4) providing a setting to teach members new life skills, and 5) helping members access needed resources. Persons with mental illness have identified all these processes as significant to maintaining wellness and preventing loss of control (Yurkovich, Buehler, & Smyer, 1997; Yurkovich et al., 2005; Yurkovich, Smyer, & Dean, 1999).

Evaluation of the effectiveness of psych-social clubs’ performance of their mission and their contribution to wellness of persons with PMI is vital in the development of best practice service models and the ability of the agencies to obtain funding resources. The evaluation must be based on and reflect members perceptions, the service mission, and goals of such clubs. Members and staff need to feel comfortable with the evaluation process, the sharing of results, and their ability to use the findings in future program planning and running of the club. In financially stretched times, it is important to learn what persons with PMI perceive as beneficial experiences related to attending and their degree of satisfaction with a psych-social club. With such knowledge, providers can continue building best practice environments that support maintenance of wellness in this community based population. Finally, a search in SCOPUS, CINHAL, and PubMed revealed no publications on tools that evaluated the effectiveness of psych-social clubs.

The tool created is complementary to and supported by the Clinical Values Compass (CVC) Model which includes four clinical service values (biological, functional, satisfaction, and cost) outlined by Beyea and Nicoll (2000). The model was developed during the authors’ evaluation of a patient service. The CVC Mode supports the psych-social club mission of helping maintain wellness and prevent loss of control. (Table 1) It contains biological outcomes, functional health status, member satisfaction, and cost measures. The assumption is that living in the community is less costly than being in a hospital treatment program.

Table 1.

Relationship of Clinical Values Compass Model (CVC) and Psych-Social Club Mission ^*

CVC Model Cost/Outcome Factors	Psych-Social Club Mission Statements
Functional	A safe place to socialize
	A place to develop healthy peer relationships
Satisfaction	Supportive staff who give feedback on behavior
	Teach/learn new life skills
	Help members access needed resources
Biological	Help members stay out of hospital
Costs	Help members stay out of hospital (maintain wellness and prevent loss of control)

^*The CVC Model Cost/Outcomes and Psych-Social Club Mission Statements are interactive and dynamic, influencing and affecting each other in both direct and indirect ways.

Instrument Development

Phase I: Research Supporting the Creation of B&ST-MPC

Phase one of the research, a mixed method design, emphasized the use of qualitative methodology and was the basis for creation of the B&ST-MPC. This study 1) examined what brought members to the club; 2) determined ways the program impacted members’ health and wellness in the community; and 3) identified other services the club could provide (Yurkovich et al., 2005). It included demographics, a semi-structured interview guide, and a 10 point satisfaction scale developed by the researchers. Semi-structured interview questions were used in audio taped interviews with 20 club members. The result was a comprehensive and rich measurement of membership use, perceived benefits, level of satisfaction with services, and effectiveness of the psych-social club for maintaining wellness, including functionality, in the community.

Additionally, phase one assisted with 1) identification of the values of a club model, and 2) identifying issues that can occur with this population during data collection. These insights aided in revision of open-ended questions, establishment of a “friendly” research process, and the development of the scales and items that became part of the B&ST-MPC. The aim was to develop an instrument that would effectively measure the members’ perceptions and experiences within their emotional and cognitive abilities/skills. (Table 2 lists sections of the instrument.) The results from phase one also showed that there is a significant role for qualitative research in providing clusters of information which health care providers can utilize clinically for program development and maintenance (Kearney, 2001) as well as instrument development for quantitative evaluation. Kessler et al. (2005) states that, “continued efforts are needed to obtain data on the effectiveness of treatment in order to increase the use of effective treatments” (p. 2515). (For a complete explication of phase one of this study, please see Yurkovich et al., 2005.)

Table 2.

Break out of Benefits and Satisfaction Tool - for Members of a Psych-Social Club (B&ST-MPC)

* Section one:	Members’ basic demographics Rate of attendance at the club
* Section Two	How they first accessed the club
* Section Three	18 items representing the healthy outcomes/benefits
* Section Four	Satisfaction scale
** Section Five	Question about knowledge of club’s services
** Section Six	Question focused on their thoughts about the club
** Section Seven	Question about areas of club improvement

^*Quantitative methodology

^**Qualitative methodology

Phase II: Instrument Development

Phase II addresses the development of the quantifiable mixed method instrument, B&STMPC, and testing of psychometric properties and member usability. The designers determined that the most effective way to represent the members’ collective wisdom and their level of language was to include items that closely represented their spoken words. Statements for the items are from the qualitative interviews (phase one) that incorporate themes and concepts from the findings; grammatical corrections were made on some consumer statements. For example, in the first phase a research participant stated, “I help them [peers] with their problems, ya know if they have a problem I listen. I’m somebody to talk to…It makes me feel good.” This was translated to two items as 1) When I am at the psych social club I help people with their problems, and 2) I feel good when I do something that helps others (e.g. listen to their problems). Because of the possibility that members might feel overwhelmed by a lengthy tool and the potential loss of attention/concentration, 18 different items make up the quantitative section. There is a proportional representation of themes and concepts based on repetition of and significance given to them by the consumers. To avoid clustering of themes, items were randomized prior to placement on the instrument (Please see table 2).

To support simplicity and ease of completion of the 18 items, the designers chose a Likert scale of Yes = 4, Sometimes =3, and No =2, Don’t Know = 1. Members were also asked to rate their overall level of satisfaction with the services of the psych-social club according to a linear scale of not satisfied = 1 to very satisfied = 10. The instrument fits on two pages back to back, thus meeting the need for brevity so as not to overwhelm the participants.

The research team that was involved in the first qualitative study reviewed the B&ST-MPC and provided feedback that directed relevant changes. A pilot test with 24 members (10 females and 14 males) created no new changes; this established internal face or content validity and usability. The pilot test of the B&ST-MPC also provided evidence that participants were comfortable with the tool and the researcher, and that the length and simplicity made it feasible to use with this vulnerable population. For best results, it is recommended that the instrument be used in a context where participants are familiar with the environment and the instrument administrator.

Testing the Instrument

Prior to testing of the instrument the project received ethical approval from the Internal Review Board of the researchers’ university. In addition, an agreement was made with each Club’s administrator to gather data and then provide a report regarding the findings. Before data collection began, a research team member spent approximately 3 to 5 days at the psych-social club interacting with members to reduce the effects of the confounding variable of fear/discomfort related to the presence of a researcher who was a stranger and etic to the environment. Once club members appeared to be comfortable with the researcher, data collection began. The research team member explained the study to the members in the following manner:

I am here at the request of the director of the club(s) to administer this survey to you in an attempt to find how satisfied or unsatisfied you are with your club; thoughts you have about the club; and, things you would like to improve within the club. If you agree to participate in this study about these topics, you will be asked to fill out the survey by answering a few questions. We can do this one of two ways: 1) I can read the survey to you, and fill in the survey for you from your responses; 2) Or, you can read and respond to the survey on your own. I will be here to answer any questions you may have on any portion of the survey. You are the experts as members of this club; and therefore there is no right or wrong answer. I will read the consent form to you and will provide you a copy of the consent form to take home with you upon completion of the survey. You will notice on the consent form, contact persons and information about the survey and study, should you desire more information.

Interested participants were given the survey and consent form (for which a waiver of signed consent had been granted by the University of North Dakota Institutional Review Board). Participants met the following criteria: adult, aged 18 years and older; able to read or understood when read to; and membership in the psych-social club, which required a self-reported mental illness diagnosis. A research team member was present to assist with any questions. The researchers who administered the tool were educated on the process according to the written directives for the instrument. No problems or complaints occurred during this process. Table 2 supplies a break-out of sections of the B&ST-MPC.

Sample Characteristics

Ninety two members, 49 males and 43 females, from three psych-social clubs in Midwest and Western regions of the rural United States participated in the study. All participants were sufficiently healthy and functional to attend the psych-social club on a regular basis. A few participants asked questions and received some assistance from the research team member. No one communicated nonverbal distress while completing the instrument.

Over half of the participants (53.3%) were males; the entire sample had an age range of 23 to 80 with a mean age of 45.4. Frequency of attendance was coded 1 = 1 to 2 times a week (37.9%), and 2 = 3 or more times a week (62.1%). Three days or more a week attendance was highest for the club in the largest city and 1–2 days a week attendance more likely in the smallest city; the club in the largest city has more diversified services. B&ST-MPC scale scores ranged from 20 to 72 ((χ̄ = 61.7); satisfaction scores ranged from 2 to 10 (χ̄ = 8.4). Mental illness diagnoses included depression, bipolar disorder, anxiety, panic disorder, post-traumatic stress disorder, attention deficit hyperactivity disorder, attention deficit disorder, obsessive compulsive disorder, schizophrenia, schizoaffective disorder, paranoid schizophrenia and brain trauma/injury. Some reported a combination of one or two disorders. This mixed membership is representative of most club populations.

Results

A number of statistics were computed to examine the psychometric characteristics of the B&ST-MPC. Coefficient alpha for the B&ST-MPC was .92. B&ST-MPC scores and satisfaction ratings were significantly correlated (r. = .243, p. = .032). An unweighted least squares factor analysis was conducted as a validity check. Unweighted least squares solutions have the advantages of running even if the correlation matrix is not positive-definite (Joreskog, 1977) and the distributions are non-normal (Nunnally & Bernstein, 1994); the method is frequently used for confirmatory factor analyses (Pett, Lackey, & Sullivan, 2003). Two items from the B&ST-MPC were deleted, “recognizing the need to take their medications” and “club environment better than other settings, e.g., bars,” because of low communalities, indicating low associations between the items. The low communalities may be due to cultural differences and or level of emphasis upon different services and activities among the three settings. Bartlett’s test of sphericity was significant, indicating that an identity matrix was not present and the matrix could be factor analyzed. The Kaiser-Meyer-Olkin Test (KMO) was .812, a value in the “meritorious” range of Kaiser’s (1974) criteria, indicating that the sample size was sufficient for factor analysis (Pett et al., 2003). Three factors had initial Eigenvalues of greater than one. A varimax rotation was used in order to have the clearest differentiation between factor loadings, and a criterion of .40 was set for acceptance of a loading on a factor.

Table 3 displays the rotated factor loadings that fit this criterion. The three factors derived from the analysis have the headings of productive socialization, belonging/building self-esteem and learning to stay well from others and the environment. The items loading on each factor represent the proportionate split established during the instrument development process based on the members’ repetitive description of behaviors. The items under the factor productive socialization include activities and feelings related to experiencing people in a positive way through a safe supportive environment that allows for personal growth during interactions, experiences, and helping others. This creates or enhances a sense of purposeful doing. The items loading on this factor measure the first three mission statements (See Table 1) of a club. The second factor is belonging/building self-esteem. Belonging included a demonstrated level of comfort (“able to relax,” “don’t feel alone”) and the presence of trust that helped members accept assistance from others at the club. Self-esteem was enhanced through altruistic processes (“help people”) and reinforced by feelings of self worth and performing as normal people. The third factor, learning to stay well from others and the environment, incorporates the identification of experiences that support learning (which builds on their belonging experience) and socialization skills by observing the environmental context and working with other club members. These skills enhance a member’s level of functioning in the community and reduce the frequency of hospital stays.

Table 3.

Factor Loadings for Exploratory Factor Analysis with Varimax Rotation^*

Tool Items	Factor 1 Productive Socialization	Factor 2 Belonging/ Building Self-esteem	Factor 3 Learning to Stay Well
Feel safe	.793
Others should show respect	.757
Comfortable, no pressure	.710
As good as anyone else	.661
Feel good – help others	.591	.515
Visit with people	.536
Don’t feel alone	.459	.423
Attending gets me food	.669
Worth something	.625	.665
Help people with problems		.591	.430
Feel good to do things		.576
Learn about paying bills		.533
Learned how to socialize		.409
Since in club been in hospital less			.686
Help me know not feeling well			.580
Gives ability to try & do things			.527

^*Rotated factor loadings of .40 or greater

“Feel good-help others”, “don’t feel alone”, and “worth something” loaded across factors one and two. This is consistent with the importance of the role of feelings of worth and feeling good by helping others in regard to socialization and belonging/self-esteem. The items loading on factors two and three provide for the evaluation of satisfaction with or benefiting from the third, fourth and fifth mission statements of a club. “Help people with problems” loaded on factors two and three; this is appropriate because members indicated that they experienced personal growth in the process of helping others. In order for club members to be successful in the maintenance of wellness in the context of community, the items involving the development and utilization of social skills appear to cross all three factors; this reinforces the central significance of this role in the club.

The results from the qualitative study also provided insight into the interactive, relationship-building process currently present in the club’s milieu (context). For example, there were statements of concern about safety, members needing to follow the club rules, staff not enforcing all the rules equally, and a prevalence of gossiping in the social milieu; all of these factors could influence the items loading on factors one and two and their correlation to satisfaction. Comments indicating a shortage of volunteers suggest problems regarding the ability of staff to “be out of their office,” thus interfering with the role modeling of social skills and sustaining offensive “gossiping.” Likewise, written comments from members may affirm the need for group education about social processes and skills as an important element of evidence based practice in club settings.

Factors 1 and 2 were significantly correlated with satisfaction (r= .331, p = .012 and r = .279, p = .036, respectively). Factor 3 correlated with satisfaction in the expected direction but the relationship was not statistically significant (this could be due to the more abstract nature of this factor).

Discussion

The results provided evidence of the acceptable internal consistency and validity of the B&ST-MPC . The findings affirm that the items reflect the benefits that the members experience while using a psych-social club. Factor analysis yielded three factors representing similar themes and concepts that logically incorporated the six sub themes that emerged from the qualitative study. The first two factors included items that demonstrated satisfaction with the benefits implied in the first two mission statements for a psych-social club (providing a place to socialize safely with peers and staff and offering opportunities to establish healthy peer relationships). The items loading on Factor 1: Productive Socialization are related to being able to experience people in a positive way through a safe supportive environment that allows for personal growth during interactions, activities, and helping others. This creates or enhances a sense of purposeful doing.

Factor 2: Belonging/Building Self-Esteem represents a proportionate split of concepts established at the beginning of the instrument development process. Belonging includes a demonstrated level of comfort (“able to relax,” “don’t feel alone”) and the presence of trust that helps members accept assistance from others at the club.

Factor 3: Learning to stay well from others and the environment incorporates the identification of experiences that support learning basic life supporting and socialization skills. These skills enhance a member’s level of functioning in the community and reduce the frequency of hospital stays. The items loading on factors two and three provide for the evaluation of satisfaction with or benefiting from the third, fourth and fifth mission statements of a club (having supportive staff, a setting to teach members’ new life skills, and helping members access needed resources).

The researchers recognize that psych-social clubs are designed to be places where community-dwelling persons with a diagnosed mental illness can socialize without the feeling of being stigmatized by their mental illness. Members participate and contribute to the programs, environment, and services as their health and circumstances permit; they can assume roles of member or leader in program planning and the administration of the club. Taking on such responsibility is empowering, furthers their sense of normalcy, contributes to recovery, and supports engagement in the citizenship of the community. Therefore, the focus of the B&ST-MPC mixed method instrument is on the assessment of wellness or desired outcomes rather than illness or deficits (Radwin, Washko, Suchy & Tyman, 2005). The focus on wellness provides a positive view of the abilities of this population and allows for development of potential rather than measuring the deficits. Such an approach is in keeping with the philosophy and mission of psych-social clubs.

Furthermore, written responses from club members demonstrated the participants’ ability to understand the open ended questions. Responses also verified that participants were capable of replying to the satisfaction scale without difficulty, displaying congruency of findings between the different instrument sections. Results confirm that these sections effectively contribute to the evaluation process and should be included with the quantitative items, thus providing a rich picture of the psych-social club’s benefits. The B&ST-MPC instrument provides different approaches to measurement of benefits from and satisfaction with experiences in a psych-social club, all of which is informed by the compass framework components (Beyea & Nicoll, 2000)

Limitations

Testing of the B&ST-MPC tool was carried out in psych-social clubs that are administered by professional staff and community volunteers. Members had input into governance of these clubs. The membership consisted of persons with PMI residing in the three clubs located in the Midwest and Western Region of United States drawn mainly from their surrounding rural areas. Participants self-selected for the study. A larger number of participants could have strengthened the statistical analysis. No adolescents participated in the study, although older adolescents and young adults could be members. The original instruments did not include the identification of cultural or ethnic variability; this has been added to the second draft. Although the sample size tested as adequate for factor analysis, a larger sample would have helped reduce the possibility of sampling error (Pett et al., 2003).

Recommendations

The authors recommend further testing for ongoing analysis of the items with more ethnically diverse groups and utilization of large urban centers in other geographical regions. Assessment of the community milieu through additional qualitative questions could enhance understanding of the needs of persons with PMI in a specific community. The authors recognize that benefits from the milieu are affected by feelings of comfort, safety and satisfaction. Thus, clubs are encouraged to add open-ended questions that may be specific to their locale.

Conclusion

The use of the clinical values compass framework in developing comprehensive best practice models for persons with PMI moves care into the wellness framework and builds on the strengths rather than the maladaptive behaviors of this population. Valid and reliable instruments that measure the benefits of and satisfaction with a psych-social club give members and administrators the evidence needed for further development of their programs and accessing funding resources. The instrument analyzed in this study presents solid psychometric properties that will be enhanced with further testing in diverse populations. The goal of wellness for persons experiencing PMI is well served by evidenced based services that mirror the perceptions and aspirations of this population.