Quality Indicators for Family Support Services and Their Relationship to Organizational Social Context

Abstract

Quality measurement is an important component of healthcare reform. The relationship of quality indicators (QIs) for parent-delivered family support services to organizational social contexts known to improve quality is unexamined. This study employs data collected from 21 child mental health programs that deliver team-based family support services. Performance on two levels of QIs —those targeting the program and staff -- were significantly associated with organizational social context profiles and dimensions. High quality program policies are associated with positive organizational cultures and engaging climates. Inappropriate staff practices are associated with resistant cultures. Implications for organizational strategies to improve service quality are discussed.

Introduction

Family participation in mental health has been an essential element of service quality since the mid-1980s, when the US federal government funded the Child and Adolescent Service System Program (CASSP) (Stroul, 1996). Family participation is viewed as essential to service quality because of the key role parents and caregivers (hereafter referred as parents) play in the lives of children. Parents often provide critical information and play an important role in service engagement and treatment adherence. Parental perceptions of the relevance of treatments and therapeutic alliance with providers influence service utilization. The degree to which parents are involved in service planning and problems with the therapeutic relationship have been found to predict premature drop-out and treatment completion in different populations (Garcia & Weisz, 2002; Kazdin, Holland, & Crowley, 1997; Koren, Paulson, Kinney, Yatchmenoff, Gordon & Dechillo, 1997; McCabe, 2002).

Efforts to increase meaningful family participation in child-serving systems have achieved only modest success due in part to divergent views of parents and professionals regarding the genesis of child emotional and behavioral problems and parents’ role in treatment (Donner, 2003; Stroul, 1996). Various technical assistance organizations involved in developing comprehensive community systems for children report significant challenges and frustrations with respect to parent and professional relationships (Peer Technical Assistance Network, 2000). In response to these challenges, parent advocates and clinicians have partnered to develop a range of new service models designed to enhance the participation of families in children’s mental health care (Donner, 2003; Winters & Metz, 2009). Parent-delivered family support services represent one such service model that has gained national prominence during the last decade (Walker & Crocker, 2008). Estimates from national surveys of family organizations suggest that the number of family support specialists (FSS) in the children’s mental health field has risen dramatically in the past decade (Hoagwood et al., 2008; Robbins et al., 2008), with a growing number of states developing certification standards (e.g., GA, MD, MN, ME, FL, NY). However, attempts to implement parent-delivered family support services have encountered a number of the same organizational, ideological, and operational barriers that hindered earlier attempts to implement CASSP and system of care initiatives.

Theories of innovation diffusion (Rogers, 2003) and organizational change (Klein & Sorra, 1996) suggest that the repeated difficulties encountered in integrating family-centered program models into routine mental health settings may be rooted in the organizational social contexts within which these service innovations are introduced. The philosophy, assumptions, and norms of family support services and other family-centered care models diverge significantly from the shared assumptions and norms (i.e., organizational cultures) of many traditional children’s mental health services. Moreover, family-centered service models require significant changes to practice processes for child-serving organizations, which are already struggling with highly stressful and sometimes minimally functional organizational climates. Thus, a potentially important but unexamined barrier to the successful implementation of parent-delivered family support services may be a poor fit between the organizational social contexts (i.e., organizational cultures and climates) of traditional children’s service settings and the underlying assumptions and behavioral norms required for high quality family support services implementation.

The present study examines these links by exploring the association between two dimensions of organizational social context—organizational culture and climate—and the quality of family support program policies and individual family support services provided by FSSs. Results from this study form a basis for understanding how organizational social context may relate to the implementation of family support services and could inform organizational strategies for enhancing the delivery of parent-delivered family support services.

Parent-Delivered Family Support Services

Parent-delivered family support services are designed to address a range of issues related to family preferences and service access (Olin et al., 2010a). Family support specialists (also known by a variety of other names including family partners, family support workers, parent partners, parent mentors, veteran parents) are typically parents of children with special needs who have moved into a professional provider role designed to help other parents of children with special needs navigate the complex children’s service system. Family support specialists serve as peers who model, coach and support parents in their journey to understand, cope with, advocate within, and negotiate across various service systems (Olin et al., 2010a). Two key goals of parent-delivered family support services are to facilitate access to services and to increase family voice in the treatment process (Olin et al., 2010b).

Prior to recent efforts by the National Federation of Families for Children’s Mental Health to establish a Certification Commission for Family Support (http://certification.ffcmh.org/), no national standards of practice existed to guide delivery of parent-delivered family support services in children’s mental health. Service models for parent-delivered family support services vary across settings, such that the roles, responsibilities and activities (and hence competencies) of FSSs can vary significantly depending on context (Obrochta, Anthony, Armstrong, Kalil, Hust, & Kernan, 2011; Olin et al., 2010b). Such variation and lack of consensus about essential elements and competencies associated with FSS services are not uncommon in the field of peer-led services (Richard, Jongbloed, &MacFarlane, 2009; Salzer, Schwenk, & Brusilovskiy, 2010), where a primary criterion for the position is based on documentation of lived experience (see National Certification application for parent support provider: http://certification.ffcmh.org/). As family members move from a consumer to a professional role, however, training and staff development to help peer parents acquire greater skills in working with parents of children with behavioral/emotional needs will be important (Pearson & Lambert, 2012; Spencer, Blau, and Mallery, 2010).

In parallel to national efforts, New York State (the context for this current study) has also been developing competency standards for this workforce, with concurrent efforts to strengthen workforce training through the dissemination of a FSS training called the Parent Empowerment Program (Jensen & Hoagwood, 2008, Olin et. al., 2010b; Rodriguez et al., 2011). The Parent Empowerment Program, and other programs like it, has begun to establish core competencies and standards of practice for this emerging profession. However, the degree to which such standards relate to service quality and outcomes is unknown. Despite the infusion of significant resources for family support services, FSSs in New York and elsewhere struggle to adhere to practice guidelines and philosophy, in part, due to the lack of structures and policies within their host organizations to support such practices. FSSs often report organizational barriers, including lack of understanding of FSS role or their value (Parsons & Lambert, 2012). Among those who work on teams, philosophical differences exist between professional providers and FSSs in the use of academic vs. personal or lived experience; this can create tensions that interfere with FSSs’ ability to be effective brokers between families and providers, and to promote family empowerment and shared decision-making. These differences create significant challenges to effectively integrating this service component within children’s mental health programs.

To date, the evidence base for parent-delivered family support services is limited although evidence on adult peer services is growing (Corrigan, 2006; Dixon, Stewart, Burland, Delahanty, Lucksted, & Hoffman, 2001; Min, Whitecraft, Rothbard, & Salzer, 2007; Sells, Davidson, Jewell, Falzer, & Rowe, 2006). This is changing, but slowly (Hoagwood et al., 2010). Despite limited evidence for family support, these services are expanding nationally. To keep pace with this expansion and align it with the new healthcare emphasis on accountability and performance standards, metrics for measuring quality of services are sorely needed (IOM, 2006; Pincus et al., 2007; Pincus et al, 2011). Towards this end, a modified Delphi approach was used to establish an initial set of quality indicators (QIs) to characterize family support programs and the skills or competencies of FSSs (Olin et al., this issue). These QIs were pilot tested among a sample of 21 Home and Community Based Services (HCBS) Wavier Programs that included parent-delivered family support services as one of six service options available to families of youth with serious emotional disturbance (SED) who are at risk for institutional placement. Within these HCBS Waiver programs, those with higher quality family support policies had individual FSS staff who exhibited higher quality standards of practice, suggesting a link between organizational factors and the delivery of effective parent-delivered family support services (Olin et al., this issue).

Role of organizational social context in service quality

An organization’s social context is an important, yet often overlooked factor in influencing the quality of services and child outcomes in human service organizations (Glisson, 2007; Glisson & Green, 2006; Glisson & Hemmelgarn, 1998). Organizational social context includes the organization’s culture (i.e., shared behavioral norms, work priorities, and expectations of frontline service providers), as well as the organization’s climate (i.e., the psychological impact of the work environment on providers’ sense of wellbeing and functioning). Organizational culture and climate have been demonstrated through multiple studies to affect service implementation and quality (Aarons & Sawitzky, 2006; Brunette et al., 2008; Carr, Schmidt, Ford, & DeShon, 2003; Glisson, 1978; Glisson, 2007; Glisson, 2008; Glisson & Durick, 1988; Glisson & Green, 2006; Glisson & Hemmelgarn, 1998; Glisson, Hemmelgarn, Green & Williams, 2013; Glisson, et al., 2008a; Glisson, et al., 2010; Greener, Joe, Simpson, Rowan-Szal, & Lehman, 2007; Guzzo, Jette, & Katzel, 1985; Neuman, Edwards, & Raju, 1989; Parker et al., 2003; Roberston, Roberts, & Porras, 1993; Sheridan, 1992; Shim, 2010). Specifically within mental health, provider attitudes toward evidence based practices or innovations have been found to be associated with organizational context (Aarons & Sawitzky, 2006). Relevant to this present study, the role of organizational culture in supporting provider practices and attitudes related to family centered care has been demonstrated in pediatric emergency rooms (Hemmelgarn, Glisson, & Dukes, 2001). Similarly, dimensions of organizational social context are likely to be important in this study because of the ideological and philosophical differences between parent-delivered family support services and more traditional clinical approaches.

The involvement of parents as providers represents a significant paradigm shift in the field of children’s mental health, where parents have traditionally been viewed as passive recipients of care at one extreme, or as primary contributors to their children’s problems at the other extreme (Donner, 2003). The involvement of parents as service providers may thus challenge the underlying assumptions and behavioral norms (i.e., organizational cultures) of traditional children’s mental health organizations that see families as contributing to children’s emotional and behavioral difficulties. Moreover, the increased work demands required of providers who try to implement new family-centered care models may be inhibited by dysfunctional organizational climates that hinder providers’ and FSSs’ attempts to integrate these new program models. Glisson, Hemmelgarn, Green, & Williams (this issue) examined the organizational social contexts of the same 21 programs used in the present study, and found significant variations in their cultures and climates. Further, FSS and other provider staff reported significantly higher individual levels of job satisfaction and organizational commitment in programs with the best cultures and climates. These variations in cultures and climates across the 21 programs point to the utility of examining how social context variations might relate to programs’ performance on a set of newly developed quality benchmarks of family support services.

In this paper, we explore whether performance on quality benchmarks targeting two levels of indicators (Program and Individual FSS practices) relate to OSC profiles and dimensions of culture and climate; this represents an initial effort to validate these quality measures. We hypothesized that HCBS Waiver programs’ performance on both quality indicator domains (program and staff level) would be significantly associated with the organizations’ cultures and climates. Specifically, organizations with positive cultures and climates are more likely to have program policies that support family support services and higher quality individual FSS practices. Understanding how specific culture and climate dimensions relate to the Program and Individual FSS delivery of family support services can identify strategies to improve integration of FSSs within these programs. An important goal of this study is to guide the development and implementation of effective organizational strategies for improving quality of care, financing, and regulation for team-based models of parent-delivered family support services.

Methods

Study Sample

All 33 OMH-funded HCBS Waiver Programs in New York State that serve children with SED were invited to participate in this study. Of these, three programs were deemed ineligible for participation because they did not have at least one half-time FSS or an FSS on staff for at least 6 months. Twenty-one (64%) programs volunteered to participate in this study. These 21 programs did not differ from the rest of the non-participating HCBS programs across the state in terms of program capacity (number of families that can be served ranged from 12–144; mean = 48.9, SD = 37.5) and region of the state, with at least half the programs in each of the five regions participating in the study. Participation rates were highest in the Western (and most rural) regions of the state. All data were collected between August and October 2011.

Measures

Measures of Quality Indicators

A modified Delphi approach was used to develop Program and Individual FSS QIs for parent-delivered family support services in HCBS Waiver Programs (see Olin et al, this issue). The Program and Individual FSS QIs are shown in Table 1. Fourteen Program QIs focus on program policies and practices for parent-delivered family support services, while 27 FSS QIs focused on individual FSS provider practices (six of the 27 QIs focus on Inappropriate practices). Each HCBS Waiver Program was rated on each QI set using two sources of data, collected between April and October 2011: Program QIs were based on ratings of semi-structured interviews with the HCBS Waiver Program Director, one Individualized Care Coordinator, and one FSS. The Program Director completed a modified MacArthur’s Survey (Schoenwald et al., 2008) including questions focused on fiscal issues, the case referral process, staff supervision and training opportunities, and staff roles and interactions. All three respondents were interviewed about the structure of family support services, access to FSS services, and ways FSS worked with other staff in the program to serve families. FSS QIs were based on independent ratings of a Standardized Walkthrough procedure with one selected FSS from each program to examine process and content of FSS services. Detailed descriptions of these QI ratings are described elsewhere (Olin et al., this issue). Psychometric properties of these quality indicator measures are acceptable, with excellent inter-rater reliability (average of 98% on Program QIs and 96% for FSS QIs) and good variation in scores across both indicator sets

Table 1.

List of Program Level QIs and FSS Level QIs

Program Level QIs by Domain

A program that provides high quality family support services includes…ROLE 1Clear roles and responsibilities for FSSs (e.g., detailed job descriptions available) 2Staff that clearly understand the role of the FSS within the goal of the program FISCAL 3A specific budget allocated for family support services 4Program flexibility in use of family support services even when such services are not billable (e.g., pre-enrollment into program and post discharge) 5Employment benefits provided (e.g., health, vacation) to FSSs STANDARDS TO GUIDE PRACTICE 6A standardized protocol or framework used by program staff with all families to guide intensity, type and progress of family support services relative to family needs, goals and strengths 7FSSs that have received training or certification in core competencies 8FSSs employed with experience of parenting a child with emotional, developmental, behavioral, substance use or mental health concerns STRUCTURAL 9Specialized supervision/consultation structures in place to help integrate FSS role on the team and with families 10Structures in place to facilitate teamwork among FSSs and other staff on team (e.g., team meetings) 11Informal communications among team members outside of planned or structured meetings (e.g., phone calls, discussions in passing) 12Experienced family members (non-employees) as part of a board within agency 13Program flexibility to allow integration of new/innovative ideas from staff 14Program integration of the FSSs as equal members and/or active members of the team in working with a family

Individual FSS Level Quality Indicators by Domain

A skilled FSS.....ROLE 1Describes FSS role and what they do in relationship to other team members. 2Describes or demonstrates boundaries of own roles and responsibilities toward families RELATIONSHIP WITH TEAM 3Demonstrates positive regard for the role of other staff on team 4Supports the development of more effective partnerships between family and other members of the team. ETHICS 5Discusses how information they learn is shared among staff and outside family. 6Establishes a quiet, confidential location to talk (whenever possible at location of caregiver’s choice, e.g., home, school, diner, etc.). KNOWLEDGE 7Demonstrates knowledge of community supports and resources. SKILLS IN PRIORITY SETTING 8Know how and when to facilitate goal oriented and solution focused problem solving to help the family accomplish their goals. 9Identified safety concerns and existence of safety plan; works with family and team to initiate a plan as appropriate and/or address the adequacy of existing plan. 10Uses systematic and standardized methods to work collaboratively with the family to identify goals of family support services based on family strengths and needs 11Uses systematic and standardized methods to work collaboratively with the family to monitor the progress of goals of family support services. 12Facilitates a family’s identification of priorities and concerns. SKILLS IN PROVIDING LINKAGES 13Works with family to identify, reconnect and or build their formal and informal support system. SKILLS IN PROVIDING EMOTIONAL SUPPORT 14Facilitates caregiver identification of ways to promote self-care. 15Uses his or her own experiences to support and or normalize a caregiver’s experience and promote hope SKILLS IN EDUCATING 16Promotes empowerment by targeting assistance to caregiver need. 17Exercises sound judgment when providing advice or recommendations. 18Models or coaches caregiver use new skills (e.g. through role plays). FAMILY ENGAGEMENT 19Promotes family voice and choice through shared decision making (e.g., helps family articulate cultural, spiritual and/or religious values and preferences). 20Reframes or clarifies a caregiver’s perspective or position in a way that avoids criticism or judgment of caregiver. 21Uses strength-based language. An inappropriate action of a family peer advocate is one that .... FAMILY ENGAGEMENT 22Uses communication that indicates blame or criticism of caregiver. 23Provides advice or service that is beyond the scope or role of FSS (e.g., legal or medical advice, transportation, providing child care). 24Uses deficit-based language. 25Uses medical jargon inappropriately (e.g., in a patronizing way). 26Is directive and makes decisions independent of the caregiver about what is good for the family. RELATIONSHIP WITH TEAM 27Works with a caregiver to go around decisions of the rest of the team

Measure of Organizational Social Context

The Organizational Social Context (OSC) Survey, a nationally-normed agency social context measure to assess dimensions of organizational culture and climate (Glisson, et al., 2008a), was used to assess the organizational social contexts of the 21 HCBS Waiver Programs.

The OSC assesses organizational culture, defined as the norms that drive the way work is done in an organization, on three dimensions: proficiency, rigidity and resistance. Proficient cultures expect that staff will place the wellbeing of each client first, and that staff will be competent and have up to date knowledge. The two subscales underlying this dimension measure responsiveness and competence. Rigid cultures have staff who provide limited input into management decisions, who have no flexibility in job functions, and whose behavior is governed by a fixed set of confining rules and regulations. The two subscales underlying this dimension measure centralization and formalization. Resistant cultures expect staff to show limited interest in new ways of service provision and to suppress any effort to change. The two underlying subscales of this dimension measure apathy and suppression.

The OSC also assesses organizational climate, defined as the shared perceptions of the psychological impact of the work environment on staff wellbeing and functioning, on three dimensions: engagement, functionality and stress. Staff in engaged climates perceive that they are capable of accomplishing worthwhile things in their job and are able to remain personally involved in their work and concerned about their clients. The two underlying subscales of engagement measure personalization and personal accomplishment. Staff in functional climates perceive that they have the support and cooperation they need from coworkers and administrators to do a good job and have a clear understanding of successful behaviors; the three underlying subscales measure growth and achievement, role clarity and cooperation. Staff in stressful climates experience emotional exhaustion and work overload, and feel conflicted in their job responsibilities; the three underlying subscales measure emotional exhaustion, role conflict, and role overload.

The OSC was administered in person to 223 HCBS Waiver staff from 21 programs. All frontline staff employed at least half-time were included; no supervisors were present. The number of staff per program ranged from 4 to 27. Out of the 223 OSCs, 14 surveys were deemed invalid due to inconsistent ratings and/or insufficient data (more than 10% missing data). OSCs profiles were generated using the 209 valid OSC surveys. All data were collected at site visits between May and October, 2011.

Organizations’ cultures and climates were characterized by aggregating staff responses within each program to the OSC on each of the six dimensions described above. These aggregated scores were transformed to t-scores based on national OSC norms for children’s mental health clinics thus producing norm-based OSC profiles on the three dimensions of culture and three dimensions of climate for each program (Glisson, et al., 2008a). Details of the aggregation process and evidence of adequate inter-staff agreement to justify aggregation of responses for each of the 21 programs in this study, is provided in Glisson et al. (this issue).

Following the generation of norm-based OSC profiles for each agency, latent profile analysis was used to classify agencies into one of three empirically-derived classes, corresponding to overall negative, neutral, or positive OSC profiles (see Glisson et al. this issue for details on the classification process). Latent profile analysis is analogous to factor analysis except that LPA classifies individuals, or in the present study, organizations (Muthén & Muthén, 2008; Vermunt, 2002). Parameter estimates from LPA provide means and variances for each class as well as the posterior probability of class membership for each individual in the sample (Muthén & Muthén, 2008). These parameter estimates can be applied to subsequent samples to determine the probability of class membership for programs not involved in the original study (Vermunt & Magidson, 2004). LPA was applied to the OSC scores of 99 children’s mental health clinics from a national probability sample (Glisson et al., 2008). These analyses provided empirical support for a three class LPA solution that was applied to the programs in the present study. The three empirically derived classes from the national sample corresponded to “best,” “average,” and “worst” OSC profiles. The class with a “best” OSC profile had culture scores that were high on proficiency and low on rigidity and resistance, and climate scores that were high on engagement and functionality and low on stress. The class with an “average” OSC profile was near the national average on all six OSC dimensions. The class with a “worst” OSC profile had culture scores high on rigidity and resistance and low on proficiency and climate scores high on stress and low on functionality and engagement. The LPA parameters (means and variances) from the national sample were applied to the programs in the present study to determine the probability of class membership for each program in the present study. This analysis produced probabilities of class membership for each program in the study. Subsequently, a weighted class membership variable was derived for each program, calculated as the probability-weighted sum of class membership in the three classes. This weighted sum was used to characterize the OSC of each program in the study. Higher scores on the weighted sum represent greater likelihood of membership in the class with the “best” OSC profile.

Data Analyses

Basic descriptive statistics and summary scores of HCBS Waiver Programs’ performance on the two sets of QIs (Program and Individual FSS) were calculated for the three clusters of programs with “worst,” “average,” and “best” OSC LPA profiles. One-way ANOVAs were used to compare the three program clusters on their performance on Program and Individual FSS QIs. Where significant groups differences on QIs were detected, posthoc analyses using Tukey’s HSD were conducted for pairwise comparisons. Spearman’s rank coefficient correlation was used to examine the relationship between HCBS Waiver Program performance on Program and Individual FSS QIs and the 2 types of OSC data: OSC LPA clusters and the 6 OSC culture and climate dimensions. For all analyses, Individual FSS QIs were further examined by separating out Inappropriate FSS practices. These bivariate relationships were also analyzed, controlling for key program characteristics including number of staff, program capacity in terms of number of slots available, and annual budget.

Results

Performance on QIs

HCBS Waiver Programs were rated on 12 of the 14 program QIs (2 Program QIs were eliminated from the analyses because of excessive “Don’t Knows” indicating insufficient information to code). The mean program performance score on the 12 Program QIs was 19.8 (SD=4.8) out of a total score of 36 (range 13 to 28). On Individual FSS QIs, the mean performance score on all 27 QIs (n= 21, plus 6 inappropriate practice indicators that were reverse scored) was 39.2 (SD=9.3) out of a total score of 63 (range 21 to 54). Across the 21 Waiver Programs, the mean total Program and Individual FSS QIs was 58.9 (SD=12.3) out of a total score of 99 (range 35–77).

OSC LPA Profiles

The 21 HCBS Waiver Programs in the present study were classified according to the three LPA clusters of “worst,” ”average,” and “best” profiles: 9 worst, 5 average, and 7 best. In general, programs with the “best” OSC LPA profiles performed significantly better than programs with the “worst” OSC LPA profiles; programs within the “average” profile cluster did not differ from those in either the “best” or “worst” clusters on any of the QI sets.

Overall Program and FSS QIs

The three program clusters performed significantly differently on the overall 39 Program and Individual FSS QIs (F = 4.57, p = 0.025). Pairwise comparisons only differentiated the “best” and the “worst” clusters of programs (t = 2.93, p <0.05).

Program Level QIs

HCBS Waiver Program performance on the 12 Program QIs did not significantly differ across the three program clusters (F = 2.50, p = 0.11).

FSS Level QIs

On the 27 FSS QIs, the three program clusters performed significantly differently (F = 3.82, p = 0.041). Pairwise comparisons showed that programs with the “worst” profile performed significantly poorer on FSS QIs than those in the “best” profile (t = 2.74; p < 0.05). When the FSS QIs were separately analyzed according to the 6 Inappropriate practices and the 21 high quality FSS practices, these differences held up. Comparing the worst, average, and best profiles, the 3 program clusters scored significantly differently on the 6 QIs focused on inappropriate practices (F = 3.795, p =0.04); differences in performance on the other 21 high quality individual FSS QIs (F = 2.29, p = 0.056) approached significance. Pairwise comparisons again showed significant differences between the clusters of programs with the “best” and “worst” profiles on these 6 Inappropriate FSS practices (t = 2.71, p < 0.04).

Correlations between OSC and QIs

Significant correlations were found between OSC profiles and dimensions and performance on both Program and FSS QIs. The pattern of results remained the same even after controlling for key program characteristics (number of staff, slot size, annual budget); due to the very small sample size and the exploratory nature of this study, only simple bivariate correlations are reported here.

OSC LPA Profiles

The weighted OSC Profiles based on LPA clusters were correlated with both Program and Individual FSS QIs, with more positive program profiles showing better performance on the two QI sets. Negative program profiles showed higher scores on the FSS QIs related to Inappropriate Practices. These results were similar using both weighted and unweighted profile classifications, with stronger correlations in general with the weighted OSC LPA profiles. One exception was that the weighted OSC LPA profile correlation with the 6 Inappropriate Practices QIs approached, yet was not, significant.

OSC Dimensions

In general, Program QIs were significantly correlated with OSC Culture but less so with OSC Climate. FSS QIs were correlated with both culture and climate dimensions. Inappropriate FSS practices were significantly associated only with the Resistant Cultures.

OSC Culture Dimensions

Overall, performance on the two QIs sets was significantly correlated with OSC Culture dimensions. These QI sets were positively correlated with Proficiency and negatively correlated with both Rigidity and Resistance. Correlations between Individual FSS QIs representing Inappropriate Practices were significantly correlated only with Resistant cultures; although not significant, these Inappropriate FSS practices were associated with the other culture dimensions in the expected direction (positive correlations with Rigidity and negative correlation with Proficiency).

OSC Climate Dimensions

Overall, performance on the 2 QI sets was significantly correlated with OSC Climate Dimensions. Climate dimensions demonstrated stronger correlations with FSS QIs than Program QIs. Program QIs were significantly correlated (positively) with only one Climate Dimension: Engagement. The highest correlation was found between Individual FSS level QIs and Functionality (rho = .734, p < .001). While the performance on the 27 Individual FSS QIs were significantly correlated with all Climate dimensions, these appear to be driven by the Individual FSS practices that are considered good quality practices; inappropriate practices were not found to be significantly related to any of the Climate dimensions.

Discussion

To our knowledge, this is the first study to examine the relationship between the organizational social contexts of programs that employ FSSs and the quality of family support services. Given the cross-sectional nature of the data, it is not possible to infer causal relationships between QIs and organizational context. However, significant and interpretable correlations were found between organizational social context and Program and FSS staff QIs. This finding is consistent with prior studies that have linked organizational social context of human service organizations with service quality (e.g., Glisson et al., 2008b).

Organizational Profile Clusters

These 21 HCBS Waiver Programs could be clustered according to “best,” “average” and “worst” organizational profiles. These program clusters did not perform differently on Program QIs. However, distinct differences emerged for staff level QIs. “Best” program profiles were associated with higher quality Individual FSS practices, while “worst” program profiles were linked with poorer quality FSS practices, particularly with inappropriate FSS practices. These findings provide concurrent validity for QIs of family support services, and suggest the potential utility of these QIs in practice. In particular, these correlations suggest that organizational social context may be important for supporting good quality FSS practices; poor cultures and climates may foster inappropriate FSS practices that can be detrimental to client well-being.

In general, clustering of programs can be helpful in distinguishing high quality from low quality programs, thus providing a potentially useful tool to target resources for program improvement within a service system. However, these profile clusters by themselves are less helpful in identifying specific strategies for organizational improvements. Associations between QIs and specific OSC dimensions suggest that more refined targets for organizational improvement is possible, even among the “best” program clusters.

Organizational Social Contexts

Program QIs were generally more closely related to organizational culture than climate. Both organizational culture and program policies are typically driven and shaped by organizational leadership. Program policies around family support services are thus likely to be influenced by the same leaders who establish how things get done in their organizations (i.e., organizational culture). Organizational climate, on the other hand, represent staffs’ responses to leadership policies and behaviors. Depending on the staff make-up of the organization, their responses to the same program policies may differ depending on their own personal beliefs, attitudes and perceptions; their aggregate perceptions contribute to organizational climate. Variance in staff perceptions may explain the less robust relationship between Program QIs and organizational climate. On the other hand, FSS QIs were both highly associated with organizational cultures and climates, suggesting a stronger influence of organizational context, or perhaps a greater vulnerability of FSS practices to their work environments, particularly organizational culture. That is, regardless of individual attitudes and beliefs, FSS practices are more likely to reflect the larger organizational context. In this study, such may be the case because program directors selected one FSS to participate in the standardized walkthrough; their choice is likely influenced by the degree to which the FSS aligned with program culture. In the next phase of this research, FSS QIs will be based on walkthroughs of all FSS in the program.

Organizational Culture Dimensions

HCBS Waiver Programs with highly proficient cultures are more likely to score high on quality program policies and to have FSS who demonstrate higher quality practices. In these highly proficient programs, policies and procedures that support the work of FSS are likely to be in place, such as standards to guide practice, supervision, and consultation; collaborative structures developed to help FSSs be more effective on teams; and program flexibility to allow integration of innovative ideas or different perspectives about families brought to the team by FSSs. By contrast, those programs with highly rigid and resistant cultures are more likely to be associated with poorer performance on program policies, where staff are bound by bureaucratic rules and regulations that hamper flexible service delivery, and more likely to resist new or alternative processes or perspectives that might be introduced by FSSs to more effectively meet the individual needs of youth and families served.

Notably, Inappropriate FSS practices were significantly associated with highly resistant organizational cultures, where staff are expected to resist new ways of providing services through suppression or apathy. This finding is significant because an important role of FSSs who work on teams is to promote family empowerment and to facilitate working relationships between families and other providers on the team. Many families served within HCBS Waiver Programs have complex situations and they are often perceived as difficult to engage or non-compliant. FSSs play a key role on provider teams to bring a different lens through which to view families, as well as to facilitate the development of service options that better meet the needs and preferences of families. It is possible that FSSs who work in programs that are resistant to new ways of working with families may feel a need to help a family work around the rest of the provider team, or may feel co-opted by the rest of the team to engage in entrenched practices that tend to be paternalistic and directive (expert knows best) or practices that pathologize or blame the caregiver.

Organizational Climate Dimensions

HCBS Waiver programs that had higher quality program and FSS staff practices were more likely to have staff that perceived positive work environments or climates. In particular, programs with high performance on Program QIs had staff that reported being more engaged, who feel a sense of personal accomplishment and have meaningful connections to their clients. Notably, Inappropriate FSS practices were not associated with any of the climate dimensions. By contrast, high quality FSS practices were found to be significantly associated with all climate dimensions, suggesting that organizational climates where staff feel a personal sense of accomplishment and have meaningful client connections tend to promote high quality FSS practices. High quality FSS practices were also associated with work environments that have lower stress and staff who are less likely to be emotionally exhausted, conflicted about their roles and responsibilities or be overloaded. In particular, high quality FSS practices showed the highest associations with functional climates where staff perceive high levels of cooperation, have clear expectations associated with their roles and responsibilities, and clear procedures in place to enhance job successes. In such highly functional work environments, individual FSS practices are most likely to be supported by collaborative team members who work towards a shared purpose, are clear about FSS roles, and value the perspective FSSs bring to create more effective service plans for the families they serve.

Implications for Integration of Family Support Services in Healthcare Reform

The relationship between HCBS Waiver Program’s organizational social context and their performance on quality benchmarks that support family support services is highly encouraging for many reasons. First, this relationship provides initial validation of this newly developed set of QIs for family support practices. The ultimate value and utility of these QIs, however, hinges on verification that these QIs are related to important outcomes of interest, such as parent and child outcomes. This is our focus in the next phase of this research. Second, the relationship between organizational social context and QIs suggests multiple organizational culture and climate dimensions that may be malleable and can be targeted to improve parent-delivered family support services. Third, addressing a program’s organizational social context has the potential to improve not only program level policies around family support services, but may likely influence the effectiveness of individual FSS practices. Thus, interventions at the organizational level could have synergistic effects for both staff and clients, improving the work environment for service providers as well as service quality for clients. Most importantly, improving a program’s organizational social context holds promise for impacting actual outcomes for families and youth in these programs.

There are several implications for research directions. One next step is to test a tailored and more targeted organizational intervention to address specific program barriers to the effective integration of parent-delivered family support services within HCBS Waiver programs. A range of organizational studies that focus on contextual common factors (culture and climate) have shown that such factors can be improved within community-based service systems to support service delivery (Carr et al., 2003; Glisson et al., 2006; Glisson et al., 2010; Glisson et al., 2012; Glisson et al., 2013; Guzzo et al., 1985; Neuman et al., 1989; Parker et al., 2003; Robertson et al., 1993). This exploratory study also established strong a strong relationship between QIs and a well-validated measure (the OSC) with proven relationships with service quality and outcomes. While the correlational nature of this data precludes determination of causality, this relationship suggests that using the QIs as a tool for assessing change may help focus organizational intervention strategies to expedite quality FSS service delivery and outcomes. Given the significant challenges associated with the development of mental health quality measures, such an effort is in line with recent calls for the field of mental health to keep pace with health care improvement efforts nationally (IOM, 2006; Kilbourne, Keyser, & Pincus, 2010; Pincus et al, 2007).

Table 2.

Performance on QIs across LPA Clusters of OSC Profiles

OSC LPA Profiles	Program Level QIs (n=12)*	Individual FSS Level QIs (n=27)		Sum of 27 Individual FSS Level QIs	Total Prog Level QI and Individual Level QIs
		Inappropriate QIs (n = 6)	Ind. Level QIs (n=21)
Worst (n = 9)	17.22	−1.56	35.44	33.89	51.11
Mean (SD)	(4.32)	(1.59)	(9.13)	(10.29)	(12.98)
Average (n = 5)	21.60	−.60	41.00	40.40	62.00
Mean (SD)	(3.58)	(.89)	(7.07)	(7.34)	(7.87)
Best (n = 7)	21.71	.00	45.29	45.29	66.71
Mean (SD)	(5.25)	(.000)	(5.22)	(5.22)	(8.34)
Total (N = 21)	19.76	−.81	40.05	39.24	58.90
Mean (SD)	(4.84)	(1.29)	(8.42)	(9.34)	(12.32)

^*Excluded 2 QIs (items 1,13) from the program level because of excessive DKs (unable to code these or 75% of programs.); Program QI scores are based on 12 indicators

Table 3.

Spearman Correlations between OSC (Profiles and Dimensions) and Program Level and Individual Level QI

	Program Level QIs (12 items+)	Individual Level QIs (27 items)		Sum of 27 Individual FPA Level QIs	Total Level QI and Individual Level QIs
		Inappr. QIs (6 items)	Ind. Level QIs (n=21 items)
OSC LPA Profiles- Weighted (N=21)
Spearman’s rho	.519	−.406	.591	.583	.607
Sig. (2 tailed)	.016	.068	.005	.005	.004
OSC Dimensions (N = 21)
Culture:
Proficiency
Spearman’s rho	.440	−.120	.576	.565
Sig. (2 tailed)	.046	.604	.006	.008	.010
Rigidity
Spearman’s rho	−.465	.284	−.483	−.469	−.511
Sig. (2 tailed)	.034	.213	.026	.032	.018
Resistance
Spearman’s rho	−.489	.459	−.544	−.536	−.557
Sig. (2 tailed)	.025	.036	.011	.012	.009
Climate:
Engagement
Spearman’s rho	.450	−.197	.441	.433	.481
Sig. (2 tailed)	.041	.437	.046	.050	.027
Functionality
Spearman’s rho	.416	−.347	.738	.734	.684
Sig. (2 tailed)	.061	.123	.000	.000	.001
Stress
Spearman’s rho	−.352	.401	−.446	−.442	−.467
Sig. (2 tailed)	.118	.072	.043	.045	.033

⁺Excluded 2 QIs (items 1,13) from the program level because of excessive DKs (unable to code these or 75% of programs.); Program QI scores are based on 12 indicators